Epic: establish a centralized observability environment

[sduranc]

• Currently we have grafana per environment/(service)owner. This is not optimal when it comes to on-call or just having a good overview of all the environments. Therefore we need to gather observability signals in one place.
• The initial idea is to rely on managed solutions but we should evaluate the feasibility of this
• This epic will have as an outcome a proposed solution, along with an RFC and a PoC.

Tasks

Preview Give feedback

1. Create centralized grafana #908
   3 of 3
   PartOfEpic area/observability kind/user-story +3
   
2. Create centralized prometheus #909
   2 of 2
   PartOfEpic area/observability kind/user-story +3
   
3. Create centralized tracing solution #910
   area/observability +1
4. Define the agent that exports obs signals to the centralized obs env #911
   4 of 4
   area/observability +1
   
5. Dashboards should be reconciled from one source of truth to all grafanas #912
   PartOfEpic area/observability +2
   
6. Define an alert manager solution for predefined alerts (slos etc) #913
   area/observability +1
   
7. Filter out metrics #917
   area/observability +1
8. Enable traces from CI/CD workflows for all teams/services #919
   area/observability +1
   
9. Deploy Pyrra in the apps clusters #995
   PartOfEpic area/observability +2
   
10. Deploy Traefik SLO similar to the one defined in the platform clusters #996
    PartOfEpic area/observability +2
    
11. Create a Dashboard to visualize the Traefik SLO from the various environments #998
    PartOfEpic area/observability +2
12. Send selected metrics from platform clusters to the centralized azure monitor #1000
    PartOfEpic area/observability +2
    
13. Add annotations in grafana for each deployment #1083
    +0
    
14. Find and alternative way to generate metrics from nginx logs #1084
    +0
    
15. Re-create the DCR and DCRA in the TT02 environments #1120
    PartOfEpic area/observability +2