[Security Solution] Performance measurement and monitoring with Elastic APM

[banderror]

Parent ticket: https://github.com/elastic/security-team/issues/3550 (internal)

Summary

Background: Since 8.1.1, the application has become hard to use due to client-side and server-side performance issues. A series of PRs were merged to address these issues, but there are still some problems.

We need to be able to profile the client- and server-side performance to be able to identify performance issues. One of the tools for that is Elastic's own APM.

Sub-tasks

Tasks

Preview Give feedback

1. [Security Solution] Instrument Security frontend code to collect performance data #129324
   8.3 candidate Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp performance v8.3.0 +6
   
2. Enable APM instrumentation for some of the Elastic's internal clusters. Start collecting APM data. Analyze gathered APM data to identify potential performance bottlenecks.
3. Enable APM instrumentation for Cypress tests locally. Cypress picks up the Kibana dev config. So if APM is configured, the performance measurements will be collected during test runs.
4. [Security Solution] Rules management performance and scalability issues #134826
   8.8 candidate Feature:ML Rule Feature:Prebuilt Detection Rules Feature:Rule Management Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp performance v8.4.0 v8.8.0 +10
   
5. [Security Solution] Document APM setup and instrumentation #135193
   Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp performance +4
   
6. Address comments in #133486
7. Implement a generic solution to execution context propagation inside Kibana #132629
   Team: SecuritySolution Team:SharedUX performance technical debt +4
   
8. Set up alerts to get notifications when performance falls below a certain threshold
9. Introduce data sets for local development with enough data for performance bottlenecks to show up
10. Enable APM instrumentation for Cypress tests on CI
11. Explore possibilities of automated performance regression testing

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)