[ML][AIOps] Telemetry: track analysis endpoint usage #166988

alvarezmelissa87 · 2023-09-21T19:38:59Z

Summary

This PR adds tracking for Log Rate Analysis and Log Pattern Analysis endpoints for AIOps.

tracks type of analysis and source (where the analysis is being run from)

elasticmachine · 2023-09-21T19:39:03Z

Pinging @elastic/ml-ui (:ml)

… routes

…fix'

alvarezmelissa87 · 2023-09-21T20:20:30Z

cc @benakansara for the infra plugin addition.

.../public/components/log_rate_analysis/log_rate_analysis_content/log_rate_analysis_content.tsx

x-pack/plugins/aiops/public/components/log_rate_analysis/log_rate_analysis_results.tsx

peteharverson · 2023-09-22T16:53:04Z

...alerting/log_threshold/components/alert_details_app_section/components/log_rate_analysis.tsx

@@ -268,6 +268,7 @@ export const LogRateAnalysis: FC<AlertDetailsLogRateAnalysisSectionProps> = ({ r
        </EuiFlexItem>
        <EuiFlexItem>
          <LogRateAnalysisContent
+            source="observability_alerts"


Maybe observability_alert_details or log_threshold_alert_details ?

Updated in 8c6bc9d

walterra

We should make the header name less generic than just source. Please have a look how Kibana structures other header names like Kbn-Build-Number, Kbn-System-Request, Kbn-Version etc.

@walterra - good call. Updated to aiops-analysis-run-origin in bc27b46

walterra · 2023-09-25T11:37:40Z

x-pack/plugins/aiops/common/constants.ts

+export const LOG_RATE_ANALYSIS_RUN = 'aiops_log_rate_analysis_run';
+
+export const LOG_PATTERN_ANALYSIS_RUN = 'aiops_log_pattern_analysis_run';
+
+export const CHANGE_POINT_DETECTION_RUN = 'aiops_change_point_detection_run';
+
+export const AIOPS_DEFAULT_SOURCE = 'ml_aiops_labs';


Suggest to make these single consts an enum like object with as const, something like:

export const AIOPS_TELEMETRY_ID = { LOG_RATE_ANALYSIS_RUN: ..., LOG_PATTERN_ANALYSIS_RUN: ... } as const;

Updated in bc27b46

walterra · 2023-09-25T11:38:54Z

.../public/components/log_rate_analysis/log_rate_analysis_content/log_rate_analysis_content.tsx

@@ -64,6 +65,8 @@ export interface LogRateAnalysisContentProps {
  barHighlightColorOverride?: string;
  /** Optional callback that exposes data of the completed analysis */
  onAnalysisCompleted?: (d: LogRateAnalysisResultsData) => void;
+  /** Optional identifier to indicate the plugin utilizing the component */
+  source?: string;


I wonder if we should make this non-optional to enforce correct telemetry on new/future embeddings?

Good call - updated in bc27b46

fkanout · 2023-09-25T12:34:23Z

...alerting/log_threshold/components/alert_details_app_section/components/log_rate_analysis.tsx

@@ -268,6 +268,7 @@ export const LogRateAnalysis: FC<AlertDetailsLogRateAnalysisSectionProps> = ({ r
        </EuiFlexItem>
        <EuiFlexItem>
          <LogRateAnalysisContent
+            source="observability_alert_details"


We have other alert details pages for other rule types. Could we add a prefix/suffix to specify this is related to Log threshold?

Added in bc27b46

alvarezmelissa87 · 2023-09-25T18:55:47Z

@elasticmachine merge upstream

alvarezmelissa87 · 2023-09-25T22:18:59Z

This has been updated and is ready for another look when you get a chance 🙏 cc @walterra, @peteharverson, @fkanout

walterra · 2023-09-26T07:04:52Z

x-pack/plugins/aiops/common/constants.ts

+  CHANGE_POINT_DETECTION_RUN: 'aiops_change_point_detection_run',
+  AIOPS_DEFAULT_SOURCE: 'ml_aiops_labs',
+  AIOPS_ANALYSIS_RUN_ORIGIN: 'aiops-analysis-run-origin',
+} as const;


Sorry for the continued nitpicking 😅 — looking at the attributes now they serve different purposes so we probably shouldn't combine them all into one enum. Suggest to break out the run ones.

If I read the code right AIOPS_ANALYSIS_RUN_ORIGIN is now the header name? Suggest to prefix with kbn- then. We now also have a mix of source and origin in the naming. Related to that, if you align this, maybe we can also make the prop name source more specific, what do you think?

The header name can't begin with 'kbn' - I got an error. But good call on renaming the prop to match up.

Renamed prop name to 'embeddingOrigin' in c973e76

fkanout

AO changes LGTM.

peteharverson · 2023-09-26T13:39:26Z

x-pack/plugins/aiops/server/routes/log_categorization.ts

@@ -32,6 +36,13 @@ export const defineLogCategorizationRoutes = (
        },
      },
      async (context, request, response) => {
+        const { headers } = request;
+        trackAIOpsRouteUsage(


@jgowdyelastic is this in the right place? We want to track the number of times the analysis is run from the ML AIOps Labs Log Pattern Analysis page, or from within Discover. Currently I don't see this counter incrementing when I run the analysis from either page.

This endpoint will be called whenever the analysis is run, but it also has the risk that it could be called without the categorization analysis being run.

Debugging this, it appears there is a bug. The AIOPS_TELEMETRY_ID.AIOPS_ANALYSIS_RUN_ORIGIN header is not being added to the request and so the telemetry is not being counted.

One alternative would be to also implement client side telemetry to count this on click of the analyse button or in the case of the Discover embedded version, on render of the component.

Fixed categorization tracking in c973e76

walterra · 2023-09-27T11:39:40Z

Did some more testing and was able to generate the following data with the latest state:

                {
                  "domainId": "aiops",
                  "counterName": "POST /internal/aiops/log_rate_analysis",
                  "counterType": "run_via_observability_log_threshold_alert_details",
                  "lastUpdatedAt": "2023-09-27T11:31:13.882Z",
                  "fromTimestamp": "2023-09-27T00:00:00Z",
                  "total": 1
                },
               {
                  "domainId": "aiops",
                  "counterName": "POST /internal/aiops/log_rate_analysis",
                  "counterType": "run_via_ml_aiops_labs",
                  "lastUpdatedAt": "2023-09-27T11:11:53.590Z",
                  "fromTimestamp": "2023-09-27T00:00:00Z",
                  "total": 2
                },
                {
                  "domainId": "aiops",
                  "counterName": "POST /internal/aiops/categorization_field_validation",
                  "counterType": "run_via_ml_aiops_labs",
                  "lastUpdatedAt": "2023-09-27T10:59:08.426Z",
                  "fromTimestamp": "2023-09-27T00:00:00Z",
                  "total": 1
                },

Looks good IMHO!

When I run log pattern analysis via Discover it still tracks it as run_via_ml_aiops_labs though.

walterra

The latest state correctly tracks usage of log pattern analysis across AIOps Labs/Discover:

                {
                  "domainId": "aiops",
                  "counterName": "POST /internal/aiops/categorization_field_validation",
                  "counterType": "run_via_discover_run_pattern_analysis",
                  "lastUpdatedAt": "2023-09-27T17:16:35.721Z",
                  "fromTimestamp": "2023-09-27T00:00:00Z",
                  "total": 1
                },
                {
                  "domainId": "aiops",
                  "counterName": "POST /internal/aiops/categorization_field_validation",
                  "counterType": "run_via_ml_aiops_labs",
                  "lastUpdatedAt": "2023-09-27T17:16:15.710Z",
                  "fromTimestamp": "2023-09-27T00:00:00Z",
                  "total": 4
                },

Just a note: I think it's good enough for now, but should the log pattern analysis flyout ever be used in multiple places we need to make sure that the origin gets passed on from the place where it is embedded from. The way the code is structured now, it would report any flyout usage as a trigger via Discover.

jgowdyelastic · 2023-09-27T18:03:54Z

x-pack/plugins/aiops/public/components/log_categorization/show_flyout.tsx

@@ -70,6 +70,7 @@ export async function showCategorizeFlyout(
                    savedSearch={null}
                    selectedField={field}
                    onClose={onFlyoutClose}
+                    embeddingOrigin={'discover_run_pattern_analysis'}


Unfortunately I don't think hardcoding the ID here is sufficient. This flyout could be (in the future) opened from other places in kibana.
To get the real origin of the action, you'll need to have the value passed into this component as a prop and set by the ui action that has trigged it.
The context that is passed to the action's execute function does have an originatingApp variable which I added just in case when this was first written.
This has the value discover when called from Discover.

benakansara

LGTM. Tested analysis endpoint with Log threshold alert details page.

peteharverson

Tested and LGTM

jgowdyelastic

LGTM

alvarezmelissa87 · 2023-09-28T21:44:26Z

@elasticmachine merge upstream

kibana-ci · 2023-09-28T22:48:41Z

💚 Build Succeeded

Buildkite Build
Commit: a0bc3c3

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`aiops`	594.8KB	595.3KB	+458.0B
`infra`	1.9MB	1.9MB	+60.0B
total			+518.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`aiops`	7.9KB	8.0KB	+129.0B

Unknown metric groups

API count

id	before	after	diff
`aiops`	66	67	+1

History

💔 Build #163555 failed e0db511
💚 Build #163182 succeeded d5584cb
💛 Build #163086 was flaky b92d018
💚 Build #163042 succeeded 4bfd9cc
💛 Build #162681 was flaky 82bc817

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @alvarezmelissa87

alvarezmelissa87 added :ml release_note:skip Skip the PR/issue when compiling release notes Feature:ML/AIOps ML AIOps features: Change Point Detection, Log Pattern Analysis, Log Rate Analysis v8.11.0 labels Sep 21, 2023

alvarezmelissa87 requested review from walterra and peteharverson September 21, 2023 19:38

alvarezmelissa87 self-assigned this Sep 21, 2023

alvarezmelissa87 requested review from a team as code owners September 21, 2023 19:39

adds telemetry tracking to log rate analysis and log pattern analysis…

a71ecb5

… routes

alvarezmelissa87 force-pushed the ml-aiops-add-telemetry branch from 55a7ce3 to a71ecb5 Compare September 21, 2023 19:58

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

0569e34

…fix'

peteharverson reviewed Sep 22, 2023

View reviewed changes

.../public/components/log_rate_analysis/log_rate_analysis_content/log_rate_analysis_content.tsx Outdated Show resolved Hide resolved

x-pack/plugins/aiops/public/components/log_rate_analysis/log_rate_analysis_results.tsx Outdated Show resolved Hide resolved

peteharverson reviewed Sep 22, 2023

View reviewed changes

alvarezmelissa87 added 2 commits September 22, 2023 12:37

rename counter type constants

67fc696

add test for tracking route usage

8c6bc9d

walterra reviewed Sep 25, 2023

View reviewed changes

fkanout reviewed Sep 25, 2023

View reviewed changes

kibanamachine and others added 2 commits September 25, 2023 14:55

Merge branch 'main' into ml-aiops-add-telemetry

77af3c8

ensure usageCollection is required

bc27b46

walterra reviewed Sep 26, 2023

View reviewed changes

fkanout approved these changes Sep 26, 2023

View reviewed changes

peteharverson reviewed Sep 26, 2023

View reviewed changes

alvarezmelissa87 added 2 commits September 26, 2023 10:59

update telemetry with path

c973e76

update jest test for latest changes

82bc817

peteharverson mentioned this pull request Sep 27, 2023

[ML] Increase Test Coverage 8.11.0 #164562

Closed

10 tasks

alvarezmelissa87 added 2 commits September 27, 2023 09:59

Merge branch 'main' into ml-aiops-add-telemetry

4bfd9cc

update discover flyout origin prop

b92d018

walterra approved these changes Sep 27, 2023

View reviewed changes

jgowdyelastic reviewed Sep 27, 2023

View reviewed changes

pass in origin plugin for categorization flyout instead of hard coding

d5584cb

alvarezmelissa87 requested a review from a team as a code owner September 27, 2023 20:48

benakansara approved these changes Sep 28, 2023

View reviewed changes

Merge branch 'main' into ml-aiops-add-telemetry

e0db511

peteharverson approved these changes Sep 28, 2023

View reviewed changes

jgowdyelastic approved these changes Sep 28, 2023

View reviewed changes

Merge branch 'main' into ml-aiops-add-telemetry

a0bc3c3

peteharverson mentioned this pull request Sep 29, 2023

[ML] AIOps Log Rate Analysis: 8.11 improvements. #163690

Closed

rshen91 approved these changes Sep 29, 2023

View reviewed changes

alvarezmelissa87 merged commit 0bdbcc0 into elastic:main Sep 29, 2023

kibanamachine added the backport:skip This commit does not require backporting label Sep 29, 2023

alvarezmelissa87 deleted the ml-aiops-add-telemetry branch September 29, 2023 16:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ML][AIOps] Telemetry: track analysis endpoint usage #166988

[ML][AIOps] Telemetry: track analysis endpoint usage #166988

alvarezmelissa87 commented Sep 21, 2023 •

edited by peteharverson

Loading

elasticmachine commented Sep 21, 2023

alvarezmelissa87 commented Sep 21, 2023

peteharverson Sep 22, 2023

alvarezmelissa87 Sep 22, 2023

walterra left a comment •

edited by alvarezmelissa87

Loading

walterra Sep 25, 2023

alvarezmelissa87 Sep 25, 2023

walterra Sep 25, 2023

alvarezmelissa87 Sep 25, 2023

fkanout Sep 25, 2023

alvarezmelissa87 Sep 25, 2023

alvarezmelissa87 commented Sep 25, 2023

alvarezmelissa87 commented Sep 25, 2023

walterra Sep 26, 2023

alvarezmelissa87 Sep 26, 2023

alvarezmelissa87 Sep 26, 2023

fkanout left a comment

peteharverson Sep 26, 2023 •

edited

Loading

jgowdyelastic Sep 26, 2023 •

edited

Loading

jgowdyelastic Sep 26, 2023

alvarezmelissa87 Sep 26, 2023

walterra commented Sep 27, 2023

walterra left a comment

jgowdyelastic Sep 27, 2023

benakansara left a comment

peteharverson left a comment

jgowdyelastic left a comment

alvarezmelissa87 commented Sep 28, 2023

kibana-ci commented Sep 28, 2023

API count

[ML][AIOps] Telemetry: track analysis endpoint usage #166988

[ML][AIOps] Telemetry: track analysis endpoint usage #166988

Conversation

alvarezmelissa87 commented Sep 21, 2023 • edited by peteharverson Loading

Summary

elasticmachine commented Sep 21, 2023

alvarezmelissa87 commented Sep 21, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

walterra left a comment • edited by alvarezmelissa87 Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alvarezmelissa87 commented Sep 25, 2023

alvarezmelissa87 commented Sep 25, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fkanout left a comment

Choose a reason for hiding this comment

peteharverson Sep 26, 2023 • edited Loading

Choose a reason for hiding this comment

jgowdyelastic Sep 26, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

walterra commented Sep 27, 2023

walterra left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benakansara left a comment

Choose a reason for hiding this comment

peteharverson left a comment

Choose a reason for hiding this comment

jgowdyelastic left a comment

Choose a reason for hiding this comment

alvarezmelissa87 commented Sep 28, 2023

kibana-ci commented Sep 28, 2023

💚 Build Succeeded

Metrics [docs]

Async chunks

Page load bundle

API count

History

alvarezmelissa87 commented Sep 21, 2023 •

edited by peteharverson

Loading

walterra left a comment •

edited by alvarezmelissa87

Loading

peteharverson Sep 26, 2023 •

edited

Loading

jgowdyelastic Sep 26, 2023 •

edited

Loading