[Security Solution] Alerts Treemap and Multi-Dimensional Alert Grouping

[andrew-goldstein]

[Security Solution] Alerts Treemap and Multi-Dimensional Alert Grouping

This PR introduces the new Treemap and Multi-Dimensional Alert Grouping to the Alerts page.

The initial commit was developed as an ON week proof of concept (POC). It has since been updated to incorporate product and UX feedback.

Alerts treemap

The new Alerts page treemap is shown in the screenshot below:

Above: The new treemap in the Alerts page

• Alerts are colored by risk score
• Clicking on a cell instantly filters the Alerts page
• Treemap legend items may be added to filters and Timeline investigations
• The new treemap supports multi-dimensional grouping and filtering
  • Alerts are grouped by kibana.alert.rule.name and host.name by default

Multi-Dimensional Alert Grouping

The table on the Alerts page, which previously supported grouping by a single field, has also been enhanced to support multi-dimensional grouping, per the screenshot below:

Above: The table in the Alerts page, enhanced to support multi-dimensional grouping

Filtering the Alerts page by risk score

Every rule, including prebuilt Elastic rules and custom rules created by users, must specify a risk score at rule creation time, per the screenshot below:

Above: Every rule has a risk score specified when it's created

The colors of the alerts displayed in the treemap are determined by the rule's risk score. This makes it easy to quickly filter the entire alerts page by clicking on the riskiest alerts.

Clicking on a cell in the treemap adds two filters, one for each group by field, per the screenshot below:

Above: Two filters, (one for each group by field), are added to the page when a cell is clicked

The Alerts page updates instantly when filters are added or removed. In the screenshot below, the 2nd filter was removed to filter the page to all mimikatz process started alerts:

Above: Removing the 2nd filter, a specific host.name, revealed all the hosts in the mimikatz process started alerts

Switching views

Users may switch between the following views:

• Table
• Trend (default)
• Treemap

per the screenshot below:

Above: View selection

The default Trend view is shown in the screenshot below:

Above: The (default) Trend view

• The Trend chart's legend has been enhanced to display counts, per the design detailed in issue [Security Solution] Suggestion. Collapse Alerting graphs into one view #120282

• The Trend view only supports visualizing a single dimension. Hovering over the disabled Group by top select in the Trend view displays the tooltip shown in the screenshot below:

Above: The Trend view only supports visualizing a single dimension

Collapsing the panel

The panel may optionally be collapsed to save space, per the screenshot below:

Above: The panel (optionally) collapsed

User preferences are persisted to local storage

Previously, the Group by selections on the Alerts page were always forgotten when users navigated away from the Alerts page. As a result, users had to re-select their preferred Group by fields every time they visited the page.

We now store all of the following preferences in local storage:

• View selection (Table, Trend, Treemap)
• Group by selections
• Panel collapse state

The preferences above are now restored the next time users return to the Alerts page.

Group by field selection is synchronized between views

Group by field selection is synchronized between views. For example, if a user changes the Group by fields in the Treemap view and then switches to the Table view, the same Group by fields will be displayed in the Table view.

Resetting Group by fields to their defaults

Users may reset the Group by fields to their defaults (kibana.alert.rule.name and host.name) for any visualization via the menu shown in the screenshot below:

Above: Resetting Group by fields to defaults via the menu

[andrew-goldstein]

This draft PR is ready for discussion, but not code review

[andrew-goldstein]

Archived: Original POC design

For reference, this comment archives the original ON week POC design from the initial commit, when the PR was opened as a [DRAFT]:

[Security Solution] Alerts by risk score treemap

This PR introduces the Alerts by risk score treemap to the Alerts page:

Above: The new Alerts by risk score treemap

• Alerts are colored by risk score
• Clicking on a cell instantly filters the alerts page
• Users may specify custom fields to group by, or quick-select from our curated sets of fields that pair well together

Feature flag

The features in this PR are enabled via the following feature flag in experimental_features.ts:

  alertsTreemapEnabled: true,

When the feature flag is disabled, the alerts page looks like main today:

Above: This branch, when the feature flag is disabled

Let's take an (EUI) tour

When the feature flag is enabled, the layout of the Alerts page incorporates the design detailed in #120287, which eliminates unusable space at the top of the Alerts page, and enables users to quickly hide and show the charts.

Hiding and showing charts

Users are introduced to the new Hide charts feature of the layout via Step 1 of an EUI Tour:

Above: Step 1 of the tour introduces the Hide charts feature

The tour is interactive

Users may (optionally) try the Hide charts feature as part of the tour, per the video below:

hide_show_chart.mov

Above: (video) The interactive tour let's users experience the new Hide charts feature

Switching views

Step 2, the final step of the tour, introduces the view selector:

Above: Step 2 of the tour introduces the view selector

Per the screenshot and video below, view selection toggles between the default Trend view and the new Alerts by risk score view:

Above: View selection

view_selection.mov

Above: (video) View selection toggles between the Trend view and the Alerts by risk score view

Preferences are persisted to local storage

We store the user's chart visibility and view preferences in local storage.

When users navigate to other views in Kibana or close the browser, their chart visibility and view preferences are restored the next time they visit the Alerts page.

Filtering the Alerts page by risk score

Every rule, including prebuilt Elastic rules and custom rules created by users, must specify a risk score at rule creation time, per the screenshot below:

Above: Every rule has a risk score specified when it's created

The colors of the alerts displayed in the treemap are determined by the rule's risk score.

This makes it easy to quickly filter the entire alerts page by clicking on the riskiest alerts, per the video below:

filtering_via_the_treemap.mov

Above: (video) Users click on the riskiest alerts to filter the page

Clicking on a cell in the treemap adds two filters, one for each group by field:

Above: Two filters, (one for each group by field), are added to the page when a cell is clicked

In the screenshot below, the 2nd filter was removed to reveal all the hosts in mimikatz process started alerts:

Above: Removing the 2nd filter, host.name, revealed all the hosts in the mimikatz process started alerts

Instant insights

We can instantly answer questions like "Did this host trigger alerts for other rules?", and at a glance, grok the severity of those alerts, by removing the first of the two filters:

Above: Removing the first filter, kibana.alert.rule.name, revealed other alerts (from different rules) triggered for a specific host

Custom group by criteria

Users may customize the group by criteria:

custom_group_by.mov

Above: (video) Entering custom group by criteria

Persistance via local storage

When users navigate to other views in Kibana or close the browser, the previously-selected group by fields are restored from local storage the next time they visit the Alerts page.

Restoring defaults

Clicking on the first menu item in the settings gear, Reset group by fields, restores the default group by fields, per the video below:

restoring_default_groups.mov

Above: (video) Restoring the default group by fields

Quick-select from Elastic-curated group by criteria

Users may quick-select from Elastic-curated sets of fields that pair well together, like parent and child processes, or process and file name:

Above: Quick-select from Elastic-curated sets of fiels

A Multi-dimensional Count table

Previous versions of the Count table were limited to grouping by one field. This PR enhances it to support grouping by two fields, per the screenshot below:

Above: The Count table now supports two group by fields

Quickly toggle between table and tree views

The group by criteria of the Count and Alerts by risk score widgets are linked together, making it easy to toggle between the two views, per the video below:

linked_group_by_criteria.mov

Above: (video) The Count and Alerts by risk score group by criteria are linked

Trend chart legend enhancements

The Trend chart's legend has been enhanced to display counts, per the design detailed in issue #120282

Above: The Trend chart legend, enhanced to display counts

Note: The original intention of adding counts to the Trend chart legend, as documented in #120282, was to combine the Trend and Count widgets into a single view via the legend.

In this PR, the Trend and Count widget queries have diverged; (the latter now supports multiple group by fields). This prevents the Trend and Count widgets from being combined into a single view.

Trend chart field selection is persisted to local storage

This PR adds persistence of the Trend chart Stack by field to local storage.

When users navigate to other views in Kibana or close the browser, the previously-selected field is restored the next time they visit the Alerts page.

[andrew-goldstein]

Archived: Recommended groups POC

In this release, users must manually select Group by fields that pair well together, e.g. parent and child processes, or process name and file name.

For reference, this comment archives the Recommended groups POC, where we may proffer curated groups that pair well together, demonstrated in the video below:

recommended_groups.mov

Above: A video demonstrating the Recommended groups (POC)

[andrew-goldstein]

Archived: Color palette selection POC

In this release, the colors of the alerts displayed in the treemap are determined by the rule's risk score.

For reference, this comment archives the Color palette selection POC, demonstrated in the video below:

color_palettes.mov

Above: A video demonstrating the Color palette selection (POC)

[michaelolo24]

Can you add a comment just detailing why the values are these. We may want to reference this for the docs team as well

[andrew-goldstein]

Can you add a comment just detailing why the values are these. We may want to reference this for the docs team as well

😂 I didn't define or change these values, so I can't speak to, or document why for example low is 21.

This PR merely takes the previous definition of these values:

export const defaultRiskScoreBySeverity: Record<Severity, number> = {
  low: 21,
  medium: 47,
  high: 73,
  critical: 99,
};

and exposes them as constants for use in other files.

[michaelolo24]

Yea, I'd say let's default plugin APP_ID in the actual useLocalStorage. It can be in a follow up PR for sure if it'll be too much of a headache here with the tests 😂

[andrew-goldstein]

Yea, I'd say let's default plugin APP_ID in the actual useLocalStorage. It can be in a follow up PR for sure if it'll be too much of a headache here with the tests 😂

done! 😄

[spong]

Checked out, tested locally, and pair reviewed with @andrew-goldstein. Thank you for taking the time to go over everything here @andrew-goldstein, and for all your efforts in getting this amazing new alert triaging component in place! 🎉 This is going to greatly impact analysts alert triage experience, and is a step forward in optimizing and enhancing their workflow!

Overall everything in testing and review LGTM, so approving as-is. In testing we did come across a few items, so I'll note them below, but no need to address these as part of this PR, so feel free to either pickup in a follow-up, or create an issue for tracking/discussion as you see fit.

Welcome TreeMap! 🙌 LGTM! 🌲 🗺️ 🚀

Notes

Tree map isn't displayed for certain `group by top` fields, like `Ransomeware.version`, or `tags`

• Consider a 'empty state' message similar to No data to display when both group by fields aren't supplied.

Extra left margins on `Table` view

Consider not refreshing/clearing data when deleting `group by` until focus is lost (or another selection is complete)

• With regards to the group by selection UX, consider not updating data until a new field is selected, or focus is lost when empty (to prevent an unnecessary refresh when selecting a new field):

Allow `enter` to select initial item in group by components

• When typing a field name we require an extra down-arrow + enter instead of just enter to confirm the first match (even if exact):

Consider mirroring same table/trend/tree component on Rule Details

• I've always kinda paired the visualization above the Alerts table with the table itself, so it was interesting to see that we kept just the trend visualization on the Rule Details page. I think it would be an added benefit to mirror this new visualization UI on the Rule Details as well, so users have access to the treemap for additional grouping at the Rule layer. Would likely want it to have their own localstorage values and default group by's.

[spong]

Side note: really loved the Color palette selection and Recommended groups features in the original POC. Hopefully we can enable these for 8.4 as well as they're fantastic QoL enhancements for this feature.

[andrew-goldstein]

The should allow a user with crud privileges to attach alerts to cases test in attach_alert_to_case.spec.ts invokes the expandFirstAlertActions() test helper.

The test is failing on CI with the following error:

AssertionError: Timed out retrying after 120000ms: expected '<svg.euiIcon.euiButtonIcon__icon.css-e8xmlg-euiIcon-m-inherit>' to have attribute 'data-is-loaded'
    at expandFirstAlertActions (http://localhost:5620/__cypress/tests?p=cypress/integration/cases/attach_alert_to_case.spec.ts:22991:60)
    at Context.eval (http://localhost:5620/__cypress/tests?p=cypress/integration/cases/attach_alert_to_case.spec.ts:21750:50)

The expandFirstAlertActions() test helper in cypress/tasks/alerts.ts was updated ~ 21 hours ago with this change: https://github.com/elastic/kibana/pull/135373/files#diff-9881e669a4305a15a96f3880d5d68906edfc15991e384a71d7ef5498ed04c0c8R68hh

Based on desk testing, the data-is-loaded attribute:

• Only indicates whether or not the button's SVG is loaded (it's an implementaiton detail of EUI: https://github.com/elastic/eui/blob/b151438410eb39a39c4540e04810a5023530d77f/src/components/icon/icon.tsx)
• Does not seem to be reliable: When inspecting the DOM via dev tools, the attribute is not present, but the icon is clearly visible / loaded

Based on the git history, it looks like the expandFirstAlertActions() test helper didn't include this check:

cy.get(TIMELINE_CONTEXT_MENU_BTN).find('svg').should('have.attr', 'data-is-loaded');

in it's original implementation. The check was first added in this PR: 683463e#diff-9881e669a4305a15a96f3880d5d68906edfc15991e384a71d7ef5498ed04c0c8R68

Since this check only determines whether or not the SVG was loaded in the button icon, and data-is-loaded does not appear to be reilable, I'm removing this cy.get.

[spong]

Sounds good 👍

I was the one that actually added that check, so for context, this is the comment that details the change: #130072 (comment)

When I ran into this I had no issue with it passing locally, but it would fail on CI, so you may have to click-loop and/or wait for the menu if there's still an issue waiting on the table to load before the icon is visible to click.

[andrew-goldstein]

The Displays version conflict error and Displays missing exception list error tests in exceptions_flyout.spec.ts invoke the editException() test helper.

This editException() helper function was added ~ 4 days ago, via this PR and is failing on CI with the following error:

CypressError: Timed out retrying after 120050ms: `cy.click()` failed because the center of this element is hidden from view:

`<button class="euiButtonIcon euiButtonIcon--primary euiButtonIcon--empty euiButtonIcon--xSmall" type="button" aria-label="Exception item actions menu" data-test-subj="exceptionItemCardHeader-actionButton">...</button>`

Fix this problem, or use `{force: true}` to disable error checking.

The test passes locally. I'm implementing the fix suggested in the error message above.

[michaelolo24]

Thanks for doing this work and awesome job! LGTM 🚀

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 2a43fa0
• Storybooks Preview
• Cloud Deployment

Failed CI Steps

• Security Solution Tests #3

Test Failures

• [job] [logs] Security Solution Tests #3 / Inspect Hosts stats and tables "before all" hook for "inspects the All Hosts Table"

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
securitySolution	3118	3146	+28

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	5.2MB	5.2MB	+22.1KB

History

• 💔 Build #57217 failed 848f8da
• 💔 Build #57187 failed 8f9805547889f64b91e7bb06bcd3d3ae01bcc0d8
• 💔 Build #56912 failed 2988064478506bc17d2c578341bc9dc56f3d12e8
• 💔 Build #56622 failed 5006af7387c5bc3b9d53425e4efc6f62fc0ba8cf
• 💔 Build #42706 failed 9bf04dd1a1b77d911beb741afbb1a04d2769ef3d

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @andrew-goldstein