[Security Solution][Detections] Rule Execution Log Feedback and Fixes Part Deux #130072

spong · 2022-04-13T00:40:57Z

Summary

Addresses feedback and fixes identified in #126215 & #129003

Feedback addressed includes:

Adds toast for restoring global query state after performing view alerts for execution action

Updates global SuperDatePicker to daterange of execution (+/- day) for view alerts for execution action (and clear all other filters)
- See above gif
Remove redundant RuleExecutionStatusType ([Security Solution][Detections] Rule Execution Log Feedback and Fixes #129003 (comment))
Persist table state (DatePicker/StatusFilter/SortField/SortOrder/Pagination) when navigating to other tabs on the same page

Fix duration hours bug (7 hours (25033167ms) as 06:417:13:000)

Support disabled rule platform error ([Security Solution][Detections] Adds rule execution log table #126215 (comment))
- Updated getAggregateExecutionEvents to fallback to platform status from event.outcome if security_status is empty, and also falls back to error.message is security_message is empty. This also now queries for corresponding event.outcome if filter is provided so that platform-only events can still be displayed when filtering.

Verify StatusFilter issue [Security Solution][Detections] Adds rule execution log table #126215 (comment)
- Unable to reproduce, I believe the query updates around first querying for status may've fixed this?
Provide helpful defaults for to/from and support datemath strings again ([Security Solution][Detections] Rule Execution Log Feedback and Fixes #129003 (comment))
- Created enhancement for this here: [Security Solution][Detections] Add defaults to Rule Execution log API for from/to #131095
Adds UI Unit tests for RuleExecutionLog Table
Finalize API Integration tests for gap remediation events
- Test methods developed for injecting arbitrary execution events while still working with event-log RBAC. See last API integration test for technique. This can further be used to inject many execution events and expand tests around pagination, sorting, filters, etc.
Fixes gap_duration's of 1-499ms showing up as - instead of 0
Fixes restore filters action to restore either absolute or relative datepicker as it originally was
Resolves [Security Solution] "Rule Execution logs" page shifts downwards on clicking on any duration for the first time only #130946
- Adds min-height to tab container
- Removes scroll-pane from ExceptionsViewer to match Alerts/Execution Log

Remaining follow-ups:

None! 🎉

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)

…log-feedback-part-deux

…-fix'

…log-feedback-part-deux

…ong/kibana into rule-execution-log-feedback-part-deux

…log-feedback-part-deux

MadameSheema · 2022-04-21T12:45:32Z

@elasticmachine merge upstream

.../detections/pages/detection_engine/rules/details/execution_log_table/execution_log_table.tsx

…log-feedback-part-deux

…uration, and adds remaining API integration test

spong · 2022-05-02T20:51:45Z

A side note; not related to this PR.

We should probably clean up the error message before writing it to the event log. We concat error messages with additional info, like rule name, id, space, etc. That information makes sense when we write to the system log (console or json), but not very useful in the context of the event log and only clutters the table:

Good catch! Opened a dedicated bug/chore for this one since this PR has grown a bit already: #131352

yctercero

Platform changes LGTM 👍🏽

…on_ms to gap_duration_s

…log-feedback-part-deux

xcrzx

Checked out and tested locally, everything works as expected, code changes LGTM 👍
Thank you, @spong, for these improvements 🎉

kibana-ci · 2022-05-03T23:31:11Z

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`securitySolution`	2787	2788	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`securitySolution`	4.8MB	4.8MB	+3.6KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`securitySolution`	249.2KB	249.3KB	+159.0B

History

💔 Build #42222 failed f8d0266
💔 Build #42160 failed 7f6ae5d
💚 Build #41915 succeeded 917d9cf
💛 Build #41351 was flaky 1bda037
💔 Build #41342 failed db4ca85

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @spong

michaelolo24 · 2022-05-04T19:01:15Z

x-pack/test/detection_engine_api_integration/utils/index_event_log_execution_events.ts

@@ -19,8 +19,9 @@ export const indexEventLogExecutionEvents = async (
  log: ToolingLog,
  events: object[]
 ): Promise<void> => {
+  const aliases = await es.cat.aliases({ format: 'json', name: '.kibana-event-log-*' });


Question, why this call versus es.indices.getAlias? The formatting?

Yup! IIRC you can't JSON format the response from getAlias -- I had tried it first then swapped over to es.cat.aliases as the response was easier to parse. Not much in documentation on the client, but the endpoint API's mention it (though don't have a sample response for getAlias... 🤷).

michaelolo24

LGTM! Thanks @spong !

kibanamachine · 2022-05-04T19:25:58Z

💔 All backports failed

Status	Branch	Result
❌	8.2	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 130072

Questions ?

Please refer to the Backport tool documentation

… Part Deux (elastic#130072) ## Summary Addresses feedback and fixes identified in elastic#126215 & elastic#129003 ##### Feedback addressed includes: * Adds toast for restoring global query state after performing `view alerts for execution` action <img width="500" src="https://user-images.githubusercontent.com/2946766/164511565-b77d3dc8-a8b5-4927-a947-54966a58c74f.gif" /> * Updates global SuperDatePicker to daterange of execution (+/- day) for `view alerts for execution` action (and clear all other filters) * See above gif * Remove redundant `RuleExecutionStatusType` (elastic#129003 (comment)) * Persist table state (DatePicker/StatusFilter/SortField/SortOrder/Pagination) when navigating to other tabs on the same page <img width="500" src="https://user-images.githubusercontent.com/2946766/164512498-59416601-d967-4a27-b0cc-0715cc0662c0.gif" /> * Fix duration hours bug (`7 hours (25033167ms)` as `06:417:13:000`) <img width="500" src="https://user-images.githubusercontent.com/2946766/164511478-bf0bb6d8-d8b7-4c86-8fbd-b60090f00555.png" /> * Support `disabled rule` platform error (elastic#126215 (comment)) * Updated `getAggregateExecutionEvents` to fallback to platform status from `event.outcome` if `security_status` is empty, and also falls back to `error.message` is `security_message` is empty. This also now queries for corresponding `event.outcome` if filter is provided so that platform-only events can still be displayed when filtering. <img width="500" src="https://user-images.githubusercontent.com/2946766/164510056-1e0bce86-8360-4d46-b591-2041457e3244.png" /> * Verify StatusFilter issue elastic#126215 (comment) * Unable to reproduce, I believe the query updates around first querying for status may've fixed this? * Provide helpful defaults for `to`/`from` and support datemath strings again (elastic#129003 (comment)) * Created enhancement for this here: elastic#131095 * Adds UI Unit tests for RuleExecutionLog Table * Finalize API Integration tests for gap remediation events * Test methods developed for injecting arbitrary execution events while still working with event-log RBAC. See last [API integration test](https://github.com/elastic/kibana/blob/22cc0c8dbd2a1300675caf4c6d471d211ed44858/x-pack/test/detection_engine_api_integration/security_and_spaces/tests/get_rule_execution_events.ts#L121-L166) for technique. This can further be used to inject many execution events and expand tests around pagination, sorting, filters, etc. * Fixes `gap_duration`'s of `1-499`ms showing up as `-` instead of `0` * Fixes restore filters action to restore either absolute or relative datepicker as it originally was * Resolves elastic#130946 * Adds `min-height` to tab container * Removes scroll-pane from ExceptionsViewer to match Alerts/Execution Log --- ##### Remaining follow-ups: None! 🎉 ### Checklist Delete any items that are not applicable to this PR. - [X] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [X] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [X] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/)) - [X] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server)) (cherry picked from commit 683463e) # Conflicts: # x-pack/plugins/security_solution/cypress/tasks/alerts.ts # x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/execution_log_table/execution_log_columns.tsx # x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/event_log/event_log_reader.ts # x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/event_log/get_execution_event_aggregation/index.test.ts # x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/event_log/get_execution_event_aggregation/index.ts # x-pack/test/detection_engine_api_integration/utils/index_event_log_execution_events.ts

… Fixes Part Deux (#130072) (#131574) * [Security Solution][Detections] Rule Execution Log Feedback and Fixes Part Deux (#130072) ## Summary Addresses feedback and fixes identified in #126215 & #129003 ##### Feedback addressed includes: * Adds toast for restoring global query state after performing `view alerts for execution` action <img width="500" src="https://user-images.githubusercontent.com/2946766/164511565-b77d3dc8-a8b5-4927-a947-54966a58c74f.gif" /> * Updates global SuperDatePicker to daterange of execution (+/- day) for `view alerts for execution` action (and clear all other filters) * See above gif * Remove redundant `RuleExecutionStatusType` (#129003 (comment)) * Persist table state (DatePicker/StatusFilter/SortField/SortOrder/Pagination) when navigating to other tabs on the same page <img width="500" src="https://user-images.githubusercontent.com/2946766/164512498-59416601-d967-4a27-b0cc-0715cc0662c0.gif" /> * Fix duration hours bug (`7 hours (25033167ms)` as `06:417:13:000`) <img width="500" src="https://user-images.githubusercontent.com/2946766/164511478-bf0bb6d8-d8b7-4c86-8fbd-b60090f00555.png" /> * Support `disabled rule` platform error (#126215 (comment)) * Updated `getAggregateExecutionEvents` to fallback to platform status from `event.outcome` if `security_status` is empty, and also falls back to `error.message` is `security_message` is empty. This also now queries for corresponding `event.outcome` if filter is provided so that platform-only events can still be displayed when filtering. <img width="500" src="https://user-images.githubusercontent.com/2946766/164510056-1e0bce86-8360-4d46-b591-2041457e3244.png" /> * Verify StatusFilter issue #126215 (comment) * Unable to reproduce, I believe the query updates around first querying for status may've fixed this? * Provide helpful defaults for `to`/`from` and support datemath strings again (#129003 (comment)) * Created enhancement for this here: #131095 * Adds UI Unit tests for RuleExecutionLog Table * Finalize API Integration tests for gap remediation events * Test methods developed for injecting arbitrary execution events while still working with event-log RBAC. See last [API integration test](https://github.com/elastic/kibana/blob/22cc0c8dbd2a1300675caf4c6d471d211ed44858/x-pack/test/detection_engine_api_integration/security_and_spaces/tests/get_rule_execution_events.ts#L121-L166) for technique. This can further be used to inject many execution events and expand tests around pagination, sorting, filters, etc. * Fixes `gap_duration`'s of `1-499`ms showing up as `-` instead of `0` * Fixes restore filters action to restore either absolute or relative datepicker as it originally was * Resolves #130946 * Adds `min-height` to tab container * Removes scroll-pane from ExceptionsViewer to match Alerts/Execution Log --- ##### Remaining follow-ups: None! 🎉 ### Checklist Delete any items that are not applicable to this PR. - [X] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [X] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [X] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/)) - [X] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server)) (cherry picked from commit 683463e) # Conflicts: # x-pack/plugins/security_solution/cypress/tasks/alerts.ts # x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/execution_log_table/execution_log_columns.tsx # x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/event_log/event_log_reader.ts # x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/event_log/get_execution_event_aggregation/index.test.ts # x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/event_log/get_execution_event_aggregation/index.ts # x-pack/test/detection_engine_api_integration/utils/index_event_log_execution_events.ts * Fixing import

… Part Deux (elastic#130072) ## Summary Addresses feedback and fixes identified in elastic#126215 & elastic#129003 ##### Feedback addressed includes: * Adds toast for restoring global query state after performing `view alerts for execution` action <img width="500" src="https://user-images.githubusercontent.com/2946766/164511565-b77d3dc8-a8b5-4927-a947-54966a58c74f.gif" /> * Updates global SuperDatePicker to daterange of execution (+/- day) for `view alerts for execution` action (and clear all other filters) * See above gif * Remove redundant `RuleExecutionStatusType` (elastic#129003 (comment)) * Persist table state (DatePicker/StatusFilter/SortField/SortOrder/Pagination) when navigating to other tabs on the same page <img width="500" src="https://user-images.githubusercontent.com/2946766/164512498-59416601-d967-4a27-b0cc-0715cc0662c0.gif" /> * Fix duration hours bug (`7 hours (25033167ms)` as `06:417:13:000`) <img width="500" src="https://user-images.githubusercontent.com/2946766/164511478-bf0bb6d8-d8b7-4c86-8fbd-b60090f00555.png" /> * Support `disabled rule` platform error (elastic#126215 (comment)) * Updated `getAggregateExecutionEvents` to fallback to platform status from `event.outcome` if `security_status` is empty, and also falls back to `error.message` is `security_message` is empty. This also now queries for corresponding `event.outcome` if filter is provided so that platform-only events can still be displayed when filtering. <img width="500" src="https://user-images.githubusercontent.com/2946766/164510056-1e0bce86-8360-4d46-b591-2041457e3244.png" /> * Verify StatusFilter issue elastic#126215 (comment) * Unable to reproduce, I believe the query updates around first querying for status may've fixed this? * Provide helpful defaults for `to`/`from` and support datemath strings again (elastic#129003 (comment)) * Created enhancement for this here: elastic#131095 * Adds UI Unit tests for RuleExecutionLog Table * Finalize API Integration tests for gap remediation events * Test methods developed for injecting arbitrary execution events while still working with event-log RBAC. See last [API integration test](https://github.com/elastic/kibana/blob/22cc0c8dbd2a1300675caf4c6d471d211ed44858/x-pack/test/detection_engine_api_integration/security_and_spaces/tests/get_rule_execution_events.ts#L121-L166) for technique. This can further be used to inject many execution events and expand tests around pagination, sorting, filters, etc. * Fixes `gap_duration`'s of `1-499`ms showing up as `-` instead of `0` * Fixes restore filters action to restore either absolute or relative datepicker as it originally was * Resolves elastic#130946 * Adds `min-height` to tab container * Removes scroll-pane from ExceptionsViewer to match Alerts/Execution Log --- ##### Remaining follow-ups: None! 🎉 ### Checklist Delete any items that are not applicable to this PR. - [X] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [X] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [X] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/)) - [X] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))

Adds toast for resetting global search filters

5e93888

spong self-assigned this Apr 13, 2022

spong added the ci:deploy-cloud label Apr 13, 2022

spong and others added 18 commits April 13, 2022 08:59

Mocks the missing mocks merrily

ead5ad4

Merge branch 'main' of github.com:elastic/kibana into rule-execution-…

0b27bad

…log-feedback-part-deux

More missed mocks

b61afbd

Merge branch 'main' of github.com:elastic/kibana into rule-execution-…

c40fa6d

…log-feedback-part-deux

Last mock to update it looks like...

dfb6374

Removes RuleExecutionStatusType

3cc941a

Merge branch 'main' of github.com:elastic/kibana into rule-execution-…

a8f16c3

…log-feedback-part-deux

Adds RuleDetailsContextProvider to maintain state between tab selection

068ccd8

Merge branch 'main' of github.com:elastic/kibana into rule-execution-…

3ec3084

…log-feedback-part-deux

Fixing new type constraints

a77a303

Merge branch 'main' of github.com:elastic/kibana into rule-execution-…

9c8ea5a

…log-feedback-part-deux

Fixing default statusFilter param to support receiving emptystring

3c4df6c

Merge branch 'main' of github.com:elastic/kibana into rule-execution-…

a1a5247

…log-feedback-part-deux

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

2126a03

…-fix'

Merge branch 'main' of github.com:elastic/kibana into rule-execution-…

9d393f5

…log-feedback-part-deux

Merge branch 'rule-execution-log-feedback-part-deux' of github.com:sp…

8207539

…ong/kibana into rule-execution-log-feedback-part-deux

Fixes RuleDurationFormat bugs and adds test coverage

22a9ca7

Merge branch 'main' of github.com:elastic/kibana into rule-execution-…

f51c6eb

…log-feedback-part-deux

kibanamachine and others added 2 commits April 21, 2022 07:45

Merge branch 'main' into rule-execution-log-feedback-part-deux

30daac6

Adds support for disabled while running rule execution error

3623419

xcrzx reviewed May 2, 2022

View reviewed changes

.../detections/pages/detection_engine/rules/details/execution_log_table/execution_log_table.tsx Outdated Show resolved Hide resolved

spong added 2 commits May 2, 2022 14:35

Merge branch 'main' of github.com:elastic/kibana into rule-execution-…

9581b83

…log-feedback-part-deux

Fixes absolute-relative datepicker replacement, fixes showing 0 gap_d…

22cc0c8

…uration, and adds remaining API integration test

spong mentioned this pull request May 2, 2022

[Security Solution][Detections] Clean up rule execution message before writing to event-log #131352

Closed

Removes scrollpane from exceptions viewer

917d9cf

spong requested a review from a team as a code owner May 2, 2022 21:03

yctercero approved these changes May 2, 2022

View reviewed changes

spong mentioned this pull request May 3, 2022

[Security Solution][Sourcerer] Delay in Alerts Security Data View initialization after Alerts index is created #131427

Open

spong added 2 commits May 3, 2022 11:01

Reverts gap duration empty value render change and renames gap_durati…

32e12d3

…on_ms to gap_duration_s

Merge branch 'main' of github.com:elastic/kibana into rule-execution-…

7f6ae5d

…log-feedback-part-deux

xcrzx approved these changes May 3, 2022

View reviewed changes

spong added the auto-backport Deprecated - use backport:version if exact versions are needed label May 3, 2022

spong added 2 commits May 3, 2022 14:05

Merge branch 'main' into rule-execution-log-feedback-part-deux

f8d0266

Merge branch 'main' into rule-execution-log-feedback-part-deux

5dc0da7

angorayc approved these changes May 4, 2022

View reviewed changes

michaelolo24 reviewed May 4, 2022

View reviewed changes

michaelolo24 approved these changes May 4, 2022

View reviewed changes

spong merged commit 683463e into elastic:main May 4, 2022

spong deleted the rule-execution-log-feedback-part-deux branch May 4, 2022 19:22

spong mentioned this pull request May 4, 2022

[8.2] [Security Solution][Detections] Rule Execution Log Feedback and Fixes Part Deux (#130072) #131574

Merged

spong mentioned this pull request May 4, 2022

[Security Solution]HTML tag are there in the message for Failed Rule Execution Logs entry #131380

Closed

joepeeples mentioned this pull request Jun 3, 2022

[DOCS] Rule execution logs: Add filter icon in Actions column elastic/security-docs#2043

Closed

spong mentioned this pull request Jul 12, 2022

[Security Solution] Alerts Treemap and Multi-Dimensional Alert Grouping #126896

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution][Detections] Rule Execution Log Feedback and Fixes Part Deux #130072

[Security Solution][Detections] Rule Execution Log Feedback and Fixes Part Deux #130072

spong commented Apr 13, 2022 •

edited

Loading

MadameSheema commented Apr 21, 2022

spong commented May 2, 2022

yctercero left a comment

xcrzx left a comment

kibana-ci commented May 3, 2022

michaelolo24 May 4, 2022

spong May 4, 2022

michaelolo24 left a comment

kibanamachine commented May 4, 2022

[Security Solution][Detections] Rule Execution Log Feedback and Fixes Part Deux #130072

[Security Solution][Detections] Rule Execution Log Feedback and Fixes Part Deux #130072

Conversation

spong commented Apr 13, 2022 • edited Loading

Summary

Feedback addressed includes:

Remaining follow-ups:

Checklist

MadameSheema commented Apr 21, 2022

spong commented May 2, 2022

yctercero left a comment

Choose a reason for hiding this comment

xcrzx left a comment

Choose a reason for hiding this comment

kibana-ci commented May 3, 2022

💚 Build Succeeded

Metrics [docs]

Module Count

Async chunks

Page load bundle

History

michaelolo24 May 4, 2022

Choose a reason for hiding this comment

spong May 4, 2022

Choose a reason for hiding this comment

michaelolo24 left a comment

Choose a reason for hiding this comment

kibanamachine commented May 4, 2022

💔 All backports failed

Manual backport

Questions ?

spong commented Apr 13, 2022 •

edited

Loading