Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SIEM] Histogram Consistency & Loading Updates #55585

Open
3 of 4 tasks
MichaelMarcialis opened this issue Jan 22, 2020 · 4 comments
Open
3 of 4 tasks

[SIEM] Histogram Consistency & Loading Updates #55585

MichaelMarcialis opened this issue Jan 22, 2020 · 4 comments
Assignees
Labels
Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:SIEM v7.6.0

Comments

@MichaelMarcialis
Copy link
Contributor

MichaelMarcialis commented Jan 22, 2020

With the upcoming release of the redesigned overview page, I've noticed some consistency issues between histograms and had some concerns with the existing loading states for each. This ticket is meant to capture these issues in the hopes of correcting them before the 7.6 release.

CCing @andrew-goldstein and @angorayc, as these requested changes could have implications on the overview page redesign.

  • Histogram bar interval (bar counts and widths) consistency
    Depending on the time range and data being shown, each histogram can potentially present the user with different bar intervals (bar counts and widths) than other histograms on the page. These bar intervals may also change when the user changes the stack dimension for the data. Apparently, these bar intervals are being set dynamically depending on the data being shown. I believe this is at the detriment to the user when we are presenting more than one histogram on a page at a time (such as the new overview page). In these situations, it makes more sense to present the user with the same bar intervals between charts, so it is easier to read and understand. Can we make the bar intervals a consistent 32 bars across the board for now and revisit if users take issue with this in the future?

    • @spong: Resolved as part of [SIEM] [Detection Engine] Fixes histogram intervals  #55969. Note: depending on the timerange the user selects, an extra interval may be created to deal with any remainder so we don't exclude any data. That said, the intervals will remain consistent across all histograms, so the ability to compare one with another will remain intact. Also of note, I recently discovered that there's actually some histogram settings in Kibana Advanced Settings that we can make use of, so we may want to explore using those.
  • Histogram x-axis tick placement consistency
    It appears that the signals histograms correctly shows the x-axis tick marks in between the bars (showing the beginning and ending of histogram interval bar's range). However, all other SIEM histograms show the ticks on the bars, which I believe is indicative of a traditional bar chart, not a histogram.

 Are we able to have the histogram ticks placement be consistent between histograms and also position them to always appear in between the bars?

  • Improve loading states
    All SIEM histograms appear to use the EuiLoadingContent component for their loading state. This is odd, as these histograms are not text-only content. Can we swap this out for something better, until I'm able to craft a loading patterns guide for consistent loading states across the SIEM app? I'm thinking something like:


    • Spinner on initial page load


      Show nothing but panel title and a vertically/horizontally centered EuiLoadingSpinner on intial load.

    • Progress bar on panel for data refreshes
      It's unnecessary to obscure content during a data refresh. In this case, rather than using a loading overlay as we have in the past, I think it would make more sense to use the EuiProgress component (absolutely positioned on the top of the panel), as shown in the EUI docs.

  • Correct color usage
    @MikePaquette correctly noted that some of the color usage for our histogram is incorrect and particularly noticeable in dark mode. This includes the colors being used for column hover, axes, ticks, labels and possibly others. I believe the fix is simply to use the Elastic Charts theme provided by EUI (https://elastic.github.io/eui/#/elastic-charts/creating-charts). Can we apply this theme and see if this corrects our color issues?

Example of inconsistent bar intervals and ticks:
image

Example of incorrect color usage:
image

@elasticmachine
Copy link
Contributor

Pinging @elastic/siem (Team:SIEM)

spong added a commit that referenced this issue Jan 27, 2020
## Summary

This PR wraps up the remaining `Detection Engine` meta tickets: #55585, #54935, and elastic/siem-team#498
- [x] Histogram bar interval (bar counts and widths) consistency (#55585)
  - [x] Make the bar intervals a consistent 32 bars across the board
  * Enabled `extended_bounds`, `min_doc_count: 0`, and now setting consistent `fixed_interval` when querying to ensure the entire daterange is displayed across all histograms.
- [x] Filter out the "untitled" timelines from both timeline selection options during rule creation (elastic/siem-team#498)
  - [ ] ~Import query from saved timeline~
    * For 7.7 tracking ticket here: #56079
  - [x] `Investigate detections using this timeline template` 
- [x] Everywhere we use "Alerts" (Overview page, Host Tab, Network Tab) we should change the term to "External Alerts"
  - [x] Updated Host Page Tab/Table/Histogram/Breadcrumbs
  - [x] Updated Network Page Tab/Table/Histogram/Breadcrumbs
- [x] Updated DE permission/index  error doc links to go to [corresponding DE docs section](https://www.elastic.co/guide/en/siem/guide/7.6/detection-engine-overview.html#detections-permissions)
- [x] Removed `frequency` in favor of `count` for remaining histograms

##### Inconsistent Histogram intervals
![image](https://user-images.githubusercontent.com/2946766/73161560-04a82300-40a9-11ea-950f-ea56f9a5bfd7.png)


##### Consistent Histogram Intervals
![image](https://user-images.githubusercontent.com/2946766/73159564-fefc0e80-40a3-11ea-9b9d-4d15899dabd2.png)


cc @MichaelMarcialis @cwurm @MikePaquette 

### Checklist

Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR.

- [ ] ~This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~
- [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)
- [ ] ~[Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials~
- [ ] ~[Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios~
- [ ] ~This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~

### For maintainers

- [ ] ~This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~
- [ ] ~This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~
spong added a commit to spong/kibana that referenced this issue Jan 27, 2020
## Summary

This PR wraps up the remaining `Detection Engine` meta tickets: elastic#55585, elastic#54935, and elastic/siem-team#498
- [x] Histogram bar interval (bar counts and widths) consistency (elastic#55585)
  - [x] Make the bar intervals a consistent 32 bars across the board
  * Enabled `extended_bounds`, `min_doc_count: 0`, and now setting consistent `fixed_interval` when querying to ensure the entire daterange is displayed across all histograms.
- [x] Filter out the "untitled" timelines from both timeline selection options during rule creation (elastic/siem-team#498)
  - [ ] ~Import query from saved timeline~
    * For 7.7 tracking ticket here: elastic#56079
  - [x] `Investigate detections using this timeline template` 
- [x] Everywhere we use "Alerts" (Overview page, Host Tab, Network Tab) we should change the term to "External Alerts"
  - [x] Updated Host Page Tab/Table/Histogram/Breadcrumbs
  - [x] Updated Network Page Tab/Table/Histogram/Breadcrumbs
- [x] Updated DE permission/index  error doc links to go to [corresponding DE docs section](https://www.elastic.co/guide/en/siem/guide/7.6/detection-engine-overview.html#detections-permissions)
- [x] Removed `frequency` in favor of `count` for remaining histograms

##### Inconsistent Histogram intervals
![image](https://user-images.githubusercontent.com/2946766/73161560-04a82300-40a9-11ea-950f-ea56f9a5bfd7.png)


##### Consistent Histogram Intervals
![image](https://user-images.githubusercontent.com/2946766/73159564-fefc0e80-40a3-11ea-9b9d-4d15899dabd2.png)


cc @MichaelMarcialis @cwurm @MikePaquette 

### Checklist

Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR.

- [ ] ~This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~
- [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)
- [ ] ~[Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials~
- [ ] ~[Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios~
- [ ] ~This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~

### For maintainers

- [ ] ~This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~
- [ ] ~This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~
spong added a commit to spong/kibana that referenced this issue Jan 27, 2020
## Summary

This PR wraps up the remaining `Detection Engine` meta tickets: elastic#55585, elastic#54935, and elastic/siem-team#498
- [x] Histogram bar interval (bar counts and widths) consistency (elastic#55585)
  - [x] Make the bar intervals a consistent 32 bars across the board
  * Enabled `extended_bounds`, `min_doc_count: 0`, and now setting consistent `fixed_interval` when querying to ensure the entire daterange is displayed across all histograms.
- [x] Filter out the "untitled" timelines from both timeline selection options during rule creation (elastic/siem-team#498)
  - [ ] ~Import query from saved timeline~
    * For 7.7 tracking ticket here: elastic#56079
  - [x] `Investigate detections using this timeline template` 
- [x] Everywhere we use "Alerts" (Overview page, Host Tab, Network Tab) we should change the term to "External Alerts"
  - [x] Updated Host Page Tab/Table/Histogram/Breadcrumbs
  - [x] Updated Network Page Tab/Table/Histogram/Breadcrumbs
- [x] Updated DE permission/index  error doc links to go to [corresponding DE docs section](https://www.elastic.co/guide/en/siem/guide/7.6/detection-engine-overview.html#detections-permissions)
- [x] Removed `frequency` in favor of `count` for remaining histograms

##### Inconsistent Histogram intervals
![image](https://user-images.githubusercontent.com/2946766/73161560-04a82300-40a9-11ea-950f-ea56f9a5bfd7.png)


##### Consistent Histogram Intervals
![image](https://user-images.githubusercontent.com/2946766/73159564-fefc0e80-40a3-11ea-9b9d-4d15899dabd2.png)


cc @MichaelMarcialis @cwurm @MikePaquette 

### Checklist

Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR.

- [ ] ~This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~
- [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)
- [ ] ~[Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials~
- [ ] ~[Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios~
- [ ] ~This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~

### For maintainers

- [ ] ~This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~
- [ ] ~This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~
@spong
Copy link
Member

spong commented Jan 28, 2020

@MichaelMarcialis -- @patrykkopycinski and I were able to address all of the above except Histogram x-axis tick placement consistency as part of 7.6.

Having spent some more time with the histograms it looks like we've got a bit of flexibility around tick placement, so that's good news. 🙂

Do you want to keep this open for tracking this remaining task, or should be move it to #55753 and close this issue?

spong added a commit that referenced this issue Jan 28, 2020
## Summary

This PR wraps up the remaining `Detection Engine` meta tickets: #55585, #54935, and elastic/siem-team#498
- [x] Histogram bar interval (bar counts and widths) consistency (#55585)
  - [x] Make the bar intervals a consistent 32 bars across the board
  * Enabled `extended_bounds`, `min_doc_count: 0`, and now setting consistent `fixed_interval` when querying to ensure the entire daterange is displayed across all histograms.
- [x] Filter out the "untitled" timelines from both timeline selection options during rule creation (elastic/siem-team#498)
  - [ ] ~Import query from saved timeline~
    * For 7.7 tracking ticket here: #56079
  - [x] `Investigate detections using this timeline template` 
- [x] Everywhere we use "Alerts" (Overview page, Host Tab, Network Tab) we should change the term to "External Alerts"
  - [x] Updated Host Page Tab/Table/Histogram/Breadcrumbs
  - [x] Updated Network Page Tab/Table/Histogram/Breadcrumbs
- [x] Updated DE permission/index  error doc links to go to [corresponding DE docs section](https://www.elastic.co/guide/en/siem/guide/7.6/detection-engine-overview.html#detections-permissions)
- [x] Removed `frequency` in favor of `count` for remaining histograms

##### Inconsistent Histogram intervals
![image](https://user-images.githubusercontent.com/2946766/73161560-04a82300-40a9-11ea-950f-ea56f9a5bfd7.png)


##### Consistent Histogram Intervals
![image](https://user-images.githubusercontent.com/2946766/73159564-fefc0e80-40a3-11ea-9b9d-4d15899dabd2.png)


cc @MichaelMarcialis @cwurm @MikePaquette 

### Checklist

Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR.

- [ ] ~This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~
- [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)
- [ ] ~[Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials~
- [ ] ~[Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios~
- [ ] ~This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~

### For maintainers

- [ ] ~This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~
- [ ] ~This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~
spong added a commit that referenced this issue Jan 28, 2020
## Summary

This PR wraps up the remaining `Detection Engine` meta tickets: #55585, #54935, and elastic/siem-team#498
- [x] Histogram bar interval (bar counts and widths) consistency (#55585)
  - [x] Make the bar intervals a consistent 32 bars across the board
  * Enabled `extended_bounds`, `min_doc_count: 0`, and now setting consistent `fixed_interval` when querying to ensure the entire daterange is displayed across all histograms.
- [x] Filter out the "untitled" timelines from both timeline selection options during rule creation (elastic/siem-team#498)
  - [ ] ~Import query from saved timeline~
    * For 7.7 tracking ticket here: #56079
  - [x] `Investigate detections using this timeline template` 
- [x] Everywhere we use "Alerts" (Overview page, Host Tab, Network Tab) we should change the term to "External Alerts"
  - [x] Updated Host Page Tab/Table/Histogram/Breadcrumbs
  - [x] Updated Network Page Tab/Table/Histogram/Breadcrumbs
- [x] Updated DE permission/index  error doc links to go to [corresponding DE docs section](https://www.elastic.co/guide/en/siem/guide/7.6/detection-engine-overview.html#detections-permissions)
- [x] Removed `frequency` in favor of `count` for remaining histograms

##### Inconsistent Histogram intervals
![image](https://user-images.githubusercontent.com/2946766/73161560-04a82300-40a9-11ea-950f-ea56f9a5bfd7.png)


##### Consistent Histogram Intervals
![image](https://user-images.githubusercontent.com/2946766/73159564-fefc0e80-40a3-11ea-9b9d-4d15899dabd2.png)


cc @MichaelMarcialis @cwurm @MikePaquette 

### Checklist

Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR.

- [ ] ~This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~
- [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)
- [ ] ~[Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials~
- [ ] ~[Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios~
- [ ] ~This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~

### For maintainers

- [ ] ~This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~
- [ ] ~This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~
@MichaelMarcialis
Copy link
Contributor Author

@MichaelMarcialis -- @patrykkopycinski and I were able to address all of the above except Histogram x-axis tick placement consistency as part of 7.6.

Having spent some more time with the histograms it looks like we've got a bit of flexibility around tick placement, so that's good news. 🙂

Do you want to keep this open for tracking this remaining task, or should be move it to #55753 and close this issue?

I say let's either keep this ticket open or spin off as a separate issue. I'd like to keep it out of #55753, as it is not strictly a detection issue. Whichever you and @patrykkopycinski prefer.

@spong
Copy link
Member

spong commented Jan 29, 2020

Sounds good -- we can just keep this one open with updating the x-axis tick placement as the last remaining task. Thanks @MichaelMarcialis!

@MindyRS MindyRS added the Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. label Oct 27, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:SIEM v7.6.0
Projects
None yet
Development

No branches or pull requests

5 participants