Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[8.9] [Security Solution] Flaky Cypress test: Detection rules, Prebuilt Rules Installation and Update workflow - Installation of prebuilt rules package via Fleet should install package from Fleet in the background #164566

Merged
merged 2 commits into from
Aug 24, 2023

Conversation

jpdjere
Copy link
Contributor

@jpdjere jpdjere commented Aug 23, 2023

NOTE: This is a manual backport of #163468 to 8.9.

Original PR description:

Fixes: #163447
Fixes: #163586

Summary

  • Fixes flaky test: x-pack/plugins/security_solution/cypress/e2e/detection_response/prebuilt_rules/prebuilt_rules_install_update_workflows.cy.ts
  • Test title: Detection rules, Prebuilt Rules Installation and Update workflow - Installation of prebuilt rules package via Fleet should install package from Fleet in the background

Details

  • Initially ran the flaky test runner with multiple iterations and all gave succesful results, i.e. no flakiness or failed tests.
  • But: after checking the logs for the failed tests in the original failed build, discovered that the reason the test failed is because:
    • when checking Fleet's response for the installation of the security_detection_engine, the API response was not as expected from the API spec:
**Expected:** [{ name: 'security_detection_engine', installSource: 'registry' }]
**Actual:** [{ name: 'security_detection_engine', installSource: undefined }]

Since we cannot rely 100% that the Fleet API will return the correct value for the installSource, this PR deletes this part of the test to prevent any type of flakiness caused by external factors such as this.

@jpdjere jpdjere added test backport release_note:skip Skip the PR/issue when compiling release notes Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Detection Rule Management Security Detection Rule Management Team labels Aug 23, 2023
@jpdjere jpdjere self-assigned this Aug 23, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@jpdjere jpdjere assigned maximpn and unassigned maximpn Aug 23, 2023
@jpdjere jpdjere requested a review from maximpn August 23, 2023 11:24
@kibana-ci
Copy link
Collaborator

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] Security Solution Tests #3 / Changing alert status Opening alerts Open one alert when more than one closed alerts are selected
  • [job] [logs] Defend Workflows Cypress Tests #1 / Endpoint Policy Response from Fleet Agent Details page should display policy response with errors

Metrics [docs]

Unknown metric groups

ESLint disabled line counts

id before after diff
enterpriseSearch 13 15 +2
securitySolution 416 420 +4
total +6

Total ESLint disabled count

id before after diff
enterpriseSearch 14 16 +2
securitySolution 497 501 +4
total +6

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @jpdjere

@jpdjere jpdjere merged commit 7b2b4a4 into elastic:8.9 Aug 24, 2023
@jpdjere jpdjere added the flake-docs Temp label to gather PRs used to create dev docs label Aug 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport flake-docs Temp label to gather PRs used to create dev docs release_note:skip Skip the PR/issue when compiling release notes Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. test
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants