[Security] Periodic testing for security against MKI, for latest Kibana commit #171859

dkirchan · 2023-11-23T12:57:55Z

Summary

In this PR the Kibana image override is introduced.

The user will be able to decide whether the pipeline will be testing against the code that is already deployed in QA OR if the project will be created with an override from the latest commit in kibana.

This override is created in the first step of the pipeline where a new freshly baked kibana image is built and pushed into docker.elastic.co/kibana-ci/kibana-serverless with tag git-${BUILDKITE_COMMIT:0:12} which is the exact commit checked out in the current build. This is then pushed to the above mentioned registry and as a result an image like docker.elastic.co/kibana-ci/kibana-serverless:git-46iuy423rn3 is being used as the override.

In order to run the tests with the override the environment variable KIBANA_MKI_USE_LATEST_COMMIT=1 needs to be passed when starting the build as shown in the following snippet.

By default, if the environment variable is not passed, the tests will run against the already deployed version in QA without any override.

If only the override flow is picked, then the Build Kibana Image will build and push the new image. On the other hand if the environment variable is absent, then the Build Kibana Image step will exit immediately with exit code 0 (passed in buildkite pipeline) and proceed to run the tests normally.

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
This was checked for cross-browser compatibility

Risk Matrix

Delete this section if it is not applicable to this PR.

Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release.

When forming the risk matrix, consider some of the following examples and how they may potentially impact the change:

Risk	Probability	Severity	Mitigation/Notes
Multiple Spaces—unexpected behavior in non-default Kibana Space.	Low	High	Integration tests will verify that all features are still supported in non-default Kibana Space and when user switches between spaces.
Multiple nodes—Elasticsearch polling might have race conditions when multiple Kibana nodes are polling for the same tasks.	High	Low	Tasks are idempotent, so executing them multiple times will not result in logical error, but will degrade performance. To test for this case we add plenty of unit tests around this logic and document manual testing procedure.
Code should gracefully handle cases when feature X or plugin Y are disabled.	Medium	High	Unit tests will verify that any feature flag or plugin combination still results in our service operational.
See more potential risk examples

For maintainers

This was checked for breaking API changes and was labeled appropriately

apmmachine · 2023-11-23T12:58:12Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
/oblt-deploy-serverless : Deploy a serverless Kibana instance using the Observability test environments.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2023-12-04T16:34:05Z

Pinging @elastic/security-solution (Team: SecuritySolution)

elasticmachine · 2023-12-04T16:34:07Z

Pinging @elastic/security-defend-workflows (Team:Defend Workflows)

MadameSheema · 2024-02-06T09:40:16Z

...te/scripts/pipelines/security_solution_quality_gate/api_integration/api-integration-tests.sh

    --header "Authorization: ApiKey $QA_API_KEY" \
    --header 'Content-Type: application/json' \
    --data '{
-        "name": "ftr-integration-tests-'$random_number'",
-        "region_id": "aws-eu-west-1"}' | jq '.')
+          "name": "ftr-integration-tests-'$random_number'",


Why are we using a random number?

This way it works both against buildkite or locally. Otherwise I should have to do if else statements in bash script to use an alternative for each case of the above. When this will be moved into the normal runner such as parallel_serverless, they can fix it.

MadameSheema · 2024-02-06T09:41:53Z

x-pack/plugins/security_solution/scripts/run_cypress/parallel_serverless.ts

+    );
+    body.overrides = {
+      kibana: {
+        docker_image: `docker.elastic.co/kibana-ci/kibana-serverless:sec-sol-qg-${kibanaOverrideImage}`,


Same as: https://github.com/elastic/kibana/pull/171859/files#r1479479038

This (docker.elastic.co/kibana-ci/kibana-serverless) is hardcoded in other places as well e.g:

kibana/.buildkite/scripts/steps/artifacts/docker_image.sh

Line 16 in 24d3619

KIBANA_BASE_IMAGE="docker.elastic.co/kibana-ci/kibana-serverless"

I guess its not a problem to be hardcoded for us as well. If you insist though, I can move it definitely to vault.

I guess it is not then :)

MadameSheema

Thanks @dkirchan :)

…-fix'

kibana-ci · 2024-02-06T14:45:49Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: eeb44ad

Failed CI Steps

Test Failures

[job] [logs] Investigations - Security Solution Cypress Tests #7 / Basic esql search and filter operations "before each" hook for "should remove the query when the back button is pressed after adding a query" "before each" hook for "should remove the query when the back button is pressed after adding a query"
[job] [logs] Defend Workflows Cypress Tests on Serverless #6 / Response console Processes operations: "before all" hook for ""processes" - should obtain a list of processes" "before all" hook for ""processes" - should obtain a list of processes"

Metrics [docs]

✅ unchanged

History

💛 Build #191401 was flaky 56ce703
💔 Build #191371 failed 9f4d0a5
💔 Build #190476 failed 9ed98e1
💔 Build #190417 failed b7bc072
💔 Build #190233 failed e7f74e3
💛 Build #189943 was flaky 74c6221

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @dkirchan

kibanamachine · 2024-02-06T15:03:33Z

💔 All backports failed

Status	Branch	Result
❌	8.12	Backport failed because of merge conflicts You might need to backport the following PRs to 8.12: - [Security Solution] Specific Cypress executions for `Entity Analytics` team (#173024)

Manual backport

To create the backport manually run:

node scripts/backport --pr 171859

Questions ?

Please refer to the Backport tool documentation

…na commit (elastic#171859) ## Summary In this PR the Kibana image override is introduced. The user will be able to decide whether the pipeline will be testing against the code that is already deployed in QA **OR** if the project will be created with an override from the latest commit in kibana. This override is created in the first step of the pipeline where a new freshly baked kibana image is built and pushed into `docker.elastic.co/kibana-ci/kibana-serverless` with tag `git-${BUILDKITE_COMMIT:0:12}` which is the exact commit checked out in the current build. This is then pushed to the above mentioned registry and as a result an image like `docker.elastic.co/kibana-ci/kibana-serverless:git-46iuy423rn3` is being used as the override. In order to run the tests with the override the environment variable `KIBANA_MKI_USE_LATEST_COMMIT=1` needs to be passed when starting the build as shown in the following snippet. ![Screenshot 2024-01-23 at 12 49 03 PM](https://github.com/elastic/kibana/assets/55240027/273fe0f9-5dcd-403d-84ce-6c79365916da) By default, if the environment variable is not passed, the tests will run against the already deployed version in QA without any override. **If only** the override flow is picked, then the Build Kibana Image will build and push the new image. On the other hand if the environment variable is absent, then the Build Kibana Image step will exit immediately with exit code 0 (passed in buildkite pipeline) and proceed to run the tests normally. ### Checklist Delete any items that are not applicable to this PR. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/)) - [ ] Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US)) - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server)) - [ ] This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers) ### Risk Matrix Delete this section if it is not applicable to this PR. Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release. When forming the risk matrix, consider some of the following examples and how they may potentially impact the change: | Risk | Probability | Severity | Mitigation/Notes | |---------------------------|-------------|----------|-------------------------| | Multiple Spaces—unexpected behavior in non-default Kibana Space. | Low | High | Integration tests will verify that all features are still supported in non-default Kibana Space and when user switches between spaces. | | Multiple nodes—Elasticsearch polling might have race conditions when multiple Kibana nodes are polling for the same tasks. | High | Low | Tasks are idempotent, so executing them multiple times will not result in logical error, but will degrade performance. To test for this case we add plenty of unit tests around this logic and document manual testing procedure. | | Code should gracefully handle cases when feature X or plugin Y are disabled. | Medium | High | Unit tests will verify that any feature flag or plugin combination still results in our service operational. | | [See more potential risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) | ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: kibanamachine <[email protected]>

kibanamachine · 2024-02-07T21:48:36Z