[Fleet][Tests] Package install signature verification API tests #136947

hop-dev · 2022-07-22T13:21:29Z

Non Fleet team reviewers 👋 : I have had to change your package registry config to point to a new directory. In Fleet we have moved to using the v2 registry docker image for integration testing which stores packages in this different directory 👍

Summary

Part of #133822.

This PR isn't as big as it seems! :) Files under fixtures/package_verification/packages/src are just test packages.

Add tests for package verification on the install package route. As part of these tests I have had to include a public and private key pair for testing purposes, the private key is needed to generate the signed zip files.

Fleet Devs: This means that in integration tests any non-test fixture package from the registry will fail signature verification and must be force installed. In general we should be using test packages for our integration tests to avoid this, bu I understand there are situations where using a real package can't be avoided.

This PR was starting to get a bit big, things I intend to do in further PRs:

package policy API tests
cypress tests to see if the force install modal is displayed
add a dev doc for package verification in general

x-pack/test/fleet_api_integration/config.ts

hop-dev · 2022-07-28T14:49:25Z

x-pack/test/security_solution_endpoint/services/endpoint_policy.ts

@@ -33,7 +33,7 @@ const INGEST_API_PACKAGE_POLICIES = `${INGEST_API_ROOT}/package_policies`;
 const INGEST_API_PACKAGE_POLICIES_DELETE = `${INGEST_API_PACKAGE_POLICIES}/delete`;
 const INGEST_API_EPM_PACKAGES = `${INGEST_API_ROOT}/epm/packages`;

-const SECURITY_PACKAGES_ROUTE = `${INGEST_API_EPM_PACKAGES}?category=security`;
+const SECURITY_PACKAGES_ROUTE = `${INGEST_API_EPM_PACKAGES}?category=security&experimental=true`;


I was having the issue where 8.4.0-dev of the endpoint package was being installed, but this function was returning 8.3.0 as the latest version, adding ?experimental=true fixes this, and it made sense that we would always want to test with the latest package, but do we only ever want to install GA packages for these tests?

cc @paul-tavares is installing experimental (prereleases) endpoint package a good idea for these tests?

somewhat unlikely that a prerelease package is going to break tests since it's mostly mappings, but transform stuff is there too.

And the prereleases are going to be more unstable now with automatic deploys.

on the face of it it doesn't seem great, but I'm having trouble thinking through the implications of what this affects

@pzl ,
I think running against the latest package here is ok. Finding and addressing failures as soon as possible rather than to wait until a release package is available would be best. I'm actually a bit surprised - have we been testing against an older package all along?

@hop-dev ,
What does experimental do from a fleet API standpoint?
Our tests all run with a custom package registry running in docker (we get the config for the FTR tests from here, which uses the Docker image defined via the fleet API integration config. Just want to ensure that your change here will not cause the tests to become flaky, especially if somehow this flag is causing the API to reach out to the "real" registry (we have had problems in the past with CI test runs making external network calls to the real registry (especially network timeout or connection issues)).

Hi @paul-tavares, adding experimental means EPR returns the latest version of a package it has whether it is GA or not, e.g it will return 8.4.0-dev instead of 8.3.0 of the endpoint package. It doesnt change where we go to get the packages.

As part of this PR I have moved to v2 of the EPR docker image which contains dev packages which it seems the previous v1 docker image we were using didn't. When the security tests start we install the latest version of the package here

the previous issue was that we were installing 8.4.0-dev but then the EPR call was returning 8.3.0 as the latest version, adding experimental solved this.

If it is too risky to use the latest version I could look at a way of only specifying the latest GA release of the package on startup

Just a note, the version of the package would only change if we update the docker image so I wouldnt expect this to add flakiness. Only the PR changing the image would fail

^ got it. So this is still pulling a particular docker image to test from, and the query here is being sent to that point-in-time image. Sounds like there will be at least some minimal checking that it isn't a completely broken package during a PR that would update the image. Seems ok to me

Thanks @hop-dev . It seems ok with me as well, so I'll 👍 it.

hop-dev · 2022-07-29T14:39:36Z

x-pack/test/functional_synthetics/apps/uptime/synthetics_integration.ts

@@ -54,6 +54,7 @@ export default function (providerContext: FtrProviderContext) {
        },
        id: `${getSyntheticsPolicy(agentFullPolicy)?.streams?.[0]?.id}`,
        name,
+        origin: 'ui',


I am not sure why this value appeared when moving to the v2 registry but I can see that there are other tests which expect the origin field to be set so it seemed acceptable to add.

elasticmachine · 2022-07-29T14:40:16Z

Pinging @elastic/fleet (Team:Fleet)

cauemarcondes

APM changes LGTM

elasticmachine · 2022-07-29T16:21:54Z

Pinging @elastic/apm-ui (Team:apm)

x-pack/test/fleet_api_integration/apis/fleet_telemetry.ts

juliaElastic

LGTM, great work!

hop-dev · 2022-08-02T15:01:31Z

@elasticmachine merge upstream

kibana-ci · 2022-08-03T17:34:32Z

💚 Build Succeeded

Buildkite Build
Commit: eb53975

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`fleet`	867.4KB	867.4KB	+44.0B

History

💔 Build #62777 failed 8901c07
💚 Build #62451 succeeded 186ae3535bf55a3976f93482ba1751565f6b0049
💔 Build #62402 failed ac5637e5825be2d43e183b67355d299c0d95a056
💛 Build #61756 was flaky e097159d654a16512375274cb919a4a9546facdc
💔 Build #61736 failed 7d5e570f394cb06c9760d00a0b15a01ee5098cd5

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @hop-dev

kibanamachine · 2022-08-03T17:58:45Z

💔 All backports failed

Status	Branch	Result
❌	8.4	Backport failed because of merge conflicts You might need to backport the following PRs to 8.4: - [Fleet Synthetics] Upgrade to 0.10.1 (#137667)

Manual backport

To create the backport manually run:

node scripts/backport --pr 136947

Questions ?

Please refer to the Backport tool documentation

hop-dev · 2022-08-03T18:18:39Z

💚 All backports created successfully

Status	Branch	Result
✅	8.4

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…tic#136947) * add valid signature test package * add test signatures and readme for signature generation * mount zip packages as part of tests * amend README * Verified package test working * Rename valid to verified * Add test for unverified content * add test for package verified with wrong key * Check error types in 400 response * Check saved object keys as part of tests * Remove wrong_ keys * use release docker image * update package path for v2 registry * force install endpoint package * fix package policy upgrade on setup test * formatting * move back to production registry * Update all registry configs to use new package directory * use specific docker image not tag * fix agent policy tests * Get latest experimental endpoint version * skip impossible fleet test * update synthetics to use same registry image as fleet * fix telemetry tests * remove experimental flag from test config * add force install confirm to synthetics tests * add origin to expected policy data * add test subj to force install modal * Install latest fleet_server package not fixed version * install latest system pkg * fix types * fix deprecated API calls (cherry picked from commit 9f8a2c6) # Conflicts: # x-pack/test/fleet_api_integration/config.ts # x-pack/test/functional_synthetics/config.js

) (#138027) * add valid signature test package * add test signatures and readme for signature generation * mount zip packages as part of tests * amend README * Verified package test working * Rename valid to verified * Add test for unverified content * add test for package verified with wrong key * Check error types in 400 response * Check saved object keys as part of tests * Remove wrong_ keys * use release docker image * update package path for v2 registry * force install endpoint package * fix package policy upgrade on setup test * formatting * move back to production registry * Update all registry configs to use new package directory * use specific docker image not tag * fix agent policy tests * Get latest experimental endpoint version * skip impossible fleet test * update synthetics to use same registry image as fleet * fix telemetry tests * remove experimental flag from test config * add force install confirm to synthetics tests * add origin to expected policy data * add test subj to force install modal * Install latest fleet_server package not fixed version * install latest system pkg * fix types * fix deprecated API calls (cherry picked from commit 9f8a2c6) # Conflicts: # x-pack/test/fleet_api_integration/config.ts # x-pack/test/functional_synthetics/config.js

hop-dev added release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team labels Jul 22, 2022

hop-dev self-assigned this Jul 22, 2022

hop-dev commented Jul 22, 2022

View reviewed changes

x-pack/test/fleet_api_integration/config.ts Outdated Show resolved Hide resolved

hop-dev force-pushed the 133822-verification-int-tests branch 4 times, most recently from abd4ad2 to d5dc7d6 Compare July 28, 2022 14:01

hop-dev commented Jul 28, 2022

View reviewed changes

hop-dev force-pushed the 133822-verification-int-tests branch 3 times, most recently from 7ce8835 to fd18a23 Compare July 29, 2022 13:04

hop-dev commented Jul 29, 2022

View reviewed changes

hop-dev marked this pull request as ready for review July 29, 2022 14:40

hop-dev requested review from a team as code owners July 29, 2022 14:40

hop-dev requested review from pzl and dasansol92 July 29, 2022 14:40

cauemarcondes approved these changes Jul 29, 2022

View reviewed changes

botelastic bot added the Team:APM - DEPRECATED Use Team:obs-ux-infra_services. label Jul 29, 2022

juliaElastic reviewed Aug 1, 2022

View reviewed changes

x-pack/test/fleet_api_integration/apis/fleet_telemetry.ts Outdated Show resolved Hide resolved

juliaElastic approved these changes Aug 1, 2022

View reviewed changes

hop-dev force-pushed the 133822-verification-int-tests branch from e097159 to ac5637e Compare August 2, 2022 13:40

hop-dev added 2 commits August 3, 2022 13:22

add valid signature test package

786ad73

add test signatures and readme for signature generation

614b6e9

hop-dev added 9 commits August 3, 2022 13:23

fix telemetry tests

e856674

remove experimental flag from test config

1d23c88

add force install confirm to synthetics tests

af3aab0

add origin to expected policy data

17dd713

add test subj to force install modal

3031fe3

Install latest fleet_server package not fixed version

594bf25

install latest system pkg

2c15f1c

fix types

acc9ae3

fix deprecated API calls

8901c07

hop-dev force-pushed the 133822-verification-int-tests branch from 186ae35 to 8901c07 Compare August 3, 2022 12:24

pzl approved these changes Aug 3, 2022

View reviewed changes

hop-dev enabled auto-merge (squash) August 3, 2022 13:05

paul-tavares approved these changes Aug 3, 2022

View reviewed changes

hop-dev removed the request for review from dasansol92 August 3, 2022 13:09

remove origin from expected policy

eb53975

hop-dev merged commit 9f8a2c6 into elastic:main Aug 3, 2022

kibanamachine added v8.5.0 backport:skip This commit does not require backporting labels Aug 3, 2022

hop-dev added backport:prev-minor Backport to (9.0) the previous minor version (i.e. one version back from main) and removed backport:skip This commit does not require backporting labels Aug 3, 2022

hop-dev deleted the 133822-verification-int-tests branch August 3, 2022 17:55

hop-dev mentioned this pull request Aug 3, 2022

[8.4] [Fleet][Tests] Package install signature verification API tests (#136947) #138027

Merged

kibanamachine added the v8.4.0 label Aug 3, 2022

This was referenced Aug 4, 2022

[Observability][SecuritySolution] Fix to prevent observability style conflict in flyout across plugins #138091

Merged

[SecuritySolution][Bug] Fix to add empty values to timeline #138510

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet][Tests] Package install signature verification API tests #136947

[Fleet][Tests] Package install signature verification API tests #136947

hop-dev commented Jul 22, 2022 •

edited by kibanamachine

Loading

hop-dev Jul 28, 2022 •

edited

Loading

pzl Aug 2, 2022

paul-tavares Aug 2, 2022

hop-dev Aug 3, 2022

hop-dev Aug 3, 2022

pzl Aug 3, 2022

paul-tavares Aug 3, 2022

hop-dev Jul 29, 2022

elasticmachine commented Jul 29, 2022

cauemarcondes left a comment

elasticmachine commented Jul 29, 2022

juliaElastic left a comment

hop-dev commented Aug 2, 2022

kibana-ci commented Aug 3, 2022

kibanamachine commented Aug 3, 2022

hop-dev commented Aug 3, 2022

[Fleet][Tests] Package install signature verification API tests #136947

[Fleet][Tests] Package install signature verification API tests #136947

Conversation

hop-dev commented Jul 22, 2022 • edited by kibanamachine Loading

Summary

hop-dev Jul 28, 2022 • edited Loading

Choose a reason for hiding this comment

pzl Aug 2, 2022

Choose a reason for hiding this comment

paul-tavares Aug 2, 2022

Choose a reason for hiding this comment

hop-dev Aug 3, 2022

Choose a reason for hiding this comment

hop-dev Aug 3, 2022

Choose a reason for hiding this comment

pzl Aug 3, 2022

Choose a reason for hiding this comment

paul-tavares Aug 3, 2022

Choose a reason for hiding this comment

hop-dev Jul 29, 2022

Choose a reason for hiding this comment

elasticmachine commented Jul 29, 2022

cauemarcondes left a comment

Choose a reason for hiding this comment

elasticmachine commented Jul 29, 2022

juliaElastic left a comment

Choose a reason for hiding this comment

hop-dev commented Aug 2, 2022

kibana-ci commented Aug 3, 2022

💚 Build Succeeded

Metrics [docs]

Async chunks

History

kibanamachine commented Aug 3, 2022

💔 All backports failed

Manual backport

Questions ?

hop-dev commented Aug 3, 2022

💚 All backports created successfully

Questions ?

hop-dev commented Jul 22, 2022 •

edited by kibanamachine

Loading

hop-dev Jul 28, 2022 •

edited

Loading