[Fleet] Verify integrations using package signatures from EPR and public key bundled with Kibana #133822
Comments
kpollich
added
the
Team:Fleet
|
Pinging @elastic/fleet (Team:Fleet) |
True. Maybe let's download each version once and keep it in the cache until we get a different version of the package from EPR. For the first time, it will be a lot of bandwidth and compute consumption. We will have to verify it somehow before a user tries to install it.
No. That would create chaos and confusion because users may not be aware that we have released such a feature. Anything installed after 8.4 upgrade should be flagged as unverified if those packages fail the validation. |
|
I found some fairly helpful documentation around some of my questions https://github.com/elastic/elastic-package/blob/main/docs/howto/use_package_storage_v2.md This doc details some of the usage of the package storage v2 API, which we'll need to leverage here in order to fetch signature files. Reading through it covered two of my questions:
I think the verification process will look something like this:
Verification will look something like $ gpg --no-default-keyring --keyring /path/to/pubkey.gpg --verify ${packageName}.zip.sig ${packageName}.zipWe'll need to use a JS implementation of GPG e.g. https://www.npmjs.com/package/openpgp#create-and-verify-detached-signatures. I don't think this is something that Kibana has currently, just from a cursory look around at dependencies and a quick grep. @joshdover might know better.
It won't be deployed to the legacy EPR API - it exists only in package storage v2. |
For SaaS, no we don't need that. For ECE, yes, we need this capability because I believe customers would not be bothered about the outdated public key for their stack but as soon as they upgrade their EPR, every tile would become yellow for them. Having said that, it's not as urgent as the signature validation because we are not going to restrict anyone from installing unverified packages and I don't think we will rotate our key pair anytime soon. So, it doesn't have to be now and we can add it in the next phase if the scope is enormous for it. |
|
I wrote a POC script to verify package signatures using a JS based PGP implementation called I roughly followed their docs and came up with this import * as openpgp from "openpgp";
import fetch from "node-fetch";
const ELASTIC_GPG_KEY_URL =
"https://artifacts.elastic.co/GPG-KEY-elasticsearch";
const EXAMPLE_PACKAGE = "elastic_agent-1.3.1";
const BASE_PACKAGE_STORAGE_URL =
"https://package-storage.elastic.co/artifacts/packages";
const EXAMPLE_PACKAGE_ZIP_URL = `${BASE_PACKAGE_STORAGE_URL}/${EXAMPLE_PACKAGE}.zip`;
const EXAMPLE_PACKAGE_SIG_URL = `${EXAMPLE_PACKAGE_ZIP_URL}.sig`;
export function streamToBuffer(stream) {
return new Promise((resolve, reject) => {
const chunks = [];
stream.on("data", (chunk) => chunks.push(Buffer.from(chunk)));
stream.on("end", () => resolve(Buffer.concat(chunks)));
stream.on("error", reject);
});
}
try {
const key = await fetch(ELASTIC_GPG_KEY_URL).then((res) => res.text());
const publicKey = await openpgp.readKey({ armoredKey: key });
const zip = await fetch(EXAMPLE_PACKAGE_ZIP_URL).then((res) =>
streamToBuffer(res.body)
);
const signature = await fetch(EXAMPLE_PACKAGE_SIG_URL).then((res) =>
res.text()
);
const message = await openpgp.createMessage({
binary: zip,
});
const parsedSignature = await openpgp.readSignature({
armoredSignature: signature,
});
const verificationResult = await openpgp.verify({
verificationKeys: publicKey,
signature: parsedSignature,
message: message,
});
const { verified, keyID } = verificationResult.signatures[0];
await verified;
console.log("Signed by key id", keyID.toHex());
} catch (e) {
console.error("Unable to verify signature");
console.error(e);
}
I'm assuming the Key ID I get here in some way corresponds to the one listed on https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html based on these digits:
I've never really used or looked at GPG stuff so this is all pretty foreign to me. Hopefully I'm on the right track. Maybe one of the folks who worked on the package signature implementation would be of help to sanity check this? |
I don't think downloading every package will ever be practical, especially as we scale to 1000k+ integrations. Some packages are already 100MB+ and this is going to consume a large amount of bandwidth to have every Kibana instance do this, both for the customers and for our registry CDN. Instead, I think we should only verify these packages on install or maybe do it lazily when the user opens the integration details view for a single integration. If the package fails verification during install, we should give the user an opportunity to continue anyway or abort. Admins should probably have the ability to not allow unverified packages at all, but maybe we can add this as a follow up feature. How we expose this setting is unclear to me, because we'd only want cluster admins to be able to modify it, so I think
IMO it'd be ok to have a flow similar to how we download bundled packages to package the latest public key from artifacts.elastic.co/GPG-KEY-elasticsearch into a file that Kibana can read. Ideally this is integrated as a
Could we start with a |
|
@elastic/kibana-security Any feedback on our plan to use opengpgjs to verify zip signatures? #133822 (comment) |
|
@joshdover I agree that we should only verify on install in the first instance. Presumably it will be very rare that there is a verification issue, and we will be wasting a lot of bandwidth even by lazily verifying the package on the package details page. Could we return an error on the backend if the verification fails, then add a URL param that allows users to ignore validation errors, the UI could then give the user the choice to proceed: sequenceDiagram
autonumber
participant User
participant UI
participant Backend
User->>UI: Install button clicked
UI->>Backend: POST /epm/packages/mypkg-1
Note right of Backend: Donwload package
Note right of Backend: Verify package
Note right of Backend: Verification fails
Backend->>UI: 400 Package Verification Failed
UI->>User: Display error modal
Note right of User: This package is unverified <br/> continue | cancel
User->>UI: Continue button clicked
UI->>Backend: POST /epm/packages/mypkg-1?allowUnverified
Note right of Backend: Donwload package
Note right of Backend: Verify package
Note right of Backend: Verification fails
Note right of Backend: Install continues
Backend->>UI: 200 Installation successful
UI->>User: Installation complete
|
|
@elastic/ecosystem - Does anyone have any feedback on our understanding of how we'll verify package signatures in Kibana here? I wrote a quick POC using https://openpgpjs.org/, but we've got some open questions about specifics, e.g.
Also, the concerns raised around verifying all integrations being an expensive, tough to scale operation would be great to get feedback on as well. |
|
Hi Kyle, let me respond to your questions.
As we're controlling the package-storage, we can upload and manage Elastic keys there. On the other hand, I'd rather use the same source of keys, so if there is any recommended place by the Infra team, I would go for that place. We haven't received any recommendations around this or key invalidation.
In our case this is MIT, but if you prefer we can change the library/implementation. It shouldn't hurt as that much at this moment.
We use https://package-storage.elastic.co/ for storing artifacts, those will be also accessible via https://epr.elastic.co/, including signatures. EPR will stream content to the CDN. Maybe we can introduce another property To be honest, I thought that Kibana will be still reaching out to EPR and the EPR will stream content from https://package-storage.elastic.co/. I'm not sure if Kibana should be aware of https://package-storage.elastic.co/ (hardcoded). |
Happy to fetch a key from package storage instead. I think we're both blocked by infra's recommendation here then. Our logic will be to fetch the key from wherever -> store it in something like a git ignored
I opened https://github.com/elastic/open-source/issues/298 to hopefully get approval from legal to use the JS OpenPGP package. It'd be great to use this package because then both Kibana and Package Storage are using PGP implementations maintained by ProtonMail. The sync on the same package maintainer across languages feels like a solid implementation choice.
This would be preferred on our end too, so I'm happy to move forward with that assumption. However I'm not seeing signatures available via EPR today e.g. https://epr.elastic.co/epr/elastic_agent/elastic_agent-1.3.1.zip.sig. Is this work still p ending and if so where we can track it? Thanks a ton for your response, @mtojek - really cleared a few things up here 👍 |
Yes, my main concern is lack of procedure in case of invalidation, but we have an issue to focus on this problem.
This is because signatures are valid only for .zip packages, and current setup of the Package Storage operates on unpacked packages (like in the Git repository). Once we switch to the bucket storage indexer (connects to GCP buckets), you will see all signatures. Do you need something for Fleet development purposes now? I guess that we can arrange it. |
It'd be great to be able to fetch signatures and the proper package |
agreed with Kyle above, something as close to how it will be in production would be great, even if its the ability to set up an EPR locally with the new structure for dev. |
|
@kpollich regarding:
Here is my breakdown of the pros/cons of build vs bootstrap: Build step:
Bootstrap step:
I think there could also be a hybrid approach where it is bundled but we also check for an updated key. I am not sure which is better from a security standpoint, e.g could requests be intercepted/bundles be messed with? |
I think this is okay as long as we allow for overriding the bundled key via
I'm in agreement here. I think what we want to do is something like The important thing with generating this during bootstrap as well is that we need to allow for package verification to run in development. I don't think bootstrap is relevant for production builds - only development. |
This is a pretty important detail, should we even keep spending time here right now if the registry doesn't support what we need? My understanding is that storage v2 is not necessarily shipping in 8.4, which is what I think @mtojek is talking about. |
|
@mtojek is there an issue for this "storage v2" work we can link to?:
And is Josh right that it won't be in 8.4? |
|
@hop-dev This is the meta issue we use to track the work: It should be done in 8.4 but may slip through a little bit. I'll let @mtojek answer as soon as he is back. |
|
@dborodyansky will you have some availability in the next couple of weeks to create a design for the following scenario (diagram above)?
let me know if there is any more info you would need 👍 |
|
Yesterday, @akshay-saraswat and @kpollich agreed to sync up and figure out how much progress the UI team can make before this is available in the production registry. Until then, I think we can defer on any additional engineering effort here. |
|
I'd rather keep works on both sides (Package Storage, Fleet) loosely coupled as the contract is pretty clear. There are many steps until we deploy it in production:
Many unpredicted blockers might occur until we deploy the improved EPR to production. If possible I would keep the signature verification optional in Fleet - enabled if PS. We have one person working on this stuff at the moment, interchangeably with the "input package" which has a higher priority. |
Thanks for clarifying the state of the project, @mtojek. I am in agreement with @joshdover here then, and we should defer most engineering work on the Fleet UI side until we can retrieve package signatures from EPR.
I think this approach is valid, but we can't realistically develop the signature verification process if @akshay-saraswat to address what engineering work can be done on the Fleet UI side of things here in the meantime:
Other than these two things I'd really rather defer on "wiring up" the signature verification process to the package registry until https://epr.elastic.co is serving package signatures at least in a staging/snapshot environment we can use during development. I think it'd be fine to use Marcin's suggestion of "only apply verification if the cc @jlind23 as well. |
|
We discussed this today in the weekly meeting. We'll capture the final plan next week when Marcin is back from PTO. My 2 cents The Fleet UI work can be divided into the following three items:
From the threat assessment report recommendations and enterprise-readiness requirements perspective, the priority for this work is pretty high. We must aim to deliver it as soon as possible. I agree with what @kpollich mentioned above. IMO, we should start working on the last two items while the dependencies for # 1 are handled by the ecosystem team during 8.4. |
|
@akshay-saraswat - I'm glad we've some time on the calendar now to align here, but I wanted to just address your comment here before we sync up next week. I captured how I'm understanding the state of work and what's blocking what here in this quick sketch:
We can't do this until EPR is wired up to the new Package Storage v2 GCP buckets, so considering Fleet UI blocked here
We could technically implement a service/module that takes in a package
Similar to number 2 above, we could deliver these UI's in an "isolated" development pattern using mock data or Storybook, but we won't be able to develop against real package data until number 1 is addressed. Completely agree on the priority here. Looking forward to chatting further next week and clearing up dependencies here. |
|
Hi folks,
As soon as the docker image is ready, @mtojek will provide an update here. |
|
Here is a freshly baked Docker image: Remember to docker-auth before :) |
|
FYI We started publishing more fresh package Package Storage distributions:
|
|
@amolnater-qasource I've written a manual testing guide here Here are the test files needed |
|
Hi @hop-dev We will be revalidating this at our end and will be sharing results here. |
|
Hi @hop-dev We had below observations:
Screenshots: Other Observations:
Screenshot: Build details: Please let us know if we are missing anything here. |
Unverified labels shouldn't show in the browse integrations view ✅ |
|
Hi Team
Please let us know if we are missing any scenario here. |
|
Hi Team We have executed 02 testcases for this feature under our Fleet Test run at link: We have observed no issues while revalidating this feature on 8.4 BC5 self-managed environment. Build details: As the testing is completed on this feature, we are marking it as QA:Validated. Thanks |
amolnater-qasource
added
QA:Validated
Summary
We want to provide Kibana users with confidence that the integrations they're installing have not been corrupted or otherwise tampered with. To facilitate this, we're adding support for signing packages: elastic/package-registry#728.
In the integrations UI, we'll need to add elements that indicate whether an integration is verified or not, as well as a service to handle verifying integrations using a public key bundled with Kibana
Implementation
kibana.ymlin cases where a new public key should be used.zip.sigfile published in Package Storage v2 e.g.https://package-storage.elastic.co/artifacts/packages/${packageName}-${packageVersion}.zip.sigDisplay a badge for each unverified integration on the "Browse integrations" grid.ziparchive in order to verifyopenpgpNPM moduleopenpgppackageDesigns
Show designs
Open questions
Answered questions
.zipfile for every integration in existence?8.4.0, will we be showing that the installed integration is unverified until they're updated? If I upgrade Kibana to 8.4, do all of my installed integrations get flagged as unverified?kibana.ymlsignature_pathreturned from EPR?signature_pathfield mentioned here to be published in EPR? e.g. https://epr.elastic.co/package/fleet_server/1.1.1/.sigfiles mean we should also switch our zip download logic to use https://package-storage.elastic.co/ instead of https://epr.elastic.co/?openpgpjsin https://github.com/elastic/open-source/issues/298References
The text was updated successfully, but these errors were encountered: