Signing packages: prepare procedure for revoking compromised certificates

[mtojek]

The goal of this issue is to describe and describe steps to perform when the Elastic signing key for packages is leaked.

Related questions:

1. Should we prepare a revocation list of keys with dates?
2. Should resign all packages (consider the scaling problem)?
3. Where should we keep public keys and CRL, so that they can be accessible for running Kibana instances (Fleet)?

Related issue:
elastic/package-registry#728

[joshdover]

@jen-huang We'll need to wait for this to be completed before we can ship the package signature verification feature

[jen-huang]

@mtojek @mrodm We've merged the UI changes for verifying signatures for 8.4. Does the ecosystem team still intend to complete this documentation task in tandem with 8.4?

[mtojek]

I guess that we can start working on this then. Based on the delivery schedule, it looks like the 8.4.0 ended up yesterday, so we will slip it to 8.5.0.

@jen-huang Do you have preferences regarding these questions or should we propose something:

1. Should we prepare a revocation list of keys with dates?
2. Should resign all packages (consider the scaling problem)?
3. Where should we keep public keys and CRL, so that they can be accessible for running Kibana instances (Fleet)?

[jen-huang]

I will let @hop-dev and @kpollich weigh in on the above... :)

From what I understand, although we have completed work on our side, package storage v2 still needs to be "turned" on for this feature to go live. Let's make sure we make sure we get this process in place prior to flipping the switch. cc @jlind23

[mtojek]

From what I understand, although we have completed work on our side, https://github.com/elastic/ingest-dev/issues/1040 still needs to be "turned" on for this feature to go live. Let's make sure we make sure we get this process in place prior to flipping the switch.

Package Storage v2 is already populated with signatures and technically ready to switch. We have these blockers to close before proceeding:

• Adapt apm package to package storage v2 apm-server#8491
• [Fleet] Test Kibana with package storage v2

I pinged also the Infosec team to comment on the revocation procedure as it's a global Elastic-wide case.

On our side, we can prepare a job that will regenerate signatures for all packages in the storage (step 2). The job will depend on the Elastic signing job, so the private key must be renewed there.

[hop-dev]

@mtojek I'll just describe the situation as it stands and we can see if any changes need to be made.

Currently we download the public key from https://artifacts.elastic.co/GPG-KEY-elasticsearch as part of the kibana build process (here). We store the SHA512 of the key so if the key were to change, the build would break until we updated the SHA.

The location of the key can be changed by config xpack.fleet.packageVerification.gpgKeyPath so if a key were to be updated then the new one could be downloaded and an existing kibana could be configured to use it.

We then store the ID of the key against every package we have verified e.g:

verification_status: 'verified',
verification_key_id: 'abcd12345'

Currently we only display a package as unverified (see below for what this looks like) if:

• it failed signature verification
• OR if the key ID it was verified with doesn't match the current elastic public key ID (@kpollich as I've written this out I think this may be the wrong stance, the package was verified when they installed it, and the key has since been invalidated, does that make the package verification status "unknown" as opposed to unverified)

So as it stands if the key changed then all of a users verified packages would show as unverified until they were upgraded or reinstalled.

[mtojek]

Currently we download the public key from https://artifacts.elastic.co/GPG-KEY-elasticsearch as part of the kibana build process (here). We store the SHA512 of the key so if the key were to change, the build would break until we updated the SHA.

👍

So as it stands if the key changed then all of a users verified packages would show as unverified until they were upgraded or reinstalled.

What will happen if we decide to resign the same .zip package?

Currently we only display a package as unverified (see below for what this looks like) if:

Will Fleet show the "unverified" notification while interacting with the Package Storage with signature paths? Let's say we decide to switch back to v1 (no signatures present).

[hop-dev]

What will happen if we decide to resign the same .zip package?

We do have a reinstall button now which would re-verify them, but it would be a manual process. I think we may need specific instructions in the UI for the case where the key has been invalidated as opposed to just telling them the package is unverified.

Will Fleet show the "unverified" notification while interacting with the Package Storage with signature paths? Let's say we decide to switch back to v1 (no signatures present).

If the package does not have a signature file then we do not perform verification. The verification status is stored on the package installation saved object. So if I installed a package from the v2 registry, then we switch back to v1 it wouldnt show as unverified as it would still have the verification details from when it was first installed.

[mtojek]

If the package does not have a signature file then we do not perform verification. The verification status is stored on the package installation saved object. So if I installed a package from the v2 registry, then we switch back to v1 it wouldnt show as unverified as it would still have the verification details from when it was first installed.

👍

We do have a reinstall button, which would re-verify them, but it would be a manual process. I think we may need specific instructions in the UI for the case where the key has been invalidated as opposed to just telling them the package is unverified.

I see a possibility that we will resign 1-2 packages for testing purposes or by accident, but we don't revoke the old signature. I guess that Fleet UI won't notify the user that the package signature has changed in the meantime, right?

[mrodm]

@hop-dev Writing the doc about the revoke/resign procedure that it should be followed, I was wondering if the "Unverified" badge/status is shown in the tiles in these two scenarios:

• a package with no signature
• a package with a bad signature

If not, is there a specific badge/status for the second scenario?

[jlind23]

Adding @kpollich and @jen-huang for awareness

[mrodm]

@hop-dev Writing the doc about the revoke/resign procedure that it should be followed, I was wondering if the "Unverified" badge/status is shown in the tiles in these two scenarios:

• a package with no signature
• a package with a bad signature

If not, is there a specific badge/status for the second scenario?

@hop-dev are both cases/scenarios shown as "unverified" in the UI?

[hop-dev]

@mrodm Sorry I missed your first message i have been on leave.

• a package with no signature - package does not show as unverified (currently no visual indication that the signature was missing)
• a package with a bad signature - package shows as unverified

[mrodm]

Thanks @hop-dev !