Add CCv1 Image security design document

[jialez0]

This document describes the design of container image security under the confidential container V1 architecture, including the encryption-decryption and signing of container image. The components involved include image-rs, ocicrypt-rs, attestation-agent and Key Broker Service.

cc: @fitzthum @arronwy @stevenhorsman @jiazhang0 @jiangliu

[fitzthum]

This looks great. Very thorough. I have a few minor suggestions.

[fitzthum]

I like that the config files can optionally be provided in the rootfs. This will go nicely with the Secure Execution platform or any other approaches where the rootfs is confidential.

[jialez0]

In the mode described in the document, image-rs will preferentially read the configuration file under the local specified directory. If it is found that the config file does not exist, it will get it from AA. In the scenario where a trusted rootfs is provided, the configuration file in the local specified directory will be preset in advance, so the service interface of AA will not be called to obtain it. In this way, it can be compatible with the scenarios of trusted rootfs and untrusted rootfs

Do you mean that we need an explicit configuration option to configure whether image-rs is obtained directly from the local specified directory or from the service interface of AA?

[Jakob-Naucke]

@jialez0 Do you see it as a strict necessity that policy.json cannot be read by untrusted entities? I fully understand they should not be able to write it, but do we consider a list of allowed container images plus a public key for verification to be confidential data? I wouldn't understand what the threat is there.

[fitzthum]

Do you mean that we need an explicit configuration option to configure whether image-rs is obtained directly from the local specified directory or from the service interface of AA?

No I think it's good as it is. If someone doesn't want to use signatures they can just include the permissive config that you describe below in their initrd. This adds additional overhead for the basic user who just wants to deploy containers, but I guess it's fine.

[jialez0]

@Jakob-Naucke At present, ccv1, there are two distribution methods for policy files. In the protected boot image scheme, the user directly presets them in the rootfs of the boot image. In the unprotected boot image scheme, they are dynamically obtained from kbs through AA. KBS will distribute the policy only after AA's attitude evidence is verified, Sending through the encrypted channel is not to prevent untrusted entities from reading him, but to prevent malicious tampering during transmission. Of course, only MAC can also be used to protect the integrity of policy files, which depends on the security requirements of specific KBC and KBS.

[Jakob-Naucke]

Sending through the encrypted channel is not to prevent untrusted entities from reading him, but to prevent malicious tampering during transmission.

Shouldn't it be this then?

Thus, policy.json is confidential information that must be controlled by users and cannot be modified by untrusted entities.

[jialez0]

My statement here may not be accurate and should be modified as follows:

The policy file must be generated and maintained by the user. Additional security means need to be used to ensure that the policy file will not be modified by malicious entities during storage and transmission. If users have high requirements for security level, they even need to use additional means to ensure the confidentiality of policy files.

[c3d]

The implication of the above is that a local rootfs policy.json takes precedence over a remote AA one. This seems wrong to me. Consider if some particular registry has been compromised or simply moved/renamed, but the rest of your encrypted images are otherwise good (possibly at a new location). Wouldn't it be preferable to add a policy.json to your KBS that would supersede the local one, rather than force all rootfs to be re-generated?

[bpradipt]

@jialez0 thanks so much for putting this design document. Very helpful. Added few minor comments and questions.

[bpradipt]

/lgtm
Thanks @jialez0. According to me this looks good to start with.

[jodh-intel]

Thanks @jialez0 - a few comments.

Also, I think an overview diagram would make this easier to understand.

Finally, you might want to tal at https://github.com/kata-containers/kata-containers/blob/main/docs/Documentation-Requirements.md for ideas on how Kata docs handle API calls, filenames, etc.

[jodh-intel]

Are there some links to these opencontainers annotations? It's a shame they aren't terribly clearly named either ;(

[c3d]

In addition to the JSON output, could you give an example of how you generated it (i.e. the commands you used to actually generate that JSON, which I assume you did not create manually)?

[jodh-intel]

How is an image resolved if multiple registries are specified and they all contain the same image? Will the local image be used in preference to the registry version?

How does "generic matching" affect this? transports is an object rather than an array, so are the entries searched in random order and the first matching image used?

[jodh-intel]

Should this document show trust boundaries as by definition if a remote registry is configured, someone must trust it, even if the local host environment itself is not trusted.

Also, since a remote entity is being used, should the system require DNS over HTTPS/TLS?

[c3d]

Can a single registry store both an encrypted and unencrypted version of the same image? Sorry if this is a very naive question, but if I specify I want to run "busybox:latest", how does the image download service request the encrypted version?

• Will the registry deliver the encrypted version if it exists, and fallback to the non-encrypted version otherwise?
• Will the image download fail if the image is not encrypted?

Or is the intent that this will somehow fall under user control, i.e. busybox:latest+encrypted or something like that?

[jialez0]

@c3d The encrypted version and the non encrypted version of the container image are identical to two completely different images in the perspective of the registry. Only the owner knows the relationship between them. The owner can distinguish them by naming the container image. For example, the encrypted busybox:latest can be named busybox-enc-byxxx:latest.

[c3d]

@jialez0 Ah, I see, so a totally different name. OK, thanks.

[c3d]

Didn't we conclude at some point late last year that our security model might not be robust for images with no encryption at all? @fitzthum were you the one raising that point (sorry, my memory is a bit fuzzy)?

[jodh-intel]

I'm also unclear about images containing both encrypted and unencrypted layers. How can CC guarantee safety in the following sort of scenario?:

layer	encrypted?	signed?	dependent layer
1	yes	yes	n/a
2	no	no	n/a
3	yes	yes	2

[jialez0]

cc @jiazhang0

[c3d]

@jodh-intel To me, the guarantee that you have in that case is that you won't be able to execute anything in that image before you decrypted the encrypted layers, which requires data from the attestation itself. With only signed layers, at least conceptually, you could run the image without going through attestation.

[fitzthum]

Because we don't explicitly verify the identity of the KBS, signature verification is only guaranteed by the successful decryption of an encrypted layer. In other words, your workload is secure only if its execution is gated by the secret. Note that this is scoped at the pod level, so you can trust signed but unencrypted containers as long as something in the pod depends on depends on the secret. This should be a pretty good fit to workloads that use some containers that access secret data and some generic containers or one container with some generic layers and some secret layers.

Also note that we do signature verification at the container level, so I don't think we can have a mix of signed and unsigned layers within one container.

[jiazhang0]

I'm also unclear about images containing both encrypted and unencrypted layers. How can CC guarantee safety in the following sort of scenario?:
layer encrypted? signed? dependent layer
1 yes yes n/a
2 no no n/a
3 yes yes 2

First, image signing takes its effect on image manifest digest (in other word, the entire image) rather than an individual layer. You could refer to https://github.com/confidential-containers/documentation/issues/18 《What container image signing means for CC？》for more explanations about it.

[c3d]

@jiazhang0 Thanks for the pointer to the documentation. My question was not about the usefulness of encryption (I have no doubt about it 😄) but about the weakness of the attestation and workload process if we have only signature.

As I wrote in my comment, the risk is running a workload without a valid attestation when you don't need any secret to decrypt the image. @fitzthum reminded me of one easy method to achieve that, which is to insert a bogus KBS in the loop.

As he also pointed out, image decryption is not the only step that may require a secret. If the workload has some encrypted volume being used only by super-standard containers (e.g. some mysql), then arguably signature is sufficient, since what you really want to protect is the data, and that data is not accessible without a secret.

In short, I'm trying to evaluate if supporting a non-encrypted signed image makes sense or not. At the moment, I believe there are valid use cases, as long as it does not break the rest

[jiazhang0]

As I wrote in my comment, the risk is running a workload without a valid attestation when you don't need any secret to decrypt the image. @fitzthum reminded me of one easy method to achieve that, which is to insert a bogus KBS in the loop.

To accurately answer this, we need to distinguish the use cases according to boot image strategy. For protected boot image strategy, the TCB state of Pod is trusted. It is safe to verify a signed but unencrypted image (w/o sensitive data) because all artifacts used to verify the signature of image and the verification result are all trusted. For unprotected boot image strategy, the initial TCB state of Pod is unverified, so the initial image to be pulled must be an encrypted, which will enforce an remote attestation to let remote party verify the TCB state of Pod and then unseal the decryption key. Once the TCB state of Pod was verified, the following images to be pulled can be unencrypted as long as it doesn't contain sensitive data.

So the TCB state of Pod is important to conclude whether unencrypted image has weakness or not. Essentially speaking, whether the verification result of signed image is trusted is determined by the TCB state of Pod. If it is unverified or untrusted, the verification result or even whether the signed image is actually occurred is doubtful. The introduction of image encryption can introduce an opportunity to check the TCB stage of Pod.

In short, I'm trying to evaluate if supporting a non-encrypted signed image makes sense or not. At the moment, I believe there are valid use cases, as long as it does not break the rest

Yes. For sidecar containers, it doesn't contain sensitive data, so it is not necessary to be encrypted.

[magowan]

We did talk in the one of calls about the possibility in the future of having another component which might have wider awareness of protected v unprotected boot strategy, TCB state of the POD, status of all container images so far with respect to signatures and encryption and where signatures/keys originated from (i.e. are they all unencrypted so far or has one been decrypted).
This component can combine all this knowledge to take a view if the Pod has any unaddressed risk that might impact security/confidentiality.

Bottom line is that something somewhere needs to depend upon a key that is either secure in hardware (closer to protected bootimage) or provided by KBS as part of the attestation flow (could be for container decryption but could also be key for data decryption - application won't work without the data). If all these are untrue then we have a problem, if one is true then answer could be "depends"

[jiazhang0]

We did talk in the one of calls about the possibility in the future of having another component which might have wider awareness of protected v unprotected boot strategy, TCB state of the POD, status of all container images so far with respect to signatures and encryption and where signatures/keys originated from (i.e. are they all unencrypted so far or has one been decrypted). This component can combine all this knowledge to take a view if the Pod has any unaddressed risk that might impact security/confidentiality.

I guess that component is Root of Chain for Measurement (RTM), consisting of the initial measurement, dynamic measurement or vTPM (w/o dynamic measurement), and event log.

[c3d]

When would a local dir be used? Is that with the intent of ultimately sending it to some network-accessible sigstore, or is there any way to use a locally-stored signature claim? Am I correct that it would be OK to use some local file if the policy allows it, which could be useful for debugging?

[jialez0]

sigstore can be a remote server or a specified local directory. The owner decides which one to use.

For example, in the protected boot image scheme of CC, the local rootfs of pod is trusted, so the owner can place the signatures in the rootfs of boot image. When verification is needed, image-rs can directly obtain the signatures from the local directory

[c3d]

That sentence seems to imply that you have to decrypt all layers before applying them. I believe it would be OK to apply each layer immediately after decrypting it (and earlier layers)?

[c3d]

Hmm, again this idea that there is a per-layer annotation. So I guess that must exist somewhere. A link would help.

[jialez0]

Annotation here refers to annotations.md, annotations can be flexibly placed in many locations in the manifest, such as:

{
    "schemaVersion": 2,
    "config": {
        ...
    },
    "layers": [
        {
            "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip,
            "digest": "sha256:f69ae40...",
            "size": 204812,
        }
    ]
   "annotations": {
                ...
    }
}

Or:

{
    "schemaVersion": 2,
    "config": {
        ...
    },
    "layers": [
        {
            "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip+encrypted",
            "digest": "sha256:f69ae40...",
            "size": 204812,
            "annotations": {
                ...
            }
        }
    ]
}

The latter is what we call pre-layer annotations.

[bpradipt]

@jialez0 @sameo @c3d I think this version should be merged since there is in-principle approval to the design and PR has landed as well. Any subsequent updates can be handled as new PR. Wdyt ?

[c3d]

@jialez0 @sameo @c3d I think this version should be merged since there is in-principle approval to the design and PR has landed as well. Any subsequent updates can be handled as new PR. Wdyt ?

Actually, I had already given my approval on that basis, stating that improvements could be done later.

[sameo]

@fitzthum @jodh-intel Are you good with that one?