CCv0: Discussion options for removing the CC branch and running confidential-containers out of kata-containers main

[stevenhorsman]

For the confidential containers project our end goal was always to merge code into the main branches of kata-containers and tests, but whilst we were doing PoCs in CCv0 we decided the simplest approach was to work on a separate branch. As we entered the 'CCv1' roadmap we thought that that fact we were using a containerd fork blocked this, but as the burden of managing two development branches is continuing we want to give it some attention and work out what the options are and what is viable.

Here is the updated plan from my perspective, but it might well be wrong, or incomplete, so other input is welcome:

Our favourite suggestion so far is to have all the content for 'normal' kata and 'CC' kata coming from the same main branch, but provide a build config to create different binaries for the different versions.

Description

• Add a PR label, or different Jenkins job configuration with a CC parameter, so that when the build/test/release? jobs run the scripts know whether they are building/testing the CC version, or normal
• If CC 'flag' is on:
  • we use the approach in

    tests/.ci/install_kata_image.sh

    Lines 52 to 54 in d70430e

    	if [ ${CCV0} == "yes" ]; then
    	sudo -E USE_DOCKER="${use_docker:-}" DISTRO="${distro}" UMOCI=yes make -e "image"
    	else

    to add in the CC components to the guest image e.g. attestation-agent, umoci (whilst still required)
  • we add a new cc-containerd external dependency in version.yaml, similar to https://github.com/kata-containers/kata-containers/blob/46522a3e46fcb44ce8bb22c35da31f7ff86329fd/versions.yaml#L187-L198 which points to the fork and use the logic from

    tests/.ci/install_cri_containerd.sh

    Lines 53 to 68 in d70430e

    	install_from_branch() {
    	containerd_repo=$(get_version "externals.containerd.url")
    	warn "Using patched Confidential Computing containerd version: see https://${containerd_repo}/tree/${containerd_branch}"
    	echo "Trying to install containerd from a branch"
    	(
    	go get -d "${containerd_repo}"
    	cd "${GOPATH}/src/${containerd_repo}" >>/dev/null
    	git fetch
    	git checkout "${containerd_branch}"
    	sudo -E PATH="$PATH" make BUILD_TAGS="${BUILDTAGS:-}" cri-cni-release
    	# SH: The PR containerd version might not match the version.yaml one, so get from build
    	containerd_version=$(_output/cri/bin/containerd --version | awk '{ print substr($3,2); }')
    	tarball_name="cri-containerd-cni-${containerd_version}-${CONTAINERD_OS}-${CONTAINERD_ARCH}.tar.gz"
    	sudo tar -xvf "./releases/${tarball_name}" -C /
    	)
    	}

    to install that version for the integration tests
• Find a solution to the forked containerd problem. This needs some investigation, but the approach that we've discussed is:
  • Rename the confidential-containers fork of containerd to have a different module name e.g. containerd-cc
  • Pull both copies of the containerd into the vendor directory
  • Use something like go build tags, or go plugins to switch between the 'main' and 'CC' versions of containerd based on the CC build parameter value when building the kata runtime. We might want to add a wrapper layer that does this switch to get the code cleaner?
• Partition up the current CCv0 changes into manageable chunks, ensuring that they have the appropriate level of testing required and creating PRs to go into main (carefully ensuring we keep the DCO history and don't break the CCv0 branch).
  • We can potentially start some of this on the tests repo, and/or the 'guest'/agent side once image-rs has been added without the containerd runtime changes. At this point it won't be much use (though we can add agent component tests using agent-ctl maybe?), but will close the gap at least and get us starting the conversation with the wider kata community

[wainersm]

Hi @stevenhorsman , just brainstorming here...maybe we don't need to duplicate the CI jobs. Keep the current Kata jobs building and testing the "normal" Kata Containers; then add just one new job for CC specific tests.

Suppose that the new job configuration is called CC_CRI_CONTAINERD then basically we need this:

diff --git a/.ci/ci_job_flags.sh b/.ci/ci_job_flags.sh
index 0eb255dc..eb8202b8 100755
--- a/.ci/ci_job_flags.sh
+++ b/.ci/ci_job_flags.sh
@@ -93,13 +93,21 @@ case "${CI_JOB}" in
        export KATA_HYPERVISOR="qemu"
        export KUBERNETES="no"
        ;;
-"CRI_CONTAINERD"|"CRI_CONTAINERD_K8S")
+"CRI_CONTAINERD"|"CRI_CONTAINERD_K8S"|"CC_CRI_CONTAINERD")
        # This job only tests containerd + k8s
        init_ci_flags
        export CRI_CONTAINERD="yes"
        export CRI_RUNTIME="containerd"
        export KATA_HYPERVISOR="qemu"
-       [ "${CI_JOB}" == "CRI_CONTAINERD_K8S" ] && export KUBERNETES="yes"
+       case "${CI_JOB}" in
+               "CRI_CONTAINERD_K8S")
+                       export KUBERNETES="yes"
+                       ;;
+               "CC_CRI_CONTAINERD")
+                       # Export any CC specific environment variables
+                       export CC="yes"
+                       ;;
+       esac
        ;;
 "CRI_CONTAINERD_K8S_INITRD")
        # This job tests initrd image + containerd + k8s
diff --git a/.ci/run.sh b/.ci/run.sh
index 8d27f4c8..4580e297 100755
--- a/.ci/run.sh
+++ b/.ci/run.sh
@@ -55,6 +55,9 @@ case "${CI_JOB}" in
                echo "INFO: Running ksm test"
                sudo -E PATH="$PATH" CRI_RUNTIME="containerd" bash -c "make ksm"
                ;;
+       "CC_CRI_CONTAINERD")
+               echo "INFO: Running Confidential Container tests"
+               sudo -E PATH="$PATH" CRI_RUNTIME="containerd" bash -c "make cc-tests"
        "CRI_CONTAINERD_K8S_COMPLETE")
                echo "INFO: Running e2e kubernetes tests"
                sudo -E PATH="$PATH" CRI_RUNTIME="containerd" bash -c "make kubernetes-e2e"

The makefile target cc-tests (just an example, ok?) will call the CC specific integration tests.

[stevenhorsman]

Hi @stevenhorsman , just brainstorming here...maybe we don't need to duplicate the CI jobs. Keep the current Kata jobs building and testing the "normal" Kata Containers; then add just one new job for CC specific tests.

Sorry, I'm probably missing something obvious, but how do we test all ~12 os/cri/hypervisor combinations that Kata supports in this way if we only add 1 new job config?

[wainersm]

Hi @stevenhorsman , just brainstorming here...maybe we don't need to duplicate the CI jobs. Keep the current Kata jobs building and testing the "normal" Kata Containers; then add just one new job for CC specific tests.

Sorry, I'm probably missing something obvious, but how do we test all ~12 os/cri/hypervisor combinations that Kata supports in this way if we only add 1 new job config?

It was just an example. The way that Kata CI is designed, one need to created entries on .ci/ci_job_flags.sh corresponding to the combinations of CRI/Hypervisors to be tested. The .ci/run.sh script contain the instructions to run the suite of tests for each combination. And finally we mix that with different Jenkins jobs, resulting on the various combinations of os/cri/hypervisor we want to test.

[stevenhorsman]

Update: on the Architecture Committee meeting today (15th Feb) we brought up the issue of getting confidential containers code merged into main. My take on it was that everyone was generally on board and we didn't get push-back on the idea of merging it in general (we pitched having a CC build config and using that to drive the differences in the build e.g. whether the attestation-agent gets included in the rootfs).
The main question is still about the containerd piece. Our best idea (from Fidencio and Samuel) so far is to re-name the confidential-containers fork of containerd to have a different module name e.g. containerd-cc or something, so then we could pull in both copies of the code into the vendor directory.
We could then create a containerd wrapper layer than switches between the main implementation and cc implementation based on the build parameter (similar to an #ifdef). My golang is very basic, but will a quick google, it looks like it might be possible with go build tags. I think the next step is to try a PoC and see how it works.

[stevenhorsman]

Wainer has also suggested that go plugins might be able to help as an alternative to build tags

[stevenhorsman]

I think the single release option is no longer being considered, so I've removed it from the description for clarity, but I'll paste it here for reference:

Single release

Description

• The CC extras e.g. attesation-agent, umoci etc are built into all kata guest images, but only activated if the CC settings are enabled in containerd and kata’s config toml files

Pros:

• Avoids all the merging we currently have to do working on a separate branch
• Only a single release, so easy for users to pick the correct version
• No/limited custom build flags

Cons:

• All the work done in Kata 2 to make the image smaller is undone and ‘non-CC’ kata users get extra bloat
• Could users more easily end up with a misconfigured runtime and think they are confidential, when they aren't?

Common issues:

Other

• Some/most of the CCv0 code is missing unit tests, so work is needed to improve them before merging (though much of the changes are going to be re-done with the image-rs crate

Questions

• How do we handle that CC uses different containerd vendor code that kata runtime pulls in
  • Can we dual vendor the same package (as long as we ensure that our cc fork on containerd keeps in step with the version that main pulls in
  • How long before equivalent function gets upstream in containerd
• Can we use the existing experimental feature of kata? https://github.com/kata-containers/kata-containers/tree/main/src/runtime/virtcontainers/experimental how does it work/apply?

[jodh-intel]

Thanks for raising this @stevenhorsman.

A few thoughts and questions. Note: these are not directed at you, but at the entire community...

Assumptions

• Minimal impact on "vanilla Kata":

  • User experience: Kata will continue to work as it does today with no additional effort required on the behalf of users/admins.

  • Performance/metrics: The overall system performance using the default Kata configuration will not be negatively impacted by these changes.

ifdef build solutions

If any sort of #ifdef plan is required, we're going to have to be extremely careful to avoid packaging mistakes and to ensure we can discriminate between the two versions. As a mimimum we will need both containerd-shim-kata-v2 --version and the announce function to log details of how that version of the runtime has been built.

Containerd fork

It would be useful if we could summarise:

• Why a containerd fork is needed.
• How the fork differs to upstream.
• How we plan to maintain that fork (wrt security fixes, etc).
• How long we are comfortable maintaining such a fork (it's potentially a huge cost).
• What the stumbling blocks and/or timeline for getting the fork changes merged upstream.

Avoid confusion

How is all this going to be documented?

It's seems odd to document CC functionality in Kata repos, but if that's where the code lives, we will have to atleast create a set of links back to the actual docs in an appropriate CC repo.

Unless handled carefully, it's going to be rather unclear to users and contributors why there are two separate GitHub organisations imho.

Merge periodicity

Is this a "one off" merge or is there a possibility of further CC code syncs into Kata?

If there is a possibility of this happening (and I can imagine there is), we need to consider some sort of merge window rules now. For example, we don't want to land a lot of CC changes in Kata close to the date of new Kata releases (even if the CC code isn't being actively used by Kata).

Maybe something as simple as this would suffice?:

• No CC merges into Kata a week before a major Kata release.

We could also consider something like this:

• If the Kata AC agrees, a CC merge could still occur close to a Kata release on the assumption that the Kata release was delayed to give sufficient time for testing, etc.

/cc @kata-containers/architecture-committee for further input.

[stevenhorsman]

Thanks so much for the feedback @jodh-intel - I really appreciate the feedback and this is something that we all have to do together and be happy with if we are going to be successful.

In the Kata CC meeting, I took an action to break this issue down into separate issues for different components, but I think it is worth having the discussion here first as it might get some of the higher-level topics dealt with first before we worry about the specific code merges. I definitely don't have all the answers and as you've said it's a discussion for the whole community, but I'll give my initial reaction to some of the questions raised.

Assumptions

• Minimal impact on "vanilla Kata":

  • User experience: Kata will continue to work as it does today with no additional effort required on the behalf of users/admins.
  • Performance/metrics: The overall system performance using the default Kata configuration will not be negatively impacted by these changes.

I agree. I think we'd prefer to minimise the '#ifdef'-ing of code (apart from in the containerd section which we can't currently avoid). This would mean that we'd just have a single version of the agent which would pull into the extra crates required for image offload. I hope that the footprint of these won't be too large, but it's something we'll want to work with some performance specialists to check before we merged anything.

ifdef build solutions

If any sort of #ifdef plan is required, we're going to have to be extremely careful to avoid packaging mistakes and to ensure we can discriminate between the two versions. As a mimimum we will need both containerd-shim-kata-v2 --version and the announce function to log details of how that version of the runtime has been built.

Yep - if we can restrict the go build tag/#ifdef type behaviour to the containerd-shim-kata-v2 and separate all the CC specific code into separate file(s), then one of the things we'll have to put on the list is to have some type of cc discriminator in the version.

Containerd fork

I'm not an expert on all the details, so @wllenyj will be able to give more details, but here is my basic understanding:

It would be useful if we could summarise:

• Why a containerd fork is needed.

I think the simple answer here is that we need to modify the containerd code to enable us to pass the pull_image request through the shim and to the agent rather than being run on the host, in order to enable image offload.

• How the fork differs to upstream.

I think the approach that the Ali team have taken is to create a new image service and pull image command in the shim and implemented a CC CRI service that handles the routing of the commands. I'm sure this is probably not completely accurate. For reference the full changes can be seen at: confidential-containers/containerd@main...confidential-containers:CC-main

• How we plan to maintain that fork (wrt security fixes, etc).

My current thinking is that we keep the CC fork of containerd tightly aligned with the version that 'vanilla Kata' is using, so when Kata updates we will rebase the CC fork on that version. That minimises the code differences in Kata and tries to ensure we don't get left too far behind.

• How long we are comfortable maintaining such a fork (it's potentially a huge cost).
• What the stumbling blocks and/or timeline for getting the fork changes merged upstream.

That's the $64,000 question. We initially hoped that containerd would have accepted our changes, or at least ideas by now and avoid this problem, but from the discussions that Samuel et al. have had with Derek and co. it seems like they have a grander to re-architect containerd to provide the ability to offload most of the containerd services when it starts. This is way beyond the scope of what we need, but means that they are not going to merge our fork and their new architecture won't be ready in the short term (my impression is we are looking at closer to 12 months than 1, or 2, but I'm definitely not 'in the know').

That's how long we are might have to maintain the fork, which we are pretty much committed to for CC, as we don't have a solution without it, but it's up to the wider Kata community to decide if they are comfortable with this.

Avoid confusion

How is all this going to be documented?

It's seems odd to document CC functionality in Kata repos, but if that's where the code lives, we will have to atleast create a set of links back to the actual docs in an appropriate CC repo.

Unless handled carefully, it's going to be rather unclear to users and contributors why there are two separate GitHub organisations imho.

At the moment, some of the limited doc we have is in the kata CCv0 fork, but as we start to pull in more repos from the confidential-containers org it might makes sense for it to migrate to that org where Kata is part of the wider 'CC' solution and then just sign post to it from the Kata repositories

Merge periodicity

Is this a "one off" merge or is there a possibility of further CC code syncs into Kata?

This depends on the Kata Architecture committee and wider community, but the way I would like to see it is that we'd have a few smaller scoped PRs to get the existing CCv0 code into main and once that is done we'd delete and 'CC' branches and work directly against main from then on integrating any 'CC' code directly rather than going via a branch. I think the aim of confidential-containers is to pull in the upstream versions of all the components we used rather than having our own forks/branches. Obviously we've come unstuck with that with containerd at the moment, but it's the only viable way to proceed long term as far as I can see without ending up with a lot of duplicated effort.

As I said - this is my own thoughts and probably not representative of the CC community and people who are much smarter and more experienced in this than me. Maybe @sameo @fitzthum @magowan, or @ariel-adam can elaborate further and correct me?

[jodh-intel]

Thanks @stevenhorsman. A few more questions:

containerd fork

fwics the ideal would be for upstream containerd to allow us to configure it to:

• not pull the image.
• pass the details of the image through to the kata-agent to allow the guest env to handle that task.

Although these features wouldn't generally be useful to most, they don't look that contentious. Is the problem then that containerd don't want us to spend time on a containerd feature that would (have to) be dropped when their new architecture lands? Is there a middle ground I wonder where we could create a new containerd plugin for the features we'd like? It looks like containerd already supports these but maybe the scope of that interface might need to be expanded to support the features we need?

Short-term alternatives?

Random thought: containerd is but one container manager. If we are going to have to wait for the upstream grand redesign work to be finished, to avoid having to maintain a fork of containerd ourselves (and all the costs and complications that brings with it), could we use an alternative such as podman or even Docker (once we have a viable shimv2 "wrapper") as a stop-gap solution?

Or are there other specific containerd features we need for Kata+CC?

/cc @c3d, @dgibson.

[fidencio]

@stevenhorsman, I'd explicitly add the need of having a kata-deploy image being shipped for the CC specific effort.

[c3d]

@jodh-intel For them, there is also the problem of having to maintain a specific feature they can't really test, and that also conflicts with their changes in progress.

[jodh-intel]

+1 to @fidencio's idea of having separate kata-deploy images. For maximum safety, we'd add a few checks somewhere to assert (positively and negatively) what we expect (and don't expect) to find in the Kata and CC output of kata-deploy.

If we end up having to use the "ifdef CC" build approach, we'll also need a similar set of checks for the CI (ensure-kata.sh and ensure-cc.sh type tests).

Metrics

As discussed in the AC meeting earlier today, we can "suck it and see" by raising a PR and just reviewing what the metrics CI thinks of it. If we have a single image that includes the CC elements, we may need to adjust the thresholds but if we are talking about separate images, we might need a separate set of CC metrics thresholds.

/cc @GabyCT who might have thoughts on this.

[GabyCT]

@jodh-intel currently for CC there is not metrics CI, in case that you are testing with CC elements you will need a new baremetal to test and enable that CI as probably the thresholds will be different

[stevenhorsman]

Whilst the conversation is on-going I've started the process of breaking down this into separate issues for different areas that we might need to separate and merge. This a still WIP and likely to change before we can run with them:

• ci: Add confidential-containers awareness to the build/test jobs #4626
• ci: Add support for installing the confidential-containers fork of containerd for integration testing #4627
• agent: Merge confidential containers agent/guest changes into main branch kata-containers#3971
• runtime: Add support for the confidential-containers changes kata-containers#3996
• ci: Add integration test for the confidential-containers changes scenarios #4628
  And fidencio's suggestion:
• kata-deploy: Provide a kata-deploy image to be used with the Confidential Containers operator kata-containers#3992

[wainersm]

Hi @stevenhorsman,

Whilst the conversation is on-going I've started the process of breaking down this into separate issues for different areas that we might need to separate and merge. This a still WIP and likely to change before we can run with them:

* [ci: Add confidential-containers awareness to the build/test jobs #4626](https://github.com/kata-containers/tests/issues/4626)

* [ci: Add support for installing the confidential-containers fork of containerd for integration testing #4627](https://github.com/kata-containers/tests/issues/4627)

* [agent: Merge confidential containers agent/guest changes into main branch kata-containers#3971](https://github.com/kata-containers/kata-containers/issues/3971)

Am I wrong assuming that kata-containers/kata-containers#3971 can be the first task we should focus on?

IIUC merging the CC agent/guest into main doesn't break the non-CC use case of Kata, although CC specific features (e.g. pull image inside the guest) won't be working until the runtime and containerd are changed.

The current chain of dependencies seems to be:

confidential-containers/guest-components#5 -> (release image-rs x.y so we stop pulling from main?) -> kata-containers/kata-containers#3970 -> kata-containers/kata-containers#3971 -> #4626 -> #4627 -> #4628

* [runtime: Add support for the confidential-containers changes #4628](https://github.com/kata-containers/tests/issues/4628) 

[stevenhorsman]

@wainersm - Firstly, thanks for taking a look at the issues and giving feedback - it's really appreciated. I also have a confession that once I read your flow I realised that I should have split out merging the runtime and integration tests into separate issues for better clarity, so have both kata-containers/kata-containers#3996 and #4628 now. Sorry if that makes you comments make less sense

Am I wrong assuming that kata-containers/kata-containers#3971 can be the first task we should focus on?

IIUC merging the CC agent/guest into main doesn't break the non-CC use case of Kata, although CC specific features (e.g. pull image inside the guest) won't be working until the runtime and containerd are changed.

The current chain of dependencies seems to be:

confidential-containers/image-rs#5 -> (release image-rs x.y so we stop pulling from main?) -> kata-containers/kata-containers#3970 -> kata-containers/kata-containers#3971 -> #4626 -> #4627 -> #4628

Sort of. I agree on the dependencies: confidential-containers/image-rs#5 -> (release image-rs x.y so we stop pulling from main?) -> kata-containers/kata-containers#3970 -> kata-containers/kata-containers#3971 -> kata-containers/kata-containers#3996 -> #4628 (also dependant on #4627)

However my thinking (which might be flawed) is that we can make progress on #4626, then #4627 before the agent and runtime changes are merged. If we did this then they would be testing regression scenarios from 'vanilla' Kata with the forked containerd installed, rather than new 'confidential-containers' integration tests, but I think there is still value in this. Also have #4626 might be useful for us if we want to try out only adding the attesation-agent to the guest image if we are creating the 'CC' build (which might be required depending on how much it bloats the image.)

I hope that splitting the flow into two slightly parallel streams helps speed up our progress, but there might well be something I've failed to take into account?

[wainersm]

@stevenhorsman I think your comments on #4441 (comment) made the plan very clear to me. IMO that's detailed enough to allow us kick off some changes on main. Thus, I assigned to myself the issues #4626 , #4627 , and #4628; but if you (or someone of your team) had the intention to work on those issues then I am glad to re-assign them. :)

[wainersm]

Linking this issue with kata-deploy: Provide a kata-deploy image to be used with the Confidential Containers operator

[stevenhorsman]

@stevenhorsman I think your comments on #4441 (comment) made the plan very clear to me. IMO that's detailed enough to allow us kick off some changes on main. Thus, I assigned to myself the issues #4626 , #4627 , and #4628; but if you (or someone of your team) had the intention to work on those issues then I am glad to re-assign them. :)

@wainersm - I was thinking of taking #4627 as it's mostly just a port of the code I wrote in CCv0, so would simplify the DCO sign-off. I think #4628 is obviously a lot larger and will require us to sync up depending on when it is unblocked, but I'm (or someone on my team) happy to be involved in that and #4626 if there is anything we can help with!

[wainersm]

@stevenhorsman I think your comments on #4441 (comment) made the plan very clear to me. IMO that's detailed enough to allow us kick off some changes on main. Thus, I assigned to myself the issues #4626 , #4627 , and #4628; but if you (or someone of your team) had the intention to work on those issues then I am glad to re-assign them. :)

@wainersm - I was thinking of taking #4627 as it's mostly just a port of the code I wrote in CCv0, so would simplify the DCO sign-off. I think #4628 is obviously a lot larger and will require us to sync up depending on when it is unblocked, but I'm (or someone on my team) happy to be involved in that and #4626 if there is anything we can help with!

I just reassigned #4627 to you as it makes complete sense.
I will try to work on #4626 first. Let's see how it goes.

[stevenhorsman]

So just an update to the merge of CCv0 into main, based on a discussion we had at the 21st April CC meeting:
Given what we heard from Derek MacGowan about the 1.7 release dates we agreed that rather than push ahead with trying to get support for our confidential-containers fork of containerd merged into main immediately, we would wait until the 1.7 beta was available at the end of next month. At this point we will try switching to that and dropping our fork (this might take a while as I believe there are some complications with rust ttrpc supporting streaming?).
The timeframe for the 1.7/2.0 release in October-ish is a little longer than we'd like and we know at this point that we also want the Kata main branch to adopt the new version of containerd, but we felt like trying to get this dual vendor support merged would be very difficult with a new version on containerd that would supercede it coming up soon.
Obviously once we learn more information about the release dates and plans and the new code this might change, but the community felt that investing much time in the dual-vendoring approach wasn't the best move currently.
With respect to the guest/agent changes, everything we've heard makes us believe the new containerd code won't change anything beyond the shim, so we can proceed with trying to get agent changes merged, once we've removed skopeo and umoci.

[wainersm]

Thanks for the details update @stevenhorsman ! I wasn't able to join that meeting but I think the decision the community took was the best one.

[stevenhorsman]

So to rehydrate this discussion 7 months on with a few thing changing I think we have the following high-level items to resolve/split up and merge before we can completely be free of the CCv0 branch:

• Once skopeo and umoci are removed from the guest image, we can try and start the process of merging the agent changes: agent: Merge confidential containers agent/guest changes into main branch kata-containers#3971
• Migrate over to the upstream containerd, when a (beta?) release is available and update the kata-runtime code to be compatible with the new transfer service. Then get this version of containerd adopted in main before we merge the image service related code - Migrate over to the upstream containerd kata-containers#6067
• Get remote hypervisor support merged
• 'Backport' the rework to make the CI use the deploy mechanism for building the components and all the great component caching improvements
• Get the new TEE configs and deploy changes merged.
• Have our CC pipeline jobs and integration tests (non-TEE and TEE) stable, consistent and the pipeline ownership identified so they can been added to the daily baseline.
  • Once merged the operator e2e CI should be changed to use tests main branch
• Update the workflows that push cc-payload to run based on the main branch
• Align release process with the main/'vanilla' Kata Containers community - Currently we aim to have releases every 6 weeks and have our own freeze period before that and create our own tags for releases. Kata containers aims to release once a month. IMHO Once we merge into main we are a normal feature of Kata containers, so would line up with them and their processes and just depend on the release builds of kata rather than tagging our own. WRT freeze period, we can't stop development on the main branch, and I think that long term the freeze isn't viable, so we need to get the CI/CD in place that at freeze time we can just take the last passing build for various components and use that rather than a freeze period with manual testing.

I'm sure there are some other items, so keep me honest and I'll try and update this list. As we get more uniform agreement we can covert them into issues.

[wainersm]

Hi @stevenhorsman , so minor changes that came to my mind while in the meeting today:

• Operator e2e CI should be changed to use tests main branch -- one line change
• The workflows that push cc-payload should be changed too -- hopefully a couple of one line changes

[stevenhorsman]

Hi @stevenhorsman , so minor changes that came to my mind while in the meeting today:

• Operator e2e CI should be changed to use tests main branch -- one line change
• The workflows that push cc-payload should be changed too -- hopefully a couple of one line changes

Thanks, I'll update the list above to try and reflect this

[surajssd]

Migrate over to the upstream containerd, when a (beta?) release is available and update the kata-runtime code to be compatible with the containerd/containerd#7320. Then get this version of containerd adopted in main before we merge the image service related code.

@stevenhorsman is there an issue in the kata repo where the work to migrate from the forked containerd to upstream containerd is being tracked?

[stevenhorsman]

Migrate over to the upstream containerd, when a (beta?) release is available and update the kata-runtime code to be compatible with the containerd/containerd#7320. Then get this version of containerd adopted in main before we merge the image service related code.

@stevenhorsman is there an issue in the kata repo where the work to migrate from the forked containerd to upstream containerd is being tracked?

Hey Suraj, not that I'm aware of. I can start one tomorrow if that's helpful, or anyone else can for that matter and I'll link to it in the above list.

[surajssd]

@stevenhorsman kata-containers/kata-containers#6067 created an issue.

I don't have a lot of context on what needs to be done there other than just switching from the forked version to the latest beta. But I have created the tracking issue. Also that issue needs to have the label area/confidential-containers.