CI/CD plan for CCv1

[wainersm]

Context

Currently we got CI pipelines to test pull requests addressed to our CC branch of the Kata Containers project, however those are tests that ensure we don't introduce regressions to the "normal" Kata containers. As part of CCv1 proposal, we are aiming to delivery a "CI/CD (...) With and without a TEE, add confidential container integration tests" (slide 5) so that we begin catching CC specific regressions. This work will be particularly valuable for checking the rebases that Steve Horsman (and others) frequently carry out on our CC branch in the Kata Containers.

Deliverables

• Specific integration tests without TEE
• Specific integration tests with TEE
• CI pipelines

Scope

• The CI pipelines will run for pull requests (PR) opened to the kata-containers and tests. Other repositories like image-rs and attestation-agent are out of the scope.
• Test with containerd + CRI
• Test with Kubernetes + Containerd
• CRI-O is out of scope
• Use QEMU and Cloud Hypervisor as Kata hypervisor. Other hypervisors are out of scope for now

Assumptions

• The Kata CI resources (Jenkins and VMs on Azure) can be used.

Risks

• Currently CC is developed in a branch of Kata Containers and it is discussed its merge into main. While that effort is important and the way to go, it might cause changes on this CI plans.
• We aim to introduce the CC specific tests on Kata Container's tests repository and leverage the current Kata CI infrastructure and framework. This will turn CC tests/CI intimately coupled in Kata Containers, which goes against the aim of decouple Kata Containers and CC (being the former just a dependency of CC).

Environments

• Ubuntu 20.04 (x86_64) VM - non-TEE tests
• Fedora 35 (x86_64) VM - non-TEE tests
• TEE tests - to be defined

Major Milestones

1. February 2022: Initial CI - all CC PRs go through CC specific integration tests
2. April 2022: TEE (At least one) backed CI

Work items

February, 2022

• tests: PoC: running a specific Confidential Containers integration test without TEE kata-containers/tests#4509
• Identify and detail more non-TEE tests #3
• Identify hardware requirements for TEE tests #4

March, 2022

• Implement the remaining non-TEE tests
  • tests | CCv0: Non-TEE, CC pod creation with Kubernetes from unencrypted image kata-containers/tests#4557
  • tests | CCv0: Non-TEE, CC pod creation with Kubernetes from the encrypted image kata-containers/tests#4685
  • tests | CCv0: Non-TEE, CC pod creation with various combinations of (un)signed image vs (un)protected registry kata-containers/tests#4572
  • Add test cases for the kata agent API configuration kata-containers/tests#4676
• Create non-TEE CI pipelines
  • ci: add Jenkins job for CC_CRI_CONTAINERD type kata-containers/ci#445
  • ci: add Jenkins job for CC_SKOPEO_CRI_CONTAINERD type kata-containers/ci#451
  • CCv1: Also test CC efforts with Cloud Hypervisor kata-containers/ci#454
• Onboard hardware for TEE tests
  • Add AMD SEV machine in the Kata CI pool
• Identify and detail TEE tests #5 (mid-March)

April, 2022

• Implement TEE tests
• Create TEE CI pipelines
  • ci: create Intel TDX CI pipeline for CCv0+QEMU+TDVF #32
  • CCv0| Add CI job to test Confidential Containers in AMD SEV kata-containers/tests#4861

The team

The list of people who will be involved on this CI effort:

Contact	Role
Wainer Moschetta (@wainersm), Red Hat	Developer - QEMU tests and CI pipelines
Steve Horsman (@stevenhorsman), IBM	Developer - tests
Agam Dua, Apple	Developer - Cloud Hypervisor
Peter(@peterzcst )/Arron(@arronwy )	Hardware owner - Intel TDX TEE
Unmesh Deodhar, AMD	Hardware owner - AMD SEV / Tests SEV
Hendrik Brueckner (@hbrueckner)	Hardware owner - IBM s390x/Secure Execution

[peterzcst]

hi @wainersm , please my latest comments in #4. We are ready to integrate first TDX pipeline using CCV0+Qemu+TDVF in CCV0 CI. So please help add task of "Add Intel TDX machine in the CC CI pool" for April deliverable

[wainersm]

hi @wainersm , please my latest comments in #4. We are ready to integrate first TDX pipeline using CCV0+Qemu+TDVF in CCV0 CI. So please help add task of "Add Intel TDX machine in the CC CI pool" for April deliverable

Hi @peterzcst , just created the issue #32 to track that work.

[wainersm]

Delivered on 1st release.