azure-e2e-test: Add test for key release in kbs #1735
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kartikjoshi21
force-pushed
the
kartikjoshi21/test-key-release-kbs
branch
from
01acaa7 to
4807306
Compare
kartikjoshi21
force-pushed
the
kartikjoshi21/test-key-release-kbs
branch
15 times, most recently
from
687ae97 to
20289d3
Compare
kartikjoshi21
force-pushed
the
kartikjoshi21/test-key-release-kbs
branch
from
20289d3 to
ff9e1c9
Compare
kartikjoshi21
force-pushed
the
kartikjoshi21/test-key-release-kbs
branch
2 times, most recently
from
73fa759 to
e609d93
Compare
kartikjoshi21
force-pushed
the
kartikjoshi21/test-key-release-kbs
branch
from
e609d93 to
0a0296a
Compare
|
This is blocked on getting #1684 in? |
This is just test update so it can be merged. |
|
Steps to test this PR: Test Results: |
kartikjoshi21
force-pushed
the
kartikjoshi21/test-key-release-kbs
branch
from
0a0296a to
78f5e1c
Compare
mkulke
reviewed
Apr 16, 2024
| pod := NewBusyboxCurlPod(E2eNamespace) | ||
| testCommands := []TestCommand{ | ||
| { | ||
| Command: []string{"curl", "http://127.0.0.1:8006/cdh/resource/reponame/workload_key/key.bin"}, |
There was a problem hiding this comment.
busybox ships wget if we want to use busybox
There was a problem hiding this comment.
The comment is reasonable to me, we don't need introduce new function NewAlpineCurlPod and can reuse NewBusyboxPodWithName directly. And change the command to:
[]string{"wget", "http://127.0.0.1:8006/cdh/resource/reponame/workload_key/key.bin"},
There was a problem hiding this comment.
As I tried, the command should be:
wget -q -O- http://127.0.0.1:8006/cdh/resource/reponame/workload_key/key.bin, we need parameters -q -O- here.
There was a problem hiding this comment.
So, command would be:
[]string{"wget", "-q", "-O-", "http://127.0.0.1:8006/cdh/resource/reponame/workload_key/key.bin"}
|
@kartikjoshi21 @mkulke I'm a little confusion, seen from the code here https://github.com/confidential-containers/cloud-api-adaptor/blob/main/src/cloud-api-adaptor/test/provisioner/provision.go#L329, I think the KBS-IP is the overlay IP assigned by CNI, it should match the log: So, IP |
Good question, this is indeed unintuitive, I also stumbled over this. afaict that's a peculiarity of AKS's network configuration, see here:
So, the podvm's root network ns is still separated, but if the podvm runs in the same vnet, we're able to address a kbs pod with its pod ip. I think this won't work for k8s service ip and other cluster configurations, though. So, if this will only work for AKS in this particular configuration, we should probably think about an alternative approach for addressing KBS from the PodVM's AA + CDH. Maybe a NodePort service for the kbs deployment? |
Thanks @mkulke , which is reasonable to me now. Yeah, I tried the KBS NodePort IP in libvirt provider, it worked. I think we should make the change so that the e2e test cases suitable for all providers also. Its definitely a new issues :-) |
|
For me the tests fails because the trustee repo is not being mapped correctly. $ make test-e2e
go test -v -tags=azure -timeout 60m -count=1 ./test/e2e
time="2024-04-18T04:26:18Z" level=info msg="Do setup"
time="2024-04-18T04:26:18Z" level=info msg="Deploying kbs"
time="2024-04-18T04:26:18Z" level=info msg="creating key.bin"
time="2024-04-18T04:26:18Z" level=error msg="creating file: open trustee/kbs/config/kubernetes/overlays/key.bin: no such file or directory\n"
F0418 04:26:18.949698 37015 env.go:369] Setup failure: creating file: open trustee/kbs/config/kubernetes/overlays/key.bin: no such file or directory
FAIL github.com/confidential-containers/cloud-api-adaptor/src/cloud-api-adaptor/test/e2e 0.023s
FAIL
make: *** [Makefile:95: test-e2e] Error 1I followed the following instructions: # Test the PR 1735
export AZURE_RESOURCE_GROUP="suraj-test-kartik-pr1735-3"
export AZURE_REGION="eastus"
az group create --name "${AZURE_RESOURCE_GROUP}" \
--location "${AZURE_REGION}"
export AZURE_SUBSCRIPTION_ID=$(az account show --query id --output tsv)
export USER_ASSIGNED_IDENTITY_NAME="caa-${AZURE_RESOURCE_GROUP}"
az identity create \
--name "${USER_ASSIGNED_IDENTITY_NAME}" \
--resource-group "${AZURE_RESOURCE_GROUP}" \
--location "${AZURE_REGION}" \
--subscription "${AZURE_SUBSCRIPTION_ID}"
export PRINCIPAL_ID="$(az identity show \
--name "${USER_ASSIGNED_IDENTITY_NAME}" \
--resource-group "${AZURE_RESOURCE_GROUP}" \
--subscription "${AZURE_SUBSCRIPTION_ID}" --query principalId -otsv)"
sleep 30
az role assignment create \
--role Contributor \
--assignee-object-id "${PRINCIPAL_ID}" \
--scope "/subscriptions/${AZURE_SUBSCRIPTION_ID}"
export AZURE_CLIENT_ID="$(az identity show \
--resource-group "${AZURE_RESOURCE_GROUP}" \
--name "${USER_ASSIGNED_IDENTITY_NAME}" --query 'clientId' -otsv)"
export CLUSTER_NAME="e2e"
export AZURE_IMAGE_ID="/CommunityGalleries/cocopodvm-d0e4f35f-5530-4b9c-8596-112487cdea85/Images/podvm_image0/Versions/2024.04.17"
# Docker image for KBS
# https://github.com/confidential-containers/kbs/pkgs/container/staged-images%2Fkbs
cat <<EOF >/tmp/provision_azure.properties
AZURE_CLIENT_ID="${AZURE_CLIENT_ID}"
AZURE_SUBSCRIPTION_ID="${AZURE_SUBSCRIPTION_ID}"
RESOURCE_GROUP_NAME="${AZURE_RESOURCE_GROUP}"
CLUSTER_NAME="${CLUSTER_NAME}"
LOCATION="${AZURE_REGION}"
SSH_KEY_ID="id_rsa.pub"
AZURE_IMAGE_ID="${AZURE_IMAGE_ID}"
AZURE_CLI_AUTH="true"
MANAGED_IDENTITY_NAME="${USER_ASSIGNED_IDENTITY_NAME}"
# Deploy the same one that is merged on the CAA main
KBS_IMAGE="ghcr.io/confidential-containers/staged-images/kbs"
KBS_IMAGE_TAG="84432a2a97da306399db5bc863c9324dbd8b95ac"
# Get the tag from: https://quay.io/repository/confidential-containers/cloud-api-adaptor?tab=tags&tag=latest
CAA_IMAGE="quay.io/confidential-containers/cloud-api-adaptor:d32c2e6f92dd610a688c78d92c463f39781b8416"
EOF
pushd src/cloud-api-adaptor/
ssh-keygen -t rsa -b 4096 -f install/overlays/azure/id_rsa -N "" -C [email protected]
pushd test/tools
git clone [email protected]:confidential-containers/trustee.git
pushd trustee
git checkout 84432a2a97da306399db5bc863c9324dbd8b95ac
popd
popd
# Now open a new terminal
export TEST_PROVISION_FILE=/tmp/provision_azure.properties
export CLOUD_PROVIDER=azure
export BUILTIN_CLOUD_PROVIDERS=azure
export DEPLOY_KBS=true
export TEST_PROVISION=true
pushd test/tools
make caa-provisioner-cli
./caa-provisioner-cli -action=provision
popd
make test-e2eI cloned the trustee repo in the folder If I cloned it in the |
|
So essentially |
In CI we do |
kartikjoshi21
force-pushed
the
kartikjoshi21/test-key-release-kbs
branch
from
c1567de to
86d1ecb
Compare
|
Tested this change along with other two changes #1812 and #1684 . Things passes in CI now. This is run on my fork: https://github.com/kartikjoshi21/cloud-api-adaptor/actions/runs/8813015156/job/24190065704 |
When I use that I cannot get the CAA running, it fails with the following error: $ kl -n confidential-containers-system cloud-api-adaptor-daemonset-bjfp6
$AZURE_SUBNET_ID is NOT setI think we have My deployment instructions: # Test the PR 1735
export AZURE_RESOURCE_GROUP="suraj-test-kartik-pr1735-1"
export AZURE_REGION="eastus"
az group create --name "${AZURE_RESOURCE_GROUP}" \
--location "${AZURE_REGION}"
export AZURE_SUBSCRIPTION_ID=$(az account show --query id --output tsv)
export USER_ASSIGNED_IDENTITY_NAME="caa-${AZURE_RESOURCE_GROUP}"
az identity create \
--name "${USER_ASSIGNED_IDENTITY_NAME}" \
--resource-group "${AZURE_RESOURCE_GROUP}" \
--location "${AZURE_REGION}" \
--subscription "${AZURE_SUBSCRIPTION_ID}"
export PRINCIPAL_ID="$(az identity show \
--name "${USER_ASSIGNED_IDENTITY_NAME}" \
--resource-group "${AZURE_RESOURCE_GROUP}" \
--subscription "${AZURE_SUBSCRIPTION_ID}" --query principalId -otsv)"
sleep 30
az role assignment create \
--role Contributor \
--assignee-object-id "${PRINCIPAL_ID}" \
--scope "/subscriptions/${AZURE_SUBSCRIPTION_ID}"
export AZURE_CLIENT_ID="$(az identity show \
--resource-group "${AZURE_RESOURCE_GROUP}" \
--name "${USER_ASSIGNED_IDENTITY_NAME}" --query 'clientId' -otsv)"
export CLUSTER_NAME="e2e"
export AZURE_IMAGE_ID="/CommunityGalleries/cocopodvm-d0e4f35f-5530-4b9c-8596-112487cdea85/Images/podvm_image0/Versions/2024.04.25"
# Docker image for KBS
# https://github.com/confidential-containers/kbs/pkgs/container/staged-images%2Fkbs
cat <<EOF >/tmp/provision_azure.properties
AZURE_CLIENT_ID="${AZURE_CLIENT_ID}"
AZURE_SUBSCRIPTION_ID="${AZURE_SUBSCRIPTION_ID}"
RESOURCE_GROUP_NAME="${AZURE_RESOURCE_GROUP}"
CLUSTER_NAME="${CLUSTER_NAME}"
LOCATION="${AZURE_REGION}"
SSH_KEY_ID="id_rsa.pub"
AZURE_IMAGE_ID="${AZURE_IMAGE_ID}"
AZURE_CLI_AUTH="true"
MANAGED_IDENTITY_NAME="${USER_ASSIGNED_IDENTITY_NAME}"
# Deploy the same one that is merged on the CAA main
KBS_IMAGE="ghcr.io/confidential-containers/staged-images/kbs"
KBS_IMAGE_TAG="dc01f454264fb4350e5f69eba05683a9a1882c41"
# Get the tag from: https://quay.io/repository/confidential-containers/cloud-api-adaptor?tab=tags&tag=latest
CAA_IMAGE="quay.io/confidential-containers/cloud-api-adaptor:d32c2e6f92dd610a688c78d92c463f39781b8416"
EOF
# Change into the CAA repo
gh pr checkout 1735
pushd src/cloud-api-adaptor/
ssh-keygen -t rsa -b 4096 -f install/overlays/azure/id_rsa -N "" -C [email protected]
git clone https://github.com/confidential-containers/trustee test/trustee
pushd test/trustee
git checkout dc01f454264fb4350e5f69eba05683a9a1882c41
popd
popd
# Now open a new terminal
export TEST_PROVISION_FILE=/tmp/provision_azure.properties
export CLOUD_PROVIDER=azure
export BUILTIN_CLOUD_PROVIDERS=azure
export DEPLOY_KBS=true
export TEST_E2E_CREATE_RG=no
export TEST_PROVISION=no
pushd src/cloud-api-adaptor
pushd test/tools
# Get changes from #1684
curl -L -o /tmp/diff https://github.com/confidential-containers/cloud-api-adaptor/commit/b672ac0c2fafff0a85d11b71ee99cedd9b91e8b7.diff
git apply /tmp/diff
make caa-provisioner-cli
./caa-provisioner-cli -action=createcluster
popd
make test-e2e
popdThe deployment never passes because CAA is in Crashloop $ make test-e2e
go test -v -tags=azure -timeout 60m -count=1 ./test/e2e
time="2024-04-26T00:24:29Z" level=info msg="Do setup"
time="2024-04-26T00:24:29Z" level=info msg="Deploying kbs"
time="2024-04-26T00:24:29Z" level=info msg="creating key.bin"
/home/surajaz/code/work/cloud-api-adaptor/src/cloud-api-adaptor/test/e2e
time="2024-04-26T00:24:29Z" level=info msg="Creating kbs install overlay"
time="2024-04-26T00:24:29Z" level=info msg="Customize the overlay yaml file"
time="2024-04-26T00:24:29Z" level=info msg="Updating kbs image with \"ghcr.io/confidential-containers/staged-images/kbs\""
time="2024-04-26T00:24:29Z" level=info msg="Updating kbs image tag with \"dc01f454264fb4350e5f69eba05683a9a1882c41\""
time="2024-04-26T00:24:29Z" level=info msg="Creating kbs install overlay"
time="2024-04-26T00:24:29Z" level=info msg="Install Kbs"
Wait pod 'kbs-67fffcc988-mwkhp' status for Ready
pod 'kbs-67fffcc988-mwkhp' is Ready
Pod IP: 10.244.0.15
time="2024-04-26T00:24:50Z" level=info msg="KBS PARAMScc_kbc::http://10.244.0.15:8080:"
time="2024-04-26T00:24:50Z" level=info msg="Install Cloud API Adaptor"
time="2024-04-26T00:24:50Z" level=info msg="Deploy the Cloud API Adaptor"
time="2024-04-26T00:24:50Z" level=info msg="Install the controller manager"
Wait for the cc-operator-controller-manager deployment be available
time="2024-04-26T00:25:10Z" level=info msg="Customize the overlay yaml file"
time="2024-04-26T00:25:10Z" level=info msg="Updating CAA image with \"quay.io/confidential-containers/cloud-api-adaptor\""
time="2024-04-26T00:25:10Z" level=info msg="Updating CAA image tag with \"d32c2e6f92dd610a688c78d92c463f39781b8416\""
time="2024-04-26T00:25:10Z" level=info msg="Updating CAA image digest with \"\""
time="2024-04-26T00:25:12Z" level=info msg="Install the cloud-api-adaptor"
Wait for the cc-operator-daemon-install DaemonSet be available
Wait for the pod cc-operator-daemon-install-hgl82 be ready
Wait for the cloud-api-adaptor-daemonset DaemonSet be available
Wait for the pod cloud-api-adaptor-daemonset-bjfp6 be ready |
huoqifeng
reviewed
Apr 26, 2024
| pod := NewBusyboxCurlPod(E2eNamespace) | ||
| testCommands := []TestCommand{ | ||
| { | ||
| Command: []string{"curl", "http://127.0.0.1:8006/cdh/resource/reponame/workload_key/key.bin"}, |
There was a problem hiding this comment.
The comment is reasonable to me, we don't need introduce new function NewAlpineCurlPod and can reuse NewBusyboxPodWithName directly. And change the command to:
[]string{"wget", "http://127.0.0.1:8006/cdh/resource/reponame/workload_key/key.bin"},
|
@kartikjoshi21 @mkulke does the kbs image support sample TEE also? I met problem when try this against libvirt provider and sample TEE. |
|
@surajssd I have used steps we use in CI locally and iam able to test this change |
I think this image have is
Can you share what problems you faced ? Also this image wont have latest nodeport change confidential-containers/trustee#371 so i think we should use |
Thanks @kartikjoshi21 I'm tracking the logs in #1824 (comment) |
Fixes: confidential-containers#1785 Signed-off-by: Kartik Joshi <[email protected]>
kartikjoshi21
force-pushed
the
kartikjoshi21/test-key-release-kbs
branch
from
86d1ecb to
cd5b4c0
Compare
|
I tested this PR with the following steps and it works with local testing: # Test the PR 1735
export AZURE_RESOURCE_GROUP="suraj-test-kartik-pr1735-1"
export AZURE_REGION="eastus"
az group create --name "${AZURE_RESOURCE_GROUP}" \
--location "${AZURE_REGION}"
export AZURE_SUBSCRIPTION_ID=$(az account show --query id --output tsv)
export USER_ASSIGNED_IDENTITY_NAME="caa-${AZURE_RESOURCE_GROUP}"
az identity create \
--name "${USER_ASSIGNED_IDENTITY_NAME}" \
--resource-group "${AZURE_RESOURCE_GROUP}" \
--location "${AZURE_REGION}" \
--subscription "${AZURE_SUBSCRIPTION_ID}"
export PRINCIPAL_ID="$(az identity show \
--name "${USER_ASSIGNED_IDENTITY_NAME}" \
--resource-group "${AZURE_RESOURCE_GROUP}" \
--subscription "${AZURE_SUBSCRIPTION_ID}" --query principalId -otsv)"
sleep 30
az role assignment create \
--role Contributor \
--assignee-object-id "${PRINCIPAL_ID}" \
--scope "/subscriptions/${AZURE_SUBSCRIPTION_ID}"
export AZURE_CLIENT_ID="$(az identity show \
--resource-group "${AZURE_RESOURCE_GROUP}" \
--name "${USER_ASSIGNED_IDENTITY_NAME}" --query 'clientId' -otsv)"
export CLUSTER_NAME="e2e"
export AZURE_IMAGE_ID="/CommunityGalleries/cocopodvm-d0e4f35f-5530-4b9c-8596-112487cdea85/Images/podvm_image0/Versions/2024.04.25"
# Docker image for KBS
# https://github.com/confidential-containers/kbs/pkgs/container/staged-images%2Fkbs
cat <<EOF >/tmp/provision_azure.properties
AZURE_CLIENT_ID="${AZURE_CLIENT_ID}"
AZURE_SUBSCRIPTION_ID="${AZURE_SUBSCRIPTION_ID}"
RESOURCE_GROUP_NAME="${AZURE_RESOURCE_GROUP}"
CLUSTER_NAME="${CLUSTER_NAME}"
LOCATION="${AZURE_REGION}"
SSH_KEY_ID="id_rsa.pub"
AZURE_IMAGE_ID="${AZURE_IMAGE_ID}"
AZURE_CLI_AUTH="true"
MANAGED_IDENTITY_NAME="${USER_ASSIGNED_IDENTITY_NAME}"
# Deploy the same one that is merged on the CAA main
KBS_IMAGE="ghcr.io/confidential-containers/staged-images/kbs"
KBS_IMAGE_TAG="dc01f454264fb4350e5f69eba05683a9a1882c41"
# Get the tag from: https://quay.io/repository/confidential-containers/cloud-api-adaptor?tab=tags&tag=latest
CAA_IMAGE="quay.io/confidential-containers/cloud-api-adaptor:d32c2e6f92dd610a688c78d92c463f39781b8416"
EOF
# Change into the CAA repo
gh pr checkout 1735
pushd src/cloud-api-adaptor/
ssh-keygen -t rsa -b 4096 -f install/overlays/azure/id_rsa -N "" -C [email protected]
git clone https://github.com/confidential-containers/trustee test/trustee
pushd test/trustee
git checkout dc01f454264fb4350e5f69eba05683a9a1882c41
popd
popd
# Now open a new terminal
export TEST_PROVISION_FILE=/tmp/provision_azure.properties
export CLOUD_PROVIDER=azure
export BUILTIN_CLOUD_PROVIDERS=azure
export DEPLOY_KBS=true
export TEST_E2E_CREATE_RG=no
export TEST_PROVISION=no
pushd src/cloud-api-adaptor
pushd test/tools
make caa-provisioner-cli
./caa-provisioner-cli -action=provision
popd
make test-e2e
popd |
surajssd
approved these changes
Apr 29, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.