Add CI script to deploy published releases #1837

tnozicka · 2024-03-15T15:00:51Z

Description of your changes:
This script will be used in periodics to test our published images. It essentially codifies what needs to be done to deploy a particular scylla-operator version. Some action could be simplified by restructuring manifest but it's out of the scope of this PR. Eventually, we should fix this and possibly have some smarter templating / publish manifests into releases with the replaced image. So take this PR as a start with what we have now, not where it should end.

Which issue is resolved by this Pull Request:
Resolves #1936

Requires

tnozicka · 2024-03-18T15:45:31Z

/retest

tnozicka · 2024-03-18T16:24:57Z

/test images

tnozicka · 2024-03-18T16:30:14Z

/retest

rzetelskik · 2024-03-19T12:52:06Z

How is this going to be set up with CI? https://github.com/scylladb/scylla-operator/blob/master/hack/.ci/run-e2e-gke.sh uses ./hack/ci-deploy.sh. Are you planning to send a separate PR configuring it? Or is this going to be used by an entirely separate script?

tnozicka · 2024-03-19T14:44:34Z

this can't be used by presubmits by definition as their merged git shas are not published - this is only for published releases

rzetelskik · 2024-03-20T12:29:58Z

this can't be used by presubmits by definition as their merged git shas are not published - this is only for published releases

I know, but the question was about how is this going to be integrated with CI, pointing at whether this PR is potentially missing any adjustments.

tnozicka · 2024-03-21T08:20:13Z

I know, but the question was about how is this going to be integrated with CI, pointing at whether this PR is potentially missing any adjustments.

This will be used only by new periodics based on already published images (we don't have these periodics yet).
(Eventually, we can add an on demand presubmit using this script against a latest release to smoke test it.)

tnozicka · 2024-03-25T10:13:22Z

added the extra presubmit in https://github.com/scylladb/scylla-operator-release/pull/157

zimnx · 2024-03-27T15:35:45Z

/retest

zimnx · 2024-03-27T16:40:02Z

Looks like there's a missing bracket in CI test script.

zimnx

/lgtm

please fix CI script
/hold

rzetelskik · 2024-03-28T13:12:52Z

https://github.com/scylladb/scylla-operator-release/pull/163

tnozicka · 2024-03-28T13:25:07Z

/retest

rzetelskik · 2024-03-28T14:21:08Z

time="2024-03-28T13:38:18Z" level=fatal msg="Error parsing image name \"docker://docker.io/scylladb/scylla-operator:latest@sha256:0c8896a666fdc0d273b92dbd1bd936140d1da34e6c25b9f7ac1260a7acc18f16\": Docker references with both a tag and digest are currently not supported"

needs a fix

tnozicka · 2024-04-02T10:42:25Z

fixed in https://github.com/scylladb/scylla-operator-release/pull/168

tnozicka · 2024-04-18T08:15:39Z

https://github.com/scylladb/scylla-operator-release/pull/168 has landed
/retest

tnozicka · 2024-04-18T14:31:12Z

cluster didn't spin up in time
/retest

tnozicka · 2024-04-19T06:56:47Z

oh, I've tested this but seems like it depends on extglob which I had enabled and the image doesn't. need to fix that in the CI config

tnozicka · 2024-04-19T08:28:58Z

https://github.com/scylladb/scylla-operator-release/pull/180 landed
/retest

tnozicka · 2024-04-19T12:12:57Z

(this need more work on artifact collection and specifying nodeconfig on CI side)

tnozicka · 2024-05-16T12:57:13Z

(this need more work on artifact collection and specifying nodeconfig on CI side)

done and fixed the script + CI setup - ready for a second round of review, thx

hack/ci-deploy.sh

hack/ci-deploy-release.sh

rzetelskik · 2024-05-20T11:19:56Z

hack/ci-deploy-release.sh

+# Wait for cert-manager crd and webhooks
+kubectl wait --for condition=established --timeout=60s crd/certificates.cert-manager.io crd/issuers.cert-manager.io
+for d in cert-manager{,-cainjector,-webhook}; do
+    kubectl -n=cert-manager rollout status --timeout=5m "deployment.apps/${d}"


missing wait-for-object-creation from ci-deploy:

scylla-operator/hack/ci-deploy.sh

Line 48 in b7ca9a4

wait-for-object-creation cert-manager deployment.apps/"${d}"

. Is it superfluous there or accidentally missing here?

Shouldn't be needed, I don't think kubectl hits the watch cache.

should it be removed from the other script then? (not in this pr)

should we have a tracker for this? Even if it's in the backlog

hack/ci-deploy-release.sh

tnozicka · 2024-05-27T12:06:37Z

@rzetelskik please have a look at the comment replies, I'd like to land this and unblock your arm64 PR

rzetelskik · 2024-05-27T12:26:32Z

@rzetelskik please have a look at the comment replies, I'd like to land this and unblock your arm64 PR

done

rzetelskik

lgtm
/assign zimnx
in case you want to have a second look at it, it changed since your review. Otherwise feel free to remove the hold.

zimnx

One nit. Feel free to cancel hold even without applying, i'm fine with having both options.

/lgtm
/hold

zimnx · 2024-05-27T16:19:56Z

hack/ci-deploy-release.sh

+wait-for-object-creation scylla-manager statefulset.apps/scylla-manager-cluster-manager-dc-manager-rack
+kubectl -n=scylla-manager rollout status --timeout=5m statefulset.apps/scylla-manager-cluster-manager-dc-manager-rack


nit: these could be replaced with waiting for our three aggregated conditions on ScyllaCluster without hardcoding names of Operator managed resources.

It's a good point but it's in ci-deploy.sh as well, I'll send a followup.
/hold cancel

scylla-operator-bot · 2024-05-27T16:20:57Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rzetelskik, tnozicka, zimnx

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [tnozicka,zimnx]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

tnozicka added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. kind/machinery Categorizes issue or PR as related to Makefile, scripts or similar changes. labels Mar 15, 2024

scylla-operator-bot bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Mar 15, 2024

scylla-operator-bot bot requested review from rzetelskik and zimnx March 15, 2024 15:01

scylla-operator-bot bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 15, 2024

zimnx approved these changes Mar 27, 2024

View reviewed changes

scylla-operator-bot bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 27, 2024

scylla-operator-bot bot assigned zimnx Mar 27, 2024

scylla-operator-bot bot added the lgtm Indicates that a PR is ready to be merged. label Mar 27, 2024

tnozicka changed the title ~~Add CI script to deploy published releases~~ [WIP] Add CI script to deploy published releases Apr 19, 2024

tnozicka force-pushed the deploy-release branch from 27b187a to 70c19c3 Compare May 16, 2024 08:48

tnozicka changed the title ~~[WIP] Add CI script to deploy published releases~~ Add CI script to deploy published releases May 16, 2024

scylla-operator-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 16, 2024

tnozicka requested a review from zimnx May 16, 2024 12:53

tnozicka mentioned this pull request May 17, 2024

Support multi-datacenter clusters in test runner and script #1881

Merged

rzetelskik requested changes May 20, 2024

View reviewed changes

zimnx mentioned this pull request May 20, 2024

Bump local volume provisioner to 3.2.0 #1927

Merged

tnozicka force-pushed the deploy-release branch from 70c19c3 to 75163e6 Compare May 22, 2024 05:44

tnozicka changed the title ~~Add CI script to deploy published releases~~ [WIP] Add CI script to deploy published releases May 22, 2024

scylla-operator-bot bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 22, 2024

tnozicka force-pushed the deploy-release branch from 75163e6 to feeb101 Compare May 22, 2024 06:05

tnozicka changed the title ~~[WIP] Add CI script to deploy published releases~~ Add CI script to deploy published releases May 22, 2024

scylla-operator-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 22, 2024

Add ci script to deploy published releases

3d3b36e

tnozicka force-pushed the deploy-release branch from feeb101 to 3d3b36e Compare May 27, 2024 13:13

tnozicka requested a review from rzetelskik May 27, 2024 13:23

rzetelskik approved these changes May 27, 2024

View reviewed changes

tnozicka mentioned this pull request May 27, 2024

Remove redundant check in ci deploy script #1945

Merged

zimnx approved these changes May 27, 2024

View reviewed changes

scylla-operator-bot bot added the lgtm Indicates that a PR is ready to be merged. label May 27, 2024

scylla-operator-bot bot closed this in #1945 May 28, 2024

tnozicka reopened this May 28, 2024

scylla-operator-bot bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 29, 2024

scylla-operator-bot bot merged commit 6c4fa9f into scylladb:master May 29, 2024
13 checks passed

tnozicka deleted the deploy-release branch May 29, 2024 07:47

tnozicka mentioned this pull request May 29, 2024

Use conditions to wait for deploying manager DB #1947

Merged

		wait-for-object-creation scylla-manager statefulset.apps/scylla-manager-cluster-manager-dc-manager-rack
		kubectl -n=scylla-manager rollout status --timeout=5m statefulset.apps/scylla-manager-cluster-manager-dc-manager-rack

Add CI script to deploy published releases #1837

Add CI script to deploy published releases #1837

Conversation

tnozicka commented Mar 15, 2024 • edited Loading

Requires

tnozicka commented Mar 18, 2024

tnozicka commented Mar 18, 2024

tnozicka commented Mar 18, 2024

rzetelskik commented Mar 19, 2024 • edited Loading

tnozicka commented Mar 19, 2024

rzetelskik commented Mar 20, 2024

tnozicka commented Mar 21, 2024

tnozicka commented Mar 25, 2024

zimnx commented Mar 27, 2024

zimnx commented Mar 27, 2024

zimnx left a comment

Choose a reason for hiding this comment

rzetelskik commented Mar 28, 2024

tnozicka commented Mar 28, 2024

rzetelskik commented Mar 28, 2024

tnozicka commented Apr 2, 2024

tnozicka commented Apr 18, 2024

tnozicka commented Apr 18, 2024

tnozicka commented Apr 19, 2024

tnozicka commented Apr 19, 2024

tnozicka commented Apr 19, 2024

tnozicka commented May 16, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tnozicka commented May 27, 2024

rzetelskik commented May 27, 2024

rzetelskik left a comment • edited Loading

Choose a reason for hiding this comment

zimnx left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

scylla-operator-bot bot commented May 27, 2024

tnozicka commented Mar 15, 2024 •

edited

Loading

rzetelskik commented Mar 19, 2024 •

edited

Loading

rzetelskik left a comment •

edited

Loading