control-service: change CI/CD to release a new version on each PR mer…

…ge (#357) Up until now, Control Service releases had to be done manually by making changes to several files and then making a new PR and merge it. With this commit, a new version of the Control Service is released automatically every time a new image of the Control Service is built, which happens every time a PR with changes to the code base of the Control Service is merged. The MAJOR and MINOR components of the version are taken from the version.txt file. The PATCH component is automatically generated. Updated the CONTRIBUTING.md file to reflect the new release process. Cleaned up the CHANGELOG.md file. Testing done: Changed the steps of interest to execute on PR rather than on merge and verified that they run correctly, i.e. verified that the version of the used control service image is correct, that the control service version itself is properly generated and that the heartbeat is released with the appropriate version. Signed-off-by: Tsvetomir Palashki <[email protected]>
vmware · Oct 11, 2021 · 3178e35 · 3178e35
1 parent 35c8ebb
commit 3178e35
Show file tree

Hide file tree

Showing 5 changed files with 31 additions and 72 deletions.
diff --git a/projects/control-service/CHANGELOG.md b/projects/control-service/CHANGELOG.md
@@ -1,7 +1,7 @@
 Changelog
 =========
 
-1.2.17 - Next
+MAJOR.MINOR - dd.MM.yyyy
 ----
 * **New feature**
 
@@ -11,39 +11,35 @@ Changelog
 
 * **Breaking Changes**
 
-1.2.16 - 05.10.2021
+
+1.3 - 08.10.2021
 ----
-* **New feature**
+* **Improvement**
+  * Switch to a new versioning model where the patch version is automatically generated.
+  * Control Service is now automatically released on every PR merge.
 
+
+1.2.16 - 05.10.2021
+----
 * **Improvement**
   * Additional deployment labels are now set using valid YAML instead of a multiline string.
-* **Bug Fixes**
 
-* **Breaking Changes**
 
 1.2.14 - 04.10.2021
 ----
-* **New feature**
-
 * **Improvement**
   * Custom labels can now be supplied to the Control Service deployment during Helm chart installation.
 
 * **Bug Fixes**
   * Fix an error which happens occasionally when listing data jobs or job deployment statuses
 
-* **Breaking Changes**
 
 1.2.13 - 30.09.2021
 ----
-* **New feature**
-
 * **Improvement**
   * Implement a new OAuth2 claim/role-based authorization model, which can work instead or in complement
     of the existing webhook-based authorization.
 
-* **Bug Fixes**
-
-* **Breaking Changes**
 
 1.2.12 - 29.09.2021
 ----
@@ -53,13 +49,14 @@ Changelog
 * **Improvement**
   * The Data Job Execution statuses synchronization takes into account SUBMITTED status.
 
+
 1.2.11 - 27.09.2021
 ----
-
 * **Bug Fixes**
   * Fixed helm chart to give correct permission to read logs
     This would fix Logging API to work and should fix the deployment notification on user error which was not sending notifications
 
+
 1.2.10 - 23.09.2021
 ----
 * **New feature**
@@ -69,8 +66,6 @@ Changelog
 
 1.2.9 - 21.09.2021
 ----
-* **New feature**
-
 * **Improvement**
   * GraphQL endpoint now does not have limit for fetching data jobs, which was previously 100 jobs per page.
 
@@ -95,73 +90,48 @@ Changelog
 
 1.2.6 - 31.08.2021
 ----
-
 * **New feature**
   * Adding support for forwarding logs of the Control Service Server (only) to syslog server <br>
     Users can now specify environment variable LOGGING_SYS_LOG_HOST and LOGGING_SYS_LOG_PORT to enable forwarding them to syslog.
 
 
 1.2.5 - 30.08.2021
 ----
-
 * **New feature**
   * implement DELETE Job execution REST API<br>
     Users can now cancel started data job execution in self-service manner using REST API
 
-
 * **Improvement**
   * Switch Job Builder image to use kaniko<br>
     This will enable Control Servie to run in more secure kubernetes (pod security - no privilege run, seccomp/apparmor profiles enabled)
     It will enable to use local docker registry (without ssl) for easier deployment for prototype purposes
 
-* **Bug Fixes**
-
-* **Breaking Changes**
-
 
 1.2.4 - 27.08.2021
 ---
-* **New feature**
-
-* **Improvement**
-
 * **Bug Fixes**
   * Non-accurate control-service image tag fix
   * Data Job deployment fix
     Fixed data jobs deployment failures that emit 'ERROR: restart transaction' due to DeploymentMonitor.recordDeploymentStatus.
 
-* **Breaking Changes**
-
 
 1.2.2 - 19.08.2021
 ----
-
-* **New feature**
-
 * **Improvement**
   * Expose data job base image as configuration
     Data job base image used to run deployed data job can be configured using helm now.
 
 * **Bug Fixes**
   * Generating javadoc fix
 
-* **Breaking Changes**
-
 
 1.2.1 - 18.08.2021
 ----
-
-* **New feature**
-
 * **Improvement**
   * Make the image registry in the helm chart configurable.
     Configure registry part of docker images used for control-service API, job builder image and VDK distribution image.
     This make it easier to use docker cache proxies.
 
-* **Bug Fixes**
-
-* **Breaking Changes**
-
 
 1.2.0 - 17.08.2021
 ----
@@ -192,9 +162,3 @@ Initial release of Control Service in the public VDK Helm repository.
   - GET Sources API implementation
   - Configurable data job template can be provided via Helm
   - Data Job Execution API that allows clients to execute their jobs remotely.
-
-* **Improvement**
-
-* **Bug Fixes**
-
-* **Breaking Changes**
diff --git a/projects/control-service/CONTRIBUTING.md b/projects/control-service/CONTRIBUTING.md
@@ -190,12 +190,10 @@ helm uninstall taurus-data-pipelines
 
 #### Control Service and Helm Chart
 
-In order to make new release of Versatile Data Kit Control Service:
-* Update version in [helm_charts/pipelines-control-service/version.txt](projects/helm_charts/pipelines-control-service/version.txt) - follows https://semver.org
-* Make sure image.tag (in [values.yaml](projects/helm_charts/pipelines-control-service/values.yaml)) is updated accordingly
-  * look at the latest successful CICD Pipeline on [main](https://gitlab.com/vmware-analytics/versatile-data-kit/-/pipelines?page=1&scope=all&ref=main) - get tag from logs of [stage pre_release](cicd/.gitlab-ci.yml) and ci job control_service_deploy_testing_data_pipelines. This job is a part of the pre_release pipeline. The new tag string should contain the short git commit hash of the last commit that changed the control-service.
-* Check if [CHANGELOG.md](CHANGELOG.md) needs to be updated.
-* Post review and merge to main. The release commit should not have other changes except those above.
-* CICD automatically triggers new release on each update of the version.txt file - so only monitor CICD pipeline.
+New releases of the Versatile Data Kit Control Service are done automatically on each PR merge. As part of the PR, make sure to do the following:
+* If necessary, update the version in [helm_charts/pipelines-control-service/version.txt](projects/helm_charts/pipelines-control-service/version.txt) - follows https://semver.org, but only the MAJOR and MINOR components are specified. The PATCH component is generated automatically during release.
+* Check if [CHANGELOG.md](CHANGELOG.md) needs to be updated. New entries should include <MAJOR>.<MINOR> - <dd>.<MM>.<yyyy> as a heading.
+* Post review and merge to main.
+* The CI/CD automatically triggers the new release so only monitor the CI/CD pipeline.
     * https://gitlab.com/vmware-analytics/versatile-data-kit/-/pipelines
     * Once released it will be uploaded in Helm repo: https://gitlab.com/api/v4/projects/28814611/packages/helm/stable
diff --git a/projects/control-service/cicd/.gitlab-ci.yml b/projects/control-service/cicd/.gitlab-ci.yml
@@ -63,9 +63,6 @@ control_service_build_image:
       - external_pull_requests
       - main
     changes: *control_service_change_locations
-  except:
-    changes:
-      - projects/control-service/projects/helm_charts/pipelines-control-service/version.txt
 
 control_service_integration_test:
   extends: .control_service_base_build
@@ -226,17 +223,20 @@ control_service_release:
   stage: release
   script:
     - apk --no-cache add bash openssl curl git
+    - export IMAGE_TAG=$(git rev-parse --short HEAD)
     - export DESIRED_VERSION=v3.6.3 # helm version 3.6.3
+    - export CHART_NAME=pipelines-control-service
+    - export CHART_VERSION="$(cat projects/control-service/projects/helm_charts/$CHART_NAME/version.txt | grep -o '^[0-9]\+\.[0-9]\+').$CI_PIPELINE_ID"
     - curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash
-    - helm plugin install https://github.com/chartmuseum/helm-push
+    - helm plugin install https://github.com/chartmuseum/helm-push --version 0.10.0
+    - helm plugin install https://github.com/thynquest/helm-pack --version 0.2.2
     - cd projects/control-service/projects/helm_charts
     - bash -ex ../../cicd/release-pipelines-service.sh
   retry: !reference [.control_service_retry, retry_options]
   only:
     refs:
       - main
-    changes:
-      - projects/control-service/projects/helm_charts/pipelines-control-service/version.txt
+    changes: *control_service_change_locations
 
 control_service_vdk_heartbeat_release:
   image: docker:19.03.8
@@ -249,7 +249,7 @@ control_service_vdk_heartbeat_release:
   stage: release
   script:
     - export CHART_NAME=pipelines-control-service
-    - export CHART_VERSION=$(cat projects/control-service/projects/helm_charts/$CHART_NAME/version.txt)
+    - export CHART_VERSION=$(cat projects/control-service/projects/helm_charts/$CHART_NAME/version.txt | grep -o '^[0-9]\+\.[0-9]\+').$CI_PIPELINE_ID
     - export VDK_HEARTBEAT_TEST_NAME=$CHART_NAME-integration-tests:$CHART_VERSION
     - export VDK_HEARTBEAT_IMAGE_NAME="$VDK_DOCKER_REGISTRY_URL/$VDK_HEARTBEAT_TEST_NAME"
     - docker login $VDK_DOCKER_REGISTRY_URL --username $VDK_DOCKER_REGISTRY_USERNAME --password $VDK_DOCKER_REGISTRY_PASSWORD
@@ -260,5 +260,4 @@ control_service_vdk_heartbeat_release:
   only:
     refs:
       - main
-    changes:
-      - projects/control-service/projects/helm_charts/pipelines-control-service/version.txt
+    changes: *control_service_change_locations
diff --git a/projects/control-service/cicd/release-pipelines-service.sh b/projects/control-service/cicd/release-pipelines-service.sh
@@ -3,17 +3,15 @@
 # Copyright 2021 VMware, Inc.
 # SPDX-License-Identifier: Apache-2.0
 
-export CHART_NAME=pipelines-control-service
-export CHART_VERSION=$(cat $CHART_NAME/version.txt)
 export HELM_REPO=versatile-data-kit-helm-registry
 # TODO: sign chart
-helm package --version $CHART_VERSION --app-version $CHART_VERSION --dependency-update $CHART_NAME
+helm pack --version "$CHART_VERSION" --app-version "$CHART_VERSION" --dependency-update "$CHART_NAME" --set image.tag="$IMAGE_TAG"
 echo "Upload $CHART_NAME version $CHART_VERSION (if version already exists upload will fail)"
-helm repo add $HELM_REPO $VDK_HELM_REGISTRY_URL \
-      --username $VDK_HELM_REGISTRY_USERNAME \
-      --password $VDK_HELM_REGISTRY_PASSWORD
+helm repo add $HELM_REPO "$VDK_HELM_REGISTRY_URL" \
+      --username "$VDK_HELM_REGISTRY_USERNAME" \
+      --password "$VDK_HELM_REGISTRY_PASSWORD"
 # in v 0.10.0 the name of the push command was changed to cm-push
-helm cm-push $CHART_NAME-$CHART_VERSION.tgz $HELM_REPO
+helm cm-push "$CHART_NAME-$CHART_VERSION.tgz" $HELM_REPO
 echo "Check helm repo and new chart is accessible"
 helm repo update
-helm search repo $CHART_NAME --version $CHART_VERSION
+helm search repo "$CHART_NAME" --version "$CHART_VERSION"
diff --git a/projects/control-service/projects/helm_charts/pipelines-control-service/version.txt b/projects/control-service/projects/helm_charts/pipelines-control-service/version.txt
@@ -1 +1 @@
-1.2.16
+1.3