Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: action to rollback cloud run if health check fails #2

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

feat: action to rollback cloud run if health check fails #2

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
667c5e6
feat: action to rollback cloud run if health check fails
rot1024 Feb 18, 2025
bb82f34
update
rot1024 Feb 18, 2025
bb8ed7b
fix
rot1024 Feb 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
81 changes: 81 additions & 0 deletions .github/actions/deploy-cloud-run/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
name: Deploy Cloud Run service
description: Create a new revision and rollback if health check fails
inputs:
region:
description: Region of the Cloud Run service
required: true
image:
description: Image name
required: true
service:
description: Name of the Cloud Run service
required: true
health_check_url:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the services are publicly accessible anyway and we don't implement special host-based routing within the service itself, We might use path instead and leverage cloud-run URL for testing instead

Suggested change
health_check_url:
health_check_path:

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using Cloud Run URLs may cause the application to not work properly. Even if you use this, the addition of the Host header to curl is still required. In other words, the need to specify a URL unfortunately cannot be eliminated.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Noted on this. If this is the hard constraint from the service side then my suggestion is not relevant.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: this might be useful if we want to explicitly shows the path or list of paths for healthcheck

description: Health check URL. Multiple URLs can be separated by line breaks. If not provided, health check will be skipped.

runs:
using: composite
steps:
- name: Get current revision
id: get_revision
shell: bash
run: |
REV=$( \
gcloud run services describe "${{ inputs.service }}" \
--region "${{ inputs.region }}" \
--format="value(status.latestReadyRevisionName)" \
)
echo "Current revision: $REV"
echo "revision=$REV" >> "$GITHUB_OUTPUT"

- name: Deploy
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OTOH and could be out of scope but to be on the safer approach, we might as well utilize revision tags to do test on service healthcheck path before it's serving live traffic. Somewhat like blue/green deployment.

Hence potentially reducing chance of having to do rollback

https://cloud.google.com/run/docs/rollouts-rollbacks-traffic-migration#tags

shell: bash
run: |
gcloud run deploy "${{ inputs.service }}" \
--image "${{ inputs.image }}" \
--region "${{ inputs.region }}" \
--platform managed \
--quiet
Comment on lines +31 to +38
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could utilize this GH actions

Comment on lines +31 to +38
Copy link

@franzramadhan franzramadhan Feb 19, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we want to utilize the deploy-cloudrun github-action we can do somethink like this

Suggested change
- name: Deploy
shell: bash
run: |
gcloud run deploy "${{ inputs.service }}" \
--image "${{ inputs.image }}" \
--region "${{ inputs.region }}" \
--platform managed \
--quiet
- id: 'deploy_green'
name: Deploy new tagged-revision in green version
uses: 'google-github-actions/deploy-cloudrun@v2'
with:
service: ${{ inputs.service }}
image: ${{ inputs.image }}
region: ${{ inputs.region }}
tag_traffic: 'green=0' # this mean the new release will be having 0% traffic directed to it
- id: 'health_check_green'
name: Check if healthcheck path in green version returns healthy responses
if: ${{ inputs.health_check_url }}
shell: bash
run: |
for url in "{{ inputs.health_check_url }}"; do
echo "Checking health of $url"
echo "Parsing host of $url"
HOST=`echo $url | sed -E 's|https?://([^/]+).*|\1|'`
RESULT=`curl -m 10 -H "Host: $HOST" -s -o /dev/null -w "%{http_code}" "$url"`
echo "Status code is $RESULT"
if [ $RESULT -ge 300 ]; then
echo "Health check failed"
echo "error=true" >> "$GITHUB_OUTPUT"
break
fi
done

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rot1024 updated the suggestion to supply the Host Header


# ensure that the traffic is updated to the latest revision always
- name: Update traffic
shell: bash
run: |
gcloud run services update-traffic "${{ inputs.service }}" \
--to-latest \
--region "${{ inputs.region }}"
Comment on lines +40 to +46
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we are using aforementioned GH actions, we could make sure that Cloud Run will route traffic to the latest revision via GH actions inputs.

Comment on lines +40 to +46
Copy link

@franzramadhan franzramadhan Feb 19, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can later direct all traffic to green tag if healthcheck is okay

Suggested change
# ensure that the traffic is updated to the latest revision always
- name: Update traffic
shell: bash
run: |
gcloud run services update-traffic "${{ inputs.service }}" \
--to-latest \
--region "${{ inputs.region }}"
- id: 'direct_green'
name: Direct all traffics to green/latest revision if healthcheck is passing
if: ${{ steps.health_check.outputs.error != "true" }} # I believe `true` in health_check output is not parsed as bool but string
uses: 'google-github-actions/deploy-cloudrun@v2'
with:
service: ${{ inputs.service }}
image: ${{ inputs.image }}
region: ${{ inputs.region }}
revision_traffic: 'LATEST=100' # this mean the new release will be having 100% traffic directed to it
- name: Remove green tag
needs: direct_green
shell: bash
run: |
gcloud run services update-traffic "${{ inputs.service }}" \
--remove-tags green \
--region "${{ inputs.region }}"


# if the URL is invalid, curl itself will return an error code
- name: Health check
id: health_check
if: ${{ inputs.health_check_url }}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could utilize aforementioned GH actions output to get URL.

But the problem is to hit / or /api/ping as we have different healthcheck path for API service and Web service.

shell: bash
run: |
for url in ${{ inputs.health_check_url }}; do
echo "Checking health of $url"
RESULT=`curl -m 10 -s -o /dev/null -w "%{http_code}" "$url"`
echo "Status code is $RESULT"
if [ $RESULT -ge 300 ]; then
echo "Health check failed"
echo "error=true" >> "$GITHUB_OUTPUT"
break
fi
done

- name: Rollback
if: ${{ steps.health_check.outputs.error && steps.get_revision.outputs.revision }}
shell: bash
run: |
gcloud run services update-traffic "${{ inputs.service }}" \
--to-revision="${{ steps.get_revision.outputs.revision }}" \
--region "${{ inputs.region }}"

- name: Finish
shell: bash
run: |
if [ -z "${{ steps.health_check.outputs.error }}" ]; then
echo "Deployment successful"
else
echo "Deployment failed"
exit 1
fi