add action for deploying oss connector catalog to GCS #18633
Conversation
| - name: Setup Cloud SDK | ||
| uses: google-github-actions/setup-gcloud@v0 | ||
| with: | ||
| service_account_key: ${{ secrets.PROD_SPEC_CACHE_SA_KEY }} |
There was a problem hiding this comment.
Added this new secret after chatting with @git-phu, who verified the prod-spec-cache-bucket-SA-key from LastPass has access to the CMS bucket
| branches: | ||
| - master | ||
| paths: | ||
| - airbyte-config/init/src/main/resources/seed |
There was a problem hiding this comment.
nice - only runs when the right directory has been changed, only on the master branch.
| - name: Upload catalog to GCS | ||
| shell: bash | ||
| run: | | ||
| gcs_bucket_name="prod-airbyte-cloud-connector-metadata-service" |
There was a problem hiding this comment.
This isn't a secret, right?
There was a problem hiding this comment.
Should there be an OSS bucket that's distinct from the Cloud bucket?
There was a problem hiding this comment.
These files are meant to be public so I think it's ok for the bucket name to be in plaintext. The spec cache bucket name is also in the publish scripts.
As for having a separate bucket, infra said it was ok for them to be in the same bucket, and I think it might make things easier down the line if we want to actually put these behind a domain if they're in the same bucket.
There was a problem hiding this comment.
👍 then!
I guess ideally the bucket name wouldn't have had ☁️ in the name... but 🤷
* master: (38 commits) New Source: Gridly (#18342) 🎉 New Source: Alpha Vantage (#18320) ci_integration_test.sh: cut GITHUB_STEP_SUMMARY (#18895) 🎉 New Source: Datadog [python cdk] (#18150) Hide Reject all button in consent dialog (#18596) feat: add doc url to track event (#18690) fix: install java in oss catalog deploy action (#18887) [CI] Speed up check_images_exist (#18873) Extract open API (#18879) Remove unused interfaces (#18880) add action for deploying oss connector catalog to GCS (#18633) feat: generate full connector catalog json (#18562) Add unsupported_protocol_version column to Connection (#18876) Extract OAuth API (#18818) update images to have non-transparent background (#18874) DiscoverSchema endpoints calculates diff and breaking change (#18571) Validate protocol version on connector update (#18639) Bmoric/extract notification api (#18812) Show version and changelog status for affected connectors (#18845) Bmoric/extract logs api (#18621) ...
What
To be able to use the remote OSS catalog as a base, we need to upload it to GCS.
to merge after #18562
close https://github.com/airbytehq/airbyte-cloud/issues/3164
How
Create an action that will be auto triggered when a catalog change is merged to master. This action will generate the new oss_catalog.json and upload it to the GCS bucket.