Put GCP-only 'scratch bucket' behind a flag #374
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Pangeo hubs have a `PANGEO_SCRATCH` env variable that points to a GCS bucket, used to share data between users. We implement that here too, but with a more generic `SCRATCH_BUCKET` env var (`PANGEO_SCRATCH` is also set for backwards compat). pangeo-data/pangeo-cloud-federation#610 has some more info on the use cases for `PANGEO_SCRATCH` Right now, we use Google Config Connector (https://cloud.google.com/config-connector/docs/overview) to set this up. We create Kubernetes CRDs, and the connector creates appropriate cloud resources to match them. We use this to provision a GCP Serivce account and a Storage bucket for each hub. Since these are GCP specific, running them on AWS fails. This PR puts them behind a switch, so we can work on getting things to AWS. Eventually, it should also support AWS resources via the AWS Service broker (https://aws.amazon.com/partners/servicebroker/) Ref 2i2c-org#366
6 tasks
|
Nice - do we have another deliverable to track implementing Scratch Buckets for AWS/Azure? If not maybe we should make one to track adding this functionality for those providers? |
damianavila
reviewed
May 4, 2021
damianavila
approved these changes
May 4, 2021
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pangeo hubs have a
PANGEO_SCRATCHenv variable thatpoints to a GCS bucket, used to share data between users.
We implement that here too, but with a more generic
SCRATCH_BUCKETenv var (
PANGEO_SCRATCHis also set for backwards compat).pangeo-data/pangeo-cloud-federation#610
has some more info on the use cases for
PANGEO_SCRATCHRight now, we use Google Config Connector
(https://cloud.google.com/config-connector/docs/overview)
to set this up. We create Kubernetes CRDs, and the connector
creates appropriate cloud resources to match them. We use this
to provision a GCP Serivce account and a Storage bucket for each
hub.
Since these are GCP specific, running them on AWS fails. This
PR puts them behind a switch, so we can work on getting things to
AWS.
Eventually, it should also support AWS resources via the
AWS Service broker (https://aws.amazon.com/partners/servicebroker/)
Ref #366