Use S3 Sync for CI deploys to avoid recopying files #7895
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We only want to use these values when using the script. Otherwise, AWS picks them up and tries to use them.
Eric-Arellano
changed the title
|
Going to run all CI via the branch build, not the PR build, since this impacts only non-PRs. I ran yesterday one shard on the PR build to confirm that this won't break PRs uploading |
benjyw
approved these changes
Jun 20, 2019
|
Big thanks to @benjyw for all the help to land this! |
Eric-Arellano
added a commit
that referenced
this pull request
Jun 21, 2019
#7895 works for branch builds, but not pull request builds due to the error `Partial credentials found in env, missing: AWS_SECRET_ACCESS_KEY`. This would happen because we always would define `AWS_SECRET_ACCESS_KEY`, but the value `AWS_SECRET_ACCESS_KEY` would only be defined on branch builds because it's encrypted. Instead, we should only define `AWS_SECRET_ACCESS_KEY` if we're on a branch build.
|
Thanks! Relatively little should already be in S3, because the path is 1) unique to the SHA, 2) unique to the wheel that is being created. I think that only universal wheels could potentially be uploaded multiple times. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We currently copy all of
dist/deployinto S3, even if the file is an exact copy of something already in S3. This poses a real cost, both monetary and time.Instead, here we use
aws s3 syncto only transfer files that are different from what is currently in the bucket.Will close #7258.