[query] Unify CloudCredentials and Simplify BatchClient #14684
Merged
+1,072
−1,012
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This stack of pull requests is managed by Graphite. Learn more about stacking. |
ehigham
force-pushed
the
ehigham/cloud-credentials
branch
from
cd38e9b to
0ac3faa
Compare
ehigham
force-pushed
the
ehigham/enclosing
branch
from
992de5f to
a11a232
Compare
ehigham
force-pushed
the
ehigham/cloud-credentials
branch
from
0ac3faa to
9c4151a
Compare
ehigham
force-pushed
the
ehigham/enclosing
branch
from
a11a232 to
098f6a4
Compare
ehigham
force-pushed
the
ehigham/cloud-credentials
branch
from
9c4151a to
16b6635
Compare
This was referenced Sep 17, 2024
ehigham
force-pushed
the
ehigham/enclosing
branch
from
098f6a4 to
2b028ba
Compare
ehigham
force-pushed
the
ehigham/cloud-credentials
branch
from
9cff2fc to
d14679d
Compare
This was referenced Sep 20, 2024
ehigham
force-pushed
the
ehigham/enclosing
branch
from
2b028ba to
8c7b1c3
Compare
ehigham
force-pushed
the
ehigham/cloud-credentials
branch
from
d14679d to
ed2e8e8
Compare
ehigham
force-pushed
the
ehigham/enclosing
branch
from
8c7b1c3 to
3c702c0
Compare
ehigham
force-pushed
the
ehigham/cloud-credentials
branch
2 times, most recently
from
fc06f02 to
a101b55
Compare
ehigham
force-pushed
the
ehigham/cloud-credentials
branch
from
a101b55 to
1682a0f
Compare
ehigham
force-pushed
the
ehigham/enclosing
branch
from
2f05ecd to
83ff2cc
Compare
ehigham
force-pushed
the
ehigham/cloud-credentials
branch
from
1682a0f to
6e63e69
Compare
1 task
ehigham
force-pushed
the
ehigham/cloud-credentials
branch
from
6e36a7a to
e121222
Compare
ehigham
commented
Nov 19, 2024
This was referenced Dec 5, 2024
patrick-schultz
requested changes
Dec 12, 2024
ehigham
force-pushed
the
ehigham/cloud-credentials
branch
2 times, most recently
from
0a14104 to
4b932f1
Compare
ehigham
force-pushed
the
ehigham/cloud-credentials
branch
from
4b932f1 to
64042bc
Compare
patrick-schultz
approved these changes
Dec 16, 2024
grohli
pushed a commit
to grohli/hail
that referenced
this pull request
Jan 27, 2025
This change combines cloud auth logic that was previously duplicated between the various `FS` implementations and the `BatchClient`. The main refactoring is to make the interface between the `ServiceBackend` more high-level and leave json serialisation to the `BatchClient`. To do this, I've added a bunch of case classes that resemble the python objects the batch service expects (or a subset of the data). To simplify the interface, I've split batch creation from job submission (update). For QoB, the python client creates the batch before handing control to the query driver; batch creation is necessary for testing only. This change has low security impact as there are minor changes to the creation and scoping of service account credentials. Note that for each `FS`, credentials are scoped to the default storage oauth2 scopes for each service.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change combines cloud auth logic that was previously duplicated
between the various
FSimplementations and theBatchClient.The main refactoring is to make the interface between the
ServiceBackendmorehigh-level and leave json serialisation to the
BatchClient. To do this, I'veadded a bunch of case classes that resemble the python objects the batch service
expects (or a subset of the data). To simplify the interface, I've split batch
creation from job submission (update). For QoB, the python client creates the
batch before handing control to the query driver; batch creation is necessary
for testing only.
This change has low security impact as there are minor changes to the creation
and scoping of service account credentials. Note that for each
FS, credentialsare scoped to the default storage oauth2 scopes for each service.