Pass conn ID to ObjectStoragePath via URI #35913
Conversation
|
I like being able to use an uri, but what did you base the format on? |
|
From AIP-48 (Data-aware scheduling):
I don’t think this has any special semantics in Dataset scheduling currently, but having the connection ID in the auth part of the URI seems reasonable since Connection does hold auth information. |
|
Can you add this to the documentation of object storage (how to use it)? Then lgtm. |
This enables an alternative ObjectStoragePath init syntax, using the auth section in the URI to supply conn ID instead of a separate keyword argument. The explicit keyword argument is honored if supplied.
uranusjr
force-pushed
the
fs-conn-id-auth
branch
from
a2c2542 to
0e6f7e0
Compare
|
I added a paragraph in the tutorial to explain the usage. |
While unlikely, it is *possible* for CPython to change how username and password is parsed from netloc. To ensure we always do the right thing, it is better to just implement the logic outselves so it always work as expected. We need this logic to get the host info anyway, so it is not possible to solely rely on CPython's parser.
|
Whatever that should go into the URI’s auth section should go into Connection since we don’t want users to put the real credentials in a DAG in any scenarios. One example is HTTP; while technically we can allow putting auth here, we should not. Whatever is the need, the actual auth should go into Connection, and be referenced via the ID in the path instead. |
|
Were you planning on making it clearer to the user when it fails (ie. a username, password probably has been provided)? |
|
I do, but we don’t currently support any fs that logically takes auth from the URI, so I figure it’s better to split that into a separate changeset. Also I’m not quite sure when that error should be implemented yet. |
ephraimbuddy
added
changelog:skip
This enables an alternative ObjectStoragePath init syntax, using the auth section in the URI to supply conn ID instead of a separate keyword argument. The explicit keyword argument is honored if supplied. (cherry picked from commit ab87cd0)
This enables an alternative ObjectStoragePath init syntax, using the auth section in the URI to supply conn ID instead of a separate keyword argument. The explicit keyword argument is honored if supplied.
This enables an alternative ObjectStoragePath init syntax, using the auth section in the URI to supply conn ID instead of a separate keyword argument. The explicit keyword argument is honored if supplied.