Use AWS Rust SDK To Source Credentials For S3

[tustvold]

Description

Problem

object_store provides a limited selection of common authentication mechanisms, with a particular focus on those used in server contexts. It does not, however, seek to replicate the entire credentials setup of the various SDKs. Polars is often run on end user devices and therefore people wish for it to support a broader range of authentication options.

object_store exposes a CredentialProvider API that can be used to provide an alternative way to source credentials.

Proposal

#18979 tracks exposing CredentialProvider in a way that it can be configured, there are, however, some design questions around what this might look like through a python API.

An alternative would be for polars to provide an option to use aws-sdk-rust to source credentials, much like datafusion-cli does.

Alternatives Considered

Users could use software like aws-vault to generate session credentials, whilst this has other security benefits, for various reasons people may not wish to do this.

We could expose the full CredentialProvider API to users. This would be more flexible, support providers other than AWS, and avoid adding some non-trivial additional dependencies, but requires more design work.

Related Context

• AWS_PROFILE should be supported in cloud storage I/O config #18757
• Issue reading S3 files #18907
• Feature request: allow user to use a boto3-native credential provider/session for s3 etc. #15838
• Cannot pass GCP access token in storage_options (Google Cloud) #13138
• [object-store]: Implement credential_process support for S3 apache/arrow-rs#6422
• object_store: Use the official AWS SDK for s3 interactions apache/arrow-rs#5143
• object_store: Move Away From SDKs  apache/arrow-rs#2176

[alamb]

BTW here is an example of how we use the AWS SKD to get credentials in datafusion-cli:

https://github.com/apache/datafusion/blob/747001a41481e0cf39dc758a85d1bdb64fdeb7c0/datafusion-cli/src/object_storage.rs#L65-L133

[kylebarron]

It looks like this is closed by #18979?