Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for GCS emulator #275

Open
o0neup opened this issue Feb 20, 2025 · 2 comments
Open

Support for GCS emulator #275

o0neup opened this issue Feb 20, 2025 · 2 comments

Comments

@o0neup
Copy link

o0neup commented Feb 20, 2025

Hi folks!

I imagine it might be quite common to use some object storage emulators wrapped in testcontainers (e.g https://github.com/fsouza/fake-gcs-server for GCS). Is there a way to point obstore to the emulator running in a container?
The google's library has a way of passing the base API endpoint to the client (either via the env variable or explicitly), and perhaps there is or could be a way to do similar things with obstore?

for reference, fake-gcs-server does something like this: https://github.com/fsouza/fake-gcs-server/blob/main/examples/python/python.py (using the google's lib)

@kylebarron
Copy link
Member

kylebarron commented Feb 20, 2025

Obstore has been tested with minio and moto for S3, but I wasn't aware of an emulator for Google Cloud Storage.

It looks like this is supported (by the upstream Rust library that obstore uses) as long as you use a fork of fake-gcs-server that supports the GCS XML API: apache/arrow-rs#5263 (comment)

@o0neup
Copy link
Author

o0neup commented Feb 20, 2025

@kylebarron thank you for the swift reply!

@kylebarron kylebarron changed the title Interface to test obstore with object storage emulators Support for GCS emulator Feb 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants