Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

more complete support for remote filesystems like GCS and GDrive #21

Open
davemfish opened this issue Mar 8, 2024 · 0 comments
Open
Labels
enhancement New feature or request on hold there's a reason we aren't working on this yet

Comments

@davemfish
Copy link
Collaborator

In #18 , we integrated fsspec to add support for reading metadata over http. And GDAL already supports reading over http.

Since much of our data is on GCS and GDrive, it might make sense to add more explicit support for those backends too. http support allows reading data with public URLs from those locations, but if we add support for gcs:// and gdrivefs then we start supporting reading from private stores, and even writing metadata directly to those stores.

Here's some things to consider first, from @phargogh

I think we should explore this option a little more before making a decision on being able to read from private filesystems and write to these filesystems. Open questions on my mind include:

Can we read all fsspec-required auth parameters from GDAL's auth configuration?
How will we read a GDAL-supported dataset from Google Drive (which is not supported by GDAL's VSIs) without downloading the whole dataset?
Are there any other cases where GDAL will handle a URI with the same protocol prefix (e.g. gs://) as fsspec, or will we have to pass GDAL VSI-style path strings?

@davemfish davemfish changed the title more complete support for remote filesystems more complete support for remote filesystems like GCS and GDrive Mar 8, 2024
@davemfish davemfish added enhancement New feature or request on hold there's a reason we aren't working on this yet labels Sep 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request on hold there's a reason we aren't working on this yet
Projects
None yet
Development

No branches or pull requests

1 participant