Support listing >1000 file directories via Fog::Storage::GoogleJSON

[plribeiro3000]

I am connecting to Google Cloud Storage where I have 20,000+ records, but when I connect to the bucket with Fog and list all files I only get 1000. Is there a way to increase the number of files returned? I was hoping to programmatically go through all of these and make modifications, but now I'm stuck. Anyone? Thanks and great work on this Gem. 😄

[plribeiro3000]

The original issue was reported by @germs12 at fog/fog#3320.

cc/ @germs12 @icco @geemus

[Temikus]

Repro:

mkdir files
cd files
dd if=/dev/urandom of=dummy_file bs=1 count=100000
split -b 10 -a 10 dummy_file
cd ..
gsutil mb gs://fog-test-files
gsutil -m cp -r ./files gs://fog-test-files

connection = Fog::Storage::Google.new
connection.directories.get('fog-test-files').files.count
=> 1000
connection.list_objects('fog-test-files', {max_results: 10000}).items.count
=> 1000

Looking into API doc:

Maximum number of items plus prefixes to return in a single page of responses.
As duplicate prefixes are omitted, fewer total results may be returned than requested.
The service will use this parameter or 1,000 items, whichever is smaller.

So, we do not support proper page token enumeration in native Fog Fog::Storage::Directory model.

[geemus]

Could you run with EXCON_DEBUG=true to ensure the parameters are getting passed over to the provider as expected? It looks like the request isn't getting back as many results as expected, and without that the model wouldn't have more data to work with either. Does that make sense?

[github-actions]

This issue has been marked inactive and will be closed if no further activity occurs.

[jgigault-pa]

It seems that the JSON API part of Fog::Google does not take "next_page_token" information when listing files.

I have pushed a PR #558 to make a proof of concept, based on the implementation I could see in the XML API part.