Start Zarr download as soon as first page of entries is obtained

[jwodder]

Closes #1447.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 88.43%. Comparing base (5c5c3a3) to head (2db4dba).
Report is 2 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1569      +/-   ##
==========================================
+ Coverage   88.35%   88.43%   +0.07%     
==========================================
  Files          78       78              
  Lines       10735    10735              
==========================================
+ Hits         9485     9493       +8     
+ Misses       1250     1242       -8     

Flag	Coverage Δ
unittests	88.43% <100.00%> (+0.07%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[yarikoptic]

cool! Indeed starts to show progress right away! one gotcha only is that

size is reported 0 and that seems to be due to initial msg somewhere:

❯ DANDI_DEVEL=1 dandi download -f debug dandi://dandi/000026/sub-I48/ses-SPIM/micr/sub-I48_ses-SPIM_sample-BrocaAreaS03_stain-Somatostatin_SPIM.ome.zarr
2025-01-28 16:07:09,518 [ WARNING] Parallel downloads are not yet implemented for non-pyout format='debug'. Download will proceed serially.
{'size': 0, 'path': 'sub-I48_ses-SPIM_sample-BrocaAreaS03_stain-Somatostatin_SPIM.ome.zarr'}
{'status': 'downloading', 'path': 'sub-I48_ses-SPIM_sample-BrocaAreaS03_stain-Somatostatin_SPIM.ome.zarr'}
{'done': 0, 'done%': 0, 'path': 'sub-I48_ses-SPIM_sample-BrocaAreaS03_stain-Somatostatin_SPIM.ome.zarr'}
2025-01-28 16:07:11,989 [    INFO] File .zgroup in Zarr 7933bc72-ec3f-4043-8076-d7aa8a70e4f7 successfully downloaded
{'done': 24, 'done%': 0, 'path': 'sub-I48_ses-SPIM_sample-BrocaAreaS03_stain-Somatostatin_SPIM.ome.zarr'}
{'message': '1 done', 'path': 'sub-I48_ses-SPIM_sample-BrocaAreaS03_stain-Somatostatin_SPIM.ome.zarr'}

do you see where that size is reported, and whether could be set to e.g. None and not reported until determined?

[jwodder]

@yarikoptic That's because the API reports that asset as having a size of zero:

$ curl -fsSL https://api.dandiarchive.org/api/dandisets/000026/versions/draft/assets/'?path=sub-I48/ses-SPIM/micr/sub-I48_ses-SPIM_sample-BrocaAreaS03_stain-Somatostatin_SPIM.ome.zarr' | jq .
{
  "count": 1,
  "next": null,
  "previous": null,
  "results": [
    {
      "asset_id": "60db4111-6d6e-4ad1-a161-63a2789cde2e",
      "blob": null,
      "zarr": "7933bc72-ec3f-4043-8076-d7aa8a70e4f7",
      "path": "sub-I48/ses-SPIM/micr/sub-I48_ses-SPIM_sample-BrocaAreaS03_stain-Somatostatin_SPIM.ome.zarr",
      "size": 0,
      "created": "2024-06-14T11:22:58.647070Z",
      "modified": "2024-06-14T11:22:58.663144Z"
    }
  ]
}

[yarikoptic]

Ah, I found the worst exactly to try on in my bag history. Thanks for digging!