remote read of NWB files using MATLAB

[bendichter]

It is supposed to be possible to do a remote read of s3 hdf5 files in MATLAB, but when I try to get this:

>> H5F.open('https://dandiarchive.s3.amazonaws.com/blobs/58c/537/58c53789-eec4-4080-ad3b-207cf2a1cac9')

Error using hdf5lib2
Unable to access
'https://dandiarchive.s3.amazonaws.com/blobs/58c/537/58c53789-eec4-4080-ad3b-207cf2a1cac9'.
The specified URL scheme is invalid.

I am guessing that "The specified URL scheme is invalid." is referring to the fact that the url does not start with "s3://". Why doesn't it? Do we have an alternative url that fits this pattern?

[satra]

this is because the hdf5 ros3 driver uses libcurl underneath which only understands the http protocol.

you can replace: https://dandiarchive.s3.amazonaws.com/ with s3://dandiarchive to get the s3 location. i believe i've tried this in the past with matlab, and i'm not sure that will work either. i don't have matlab on my current machine to try.

here is a bit of code:

    s3loc = [val for val in contenturl if ".s3.amazonaws" in val]
    assert len(s3loc) == 1
    bucket = s3loc[0].split("//")[1].split(".s3.amazonaws")[0]

[nvishnoiMW]

Since it is a publicly accessible bucket we want to reset them.

getenv('AWS_ACCESS_KEY_ID')

ans =

0×0 empty char array

getenv('AWS_SECRET_ACCESS_KEY')

ans =

0×0 empty char array

[bendichter]

>> getenv('AWS_ACCESS_KEY_ID')


ans =

  0×0 empty char array

>> 
>> getenv('AWS_SECRET_ACCESS_KEY')


ans =

  0×0 empty char array

>> 
>> H5F.open(s3_url)
Error using hdf5lib2
Unable to open or create
's3://dandiarchive/blobs/e08/02c/e0802c3e-5492-42a6-960a-bcf5b3cfd239'
due to internal error.

Error in H5F.open (line 130)
file_id = H5ML.hdf5lib2('H5Fopen', filename, flags, fapl, is_remote);

[nvishnoiMW]

is it same for @lawrence-mbf?

[nvishnoiMW]

@bendichter I am assuming the issue persists with fopen as well even after resetting the variables?

[lawrence-mbf]

@nvishnoiMW

As mentioned above, this is on Windows 10 using MATLAB R2021b.

[bendichter]

The solution involved 2 fixes.

First to use this url: s3_url = "s3://dandiarchive/blobs/e08/02c/e0802c3e-5492-42a6-960a-bcf5b3cfd239"

Second, to fix a credentials issue, @nvishnoiMW proposed in another thread to delete stale credentials by deleting the file ~/.aws/credentials (on Windows something like C:/Users/username/.aws/credentials)

[bendichter]

@satra could you help me figure out what's going on here?

from h5py import File

s3_url = "https://dandiarchive.s3.amazonaws.com/blobs/58c/537/58c53789-eec4-4080-ad3b-207cf2a1cac9"

file = File(s3_url, mode = "r", driver="ros3")
file.attrs['nwb_version']

2.2.5

bucket = s3_url.replace("https://dandiarchive.s3.amazonaws.com", "s3://dandiarchive")
print(bucket)

s3://dandiarchive/blobs/58c/537/58c53789-eec4-4080-ad3b-207cf2a1cac9

file = File(bucket, mode = "r", driver="ros3")

---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
Input In [28], in <module>
      9 bucket = s3_url.replace("https://dandiarchive.s3.amazonaws.com/", "s3://dandiarchive")
     10 print(bucket)
---> 11 file = File(bucket, mode = "r", driver="ros3")

File /opt/conda/lib/python3.9/site-packages/h5py/_hl/files.py:507, in File.__init__(self, name, mode, driver, libver, userblock_size, swmr, rdcc_nslots, rdcc_nbytes, rdcc_w0, track_order, fs_strategy, fs_persist, fs_threshold, fs_page_size, page_buf_size, min_meta_keep, min_raw_keep, locking, **kwds)
    502     fapl = make_fapl(driver, libver, rdcc_nslots, rdcc_nbytes, rdcc_w0,
    503                      locking, page_buf_size, min_meta_keep, min_raw_keep, **kwds)
    504     fcpl = make_fcpl(track_order=track_order, fs_strategy=fs_strategy,
    505                      fs_persist=fs_persist, fs_threshold=fs_threshold,
    506                      fs_page_size=fs_page_size)
--> 507     fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
    509 if isinstance(libver, tuple):
    510     self._libver = libver

File /opt/conda/lib/python3.9/site-packages/h5py/_hl/files.py:220, in make_fid(name, mode, userblock_size, fapl, fcpl, swmr)
    218     if swmr and swmr_support:
    219         flags |= h5f.ACC_SWMR_READ
--> 220     fid = h5f.open(name, flags, fapl=fapl)
    221 elif mode == 'r+':
    222     fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl)

File h5py/_objects.pyx:54, in h5py._objects.with_phil.wrapper()

File h5py/_objects.pyx:55, in h5py._objects.with_phil.wrapper()

File h5py/h5f.pyx:106, in h5py.h5f.open()

OSError: Unable to open file (invalid SCHEME construction)

[satra]

@bendichter - as noted earlier in this thread h5py is based on hdf5 which is based on libcurl and only accepts the http protocol, not the s3 protocol.

[bendichter]

ah, ok, I understood that MATLAB couldn't take the https, but not that h5py couldn't take s3