two papers with overlapping data

[bendichter]

Hi team,

Generally, I create dandisets that are collections of sessions associated with a specific paper. I have encountered my first situation where there are two papers that have overlapping data. I expect this to come up quite a bit as more and more data is published on here and reused. What is the practice for creating dandisets in this case? I can easily upload the data twice, once in each place, but that would duplicate data on the DANDI archive so maybe there is a better solution.

Ben

[satra]

DANDI supports this and also allows you to create relationships between dandisets through the metadata. You can upload the same asset to two dandisets, and DANDI won't duplicate data if the same data blob is uploaded. In a future iteration you will simply be able to "copy" an asset from one dandiset to another or construct a dandiset from the result of a search on the archive.

[kabilar]

Perfect, thank you all

[kabilar]

Hi team,

I was able to link a json file to a new dandiset, but am unable to link a zarr file. I think I am missing something simple since blobs are handled differently than zarr (zarr developer docs).

For a zarr file, I changed line 29 in the above gist script from "blob_id": asset.blob to "zarr_id": asset.zarr, but am receiving the error below at line 32. The error message appears to be getting invoked in the dandiapi here.

Error message

Error 400 while sending POST request to https://api.dandiarchive.org/api/dandisets/000724/versions/draft/assets/: "The zarr archive belongs to a different dandiset"

Entire traceback

HTTPError                                 Traceback (most recent call last)
File ~/Desktop/add_asset_to_dandiset.py:32
     31     # this will generate an exception if the path already exists
---> 32     out = da.post(path=ds.api_path + 'versions/draft/assets/', data=data)

File ~/miniconda3/envs/dandi/lib/python3.10/site-packages/dandi/dandiapi.py:308, in RESTFullAPIClient.post(self, path, **kwargs)
    304 def post(self, path: str, **kwargs: Any) -> Any:
    305     """
    306     Convenience method to call `request()` with the 'POST' HTTP method.
    307     """
--> 308     return self.request("POST", path, **kwargs)

File ~/miniconda3/envs/dandi/lib/python3.10/site-packages/dandi/dandiapi.py:274, in RESTFullAPIClient.request(self, method, path, params, data, files, json, headers, json_resp, retry_statuses, retry_if, **kwargs)
    272         raise HTTP404Error(msg, response=result)
    273     else:
--> 274         raise requests.HTTPError(msg, response=result)
    276 if json_resp:
    277     if result.text.strip():

HTTPError: Error 400 while sending POST request to https://api.dandiarchive.org/api/dandisets/000724/versions/draft/assets/: "The zarr archive belongs to a different dandiset"

[yarikoptic]

eh, yet another "zarrs are special" case I guess. We really need to approach finalizing their design in terms of supporting versioning/publishing and this current 1-to-1 mapping of them to dandisets. attn @satra and @waxlamp

[kabilar]

Thank you @yarikoptic. That is good to know. For now as a workaround, I will point users of this new dandiset 724 to the zarr assets in the existing dandiset 26.

[kabilar]

Continuing the discussion in dandi/dandi-archive#1792.