Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: download HAMT-sharded directories #1890

Merged
merged 2 commits into from
Feb 1, 2022
Merged

fix: download HAMT-sharded directories #1890

merged 2 commits into from
Feb 1, 2022

Conversation

hacdias
Copy link
Member

@hacdias hacdias commented Jan 14, 2022

fixes #1885

License: MIT
Signed-off-by: Henrique Dias [email protected]

License: MIT
Signed-off-by: Henrique Dias <[email protected]>
@hacdias hacdias requested a review from lidel January 14, 2022 09:55
@hacdias hacdias self-assigned this Jan 14, 2022
@hacdias hacdias temporarily deployed to Deploy January 14, 2022 10:01 Inactive
Copy link
Member

@lidel lidel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Disclaimer: this is the best we could do, given what go-ipfs 0.11 provides:

  • object patch is deprecated. It is not aware of HAMT-sharding, nor it will keep block size under 1MB
    • unsafe to use, we have to remove this ASAP
  • Tracking issue for UnixFS automatic sharding kubo#8106 made MFS pretty smart: it will autoshard/unshard and keep block sizes in check
    • the downside is that this MFS dance looks a bit awkward: it "pollutes" MFS root with temporary directory, but we prefix it with zzzz so it should not impact GUIs even when Files screen is opened at the time of execution.
    • we could pin the ephemeral CID to ensure its not GC'd after removal from MFS, but in practice it's not worth bothering given this is only for download action in the desktop app
  • this only solves block/hamt problem (when object patch mutaterd HAMT without knowing what HAMT is). The entire payload is still buffered in memory – we track that in File → menu → Download buffers entire file into a blob #1887

In the future:

  • we will have dag patch ('ipfs dag patch' kubo#4782), but it will work at the DAG level (too low, does not abstract sharding)
  • .. so we need to find a better abstraction for creating ephemeral directories for use cases like this one
    • could be ipfs files patch that works on MFS paths but also on /ipfs/{cid}, becoming true replacement for ipfs object patch
    • ..or ability to have more than one MFS root
    • .. or something else

@lidel lidel merged commit 22a1112 into main Feb 1, 2022
@lidel lidel deleted the fix/1885 branch February 1, 2022 02:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
No open projects
Archived in project
Development

Successfully merging this pull request may close these issues.

Downloading Multiple Large Files Fails
2 participants