Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

azure_storage should use azure_core::Body (or some variant) instead of Bytes #646

Closed
bmc-msft opened this issue Jan 28, 2022 · 2 comments · Fixed by #943
Closed

azure_storage should use azure_core::Body (or some variant) instead of Bytes #646

bmc-msft opened this issue Jan 28, 2022 · 2 comments · Fixed by #943
Labels
Storage Storage Service (Queues, Blobs, Files)

Comments

@bmc-msft
Copy link
Contributor

Right now, all of the blob storage operations require passing Bytes, which translates to having everything in memory to make a request. This makes uploading large files exceedingly difficult, even uploading the file using multiple PutBlock calls.

Right now, Blob storage allows blobs to have up to 50,000 blocks, with each block up to 4000 MiB, for a total of approximately 190.7TiB. 1

Right now, in order to upload such a file, each 4000MiB block must be entirely read into memory before uploading. Both reqwest 2 and hyper 3 support a mechanism to wrap a Stream, which would enable streaming the input, rather than reading it entirely in memory first.

Work added in #230 started the work to support this by adding azure_core::SeekableStream.

Footnotes

  1. https://docs.microsoft.com/en-us/azure/storage/blobs/scalability-targets#scale-targets-for-blob-storage

  2. https://docs.rs/reqwest/0.11.9/reqwest/struct.Body.html#method.wrap_stream

  3. https://docs.rs/hyper/latest/hyper/body/struct.Body.html#method.wrap_stream

@cataggar
Copy link
Member

May be we should look at building off of https://crates.io/crates/http-types instead of https://crates.io/crates/http.

Body represents the HTTP body of both Request and Response. It’s completely streaming, and implements AsyncBufRead to make reading from it both convenient and performant.
https://docs.rs/http-types/latest/http_types/struct.Body.html

cc @yoshuawuyts

@evanxg852000
Copy link
Contributor

This is pretty much a needed improvement that can allow things like AsyncRead instead of loading the bytes.

@bmc-msft bmc-msft added the Storage Storage Service (Queues, Blobs, Files) label Jul 14, 2022
@bmc-msft bmc-msft linked a pull request Jul 19, 2022 that will close this issue
@github-actions github-actions bot locked and limited conversation to collaborators Feb 4, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Storage Storage Service (Queues, Blobs, Files)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants