You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Stream object downloads instead of buffering objects in memory (#829)
* Stream object downloads instead of buffering object in memory
Fixes#828.
This PR makes it so objects in the FS backed are streamed from disk
instead of being loaded into memory. This drastically reduces memory
usage when working with large files, especially when performing small
concurrent range reads on a large file.
A StreamingObject type has been added to both the fakestorage and API
packages. This type contains a io.ReadSeekCloser that can be used to
access object data.
This change sets the stage for fixing #669 and #397. #669 isn't fixed
because it will require re-working of how resumable uploads are stored
(though other upload types have been made streaming in this PR). Also,
fixing #397 would require changing how initial objects are specified
when starting the server, which with this approach would require
changing the public API.
Regarding the public API: the server methods for working with objects
have been changed to use StreamingObject instead of Object. This is a
breaking change. It would be possible to make streaming backward
compatible by adding some metadata and methods to Object (to distinguish
between streaming and buffered objects), but I went with this approach
because it made identifying the code that needed to be updated easier;
the compiler pointed out the locations to update instead of needing to
track down runtime and test errors. An additional improvement, which is
compatible with the backward compatible approach, would be to add an
OpenForReading method on Object so callers that are currently
responsible for closing objects also explicitly open objects (which
would make the responsibility clearer). Anyway, I wanted to get this
initial approach up for feedback before putting in additional work.
* Fix typo in xattr_windows.go
* Fix staticcheck lint errors.
* Attempt to fix renaming over open files on Windows
* Add build directive to rename_unix.go
* Close objects when listing objects and in tests
* Use compatOpenForWritingAllowingRename for temp file
* Close temp file
* Move the Windows "temporary delete path" to the temp dir
* Close initial objects
* Bring back original non-streaming public APIs
* Apply suggestions from code review
Applying it myself so we can ship this. I'll run `go fix` after merging.
Co-authored-by: Steve Coffman <[email protected]>
Co-authored-by: fsouza <[email protected]>
Co-authored-by: Steve Coffman <[email protected]>
0 commit comments