Add PieceCache::read_pieces()
method for more efficient piece cache reads in batches
#3147
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This builds on #3135 and #3141 to implement more efficient piece reading.
For local cache we now have a pool of open files.
DirectIoFile
has mutex internally due to need to manage its read/write buffer for aligned I/O and without pool concurrent reads are not actually happening despite&self
of theFileExt
trait it implements. Also it is from our experience with farmer faster to read from multiple handles in general, especially on Windows (though due to memory usage issues Windows is still doing blocking non-concurrent reads).For cluster cache we now have stream request for faster retrieval of pieces, which was much easier to implement after refactoring done in #3141
I also clarified guarantees about returned values in a few places.
The biggest remaining topic is implementing
PieceGetter::get_pieces()
forClusterPieceGetter
and remove dummy implementation fromPieceGetter
.Not particularly happy with number of hashmaps required in piece cache handling, but I didn't want to force all results being returned in-order either and without guaranteed order we need an efficient way to track pieces, hence hashmaps 😞
Code contributor checklist: