pipelined extraction #236

cosmicexplorer · 2024-08-17T18:42:23Z

Recreation of #208 to work around github issues.

Problem

ZipArchive::extract() corresponds to the way most zip implementations perform the task, but it's single-threaded. This is appropriate under the assumptions imposed by rust's Read and Seek traits, where mutable access is necessary and only one reader can extract file contents at a time, but most unix-like operating systems offer a pread() operation which avoids mutating OS state like the file offset, so multiple threads can read from a file handle at once. The go programming language offers io.ReaderAt in the stdlib to codify this ability.

Solution

This is a rework of #72 which avoids introducing unnecessary thread pools and creates all output file handles and containing directories up front. For large zips, we want to:

create output handles and containing directories up front,
split the input file handle into chunks to process the constituent file entries in parallel,
for large compressed entries, pipe their content into a dedicated stream to avoid intermixing i/o and decompression and blocking quick small entries later in the file.

src/read/split.rs was created to cover pread() and other operations, while src/read/pipelining.rs was created to perform the high-level logic to split up entries and perform pipelined extraction.

Result

The parallelism feature was added to the crate to gate the newly added code + API.
A dependency on the libc crate was added for #[cfg(all(unix, feature = "parallelism"))] in order to make use of OS-specific functionality.
zip::read::split_extract() was added as a new external API to extract &ZipArchive<fs::File> when #[cfg(all(unix, feature = "parallelism"))].

Note that this does not handle symlinks yet, which I plan to add in a followup PR.

CURRENT BENCHMARK STATUS

On a linux host (with splice() and optionally copy_file_range()), we get about a 6.5x speedup with 12 decompression threads:

> cargo bench --features parallelism -- extract
running 2 tests
test extract_basic           ... bench: 104,389,978 ns/iter (+/- 5,715,453) = 85 MB/s
test extract_split           ... bench:  16,274,974 ns/iter (+/- 1,530,257) = 546 MB/s

The performance should keep increasing as we increase thread count, up to the number of available CPU cores (this was running with a parallelism of 12 on my 16-core laptop). This also works on macOS and BSDs, and other #[cfg(unix)] platforms.

src/read/pipelining.rs

src/read/split.rs

src/types.rs

src/read/pipelining.rs

cosmicexplorer · 2025-01-16T21:01:34Z

Going to try to get this one in before figuring out the cli PR.

- initial sketch of lexicographic trie for pipelining - move path splitting into a submodule - lex trie can now propagate entry data - outline handle allocation - mostly handle files - mostly handle dirs - clarify symlink FIXMEs - do symlink validation - extract writable dir setting to helper method - modify args to handle allocation method - handle allocation test passes - simplify perms a lot - outline evaluation - handle symlinks - BIGGER CHANGE! add EntryReader/etc - make initial pipelined extract work - fix file perms by writing them after finishing the file write - support directory entries by unix mode as well - impl split extraction - remove dependency on reader refactoring - add dead_code to methods we don't use yet

Pr0methean

Here's a review of what I've read so far. Still needs a fair bit of work, but I'm happy with the overall concept.

benches/extract.rs

src/read/pipelining.rs

Pr0methean · 2025-01-17T20:52:23Z

src/read/pipelining.rs

+    #[derive(PartialEq, Eq, Debug, Clone)]
+    pub(crate) struct DirEntry<'a, Data> {
+        pub properties: Option<Data>,
+        pub children: BTreeMap<&'a str, Box<FSEntry<'a, Data>>>,


If we were to Box the BTreeMap instead of its individual entries, what difference would that make to pointer chasing and heap fragmentation?

So this data structure only ever exists temporarily, and is freed by transform_entries_to_allocated_handles(), which runs to completion before we engage in any parallelism. The reason for this trie is purely to deduplicate subdirectories to create and to ensure uniqueness of output file handles.

...however, after trying out this change, I think I prefer this approach anyway. I'm under the impression that pointer chasing and heap fragmentation are less of an issue given that this is a temporary data structure, but this seems to make more sense to me.

Let me know if I'm correct in thinking that the temporary nature of this data structure means we can avoid analyzing heap fragmentation too deeply!

Yes, that's fine, as long as we're not interleaving expansions of it with expansions of anything longer-lasting. It's when the fragmented structure is dropped, but other objects in between its elements don't, that fragmentation can be a problem.

src/read/pipelining.rs

Pr0methean · 2025-01-18T18:22:52Z

src/read/pipelining.rs

+        pub file_range_copy_buffer_length: usize,
+        /// Size of buffer used to splice contents from a pipe into an output file handle.
+        ///
+        /// Used on non-Linux platforms without [`splice()`](https://en.wikipedia.org/wiki/Splice_(system_call)).


This buffer isn't necessary on any Unix; see https://stackoverflow.com/a/10330172.

I'm not sure I understand your meaning here. That answer seems to say that on non-linux platforms, read()/write() with an explicit buffer (as we do here) is the way to go. Our PipeReadBufferSplicer struct performs read() then pwrite_all() with an explicit buffer, because we can't use splice().

Do I misunderstand you here?

What it's saying is that on Unix, when you don't have splice() you should memmap() the file directly and pass the mapped region to write(). The memmap2 crate will provide the wrapper we need.

src/read/pipelining.rs

cosmicexplorer · 2025-02-05T22:04:17Z

thank you so much for these wonderful comments!!

- initialize the test archives exactly once in statics - add benchmarks for dynamic and static test data - use lazy_static

- fix deflate feature flag in test

cosmicexplorer · 2025-02-12T02:48:37Z

Was able to remove the non_local_definitions lint ignore after updating displaydoc to 0.2.5!

src/read/pipelining.rs

this may technically reduce heap fragmentation, but since this data structure only exists temporarily, that's probably not too important. instead, this change just reduces the amount of coercion and unboxing we need to do

src/read/handle_creation.rs

cosmicexplorer · 2025-02-13T15:02:10Z

Hey @Pr0methean -- think I got to all of your comments! I proposed a couple compromises to do in follow-up PRs (supporting absolute extraction paths and symlinks)--let me know if you agree! I am hoping to spend more time on this in the next few weeks to get it in and then do the follow-ups. No rush as usual, and I really appreciate your comments.

Pr0methean · 2025-02-25T20:32:20Z

benches/extract.rs


    let params = ExtractionParameters {
-        decompression_threads: DECOMPRESSION_THREADS,
+        decompression_threads: num_cpus::get() / 3,


What will the other 2/3 of the CPUs be doing? Also, does this need to be clamped to at least 1?

Pr0methean

Looks good; just 2 minor comments.

Pr0methean · 2025-02-25T20:37:19Z

src/spec.rs

-        let block = Self::from_le(block);
+    /// Convert endianness and check the magic value.
+    #[allow(clippy::wrong_self_convention)]
+    fn validate(self) -> ZipResult<Self> {


Call this function from_le_validated to make its combined functionality more clear, or separate out the from_le call and call it with_checked_magic.

Signed-off-by: Chris Hennick <[email protected]>

github-advanced-security bot found potential problems Aug 17, 2024

View reviewed changes

cosmicexplorer mentioned this pull request Aug 17, 2024

parallel/pipelined extraction #208

Closed

github-advanced-security bot found potential problems Aug 17, 2024

View reviewed changes

src/read/pipelining.rs Fixed Show fixed Hide fixed

src/read/pipelining.rs Fixed Show fixed Hide fixed

cosmicexplorer force-pushed the pipelined-extract-v2 branch 4 times, most recently from 7a45b32 to 5cec332 Compare August 21, 2024 04:21

cosmicexplorer added 8 commits January 16, 2025 16:03

bzip2 support needed for benchmark test

a1734b1

more review comments

3f0a887

fix merge errors

ec63cdd

correctly handle backslashes in entry names (i.e. don't)

c311784

make PathSplitError avoid consing a String until necessary

8efb409

add repro_old423 test for pipelining

3c19c28

silence dead code warnings for windows

6ccecd0

cosmicexplorer force-pushed the pipelined-extract-v2 branch 4 times, most recently from b90c9e2 to fa18aa3 Compare January 16, 2025 21:23

fix ci error

7332eb6

cosmicexplorer force-pushed the pipelined-extract-v2 branch from fa18aa3 to 7332eb6 Compare January 16, 2025 21:24

avoid erroring for top-level directory entries

c9dd876

cosmicexplorer marked this pull request as ready for review January 17, 2025 00:48

cosmicexplorer requested a review from Pr0methean January 17, 2025 00:50

Pr0methean requested changes Jan 18, 2025

View reviewed changes

cosmicexplorer added 3 commits February 9, 2025 18:25

use num_cpus by default for parallelism

17d611e

we spawn three threads per chunk

5bd512e

add dynamically-generated test archive

8dd83d3

- initialize the test archives exactly once in statics - add benchmarks for dynamic and static test data - use lazy_static

cosmicexplorer added 2 commits February 11, 2025 21:25

remove some lint ignores

7ea1b85

- fix deflate feature flag in test

add back default features for displaydoc and update version

6f9d3d6

cosmicexplorer force-pushed the pipelined-extract-v2 branch from 15c3ef5 to 6f9d3d6 Compare February 12, 2025 02:40

add FIXME for follow-up work to support absolute paths

f4c43cb

github-advanced-security bot found potential problems Feb 12, 2025

View reviewed changes

src/read/pipelining.rs Dismissed Show dismissed Hide dismissed

cosmicexplorer added 6 commits February 12, 2025 00:00

box each level of the b-tree together with its values

1ae44f0

this may technically reduce heap fragmentation, but since this data structure only exists temporarily, that's probably not too important. instead, this change just reduces the amount of coercion and unboxing we need to do

impl From<DirEntry<...>> for FSEntry

c92d018

make some tests return Result

d6da333

simplify some btreemap creation

41381b1

move handle_creation module to a separate file

4fc051a

downgrade HandleCreationError to io::Error

730f18f

github-advanced-security bot found potential problems Feb 13, 2025

View reviewed changes

src/read/handle_creation.rs Dismissed Show dismissed Hide dismissed

src/read/handle_creation.rs Dismissed Show dismissed Hide dismissed

src/read/handle_creation.rs Dismissed Show dismissed Hide dismissed

src/read/handle_creation.rs Dismissed Show dismissed Hide dismissed

cosmicexplorer added 9 commits February 13, 2025 08:17

use ByAddress over ZipDataHandle

2f8e5b4

fix lint error

0378ab9

make an assert into a debug assert

295c7f0

remove extraneous error case

436570f

replace unsafe transmutes with Pod methods

a22c53b

add dead code ignore

24c4425

use if let for matching

11321bd

fix dead code ignores

06f7b89

add note about shared future dependency task DAG

d6bbc9c

Pr0methean reviewed Feb 25, 2025

View reviewed changes

Pr0methean requested changes Feb 25, 2025

View reviewed changes

Pr0methean added 2 commits March 16, 2025 22:03

Merge branch 'master' into pipelined-extract-v2

7bbe4ae

Signed-off-by: Chris Hennick <[email protected]>

Merge branch 'master' into pipelined-extract-v2

1369bfe

Signed-off-by: Chris Hennick <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pipelined extraction #236

pipelined extraction #236

cosmicexplorer commented Aug 17, 2024 •

edited

Loading

cosmicexplorer commented Jan 16, 2025

Pr0methean left a comment

Pr0methean Jan 17, 2025

cosmicexplorer Feb 12, 2025

Pr0methean Mar 17, 2025 •

edited

Loading

Pr0methean Jan 18, 2025

cosmicexplorer Feb 13, 2025

Pr0methean Mar 17, 2025 •

edited

Loading

cosmicexplorer commented Feb 5, 2025 •

edited

Loading

cosmicexplorer commented Feb 12, 2025

cosmicexplorer commented Feb 13, 2025

Pr0methean Feb 25, 2025

Pr0methean left a comment

Pr0methean Feb 25, 2025

pipelined extraction #236

Are you sure you want to change the base?

pipelined extraction #236

Conversation

cosmicexplorer commented Aug 17, 2024 • edited Loading

Problem

Solution

Result

CURRENT BENCHMARK STATUS

cosmicexplorer commented Jan 16, 2025

Pr0methean left a comment

Choose a reason for hiding this comment

Pr0methean Jan 17, 2025

Choose a reason for hiding this comment

cosmicexplorer Feb 12, 2025

Choose a reason for hiding this comment

Pr0methean Mar 17, 2025 • edited Loading

Choose a reason for hiding this comment

Pr0methean Jan 18, 2025

Choose a reason for hiding this comment

cosmicexplorer Feb 13, 2025

Choose a reason for hiding this comment

Pr0methean Mar 17, 2025 • edited Loading

Choose a reason for hiding this comment

cosmicexplorer commented Feb 5, 2025 • edited Loading

cosmicexplorer commented Feb 12, 2025

cosmicexplorer commented Feb 13, 2025

Pr0methean Feb 25, 2025

Choose a reason for hiding this comment

Pr0methean left a comment

Choose a reason for hiding this comment

Pr0methean Feb 25, 2025

Choose a reason for hiding this comment

cosmicexplorer commented Aug 17, 2024 •

edited

Loading

Pr0methean Mar 17, 2025 •

edited

Loading

Pr0methean Mar 17, 2025 •

edited

Loading

cosmicexplorer commented Feb 5, 2025 •

edited

Loading