Consider zstd for compression of shipped artifacts #1660
Comments
|
Looking at our pipelines the compression step takes around 30m: So we could possibly save 6-8m per run just on that. This could then compound because each of our CI runs may or may not run |
|
Another thing to mention here is that in my tests I used We could experiment with different levels to see what the differences are in size versus speed, but I assumed we wanted to increase the size as little as possible so I used |
|
Huh, I was expecting more drastic differences in compression/decompression times. IMO spending an extra 6-8 minutes for 8% smaller images is worth it. |
I implemented this to investigate it as an option for coreos/fedora-coreos-tracker#1660 so figured I may as well post the code up for inclusion.
I implemented this to investigate it as an option for coreos/fedora-coreos-tracker#1660 so figured I may as well post the code up for inclusion.
|
Some more data:
|
|
If we went with something like level 10 we'd get a 90% speedup in compression which I think would take our |
Based on the discussion in coreos/fedora-coreos-tracker#1660 level 10 seems to give us a good speedup versus size tradeoff.
We'll experiment with this in `rawhide` to see what kind of real world gains we get from using zstd compression. Also see if there are any bugs that crop up. This is to further the discussion in coreos/fedora-coreos-tracker#1660
|
This was discussed in today's community meeting: |
|
I did some additionnal testing on the live qemu file
Another note: zstd was not installed in my f39 toolbox by default |
|
As a corollary, I also did some testing yesterday with the qemu image. The datasize compressed and uncompressed (cols 4 &5) were equivalent. No surprise there. Where the results differed for me was the decompression time. Mine was consistently double that. Were you passing any additional command-line switches? |
Simply running |
|
Given that the image files tested are very large, |
|
So... with the recent xz news, a lot of trust was lost in that project. Apart from the other benefits listed in this ticket, switching to zstd would now also avoid forcing people to use xz to use our artifacts if they're not comfortable with that. |
|
we have been using zstd with fcos images in podman machine now for a couple of months. lots of upside comments about the quicker decompression |
Based on the discussion in coreos/fedora-coreos-tracker#1660 level 10 seems to give us a good speedup versus size tradeoff.
|
Testing decompression speeds locally on a local 1.6G qcow2, I get 17.8s for xz and 0.99s for zstd. Weird that the decompression difference in #1660 (comment) between xz and zstd isn't much larger. As mentioned I think in the last community meeting where we discussed this, it's possibly hardware-related. |
I did some investigation into zstd as our default compression algorithm. I set the compression level of zstd to
19andxzto9(what we use today). Here is what I see for times on compress and decompress using xz of themetalandqemuartifacts:and here is what I see for zstd:
and here is what the difference in sizes look like:
To summarize:
So we get about a 20% speedup in compression and 30% speedup in decompression with the tradeoff of 8-9% larger compressed files.
The text was updated successfully, but these errors were encountered: