Feature request: zstd archive support

[warfery]

I would like Rufus to support .zst archives. I receive .zst files created on Linux, which contain .img files in sparse format. Unpacking these .zst files on Windows using 7-Zip or zstd loses the advantage of the sparse format, requiring the full size of the .img file. The original .img file is over 100GB, while the .zst file and the sparse .img file are around 10GB.

It would be great if Rufus could directly unpack .img files from .zst archives and deploy the image to a USB disk on the fly.

zstd project: https://github.com/facebook/zstd

Thank you
Frank

[pbatard]

Dupe of #2620 and #2590.

Same answer. Same question: Do you have link to public images that use zstd compression?

[pbatard]

Also, you literally gave a thumbs up on the previous issue that asks for exactly the same thing. Please don't create pointless duplicates.

[warfery]

Yes sorry, I noticed the same request when I have send the request.

For example here: https://static.redox-os.org/img/x86_64/

[warfery]

And maybe important info, .zst archives can be extracted with 7zip.

[pbatard]

You may want to push for busybox to consider the 2021 zstd proposal for integration by posting on their mailing list then, and pointing that busybox is likely to have to interract with ztsd compressed archives soon then.

I'll see what I can do for Bled, but please don't expect anything, or anything fast, as I would much rather see zstd support integrated in busybox first, and it's unlikely to happen if people who want zstd support only ask for it in downstream projects.

[warfery]

I will do it. Thank you Pete for your time and for your great work.

[pbatard]

Okay, since I have just added zstd compression support in Bled, based on the original 2021 patch (which still required some time consuming modifications) I will be reopening this issue so that I can close it in the commit that adds zstd in Rufus.

[pbatard]

Note that if you want to test this, you can find Rufus executable with zstd support in the artifacts archive from https://github.com/pbatard/rufus/actions/runs/11981708130.

[warfery]

Wow, thank you Pete! It is 2nd time you have quickly solved my problem. I know you are rejecting donations but what about coffee on buymeacoffee.com?

[pbatard]

I appreciate the proposal, and I'm happy to hear that I (but, mostly, the code that I lifted from other people) could be useful to you, but I'm afraid that I don't drink coffee, and my answer to any kind of donation is still the same as the one you find at https://rufus.ie/en/#donate, i.e. I'm currently lucky enough to be living comfortably, so I'd rather see any form of donation go to people who need it a lot more than I do.

Also, you may want to make sure that I actually did solve your problem, because I only ran some limited testing writing a single zstd compressed image, which looked okay, but it's still very possible the feature is not working as well as you hope...

[warfery]

Ok, I will test it next week and let you know. I expected that you reject my offer so I will pay it forward. (Pay it forward, 2000 movie). Thank you once again.

[warfery]

I tested the prototype and it works!
I was using dd sparse image (.img) packed with zstd. Sparse size of the file is around 8BG, unsparsed 120GB. Windows can't work with sparse files and when you unpack linux sparse file on Windows, the file always get the full size (but the file is correct).
There is cosmetic issue with progress bar. It showed 100% after 3-4 minutes and then it took additional 10 minutes to complete the task. But I think this is related to the sparse image because Windows can't detect the correct unsparsed size of the image inside the archive (even total commander shows ??? size of the img file inside the zst archive).
Thank you (also from my disks)!

[pbatard]

Thanks for the report.

There is cosmetic issue with progress bar. It showed 100% after 3-4 minutes and then it took additional 10 minutes to complete the task.

That's the expected behaviour, because a lot of compression formats just don't provide easy access to the size of the decompressed data (and the BusyBox library we use doesn't really provide it), so Rufus doesn't have it. Which means that we use the size of the compressed data against how much data we have read to display the progress.

Thus, if you have data that is mostly 20 GB of actual payload and 100 GB of zeroed data, the zeroed data will compress in a few bytes, but will still take a very long time to write, so you will pretty much stay at 99.99% progress for the time it takes to write 100 GB, which could indeed take minutes.

At this stage, I have no plan to address this, especially as, again, some compression formats simply do not store the expected decompressed size of the data, so you really have no way to know how many bytes you should have written until you have actually uncompressed them all.