[QUESTION] How to stream zip without first loading all of it in memory

[ZefQ]

First of, thank you for this lib. I found this while browsing a zip64 feature request for the Conflux project.

Anyway I have a large number of ~100 Mb files lets say a total of 40 Gb that i would like to stream to the client as a zip, but i can´t figure out how to achieve that. i have read some of your closed issues to try to get an idea but i cant seem to get it to work.
This is kind of what i got but i tried a couple of different inputs to the generate function your DownloadStream for example.

const downloadAllClientZip = async () => {
  const files = await getFiles(); // load some file objects from a rest api

  const blob = await downloadZip(generate(files)).blob();

  const link = document.createElement("a");
  link.href = URL.createObjectURL(blob);
  link.download = `${params[0]}.zip`;
  link.click();
  URL.revokeObjectURL(link.href);
  link.remove();
};

async function* generate(files) {
  for (const { href, folder, title } of files)
    yield {
      input: await fetch(href),
      name: `${folder}/${title}`,
    };
}

🙇

[Touffy]

You won't manage your 40GB download with the Blob approach (but the input generator is a good start). You absolutely have to stream the response (through a Service Worker, because that's the only way).

There's a demo showing one way to do that. You can try it here and the source is also included in the repo. In that demo, client-zip is used in the Service Worker, not the client window.

The window shows a button inside a form with method=POST and action pointing to an arbitrary same-origin URL. Unlike most forms these days, it does not prevent the default action because we want the browser to (try to) navigate to that URL, and we're using a form because it's a good way to add a request payload (as opposed to a simple download link).

The Service Worker intercepts the request and "respondsWith" the response generated by downloadZip, which always includes a Content-Disposition header that tells the browser to download instead of navigating (so in the end you remain on the current page even though you didn't prevent the submit event's default). The request payload allows the client to tell the Service Worker which files to include in the zip, though of course there are many other ways to communicate (but I've found this is the easiest).

[ZefQ]

Awesome, i will try it out tomorrow.

So you need to do this via a service worker? I got the stream working with Conflux but the file was to large for that library so in the end when i tried to open it i just got a broken 40 Gb zip file.

[Touffy]

Yeah, as you saw, Conflux hasn't implemented Zip64 so it's limited to 4GB.

It's possible to stream client-zip's output the same way you did with Conflux. I guess you used streamSaver as shown in their example. streamSaver also goes through a Service Worker and can handle any stream, including that produced by makeZip (the secondary function exported by client-zip).

The way I show in the demo is "simpler" in terms of architecture — the stream doesn't need to transfer to the Service Worker before coming back to the client — but takes a little more code because it's not using streamSaver to create the Service Worker for you.

[j0sh]

The browser filesystem APIs should work here, assuming you want to stream straight to disk.

However I haven't been able to actually get that working with client-zip. An example would be very much appreciated; just doing something like writableStream.write(downloadZip(files)) has not worked for me.

[Touffy]

Looks like a simple mistake : the Filesystem API consumes Streams (or discrete chunks of data, when you call write on the stream). downloadZip returns a Response, which is neither.

Of course there's a Stream inside the Response, or you can call makeZip, which is also exported by client-zip, to get the Stream directly. Then just pipe it to the filesystem :

const saver = await window.showSaveFilePicker({
  suggestedName: "bundle.zip",
  types: [{ description: "ZIP file", accept: { "application/zip": [".zip"] }}]
})
const writableStream = await saver.createWritable()
await makeZip(files).pipeTo(writableStream)

[j0sh]

Amazing, that worked perfectly - thank you.