Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

zlib: add zstd support #52100

Closed
wants to merge 2 commits into from
Closed

zlib: add zstd support #52100

wants to merge 2 commits into from

Conversation

jkrems
Copy link
Contributor

@jkrems jkrems commented Mar 15, 2024

Adds ZstdCompress and ZsdDecompress to the zlib module which can be used to compress/decompress with the Zstandard ("zstd") algorithm.

Notable omissions:

  • Providing dictionaries isn't implemented.
  • The docs in zlib.md don't call out any params beyond the basic compression level.

The code follows similar patterns to the PR that added Brotli support. Just that instead of brotli, it adds the equivalent zstd APIs. Just like Brotli, this required separate compression/decompression context objects.

Zstd itself has been around and stable for multiple years but this PR is early in terms of web support: It only just starts shipping by default in Chrome 123. On the other hand, by shipping in Chrome it will soon be supported quite widely on the web. Firefox also signaled support (https://bugzilla.mozilla.org/show_bug.cgi?id=1301878#c65). Zstd is now enabled on the web and supported by both Chrome and Firefox: https://caniuse.com/zstd

Official support in node.js would allow passing additional WPTs around fetch (nodejs/undici#2847).

@nodejs-github-bot
Copy link
Collaborator

Review requested:

  • @nodejs/security-wg

@nodejs-github-bot nodejs-github-bot added dependencies Pull requests that update a dependency file. needs-ci PRs that need a full CI run. labels Mar 15, 2024
@jkrems jkrems force-pushed the zstd branch 4 times, most recently from ec19b6e to 28914c7 Compare March 17, 2024 00:07
@jkrems
Copy link
Contributor Author

jkrems commented Mar 17, 2024

Alright, got up to making the basics (seemingly) work:

$ out/Debug/node -p 'zlib.zstdDecompressSync(zlib.zstdCompressSync("Hello World")).toString()'
Hello World

@jkrems jkrems force-pushed the zstd branch 6 times, most recently from 383ff46 to 47c7ab8 Compare March 17, 2024 20:59
@jkrems jkrems marked this pull request as ready for review March 17, 2024 21:00
jkrems added a commit to jkrems/node that referenced this pull request Mar 17, 2024
jkrems added a commit to jkrems/node that referenced this pull request Mar 17, 2024
jkrems added a commit to jkrems/node that referenced this pull request Mar 17, 2024
jkrems added a commit to jkrems/node that referenced this pull request Mar 17, 2024
jkrems added a commit to jkrems/node that referenced this pull request Mar 17, 2024
jkrems added a commit to jkrems/node that referenced this pull request Mar 17, 2024
jkrems added a commit to jkrems/node that referenced this pull request Mar 17, 2024
jkrems added a commit to jkrems/node that referenced this pull request Mar 17, 2024
@jasnell
Copy link
Member

jasnell commented Nov 5, 2024

@jkrems ... can you give this a rebase?

@jkrems
Copy link
Contributor Author

jkrems commented Nov 5, 2024

@jasnell I can but at this point I have spent quite bit of time on this PR without a clear signal on whether this would be likely to land (in this form or another). So it would be easier to justify if there was a tentative "yep, the high level approach looks good". By high level I mean something like "roughly following brotli integration in terms of API, using the facebook zstandard native implementation as the backing".

@bricss
Copy link

bricss commented Feb 2, 2025

That's will be great to have this in a main branch 🌿after all 🫠

@KhafraDev
Copy link
Member

@nodejs/tsc There aren't any outstanding objections, but this PR does appear to be at an impasse. @jkrems did not receive an answer to their question #52100 (comment). I have held off for months from doing this, hoping someone could review and collaborate since I couldn't.

@KhafraDev KhafraDev added the tsc-agenda Issues and PRs to discuss during the meetings of the TSC. label Feb 7, 2025
@marco-ippolito
Copy link
Member

marco-ippolito commented Feb 7, 2025

Nobody reviewed and approved the PR, there is nothing officially blocking it. If someone wants to volunteer to review it and approve it without objections, then its gonna land.
I don't think there is anything the TSC can do about it, or to be discussed TSC side

@KhafraDev
Copy link
Member

Maybe not, but I don't think I'm in a position to give guidance or give definitive instructions on the approach. I have pinged the zlib team, I pinged the security team, I have asked people to review it in private, so I didn't know what else to do lol.

@anonrig
Copy link
Member

anonrig commented Feb 7, 2025

I'm interested in landing this PR as well. If we can have one last push (and rebase this PR), I'm more than happy to review and land this PR.

@jkrems
Copy link
Contributor Author

jkrems commented Feb 7, 2025

@anonrig Thanks! I'll work on a rebase.

jkrems added a commit to jkrems/node that referenced this pull request Feb 7, 2025
@anonrig anonrig added request-ci Add this label to start a Jenkins CI on a PR. commit-queue-rebase Add this label to allow the Commit Queue to land a PR in several commits. and removed tsc-agenda Issues and PRs to discuss during the meetings of the TSC. labels Feb 8, 2025
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Feb 8, 2025
@nodejs-github-bot
Copy link
Collaborator

@nodejs-github-bot
Copy link
Collaborator

@anonrig anonrig added the commit-queue Add this label to land a pull request using GitHub Actions. label Feb 8, 2025
@nodejs-github-bot nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Feb 8, 2025
@nodejs-github-bot
Copy link
Collaborator

Landed in 0782c91...bf12d72

nodejs-github-bot pushed a commit that referenced this pull request Feb 8, 2025
PR-URL: #52100
Reviewed-By: Yagiz Nizipli <[email protected]>
nodejs-github-bot pushed a commit that referenced this pull request Feb 8, 2025
Fixes: #48412
PR-URL: #52100
Reviewed-By: Yagiz Nizipli <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c++ Issues and PRs that require attention from people who are familiar with C++. commit-queue-rebase Add this label to allow the Commit Queue to land a PR in several commits. dependencies Pull requests that update a dependency file. needs-ci PRs that need a full CI run. notable-change PRs with changes that should be highlighted in changelogs. review wanted PRs that need reviews. semver-minor PRs that contain new features and should be released in the next minor version. zlib Issues and PRs related to the zlib subsystem.
Projects
None yet
Development

Successfully merging this pull request may close these issues.