NodeJS 22 issues/feedbacks tracker

[robertsLando]

NodeJS 22 support has been added starting from pkg 5.14.0 but many things have changed on NodeJS source and so also on our patch file so it needs some tests. Use this issue to submit your feedbacks and issues

KNOWN ISSUES:

• MacOS ARM64 not available: feat: add Node 22 patch (#45) (by @faulpeltz) pkg-fetch#45. Reason is that ATM no github runner is able to produce a valid binary without going out of RAM or exceeding 6h max build time. Fixed in 5.14.1 - Ref fix: node 22 module filename resolution in win32 (#49) by @faulpeltz pkg-fetch#49
• Module Loading not working with Windows Error: Cannot find module - Ref fix: restore mac arm64 build pkg-fetch#46
• Cross OS compilation of node 22 has issues with Windows: NodeJS 22 issues/feedbacks tracker #87 (comment). The build succeeds but when running the binary it exits without any output.
  • Actual workaround: use --public flag (to include sources)

cc @faulpeltz

[faulpeltz]

I managed to complete a Node 22 build with macos arm64 after removing the android SDK (the change from the node/build repo), and then also removed the iOS simulator cache (which is nearly 50GB)

yao-pkg/pkg-fetch@main...faulpeltz:pkg-fetch:main#diff-a1654f4357833dfc4d0c6a8d39c2e3ee30ceea604c29238846903993434ab384R93

[robertsLando]

@faulpeltz That's super! PR? 🚀

[robertsLando]

I would extend that also on x64 build to prevent any future issue with this :)

[faulpeltz]

@faulpeltz That's super! PR? 🚀

I'm trying to find out why the x64 macos build on -13 is so fragile, I had it crash or timeout multiple times. But the cleanup seems not necessary for now because it has ~200GB free disk space, and I assume it will imrpove in the future 🤷‍♂️

I really don't want to complain about free Apple build infrastructure, but 14GB of free disk space is a bit of a stretch...

[robertsLando]

I really don't want to complain about free Apple build infrastructure, but 14GB of free disk space is a bit of a stretch...

Yeah definetely 😆 Also because storage nowdays doesn't cost that much, RAM and CPU does...

[cage1618]

Can I use this release in production now?

[robertsLando]

@cage1618 If you are speaking about nodejs 22 I would say nope, but you can use nodejs 18/20 safely with latest release

[faulpeltz]

@robertsLando The Node 22 patch applies to the just-released 22.9 version without problems and all builds went through

In this release they disabled Maglev (one of the JIT compilers) because of reported crashes
https://github.com/nodejs/node/releases/tag/v22.9.0

So random crashes might be caused by latent Node 22 issues before 22.9, might make sense to bump the Node 22 version (but not urgent)

[robertsLando]

@faulpeltz thanks for the info, I have an action to automatically create the patch, let me run it 🚀

[robertsLando]

I have created a workflow that automatically check for new nodejs versions: https://github.com/yao-pkg/pkg-fetch/actions/workflows/check-latest-node.yml

[cage1618]

release 5.15.0 is stable now?

[balisaurus]

Using 5.15.0 to use latest nodejs 22 doesn't seem to work for my project.
I create a simple test code with a single file writing hello to console:

pkg --debug hello.js --targets node22-win-x64
> [email protected]
(node:27328) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
> [debug] Bytecode of D:\temp\testnode\hello.js is added to queue.
> [debug] Stat info of D:\temp\testnode\hello.js is added to queue.
> [debug] Directory D:\temp\testnode is added to queue.
> [debug] Stat info of D:\temp\testnode is added to queue.
> [debug] Directory D:\temp is added to queue.
> [debug] Stat info of D:\temp is added to queue.
> [debug] Directory D:\ is added to queue.
> [debug] Stat info of D:\ is added to queue.
> [debug] Deleting record file : D:\
> [debug] The file was included as bytecode (no sources)
  D:\temp\testnode\hello.js
> [debug] The file was included as bytecode (no sources)
  D:\temp\testnode\hello.js
> [debug] files & folders deduped =
  hello.js
> [debug] The directory files list was included (1 item)
  D:\temp\testnode
> [debug] The directory files list was included (1 item)
  D:\temp\testnode
> [debug] files & folders deduped =
  testnode
> [debug] The directory files list was included (1 item)
  D:\temp
> [debug] The directory files list was included (1 item)
  D:\temp
> [debug] Targets:
  [
  {
    "nodeRange": "node22",
    "platform": "win",
    "arch": "x64",
    "output": "d:\\temp\\testnode\\hello.exe",
    "forceBuild": false,
    "fabricator": {
      "nodeRange": "node22",
      "platform": "win",
      "arch": "x64",
      "binaryPath": "C:\\Users\\tekno\\.pkg-cache\\v3.5\\fetched-v22.9.0-win-x64"
    },
    "binaryPath": "C:\\Users\\tekno\\.pkg-cache\\v3.5\\fetched-v22.9.0-win-x64"
  }
]

Running it:

hello.exe
------------------------------- virtual file system
C:\snapshot
  testnode                                29 Bytes
    hello.js                              29 Bytes
Total size =  29 Bytes
node:internal/modules/cjs/loader:1254
  throw err;
  ^

Error: Cannot find module 'C:\snapshot\testnode\hello.js'
    at Module._resolveFilename (node:internal/modules/cjs/loader:1251:15)
    at Function._resolveFilename (pkg/prelude/bootstrap.js:1955:46)
    at Module._load (node:internal/modules/cjs/loader:1077:27)
    at Function.runMain (pkg/prelude/bootstrap.js:1983:12)
    at node:internal/main/run_main_module:30:49 {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}

Node.js v22.9.0

[robertsLando]

@balisaurus does it work with others nodejs version?

[BeneRasche]

Same problem here:

node:internal/modules/cjs/loader:1254
throw err;
^

Error: Cannot find module 'C:\snapshot\appname\dist\apps\server\main.js'
at Module._resolveFilename (node:internal/modules/cjs/loader:1251:15)
at Function._resolveFilename (pkg/prelude/bootstrap.js:1955:46)
at Module._load (node:internal/modules/cjs/loader:1077:27)
at Function.runMain (pkg/prelude/bootstrap.js:1983:12)
at node:internal/main/run_main_module:30:49 {
code: 'MODULE_NOT_FOUND',
requireStack: []
}

Node.js v22.9.0

---

works with -t node20-win-x64 but not node22

[faulpeltz]

Seems that the snapshot fs doesn't work for module loading on Windows in the Node 22 patch

[robertsLando]

@BeneRasche The window issue related to nodejs 22 should be fixed on next release, please be patient. Thanks to @faulpeltz for the patch!

[thisjt]

We're waiting for this, right?

@faulpeltz Unfortunately we need to wait a new nodejs 22 release in order to test this as I have no way with current workflows to force build a specific version, I need to implement this

Based on how often nodejs releases, I think it will be either next week or mid October

[robertsLando]

@thisjt yep, I'm sorry!

Problem is that actually I cannot release patches for nodejs patches as them would become broke for actual pkg versions as the shas are shipped with npm so updating them would change them and work only for latest vesrsions. I already have a workflow that runs every day at midnight and opens a PR when a new patch is available

[thisjt]

@robertsLando nodejs had a release 4 days ago. I think it's time to patch 😬 let's gooo- or does it need to have a v22 release?
https://github.com/nodejs/node/releases/tag/v20.18.0

---

Oh lol nvm my bad it needs a v22 release

[robertsLando]

@thisjt thanks for letting me know! Seems my automated workflows picked up it correctly 4 days ago but the patches do not apply cleanly, let me try to fix this

https://github.com/yao-pkg/pkg-fetch/actions/runs/11171900799/job/31057489528

BTW seems only node 20 has a new release for now

[thisjt]

node v22.10 is finally out! https://github.com/nodejs/node/releases/tag/v22.10.0

And with that node v23 is also out, hoping that we will also be supporting this. Not in a hurry tho, we're not sure yet on how stable v23 is.

[dingiyan]

I often follow this library and hope to keep up with the latest version of Node.js. This morning, I tested the v22 version, but it is not yet available. I hope excellent authors can continue to follow up and look forward to the latest available v22 version. Finally, I would like to thank the contributors of yao-pkg for their hard work

[robertsLando]

New version is coming soon. The average delay I would like to keep between nodejs releases and pkg support for them is around 1 week, it depends on how many breaks there are on patches.

[robertsLando]

pkg 5.16.0 is out now with node 20.18.0 and 22.10.0 support

[BeneRasche]

pkg-fetch probably needs an update for 22.10 binaries, too?

[robertsLando]

@BeneRasche It's out already!

[BeneRasche]

@BeneRasche It's out already!

Where?

https://github.com/yao-pkg/pkg-fetch/releases

[robertsLando]

Oh you were speaking about patches so, anyway seems not an issue as we dropped that many patches ago and this only happens on node 22 :(

[faulpeltz]

In ContextifyScript::New() it does set the V8 flags "lazy" to false and "predictable" to true (this is code added by pkg),
the fabricator just uses vm.Script with "produceCachedData: true".

If there is another mechanism I don't know where it can be found - but maybe this is either just a result of a breaking change in Node 22 (or the new V8 version it uses) or maybe some new flags need to be set

When looking at the produced code cache blob on Linux and Windows, the generated code is exactly the same for a small but non-working example, but the hashes for the read-only snapshots are different - if someone has an idea where these are coming from or where they are generated during build, maybe I can find out whats different

[robertsLando]

Back from holidays now. Just a quick summary here.

Seems the issue is related to Node 22 that added a check on the snapshot hash that fails when cross-building.

disabling the hash check will result in a hard startup segfault/crash

@faulpeltz have you been able to detect why this happens? I think that's the only available workaround we have here

[faulpeltz]

Not yet, I think comparing the snapshot blob data generated on different platforms might be worth investigating - the generated bytecode itself looks identical, so the crash must have something to do with the layout of the startup/v8 readonly snapshot

This might be by design in V8 however - I wonder now this is handled in Node SEA for different platforms, or maybe it just ignores the code snapshot altogether and just starts from the source script again, because that fallback works in pkg too.

[robertsLando]

@faulpeltz As I wrote above from what I know SEA blob is not using bytecode so it's not the same

[mbidave]

@robertsLando @faulpeltz Is there any workaround for this cross-platform build issue?

[robertsLando]

just disable bytecode generation

[tariksogukpinarperwatch]

Is anyone experiencing performance problems with Node 22? Is it right to use 600-700 MB RAM in an average NestJS app?

[robertsLando]

@tariksogukpinarperwatch could you please try to pakcage your application with nodejs 20/18 and see if the memory usage is the same?

[tariksogukpinarperwatch]

When I tried it with Node 20, the memory usage is the same, around 600MB on average. The application is running in cluster mode. Can you give any suggestions, thank you

[robertsLando]

what is the memory usage of the same application not packaged? Anyway would be helpful to understand if this has been a regression or not as this doesn't seems to be an issue related to nodejs 22. I suggest to open a separated issue

[dingiyan]

Hello, I've found an issue. On my Windows computer, I can compile Node.js 22 (the latest version 22.13.1) without any problems, generate Windows and Linux binaries, and run them successfully. However, when I build the binaries on WSL, the Windows program fails to run. It terminates immediately upon execution, with no output whatsoever. On the other hand, the Linux program can run properly.
I've also tested Node.js version 20, and it works fine.
In addition, I've tested building a simple JavaScript file, and that also works as expected.
Therefore, I suspect that some features of version 22 interact with certain special dependencies, triggering this problem.

[robertsLando]

Hi @dingiyan, could you try to add --report-uncaught-exception --report-on-signal --report-on-fatalerror options to see if you see some more informations ?

[dingiyan]

Hi @dingiyan, could you try to add --report-uncaught-exception --report-on-signal --report-on-fatalerror options to see if you see some more informations ?

I try it, bake the options to --options , and got same result .
eg: await exec(["pkg-start.js", "--options","--report-uncaught-exception --report-on-signal --report-on-fatalerror", "--config", "./package.json", "--compress", "GZip", "--output", ./pkg/${exeName}]);

I think it may be that some low-level code behaves inconsistently across platforms
It's difficult to find the cause of this problem

[robertsLando]

@dingiyan I'm sorry for that but without any information about the error it's hard to identify the issue.

@faulpeltz any clue about this?

[faulpeltz]

@faulpeltz any clue about this?

Isn't that the known existing issue with cross platform? Building on WSL is using the Linux binary and then running that on Windows

[robertsLando]

@faulpeltz Yeah you are right, I thought the actual issue was causing some output but it was another. In order to keep track of this issue I have added it to the list here

@dingiyan only available workaround for now is to use --public flag while compiling

[dingiyan]

@robertsLando I use the --public flag, and windows app can run, but it's only output the entry file log, my main js couldn't exec, and the app is terminate.
In my entry file, require the main js code like this:require(__dirname + "/node_modules/egg-scripts/bin/egg-scripts.js");
And, My js code already set to pkg.scripts or pkg.assets config.
It's look like the --public flag didn't affect the dynamic import code?

[faulpeltz]

I seems cross compilation without --public is currently completely broken in Node 22 between most platform/arch combinations due to the differences in the node readonly snapshots

[robertsLando]

Yeah I saw also the report on #137. I'm a bit lost here... We would need some support maybe from someone working on nodejs core...

[faulpeltz]

Just to summarize the issue:

• cross platform builds fail to run on Node >=22
• the generated byte code is identical as far I can tell (it is the same for some basic test apps where the issue can be reproduced)
• if code is included with --public the runtime falls back on the script source instead of the bytecode, causing no issue
• the patches included with pkg allow Node to run bytecode without any script source
• any scenarios where the script source is needed by the runtime, execution fails (or is a no-op because of the empty script)

Regarding the bytecode/snapshots:

• the bytecode blobs contain various checksums to ensure a sane runtime state e.g. v8 flags/snapshots
• the "readonly" snapshot checksum is no longer the same on different platforms/arches in Node >=22
• when building on one platform but running the bytecode on another platform, the snapshot checksum check fails, causing the runtime to discard the bytecode and fall back to the script source
• when removing this check, the runtime segfaults - the bytecode references something in the readonly snapshot which is not there or in the right place
• there might be a way to compile node to create the same bit-for-bit readonly snapshot on most platforms/arches again, but it might be impossible because of the way V8 works on different platforms

[robertsLando]

I tried to reach out @RaisinTen (the dev who implemented node sea feature on Node.js core) via PM and maybe he will take a look at this 🙏🏼

[xcjs]

Not that it was brought up as an alternative, but I would love to use the SEA feature, but it still fails on certain NPM packages. I don't recall the exact issue I had, but I just couldn't get it to work.

This project has been my only alternative so far, and I'd really love it if we could figure it out. Unfortunately, the issue is also a bit beyond my own skill set as well. 😅

[robertsLando]

but I would love to use the SEA feature, but it still fails on certain NPM packages

The only way to use sea is to use a bundler like esbuild to create a single file of your application. Of course this will not work for npm modules that have native .node addons. Also there are other limitations related to assets management that should be done by user by checking if the asset is present on sea exe or not using sea functions like sea.getAsset (https://nodejs.org/api/single-executable-applications.html#seagetassetkey-encoding)

[xcjs]

but I would love to use the SEA feature, but it still fails on certain NPM packages

The only way to use sea is to use a bundler like esbuild to create a single file of your application. Of course this will not work for npm modules that have native .node addons. Also there are other limitations related to assets management that should be done by user by checking if the asset is present on sea exe or not using sea functions like sea.getAsset (https://nodejs.org/api/single-executable-applications.html#seagetassetkey-encoding)

Correct. My current project uses esbuild and none of the packages I'm using includes a native Node module. SEA still fails for me. I'll have to try again soon to get the error message again - I think it has to do with a require statement, but also this isn't the issue to explore my issue in. 😅

[RaisinTen]

Hi all, some questions regarding the sourceless builds bug:

Is Node.js v22 the first one where this error started happening? Does it repro on earlier versions like v21 or older minor / patch versions in the v22 line? The first approach that comes to my mind is to bisect and find out the failing Node.js or V8 commit.

the generated byte code is identical as far I can tell (it is the same for some basic test apps where the issue can be reproduced)

@faulpeltz Could you clarify this part? The sourceless code cache generated on platform X runs on X but fails on platform Y but it is identical to the sourceless code cache generated on platform Y which runs on platform Y but fails on X? If that's true, it would mean that the snapshot checksum embedded in the code cache that works on each platforms are identical but is still different from the expected snapshot checksum. In that case, I'd be interested in looking into what the expected snapshot checksum is.

Re.

for Node 22 when loading on e.g. Windows the hash check for kReadOnlySnapshot will fail because the snapshot hash is different on linux/windows/Mac

and

when building on one platform but running the bytecode on another platform, the snapshot checksum check fails, causing the runtime to discard the bytecode and fall back to the script source

IIUC the issue might be fixed if the snapshot checksum embedded in the generated code cache matches another snapshot checksum. @faulpeltz might be interesting to trigger a crash from the snapshot checksum check maybe by adding a CHECK(false), which would give us the stack trace which might help in identifying which 2 snapshots are being compared and where these are generated.

[robertsLando]

Is Node.js v22 the first one where this error started happening? Does it repro on earlier versions like v21 or older minor / patch versions in the v22 line? The first approach that comes to my mind is to bisect and find out the failing Node.js or V8 commit.

AFAIK yes, this is only happening with Node.js 22

[faulpeltz]

Is Node.js v22 the first one where this error started happening? Does it repro on earlier versions like v21 or older minor / patch versions in the v22 line? The first approach that comes to my mind is to bisect and find out the failing Node.js or V8 commit.

I didnt try any versions older than v22.8 (yet), bisecting might be quite a lot of work because the patches need to be tweaked to apply cleanly

Could you clarify this part? The sourceless code cache generated on platform X runs on X but fails on platform Y but it is >identical to the sourceless code cache generated on platform Y which runs on platform Y but fails on X? If that's true, it >would mean that the snapshot checksum embedded in the code cache that works on each platforms are identical but is >still different from the expected snapshot checksum. In that case, I'd be interested in looking into what the expected >snapshot checksum is.

So the code cache generated on platform X runs on X but fails on Y (and generated on Y runs on Y and fails on X), and is nearly identical except the header with the kReadOnlySnapshotChecksum (at offset 0x10) and another few bytes which look like checksums in the middle of the blob

Code cache generated on linux x64 for win x64
fromlinux-win.txt
Code cache generated with same node version (22.12 I believe) on win x64 for win x64
fromwin-win.txt

So the expected checksum is different on each platform.

The check which fails is in SerializedCodeData::SanityCheckWithoutSource here:
https://github.com/nodejs/node/blob/5d2feb257bcee090e57900eb51720171a6aa92f3/deps/v8/src/snapshot/code-serializer.cc#L730