Yarn, PNPM, others (was: Cannot find module './build/Release/re2')

[feliperobledo]

Hi! So I noticed you have some unusual errors with yarn, but I just encountered this same problem with the pnpm package manager. To me it seems that the package manager is not running the install script for some reason. I've pinged their discord channel to see if they can help on their end.

In my case I'm trying to install get-urls, which depends on url-refex-safe, which ultimately depends on node-re2.

[uhop]

I never used pnpm but heard about it. I'll try it to see what the problem is.

Is it 100% reproducible or happens randomly? Did you try it in some other configurations, e.g., making node-re2 a direct dependency?

[uhop]

Including it directly WFM:

$ npm init -y
Wrote to /home/xxx/temp/package.json:

{
  "name": "temp",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC"
}


$ pnpm add re2
Packages: +101
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Packages are hard linked from the content-addressable store to the virtual store.
  Content-addressable store is at: /home/xxx/.pnpm-store/v3
  Virtual store is at:             node_modules/.pnpm
Progress: resolved 101, reused 0, downloaded 101, added 101, done
node_modules/.pnpm/[email protected]/node_modules/re2: Running install script, done in 985ms

dependencies:
+ re2 1.15.9
$ node
Welcome to Node.js v15.8.0.
Type ".help" for more information.
> const RE2 = require('re2');
undefined
> RE2
[Function: RE2] {
  getUtf8Length: [Function (anonymous)],
  getUtf16Length: [Function (anonymous)],
  unicodeWarningLevel: 'nothing'
}
> 

[uhop]

Including it indirectly (adding url-regex-safe) doesn't WFM:

$ npm init -y
Wrote to /home/xxx/temp/package.json:

{
  "name": "temp",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC"
}


$ pnpm add url-regex-safe
Packages: +104
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Packages are hard linked from the content-addressable store to the virtual store.
  Content-addressable store is at: /home/xxx/.pnpm-store/v3
  Virtual store is at:             node_modules/.pnpm
Progress: resolved 104, reused 101, downloaded 3, added 104, done
node_modules/.pnpm/[email protected]/node_modules/re2: Running install script, done in 939ms

dependencies:
+ url-regex-safe 1.0.2
$ node
Welcome to Node.js v15.8.0.
Type ".help" for more information.
> const RE2 = require('re2');
Uncaught Error: Cannot find module 're2'
Require stack:
- <repl>
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:925:15)
    at Function.Module._load (node:internal/modules/cjs/loader:769:27)
    at Module.require (node:internal/modules/cjs/loader:997:19)
    at require (node:internal/modules/cjs/helpers:92:18) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [ '<repl>' ]
}
> 

PS: I do see this line:

node_modules/.pnpm/[email protected]/node_modules/re2: Running install script, done in 939ms

But I don't see any link for it.

[uhop]

The versions:

$ pnpm --version
5.17.1
$ node --version
v15.8.0

[uhop]

I inspected the created node_modules and it is not supposed to work as is. Given my 0-level experience in pnpm I am not sure node will work with it as is. For example, the latest yarn uses a special module resolution and requires a special wrapper to run node: yarn node. Do I have to use anything special to run node?

[mschipperheyn]

AFAIK it's npm rebuild and not npm run rebuild

[uhop]

That sounds like my bug. I'll do a fix and test it, thx.

[jasonswearingen]

fyi I am also having the same error with yarn version 1.22.10

using node 14.x

[jasonswearingen]

fyi I came across this error when upgrading from node v10.x to node v14.x

downgrading back to node v10.x works. I'll stay there till this issue is resolved.

[SebDuf]

FWIW, I get the same problem with yarn. On Mac, deleting node_modules and reinstalling usually fixes the problem, but it does not work on Windows. Node versions are also v14.x.

[jasonswearingen]

I also noticed the same issue with node 10.x occasionally. It seems the proper workaround is to delete the node_modules folder.

I would bet that the bug is in RE2's "download binary or rebuild" workflow.

[uhop]

What bothers me is that according to reports from you guys the problem is intermittent. It is not like "RE2's 'download binary or rebuild' workflow" runs a generator of random numbers. It must be something related to race conditions when yarn/pnpm running initialization for multiple packages in parallel.

One obvious problem on my side is calling npm run rebuild instead of npm rebuild. More subtle problem is calling npm rebuild instead of yarn rebuild or pnpm rebuild — who knows how old npm is in your environment. It is very likely that yarn/pnpm use a non-standard layout + a custom module loader, so npm just fails.

I'll try to detect what package manager is used and call appropriately or just bypass a download completely.

[uhop]

One obvious problem on my side is calling npm run rebuild instead of npm rebuild.

Actually it is not a bug: a script called rebuild is used, which should be used with npm run. Maybe it should be renamed to avoid an impression of a name clash.

[jasonswearingen]

The issue appears to occur when a module rebuild occurs. It seems that yarn will delete built modules more aggressively than npm when some other module in the dependency tree changes, and it tries to rebuild all native-compiled modules when that occurs. When that happens, it seems like re2 doesn't rebuild itself and assumes it's built binary still exists.

So not "intermittent" but just under specific situations.

[uhop]

So not "intermittent" but just under specific situations.

Do we have a minimal repro case?

[jasonswearingen]

i will try making one this weekend. feel free to ping me monday I don't post by then.

[uhop]

@jasonswearingen ping!

[jasonswearingen]

thanks, I will get this going in the morning!

[jasonswearingen]

hello, repro here: https://github.com/jasonswearingen/20210416-re2-bug-repro/tree/main/bugrepro

using node 14.x and yarn 1.x

[uhop]

Status update: I spent some time analyzing the problem. It seems that I have to detect what package manager is used and invoke it to rebuild the package.

Right now: if for some reason a prebuilt binary release has failed to be properly installed for whatever reasons, then npm run rebuild is invoked, which is an alias for node-gyp rebuild. Note that node-gyp was recently added as a direct dependency to the package as recommended by yarn's guidelines.

Potential problem: the new version of yarn and pnpm use a different way to keep dependencies, while npm assumes the standard node_modules layout. Calling npm could break the rebuild because various assumptions can be violated (location of modules, the current directory, and so on).

[dezfowler]

Having this same issue with node 10.x and yarn 1.22.x on Windows.

The issue for us is that it's failing to find prebuilt binaries due to looking for a modules version of "64" where the earliest available on GitHub for release 1.16.0 is "72".

So, referring to this I guess we need Node v12 or newer to use those binaries.

Update: After exploring the other option of getting node-gyp plus it's dependencies installed and working I've identified that yarn isn't running the post install script to do the download and compile at all. I think this is because it's called install and yarn only supports postinstall (which npm does also support).

I raised a PR to change the install script name and to downgrade the node-gyp dependency to v7 (which supports both Python 2 and 3 and works better on Windows). This is working reliably for me using yarn on Windows with Node v10.

You have to manually install some other dependencies for node-gyp to work and on Windows I just installed the windows-build-tools npm package which installs a C++ compiler and Python 2. Refer to the node-gyp readme for details on what to install for your platform.

[uhop]

On Node 10: it is EOLed already. See the wiki for more details. Node 15 will be EOLed in a few weeks. It should still work for those and other versions, but no precompiled binaries. It will compile locally.

Generally, the idea of precompiled binaries is not about avoiding setting up the necessary node-gyp environment, but to save users of common configurations some time and CPU ticks, if possible.

I am looking forward to review and test the PR when I am back in a week. Thank you very much!

[dezfowler]

On Node 10: it is EOLed already.

Sure, that is a valid point and we need to upgrade.

I think the challenge for us is that node-gyp is not part of a standard Node install and none of our other dependencies require it, mainly because this is a browser-based application and Node is only used for bundling/linting JavaScript and CSS.

[maccman]

I see this all the time on Node v14.16.1. Rming node_modules and reinstalling all my packages seems to work. On a m1 air.

[byteab]

I have removed the node_moduels and reinstalled them but the issue still exists

[vicary]

I have this issue when running our CI/CD pipelines in Alpine, node 12. Our dev machines, which are Macbooks, have no problem until codes are pushed up there.

[mschipperheyn]

This went from intermittent to now happening all the time for me. Lost already an entire day on this. Is this library absolutely necessary to have at all? The cumulative time for this issue across our project life is beyond belief.

[vicary]

@mschipperheyn If you are using url-regex-safe, I have made an adaptation in spamscanner/url-regex-safe#19 to allow a workaround, just upgrade to 2.1.0 and mark re2 as external. If you need get-urls, you may make a PR there to do the same.

Is this library absolutely necessary to have at all?

re2 is the one being depended on, not the other way round. The owner also said above that he needs help, I don't think this sentence makes any sense.

[uhop]

Note: corepack could be of help communicating with different package managers. At the moment it is an experimental feature of Node.

[dmcd]

I'm also seeing this when using esbuild.

Here's an example repo demonstrating the issue:
https://github.com/dmcd/metascraper-esbuild-example

[uhop]

@dmcd I tried the demo repo you created (thanks!) and I can reproduce what you see. Specifically, I see this error:

✘ [ERROR] Could not resolve "./build/Release/re2"

    node_modules/re2/re2.js:3:20:
      3 │ const RE2 = require('./build/Release/re2');
        ╵                     ~~~~~~~~~~~~~~~~~~~~~

Yet, when I inspect the directory referenced above it is there:

$ ll node_modules/re2/build/Release/
total 1360
drwxr-xr-x  5 eugene  staff     160 Jun 12 09:42 ./
drwxr-xr-x  8 eugene  staff     256 Jun 12 09:41 ../
drwxr-xr-x  3 eugene  staff      96 Jun 12 09:41 .deps/
drwxr-xr-x  3 eugene  staff      96 Jun 12 09:41 obj.target/
-rwxr-xr-x  1 eugene  staff  695288 Jun 12 09:42 re2.node*

The file that refers to it is there too:

$ ll node_modules/re2/re2.js
-rw-r--r--  1 eugene  staff  588 Jun 12 09:41 node_modules/re2/re2.js

Must be something wrong with a module loader used by yarn.

[uhop]

I opened a PR: yarnpkg/berry#4546

[uhop]

The PR was merged.

I understand that there is still an issue with PNPM. If it is still important/critical, please open another PNPM-specific ticket.

[rishavs]

I'm also seeing this when using esbuild.

Here's an example repo demonstrating the issue: https://github.com/dmcd/metascraper-esbuild-example

I am also getting it for esbuild.
I use esbuild to compile my typescript files to a single js file, which I then run with nodejs.
I am using npm for the package management and npm throws no errors during install. My env is wsl2 on windows 11.

$ npm run comp

> [email protected] comp
> esbuild --bundle ./src/compiler/compter.ts --format=esm --log-level=warning --platform=node --outfile=dist/compiled.js && node dist/compiled.js

✘ [ERROR] Could not resolve "./build/Release/re2"

    node_modules/re2/re2.js:3:20:
      3 │ const RE2 = require('./build/Release/re2');        
        ╵                     ~~~~~~~~~~~~~~~~~~~~~

[MindRave]

I'm running into the same issue with esbuild and I'm wondering whether it's due to re.js in "build" importing re2 like so:

const RE2 = require('./build/Release/re2');

Inspecting the "Release" folder shows an re2.node file - which is a native node module. esbuild does not support these, and you have to add specific rules for them in the esbuild config, e.g:

loader: {
  ['.node']: 'copy'
}

Which is then going to use the copy loader for any native node files, and automatically copy them next to your bundles & correctly reference them in your compiled JS.

Seeing as the re2 "require" above doesn't have the .node extension, esbuild doesn't seem to know what to do with it and fails. Changing this line to include the node extension (locally) allows esbuild to correctly handle the file.

@uhop can we change the import to explicitly state const RE2 = require('./build/Release/re2.node'); ? This shouldn't impact the library, right? And it would let us properly handle them in our bundler configs.

Thank you!

[uhop]

Thank you for the suggestion. I'll test it and if it doesn't affect other environments I'll update the import.

[boomsi]

Meet the same problem.
How can we temporarily resolve this issue? I might not be able to wait for the next version.:)
Thx.

[boomsi]

env:
[email protected]
re2: 1.18.0
[email protected]

[moltar]

Getting same issues with pnpm and node v18.19.0

Error: Cannot find module './build/Release/re2.node'
Require stack:
- /.../node_modules/.pnpm/[email protected]/node_modules/re2/re2.js
 ❯ Object.<anonymous> ../../node_modules/.pnpm/[email protected]/node_modules/re2/re2.js:3:13