scanImports() function reads all files at once, can cause EMFILE in larger projects

[FredKSchott]

Original Discussion: #1244
/cc @N8-B

The scanner appears to be causing trouble in the scanImports() function, where we call fs.promises.readFile() on all scanned files at once. You appear to be the first to hit this limit in a breaking fashion! (others at your size are probably just seeing degraded performance).

We use p-queue in the site builder for this exact reason, we should bring that same library into scanImports() to make sure that we are never loading more than X files at once in parallel. Node.js is great at I/O, so I think this number can be pretty high.

This is a great issue for anyone to tackle since it doesn't rely on any deeper understanding of Snowpack! Also should be covered by existing tests (a test directory with 10k files would be great to have, but probably out of scope for now).

[N8-B]

Thanks for opening up this issue. I'll look into it today and run tests on our code base and deploy environment.

[FredKSchott]

Fixed by #1256

[alexeychikk]

I am still having this issue:
EMFILE: too many open files, open ...
"snowpack": "^2.18.4"
My config:

/** @type {import("snowpack").SnowpackUserConfig } */
module.exports = {
  mount: {
    public: { url: '/', static: true },
    src: { url: '/dist' },
  },
  plugins: [
    '@snowpack/plugin-webpack',
    '@snowpack/plugin-react-refresh',
    '@snowpack/plugin-dotenv',
    '@snowpack/plugin-typescript',
  ],
  install: [
    /* ... */
  ],
  installOptions: {
    namedExports: ['react-google-login'],
  },
  devOptions: {
    /* ... */
  },
  buildOptions: {
    /* ... */
  },
  proxy: {
    /* ... */
  },
  alias: {
    '@src': './src',
  },
};

[alexeychikk]

If someone hits this, I fixed it like this:

1. install graceful-fs

npm i -D graceful-fs

2. create build.js in your projects folder:

require('graceful-fs').gracefulify(require('fs'));

const snowpack = require('snowpack');

(async () => {
  try {
    await snowpack.cli(['', '', 'build']);
    process.exit(0);
  } catch (err) {
    console.error(err);
    process.exit(1);
  }
})();

3. replace your build script with:

  "scripts": {
    "build": "node ./build.js"
  }