(core): CustomResourceProvider writes to node_modules

[bsweeney-gingerio]

What is the problem?

On instantiation, the CustomResourceProvider class writes a file into the node_modules directory instead of creating a safe temporary directory to stage local assets. What's more, for users of Yarn berry with PnP, node_modules is a read-only ZipFS filesystem, which raises an exception when the write is attempted.

Reproduction Steps

const bucket = new s3.Bucket(this, 'MyBucket', {
  removalPolicy: cdk.RemovalPolicy.DESTROY,
  autoDeleteObjects: true, // <-- this line causes synth to fail, uses CustomResourceProvider
});

What did you expect to happen?

A stack would be synthesized containing a custom resource to delete objects from the above bucket in the event the bucket is ever removed from the stack.

What actually happened?

me@host % yarn cdk synthesize
/Users/me/workspace/cdk/.pnp.cjs:17448
  return Object.assign(new Error(`${code}: ${message}`), {
                       ^
Error: EROFS: read-only filesystem, open '/node_modules/@aws-cdk/aws-s3/lib/auto-delete-objects-handler/__entrypoint__.js'
    at makeError (/Users/me/workspace/cdk/.pnp.cjs:17448:24)
    at EROFS (/Users/me/workspace/cdk/.pnp.cjs:17478:10)
    at ZipFS.prepareWriteFile (/Users/me/workspace/cdk/.pnp.cjs:19787:30)
    at ZipFS.writeFileSync (/Users/me/workspace/cdk/.pnp.cjs:19775:14)
    at fallback (/Users/me/workspace/cdk/.pnp.cjs:20617:14)
    at /Users/me/workspace/cdk/.pnp.cjs:20637:18
    at /Users/me/workspace/cdk/.pnp.cjs:20951:58
    at ZipOpenFS.getZipSync (/Users/me/workspace/cdk/.pnp.cjs:21096:14)
    at ZipOpenFS.makeCallSync (/Users/me/workspace/cdk/.pnp.cjs:20951:17)
    at /Users/me/workspace/cdk/.pnp.cjs:20631:19 {
  code: 'EROFS'
}
Subprocess exited with error 1

CDK CLI Version

1.131.0 (build 7560c79)

Framework Version

1.131.0

Node.js Version

v16.3.0

OS

macOS Big Sur 11.6.1

Language

Typescript

Language Version

TypeScript (4.4.4)

Other information

Previously reported as a problem in the aws-s3 module, closed by that module's maintainer:

#16552

Most likely source of the problem:

aws-cdk/packages/@aws-cdk/core/lib/custom-resource-provider/custom-resource-provider.ts

Lines 165 to 166 in 5aa6ac0

	// copy the entry point to the code directory
	fs.copyFileSync(ENTRYPOINT_NODEJS_SOURCE, path.join(props.codeDirectory, `${ENTRYPOINT_FILENAME}.js`));

A temporary directory should be created (the core module provides FileSystem.mkdtemp) and the code should be written to that instead of assuming that the module directory can accept writes. Specifics for Yarn 2.x can be viewed at:

https://yarnpkg.com/features/pnp

[cuongvo]

Unsure why issue #16552 was closed. This is non-standard behavior (leading to breaking users in unexpected ways) and should be modified by the CDK.

The only workaround so far is to unplug the dependencies: yarn unplug monocdk -A for example if you're using monocdk otherwise you'll need to explore recursively removing all of the CDK (docs).

[rix0rrr]

I agree this is poor behavior.

[bsweeney-gingerio]

Problem still exists in CDK 1.134.0 and 2.1.0.

[izaakschroeder]

A monkey-patch that you can fix this error with:

import {CustomResourceProvider, CustomResourceProviderProps} from 'aws-cdk-lib';
import {Construct} from 'constructs';
import * as path from 'path';
import * as os from 'os';
import * as fs from 'fs';
import * as crypto from 'crypto';

const tmp = os.tmpdir();
const getOrCreateProvider = CustomResourceProvider.getOrCreateProvider;
CustomResourceProvider.getOrCreateProvider = (
  scope: Construct,
  id: string,
  props: CustomResourceProviderProps,
) => {
  const hash = crypto
    .createHash('sha1')
    .update(props.codeDirectory)
    .digest('hex');
  const codeDirectory = path.join(tmp, `cdk-crp-${hash}`);
  if (!fs.existsSync(codeDirectory)) {
    fs.mkdirSync(codeDirectory);
    fs.writeFileSync(
      path.join(codeDirectory, 'index.js'),
      `module.exports = require(${JSON.stringify(props.codeDirectory)});`,
      'utf8',
    );
  }
  return getOrCreateProvider(scope, id, {
    ...props,
    codeDirectory,
  });
};

Works with at least yarn@3.

[izaakschroeder]

I lied. Referencing the original version above only passes node's cursory checks and doesn't actually build properly. The files need to be copied. A better version follows (and appears to work):

import {CustomResourceProvider, CustomResourceProviderProps} from 'aws-cdk-lib';
import {Construct} from 'constructs';
import * as path from 'path';
import * as os from 'os';
import * as fs from 'fs';
import * as crypto from 'crypto';

const copyFilesSync = (srcDir: string, dstDir: string) => {
  fs.readdirSync(srcDir).forEach((file) => {
    const src = srcDir + '/' + file;
    const dst = dstDir + '/' + file;
    var stat = fs.statSync(src);
    if (stat && stat.isDirectory()) {
      fs.mkdirSync(dst);
      copyFilesSync(src, dst);
    } else {
      fs.writeFileSync(dst, fs.readFileSync(src));
    }
  });
};

const tmp = os.tmpdir();
const getOrCreateProvider = CustomResourceProvider.getOrCreateProvider;
CustomResourceProvider.getOrCreateProvider = (
  scope: Construct,
  id: string,
  props: CustomResourceProviderProps,
) => {
  const hash = crypto
    .createHash('sha1')
    .update(props.codeDirectory)
    .digest('hex');
  const codeDirectory = path.join(tmp, `cdk-crp-${hash}`);
  if (!fs.existsSync(codeDirectory)) {
    fs.mkdirSync(codeDirectory);
    copyFilesSync(props.codeDirectory, codeDirectory);
  }
  return getOrCreateProvider(scope, id, {
    ...props,
    codeDirectory,
  });
};

[blimmer]

This is related to #19452

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.