Define spec for development environment field

[GeoffreyBooth]

This is a draft spec for nodejs/node#51888 (comment), to define the runtime and package manager for developing a project or package (not consuming/installing it as a dependency in another project). We could use the name devEngines and it would be a new top-level field defined with this schema:

interface DevEngines {
  os?: DevEngineDependency | DevEngineDependency[];
  cpu?: DevEngineDependency | DevEngineDependency[];
  libc?: DevEngineDependency | DevEngineDependency[];
  runtime?: DevEngineDependency | DevEngineDependency[];
  packageManager?: DevEngineDependency | DevEngineDependency[];
}

interface DevEngineDependency {
  name: string;
  version?: string;
  onFail?: 'ignore' | 'warn' | 'error' | 'download';
}

The os, cpu, libc, runtime, and packageManager sub-fields could either be an object or an array of objects, if the user wants to define multiple acceptable OSes, CPUs, C compilers, runtimes or package managers. The first acceptable option would be used, and onFail would be triggered for the last defined option if none validate. If unspecified, onFail defaults to error for the non-array notation; or it defaults to error for the last element in the array and ignore for all prior elements, for the array notation. Validation would check the name and version ranges.

The name field would be a string, corresponding to different sources depending on the parent field:

• os: NPM docs for engines.os.
• cpu: NPM docs for engines.cpu.
• libc: glibc or musl; see NPM docs for config.libc.
• runtime: WinterCG Runtime Keys.
• packageManager . . . I don’t know, we could define the list as part of the spec, or perhaps it would need to correspond to an npm registry package name. Suggestions welcome.

The version field syntax would match that defined for engines.node, so something like ">= 16.0.0 < 22" or ">= 20". If unspecified, any version matches.

The onFail field defines what should happen if validation fails:

• ignore: nothing.
• warn: print something and continue.
• error: print something and exit.
• download: remediate the validation failure by downloading the requested tool/version.

In the event of onFail: 'download', it would be the responsibility of the tool to determine what and how to download, perhaps by looking in the tool’s associated lockfile for a specific version and integrity hash. It could also be supported on a case-by-case basis, like perhaps Yarn and pnpm could support downloading a satisfactory version while npm would error.

Typical example:

"devEngines": {
  "runtime": {
    "name": "node",
    "version": ">= 20.0.0",
    "onFail": "error"
  },
  "packageManager": {
    "name": "yarn",
    "version": "3.2.3",
    "onFail": "download"
  }
}

“Uses every possible field” example:

"devEngines": {
  "os": {
    "name": "darwin",
    "version": ">= 23.0.0"
  },
  "cpu": [
    {
      "name": "arm"
    }, {
      "name": "x86"
    }
  ],
  "libc": {
    "name": "glibc"
  },
  "runtime": [
    {
      "name": "bun",
      "version": ">= 1.0.0",
      "onFail": "ignore"
    },
    {
      "name": "node",
      "version": ">= 20.0.0",
      "onFail": "error"
    },
  ],
  "packageManager": [
    {
      "name": "bun",
      "version": ">= 1.0.0",
      "onFail": "ignore"
    },
    {
      "name": "yarn",
      "version": "3.2.3",
      "onFail": "download"
    }
  ]
}

Some potential future expansions of this spec that have been discussed are:

• runtime and packageManager might take shorthand string values defining the desired name or name with version/version range.

[GeoffreyBooth]

onFail is a user directive, not a maintainer directive.

But in this case, the user is the maintainer. This whole configuration block only applies when it’s the developer of the package/project.

[wraithgar]

How does download even work with a semver range? If these are all npm registry packages then there is already a well defined discovery mechanism for downloading a given named package at a given version.

I'd love to hear from yarn or pnpm if and how they'd plan on handling the failure state where the user has indicated they want to download the correct version. How do you pick the version? Do you spawn it yourself? Why not fail and let the user remediate?

[wraithgar]

But in this case, the user is the maintainer

Good point. As you can see the lines between "installing as module" and "interacting as app" are so blurry that even I missed that.

[GeoffreyBooth]

How does download even work with a semver range?

If a URL is provided, we’re just downloading that exact URL. The version constraint wouldn’t apply.

If we want to allow onFail: 'download' and a missing download field, the only behavior I can think of would be to query the npm registry for the requested package manager and download the newest version that fits within the range.

[ljharb]

This is a great start. I think it's important that it not just be limited to the runtime and the package manager - process.versions has a lot of things in it that would be really nice to also constrain in various cases, especially openssl which has lots of CVEs.

[styfle]

Tagging relevant people to review the draft spec: @Ethan-Arrowood @arcanis @zkochan @aduh95

[wraithgar]

I am not very good at reading specs like this. What would it look like in implementation to limit the local dev environment to a given os or cpu for example? From what I see it doesn't look like that field is sufficiently decoupled from "named" which doesn't make sense in that context.

[aduh95]

packageManager is already used in the wild as a string, we'd need to define what happens when the value is a string to allow a smooth transition – another option is to use a different key, but packageManager is quite good a describing what it is.

[wesleytodd]

but packageManager is quite good a describing what it is .... is already used in the wild as a string

This proposal covers way more than package manager and would not be under that field name (although it is not mentioned above). I think that should have always been the goal, so that is why keeping the field "experimental" in node core was important. I agree we would want to smooth the path. But I think that is up to the folks who implemented the field in their tools. Maybe here we would just discuss goals for ways to make that transition?

[ljharb]

The first step should be figuring out (such that they can be clearly documented and conveyed) the problems we're solving, and the goals and non goals - and then to sketch out a solution that seems viable to all stakeholders.

At that point, those interested in the migration path from the experimental packageManager field to the new thing can explore the best way to achieve that - but until the solution design is settled I'm not sure how any progress could be made on migration.

[Ethan-Arrowood]

Cross-posting from Slack:

I’ve attempted to read all of the various GitHub and Twitter threads that have led us here. I think there is a lot going on in general, and I don’t have a clear, concise reply just yet. For now, all I will say is that I encourage all invested parties to join the monthly call on March 5th. We rarely have much to discuss regarding the group’s active work streams, so we can reasonably spend the entire hour discussing the corepack issue.

For those looking for the upcoming meeting invite, please copy the invite from the OpenJS Calendar

[GeoffreyBooth]

What would it look like in implementation to limit the local dev environment to a given os or cpu for example?

I added those fields because you had mentioned them on the other thread. I'm not sure what the use case would be; run npm on Windows and Bun on macOS? I can just remove them if you can't think of a use case either. We can always add fields later.

[ljharb]

I think that os/cpu etc are in the category of "general engines worth explicitly giving a range for", and altho "runtime" and "package manager" are reasonable to call special, the schema needs a mostly free-for-all dumping ground too (like engines, import.meta, etc are all now) for things like this.

[wraithgar]

Yeah, remember we're defining a spec for all development runtime constraints, not just package manager. Package manager is a subset of constraints. Node version, os, cpu, and libc are other constraints.

[zkochan]

Why would you allow specifying multiple different package managers? I assume there are still people in this group that are strongly against lockfiles. But at least don't motivate devs to use different package managers on a project. It will make everyone's life a lot harder.

[wesleytodd]

Why would you allow specifying multiple different package managers?

My guess is this is done to remain flexible, not make a recommendation that folks should. That said, I could think of a few cases where maybe you want to use pnpm for local dev but npm for publish or something of that nature?

I assume there are still people in this group that are strongly against lockfiles.

I don't think that would be a driving part of the decision is it? I assume you are meaning that if a package declared multiple dev package managers they would then have to resolve the lockfile differences? I think at this layer of abstraction (a spec for the package.json) we would not want to exclude package managers from agreeing on a shared lockfile format (in fact we might want to encourage that 😎) so I think it is reasonable to keep.

[styfle]

How about changing “hash” into two fields: “algorithm” and “digest”?

[GeoffreyBooth]

How about changing “hash” into two fields: “algorithm” and “digest”?

That’s fine, what would be an example? This?

"download": {
  "url": "https://registry.npmjs.org/@yarnpkg/cli-dist/-/cli-dist-3.2.3.tgz",
  "algorithm": "sha224",
  "digest": "16a0797d1710d1fb7ec40ab5c3801b68370a612a9b66ba117ad9924b"
}

[wesleytodd]

algorithm and digest should also be optional (but encouraged for the implementing tools to add/update them).

[wraithgar]

What's the "hash" currently? In npm lockfiles it's the output of ssri.stringify which includes both algorithm and digest

[wesleytodd]

For folks who need it (me) here is what ssri references: https://www.npmjs.com/package/ssri

EDIT: I should have added, this is an implementation of the subresource integrity spec used in browsers as well: https://w3c.github.io/webappsec-subresource-integrity/

So with that in mind, I would say that seems like it could be a good spec to just directly here.

[ljharb]

At runtime, a package would support 1 or N runtimes, and pnpm would use whatever's first in the list, I suppose, but it also shouldn't matter since any runtime in engines should work.

devEngines has nothing to do with runtime, though.

[zkochan]

Yes, you are right, you can ignore my points about the runtime part. However, the part about dependencies could be still valid.

[wesleytodd]

This definitely something that exists today. Like we have users that use different node.js versions in different projects of the monorepo. Sometimes those projects use the same dependencies.

I am highly suspicious that long term this is a really bad decision. I would happy to be proven wrong, but my spidy senses tell me this is asking for problems.

[styfle]

This definitely something that exists today. Like we have users that use different node.js versions in different projects of the monorepo. Sometimes those projects use the same dependencies.

If each project is a subdirectory in the monorepo, then each subdirectory could have its own package.json with engines/devEngines to define a different Node.js version.

[zkochan]

Two projects use the same dependency (inside node_modules). One project has devEngines set to Node.js 18 and the other one to Node.js 20. What Node.js version should be used to run the dependency's postinstall script? We don't build it separately for each project.

[ljharb]

devEngines should not have any impact on anything unless it's in the current project - so things inside node_modules only use engines, never devEngines.

[darcyclarke]

@zkochan I love that pnpm is going to support runtime management at the project level (also love that global management of node is already supported today - big props for leading the way here with pnpm env).

That said, I'd encourage you - & others - to make any runtime values/definitions equivalent to dependency specs if not rely on the existing dependency schemas (ie. dependencies, devDependencies, peerDependencies, optionalDependencies, bundledDependencies). It seems to me that a DevEngineDependency should be equivalent to whatever a Dependency is in the eyes of each package manager today. This would unlock every permutation of software package/distribution supported by the existing specs (including: remote http refs, dist-tags, semver range grammars, git repositories & more). It would also mean that the current management capabilities (ie. parsing, fetching, resolving, downloading, verifying, unpacking, linking, erroring, locking, updating etc.) are already well-known, paved-paths.

Although node & deno do not officially distribute themselves on a JavaScript registry - yet - there are faithful distributions living at npm:node, npm:deno-bin & bun is - officially - distributed at npm:bun. Runtimes are dependencies for projects & they should be able to be managed in all the same ways we expect for our other dependencies (which is why we seem to be trying to recreate all the same mechanics to manage them that we already have for our existing dependencies).

Notably, I could see this effort/work pushing projects like node/deno to finally, officially, add distribution to a JS package registry as part of their publishing workflows.

• how does a CLI app specify which js runtime it wants to run with?

I'd argue that a CLI should be a self-contained/standalone executable & if there is any runtime requirement it should be defined as a dep &/or peer-dep. That said, I totally understand this has not been the standard practice in our ecosystem (ie. CLI tools have come to rely on npx / node being available globally on the system & don't define them as dependencies at all). I think we could do the ecosystem some good by encouraging the practice of defining required runtimes as deps/peer-deps. We should also make it common place/known to authors that they can't expect their software to work if they don't define the things it needs to run properly.

• how does a package that needs to be built tell the package manager which exact js runtime to use for running the postinstall scripts?

I would be mindful not to optimize any future APIs/tooling for a legacy feature like install scripts. Instead of leaning further into them, I think we'd benefit much more from working together on something like package distributions where the expectation is that a package is consumable without the need for arbitrary script execution. If there's a use case that isn't handled by the linked RFC, I'd love to flush it out with you so we have a consistent experience for end-users going forward (binding/linking native addons via the distribution config is definitely one area that requires more thought).

• as a package consumer, how do I tell the package manager (or script/task runner) what js runtime to use for a specific dependency?

It's extremely rare that you run a dependency's script definitions outside of lifecycle scripts & even then your script/task should be agnostic of any executables not defined as dependencies (see my previous comment on defining runtimes as deps/peer-deps).

Maybe I still use an older version of node.js but I want to install a newer eslint that only supports the latest node.js version.

This sounds a lot like overrides / resolutions where consumers find themselves in situations where they are forced to resolve unresolvable dependency conflicts at the project-level. I believe overrides / resolutions are likely the right tools for this job although I know the experience around creation of these could be much friendlier/interactive (that's on us as tooling authors though). Of course, this means executables cannot rely on being hoisted to ./node_modules/bin if they must be nested to satisfy/resolve dep requirements.

This question reaffirms my belief that we should support multiple runtimes & runtime versions at the project level (just like we do with all other dependencies).

The fact that npm, yarn & pnpm have historically had an undocumented peer-dep relationship with whatever node version was available on the system is sort of the worst case scenario. Anything we do here or encourage here in the way of documenting these relationships is better then the status quo (ie. yolo). I'm excited that we're looking at this space seriously & thinking about ways to clearly document the undocumented software our project's require to run.

devEngines should not have any impact on anything unless it's in the current project - so things inside node_modules only use engines, never devEngines.

What's a "current project"? What prevents a tool from reading devEngines inside node_modules? If I cd into a node_modules directory, should the tool I'm using not read from a local package.json? I think this assumes tools are going to do some kind of project root resolution & either nopt or do something else special in those situations. For me, this just highlights that context is required no matter what to interpret a dev* configuration, devoid of its name (ie. we could be using an existing field since the true feature/requirement here is additional logic/safeguards).

I harp on this point time & time again because I've seen, firsthand, how development-specific configuration becomes indistinguishable from "production"/distribution configuration devoid of context. The most egregious of these is dependencies vs. devDependencies. Ideally, something like package distributions/variants would help to reclaim some sanity in regards to context.

[ljharb]

@darcyclarke the current project is the package.json whose dependency list is being installed. nothing prevents a tool from reading devEngines inside node_modules except that it'd be a categorically incorrect and nonsensical thing to do.

If you cd into a node_modules/foo or node_modules/@foo/bar dir, then that becomes the current project, which is how npm treats it as well.

Tools always must do "project root resolution", as npm does, and without that there's no way to meaningfully run the majority of useful commands.

[darcyclarke]

Depending on how a tool has implemented monorepos/workspaces, file or linked-deps (ie. if the tool allows for those to live in node_modules) I can easily see a situation where reading & respecting a devEngines definition inside that folder might be totally valid/useful. My point there is that there are some assumptions here but a lot of nuance/complexity when we get to implementation.

[ljharb]

Certainly if a tool has created its own concept of layout, then it'll have to figure out how devEngines works with that. The intention/purpose/design of it, though, is to only apply when a project is being developed, never when it's being consumed/installed.

[Ethan-Arrowood]

Hi folks!

This thread is getting quite long and very hard to follow. During the monthly call, we discussed the following steps:

1. Contribute a document to this repo summarizing the current state of the devEngine field
2. Close this issue
3. Create new, separate issues for any outstanding questions and continue those discussions

Based on the new PRs from npm, I think the timing for this couldn't be better!

Would anyone like to volunteer to do step 1? Otherwise, I will do my best to draft something and then ask folks to review it.

Thank you for the great work everyone!

[GeoffreyBooth]

Would anyone like to volunteer to do step 1?

I can open a PR with the top post as a file.

[reggi]

Not sure if this has been touched on. This does have some interesting consequences for runtimes which do node interop if someone specifies node but is running within the deno, it supports an emulated version of node as (await import('node:process')).version "v20.11.1". If you put node do you really mean truenode, or is emulated node ok? If deno were to implement a devEnginescheck on deno install should it fail on a project that specifies node? Also bun and deno are both runtimes and package managers unlike yarn and pnpm, not sure if this has any consequences as well, except for the duplication of the fields.

[ljharb]

I think that'd be up to the runtime providing emulation - if it's confident it's the same, then it'd be ok to pretend it's the same, and if it's not exactly the same, it wouldn't count.

I think one runtime actually emulating another is a rare enough scenario that we don't have to plan for it much.

[wraithgar]

[email protected] has an initial implementation of this spec.

[ljharb]

Awesome! Within the next week, I'll try to make a PR to npm to add a way to make it useful in a CI matrix.

[Ethan-Arrowood]

With #27 merged, lets continue and start new discussions regarding this field through separate issues or PRs to that document. Everyone is welcome to contribute details to that document as well if there are certain discussions/decisions you'd like to be documented. Thank you all for the collaboration here 🚀 together we are making JS development better and better