Update npm to v1.2.8000

[isaacs]

Addresses the issues raised in #7789 (comment)

[tjfontaine]

I will review this PR, but based on the diff stat it appears that your solution is to only update the dependencies of npm as opposed to the top level npm module?

Is there no smaller more precise way to fix this issue?

To be clear, me rolling a new version of node 0.8 with this update does not fix people in production where their version of npm is broken, this is even true of the versions of npm in 0.10.X that were broken by this.

This only provides a bootstrap for people who can upgrade their version, but doesn't solve for those people who are broken in production today.

[isaacs]

People using 0.8 in production are generally using 0.8.latest in development, and 0.8.latest in Travis-ci to test their modules.

This means that there is a year of bug fixes that they aren't getting. Their CI tests are failing, and they're unable to properly publish new versions of their modules, without going through an extra out-of-band process to update npm. This extra step is especially difficult on a CI system.

This change is a version of npm with all of the same improvements that are in 1.4, but with the spinner and loglevel set to the 1.2.x defaults, and sets the version number to 1.2.x. You had mentioned in #7789 that these were the blockers. If there are other reasons to object to this update, please list them so that they can be addressed.

[isaacs]

Also, anyone using 0.8 in production, and not pulling in the latest 0.8 update, is asking for trouble, since they are passing up two security fixes.

[rlidwka]

People who is still using 0.8 value stability above all else (why use it otherwise?), so they probably won't be happy with newest npm (I see compatible settings, but an amount of changed lines looks the same in two PRs).

If it's about carets, maybe just update semver module leaving anything else as is?

[isaacs]

@rlidwka I'd tend to agree with you, except in this case, there are some problems with that view of things.

1. npm 1.2 was developed back when we could assume that "the registry" = "single couchdb instance". That hasn't been the case for a long time. As a result, npm 1.2.30 is incapable of properly publishing, and has serious problems even logging in, because it relies on immediately updated state and is susceptible to race conditions. Since the registry is now actually a collection of databases behind a CDN, this is a huge problem. It cannot be fixed without significantly refactoring many internal parts of npm, which has happened over the last year.
2. Node users who are relying on consistent APIs don't generally run npm at run-time. It might be part of the deployment story, but it's almost never part of the actual run-time API that matters. So, internal changes to the structure of the cache folder, for example, are not relevant. The Node module system semantics haven't changed since Node 0.4, and npm 1.0. Viewing npm in the same category as V8 and libuv is fundamentally missing the point about the role that npm plays in the Node experience, vs the role that V8 and libuv play.
3. It's not that people using 0.8 "value stability above all else", but rather that the costs of upsetting their run-time environment are high enough that any benefits of use Node 0.10 are not justifiable. However, if they're publishing shared modules, they'd still like that process to work. (For evidence of this, just see the npm issues list.)

It is actually not possible to "just make the registry work with old npm". We've increased in size by well over 20x since 1.2.30 was released. The assumptions that made sense a year ago no longer make sense today. Users are coming to me understandably confused and frustrated that the npm that a new release of Node ships with a year-old version of npm that no longer works reliably.

[mgol]

Another 0.8 release is needed anyway as 0.8.27 is broken; see #7805.

EDIT: seems that issue didn't require a new release.

[tjfontaine]

We want the module ecosystem and Node.js experience to continue to thrive -- it has been and will continue to be one of the crucial selling points of Node.

Clearly npm needs to move at its own pace, and a lot of things have been added that can improve the developer and publisher experience. Since npm is moving at a faster pace than Node, the community should know and understand how to upgrade their experience out of the normal release cycle of Node, similar to how people are used to doing so with say CPAN or gem.

The log level and spinner were just examples of changes that are hard to decipher in this 7.2MB patch, in which there are a lot of moving parts. What I'm trying to solve for is a mechanism that allows people to install a version of Node v0.8 and do npm install -g npm (or install any other package) without running afoul of the ^ change.

The Node installer therefore can be a bootstrap into a known good, working, and acceptable version of npm that we've decided to ship, with the added benefit if there's a feature that a user is looking for that is in a newer version of npm they are able to upgrade to that.

It is clear that the challenges for npm weren't sufficiently appreciated in the past and design decisions were made that make that compatibility especially difficult now, but solving constrained problems is the nature of engineering. We have together learned that lesson the hard way in Node. In the same way we've worked hard to solve compatibility issues in Node, let's work together to find a solution now.

[othiym23]

Since npm is moving at a faster pace than Node, the community should know and understand how to upgrade their experience out of the normal release cycle of Node, similar to how people are used to doing so with say CPAN or gem.

"Should" is being asked to a lot of work here. I don't think this statement matches reality:

• Travis does not, by default, upgrade to the newest npm
• the user community, as attested by the npm issue tracker, is not quick to upgrade their npm to the latest version; many users are primarily front-end devs following tutorials on using Yeoman or bower and barely know what npm is, much less the importance of upgrading to the newest version
• many, if not most, downstream distributions do not package npm based on the latest npm version, but by the version bundled with a given release of node

It is clear that the challenges for npm weren't sufficiently appreciated in the past and design decisions were made that make that compatibility especially difficult now, but solving constrained problems is the nature of engineering. We have together learned that lesson the hard way in Node. In the same way we've worked hard to solve compatibility issues in Node, let's work together to find a solution now.

I still believe that it is insufficient merely to include a patch for ^; npm < 1.4 cannot safely or reliably publish to the current registry architecture, and a number of other serious bugs have been addressed in the CLI since 1.2.30 was released. Nor does a backport for ^ solve the problem of Travis using a known broken version of npm.

It is not realistic for us (by which I mean myself, as the chief maintainer of the CLI, and what time @isaacs can spend) to cherrypick those bugs onto a branch forked from 1.2.30, as there have been ~600 commits between then and the release of 1.4.16. This is very different from backporting targeted fixes from the master or 0.10 branches of node to 0.8. It's been over a year since 1.2.30, and the intention at the time of its release was not for there to be a separate maintenance series tracked alongside node.

Here are my suggestions for getting us past this impasse, in order of my preference. None strike me as particularly pleasant, although I think Isaac's proposal is the least painful overall:

1. Accept this PR. There is plenty of new functionality in 1.4.16, but it is still backwards-compatible with 1.2.30, and addresses the issues we've discussed previously.
2. Remove npm from the 0.8 distribution altogether, and add prominent instructions on how to install npm without npm to the README / homepage. This is user-hostile, but it clearly and unambiguously prevents users from relying upon a broken version of npm.
3. Add a patch to npm 1.2.30 that causes it to noisily log to stderr that it is unsupported, buggy, and out of date, with clear instructions on how to upgrade (and how to do that for Travis). I'm not wild about the idea of making this patch, but if nobody on the node side is willing to do it, I can make it happen.

I don't consider it an effective use of the npm team's time to try to figure out how to backport all of the important fixes from 1.4.16 to a new 1.2.x release.

[rlidwka]

Remove npm from the 0.8 distribution altogether, and add prominent instructions on how to install npm without npm to the README / homepage.

Can npm be reduced to a tiny script capable of only installing a package from npm registry without any (non-bundled) dependencies? I'm talking about something like scripts/install.sh rewritten in node.js for portability and renamed to npm-tiny or something like that.

[othiym23]

Can npm be reduced to a tiny script capable of only installing a package from npm registry without any (non-bundled) dependencies?

It can, but not easily. Even without the caching or npm-registry-client components, there's still a fair amount of code to deal with unpacking the tarball, setting permissions, and doing all the work of installing. It's complex enough that I believe it would run afoul of TJ's concerns about dropping a large volume of new code into a legacy distribution.

@isaacs and I have discussed the possibility of replacing the npm in the distribution with just install.sh. It's a poor solution for many Windows users, but it would do the job for Unix and Git Bash.

[tjfontaine]

I don't think we need the npm team to backport many fixes into 1.2, we're only looking for help with the semver fix, as it's a blocker for people using the legacy distribution for installing.

If the concern is around old npm's publishing in a new world, then I think it's reasonable (albeit unfortunate) to message to the user that this version of npm can no longer safely publish and that they should upgrade, and here's a link on how to do that -- but you can start with "npm install -g npm".

Are you looking for me to do a PR with these changes?

[dougwilson]

but you can start with "npm install -g npm".

It would be nice if npm bundled with core could be fixed such that Windows users could actually do that. Right now the first time you install a global module, %APPDATA%\npm is added to the end of PATH. Since this is where npm will install global libraries, doing npm install -g npm will basically do nothing, because the old npm is earlier in the search path (under C:\Program Files\nodejs, which was added by the Windows installer)...

[dougwilson]

Basically on Windows, if you npm install -g npm, you'll only be able to use the new version of npm if you open the "Node.js command prompt", but I'm not sure how many people actually use that, especially with all the people using Git Bash as their command prompt.

[isaacs]

We want the module ecosystem and Node.js experience to continue to thrive -- it has been and will continue to be one of the crucial selling points of Node.

Prove it.

If this is indeed "one of the crucial selling points of Node", then treat npm like it matters. We are here telling you that the arbitrary release scheme is making life harder for Node users, including people who are using Node and npm as part of grunt or yeoman, and don't even know what "npm" is.

You have an opportunity to improve their lives, and we are trying to help you do so. There is no one saying that they'd prefer to use npm 1.2.

I'm sorry to come across confrontational here, but frankly, when you say that you "value Node's module ecosystem", and don't follow that up with action, these words sound hollow to me.

Leading a project is more than just processing the rules arbitrarily. You are shipping broken software because the rules say so, yet you are the one in a position to change that rule. What is your priority?

I don't think we need the npm team to backport many fixes into 1.2, we're only looking for help with the semver fix, as it's a blocker for people using the legacy distribution for installing.

There is no way to reasonably backport "just" support for ^ into the SemVer util. The node-semver module was effectively rewritten for the SemVer 2.0 specification, and is a significant version bump for that reason. npm uses this module in many many places, as do the other modules throughout npm's dependency tree.

If you want to attempt this, go for it. The fork button is in the upper right.

Since npm is moving at a faster pace than Node

That is indeed a problem. Node's pace has slowed, and npm's has increased. How can we address that? Can we make Node's pace increase?

Is this just a resource problem? How can we help? What do you need for this to be successful?

Even that doesn't have to be a blocker, though.

The Node installer therefore can be a bootstrap into a known good, working, and acceptable version of npm that we've decided to ship, with the added benefit if there's a feature that a user is looking for that is in a newer version of npm they are able to upgrade to that.

That's what #7789 did. npm 1.2 is ancient. It is not supported. It is broken in half a dozen ways.

I urge you to take a hard look at what's actually stopping you from shipping a Node v0.8 with a more modern npm version.

In the past, we always shipped new versions of npm with Node releases, and we have been running the npm tests regularly with each of the supported versions of Node. I see no reasonable argument for Node v0.8 sticking to npm 1.2.

What is the difference between taking the approach of "get to where a user can npm i -g npm", and just giving them the right thing in the first place? As others on this thread have pointed out, that doesn't even work reliably on Windows, and it is problematic for CI systems. On top of any of that, it's an unnecessary bit of friction in the Node user experience, yet another hoop that they have to jump through to be successful.

What is the priority? Why does Node ship with npm in the first place?

It would be a lot easier to work together to engage in this discussion if you gave me something to work with, rather than platitudes about the nature of engineering. I can't help you reach your goal unless you tell me what it is.

[jasnell]

It's likely safe to assume it's ok to close this one ;-)