Updating npm?

[jdalton]

I noticed that nvs errors when attempting to update npm for a given version of node

npm i -g npm

errors with:

npm ERR! path C:\Users\appveyor\AppData\Roaming\nvs\node\8.4.0\x64\npm.cmd
npm ERR! code EEXIST
npm ERR! Refusing to delete C:\Users\appveyor\AppData\Roaming\nvs\node\8.4.0\x64\npm.cmd: is outside C:\Users\appveyor\AppData\Roaming\nvs\node\8.4.0\x64\node_modules\npm and not a link
npm ERR! File exists: C:\Users\appveyor\AppData\Roaming\nvs\node\8.4.0\x64\npm.cmd
npm ERR! Move it away, and try again.

Is there a recommended way to update npm with nvs?
This is in AppVeyor where I specify a couple node versions like node/8.

[jasongin]

This appears to be a bug in npm v5. I can reproduce it without nvs, by simply installing the node 8.x MSI, then running npm i -g [email protected]. (The latest npm is installed in a fresh install of node, so the repro requires specifying a different version.) Looks like npm/npm#18447 or npm/npm#16833

Earlier versions of npm (< 5.x) didn't have this problem.

[jdalton]

\cc @iarna

[jdalton]

It's still happening on nvs with Node 10+ and npm 6+.

[iarna]

This does not reproduce on all Windows systems (for instance, mine, because of course not, that would be too easy).

I do not believe the linked issues are in any way related. (They're error messages from the same validation function, but the details are actually quite a bit different.)

This error is produced when the updated npm is trying to replace an existing npm.cmd, but the existing npm.cmd points at an source code location other than the one npm is installed to. This is typically caused by conflicting system paths and npm configs.

I'm not familiar with nvs, but I would try npx npm-windows-upgrade and see if that doesn't get things into a state that npm knows how to deal with.

To diagnose this in npm, we'd need:

The output of: npm config get prefix (which is where the global install is going)

The contents the referenced .cmd file (so we can see where it's pointing to).

And finally, the npm-debug.log that was produced when the error was generated would likely be helpful.

[jdalton]

Thanks @iarna!

I'll try and get that info.
For reference here is an example AppVeyor job file (it's light on details at the moment).

Update:

Using npx npm-windows-upgrade --npm-version latest made it work for Node v10.4.1 and Node v6.14.3, but it still failed on Node v8.11.3 which defaults to npm v5.6.0.

Update:

In the fail case of Node v8.11.3:

npm config get prefix is:

C:\Users\appveyor\AppData\Roaming\nvs\node\8.11.3\x64

The contents of C:\Users\appveyor\AppData\Roaming\nvs\node\8.11.3\x64\npm.cmd is:

@ECHO OFF
SETLOCAL
SET "NODE_EXE=%~dp0\node.exe"
IF NOT EXIST "%NODE_EXE%" (
  SET "NODE_EXE=node"
)
SET "NPM_CLI_JS=%~dp0\node_modules\npm\bin\npm-cli.js"
FOR /F "delims=" %%F IN ('CALL "%NODE_EXE%" "%NPM_CLI_JS%" prefix -g') DO (
  SET "NPM_PREFIX_NPM_CLI_JS=%%F\node_modules\npm\bin\npm-cli.js"
)
IF EXIST "%NPM_PREFIX_NPM_CLI_JS%" (
  SET "NPM_CLI_JS=%NPM_PREFIX_NPM_CLI_JS%"
)
"%NODE_EXE%" "%NPM_CLI_JS%" %*

Update:

Getting the npm-debug.log may be a bit tricker but in lieu of that I have the debug logged output of npm-windows-upgrade:

npx: installed 87 in 8.369s
C:\Users\appveyor\AppData\Roaming\npm-cache\_npx\2780\node_modules\npm-windows-upgrade\bin\npm-windows-upgrade.js
npm-windows-upgrade v4.1.0
2018-06-17T17:41:23.377Z npm-windows-upgrade Powershell: Attempting to spawn PowerShell child
2018-06-17T17:41:23.690Z npm-windows-upgrade PowerShell: Stdout received: RemoteSigned
2018-06-17T17:41:23.728Z npm-windows-upgrade PowerShell: Resolving unrestricted (true)
2018-06-17T17:41:25.748Z npm-windows-upgrade PowerShell: Stdout received: C:\Users\appveyor\AppData\Roaming\nvs\node\8.11.3\x64\npm.cmd
2018-06-17T17:41:25.748Z npm-windows-upgrade PowerShell: Stdout received: 
2018-06-17T17:41:25.772Z npm-windows-upgrade PowerShell: _getPathFromPowerShell() resolving with: C:\Users\appveyor\AppData\Roaming\nvs\node\8.11.3\x64
2018-06-17T17:41:26.162Z npm-windows-upgrade PowerShell: _getPathFromNpm() resolving with: C:\Users\appveyor\AppData\Roaming\nvs\node\8.11.3\x64
2018-06-17T17:41:26.162Z npm-windows-upgrade Utils: isPathAccessible(): C:\Users\appveyor\AppData\Roaming\nvs\node\8.11.3\x64\node_modules\npm exists
2018-06-17T17:41:26.162Z npm-windows-upgrade Utils: isPathAccessible(): C:\Users\appveyor\AppData\Roaming\nvs\node\8.11.3\x64\node_modules\npm exists
Checked system for npm installation:
According to PowerShell: C:\Users\appveyor\AppData\Roaming\nvs\node\8.11.3\x64
According to npm:        C:\Users\appveyor\AppData\Roaming\nvs\node\8.11.3\x64
Decided that npm is installed in C:\Users\appveyor\AppData\Roaming\nvs\node\8.11.3\x64
2018-06-17T17:41:26.162Z npm-windows-upgrade Upgrader: Chosen npm path: C:\Users\appveyor\AppData\Roaming\nvs\node\8.11.3\x64
2018-06-17T17:41:26.163Z npm-windows-upgrade Starting upgrade
Upgrading npm... -2018-06-17T17:41:26.560Z npm-windows-upgrade PowerShell: Stdout received: Assuming npm in C:\Users\appveyor\AppData\Roaming\nvs\node\8.11.3\x64\node_modules\npm
Upgrading npm... \2018-06-17T17:41:26.604Z npm-windows-upgrade PowerShell: Stdout received: 
2018-06-17T17:41:26.604Z npm-windows-upgrade PowerShell: Stdout received: 
2018-06-17T17:41:26.606Z npm-windows-upgrade PowerShell: Stdout received:     Directory: C:\Users\appveyor\AppData\Local\Temp\1
2018-06-17T17:41:26.606Z npm-windows-upgrade PowerShell: Stdout received: 
2018-06-17T17:41:26.607Z npm-windows-upgrade PowerShell: Stdout received: Mode                LastWriteTime         Length Name                                                                  
2018-06-17T17:41:26.607Z npm-windows-upgrade PowerShell: Stdout received: ----                -------------         ------ ----                                                                  
2018-06-17T17:41:26.607Z npm-windows-upgrade PowerShell: Stdout received: d-----        6/17/2018  10:41 AM                npm_upgrade                                                           
Upgrading npm... \2018-06-17T17:41:27.343Z npm-windows-upgrade PowerShell: Stderr received: npm
2018-06-17T17:41:27.343Z npm-windows-upgrade PowerShell: Stderr received:  WARN invalid config loglevel="win"
Upgrading npm... \2018-06-17T17:41:44.826Z npm-windows-upgrade PowerShell: Stderr received: npm 
2018-06-17T17:41:44.827Z npm-windows-upgrade PowerShell: Stderr received: ERR! path C:\Users\appveyor\AppData\Roaming\nvs\node\8.11.3\x64\npm.cmd
npm ERR! code
2018-06-17T17:41:44.828Z npm-windows-upgrade PowerShell: Stderr received:  EEXIST
2018-06-17T17:41:44.829Z npm-windows-upgrade PowerShell: Stderr received: npm ERR!
2018-06-17T17:41:44.829Z npm-windows-upgrade PowerShell: Stderr received:  Refusing to delete C:\Users\appveyor\AppData\Roaming\nvs\node\8.11.3\x64\npm.cmd: is outside C:\Users\appveyor\AppData\Roaming\nvs\node\8.11.3\x64\node_modules\npm and not a link
npm ERR! File exists: C:\Users\appveyor\AppData\Roaming\nvs\node\8.11.3\x64\npm.cmd
npm ERR!
2018-06-17T17:41:44.829Z npm-windows-upgrade PowerShell: Stderr received:  Move it away, and try again.
2018-06-17T17:41:44.842Z npm-windows-upgrade PowerShell: Stderr received: 
2018-06-17T17:41:44.842Z npm-windows-upgrade PowerShell: Stderr received: npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\appveyor\AppData\Roaming\npm-cache\_logs\2018-06-17T17_41_44_837Z-debug.log
Upgrading npm... |2018-06-17T17:41:44.863Z npm-windows-upgrade PowerShell: Stdout received: All done!
2018-06-17T17:41:44.864Z npm-windows-upgrade PowerShell: Stdout received: 
2018-06-17T17:41:44.864Z npm-windows-upgrade PowerShell: Stdout received: 
2018-06-17T17:41:44.864Z npm-windows-upgrade PowerShell: Stdout received: 
Upgrading npm (fallback method)... |
You wanted to install npm 6.1.0, but the installed version is 5.6.0.
A common reason is an attempted "npm install npm" or "npm upgrade npm". As of today, the only solution is to completely uninstall and then reinstall Node.js. For a small tutorial, please see http://aka.ms/fix-npm-upgrade.
Please consider reporting your trouble to http://aka.ms/npm-issues.
Debug Information:
http_parser: 2.8.0 | node: 8.11.3 | v8: 6.2.414.54 | uv: 1.19.1 | zlib: 1.2.11 | ares: 1.10.1-DEV | modules: 57 | nghttp2: 1.32.0 | napi: 3 | openssl: 1.0.2o | icu: 60.1 | unicode: 10.0 | cldr: 32.0 | tz: 2017c | os: win32 x64 | OS Name: Microsoft Windows Server 2016 Datacenter
OS Version: 10.0.14393 N/A Build 14393

Update:

It looks like npm-windows-upgrade has a known issue upgrading npm 5.6.0:
felixrieseberg/npm-windows-upgrade#124

[iarna]

I meant the npm-debug.log from npm install -g npm, not npm-windows-upgrade, but...

I had entirely forgotten that Node ships its own npm wrapper, one that's different than the one that npm itself creates and THAT is probably the heart of the problem here.

Good news: I think I may be able to finally fix this (and also setup reproducers)!

Bad news: The fix will only help if you're already running the newer version of npm.

Inbetween news: npx npm@latest i -g npm ought to work, as weird as that is.

I'm thinking about ways we might be able to work around that though. Like making newer versions of npm not let npm itself install its binaries and instead do them in a postinstall script.

[iarna]

It is reproducible. It happens when the following are true:

1. You are using the Node.js version of npm.
2. You machine is configured to install global node modules in the same directory that Node.js is installed in.

npm/read-cmd-shim#4

The stock Windows Node.js install is supposed to include with it an npmrc that puts global modules in %APPDATA%\npm. This seems to work, unless Node.js is also installed there or the prefix is explicitly set elsewhere. (Given the other errors here, it seems like there are probably more scenarios where it doesn't work, but what they are isn't clear to me.)

The one shipped with npm on my machine is here:
C:\Program Files\nodejs\node_modules\npm\npmrc

[jdalton]

Inbetween news: npx npm@latest i -g npm ought to work, as weird as that is.

I just tried it, but it fails with the same Refusing to delete error.

npm/read-cmd-shim#4

Would a fix be back-portable to npm 5 so that a patch could land in a Node 8 patch-release?

[iarna]

Sorry for the confusion, I meant npx npm@latest i -g npm would work once I fix the bug.

The plan is that Node 8 will be getting npm 6, very soon, so it shouldn't be necessary to backport. But if need be, certainly.

[jdalton]

Sorry for the confusion, I meant npx npm@latest i -g npm would work once I fix the bug.

Oh rad!

The plan is that Node 8 will be getting npm 6, very soon, so it shouldn't be necessary to backport.

Awesome!

[joshk]

👋 Just wanted to check-in and see what the status of this was?

[DanielRuf]

Still the same: travis-ci/travis-ci#10245

[DanielRuf]

So it seems the only solution is to switch to Yarn for now.

[ljharb]

@DanielRuf you don’t ever need to do that :-) the current solution is to detect windows, run nvm install-latest-npm on everything else, and try to replicate that logic manually for windows (which is possible but painful, depending on which node versions you’re building on).

[jasongin]

@ljharb So nvm has a special command just to workaround this problem?

[ljharb]

Yes, with a bunch of logic to handle that you can’t alwsys jump from stock to latest in every node version.

[jasongin]

Actually this particular problem ("refusing to delete" the npm executable) seems to be a Windows-only issue. And I don't see anything that looks like a workaround for this in nvm install-latest-npm.

The issue of potentially having to hop between multiple versions to get to the latest is unrelated.

I'm still investigating. I'll try to figure out some kind of fix/workaround for this problem on Windows.

[ljharb]

Fair; nvm only supports windows on WSL which doesn’t seem to have this problem.

Note that the difficulty isn’t just having to hop around, it’s also knowing where to stop.

[jasongin]

The root cause appears to be that the npm.cmd file in the root of the Windows Node.js distributions is the wrong one. It is identical to one at node_modules/npm/bin/npm.cmd:

:: Created by npm, please don't edit manually.
@ECHO OFF

SETLOCAL

SET "NODE_EXE=%~dp0\node.exe"
IF NOT EXIST "%NODE_EXE%" (
  SET "NODE_EXE=node"
)

SET "NPM_CLI_JS=%~dp0\node_modules\npm\bin\npm-cli.js"
FOR /F "delims=" %%F IN ('CALL "%NODE_EXE%" "%NPM_CLI_JS%" prefix -g') DO (
  SET "NPM_PREFIX_NPM_CLI_JS=%%F\node_modules\npm\bin\npm-cli.js"
)
IF EXIST "%NPM_PREFIX_NPM_CLI_JS%" (
  SET "NPM_CLI_JS=%NPM_PREFIX_NPM_CLI_JS%"
)

"%NODE_EXE%" "%NPM_CLI_JS%" %*

However, if you manage to install some version of npm using npm install, then the npm.cmd at the root of the Node.js installation directory is different from the one under node_modules; it looks like this instead:

@IF EXIST "%~dp0\node.exe" (
  "%~dp0\node.exe"  "%~dp0\node_modules\npm\bin\npm-cli.js" %*
) ELSE (
  @SETLOCAL
  @SET PATHEXT=%PATHEXT:;.JS;=;%
  node  "%~dp0\node_modules\npm\bin\npm-cli.js" %*
)

If I install a fresh Node.js version, then replace the npm.cmd contents with the second file above (and do the same for npx.cmd, npm, and npx files), then npm install -g npm will successfully install, and any further installs of npm versions continue to work.

@iarna Does this make sense to you?

I can look into having nvs automatically fix these files after installing a version of node, so that npm can be successfully upgraded afterward.

[jasongin]

I've implemented a workaround in nvs at #109. When you nvs add a Node version, it will use npm's cmd-shim module to fix up the shims to be the way npm expects during later upgrades. I want to do a little more testing before I merge and release it in an nvs update. If anyone else would like to try, you can enlist in the npmshim branch.

[jasongin]

Fixed in [email protected]

Note the fix gets applied during nvs add, when the node version zip is extracted. So if you have previously added some node version(s) on the system, they still won't be able to successfully upgrade npm. You'll need to remove and re-add the node version first. (Of course this won't be an issue for CI environments that always start clean.)

[jdalton]

Thank you @jasongin!

It works great!