installing missing npm modules repeatedly fails and then succeeds with no change #273

arrowplum · 2017-10-12T09:00:23Z

installing packages often fail and must be reinstalled repeatedly before they are actually installed without errors. n4js-cli package failed > 10 times over two days before it finally succeeded.

Example error log:

0 info it worked if it ends with ok
1 verbose cli [ '/Users/joem/.nvm/versions/node/v8.1.0/bin/node',
1 verbose cli '/Users/joem/.nvm/versions/node/v8.1.0/bin/npm',
1 verbose cli 'install',
1 verbose cli 'n4js-cli',
1 verbose cli '--save' ]
2 info using [email protected]
3 info using [email protected]
4 verbose npm-session 615e9e6e62bafe94
5 silly install loadCurrentTree
6 silly install readLocalPackageData
7 http fetch GET 200 http://nexus3-aws.corp.numberfour.eu/repository/npm-public/n4js-cli 169ms
8 http fetch GET 304 http://nexus3-aws.corp.numberfour.eu/repository/npm-public/n4js-cli/-/n4js-cli-0.6.0-alpha.30dc7fef.tgz 1063ms
9 silly fetchPackageMetaData error for n4js-cli@latest unexpected end of file
10 verbose stack Error: unexpected end of file
10 verbose stack at Gunzip.zlibOnError (zlib.js:152:15)
11 verbose cwd /Volumes/devel/werkschaft/.metadata/.plugins/org.eclipse.n4js.external.libraries/.n4npm
12 verbose Darwin 16.7.0
13 verbose argv "/Users/joem/.nvm/versions/node/v8.1.0/bin/node" "/Users/joem/.nvm/versions/node/v8.1.0/bin/npm" "install" "n4js-cli" "--save"
14 verbose node v8.1.0
15 verbose npm v5.3.0
16 error code Z_BUF_ERROR
17 error errno -5
18 error unexpected end of file
19 verbose exit [ -5, true ]

yoosiba · 2017-10-12T12:28:31Z

how did you invoke installation? this does not look like log from IDE, nor N4JSC.
Could you provide more details about the steps you take to when failures are observed?
e.g. command line call, or actions in the IDE.

arrowplum · 2017-10-12T12:32:25Z

Sure thing @yoosiba. Here are the steps I took:

Import project with some npm dependencies (in this case eu.numberfour.mangelhaft.runner.node).
Find errors in Manifest (in this case n4js-cli.
Hover over error to get the "install npm" message.
Click on it.
See errors in alert and in log with a pointer to the more detailed log.
Open detailed log.
Paste it into a github issue on the eclipse n4js project.

yoosiba · 2017-10-12T12:52:16Z

that package is quite large. Although it works for me:

[12-10-2017 02:36:21] Fetching 'n4js-cli' package... [package 1 of 1]
+ [email protected]
added 1 package in 89.864s
[12-10-2017 02:37:51] Package 'n4js-cli' has been successfully fetched.

similar issue is discussed here npm/npm#11993

maybe it is combination of :

what is in that package specifically
your node / npm versions
your npm cache

Could you refer to the linked ticket and try solutions with npm cache / shrinkwrap helps?

If not than this might be issues with public / private packages, as stated in npm/npm#11993 (comment)

But the "publicly registered package depends on a privately registered package" case didn't work. Changing the registries to both be either private or public worked, though.

arrowplum · 2017-10-12T13:03:54Z

Please note that I am using npm5 I have no control over what parameters get called since it comes from the ide right click and I have cleaned my cache already today. So what workaround are you talking about exactly @yoosiba

arrowplum · 2017-10-12T13:05:36Z

Maybe if the command line n4js compiler could come in at something less than 50 megs it would help.

yoosiba · 2017-10-12T13:13:21Z

It would be nice if the size compiler would be smaller. If it could be smaller, it would be smaller. Unfortunately it is not possible now.

I was thinking about the manually doing:

$ npm cache clean
$ rm -rf node_modules
$ rm npm-shrinkwrap.json
$ npm shrinkwrap

in your external libraries folder.

If that does not help it looks like the issue with the private / public packages, in which case you should ask @dbo
npm/npm#11993 (comment)

But the "publicly registered package depends on a privately registered package" case didn't work. Changing the registries to both be either private or public worked, though.

arrowplum · 2017-10-12T13:28:29Z

Unfortunately if it could be smaller, it would be smaller.

I am not sure I agree with you there @yoosiba I look at other compilers, for example typescript, and they are an order of magnitude smaller eg https://registry.npmjs.org/typescript/-/typescript-2.6.0-dev.20170929.tgz

As for the manual steps, that would be fine, but can we also install the packages manually?

arrowplum · 2017-10-12T13:34:04Z

In general, though, I doubt this is the private/public issue since it works on the 11th try. Also I do not see an npm-shrinkwrap.json. Also this seems like a lot of guessing.

arrowplum · 2017-10-12T13:50:59Z

Where even are the packages that get installed in this way? They are not in the local repository.

I really think this is an internal issue to the ide and you should look at it.

yoosiba · 2017-10-12T13:51:45Z

Manual installation can work but requires extra steps
1) in your (...)/workspace/.metadata/.plugins/org.eclipse.n4js.external.libraries/.n4npm
do cleanup and manual installation

2) in .../.n4npm/node_modules/n4js-cli
add manifest.n4mf

ProjectId: n4js-cli
ProjectType: library
ProjectVersion: 0.6.0-alpha.30dc7fef
VendorId: npm
VendorName: "npm"
Output: "."
MainModule: "index"
ModuleLoader: commonjs
Sources {
	external {
		"."
	}
}

3) in IDE do "Run maintenance Actions" -> "Reload npm libraries from disk"

Then it should work.

I have hard time reproducing this issue, I have reinstalled n4js-cli few times from IDE and it worked every time.

[12-10-2017 03:44:46] ================================================================
[12-10-2017 03:44:46] Installing  npm packages : n4js-cli
[12-10-2017 03:44:46] ================================================================
[12-10-2017 03:44:46] ----------------------------------------------------------------
[12-10-2017 03:44:46] Installing packages... [step 1 of 4]
[12-10-2017 03:44:46] Fetching 'n4js-cli' package... [package 1 of 1]
npm notice created a lockfile as package-lock.json. You should commit this file.
+ [email protected]
added 1 package in 1.227s
[12-10-2017 03:44:48] Package 'n4js-cli' has been successfully fetched.
[12-10-2017 03:44:48] ----------------------------------------------------------------
[12-10-2017 03:44:48] Calculating dependency changes... [step 2 of 4]
[12-10-2017 03:44:48] Dependency changes have been successfully calculated.
[12-10-2017 03:44:48] ----------------------------------------------------------------
[12-10-2017 03:44:48] Adapting npm package structure to N4JS project structure... [step 3 of 4]
[12-10-2017 03:44:49] Packages structures has been adapted to N4JS project structure.
[12-10-2017 03:44:49] ----------------------------------------------------------------
[12-10-2017 03:44:49] Registering new projects... [step 4 of 4]
[12-10-2017 03:44:49] Platform is running.
[12-10-2017 03:44:49] Call org.eclipse.n4js.ui.external.EclipseExternalLibraryWorkspace@3606eee9 to register [file:/Users/jakub.siberski/n4p/devtools/opr/Eclipse.app/Contents/MacOS/workspace2/.metadata/.plugins/org.eclipse.n4js.external.libraries/.n4npm/node_modules/n4js-cli/] and de-register []
[12-10-2017 03:44:49] Finished registering projects.
[12-10-2017 03:44:49] Successfully finished installing  packages.
[12-10-2017 03:44:49] ================================================================
[12-10-2017 03:45:16] ================================================================
[12-10-2017 03:45:16] Installing  npm packages : n4js-cli
[12-10-2017 03:45:16] ================================================================
[12-10-2017 03:45:16] ----------------------------------------------------------------
[12-10-2017 03:45:16] Installing packages... [step 1 of 4]
[12-10-2017 03:45:16] Fetching 'n4js-cli' package... [package 1 of 1]
npm notice created a lockfile as package-lock.json. You should commit this file.
+ [email protected]
added 1 package in 1.163s
[12-10-2017 03:45:18] Package 'n4js-cli' has been successfully fetched.
[12-10-2017 03:45:18] ----------------------------------------------------------------
[12-10-2017 03:45:18] Calculating dependency changes... [step 2 of 4]
[12-10-2017 03:45:18] Dependency changes have been successfully calculated.
[12-10-2017 03:45:18] ----------------------------------------------------------------
[12-10-2017 03:45:18] Adapting npm package structure to N4JS project structure... [step 3 of 4]
[12-10-2017 03:45:19] Packages structures has been adapted to N4JS project structure.
[12-10-2017 03:45:19] ----------------------------------------------------------------
[12-10-2017 03:45:19] Registering new projects... [step 4 of 4]
[12-10-2017 03:45:19] Platform is running.
[12-10-2017 03:45:19] Call org.eclipse.n4js.ui.external.EclipseExternalLibraryWorkspace@3606eee9 to register [file:/Users/jakub.siberski/n4p/devtools/opr/Eclipse.app/Contents/MacOS/workspace2/.metadata/.plugins/org.eclipse.n4js.external.libraries/.n4npm/node_modules/n4js-cli/] and de-register []
[12-10-2017 03:45:19] Finished registering projects.
[12-10-2017 03:45:19] Successfully finished installing  packages.
[12-10-2017 03:45:19] ================================================================
[12-10-2017 03:45:39] ================================================================
[12-10-2017 03:45:39] Installing  npm packages : n4js-cli
[12-10-2017 03:45:39] ================================================================
[12-10-2017 03:45:39] ----------------------------------------------------------------
[12-10-2017 03:45:39] Installing packages... [step 1 of 4]
[12-10-2017 03:45:39] Fetching 'n4js-cli' package... [package 1 of 1]
npm notice created a lockfile as package-lock.json. You should commit this file.
+ [email protected]
added 1 package in 1.214s
[12-10-2017 03:45:41] Package 'n4js-cli' has been successfully fetched.
[12-10-2017 03:45:41] ----------------------------------------------------------------
[12-10-2017 03:45:41] Calculating dependency changes... [step 2 of 4]
[12-10-2017 03:45:41] Dependency changes have been successfully calculated.
[12-10-2017 03:45:41] ----------------------------------------------------------------
[12-10-2017 03:45:41] Adapting npm package structure to N4JS project structure... [step 3 of 4]
[12-10-2017 03:45:42] Packages structures has been adapted to N4JS project structure.
[12-10-2017 03:45:42] ----------------------------------------------------------------
[12-10-2017 03:45:42] Registering new projects... [step 4 of 4]
[12-10-2017 03:45:42] Platform is running.
[12-10-2017 03:45:42] Call org.eclipse.n4js.ui.external.EclipseExternalLibraryWorkspace@3606eee9 to register [file:/Users/jakub.siberski/n4p/devtools/opr/Eclipse.app/Contents/MacOS/workspace2/.metadata/.plugins/org.eclipse.n4js.external.libraries/.n4npm/node_modules/n4js-cli/] and de-register []
[12-10-2017 03:45:42] Finished registering projects.
[12-10-2017 03:45:42] Successfully finished installing  packages.
[12-10-2017 03:45:42] ================================================================
[12-10-2017 03:45:56] ================================================================
[12-10-2017 03:45:56] Installing  npm packages : n4js-cli
[12-10-2017 03:45:56] ================================================================
[12-10-2017 03:45:56] ----------------------------------------------------------------
[12-10-2017 03:45:56] Installing packages... [step 1 of 4]
[12-10-2017 03:45:56] Fetching 'n4js-cli' package... [package 1 of 1]
npm notice created a lockfile as package-lock.json. You should commit this file.
+ [email protected]
added 1 package in 1.228s
[12-10-2017 03:45:58] Package 'n4js-cli' has been successfully fetched.
[12-10-2017 03:45:58] ----------------------------------------------------------------
[12-10-2017 03:45:58] Calculating dependency changes... [step 2 of 4]
[12-10-2017 03:45:58] Dependency changes have been successfully calculated.
[12-10-2017 03:45:58] ----------------------------------------------------------------
[12-10-2017 03:45:58] Adapting npm package structure to N4JS project structure... [step 3 of 4]
[12-10-2017 03:45:59] Packages structures has been adapted to N4JS project structure.
[12-10-2017 03:45:59] ----------------------------------------------------------------
[12-10-2017 03:45:59] Registering new projects... [step 4 of 4]
[12-10-2017 03:45:59] Platform is running.
[12-10-2017 03:45:59] Call org.eclipse.n4js.ui.external.EclipseExternalLibraryWorkspace@3606eee9 to register [file:/Users/jakub.siberski/n4p/devtools/opr/Eclipse.app/Contents/MacOS/workspace2/.metadata/.plugins/org.eclipse.n4js.external.libraries/.n4npm/node_modules/n4js-cli/] and de-register []
[12-10-2017 03:45:59] Finished registering projects.
[12-10-2017 03:45:59] Successfully finished installing  packages.
[12-10-2017 03:45:59] ================================================================
[12-10-2017 03:46:14] ================================================================
[12-10-2017 03:46:14] Installing  npm packages : n4js-cli
[12-10-2017 03:46:14] ================================================================
[12-10-2017 03:46:14] ----------------------------------------------------------------
[12-10-2017 03:46:14] Installing packages... [step 1 of 4]
[12-10-2017 03:46:14] Fetching 'n4js-cli' package... [package 1 of 1]
npm notice created a lockfile as package-lock.json. You should commit this file.
+ [email protected]
added 1 package in 1.524s
[12-10-2017 03:46:16] Package 'n4js-cli' has been successfully fetched.
[12-10-2017 03:46:16] ----------------------------------------------------------------
[12-10-2017 03:46:16] Calculating dependency changes... [step 2 of 4]
[12-10-2017 03:46:16] Dependency changes have been successfully calculated.
[12-10-2017 03:46:16] ----------------------------------------------------------------
[12-10-2017 03:46:16] Adapting npm package structure to N4JS project structure... [step 3 of 4]
[12-10-2017 03:46:16] Packages structures has been adapted to N4JS project structure.
[12-10-2017 03:46:16] ----------------------------------------------------------------
[12-10-2017 03:46:16] Registering new projects... [step 4 of 4]
[12-10-2017 03:46:16] Platform is running.
[12-10-2017 03:46:16] Call org.eclipse.n4js.ui.external.EclipseExternalLibraryWorkspace@3606eee9 to register [file:/Users/jakub.siberski/n4p/devtools/opr/Eclipse.app/Contents/MacOS/workspace2/.metadata/.plugins/org.eclipse.n4js.external.libraries/.n4npm/node_modules/n4js-cli/] and de-register []
[12-10-2017 03:46:17] Finished registering projects.
[12-10-2017 03:46:17] Successfully finished installing  packages.
[12-10-2017 03:46:17] ================================================================

P.S.
"I look at other compilers, and they are an order of magnitude smaller". Not sure if that is the way to look at this. Linux distributions are different in size, yet they are all "linux".

yoosiba · 2017-10-12T16:45:12Z

So, I have hard time reproducing that. I am not saying that there is no issue. Since you experience it, there is some issue somewhere, it is just I can't reproduce it. Needs more investigation / input.

(example of installing n4js-cli from IDE for eu.numberfour.mangelhaft.runner.node)

yoosiba · 2017-10-13T08:02:01Z

I can reproduce this issue by turning off internet connection while package is downloading.

But this is not internal IDE issue, the same happens in the terminal when using npm directly

When your machine drops connection and npm throws error, IDE is handling it correctly.

Size of the n4js-cli gives bigger window for network errors to happen.
You can have the same errors for other packages when working on unstable internet connection.

n4js-cli size aside (which should be addressed separately),
I don't see how IDE should handle npm errors caused by machine dropping network connection.

kduske-n4 · 2018-01-05T15:26:28Z

I'm having a similar problem where I get a timeout error after 240 seconds when installing n4js-cli. Maybe it's possible to increase the timeout limit?

!ENTRY org.apache.log4j 4 0 2018-01-05 16:22:26.867
!MESSAGE org.eclipse.n4js.utils.process.ProcessExecutor - Process didn't finish after 240 SECONDS

!ENTRY org.apache.log4j 4 0 2018-01-05 16:22:26.892
!MESSAGE org.eclipse.n4js.external.NpmLogger - Error while installing npm package.

!STACK 0
java.lang.Exception: Error while installing npm package.

!ENTRY org.eclipse.n4js 4 0 2018-01-05 16:22:28.061
!MESSAGE Status of installing npm dependencies.
!SUBENTRY 1 org.eclipse.n4js 4 0 2018-01-05 16:22:28.061
!MESSAGE Error while installing npm package.

!STACK 0
java.lang.Exception: Error while installing npm package.

kduske-n4 · 2018-01-05T15:28:31Z

I can observe that my downstream is quite slow when the IDE is trying to download the package, so I suppose that there is either a network problem or something is happening with our Nexus. Either way, a longer time out may be useful.

grrowl · 2018-01-11T06:26:04Z

If you haven't tried yet, update to Node 8.9.x or 9.x: nodejs/node#13391

mmews-n4 · 2018-10-23T14:24:08Z

All the installation methods of npms were completely reworked in the meantime. This issue would not happen anymore. Closing this as solved/invalid.

mor-n4 added the bug label Jan 23, 2018

mmews-n4 closed this as completed Oct 23, 2018

mmews-n4 added invalid and removed bug labels Oct 23, 2018

mor-n4 mentioned this issue Jan 23, 2019

Discovery: As a smith I want to reconsider handling of "empty" intersection types #1224

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

installing missing npm modules repeatedly fails and then succeeds with no change #273

installing missing npm modules repeatedly fails and then succeeds with no change #273

arrowplum commented Oct 12, 2017

yoosiba commented Oct 12, 2017

arrowplum commented Oct 12, 2017

yoosiba commented Oct 12, 2017

arrowplum commented Oct 12, 2017

arrowplum commented Oct 12, 2017

yoosiba commented Oct 12, 2017 •

edited

Loading

arrowplum commented Oct 12, 2017

arrowplum commented Oct 12, 2017

arrowplum commented Oct 12, 2017

yoosiba commented Oct 12, 2017 •

edited

Loading

yoosiba commented Oct 12, 2017

yoosiba commented Oct 13, 2017

kduske-n4 commented Jan 5, 2018

kduske-n4 commented Jan 5, 2018

grrowl commented Jan 11, 2018

mmews-n4 commented Oct 23, 2018

installing missing npm modules repeatedly fails and then succeeds with no change #273

installing missing npm modules repeatedly fails and then succeeds with no change #273

Comments

arrowplum commented Oct 12, 2017

yoosiba commented Oct 12, 2017

arrowplum commented Oct 12, 2017

yoosiba commented Oct 12, 2017

arrowplum commented Oct 12, 2017

arrowplum commented Oct 12, 2017

yoosiba commented Oct 12, 2017 • edited Loading

arrowplum commented Oct 12, 2017

arrowplum commented Oct 12, 2017

arrowplum commented Oct 12, 2017

yoosiba commented Oct 12, 2017 • edited Loading

yoosiba commented Oct 12, 2017

yoosiba commented Oct 13, 2017

kduske-n4 commented Jan 5, 2018

kduske-n4 commented Jan 5, 2018

grrowl commented Jan 11, 2018

mmews-n4 commented Oct 23, 2018

yoosiba commented Oct 12, 2017 •

edited

Loading

yoosiba commented Oct 12, 2017 •

edited

Loading