Enable package lock to reduce duration & CPU usage of "npm install" 🏎🌱

[bajtos]

Historically, we were avoiding package locks to ensure our CI builds are using the latest version of all dependencies, the same versions that our consumers would receive when running npm install in their projects.

Now that we have Greenkeeper enabled to detect new versions in our dependency tree and semi-automatically update our package.json & package-lock.json files, I feel it's safe to use package-lock feature.

This pull request enables package-locks in the entire monorepo in the first commit, and adds individual package-lock.json files in the second commit - I hope this will make the change easier to review. The third commit removes Node.js version 10.0.0 from the build matrix on Windows, because npm ci is not available there. (We cannot upgrade to a newer 10.x version - see #1431.)

Note: To fully leverage the power of package locks, the CI is using npm ci instead of npm install now. Similarly, lerna is configured to use npm ci when installing dependencies in individual monorepo packages.

As a result, the time to install dependencies was cut by half - from 1m44s to 0m52s on my machine. Not only the installation is faster, it's also burning less CPU cycles (and energy).

before

# clean all dependencies
$ npm run clean:lerna && rm -rf node_modules 
# measure how long it takes to install them
$ time npm i

real  1m43.710s
user  6m44.097s
sys   1m37.536s

after

# clean all dependencies
$ npm run clean:lerna && rm -rf node_modules 
# measure how long it takes to install them using "npm ci"
$ time npm ci

real  0m51.985s
user  3m56.915s
sys   1m18.982s

Checklist

👉 Read and sign the CLA (Contributor License Agreement) 👈

• npm test passes on your machine
• New tests added or existing tests modified to cover all changes
• Code conforms with the style guide
• API Documentation in code was updated
• Documentation in /docs/site was updated
• Affected artifact templates in packages/cli were updated
• Affected example projects in examples/* were updated

[bajtos]

I am intentionally keeping .npmrc and forcing package-lock on all loopback-next developers.

Personally, I have a global .npmrc file setting package-lock to false, because that's what I need when working on LB3 repositories. If there was no .npmrc file in loopback-next, then I would not be using package-lock functionality because of my global config.

IMO, it's best to not make any assumptions about the global npm configuration and explicitly tell npm that we want to use package-lock files.

[bajtos]

Bummer, Node.js 10.0.0 used on AppVeyor does not support npm ci. I think it's best to remove 10.0.0 from the build matrix on Windows. The version 10.0.0 is very outdated when compared to current 10.x LTS versions.

[bajtos]

@raymondfeng if the patch LGTY and there are no objections from other reviewers, then feel free to land it yourself. The sooner we speed up our CI builds, the better for us and the environment too.

[marioestradarosa]

LGTM 👍 .
@bajtos , as a side note, npm ci automatically do the rm -rf node_modules for us. According to its documentation. Also it mentions that it is faster because it skips certain user-oriented features.

[raymondfeng]

We need to try out a few things:

1. Make sure package-lock.json works nicely with lerna including our release process.
2. Figure out the impact of released @loopback/* modules with package-lock.json, such as:

• @loopback/[email protected] depends on @loopback/context@^2.0.0 (locked)
• Does @loopback/[email protected] pick up @loopback/[email protected] for LB4 apps if npm i is used?

[raymondfeng]

I'm seeing a diff in package-lock.json such as:

-          "resolved": "https://registry.npmjs.org/callsites/-/callsites-2.0.0.tgz",
+          "resolved": "http://registry.npmjs.org/callsites/-/callsites-2.0.0.tgz",

npm config get registry prints https://registry.npmjs.org/.

I had to do the following to resolve the problem:

1. npm cache clean --force
2. rm -rf node_modules
3. npm i

[b-admike]

LGTM 👏

[raymondfeng]

It's surprising that I don't see entries for @loopback/* dependencies in the generated package-lock.json. Is it by design as they are symbolically linked? If so, what's the impact for published modules?

[raymondfeng]

Figure out the impact of released @loopback/* modules with package-lock.json

I just realized that package-lock.json won't be published to npm.

• package-lock.json is never published to npm, whereas npm-shrinkwrap is by default
• package-lock.json files that are not in the top-level package are ignored, but shrinkwrap files belonging to dependencies are respected

Basically, package-lock.json is used to ensure repeatable build while shrinkwrap.json is used for repeatable installation/runtime.

[bajtos]

Basically, package-lock.json is used to ensure repeatable build while shrinkwrap.json is used for repeatable installation/runtime.

Exactly 👍