[BUG] Platform-specific optional dependencies not being included in package-lock.json
when reinstalling with node_modules
present #4828
Description
Is there an existing issue for this?
- I have searched the existing issues
This issue exists in the latest npm version
- I am using the latest npm
Current Behavior
[user@host:foo] $ npm -v
8.8.0
[user@host:foo] $ node
Welcome to Node.js v16.14.2.
Type ".help" for more information.
> process.arch
'arm64'
I'm working on a team that utilizes a mix of x64-based and m1-based macs, and has CI build processes that uses musl. We're seeing that npm
is skipping platform-specific optional dependencies for packages such as @swc/core
as a result of the package-lock.json
file being generated without all of them included. In our case, this then causes linting to throw an exception, because one of our eslint plugins depends on @swc, which depends on having the platform specific @swc package also installed.
There seems to be at least two stages of cause to this. Firstly, when installing @swc/core
from a clean slate working directory npm
generates a package-lock.json
with all of the optional dependencies for @swc/core
listed:
[user@host:foo] $ npm install @swc/core
[user@host:foo] $ grep 'node_modules/@swc/core-*' package-lock.json
"node_modules/@swc/core": {
"node_modules/@swc/core-android-arm-eabi": {
"node_modules/@swc/core-android-arm64": {
"node_modules/@swc/core-darwin-arm64": {
"node_modules/@swc/core-darwin-x64": {
"node_modules/@swc/core-freebsd-x64": {
"node_modules/@swc/core-linux-arm-gnueabihf": {
"node_modules/@swc/core-linux-arm64-gnu": {
"node_modules/@swc/core-linux-arm64-musl": {
"node_modules/@swc/core-linux-x64-gnu": {
"node_modules/@swc/core-linux-x64-musl": {
"node_modules/@swc/core-win32-arm64-msvc": {
"node_modules/@swc/core-win32-ia32-msvc": {
"node_modules/@swc/core-win32-x64-msvc": {
And it only installs the platform specific package:
[user@host:foo] $ ls -l node_modules/@swc/
total 0
drwxr-xr-x 22 user staff 704 Apr 29 15:39 core
drwxr-xr-x 6 user staff 192 Apr 29 15:39 core-darwin-arm64
If I then remove my package-lock.json
, leave my node_modules
directory as-is, and then reinstall, I get:
[user@host:foo] $ rm -rf package-lock.json
[user@host:foo] $ npm install
[user@host:foo] $ grep 'node_modules/@swc/core-*' package-lock.json
"node_modules/@swc/core": {
"node_modules/@swc/core-darwin-arm64": {
That is, it then generates a package-lock.json with only the platform-specific dependency that was installed on this machine, and not with the other optional dependencies that should also be listed.
If you delete both node_modules
AND package-lock.json
, and then re-run npm install
, it generates the correct lockfile with all of those optional dependencies listed.
The problem is that then, If the package-lock.json
with the missing optional platform-specific dependencies gets checked into git and an x64 user pulls it down, or vice-versa, npm
fails to detect that your platform's optional dependencies are missing in the lockfile and just silently skips installing the platform-specific dependency. For example, when I've got a package-lock.json that only contains the x64 @swc package because of the above problem (generated by my coworker on his x64 machine):
[user@host:foo] $ node
Welcome to Node.js v16.14.2.
Type ".help" for more information.
> process.arch
'arm64'
>
[user@host:foo] $ grep 'node_modules/@swc/core-*' package-lock.json
"node_modules/@swc/core": {
"node_modules/@swc/core-darwin-x64": {
[user@host:foo] $ ls
package-lock.json package.json
And I then install:
[user@host:foo] $ npm install
added 1 package in 341ms
1 package is looking for funding
run `npm fund` for details
[user@host:foo] $ ls node_modules/@swc/
core
You can see that it fails to install the arm64 dependency or warn me in any way that the package-lock.json
is missing my platform's dependency.
So yeah, two problems:
- npm is generating an inconsistent package-lock.json when node_modules has your platform-specific dependency installed.
- When installing from this inconsistent package-lock.json, npm fails to try to correct the problem by comparing the optional dependencies to what's listed upstream
Expected Behavior
npm
should preserve the full set of platform-specific optional deps for a package like @swc when rebuildingpackage-lock.json
from an existingnode_modules
treenpm install
should warn if thepackage-lock.json
becomes inconsistent because of the first case
Steps To Reproduce
See above.
Environment
- npm: 8.8.0
- Node.js:
- OS Name: OSX
- System Model Name: Macbook Pro
[user@host:foo] $ npm -v
8.8.0
[user@host:foo] $ node -v
v16.14.2
[user@host:foo] $ uname -a
Darwin host.foo.com. 21.3.0 Darwin Kernel Version 21.3.0: Wed Jan 5 21:37:58 PST 2022; root:xnu-8019.80.24~20/RELEASE_ARM64_T8101 arm64
[user@host] $ npm config ls
; "user" config from /Users/user/.npmrc
; node bin location = /Users/user/.nvm/versions/node/v16.14.2/bin/node
; node version = v16.14.2
; npm local prefix = /Users/user/Development/foo
; npm version = 8.8.0
; cwd = /Users/user/Development/foo
; HOME = /Users/user
; Run `npm config ls -l` to show all defaults.