Skip to content

[BUG] Platform-specific optional dependencies not being included in package-lock.json when reinstalling with node_modules present #4828

Open
@JustinChristensen

Description

Is there an existing issue for this?

  • I have searched the existing issues

This issue exists in the latest npm version

  • I am using the latest npm

Current Behavior

[user@host:foo] $ npm -v
8.8.0
[user@host:foo] $ node
Welcome to Node.js v16.14.2.
Type ".help" for more information.
> process.arch
'arm64'

I'm working on a team that utilizes a mix of x64-based and m1-based macs, and has CI build processes that uses musl. We're seeing that npm is skipping platform-specific optional dependencies for packages such as @swc/core as a result of the package-lock.json file being generated without all of them included. In our case, this then causes linting to throw an exception, because one of our eslint plugins depends on @swc, which depends on having the platform specific @swc package also installed.

There seems to be at least two stages of cause to this. Firstly, when installing @swc/core from a clean slate working directory npm generates a package-lock.json with all of the optional dependencies for @swc/core listed:

[user@host:foo] $ npm install @swc/core
[user@host:foo] $ grep 'node_modules/@swc/core-*' package-lock.json
    "node_modules/@swc/core": {
    "node_modules/@swc/core-android-arm-eabi": {
    "node_modules/@swc/core-android-arm64": {
    "node_modules/@swc/core-darwin-arm64": {
    "node_modules/@swc/core-darwin-x64": {
    "node_modules/@swc/core-freebsd-x64": {
    "node_modules/@swc/core-linux-arm-gnueabihf": {
    "node_modules/@swc/core-linux-arm64-gnu": {
    "node_modules/@swc/core-linux-arm64-musl": {
    "node_modules/@swc/core-linux-x64-gnu": {
    "node_modules/@swc/core-linux-x64-musl": {
    "node_modules/@swc/core-win32-arm64-msvc": {
    "node_modules/@swc/core-win32-ia32-msvc": {
    "node_modules/@swc/core-win32-x64-msvc": {

And it only installs the platform specific package:

[user@host:foo] $ ls -l node_modules/@swc/
total 0
drwxr-xr-x  22 user  staff  704 Apr 29 15:39 core
drwxr-xr-x   6 user  staff  192 Apr 29 15:39 core-darwin-arm64

If I then remove my package-lock.json, leave my node_modules directory as-is, and then reinstall, I get:

[user@host:foo] $ rm -rf package-lock.json
[user@host:foo] $ npm install
[user@host:foo] $ grep 'node_modules/@swc/core-*' package-lock.json
    "node_modules/@swc/core": {
    "node_modules/@swc/core-darwin-arm64": {

That is, it then generates a package-lock.json with only the platform-specific dependency that was installed on this machine, and not with the other optional dependencies that should also be listed.

If you delete both node_modules AND package-lock.json, and then re-run npm install, it generates the correct lockfile with all of those optional dependencies listed.

The problem is that then, If the package-lock.json with the missing optional platform-specific dependencies gets checked into git and an x64 user pulls it down, or vice-versa, npm fails to detect that your platform's optional dependencies are missing in the lockfile and just silently skips installing the platform-specific dependency. For example, when I've got a package-lock.json that only contains the x64 @swc package because of the above problem (generated by my coworker on his x64 machine):

[user@host:foo] $ node
Welcome to Node.js v16.14.2.
Type ".help" for more information.
> process.arch
'arm64'
>
[user@host:foo] $ grep 'node_modules/@swc/core-*' package-lock.json
    "node_modules/@swc/core": {
    "node_modules/@swc/core-darwin-x64": {
[user@host:foo] $ ls
package-lock.json package.json

And I then install:

[user@host:foo] $ npm install
added 1 package in 341ms

1 package is looking for funding
  run `npm fund` for details
[user@host:foo] $ ls node_modules/@swc/
core

You can see that it fails to install the arm64 dependency or warn me in any way that the package-lock.json is missing my platform's dependency.

So yeah, two problems:

  1. npm is generating an inconsistent package-lock.json when node_modules has your platform-specific dependency installed.
  2. When installing from this inconsistent package-lock.json, npm fails to try to correct the problem by comparing the optional dependencies to what's listed upstream

Expected Behavior

  1. npm should preserve the full set of platform-specific optional deps for a package like @swc when rebuilding package-lock.json from an existing node_modules tree
  2. npm install should warn if the package-lock.json becomes inconsistent because of the first case

Steps To Reproduce

See above.

Environment

  • npm: 8.8.0
  • Node.js:
  • OS Name: OSX
  • System Model Name: Macbook Pro
[user@host:foo] $ npm -v
8.8.0
[user@host:foo] $ node -v
v16.14.2
[user@host:foo] $ uname -a
Darwin host.foo.com. 21.3.0 Darwin Kernel Version 21.3.0: Wed Jan  5 21:37:58 PST 2022; root:xnu-8019.80.24~20/RELEASE_ARM64_T8101 arm64
[user@host] $ npm config ls
; "user" config from /Users/user/.npmrc
; node bin location = /Users/user/.nvm/versions/node/v16.14.2/bin/node
; node version = v16.14.2
; npm local prefix = /Users/user/Development/foo
; npm version = 8.8.0
; cwd = /Users/user/Development/foo
; HOME = /Users/user
; Run `npm config ls -l` to show all defaults.

Metadata

Assignees

No one assigned

    Labels

    Bugthing that needs fixingNeeds Triageneeds review for next stepsRelease 8.xwork is associated with a specific npm 8 release

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions