Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Loader .node: copy don't copy files into outdir #3582

Closed
1bye opened this issue Jan 6, 2024 · 4 comments
Closed

Loader .node: copy don't copy files into outdir #3582

1bye opened this issue Jan 6, 2024 · 4 comments

Comments

@1bye
Copy link

1bye commented Jan 6, 2024

Ubuntu 22.04
Used runtimes: Bun 1.0.21v / Node.js 18

Hello!

I'm attempting to bundle the faiss-node package with esbuild. Unfortunately, after building, it doesn't copy the files (.node) from faiss-node:

Copy code
error: Could not locate the bindings file. Tried:
  /home/mrbye/Documents/GitHub/package-builder/build/faiss-node.node
  /home/mrbye/Documents/GitHub/package-builder/build/Debug/faiss-node.node
  /home/mrbye/Documents/GitHub/package-builder/build/Release/faiss-node.node
  /home/mrbye/Documents/GitHub/package-builder/out/Debug/faiss-node.node
  /home/mrbye/Documents/GitHub/package-builder/Debug/faiss-node.node
  /home/mrbye/Documents/GitHub/package-builder/out/Release/faiss-node.node
  /home/mrbye/Documents/GitHub/package-builder/Release/faiss-node.node
  /home/mrbye/Documents/GitHub/package-builder/build/default/faiss-node.node
  /home/mrbye/Documents/GitHub/package-builder/compiled/20.8.0/linux/x64/faiss-node.node
  /home/mrbye/Documents/GitHub/package-builder/addon-build/release/install-root/faiss-node.node
  /home/mrbye/Documents/GitHub/package-builder/addon-build/debug/install-root/faiss-node.node
  /home/mrbye/Documents/GitHub/package-builder/addon-build/default/install-root/faiss-node.node
  /home/mrbye/Documents/GitHub/package-builder/lib/binding/node-v115-linux-x64/faiss-node.node
      at B (/home/mrbye/Documents/GitHub/package-builder/dir/index.js:1:2455)
      at /home/mrbye/Documents/GitHub/package-builder/dir/index.js:3:706
      at /home/mrbye/Documents/GitHub/package-builder/dir/index.js:1:286
      at /home/mrbye/Documents/GitHub/package-builder/dir/index.js:3:1208

To reproduce the issue, you should install faiss-node and copy the settings and code from here
The output dir structure is:

dir
└── index.js
# nothing here

Also I've tried your plugin native-node-modules but it don't work :(
Thanks

EDIT:

My package.json

{
  "name": "package-builder",
  "module": "index.ts",
  "type": "module",
  "devDependencies": {
    "@types/bun": "latest"
  },
  "peerDependencies": {
    "typescript": "^5.0.0"
  },
  "dependencies": {
    "esbuild": "^0.19.11",
    "faiss-node": "^0.5.1"
  }
}
@evanw
Copy link
Owner

evanw commented Jan 6, 2024

This has nothing to do with the file loader. The .node file is never loaded in the first place. You can tell because removing --loader:.node=copy doesn't cause the build to fail. You can see the problem with --log-level=debug:

● [DEBUG] Indirect calls to "require" will not be bundled [indirect-require]

    node_modules/bindings/bindings.js:94:8:
      94 │       : require;
         ╵         ~~~~~~~

You are trying to bundle something that is not designed to be bundled. The solution is to not do that. Quoting from esbuild's getting started instructions for node:

You also may not want to bundle your dependencies with esbuild. There are many node-specific features that esbuild doesn't support while bundling such as __dirname, import.meta.url, fs.readFileSync, and *.node native binary modules. You can exclude all of your dependencies from the bundle by setting packages to external

Indeed, using --packages=external does fix your code sample. Sometimes libraries just aren't bundler-friendly. It's reasonable for a package containing platform-specific binary executables to not support bundling. For example, esbuild's JS API itself also doesn't support being bundled for a similar reason.

@1bye
Copy link
Author

1bye commented Jan 6, 2024

Thank you. However, could you also assist me with the following issue?

I am attempting to move the build folder to the outdir if it exists in the package. However, when I run the bundled file, it attempts to read from the root folder (I only need resolve issue with importing from root to outdir). Here is my code:

import { build } from "esbuild";
import fs from 'fs-extra';

const nativeNodeModulesPlugin = {
  name: 'native-node-modules',
  setup(build) {
    // If a ".node" file is imported within a module in the "file" namespace, resolve 
    // it to an absolute path and put it into the "node-file" virtual namespace.
    build.onResolve({ filter: /\.node$/g, namespace: 'file' }, args => {
    	console.log(args)
    	return ({
      path: require.resolve(args.path, { paths: [args.resolveDir] }),
      namespace: 'node-file',
    })})

    // Files in the "node-file" virtual namespace call "require()" on the
    // path from esbuild of the ".node" file in the output directory.
    build.onLoad({ filter: /.*/, namespace: 'node-file' }, args => ({
      contents: `
        import path from ${JSON.stringify(args.path)}
        try { module.exports = require(path) }
        catch {}
      `,
    }))

    // If a ".node" file is imported within a module in the "node-file" namespace, put
    // it in the "file" namespace where esbuild's default loading behavior will handle
    // it. It is already an absolute path since we resolved it to one above.
    build.onResolve({ filter: /\.node$/, namespace: 'node-file' }, args => ({
      path: args.path,
      namespace: 'file',
    }))

    // Tell esbuild's default loading behavior to use the "file" loader for
    // these ".node" files.
    let opts = build.initialOptions
    opts.loader = opts.loader || {}
    opts.loader['.node'] = 'file'
  },
}
/* Plugin to move build folder from packages to outdir  */
async function copyBuildFolderPlugin() {
  const pathModule = await import('path');

  return {
    name: 'copy-build-folder',
    setup(build) {
      build.onResolve({ filter: /./ }, (args) => {
        const buildFolderPath = pathModule.join(`${process.cwd()}/node_modules/${args.path}`, 'build');
        if (fs.existsSync(buildFolderPath) && fs.statSync(buildFolderPath).isDirectory()) {

          fs.copySync(buildFolderPath, `${process.cwd()}/dir/build`);
        }

        return null;
      });
    },
  };
}

await build({
	entryPoints: ['./source/index.js'],
	minify: false,
	bundle: true,
	platform: 'node',
	plugins: [nativeNodeModulesPlugin, await copyBuildFolderPlugin()],
	outdir: 'dir'
});

Log from run

120 |             throw e;
121 |           }
122 |         }
123 |       }
124 |       err = new Error(
                  ^
error: Could not locate the bindings file. Tried:
  /home/mrbye/Documents/GitHub/package-builder/build/faiss-node.node
  /home/mrbye/Documents/GitHub/package-builder/build/Debug/faiss-node.node
  /home/mrbye/Documents/GitHub/package-builder/build/Release/faiss-node.node
  /home/mrbye/Documents/GitHub/package-builder/out/Debug/faiss-node.node
  /home/mrbye/Documents/GitHub/package-builder/Debug/faiss-node.node
  /home/mrbye/Documents/GitHub/package-builder/out/Release/faiss-node.node
  /home/mrbye/Documents/GitHub/package-builder/Release/faiss-node.node
  /home/mrbye/Documents/GitHub/package-builder/build/default/faiss-node.node
  /home/mrbye/Documents/GitHub/package-builder/compiled/20.8.0/linux/x64/faiss-node.node
  /home/mrbye/Documents/GitHub/package-builder/addon-build/release/install-root/faiss-node.node
  /home/mrbye/Documents/GitHub/package-builder/addon-build/debug/install-root/faiss-node.node
  /home/mrbye/Documents/GitHub/package-builder/addon-build/default/install-root/faiss-node.node
  /home/mrbye/Documents/GitHub/package-builder/lib/binding/node-v115-linux-x64/faiss-node.node
      at bindings (/home/mrbye/Documents/GitHub/package-builder/dir/index.js:124:13)
      at node_modules/faiss-node/lib/index.js (/home/mrbye/Documents/GitHub/package-builder/dir/index.js:184:17)
      at __require (/home/mrbye/Documents/GitHub/package-builder/dir/index.js:4:21)
      at /home/mrbye/Documents/GitHub/package-builder/dir/index.js:203:55

Here is the structure of folder:

dir
├── build
│   └── Release
│       ├── faiss-node.node
│       ├── libfaiss.a
│       ├── libgfortran.so.5
│       ├── libgomp.so.1
│       ├── libopenblas.so.0
│       └── libquadmath.so.0
└── index.js

Thanks :)

@evanw
Copy link
Owner

evanw commented Jan 6, 2024

Sorry, I don't understand. Why are you still trying to bundle this library? You can get it to work by dropping all of the plugin stuff and using packages: 'external' instead. This library is not designed to be bundled.

@1bye
Copy link
Author

1bye commented Jan 6, 2024

Sorry and thank you. I'm closing this issue.

@1bye 1bye closed this as completed Jan 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants