Vite: How to handle `.node` files

At Prosopo, we're making the next generation of CAPTCHA. Our tech stack is heavy on TypeScript, relying on Vite for bundling our code into a usable package.

Vite is a relatively new bundler for ESM based javascript projects. At Prosopo, we're using this as our default bundler in our typescript and nodejs oriented tech stack.

Recently we discovered a flaw in vite: the inability to handle .node files! We're using nodejs-polars, which is a package providing dataframe functionality. It's super helpful for our use case, but to remain speedy on all architectures the authors compile code to native binaries and provide various packages for different architectures (e.g. x64_86, arm64, etc).

These packages distribute .node files, which is the nodejs way of providing platform specific code. This sounds great, until vite enters the mix. Vite has no ability to handle native binaries whilst bundling, interpreting .node files as raw js and producing this error:

[config.vite.vite-plugin-close.js 17:40:50]  ERROR  ../../node_modules/nodejs-polars-linux-x64-gnu/nodejs-polars.linux-x64-gnu.node (1:0): Unexpected character '\u{7f}' (Note that you need plugins to import files that are not JavaScript)

Vite assumes that all files it encounters are js files, and attempts to use them as such. We need a plugin, according to the error message, to make this work. We tried:

• vite-plugin-native
• vavite-node-loader
• rollup-plugin-natives

But alas, none of these worked. They either interpreted the .node file correctly but did not copy it to the output directory, or flat out did nothing.

So we posted in vite's github for help only to discover there is no good solution, and we're not alone in experiencing this problem.

This was a major blocker for us, so we rolled our own (pun intended) rollup plugin (vite is built on top of rollup - in fact a rollup plugin is also a vite plugin!). In fact, we made two:

• nodejs-polars-native-file-plugin
• nodejs-polars-dirname-plugin

The dirname plugin is simply, any reference to __dirname is replaced with the esm equivalent: import.meta.url. We use esm everywhere, whereas nodejs-polars use cjs, so this is a quick fix for that.

The native file plugin is the crux of the solution. It's a rollup plugin which intercepts the loading of .node files. The .node files need to be known ahead of time, which is the only gotcha with this solution, though this could be automated.

The plugin detects when rollup attempts to load a .node file and returns a snippet of code instead. This snippet requires the .node file at runtime, expecting it to be beside the bundle itself.

// rewrite the code to import the .node file
if (path.basename(id) === path.basename(file)) {
    logger.debug(name, 'transform', id)
    // https://stackoverflow.com/questions/66378682/nodejs-loading-es-modules-and-native-addons-in-the-same-project
    // this makes the .node file load at runtime from an esm context. .node files aren't native to esm, so we have to create a custom require function to load them. The custom require function is equivalent to the require function in commonjs, thus allowing the .node file to be loaded.
    return `
        // create a custom require function to load .node files
        import { createRequire } from 'module';
        const customRequire = createRequire(import.meta.url)

        // load the .node file expecting it to be in the same directory as the output bundle
        const content = customRequire('./${file}')

        // export the content straight back out again
        export default content
        `
}

Vite is happy with this, the require is kicked down the road until runtime, so vite can compile the bundle as usual, forgetting about .node files in favour of the above js snippet.

The plugin then intercepts the generateBundle hook to detect when vite is done. This is when we want to copy the .node files to be adjacent to the output bundle, like so:

const file = path.basename(fileAbs)
// copy the .node file to the output directory
const out = `${outDir}/${file}`
const src = `${fileAbs}`
logger.debug(name, 'copy', src, 'to', out)
const nodeFile = fs.readFileSync(src)
fs.mkdirSync(path.dirname(out), { recursive: true })
fs.writeFileSync(out, nodeFile)

It's a very simple snippet which reads the .node file and writes it back out to a file adjacent to the bundle.

It's important that we do this after vite has built the bundle because part of the build step is to clear the output directory!

Now we can add the two plugins to the rollup plugins section of the vite config file:

plugins: [
    css(),
    wasm(),
    nodeResolve(),
    nodejsPolarsDirnamePlugin(logger),
    nodejsPolarsNativeFilePlugin(logger, nodeFiles, outDir),
],

And voila, our vite build works again, producing the bundle with the nodejs-polars .node file alongside.

In a perfect world, vite would examine the extension of the file (.node) and act accordingly. In most cases, .node files simply need to be made available to the output bundle by copying it into the same directory of the bundle. We definitely need better .node file handling in vite or a plugin of some sort. If/when we have time, we may extend our solution into a fully fledged plugin, but at the moment this works for us as a temporary fix and we look forward to the community building upon this to solve a shortcoming of vite.

This all seems pretty interesting to me...

Nice! Want to work with us? We're hiring! We've got a great team and are looking for talented engineers to join us.