Bun binding problems: dyld[XXX]: missing symbol called and error: could not locate the bindings file causing unkillable process!

[buzzy]

Running the migrator.make() completely freezez the process and I am not even able to kill it using "kill -9"!

Diffing schema:
  • Capturing pragmas...dyld[80832]: missing symbol called

I am running this on an empty database + a very simple schema:

PRAGMA foreign_keys = off;
BEGIN TRANSACTION;

-- Table: sessions
CREATE TABLE "sessions" (
  id blob PRIMARY KEY NOT NULL,
  user blob NOT NULL,
  lastChecked timestamp NOT NULL
) WITHOUT ROWID;

-- Index: user
CREATE INDEX user ON sessions (user);

COMMIT TRANSACTION;
PRAGMA foreign_keys = on;

I am running this using Bun on a M2 Mac. Seems it's not compatible at all. Most likely this library (or a dependency) does not support non-x86 cpus yet. You might want to look into that.

[SanderGi]

Thanks for making me aware of this issue! I have been running things on an M-series Mac as well but using Node.js (compiled for arm64). Does it work for you with Node.js? I will look into the Bun compatibility to see if that might be the cause

[SanderGi]

Another note is, you don’t need the transaction in your schema. The migrator takes care of applying your schema within a transaction and temporarily disabling the foreign keys if they are enabled in your schema so you can remove those parts and just leave the one “PRAGMA foreign_keys = on;” at the top

[buzzy]

Thanks! I just exported the schema using SQLiteStudio without editing it as a test. Node might work. Haven't tested it as I don't want to mix runtimes in my environment. I also write in typescript, so running it in node would mean adding extra layers like transpilers etc, which I don't need with Bun. My CI-environment runs "normal" x86, so I am testing if it works fine there, but it would be nice to be able to test things locally on the Mac :)

[buzzy]

Nope, actually does not work on Bun on x86 either. My CI creates a production ready Bun-binary with all code and Bun-runtime in one file. Running that file gives error that package.json is missing. I do not get that problem if I don't import the "sqlite-auto-migrator" library, so something is definitely not playing nice with Bun.

Basically I am including the current .sql file inside my project (Bun binary file) and on first run, I have a "--update-database" flag, that compares the embedded .sql-file with the current database and performs the changes needed. That run fails with Bun.

[SanderGi]

Thanks for taking the time to investigate with x86 Bun. I just confirmed your schema works on M1 Mac with Node.js but the migrator breaks with Bun. I have pin-pointed the issue to be with running node-sqlite3 with Bun. Specifically, the Database.each() method does not bind correctly with Bun. The error can be recreated with:

import sqlite3 from 'sqlite3';

const db = new sqlite3.Database('data.db');
db.each('SELECT * FROM sessions', (err, row) => {
    console.log(row);
});

Which yields dyld[65381]: missing symbol called.

I have filed an issue with node-sqlite3. In the mean time, I have pushed a temporary patch to sqlite-auto-migrator. Let me know if upgrading sqlite-auto-migrator to version 1.1.4 does not fix things for you. It seems to work with my Bun version (1.1.29)

[buzzy]

Wow, that was quick work! I will test it now. Btw, Bun uses a own built-in sqlite library that is much faster than node-sqlite3, so I am not sure if the node-sqlite3 team will care :)

[buzzy]

Not sure if I am doing something wrong, but this is what I get:

[SanderGi]

Weird. What is your version of Bun? If it is an older version, you might need to add sqlite3 as a trusted dependency (oven-sh/bun#5187)

[buzzy]

I tried it with 1.1.20 and latest 1.1.29. Same issue.
I even tried clearing the Bun cache.
Looks like it searches for the bindings in the wrong path:

I also tried removing my node_modules and bun.lockb and re-downloaded all libs.

[SanderGi]

Hmm, not sure why it searches in the wrong path for you but not for me. Did you install bun with curl -fsSL https://bun.sh/install | bash? Does the process.arch say 'arm64' and what is your macOS version? I'll try to see if I can reproduce your issue by clearing my cache

[buzzy]

Yes, I installed the official way with curl. Yes, it's a arm64 binary. The x86 Bun binary does not even start. Not even if I would run it with Rosetta.

[buzzy]

My test-script is very basic:

[buzzy]

Ah, for some reason, Bun is trying to get sqlite3 from it's cache instead of node_modules and in the cache, there is no sqlite3 binary:

[buzzy]

If I copy the sqlite3 binary from node_modules to cache, I get error, so I guess something is wrong with Bun.

[SanderGi]

It might be worth submitting a report to the Bun team. How did you copy things over to the cache? This is how my working cache looks:
cache.zip

What does your bunfig.toml look like? Perhaps disabling the cache will fix things:

[install.cache]
# the directory to use for the cache
dir = "~/.bun/install/cache"

# when true, don't load from the global cache.
# Bun may still write to node_modules/.cache
disable = true

# when true, always resolve the latest versions from the registry
disableManifest = true

[buzzy]

Your cache looks exactly like mine. You can clearly see that there is no node_sqlite3.node binary in there. So if it works for you, it is definitely not using your cache, but the binary that exists in your node_modules dir. I do not use a bunfig.toml at all, so I use all defaults. I can try to disable the cache, but as the config says, it will still write to the .cache dir, so it sounds more like it just switches to another location.

For my test, I simply copied the entire contents from node_modules/sqlite3/* to the bun cache dir.

[SanderGi]

I see. I think the better way to get the binary into the cache directory is to cd ~/.bun/install/cache/[email protected]@@@1 and then run bun install (should be equivalent to copy pasting the installed node_modules/sqlite3 or setting the trustedDependencies but perhaps it makes a difference). I don't usually use Bun so I cannot explain why my installation finds the correct binary in the node_modules and yours does not, esp. since we both have no bunfigs. Perhaps you can try forcing it to use the node_modules one by providing a relative path? I'll keep trying to reproduce the issue on my M1 Mac

[SanderGi]

Update - @buzzy I reproduced your error message (from before messing with the missing binary in the cache) by running Bun from the parent directory:

And then reproduced the segmentation fault by running Bun globally with bunx:

As well as by adding the binary to the cache with cd ~/.bun/install/cache/[email protected]@@@1 && bun install and running from the parent directory:

A similar (but more informative) error happens with Node.js (since it does not default to running the global cached package and instead complains there is no local package found when running this way):

And no error occurs running using the global Node.js package (since it installs and binds the binary correctly):

Now that I can somewhat reproduce the error, I have found a potential work-around. Ideally Bun would work with binaries in the cache, but since it doesn't, you can make it aware of the node_modules binary by setting its working directory relative to the file being executed to guarantee it points at the folder with the package.json and node_modules:

import path from 'path';

process.chdir(path.join(__dirname, '..'));

import { Migrator } from 'sqlite-auto-migrator';

const migrator = new Migrator();

await migrator.make();
await migrator.migrate();

Something similar might work for you. Let me know if it does.

[buzzy]

Thank you! That actually solved the problem! So I guess Bun has a bug. I haven't tested it on "bun build" yet though. Hopefully it will still work when Bun bundles everything to one binary.

One question; Do I really need that "migration" table in the database? Is there a way to turn that off? As my CI simply has a sql-file with the latest schema and the CI always compares the "production" database with the schema and changes it, what exactly would I need the migration table for? I take a backup of the database before, so I don't really need any revert functionality.

[buzzy]

I have issues with it getting stuck and never finishes. It only happens randomly though.

When that happens, the tables are left in a locked state:

Also, is there a way to prevent it from creating the "migrations" table in the database and "migrations" directory? I do not need those as I am not checking in any versions or rollback features in git. I simply want it to compare the .sql-file with the database-file and create a "migration-file" in RAM and then execute it without leaving any traces.

[SanderGi]

Great! Happy to hear the workaround worked for you as well. Let me know whether it works with "bun build" when you test it

These are excellent inquiries. I have split them up into separate issues to make the conversation easier to follow. This issue will remain for the bun binary binding, #2 will be for the deadlock, #3 for the "migrations" table and directory issue

[buzzy]

I'm afraid that I get another issue regarding the sqlite3 library when using "bun build". When running the resulting binary, I get this:

You should be able to reproduce it by running this to generate a binary:

bun build --compile --minify --sourcemap ./src/server.ts --outfile ./src/server.exe

I have reported it to the Bun team as well: oven-sh/bun#14301

[SanderGi]

Thanks for trying this out and reporting with bun. After looking into it, I don't think Bun build with --compile supports the necessary options to bundle in the node-sqlite3 binaries. The workaround I've settled for is to use the built-in bun:sqlite when running via bun and use node-sqlite3 with node/deno. You should be able to try it out via the latest commit on the main branch bun install github:SanderGi/sqlite-auto-migrator. Let me know if this fixes the error for you and I can publish a package version to npm

[buzzy]

Seems you are using some post-install-script that is not trusted by Bun by default. That script also seems to require "husky"?

[buzzy]

Seems like my source .sql file can't be found by sqlite-auto-migrator when it's embedded inside the Bun binary:

I am still importing it like this and it works fine if I run the .ts file manually without building:

[buzzy]

btw, the deadlock seems to be fixed by switching to Buns own sqlite lib! So all works fine now, except reading the .sql file from within the Bun binary after using "bun build". Thank you so much for your efforts! I did not even expect that you would make special cases for Bun, that's why I did not even ask for it before :)

[SanderGi]

Seems you are using some post-install-script that is not trusted by Bun by default. That script also seems to require "husky"?

Don't worry about the prepare script, it is only supposed to run for local development, not when installing the package but installing from Github makes Bun think it should run it. I use Husky to automatically run tests, linting, and formatting before commits

[SanderGi]

Seems like my source .sql file can't be found by sqlite-auto-migrator when it's embedded inside the Bun binary:

I am still importing it like this and it works fine if I run the .ts file manually without building:

It seems Bun does not support the Node.js file functions with $bunfs:

import sessions from "./schema1.sql" with { type: "file"};

const file = Bun.file(sessions);
console.log(await file.exists()); // true, successfully finds the file

import { accessSync, constants } from "node:fs";

accessSync("./schema1.sql", constants.R_OK); // Works, finds the file

accessSync(sessions, constants.R_OK) // Errors, does not find the file

As a quick workaround, you can keep the schema file separate from the compiled binary and pass in its local path directly to the migrator. I'll also go through and make exceptions for Bun in the code so it uses Bun.file when run via Bun

[SanderGi]

btw, the deadlock seems to be fixed by switching to Buns own sqlite lib! So all works fine now, except reading the .sql file from within the Bun binary after using "bun build". Thank you so much for your efforts! I did not even expect that you would make special cases for Bun, that's why I did not even ask for it before :)

Happy to hear that the deadlock is fixed for you. Thank you for your help debugging these issues! Would you be okay having your Github username credited in the readme for your help with Bun compatibility, deadlocks, and testing the declarative migrations?

Ideally we would not need special cases for Bun, but it seems it is the only way to make it work (perhaps in the future, Bun will be more compatible with Node.js)

[buzzy]

Sure, it would be an honor!

I would prefer to not need to deploy any additional files, except the binary, so I will simply wait for your change to Bun builtin fs lib. Really awesome work so far!

[SanderGi]

Makes sense. You should be able to use bun embedded files for the schemas using the latest commit now. Lmk if it works for you

[buzzy]

I just performed my first 100% successful database upgrade through my CI system! Awesome product :)

[SanderGi]

Congrats! I'm glad we could get it to work in the end. I have published the Bun compatibility fixes to npm (version 1.2.0) and will close this issue. Don't hesitate to open another issue if you run into problems. Thanks for all your help and good luck with your projects

[lutfi-haslab]

I'm afraid that I get another issue regarding the sqlite3 library when using "bun build". When running the resulting binary, I get this:

You should be able to reproduce it by running this to generate a binary:

bun build --compile --minify --sourcemap ./src/server.ts --outfile ./src/server.exe

I have reported it to the Bun team as well: oven-sh/bun#14301

sorry to ask this after issue close, i have same problem with building into binary, how you handle this bindings not found?

[SanderGi]

sorry to ask this after issue close, i have same problem with building into binary, how you handle this bindings not found?

@lutfi-haslab, looks like your error is not related to sqlite-auto-migrator but to another library you are using, hnswlib. They might not support bun bindings. I suggest posting an issue in their repo. If you can reproduce the error using just sqlite-auto-migrator or have an example that shows this error only occurs when hnswlib is used in conjunction with sqlite-auto-migrator, feel free to open a new issue here with example code to reproduce the error. If hnswlib is not interested in supporting bun bindings, you might be able to use a workaround similar to what we tried for sqlite-auto-migrator before deciding to use the builtin bun sqlite, but again, they would best be able to advice so definitely ask in their repo.