Postgres support #5355
Postgres support #5355
Conversation
Fix insert return id
Fix blobs
Blob test fixes
pkg/sqlite/database_postgres.go
Outdated
| if disableForeignKeys { | ||
| _, err = conn.Exec("SET session_replication_role = replica;") | ||
|
|
||
| if err != nil { | ||
| return nil, fmt.Errorf("conn.Exec(): %w", err) | ||
| } | ||
| } | ||
| if !writable { | ||
| _, err = conn.Exec("SET default_transaction_read_only = ON;") | ||
|
|
||
| if err != nil { | ||
| return nil, fmt.Errorf("conn.Exec(): %w", err) | ||
| } | ||
| } |
There was a problem hiding this comment.
I would suggest not supporting "disable foreign key" and "read-only" in Postgres in the application layer.
They exist for SQLite because of limitations in the DB, such as some messiness with foreign keys. For connecting to Postgres as read-only, I'd strongly recommend that admins create a user with read-only permissions to the DB, rather than doing that in Stash
There was a problem hiding this comment.
I agree that this is ugly and should be removed.
But that would necessitate adding another connection string.
Im gonna have to think about it.
Or simply add "database_string_ro" 🤣
There was a problem hiding this comment.
If you absolutely need to tell Stash to make this RO, you could add a query stirng parameter to the connection string.
Something like ?readonly=1 to the URL... and then we remove it
There was a problem hiding this comment.
Im gonna let a reviewer weigh-in on this
Its slightly more performant, and slightly prettier
…in postgresql. Postgres is stricter and doesnt allow excess args.
So we initialize the specific database type in PostInit, which also makes it unavailable to the manager.
Fix a bug where phash distance didnt work. Fix a bug where i wrongly wrote getDBLike in a string in captionCriterionHandler.
…od_time. Add schema update 4/70 to postgres.
This caused an PostgreSQL error where "IN ()" was invalid SQL.
|
Im cutting my losses on this and migrating my own setup back to sqlite. |
|
Hi, Thanks for your significant effort on this, even though it hasn't been merged yet. I'm not super experienced in Go, but I am sure either myself or someone else in the community would like to pick this up eventually. |
If i remember right, the missing things are: Automated testing on git. That one bug that causes tests on postgres testing to sometimes fail. It would be nice to warn if the dll isnt installed I had to add filetype to the SQL schema for custom fields, but i haven't written the migration yet (SQLite remembers the type by itself, but postgres doesnt, so we need to deal with it). The SQL migration system relies on the schema version, but i simply reduced the number for postgres, meaning that a wrong migration will trigger when the first "code" migration version happens. Funny thing, we might need to expose the sql backend to clients since graphql exposes a execsql, which might have sql backend dependent code. There is a part of the SQL code that executes both a count and a fetch, meaning 2 sql transactions pr request, which seems unnecessary and makes postgres slower, you should investigate if this could be done differently. And after you have done all that, and whatever i forgot, you need to convince the maintainers that they should take on all this technical dept. |
Hello, this is an attempt at implementing postgres support.
If you want to test this branch, you can add these changes to your config.yml:
In the ui setup you must set the database path as the connection string to use this.
Also if you want anonymise to work a backup path has to be set
There are alot of changes, highlights:
lastInsertId is a SQLite specific feature, other databases need you to specify "returning id". Which our version of SQLite supports, but goqu doesnt mark it as supported, so we update to a dev version.
Call regexp as a function, as postgresql doesnt support named operators.
Postgresql C/CGo extension to support missing functionality (basename, phash_distance, regexp):
https://github.com/NodudeWasTaken/stash_pge
https://github.com/NodudeWasTaken/stash_sqlite_to_pgsql
Some particularly egregious changes are:
This will be the admins responsibility
We should probably add a postgres test
PGSQL_TEST='postgresql://username:password@localhost/dbname_for_testing' go test -tags "sqlite_stat4 sqlite_math_functions integration" ./...Some particularly lackings are:
I added testmode which mitigated it a little, but it still happens sometimes.
It happens to file_test's for whatever reason.
But i fear that it may cause insertion issues, maybe, since the thing that updates it doesnt clamp the value, and it unlike sqlite actually returns an error on overflow.
To look at (help me):
You can use dbeaver for testing the database, its nice for browsing various databases like sqlite and postgresql.
To support disable foreign keys we do (if anybody knows a better way, let me know):
"SET session_replication_role = replica;"
which requires extra postgres user permissions, just a note.
This is mainly a request for comments, or help, mainly help finishing this.
Fixes #3892