Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Postgres support #5355

Closed
Closed
Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
100 commits
Select commit Hold shift + click to select a range
a1189b4
Retry postgres
NodudeWasTaken Oct 2, 2024
a45658f
Better init
NodudeWasTaken Oct 3, 2024
876ad84
Query fixes
NodudeWasTaken Oct 3, 2024
2ac9768
Migration changes
NodudeWasTaken Oct 3, 2024
9f42dd1
Remove prepared
NodudeWasTaken Oct 3, 2024
5790ec1
SQL Boolean fixes
NodudeWasTaken Oct 4, 2024
61c3605
PGSQL Fingerprints fix
NodudeWasTaken Oct 4, 2024
17c5036
PGSQL fix duplicate ids in the wake of removing distinctIDs select
NodudeWasTaken Oct 4, 2024
08750b3
Undo remove prepared
NodudeWasTaken Oct 4, 2024
9d72fdd
Change rebind func
NodudeWasTaken Oct 4, 2024
b743bf5
fix prepared for postgresql
NodudeWasTaken Oct 4, 2024
0332276
nicer init
NodudeWasTaken Oct 4, 2024
58603ca
Dont fuck up the sorting
NodudeWasTaken Oct 6, 2024
0cdc191
regexp changes
NodudeWasTaken Oct 8, 2024
ae1759c
PGSQL migration fix
NodudeWasTaken Oct 8, 2024
bae1463
PGSQL Revert go.mod changes (or try to)
NodudeWasTaken Oct 8, 2024
b94e7b7
Add the lock back
NodudeWasTaken Oct 9, 2024
0ce1226
Fix linter issues
NodudeWasTaken Oct 9, 2024
f810b3b
Close being nil fix
NodudeWasTaken Oct 9, 2024
aaaad79
fix anonymiser
NodudeWasTaken Oct 9, 2024
c54540b
Readbility nits around getDBBoolean
NodudeWasTaken Oct 9, 2024
1d50d45
Infer db type
NodudeWasTaken Oct 9, 2024
16e0e36
Missed a readability nit
NodudeWasTaken Oct 9, 2024
9b55dcd
Another nit
NodudeWasTaken Oct 9, 2024
21f6d16
No more psql lock
NodudeWasTaken Oct 9, 2024
8e4ce3a
Multiple lines in pgsql schema nit
NodudeWasTaken Oct 9, 2024
5403431
Use a single "database" configuration as connection string too
its-josh4 Oct 9, 2024
1520e47
Fix values criterion for pgsql.
NodudeWasTaken Oct 10, 2024
36c8b8d
Merge pull request #1 from its-josh4/connstring
NodudeWasTaken Oct 10, 2024
edc3c7d
Better pgsql open
NodudeWasTaken Oct 10, 2024
817da3e
Fix DISTINCT aggregates must have exactly one argument
NodudeWasTaken Oct 10, 2024
2a79c63
Keep disableForeignKeys
NodudeWasTaken Oct 10, 2024
3959e4f
Better DB abstraction, still ugly though
NodudeWasTaken Oct 11, 2024
688e365
Added distinctIDs back and obey group by rules (if postgresql)
NodudeWasTaken Oct 13, 2024
2260ab5
Some small cosmetic query changes
NodudeWasTaken Oct 14, 2024
56fdfff
postgres backup system
NodudeWasTaken Oct 14, 2024
878c8e8
Cleanup PGSQL test
NodudeWasTaken Oct 14, 2024
088dd14
groupby
NodudeWasTaken Oct 14, 2024
235b2bc
Distinct on
NodudeWasTaken Oct 14, 2024
b019f3f
Fix images pagination bug
NodudeWasTaken Oct 14, 2024
4bf5d2a
Git fix
NodudeWasTaken Oct 14, 2024
f3c830b
pagination and sort
NodudeWasTaken Oct 14, 2024
1c537b8
Revert: postgres backup system
NodudeWasTaken Oct 15, 2024
992f481
Fix some tests for pgsql
NodudeWasTaken Oct 15, 2024
390059d
more test fixes for pgsql
NodudeWasTaken Oct 15, 2024
9cd1e5d
for this compilation fix for tests for pgsql
NodudeWasTaken Oct 15, 2024
2edc214
tests for sqlite bugfix
NodudeWasTaken Oct 15, 2024
8fa2b38
pgsql fix tests (suffering)
NodudeWasTaken Oct 17, 2024
aa4f257
pgsql fix more tests
NodudeWasTaken Oct 18, 2024
df48fc7
pgsql more test fixes
NodudeWasTaken Oct 19, 2024
45bbb76
pgsql: more work on passing tests
NodudeWasTaken Oct 20, 2024
2a9b8db
small remove
NodudeWasTaken Oct 20, 2024
db0cbc2
pgsql remove distinct (again), and use group by instead
NodudeWasTaken Oct 21, 2024
1ab44eb
pgsql missing some groupby's (some like title are unnecessary, but it…
NodudeWasTaken Oct 21, 2024
7540304
pgsql fix more sql bugs
NodudeWasTaken Oct 21, 2024
b50070c
missed
NodudeWasTaken Oct 21, 2024
45f2757
in postgresql errors cause the transaction to cancel, fix this by usi…
NodudeWasTaken Oct 22, 2024
9654813
rollback wrapper
NodudeWasTaken Oct 22, 2024
a27710b
IsLocked for pgsql (i think)
NodudeWasTaken Oct 22, 2024
3487545
Fix ismissing for pgsql
NodudeWasTaken Oct 24, 2024
2239acd
pgsql bugfixes
NodudeWasTaken Oct 24, 2024
49cea00
pgsql more bugfixes
NodudeWasTaken Oct 25, 2024
2c0c951
Fix case insensitive LIKE for postgres
NodudeWasTaken Oct 26, 2024
4e34a50
Fingerprints fix for postgres
NodudeWasTaken Oct 28, 2024
ac7456c
Fix rowid missing in pgsql
NodudeWasTaken Oct 28, 2024
cb01336
pgsql fix image test (sorting by nothing).
NodudeWasTaken Oct 28, 2024
d681d77
tag queryforautotag forgot to replace like
NodudeWasTaken Oct 28, 2024
1cd2c0d
fingerprint correct make the linter happy
NodudeWasTaken Oct 28, 2024
26ae603
Some commit removed the sqlite dialect init
NodudeWasTaken Oct 29, 2024
39e9b3e
Remove custom sqlite dialect in favor of upgrading goqu
NodudeWasTaken Oct 29, 2024
6fb7bcf
Ensure order on findfilter to prevent unstable db return order
NodudeWasTaken Oct 29, 2024
05f4d85
made TagStore Merge compatible with pgsql
NodudeWasTaken Oct 30, 2024
cdd192d
Made anonymise cross-db compatible
NodudeWasTaken Nov 1, 2024
a406807
Merge https://github.com/stashapp/stash into postgres-support-2
NodudeWasTaken Nov 1, 2024
9f757ca
fix misspelling of occurred
NodudeWasTaken Nov 1, 2024
2436f6f
Added migrations for pgsql
NodudeWasTaken Nov 1, 2024
2b55779
dbConfig as string
NodudeWasTaken Nov 1, 2024
3f457e3
change most varchar to text
NodudeWasTaken Nov 1, 2024
d3771bf
performance fix for pgsql
NodudeWasTaken Nov 1, 2024
3f1da4d
Optimize better
NodudeWasTaken Nov 1, 2024
393ad59
Replace ilike with LOWER(?) LIKE LOWER(?).
NodudeWasTaken Nov 2, 2024
192f757
Fix blob insert bug
NodudeWasTaken Nov 6, 2024
4d729b0
Fix bug in tags that made finding ancestors and descendants not work …
NodudeWasTaken Nov 7, 2024
23ba774
Setup now works
NodudeWasTaken Nov 8, 2024
cd0f2c5
Fix a bug where multiple scene files caused duplicate ids.
NodudeWasTaken Nov 11, 2024
7dd8643
Escape primary keyword for sqlite
NodudeWasTaken Nov 11, 2024
9b3ed54
Merge branch 'develop' into postgres-support-2
DogmaDragon Nov 12, 2024
0a7a340
Merge https://github.com/stashapp/stash into postgres-support-2
NodudeWasTaken Nov 20, 2024
cedcd28
Add timezone to timestamps for postgres to correct for 1-2 hour off m…
NodudeWasTaken Nov 21, 2024
d203f0c
Add github action (i hope)
NodudeWasTaken Jan 14, 2025
023dfb4
Merge branch 'develop' of https://github.com/stashapp/stash into post…
NodudeWasTaken Jan 14, 2025
f630f83
PG custom fields
NodudeWasTaken Jan 22, 2025
1970831
github action update
NodudeWasTaken Jan 22, 2025
d760fc7
Increment db version
NodudeWasTaken Jan 22, 2025
8e57a43
Include pg headers
NodudeWasTaken Jan 22, 2025
4e18923
Use AGE function for pgsql age filter
NodudeWasTaken Jan 23, 2025
afafff5
Fix use UTC time for tests
NodudeWasTaken Jan 23, 2025
564d00b
Update action
NodudeWasTaken Jan 23, 2025
1bd5fb9
Performance enhancement for pgsql
NodudeWasTaken Jan 31, 2025
fa83af9
Adds a check to FindMany such that we dont search for nothing.
NodudeWasTaken Feb 7, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 0 additions & 5 deletions internal/manager/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,6 @@ const (
DefaultMaxSessionAge = 60 * 60 * 1 // 1 hours

Database = "database"
DatabaseType = "database_type"
DatabaseConnectionString = "database_string"

Exclude = "exclude"
Expand Down Expand Up @@ -697,10 +696,6 @@ func (i *Config) GetDatabasePath() string {
return i.getString(Database)
}

func (i *Config) GetDatabaseType() string {
return i.getString(DatabaseType)
}

func (i *Config) GetDatabaseConnectionString() string {
return i.getString(DatabaseConnectionString)
}
Expand Down
9 changes: 3 additions & 6 deletions internal/manager/init.go
Original file line number Diff line number Diff line change
Expand Up @@ -38,13 +38,10 @@ func Initialize(cfg *config.Config, l *log.Logger) (*Manager, error) {
var db *sqlite.Database

{
var dbType = sqlite.DatabaseType(strings.ToUpper(cfg.GetDatabaseType()))
var dbType = sqlite.SqliteBackend

switch dbType {
case sqlite.SqliteBackend, sqlite.PostgresBackend:
// Valid case
default:
dbType = sqlite.SqliteBackend
if strings.HasPrefix(strings.ToUpper(cfg.GetDatabaseConnectionString()), string(sqlite.PostgresBackend)) {
dbType = sqlite.PostgresBackend
}

switch dbType {
Expand Down
16 changes: 0 additions & 16 deletions pkg/sqlite/database.go
Original file line number Diff line number Diff line change
Expand Up @@ -253,22 +253,6 @@ func (db *Database) Open() error {
return nil
}

// lock locks the database for writing. This method will block until the lock is acquired.
func (db *Database) lock() {
db.lockChan <- struct{}{}
}

// unlock unlocks the database
func (db *Database) unlock() {
// will block the caller if the lock is not held, so check first
select {
case <-db.lockChan:
return
default:
panic("database is not locked")
}
}

func (db *Database) Close() error {
db.lock()
defer db.unlock()
Expand Down
5 changes: 4 additions & 1 deletion pkg/sqlite/database_postgres.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@ func NewPostgresDatabase(dbConnector string) *Database {

db := &PostgresDB{
storeRepository: newDatabase(),
lockChan: make(chan struct{}, 1),
dbConfig: dbConnector,
}
db.dbInterface = db
Expand All @@ -27,6 +26,10 @@ func NewPostgresDatabase(dbConnector string) *Database {
return (*Database)(db)
}

// Does nothing
func (db *PostgresDB) lock() {}
func (db *PostgresDB) unlock() {}

func (db *PostgresDB) DatabaseType() DatabaseType {
return PostgresBackend
}
Expand Down
16 changes: 16 additions & 0 deletions pkg/sqlite/database_sqlite.go
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,22 @@ func NewSQLiteDatabase(dbPath string) *Database {
return (*Database)(db)
}

// lock locks the database for writing. This method will block until the lock is acquired.
func (db *SQLiteDB) lock() {
db.lockChan <- struct{}{}
}

// unlock unlocks the database
func (db *SQLiteDB) unlock() {
// will block the caller if the lock is not held, so check first
select {
case <-db.lockChan:
return
default:
panic("database is not locked")
}
}

func (db *SQLiteDB) DatabaseType() DatabaseType {
return SqliteBackend
}
Expand Down
6 changes: 3 additions & 3 deletions pkg/sqlite/gallery_filter.go
Original file line number Diff line number Diff line change
Expand Up @@ -378,12 +378,12 @@ func (qb *galleryFilterHandler) performerFavoriteCriterionHandler(performerfavor
if *performerfavorite {
// contains at least one favorite
f.addLeftJoin("performers", "", "performers.id = performers_galleries.performer_id")
f.addWhere(fmt.Sprintf("performers.favorite = %s", getDBBoolean(true)))
f.addWhere("performers.favorite = " + getDBBoolean(true))
} else {
// contains zero favorites
f.addLeftJoin(fmt.Sprintf(`(SELECT performers_galleries.gallery_id as id FROM performers_galleries
f.addLeftJoin(`(SELECT performers_galleries.gallery_id as id FROM performers_galleries
JOIN performers ON performers.id = performers_galleries.performer_id
GROUP BY performers_galleries.gallery_id HAVING SUM(performers.favorite) = %s)`, getDBBoolean(false)), "nofaves", "galleries.id = nofaves.id")
GROUP BY performers_galleries.gallery_id HAVING SUM(performers.favorite) = `+getDBBoolean(false)+")", "nofaves", "galleries.id = nofaves.id")
f.addWhere("performers_galleries.gallery_id IS NULL OR nofaves.id IS NOT NULL")
}
}
Expand Down
7 changes: 3 additions & 4 deletions pkg/sqlite/image_filter.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@ package sqlite

import (
"context"
"fmt"

"github.com/stashapp/stash/pkg/models"
)
Expand Down Expand Up @@ -255,12 +254,12 @@ func (qb *imageFilterHandler) performerFavoriteCriterionHandler(performerfavorit
if *performerfavorite {
// contains at least one favorite
f.addLeftJoin("performers", "", "performers.id = performers_images.performer_id")
f.addWhere(fmt.Sprintf("performers.favorite = %s", getDBBoolean(true)))
f.addWhere("performers.favorite = " + getDBBoolean(true))
} else {
// contains zero favorites
f.addLeftJoin(fmt.Sprintf(`(SELECT performers_images.image_id as id FROM performers_images
f.addLeftJoin(`(SELECT performers_images.image_id as id FROM performers_images
JOIN performers ON performers.id = performers_images.performer_id
GROUP BY performers_images.image_id HAVING SUM(performers.favorite) = %s)`, getDBBoolean(false)), "nofaves", "images.id = nofaves.id")
GROUP BY performers_images.image_id HAVING SUM(performers.favorite) = `+getDBBoolean(false)+")", "nofaves", "images.id = nofaves.id")
f.addWhere("performers_images.image_id IS NULL OR nofaves.id IS NOT NULL")
}
}
Expand Down
5 changes: 4 additions & 1 deletion pkg/sqlite/migrationsPostgres/1_initial.up.sql
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,10 @@ CREATE TABLE IF NOT EXISTS images (
organized boolean not null default FALSE,
created_at timestamp not null,
updated_at timestamp not null,
date date, code text, photographer text, details text,
date date,
code text,
photographer text,
details text,
foreign key(studio_id) references studios(id) on delete SET NULL
);
CREATE TABLE image_urls (
Expand Down
6 changes: 3 additions & 3 deletions pkg/sqlite/repository.go
Original file line number Diff line number Diff line change
Expand Up @@ -461,10 +461,10 @@ func idToIndexMap(ids []int) map[int]int {
func (r *filesRepository) getMany(ctx context.Context, ids []int, primaryOnly bool) ([][]models.FileID, error) {
var primaryClause string
if primaryOnly {
primaryClause = fmt.Sprintf(" AND \"primary\" = %s", getDBBoolean(true))
primaryClause = ` AND "primary" = ` + getDBBoolean(true)
}

query := fmt.Sprintf("SELECT %s as id, file_id, \"primary\" from %s WHERE %[1]s IN %[3]s%s", r.idColumn, r.tableName, getInBinding(len(ids)), primaryClause)
query := fmt.Sprintf(`SELECT %s as id, file_id, "primary" from %s WHERE %[1]s IN %[3]s%s`, r.idColumn, r.tableName, getInBinding(len(ids)), primaryClause)

idi := make([]interface{}, len(ids))
for i, id := range ids {
Expand Down Expand Up @@ -505,7 +505,7 @@ func (r *filesRepository) getMany(ctx context.Context, ids []int, primaryOnly bo
}

func (r *filesRepository) get(ctx context.Context, id int) ([]models.FileID, error) {
query := fmt.Sprintf("SELECT file_id, \"primary\" from %s WHERE %s = ?", r.tableName, r.idColumn)
query := fmt.Sprintf(`SELECT file_id, "primary" from %s WHERE %s = ?`, r.tableName, r.idColumn)

type relatedFile struct {
FileID models.FileID `db:"file_id"`
Expand Down
6 changes: 3 additions & 3 deletions pkg/sqlite/scene_filter.go
Original file line number Diff line number Diff line change
Expand Up @@ -455,12 +455,12 @@ func (qb *sceneFilterHandler) performerFavoriteCriterionHandler(performerfavorit
if *performerfavorite {
// contains at least one favorite
f.addLeftJoin("performers", "", "performers.id = performers_scenes.performer_id")
f.addWhere(fmt.Sprintf("performers.favorite = %s", getDBBoolean(true)))
f.addWhere("performers.favorite = " + getDBBoolean(true))
} else {
// contains zero favorites
f.addLeftJoin(fmt.Sprintf(`(SELECT performers_scenes.scene_id as id FROM performers_scenes
f.addLeftJoin(`(SELECT performers_scenes.scene_id as id FROM performers_scenes
JOIN performers ON performers.id = performers_scenes.performer_id
GROUP BY performers_scenes.scene_id HAVING SUM(performers.favorite) = %s)`, getDBBoolean(false)), "nofaves", "scenes.id = nofaves.id")
GROUP BY performers_scenes.scene_id HAVING SUM(performers.favorite) = `+getDBBoolean(false)+")", "nofaves", "scenes.id = nofaves.id")
f.addWhere("performers_scenes.scene_id IS NULL OR nofaves.id IS NOT NULL")
}
}
Expand Down
2 changes: 1 addition & 1 deletion pkg/sqlite/tag.go
Original file line number Diff line number Diff line change
Expand Up @@ -584,7 +584,7 @@ func (qb *TagStore) QueryForAutoTag(ctx context.Context, words []string) ([]*mod

whereOr := "(" + strings.Join(whereClauses, " OR ") + ")"
where := strings.Join([]string{
fmt.Sprintf("tags.ignore_auto_tag = %s", getDBBoolean(false)),
"tags.ignore_auto_tag = " + getDBBoolean(false),
whereOr,
}, " AND ")
return qb.queryTags(ctx, query+" WHERE "+where, args)
Expand Down
Loading