[Bug]: Helm upgrade fails because the databases already exists (zitadel-setup)

[samip5]

Preflight Checklist

• I could not find a solution in the documentation, the existing issues or discussions
• I have joined the ZITADEL chat

Version

7.3.0

App version

No response

Describe the problem caused by this bug

I'm trying to upgrade from chart 7.3.0 to 7.5.0 but unable due to zitadel-setup trying to create the database again, despite it existing already.

To reproduce

1. Deploy Chart 7.3.0 with Postres
2. Try to upgrade to chart 7.5.0 without making changes to anything
3. See it fail the zitadel-setup, and see the actual problem in Postgres logs.

Logs

zitadel-setup:

time="2024-01-26T10:11:42Z" level=info msg="setup started" caller="/home/runner/work/zitadel/zitadel/cmd/setup/setup.go:67"
time="2024-01-26T10:11:42Z" level=info msg="verify migration" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:44" name=14_events_push
time="2024-01-26T10:11:42Z" level=info msg="verify migration" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:44" name=01_tables
time="2024-01-26T10:11:42Z" level=info msg="verify migration" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:44" name=02_assets
time="2024-01-26T10:11:42Z" level=info msg="verify migration" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:44" name=03_default_instance
time="2024-01-26T10:11:42Z" level=info msg="verify migration" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:44" name=05_last_failed
time="2024-01-26T10:11:42Z" level=info msg="verify migration" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:44" name=06_resource_owner_columns
time="2024-01-26T10:11:42Z" level=info msg="verify migration" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:44" name=07_logstore
time="2024-01-26T10:11:42Z" level=info msg="verify migration" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:44" name=08_auth_token_indexes
time="2024-01-26T10:11:42Z" level=info msg="verify migration" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:44" name=12_auth_users_otp_columns
time="2024-01-26T10:11:42Z" level=info msg="verify migration" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:44" name=13_fix_quota_constraints
time="2024-01-26T10:11:42Z" level=info msg="verify migration" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:44" name=15_current_projection_state
time="2024-01-26T10:11:42Z" level=info msg="verify migration" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:44" name=16_unique_constraint_lower
time="2024-01-26T10:11:42Z" level=info msg="verify migration" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:44" name=17_add_offset_col_to_current_states
time="2024-01-26T10:11:42Z" level=info msg="verify migration" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:44" name=19_add_current_sequences_index
time="2024-01-26T10:11:42Z" level=info msg="verify migration" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:44" name=config_change
time="2024-01-26T10:11:42Z" level=info msg="verify migration" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:44" name=projection_tables
time="2024-01-26T10:11:42Z" level=warning msg="migration already started, will check again in 5 seconds" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:136" migration step=projection_tables
time="2024-01-26T10:11:47Z" level=warning msg="migration already started, will check again in 5 seconds" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:136" migration step=projection_tables
time="2024-01-26T10:11:53Z" level=warning msg="migration already started, will check again in 5 seconds" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:136" migration step=projection_tables
time="2024-01-26T10:11:58Z" level=warning msg="migration already started, will check again in 5 seconds" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:136" migration step=projection_tables
time="2024-01-26T10:12:03Z" level=warning msg="migration already started, will check again in 5 seconds" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:136" migration step=projection_tables
time="2024-01-26T10:12:08Z" level=warning msg="migration already started, will check again in 5 seconds" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:136" migration step=projection_tables
time="2024-01-26T10:12:13Z" level=warning msg="migration already started, will check again in 5 seconds" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:136" migration step=projection_tables
time="2024-01-26T10:12:18Z" level=warning msg="migration already started, will check again in 5 seconds" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:136" migration step=projection_tables
time="2024-01-26T10:12:23Z" level=warning msg="migration already started, will check again in 5 seconds" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:136" migration step=projection_tables
time="2024-01-26T10:12:28Z" level=warning msg="migration already started, will check again in 5 seconds" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:136" migration step=projection_tables
time="2024-01-26T10:12:33Z" level=warning msg="migration already started, will check again in 5 seconds" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:136" migration step=projection_tables
time="2024-01-26T10:12:38Z" level=warning msg="migration already started, will check again in 5 seconds" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:136" migration step=projection_tables
time="2024-01-26T10:12:43Z" level=warning msg="migration already started, will check again in 5 seconds" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:136" migration step=projection_tables
time="2024-01-26T10:12:48Z" level=warning msg="migration already started, will check again in 5 seconds" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:136" migration step=projection_tables

Expected behavior

I would have expected it to be smart and see that it already exists, so it shouldn't try to re-create the database on upgrade but rather just run migrations if needed.

Relevant Configuration

My Helm values are here: https://github.com/samipsolutions/infra/blob/main/kubernetes/apps/security/zitadel/app/hr.yaml

Additional Context

No response

[eliobischof]

Hi @samip5

The message migration already started means that your setup step got stuck for some reason. In this case, you should run the command zitadel setup cleanup manually, so the step is retried on the next setup.

I will try to find a way to define a job in kubernetes without starting it that can be executed manually instead.