Up command don't run all pending migrations

[Anochin]

Just recently I had an issue where I merged a branch and there was a migration in the merged branch that was older than the migration I had run in my current branch.

So I had missed a migration and had no way of knowing other than goose status.

It would be great to have an option to always run all the pending migrations regardless of whether they are the latest or not.

[LucasHild]

I could create a pull request for this issue and take care of this feature.

Are there any preferences about how I should implement it? Do you want me to change the behavior of the "up" command or do you want me to create a new command "up-all"? Another option would be to add a command-line flag to the "up" command.

Personally, I'd prefer to change the behavior of the "up" command, since it's the most logical approach for me to always run all pending migrations. But I am not sure whether this would be seen as a breaking change.

What do you think?

[MrDOS]

What would the performance implications be of modifying the up command to apply all pending migrations? I ask because for one of my projects, which has accumulated 208 migrations over the last half-decade, status takes ~55s to run, even though up still runs fairly quickly (~a few seconds overhead at most). My functional preference would be for up to apply all pending migrations, but not if it would add that much execution time.

[tvarsis]

I think adding the option up-all or up -a etc would be the safest way to do it. We have also ran into issues when we spin up new environments where we need to run the up command multiple times to get all migrations in, which will scale bad as the migrations keeps increasing, so it would be appreciated to at least have a way for it to always run all migrations.

[mfridman]

What would the performance implications be of modifying the up command to apply all pending migrations? I ask because for one of my projects, which has accumulated 208 migrations over the last half-decade, status takes ~55s to run, even though up still runs fairly quickly (~a few seconds overhead at most). My functional preference would be for up to apply all pending migrations, but not if it would add that much execution time.

Interesting, we've never observed status taking this long.. for contrast, we have 2 projects with ~700 and ~200 migrations and they run status quickly, within 2-3 seconds.

Having said that, we can certainly improve the performance of that status command.

---

For the initial question, I assume you've read why this is not supported, more details here: hybrid-versioning

Having said that, over the years this has been the number one requested feature.

My proposal is to maintain the existing functionality as the default, call it "strict mode", where migrations behave as they do today.

However, give users ability to opt-out of "strict mode" whereby goose applies all migrations regardless of whether the timestamp or sequence are appropriately ordered.

-disable-strict or -force-all <- I'd prefer if this flag be more explicit than just -a or -all

@VojtechVitek any objections/thoughts?

[MrDOS]

For the initial question, I assume you've read why this is not supported, more details here: hybrid-versioning

In my case, I've only just recently started working on this project which has used timestamp versioning for years, and this prescription is a somewhat bitter pill:

• Sequential versioning in production has maybe always been Pressly's advice, but some projects clearly have not followed it. (In our case, our project used the original, upstream version of Goose until late 2019, and so we were unaware of this recommendation.) It's unclear to me, in technical terms, how to migrate (ha) an existing project to a “straight and narrow” sequential versioning scheme. Is it enough to (manually) update the version_id on each row in the goose_db_version table to replace the timestamps with the serial migration numbers? And how do I automate that across all our deployment environments – not to mention every developer's local instance – in a way that doesn't add too much more code which needs to be tested and maintained while the change is incomplete?
• Even then, the advice for how to use goose fix is not very concrete. Our CI process doesn't commit back to the project repository, so even if we ran goose fix there, we'd still be subject to out-of-order migrations being merged. For us, a GitHub workflow to check that PRs don't include any migrations older than the newest one in the target branch would be much more effective.

I'm painfully aware that supporting all the multifaceted ways in which users (ab)use an open source project is not the responsibility of its maintainers. To be clear, I'm grateful for the existence of this project in its current form, and I really hope this feedback doesn't come off as stroppy and entitled. But I don't think sequential migration versioning is a complete panacea. Maybe the best way out of this is not to write any code to support a “loose mode” at all, but rather better documentation on how to safely:

• manually create Goose version records corresponding to the old migrations to correspond with manually applying the schema changes
• switch an established codebase and multiple production environments from timestamped to sequentially-versioned migrations
• enforce timestamp-to-serial version changes in a modern/common CI pipeline (e.g., GitHub workflows, Bitbucket Pipelines)

“Safely” meaning at least as far as Goose's audit records are concerned, not the contents of the migrations. Out-of-order migrations clearly carry some safety concerns with respect to schema dependencies, and that's reason enough alone to want to avoid applying old ones carelessly.

[mfridman]

Superseded by #262. Closing in favor of that issue.

In short, we plan to add support for "apply all migrations, always, even if out-of-order" which will be an opt-in flag.