Ubuntu 24.04: How to handle upgrades that jump multiple minors (e.g. when an old version was installed from standard package repositories)

[dkbarn]

I recently upgraded Ubuntu 22.04 to 24.04. As part of this upgrade, rabbitmq-server was upgraded to v3.12.1, the apt version for noble.

Previously the version of rabbitmq-server must have been <3.11, although I'm not certain of the exact version. But the service now refuses to start, because feature flags were not enabled before upgrading (which I was not aware of, and received no warnings about when upgrading Ubuntu).

2024-06-08 21:50:45.613158-07:00 [notice] <0.44.0> Application rabbit exited with reason: {failed_to_initialize_feature_flags_registry,{rabbit,start,[normal,[]]}}
{"Kernel pid terminated",application_controller,"{application_start_failure,rabbit,{failed_to_initialize_feature_flags_registry,{rabbit,start,[normal,[]]}}}"}
2024-06-08 21:50:59.473893-07:00 [error] <0.230.0> Feature flags: `classic_mirrored_queue_version`: required feature flag not enabled! It must be enabled before upgrading RabbitMQ.
2024-06-08 21:50:59.488495-07:00 [error] <0.230.0> Failed to initialize feature flags registry: {disabled_required_feature_flag,
2024-06-08 21:50:59.488495-07:00 [error] <0.230.0>                                               classic_mirrored_queue_version}
2024-06-08 21:50:59.501209-07:00 [error] <0.230.0>
2024-06-08 21:50:59.501209-07:00 [error] <0.230.0> BOOT FAILED
2024-06-08 21:50:59.501209-07:00 [error] <0.230.0> ===========
2024-06-08 21:50:59.501209-07:00 [error] <0.230.0> Error during startup: {error,failed_to_initialize_feature_flags_registry}
2024-06-08 21:50:59.501209-07:00 [error] <0.230.0>
2024-06-08 21:51:00.502146-07:00 [error] <0.229.0>   crasher:
2024-06-08 21:51:00.502146-07:00 [error] <0.229.0>     initial call: application_master:init/4
2024-06-08 21:51:00.502146-07:00 [error] <0.229.0>     pid: <0.229.0>
2024-06-08 21:51:00.502146-07:00 [error] <0.229.0>     registered_name: []
2024-06-08 21:51:00.502146-07:00 [error] <0.229.0>     exception exit: {failed_to_initialize_feature_flags_registry,
2024-06-08 21:51:00.502146-07:00 [error] <0.229.0>                         {rabbit,start,[normal,[]]}}
2024-06-08 21:51:00.502146-07:00 [error] <0.229.0>       in function  application_master:init/4 (application_master.erl, line 142)
2024-06-08 21:51:00.502146-07:00 [error] <0.229.0>     ancestors: [<0.228.0>]
2024-06-08 21:51:00.502146-07:00 [error] <0.229.0>     message_queue_len: 1
2024-06-08 21:51:00.502146-07:00 [error] <0.229.0>     messages: [{'EXIT',<0.230.0>,normal}]
2024-06-08 21:51:00.502146-07:00 [error] <0.229.0>     links: [<0.228.0>,<0.44.0>]
2024-06-08 21:51:00.502146-07:00 [error] <0.229.0>     dictionary: []
2024-06-08 21:51:00.502146-07:00 [error] <0.229.0>     trap_exit: true
2024-06-08 21:51:00.502146-07:00 [error] <0.229.0>     status: running
2024-06-08 21:51:00.502146-07:00 [error] <0.229.0>     heap_size: 233
2024-06-08 21:51:00.502146-07:00 [error] <0.229.0>     stack_size: 28
2024-06-08 21:51:00.502146-07:00 [error] <0.229.0>     reductions: 168
2024-06-08 21:51:00.502146-07:00 [error] <0.229.0>   neighbours:
2024-06-08 21:51:00.502146-07:00 [error] <0.229.0>
2024-06-08 21:51:00.509790-07:00 [notice] <0.44.0> Application rabbit exited with reason: {failed_to_initialize_feature_flags_registry,{rabbit,start,[normal,[]]}}
{"Kernel pid terminated",application_controller,"{application_start_failure,rabbit,{failed_to_initialize_feature_flags_registry,{rabbit,start,[normal,[]]}}}"}

My question is, what is the recommended approach for rectifying this situation? I am unable to downgrade rabbitmq because there is no previous version available to me in any apt repository, even the ones maintained by rabbitmq. I've already tried this:

curl -1sLf 'https://dl.cloudsmith.io/public/rabbitmq/rabbitmq-erlang/setup.deb.sh' | sudo -E bash

I am already aware of #9677 but these future plans do not help those of us in a currently broken state.

I have searched through all discussions related to this issue but every response just says "you need to downgrade to X" without actually explaining how to downgrade. Is dpkg the only option? Because it looks like an absolute nightmare to manage all the necessary dependencies and their appropriate versions manually.

Please help :(

[mkuratczyk]

Is this a production server or a dev environment? If a dev env (or any other kind of environment where you don't care about the data currently in RabbitMQ), the easiest option is to delete the data folder (/var/lib/rabbitmq/* probably). This will delete all definitions (queues, exchanges, etc) and all messages, if there are any. But if that's ok with you, that's the easiest option - afterwards it will start a fresh node without issues.

[dkbarn]

This is a production server, so retaining the definition of my queues and messages in those queues is important.

[dkbarn]

Would following the backup and restore procedure outlined here work? https://www.rabbitmq.com/docs/backup

Or would the restoration just restore the problematic feature flags?

[dkbarn]

I've tried used docker to get access to a previous version of rabbitmq. Here's what I did:

1. Pull down a previous version of rabbitmq (in my case, version 3.9.13):

sudo docker pull rabbitmq:3.9.13

2. Make a backup copy of /var/lib/rabbitmq and mount it as a volume inside the docker container:

sudo cp -r /var/lib/rabbitmq /var/lib/rabbitmq-backup
sudo docker run -d --hostname <my_host_name> --name rabbit-backup -v /var/lib/rabbitmq-backup:/var/lib/rabbitmq -p 15672:15672 -p 5672:5672 rabbitmq:3.9.13

3. Enter the docker container and enable all feature flags via rabbitmqctl, then shutdown the server

sudo docker exec -it rabbit-backup bash
$ rabbitmqctl status # verify it's running
$ rabbitmqctl enable_feature_flag all
$ rabbitmqctl shutdown

4. Copy backup directory over original directory

sudo cp -r /var/lib/rabbitmq-backup /var/lib/rabbitmq

5. Start up rabbitmq-server v3.12.1 with this /var/lib/rabbitmq

But when I do this, I get the same errors:

2024-06-09 10:41:30.405634-07:00 [notice] <0.44.0> Application rabbit exited with reason: {failed_to_initialize_feature_flags_registry,{rabbit,start,[normal,[]]}}
2024-06-09 10:41:41.659518-07:00 [error] <0.235.0> Feature flags: `classic_mirrored_queue_version`: required feature flag not enabled! It must be enabled before upgrading RabbitMQ.
2024-06-09 10:41:41.663967-07:00 [error] <0.235.0> Failed to initialize feature flags registry: {disabled_required_feature_flag,
2024-06-09 10:41:41.663967-07:00 [error] <0.235.0>                                               classic_mirrored_queue_version}
2024-06-09 10:41:41.668151-07:00 [error] <0.235.0>
2024-06-09 10:41:41.668151-07:00 [error] <0.235.0> BOOT FAILED
2024-06-09 10:41:41.668151-07:00 [error] <0.235.0> ===========
2024-06-09 10:41:41.668151-07:00 [error] <0.235.0> Error during startup: {error,failed_to_initialize_feature_flags_registry}
2024-06-09 10:41:41.668151-07:00 [error] <0.235.0>
2024-06-09 10:41:42.670257-07:00 [error] <0.234.0>   crasher:
2024-06-09 10:41:42.670257-07:00 [error] <0.234.0>     initial call: application_master:init/4
2024-06-09 10:41:42.670257-07:00 [error] <0.234.0>     pid: <0.234.0>
2024-06-09 10:41:42.670257-07:00 [error] <0.234.0>     registered_name: []
2024-06-09 10:41:42.670257-07:00 [error] <0.234.0>     exception exit: {failed_to_initialize_feature_flags_registry,
2024-06-09 10:41:42.670257-07:00 [error] <0.234.0>                         {rabbit,start,[normal,[]]}}
2024-06-09 10:41:42.670257-07:00 [error] <0.234.0>       in function  application_master:init/4 (application_master.erl, line 142)
2024-06-09 10:41:42.670257-07:00 [error] <0.234.0>     ancestors: [<0.233.0>]
2024-06-09 10:41:42.670257-07:00 [error] <0.234.0>     message_queue_len: 1
2024-06-09 10:41:42.670257-07:00 [error] <0.234.0>     messages: [{'EXIT',<0.235.0>,normal}]
2024-06-09 10:41:42.670257-07:00 [error] <0.234.0>     links: [<0.233.0>,<0.44.0>]
2024-06-09 10:41:42.670257-07:00 [error] <0.234.0>     dictionary: []
2024-06-09 10:41:42.670257-07:00 [error] <0.234.0>     trap_exit: true
2024-06-09 10:41:42.670257-07:00 [error] <0.234.0>     status: running
2024-06-09 10:41:42.670257-07:00 [error] <0.234.0>     heap_size: 233
2024-06-09 10:41:42.670257-07:00 [error] <0.234.0>     stack_size: 28
2024-06-09 10:41:42.670257-07:00 [error] <0.234.0>     reductions: 172
2024-06-09 10:41:42.670257-07:00 [error] <0.234.0>   neighbours:
2024-06-09 10:41:42.670257-07:00 [error] <0.234.0>
2024-06-09 10:41:42.680731-07:00 [notice] <0.44.0> Application rabbit exited with reason: {failed_to_initialize_feature_flags_registry,{rabbit,start,[normal,[]]}}

What am I missing here? Shouldn't me running rabbitmqctl enable_feature_flag all be making the necessary changes to the contents of /var/lib/rabbitmq that would address the errors?

[mkuratczyk]

We don't support upgrades that skip minor releases. There were feature flags introduced in 3.10/3.11, so if you enable all 3.9 feature flags and then start 3.12, there are still some flags that 3.12 expects that were not enabled (since 3.9 was unaware of them). If you follow the same process with 3.10 and then 3.11, I think it should work. https://www.rabbitmq.com/docs/upgrade#rabbitmq-version-upgradability

Relying on distro-provided packages is a terrible idea by the way. 3.12 is already out of community support and 3.12.1 is 13 patch releases behind anyway...

[dequis]

I followed a variant of the docker instructions above to upgrade from 3.8 to 3.13, one minor version at a time.

I use the recommended apt repos, I just don't update very often. And unfortunately due to erlang versions it is difficult to upgrade to a newer version through apt packages.

I had these issues with the instructions above:

• cp -r /var/lib/rabbitmq-backup /var/lib/rabbitmq copies the entire -backup directory inside the rabbitmq directory, so that won't work
• My /etc/rabbitmq/rabbitmq-env.conf was setting RABBITMQ_MNESIA_BASE to /var/lib/rabbitmq, which meant that it wasn't actually mounting the correct data directory
• The docker container will change the UID:GID of the files to 999:999, should chown it to rabbitmq after it's done

What I did instead was using the real /var/lib/rabbitmq to not have to worry about details like chmod of individual files or skipping dotfiles or something:

• Stop the non-docker system-wide rabbitmq that fails to start
• Make the backup just a backup

cp -r /var/lib/rabbitmq /var/lib/rabbitmq-backup

• chown the real rabbitmq to what the docker image wants

chown 999:999 -R /var/lib/rabbitmq

• Run docker mounting the real /var/lib/rabbitmq (not exposing ports as above since receiving connections is not desirable)

# base version
docker run -d --hostname $HOSTNAME --name rabbit-upgrade -v /var/lib/rabbitmq:/var/lib/rabbitmq rabbitmq:3.10

# with debug (shows enabled feature flags)
docker run -d --hostname $HOSTNAME --name rabbit-upgrade -v /var/lib/rabbitmq:/var/lib/rabbitmq -e RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="-rabbit log [{console,[{level,debug}]}]" rabbitmq:3.10

# with an explicit RABBITMQ_MNESIA_BASE
docker run -d --hostname $HOSTNAME --name rabbit-upgrade -v /var/lib/rabbitmq:/var/lib/rabbitmq -e RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="-rabbit log [{console,[{level,debug}]}]" -e RABBITMQ_MNESIA_BASE=/var/lib/rabbitmq rabbitmq:3.10

• Follow logs until it's done starting up

docker logs --follow rabbit-upgrade

• Go inside, list and enable flags, shutdown

docker exec -it rabbit-upgrade bash
rabbitmqctl list_feature_flags
rabbitmqctl enable_feature_flag all
rabbitmqctl shutdown

# this exits into the outer shell btw

• Remove the docker image

docker rm rabbit-upgrade

• Repeat from docker run step above with the next minor version.
• When done, chown to the correct version

chown rabbitmq:rabbitmq -R /var/lib/rabbitmq

• Check your /var/lib/rabbitmq/mnesia/rabbit@$HOSTNAME-feature_flags. Should look longer than the one in the backup.
• Start the system-wide rabbitmq
• If it works, delete the backup.

[michaelklishin]

FTR, 3.8.0 and 3.13.0 are over four years apart.

"Not very often" indeed.

[michaelklishin]

As our Debian and Ubuntu installation guide mentions at the very first paragraph, Ubuntu is always — yes, always — at least six, usually 12+ months behind latest RabbitMQ releases.

And when they upgrade, it's often a jump of multiple minors, which in recent releases with feature flag graduation means the node will refuse to start in many cases (as long as at least one minor series in between graduates a feature flag).