rabbitmq-queues rebalance strange behavior

[fiksn]

Hi,

can somebody please explain me this:

$ rabbitmqctl eval "rabbit_mnesia:cluster_nodes(all)."
['rabbit@ip-10-48-2-30','rabbit@ip-10-48-7-41','rabbit@ip-10-48-10-4']

$ rabbitmq-queues rebalance classic
Re-balancing leaders of classic queues...
Error:
{:nodedown, :"rabbit@ip-10-48-6-39"}

Running 3.8.14 RabbitMQ.

[michaelklishin]

Either the command or node failed to contact node rabbit@ip-10-48-6-39. There are no any other details for CLI tools to log.

[fiksn]

Ok, but that node is not part of the cluster anymore, so it seems like rebalance has a stale state?

[mkuratczyk]

If that node used to be a part of this cluster but has been permanently removed, you need to tell RabbitMQ about it with rabbitmqctl forget_cluster_node. More info: https://www.rabbitmq.com/clustering.html#removing-nodes.

[fiksn]

Yeah, it was removed with rabbitmqctl forget_cluster_node so also if I try rabbitmqctl cluster_status that node is gone, but rebalance still wants to use it. Note that on Management UI under "Ports and contexts" I still see Protocol "clustering" and the long gone server(s) on port 25672. Might be related. How to clear that?

[fiksn]

I can see all the 25672 clustering listeners also with rabbitmqctl eval 'rabbit_misc:dirty_read_all(rabbit_listener).'. Perhaps it's worth mentioning that we do upgrades in a rolling fashion and use AWS autocluster plugin. One node gets killed then rabbitmqctl forget_cluster_node <old_node> and later a new node joins the cluster.

Are those clustering listeners ever deleted? It seems like a bug that they aren't.

How can this affect rebalancing? I've checked https://github.com/rabbitmq/rabbitmq-server/pull/2106/files and it seems nodes are obtained through rabbit_mnesia:cluster_nodes(running) (which like I've said appears to be correct in our case). I am also not aware of any special policies specifying node name.