RabbitMQ restart automatically after ERROR "Timed out geting channel"

[mxmcarl]

Hi,
I have a 3-node rabbitmq cluster, one node of them restart automatically after error "Timed out geting channel". Who know the reason?（rabbitmq V3.8.0 Erlang 22.1.5）

2021-06-15 15:32:13.642 [warning] <0.20389.190> Mirrored queue 'reply_15955195c647412393a9a5c38861bc67' in vhost '/': Missing 'DOWN' message from <11076.13688.50> in node '[email protected]'
2021-06-15 15:32:15.573 [error] <0.25262.191> Timed out getting channel <0.14882.188> info
2021-06-15 15:32:38.620 [error] <0.6022.5> closing AMQP connection <0.6022.5> (192.168.10.7:36630 -> 10.233.64.199:5672 - neutron-server:57:47946e1f-18dd-4796-85f7-04090d3fa520):
missed heartbeats from client, timeout: 60s
2021-06-15 15:32:40.405 [warning] <0.20518.190> Mirrored queue 'reply_68aadd59a6424b5d8f8d7184369dfa0a' in vhost '/': Missing 'DOWN' message from <11076.13967.50> in node '[email protected]'
2021-06-15 15:32:41.833 [info] <0.25578.191> accepting AMQP connection <0.25578.191> (192.168.10.14:49940 -> 10.233.64.199:5672)
2021-06-15 15:32:41.836 [info] <0.25578.191> Connection <0.25578.191> (192.168.10.14:49940 -> 10.233.64.199:5672) has a client-provided name: nova-compute:1220:66f661bd-a2ba-4801-b2fc-c80cc74019f2
2021-06-15 15:32:41.837 [info] <0.25578.191> connection <0.25578.191> (192.168.10.14:49940 -> 10.233.64.199:5672 - nova-compute:1220:66f661bd-a2ba-4801-b2fc-c80cc74019f2): user 'rabbitmq' authenticated and granted access to vhost '/'
2021-06-15 15:32:53.014 [info] <0.25675.191> accepting AMQP connection <0.25675.191> (192.168.10.7:33646 -> 10.233.64.199:5672)
2021-06-15 15:32:53.020 [info] <0.25675.191> Connection <0.25675.191> (192.168.10.7:33646 -> 10.233.64.199:5672) has a client-provided name: nova-conductor:38:505bc414-70c7-4f39-bdb7-7b0b56097b50
2021-06-15 15:32:53.026 [info] <0.25675.191> connection <0.25675.191> (192.168.10.7:33646 -> 10.233.64.199:5672 - nova-conductor:38:505bc414-70c7-4f39-bdb7-7b0b56097b50): user 'rabbitmq' authenticated and granted access to vhost '/'
2021-06-15 15:32:55.406 [warning] <0.20518.190> Mirrored queue 'reply_68aadd59a6424b5d8f8d7184369dfa0a' in vhost '/': Missing 'DOWN' message from <7595.18899.311> in node '[email protected]'
2021-06-15 15:32:55.669 [error] <0.11860.191> closing AMQP connection <0.11860.191> (192.168.10.7:49058 -> 10.233.64.199:5672 - neutron-server:74:157f160b-92f5-4773-ac31-5f9d7f9d69cd):
missed heartbeats from client, timeout: 60s
2021-06-15 15:33:10.186 [info] <0.25775.191> accepting AMQP connection <0.25775.191> (192.168.10.7:34066 -> 10.233.64.199:5672)
2021-06-15 15:33:10.265 [info] <0.25775.191> Connection <0.25775.191> (192.168.10.7:34066 -> 10.233.64.199:5672) has a client-provided name: neutron-server:53:b71a9365-7978-4f18-821d-d0f4eb98061e
2021-06-15 15:33:10.316 [info] <0.25775.191> connection <0.25775.191> (192.168.10.7:34066 -> 10.233.64.199:5672 - neutron-server:53:b71a9365-7978-4f18-821d-d0f4eb98061e): user 'rabbitmq' authenticated and granted access to vhost '/'
2021-06-15 15:33:12.211 [info] <0.25803.191> accepting AMQP connection <0.25803.191> (10.233.69.196:35436 -> 10.233.64.199:5672)
2021-06-15 15:33:12.216 [info] <0.25803.191> Connection <0.25803.191> (10.233.69.196:35436 -> 10.233.64.199:5672) has a client-provided name: ceilometer-agent-notification:14:64a0f771-fe44-4c20-9edc-6708cf2fb576
2021-06-15 15:33:12.221 [info] <0.25803.191> connection <0.25803.191> (10.233.69.196:35436 -> 10.233.64.199:5672 - ceilometer-agent-notification:14:64a0f771-fe44-4c20-9edc-6708cf2fb576): user 'rabbitmq' authenticated and granted access to vhost '/'
2021-06-15 15:33:15.624 [error] <0.25708.191> Timed out getting channel <0.14882.188> info
2021-06-15 15:33:16.118 [info] <0.25832.191> accepting AMQP connection <0.25832.191> (192.168.10.7:34228 -> 10.233.64.199:5672)
2021-06-15 15:33:16.263 [info] <0.25836.191> accepting AMQP connection <0.25836.191> (192.168.10.7:34244 -> 10.233.64.199:5672)
2021-06-15 15:33:16.263 [info] <0.25832.191> Connection <0.25832.191> (192.168.10.7:34228 -> 10.233.64.199:5672) has a client-provided name: neutron-server:74:bf2e3aff-7f7c-4a97-9fec-93de2c9c883d
2021-06-15 15:33:16.269 [info] <0.25836.191> Connection <0.25836.191> (192.168.10.7:34244 -> 10.233.64.199:5672) has a client-provided name: neutron-server:74:157f160b-92f5-4773-ac31-5f9d7f9d69cd
2021-06-15 15:33:16.271 [info] <0.25836.191> connection <0.25836.191> (192.168.10.7:34244 -> 10.233.64.199:5672 - neutron-server:74:157f160b-92f5-4773-ac31-5f9d7f9d69cd): user 'rabbitmq' authenticated and granted access to vhost '/'
2021-06-15 15:33:16.279 [info] <0.25832.191> connection <0.25832.191> (192.168.10.7:34228 -> 10.233.64.199:5672 - neutron-server:74:bf2e3aff-7f7c-4a97-9fec-93de2c9c883d): user 'rabbitmq' authenticated and granted access to vhost '/'
2021-06-15 15:33:16.432 [info] <0.25857.191> accepting AMQP connection <0.25857.191> (192.168.10.7:34256 -> 10.233.64.199:5672)
2021-06-15 15:33:16.434 [info] <0.25857.191> Connection <0.25857.191> (192.168.10.7:34256 -> 10.233.64.199:5672) has a client-provided name: nova-conductor:32:028de250-cbdf-455e-aca0-6008f8570c00
2021-06-15 15:33:16.435 [info] <0.25857.191> connection <0.25857.191> (192.168.10.7:34256 -> 10.233.64.199:5672 - nova-conductor:32:028de250-cbdf-455e-aca0-6008f8570c00): user 'rabbitmq' authenticated and granted access to vhost '/'
2021-06-15 15:33:16.611 [info] <0.25869.191> RabbitMQ is asked to stop...
2021-06-15 15:33:16.691 [info] <0.25869.191> Stopping RabbitMQ applications and their dependencies in the following order:
rabbitmq_management
amqp_client
rabbitmq_web_dispatch
cowboy
cowlib
rabbitmq_management_agent
rabbit
mnesia
rabbit_common
sysmon_handler
os_mon
ra
2021-06-15 15:33:16.691 [info] <0.25869.191> Stopping application 'rabbitmq_management'
...

Regards
xianming. M

[michaelklishin]

missed heartbeats from client, timeout: 60s

has a dedicated doc guide.

RabbitMQ 3.8.0 is 19 months and 18 patch releases behind, please upgrade.

[michaelklishin]

There is no causation between the missed heartbeats and node restart. The log clearly indicates that RabbitMQ was asked to stop,
most likely by a monitoring system of some kind. Maybe the missed heartbeats and this monitoring-driven restart have the same
root cause, we can't know (e.g. a genuine network connectivity disruption on the host).

[mxmcarl]

OK, thanks for ur reply.