rmq0-1 | 2024-09-11 00:29:07.214709+00:00 [error] <0.490.0> ** Generic server <0.490.0> terminating rmq0-1 | 2024-09-11 00:29:07.214709+00:00 [error] <0.490.0> ** Last message in was {'DOWN',#Ref<0.1105274211.656408577.104353>,process, rmq0-1 | 2024-09-11 00:29:07.214709+00:00 [error] <0.490.0> <14541.487.0>,noconnection} rmq0-1 | 2024-09-11 00:29:07.214709+00:00 [error] <0.490.0> ** When Server state == {state,<0.488.0>,<0.489.0>, rmq0-1 | 2024-09-11 00:29:07.214709+00:00 [error] <0.490.0> rabbit_federation_exchange_link_sup_sup, rmq0-1 | 2024-09-11 00:29:07.214709+00:00 [error] <0.490.0> undefined,[],[]} rmq0-1 | 2024-09-11 00:29:07.214709+00:00 [error] <0.490.0> ** Reason for termination == rmq0-1 | 2024-09-11 00:29:07.214709+00:00 [error] <0.490.0> ** {function_clause, rmq0-1 | 2024-09-11 00:29:07.214709+00:00 [error] <0.490.0> [{lists,sort, rmq0-1 | 2024-09-11 00:29:07.214709+00:00 [error] <0.490.0> [#Fun,{error,timeout}], rmq0-1 | 2024-09-11 00:29:07.214709+00:00 [error] <0.490.0> [{file,"lists.erl"},{line,1205}]}, rmq0-1 | 2024-09-11 00:29:07.214709+00:00 [error] <0.490.0> {mirrored_supervisor,handle_info,2, rmq0-1 | 2024-09-11 00:29:07.214709+00:00 [error] <0.490.0> [{file,"mirrored_supervisor.erl"},{line,350}]}, rmq0-1 | 2024-09-11 00:29:07.214709+00:00 [error] <0.490.0> {gen_server2,handle_msg,2,[{file,"gen_server2.erl"},{line,1056}]}, rmq0-1 | 2024-09-11 00:29:07.214709+00:00 [error] <0.490.0> {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,241}]}]} rmq0-1 | 2024-09-11 00:29:07.214709+00:00 [error] <0.490.0> rmq0-1 | 2024-09-11 00:29:07.220345+00:00 [error] <0.493.0> ** Generic server <0.493.0> terminating rmq0-1 | 2024-09-11 00:29:07.220345+00:00 [error] <0.493.0> ** Last message in was {'DOWN',#Ref<0.1105274211.656408579.101115>,process, rmq0-1 | 2024-09-11 00:29:07.220345+00:00 [error] <0.493.0> <14541.490.0>,noconnection} rmq0-1 | 2024-09-11 00:29:07.220345+00:00 [error] <0.493.0> ** When Server state == {state,<0.491.0>,<0.492.0>, rmq0-1 | 2024-09-11 00:29:07.220345+00:00 [error] <0.493.0> rabbit_federation_queue_link_sup_sup,undefined, rmq0-1 | 2024-09-11 00:29:07.220345+00:00 [error] <0.493.0> [],[]} rmq0-1 | 2024-09-11 00:29:07.220345+00:00 [error] <0.493.0> ** Reason for termination == rmq0-1 | 2024-09-11 00:29:07.220345+00:00 [error] <0.493.0> ** {function_clause, rmq0-1 | 2024-09-11 00:29:07.220345+00:00 [error] <0.493.0> [{lists,sort, rmq0-1 | 2024-09-11 00:29:07.220345+00:00 [error] <0.493.0> [#Fun,{error,timeout}], rmq0-1 | 2024-09-11 00:29:07.220345+00:00 [error] <0.493.0> [{file,"lists.erl"},{line,1205}]}, rmq0-1 | 2024-09-11 00:29:07.220345+00:00 [error] <0.493.0> {mirrored_supervisor,handle_info,2, rmq0-1 | 2024-09-11 00:29:07.220345+00:00 [error] <0.493.0> [{file,"mirrored_supervisor.erl"},{line,350}]}, rmq0-1 | 2024-09-11 00:29:07.220345+00:00 [error] <0.493.0> {gen_server2,handle_msg,2,[{file,"gen_server2.erl"},{line,1056}]}, rmq0-1 | 2024-09-11 00:29:07.220345+00:00 [error] <0.493.0> {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,241}]}]} rmq0-1 | 2024-09-11 00:29:07.220345+00:00 [error] <0.493.0> rmq0-1 | 2024-09-11 00:29:07.221073+00:00 [error] <0.493.0> crasher: rmq0-1 | 2024-09-11 00:29:07.221073+00:00 [error] <0.493.0> initial call: mirrored_supervisor:init/1 rmq0-1 | 2024-09-11 00:29:07.221073+00:00 [error] <0.493.0> pid: <0.493.0> rmq0-1 | 2024-09-11 00:29:07.221073+00:00 [error] <0.493.0> registered_name: [] rmq0-1 | 2024-09-11 00:29:07.221073+00:00 [error] <0.493.0> exception exit: {function_clause, rmq0-1 | 2024-09-11 00:29:07.221073+00:00 [error] <0.493.0> [{lists,sort, rmq0-1 | 2024-09-11 00:29:07.221073+00:00 [error] <0.493.0> [#Fun, rmq0-1 | 2024-09-11 00:29:07.221073+00:00 [error] <0.493.0> {error,timeout}], rmq0-1 | 2024-09-11 00:29:07.221073+00:00 [error] <0.493.0> [{file,"lists.erl"},{line,1205}]}, rmq0-1 | 2024-09-11 00:29:07.221073+00:00 [error] <0.493.0> {mirrored_supervisor,handle_info,2, rmq0-1 | 2024-09-11 00:29:07.221073+00:00 [error] <0.493.0> [{file,"mirrored_supervisor.erl"},{line,350}]}, rmq0-1 | 2024-09-11 00:29:07.221073+00:00 [error] <0.493.0> {gen_server2,handle_msg,2, rmq0-1 | 2024-09-11 00:29:07.221073+00:00 [error] <0.493.0> [{file,"gen_server2.erl"},{line,1056}]}, rmq0-1 | 2024-09-11 00:29:07.221073+00:00 [error] <0.493.0> {proc_lib,init_p_do_apply,3, rmq0-1 | 2024-09-11 00:29:07.221073+00:00 [error] <0.493.0> [{file,"proc_lib.erl"},{line,241}]}]} rmq0-1 | 2024-09-11 00:29:07.221073+00:00 [error] <0.493.0> in function gen_server2:terminate/3 (gen_server2.erl, line 1172) rmq0-1 | 2024-09-11 00:29:07.221073+00:00 [error] <0.493.0> ancestors: [rabbit_federation_queue_link_sup_sup,rabbit_federation_sup, rmq0-1 | 2024-09-11 00:29:07.221073+00:00 [error] <0.493.0> rabbit_sup,<0.254.0>] rmq0-1 | 2024-09-11 00:29:07.221073+00:00 [error] <0.493.0> message_queue_len: 0 rmq0-1 | 2024-09-11 00:29:07.221073+00:00 [error] <0.493.0> messages: [] rmq0-1 | 2024-09-11 00:29:07.221073+00:00 [error] <0.493.0> links: [<0.491.0>] rmq0-1 | 2024-09-11 00:29:07.221073+00:00 [error] <0.493.0> dictionary: [{{khepri_machine,can_skip_fence_preliminary_query, rmq0-1 | 2024-09-11 00:29:07.221073+00:00 [error] <0.493.0> rabbitmq_metadata}, rmq0-1 | 2024-09-11 00:29:07.221073+00:00 [error] <0.493.0> true}] rmq0-1 | 2024-09-11 00:29:07.221073+00:00 [error] <0.493.0> trap_exit: true rmq0-1 | 2024-09-11 00:29:07.221073+00:00 [error] <0.493.0> status: running rmq0-1 | 2024-09-11 00:29:07.221073+00:00 [error] <0.493.0> heap_size: 28690 rmq0-1 | 2024-09-11 00:29:07.221073+00:00 [error] <0.493.0> stack_size: 28 rmq0-1 | 2024-09-11 00:29:07.221073+00:00 [error] <0.493.0> reductions: 1823896 rmq0-1 | 2024-09-11 00:29:07.221073+00:00 [error] <0.493.0> neighbours: rmq0-1 | 2024-09-11 00:29:07.221073+00:00 [error] <0.493.0> rmq0-1 | 2024-09-11 00:29:07.215899+00:00 [error] <0.490.0> crasher: rmq0-1 | 2024-09-11 00:29:07.215899+00:00 [error] <0.490.0> initial call: mirrored_supervisor:init/1 rmq0-1 | 2024-09-11 00:29:07.215899+00:00 [error] <0.490.0> pid: <0.490.0> rmq0-1 | 2024-09-11 00:29:07.215899+00:00 [error] <0.490.0> registered_name: [] rmq0-1 | 2024-09-11 00:29:07.215899+00:00 [error] <0.490.0> exception exit: {function_clause, rmq0-1 | 2024-09-11 00:29:07.215899+00:00 [error] <0.490.0> [{lists,sort, rmq0-1 | 2024-09-11 00:29:07.215899+00:00 [error] <0.490.0> [#Fun, rmq0-1 | 2024-09-11 00:29:07.215899+00:00 [error] <0.490.0> {error,timeout}], rmq0-1 | 2024-09-11 00:29:07.215899+00:00 [error] <0.490.0> [{file,"lists.erl"},{line,1205}]}, rmq0-1 | 2024-09-11 00:29:07.215899+00:00 [error] <0.490.0> {mirrored_supervisor,handle_info,2, rmq0-1 | 2024-09-11 00:29:07.215899+00:00 [error] <0.490.0> [{file,"mirrored_supervisor.erl"},{line,350}]}, rmq0-1 | 2024-09-11 00:29:07.215899+00:00 [error] <0.490.0> {gen_server2,handle_msg,2, rmq0-1 | 2024-09-11 00:29:07.215899+00:00 [error] <0.490.0> [{file,"gen_server2.erl"},{line,1056}]}, rmq0-1 | 2024-09-11 00:29:07.215899+00:00 [error] <0.490.0> {proc_lib,init_p_do_apply,3, rmq0-1 | 2024-09-11 00:29:07.215899+00:00 [error] <0.490.0> [{file,"proc_lib.erl"},{line,241}]}]} rmq0-1 | 2024-09-11 00:29:07.215899+00:00 [error] <0.490.0> in function gen_server2:terminate/3 (gen_server2.erl, line 1172) rmq0-1 | 2024-09-11 00:29:07.215899+00:00 [error] <0.490.0> ancestors: [rabbit_federation_exchange_link_sup_sup, rmq0-1 | 2024-09-11 00:29:07.215899+00:00 [error] <0.490.0> rabbit_federation_sup,rabbit_sup,<0.254.0>] rmq0-1 | 2024-09-11 00:29:07.215899+00:00 [error] <0.490.0> message_queue_len: 0 rmq0-1 | 2024-09-11 00:29:07.215899+00:00 [error] <0.490.0> messages: [] rmq0-1 | 2024-09-11 00:29:07.215899+00:00 [error] <0.490.0> links: [<0.488.0>] rmq0-1 | 2024-09-11 00:29:07.215899+00:00 [error] <0.490.0> dictionary: [{{khepri_machine,can_skip_fence_preliminary_query, rmq0-1 | 2024-09-11 00:29:07.215899+00:00 [error] <0.490.0> rabbitmq_metadata}, rmq0-1 | 2024-09-11 00:29:07.215899+00:00 [error] <0.490.0> true}] rmq0-1 | 2024-09-11 00:29:07.215899+00:00 [error] <0.490.0> trap_exit: true rmq0-1 | 2024-09-11 00:29:07.215899+00:00 [error] <0.490.0> status: running rmq0-1 | 2024-09-11 00:29:07.215899+00:00 [error] <0.490.0> heap_size: 28690 rmq0-1 | 2024-09-11 00:29:07.215899+00:00 [error] <0.490.0> stack_size: 28 rmq0-1 | 2024-09-11 00:29:07.215899+00:00 [error] <0.490.0> reductions: 1823998 rmq0-1 | 2024-09-11 00:29:07.215899+00:00 [error] <0.490.0> neighbours: rmq0-1 | 2024-09-11 00:29:07.215899+00:00 [error] <0.490.0> rmq0-1 | 2024-09-11 00:29:07.234041+00:00 [error] <0.491.0> supervisor: {local,rabbit_federation_queue_link_sup_sup} rmq0-1 | 2024-09-11 00:29:07.234041+00:00 [error] <0.491.0> errorContext: child_terminated rmq0-1 | 2024-09-11 00:29:07.234041+00:00 [error] <0.491.0> reason: {function_clause, rmq0-1 | 2024-09-11 00:29:07.234041+00:00 [error] <0.491.0> [{lists,sort, rmq0-1 | 2024-09-11 00:29:07.234041+00:00 [error] <0.491.0> [#Fun,{error,timeout}], rmq0-1 | 2024-09-11 00:29:07.234041+00:00 [error] <0.491.0> [{file,"lists.erl"},{line,1205}]}, rmq0-1 | 2024-09-11 00:29:07.234041+00:00 [error] <0.491.0> {mirrored_supervisor,handle_info,2, rmq0-1 | 2024-09-11 00:29:07.234041+00:00 [error] <0.491.0> [{file,"mirrored_supervisor.erl"},{line,350}]}, rmq0-1 | 2024-09-11 00:29:07.234041+00:00 [error] <0.491.0> {gen_server2,handle_msg,2, rmq0-1 | 2024-09-11 00:29:07.234041+00:00 [error] <0.491.0> [{file,"gen_server2.erl"},{line,1056}]}, rmq0-1 | 2024-09-11 00:29:07.234041+00:00 [error] <0.491.0> {proc_lib,init_p_do_apply,3, rmq0-1 | 2024-09-11 00:29:07.234041+00:00 [error] <0.491.0> [{file,"proc_lib.erl"},{line,241}]}]} rmq0-1 | 2024-09-11 00:29:07.234041+00:00 [error] <0.491.0> offender: [{pid,<0.493.0>}, rmq0-1 | 2024-09-11 00:29:07.234041+00:00 [error] <0.491.0> {id,mirroring}, rmq0-1 | 2024-09-11 00:29:07.234041+00:00 [error] <0.491.0> {mfargs, rmq0-1 | 2024-09-11 00:29:07.234041+00:00 [error] <0.491.0> {mirrored_supervisor,start_internal, rmq0-1 | 2024-09-11 00:29:07.234041+00:00 [error] <0.491.0> [rabbit_federation_queue_link_sup_sup,[]]}}, rmq0-1 | 2024-09-11 00:29:07.234041+00:00 [error] <0.491.0> {restart_type,permanent}, rmq0-1 | 2024-09-11 00:29:07.234041+00:00 [error] <0.491.0> {shutdown,4294967295}, rmq0-1 | 2024-09-11 00:29:07.234041+00:00 [error] <0.491.0> {child_type,worker}] rmq0-1 | 2024-09-11 00:29:07.234138+00:00 [error] <0.488.0> supervisor: {local,rabbit_federation_exchange_link_sup_sup} rmq0-1 | 2024-09-11 00:29:07.234138+00:00 [error] <0.488.0> errorContext: child_terminated rmq0-1 | 2024-09-11 00:29:07.234138+00:00 [error] <0.488.0> reason: {function_clause, rmq0-1 | 2024-09-11 00:29:07.234138+00:00 [error] <0.488.0> [{lists,sort, rmq0-1 | 2024-09-11 00:29:07.234138+00:00 [error] <0.488.0> [#Fun,{error,timeout}], rmq0-1 | 2024-09-11 00:29:07.234138+00:00 [error] <0.488.0> [{file,"lists.erl"},{line,1205}]}, rmq0-1 | 2024-09-11 00:29:07.234138+00:00 [error] <0.488.0> {mirrored_supervisor,handle_info,2, rmq0-1 | 2024-09-11 00:29:07.234138+00:00 [error] <0.488.0> [{file,"mirrored_supervisor.erl"},{line,350}]}, rmq0-1 | 2024-09-11 00:29:07.234138+00:00 [error] <0.488.0> {gen_server2,handle_msg,2, rmq0-1 | 2024-09-11 00:29:07.234138+00:00 [error] <0.488.0> [{file,"gen_server2.erl"},{line,1056}]}, rmq0-1 | 2024-09-11 00:29:07.234138+00:00 [error] <0.488.0> {proc_lib,init_p_do_apply,3, rmq0-1 | 2024-09-11 00:29:07.234138+00:00 [error] <0.488.0> [{file,"proc_lib.erl"},{line,241}]}]} rmq0-1 | 2024-09-11 00:29:07.234138+00:00 [error] <0.488.0> offender: [{pid,<0.490.0>}, rmq0-1 | 2024-09-11 00:29:07.234138+00:00 [error] <0.488.0> {id,mirroring}, rmq0-1 | 2024-09-11 00:29:07.234138+00:00 [error] <0.488.0> {mfargs, rmq0-1 | 2024-09-11 00:29:07.234138+00:00 [error] <0.488.0> {mirrored_supervisor,start_internal, rmq0-1 | 2024-09-11 00:29:07.234138+00:00 [error] <0.488.0> [rabbit_federation_exchange_link_sup_sup,[]]}}, rmq0-1 | 2024-09-11 00:29:07.234138+00:00 [error] <0.488.0> {restart_type,permanent}, rmq0-1 | 2024-09-11 00:29:07.234138+00:00 [error] <0.488.0> {shutdown,4294967295}, rmq0-1 | 2024-09-11 00:29:07.234138+00:00 [error] <0.488.0> {child_type,worker}] rmq0-1 | 2024-09-11 00:29:07.235079+00:00 [error] <0.491.0> supervisor: {local,rabbit_federation_queue_link_sup_sup} rmq0-1 | 2024-09-11 00:29:07.235079+00:00 [error] <0.491.0> errorContext: shutdown rmq0-1 | 2024-09-11 00:29:07.235079+00:00 [error] <0.491.0> reason: reached_max_restart_intensity rmq0-1 | 2024-09-11 00:29:07.235079+00:00 [error] <0.491.0> offender: [{pid,<0.493.0>}, rmq0-1 | 2024-09-11 00:29:07.235079+00:00 [error] <0.491.0> {id,mirroring}, rmq0-1 | 2024-09-11 00:29:07.235079+00:00 [error] <0.491.0> {mfargs, rmq0-1 | 2024-09-11 00:29:07.235079+00:00 [error] <0.491.0> {mirrored_supervisor,start_internal, rmq0-1 | 2024-09-11 00:29:07.235079+00:00 [error] <0.491.0> [rabbit_federation_queue_link_sup_sup,[]]}}, rmq0-1 | 2024-09-11 00:29:07.235079+00:00 [error] <0.491.0> {restart_type,permanent}, rmq0-1 | 2024-09-11 00:29:07.235079+00:00 [error] <0.491.0> {shutdown,4294967295}, rmq0-1 | 2024-09-11 00:29:07.235079+00:00 [error] <0.491.0> {child_type,worker}] rmq0-1 | 2024-09-11 00:29:07.235478+00:00 [error] <0.488.0> supervisor: {local,rabbit_federation_exchange_link_sup_sup} rmq0-1 | 2024-09-11 00:29:07.235478+00:00 [error] <0.488.0> errorContext: shutdown rmq0-1 | 2024-09-11 00:29:07.235478+00:00 [error] <0.488.0> reason: reached_max_restart_intensity rmq0-1 | 2024-09-11 00:29:07.235478+00:00 [error] <0.488.0> offender: [{pid,<0.490.0>}, rmq0-1 | 2024-09-11 00:29:07.235478+00:00 [error] <0.488.0> {id,mirroring}, rmq0-1 | 2024-09-11 00:29:07.235478+00:00 [error] <0.488.0> {mfargs, rmq0-1 | 2024-09-11 00:29:07.235478+00:00 [error] <0.488.0> {mirrored_supervisor,start_internal, rmq0-1 | 2024-09-11 00:29:07.235478+00:00 [error] <0.488.0> [rabbit_federation_exchange_link_sup_sup,[]]}}, rmq0-1 | 2024-09-11 00:29:07.235478+00:00 [error] <0.488.0> {restart_type,permanent}, rmq0-1 | 2024-09-11 00:29:07.235478+00:00 [error] <0.488.0> {shutdown,4294967295}, rmq0-1 | 2024-09-11 00:29:07.235478+00:00 [error] <0.488.0> {child_type,worker}] perf-test-1 | id: test-002643-562, time 132.000 s, sent: 1000 msg/s, returned: 0 msg/s, confirmed: 1000 msg/s, nacked: 0 msg/s, received: 1000 msg/s, min/median/75th/95th/99th consumer latency: 5401/7388/8361/12549/13328 µs, confirm latency: 5324/7331/8241/12488/13287 µs perf-test-1 | id: test-002643-562, time 133.000 s, sent: 1000 msg/s, returned: 0 msg/s, confirmed: 1004 msg/s, nacked: 0 msg/s, received: 1004 msg/s, min/median/75th/95th/99th consumer latency: 5828/8380/9773/14468/19601 µs, confirm latency: 5623/8257/9712/14195/19353 µs perf-test-1 | id: test-002643-562, time 134.000 s, sent: 1001 msg/s, returned: 0 msg/s, confirmed: 1000 msg/s, nacked: 0 msg/s, received: 1000 msg/s, min/median/75th/95th/99th consumer latency: 5883/8085/9370/14209/19053 µs, confirm latency: 5808/8003/9288/13904/19075 µs rmq2-1 | 2024-09-11 00:29:10.084227+00:00 [error] <0.181.0> rmq2-1 | 2024-09-11 00:29:10.084227+00:00 [error] <0.181.0> ** Cannot get connection id for node 'rabbit@rmq2.local' rmq2-1 | 2024-09-11 00:29:10.084227+00:00 [error] <0.181.0> rmq1-1 | 2024-09-11 00:29:10.096091+00:00 [error] <0.181.0> rmq1-1 | 2024-09-11 00:29:10.096091+00:00 [error] <0.181.0> ** Cannot get connection id for node 'rabbit@rmq1.local' rmq1-1 | 2024-09-11 00:29:10.096091+00:00 [error] <0.181.0>