Skip to content
This repository has been archived by the owner on Nov 17, 2020. It is now read-only.

Improve logging on unexpected behaviour during partial partitions #53

Closed
dcorbacho opened this issue Jan 23, 2017 · 0 comments
Closed
Assignees
Milestone

Comments

@dcorbacho
Copy link
Contributor

During partial network partitions on the downstream cluster, nodes on different sides of the partition might connect simultaneously to the upstream. That causes upstream exchanges to be recreated with different names (A/B) and links trying to add bindings to exchanges that don't exist anymore. That causes crashes like the one below and restarts the link.

Apparently, the best way to recover the link is to allow it to restart, but the crash can be captured and replaced by a more informative log message.

=SUPERVISOR REPORT==== 19-Jan-2017::15:34:44 ===
     Supervisor: {<0.298.0>,rabbit_federation_link_sup}
     Context:    child_terminated
     Reason:     {{shutdown,
                      {server_initiated_close,404,
                          <<"NOT_FOUND - no exchange 'federation: test_5 -> rabbit@ubuntu-c3 B' in vhost '/'">>}},
                  {gen_server,call,
                      [<0.671.0>,
                       {call,
                           {'exchange.bind',0,
                               <<"federation: test_5 -> rabbit@ubuntu-c3 B">>,
                               <<"test_5">>,<<"test_5">>,false,
                               [{<<"x-bound-from">>,array,
                                 [{table,
                                      [{<<"cluster-name">>,longstr,
                                        <<"rabbit@ubuntu-c3">>},
                                       {<<"exchange">>,longstr,
                                        <<"/:test_5 B">>},
                                       {<<"hops">>,short,1}]}]}]},
                           none,<0.299.0>},
                       infinity]}}
     Offender:   [{pid,<0.299.0>},
                  {name,
                      {upstream,
                          [<<"amqp://192.168.56.108">>],
                          <<"test_5">>,<<"test_5">>,1000,1,5,none,none,false,
                          'on-confirm',none,<<"ubuntu-c4">>,false}},
                  {mfargs,
                      {rabbit_federation_exchange_link,start_link,
                          [{{upstream,
                                [<<"amqp://192.168.56.108">>],
                                <<"test_5">>,<<"test_5">>,1000,1,5,none,none,
                                false,'on-confirm',none,<<"ubuntu-c4">>,false},
                            {resource,<<"/">>,exchange,<<"test_5">>}}]}},
                  {restart_type,{permanent,5}},
                  {shutdown,30000},
                  {child_type,worker}]
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

2 participants