Late answer crash in rabbit_channel

[dcorbacho]

While testing partial partitions, I often find the following function_clause in rabbit_channel:handle_info. It kind of seems like the late answer of a gen_server after a timeout, but instead of be of the form{Ref, LateAnswer} it is {{Ref, Node}, LateAnswer}. In most of the cases, the LateAnswer is {ok, Pid, ok} which could correspond to some of the messages handled in delegate calls, as to rabbit_amqqueue:basic_consume. The second snippet contains an unusal crash in the handle_info, where we get a better trace of this call.

If for any reason a call has timeout but the channel is able to continue working, there should be no harm on ignoring late messages.

** Reason for termination ==
** {function_clause,
       [{rabbit_channel,handle_info,
            [{{#Ref<0.0.262150.167475>,'rabbit@rb-a7674cb60661'},
              [{ok,<7956.19508.1>,{ok,0,0}}]},
             {ch,running,rabbit_framing_amqp_0_9_1,5,<0.11855.19>,
                 <0.21058.20>,<0.11855.19>,
                 <<"172.24.168.11:50873 -> 172.24.173.33:5671">>,
                 {lstate,<0.21059.20>,false},
                 none,1,
                 {[],[]},
                 {user,<<"f90c2e89-5d91-430c-8a1f-0cf29a4f4496">>,[],
                     [{rabbit_auth_backend_internal,none}]},
                 <<"f90c2e89-5d91-430c-8a1f-0cf29a4f4496">>,
                 <<"excelsior-thranaqengx02.RESPONSE.72451d59-3d80-45ec-bab0-d33961172de4">>,
                 {dict,0,16,16,8,80,48,
                     {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                     {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                 {state,
                     {dict,0,16,16,8,80,48,
                         {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                         {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                     erlang},
                 {dict,0,16,16,8,80,48,
                     {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                     {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                 {dict,0,16,16,8,80,48,
                     {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                     {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                 {set,0,16,16,8,80,48,
                     {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                     {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                 <0.11860.19>,
                 {state,fine,30000,undefined},
                 true,1,
                 {{0,nil},{0,nil}},
                 [],

=ERROR REPORT==== 24-Oct-2016::10:31:40 ===
Error on AMQP connection <0.2505.0> (192.168.56.1:53972 -> 192.168.56.105:5672, vhost: '/', user: 'test', state: blocking), channel 2:error while terminating:
{function_clause,
 [{rabbit_channel,handle_info,
   [{{#Ref<0.0.1.50703>,'rabbit@ubuntu-c2'},
     [{error,<9278.550.0>,
       {exit,
        {shutdown,
         {gen_server2,call,
          [<9278.550.0>,
           {basic_consume,false,<0.2519.0>,<0.2518.0>,false,0,
            <<"amq.ctag-i4Oh6mc1B_njNPtMVkNgwQ">>,false,[],undefined},
           infinity]}},
        [{gen_server2,call,3,[{file,"src/gen_server2.erl"},{line,325}]},
         {delegate,safe_invoke,2,[{file,"src/delegate.erl"},{line,199}]},
         {delegate,'-safe_invoke/2-lc$^0/1-0-',2,
          [{file,"src/delegate.erl"},{line,196}]},
         {delegate,handle_call,3,[{file,"src/delegate.erl"},{line,214}]},
         {gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1011}]},
         {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,257}]}]}}]},
    {ch,running,rabbit_framing_amqp_0_9_1,2,<0.2505.0>,<0.2517.0>,<0.2505.0>,
     <<"192.168.56.1:53972 -> 192.168.56.105:5672">>,
     {lstate,<0.2518.0>,false},
     none,61043,