Quorum queue crash during rolling upgrade to 3.13 (install_snapshot_rpc)

[gomoripeti]

During a rolling upgrade from 3.12.13 to 3.13.2 (Erlang 26.2.5) of a 3-node cluster a quorum queue follower crashed with the following, when

• mixed version: node 01 already on new version, 02, 03 on old version
• leader is on 01, followers on 02, 03
• after 03 crashed and status became noproc I tried to delete_member then add_member on 03 but it crashed again the same way

I suspect the reason is that PR ra#375 changed the cluster field from a list to a map that is included in the #install_snapshot_rpc. In this case the the RPC message is sent from 01 running the new code to a follower on 03 running the old code which cannot handle a map.

Is this really a bug in rabbitmq/ra?

Is there any workaround to push through an upgrade? Maybe using rabbit_quorum_queue:force_shrink_member_to_current_member/1. to temporarily shrink the queue to the new node 01, then add members 02 and 03 after the rolling upgrade finished?

tagging @illotum as you have a deep understanding of this part of the code (btw thanks for the great feature of non-voters)

** State machine 'vhost1_qq1' terminating
** Last event = {{call,
                     {<27487.7215.0>,
                      [alias|
                       #Ref<27487.0.923523.1478895759.3268214785.253450>]}},
                 {install_snapshot_rpc,13,
                     {'vhost1_qq1',
                         'rabbit@host-01'},
                     #{index => 7511,term => 13,machine_version => 3,
                       cluster =>
                           #{{'vhost1_qq1',
                                 'rabbit@host-01'} =>
                                 #{},
                             {'vhost1_qq1',
                                 'rabbit@host-02'} =>
                                 #{},
                             {'vhost1_qq1',
                                 'rabbit@host-03'} =>
                                 #{}}},
                     {1,last},
                     <<...>>}}
...
** Reason for termination = error:function_clause
** Callback modules = [ra_server_proc]
** Callback mode = [state_functions,state_enter]
** Stacktrace =
**  [{ra_server,'-make_cluster/2-lists^foldl/2-0-',
         [#Fun<ra_server.18.7275718>,#{},
          #{{'vhost1_qq1','rabbit@host-01'} =>
                #{},
            {'vhost1_qq1','rabbit@host-02'} =>
                #{},
            {'vhost1_qq1','rabbit@host-03'} =>
                #{}}],
         [{file,"src/ra_server.erl"},{line,2210}]},
     {ra_server,make_cluster,2,[{file,"src/ra_server.erl"},{line,2210}]},
     {ra_server,handle_receive_snapshot,2,
         [{file,"src/ra_server.erl"},{line,1270}]},
     {ra_server_proc,handle_receive_snapshot,2,
         [{file,"src/ra_server_proc.erl"},{line,1064}]},
     {ra_server_proc,receive_snapshot,3,
         [{file,"src/ra_server_proc.erl"},{line,817}]},
     {gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1395}]},
     {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,241}]}]

[illotum]

😮Indeed, there is nothing prohibiting leaders from sending out snapshots in the new format. The opposite works, followers will accept both old and new.

Perhaps the best way to fix is make it send both values under different keys.

[LoisSotoLopez]

We've seen that there were some leader abdications during this rolling upgrade. We were wondering if and how it could be related to the generation of the snapshot at such unfortunate moment, and more precisely why did 01 jump to term 13?

Logs below are a combination of the 3 nodes logs. They've been simplified and marked after the time stamp with the correspondent node number.

11:26:22.038704+00:00 02 <0.2685.0> detected a new leader 03 in term 11
11:26:22.051113+00:00 03 <0.2746.0> candidate -> leader in term: 11 machine version: 3

11:28:05.936751+00:00 01 <0.6140.0> detected a new leader 03 in term 11

11:30:24.009948+00:00 03 <0.2746.0> leader saw append_entries_reply from 02 for term 12 abdicates term: 11!
11:30:24.019823+00:00 03 <0.2746.0> leader -> follower in term: 12 machine version: 3
11:30:24.027562+00:00 02 <0.7009.0> candidate -> leader in term: 12 machine version: 3
11:30:24.027700+00:00 03 <0.2746.0> detected a new leader 02 in term 12
11:30:24.045665+00:00 01 <0.6140.0> detected a new leader 02 in term 12

11:32:42.220041+00:00 03 <0.7836.0> detected a new leader 02 in term 12

11:36:00.053418+00:00 02 <0.7009.0> leader saw append_entries_reply from 01 for term 13 abdicates term: 12!

11:36:00.067650+00:00 02 <0.7009.0> leader -> follower in term: 13 machine version: 3
11:36:00.085412+00:00 01 <0.6469.0> candidate -> leader in term: 13 machine version: 3
11:36:00.086400+00:00 02 <0.7009.0> detected a new leader 01 in term 13
11:36:00.139657+00:00 03 <0.7836.0> detected a new leader 01 in term 13

13:24:41.541422+00:00 01 <0.6469.0> leader -> terminating_leader in term: 13 machine version: 3

13:26:45.571680+00:00 03 <0.2010069.0> candidate -> leader in term: 1 machine version: 3
13:26:45.571976+00:00 02 <0.1691756.0> detected a new leader 03 in term 1
13:26:45.573625+00:00 01 <0.2120331.0> detected a new leader 03 in term 1

13:52:17.673825+00:00 02 <0.3032.0> detected a new leader 03 in term 1
13:52:55.845962+00:00 01 <0.2120331.0> detected a new leader 02 in term 3
13:52:55.846592+00:00 02 <0.3032.0> candidate -> leader in term: 3 machine version: 3
13:54:00.189128+00:00 03 <0.2556.0> detected a new leader 02 in term 3
13:54:05.763637+00:00 03 <0.2556.0> detected a new leader 02 in term 3

[lukebakken]

For what it's worth, I've re-visited this discussion several times, and have tried to reproduce this issue using this project - https://github.com/lukebakken/rabbitmq-server-11441

No luck! I think I'll close this discussion but if someone is able to reproduce, please follow up.