feat(meta): Failover handling meta service

[CAJan93]

I hereby agree to the terms of the Singularity Data, Inc. Contributor License Agreement.

What's changed and what's your intention?

• Tracing issue: Tracking: high availability for Meta service #5943
• PR introduces leader election of meta nodes
• A leader starts execution as normal. Followers idle
• A follower can become leader once the leader crashes/times out

TODOs:

• Unittests
• Dummy follower service
• How do we test that failover handling works?
• Delete draft file

Checklist

• I have written necessary rustdoc comments
• I have added necessary unit tests and integration tests
• All checks passed in ./risedev check (or alias, ./risedev c)

Documentation

If your pull request contains user-facing changes, please specify the types of the changes, and create a release note. Otherwise, please feel free to remove this section.

Types of user-facing changes

#5943

Please keep the types that apply to your changes, and remove those that do not apply.

• Installation and deployment
• Connector (sources & sinks)
• SQL commands, functions, and operators
• RisingWave cluster configuration changes
• Other (please specify in the release note below)

Release note

Please create a release note for your changes. In the release note, focus on the impact on users, and mention the environment or conditions where the impact may occur.

Refer to a related PR or issue link (optional)

[codecov]

Codecov Report

Merging #6466 (aa20b19) into main (8b46da1) will increase coverage by 0.04%.
The diff coverage is n/a.

❗ Current head aa20b19 differs from pull request most recent head 5748a16. Consider uploading reports for the commit 5748a16 to get more accurate results

@@            Coverage Diff             @@
##             main    #6466      +/-   ##
==========================================
+ Coverage   73.20%   73.25%   +0.04%     
==========================================
  Files        1024     1023       -1     
  Lines      163741   163598     -143     
==========================================
- Hits       119870   119844      -26     
+ Misses      43871    43754     -117     

Flag	Coverage Δ
rust	73.25% <0.00%> (+0.04%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/common/src/config.rs	58.91% <0.00%> (-3.07%)	⬇️
src/frontend/src/handler/create_sink.rs	93.75% <0.00%> (-2.83%)	⬇️
src/frontend/src/handler/util.rs	83.10% <0.00%> (-2.21%)	⬇️
src/expr/src/vector_op/cast.rs	87.92% <0.00%> (-1.88%)	⬇️
src/frontend/src/handler/mod.rs	63.47% <0.00%> (-1.53%)	⬇️
src/frontend/src/handler/explain.rs	52.88% <0.00%> (-0.89%)	⬇️
...ontend/src/optimizer/plan_node/stream_hash_join.rs	85.71% <0.00%> (-0.60%)	⬇️
src/frontend/src/handler/create_source.rs	66.66% <0.00%> (-0.14%)	⬇️
src/frontend/src/handler/create_index.rs	88.41% <0.00%> (-0.12%)	⬇️
src/frontend/planner_test/src/lib.rs	73.55% <0.00%> (-0.09%)	⬇️
... and 25 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[CAJan93]

fencing mechanism

[yezizp2012]

Besides, Why are you maintaining this PR now? 🤔
It seems that you will need to submit all the commits to both the PR you're working on and this one. I think the whole HA tasks is well split, if for test purpose you can test in the PR you're working on. And this PR contains some confusing changes that was introduced maybe in the very early stage. It is difficult or increases your workload to keep up with the main or your new PR.
Well, just some friendly advice. Hope it can save some of your time.

[CAJan93]

Besides, Why are you maintaining this PR now? 🤔

I am keeping this PR more or less up to date to test how the current implementations work with the other things we will implement. Furthermore I am using this PR to copy paste things from into the new PRs.

Totally agree with you. Only keeping this PR around for myself. We will not merge it and I am focusing on the actual feature PRs and not on this one.