feat(risedev): support multiple etcd and multiple meta node

[BugenZhao]

Signed-off-by: Bugen Zhao 2 [email protected]

I hereby agree to the terms of the Singularity Data, Inc. Contributor License Agreement.

What's changed and what's your intention?

This PR adds support for multiple etcd and multiple meta node. Use ./risedev d 3etcd-3meta-1cn-1fe to set up this profile.

Checklist

• I have written necessary rustdoc comments
• I have added necessary unit tests and integration tests
• All checks passed in ./risedev check (or alias, ./risedev c)

Documentation

If your pull request contains user-facing changes, please specify the types of the changes, and create a release note. Otherwise, please feel free to remove this section.

Types of user-facing changes

Please keep the types that apply to your changes, and remove those that do not apply.

• Installation and deployment
• Connector (sources & sinks)
• SQL commands, functions, and operators
• RisingWave cluster configuration changes
• Other (please specify in the release note below)

Release note

Please create a release note for your changes. In the release note, focus on the impact on users, and mention the environment or conditions where the impact may occur.

Refer to a related PR or issue link (optional)

ref #5943
cc @CAJan93

[codecov]

Codecov Report

Merging #6421 (c67bbdc) into main (37ae311) will decrease coverage by 0.01%.
The diff coverage is 66.04%.

@@            Coverage Diff             @@
##             main    #6421      +/-   ##
==========================================
- Coverage   74.05%   74.04%   -0.02%     
==========================================
  Files         966      972       +6     
  Lines      157832   158105     +273     
==========================================
+ Hits       116887   117071     +184     
- Misses      40945    41034      +89     

Flag	Coverage Δ
rust	74.04% <66.04%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/batch/src/executor/row_seq_scan.rs	20.20% <ø> (ø)
src/batch/src/task/task_execution.rs	50.29% <0.00%> (ø)
src/compute/src/lib.rs	2.56% <0.00%> (-0.07%)	⬇️
src/compute/src/server.rs	0.00% <0.00%> (ø)
src/connector/src/aws_utils.rs	0.00% <0.00%> (ø)
src/connector/src/source/cdc/enumerator/mod.rs	0.00% <0.00%> (ø)
src/connector/src/source/cdc/source/message.rs	0.00% <0.00%> (ø)
src/connector/src/source/cdc/source/reader.rs	0.00% <0.00%> (ø)
src/connector/src/source/cdc/split.rs	0.00% <0.00%> (ø)
src/connector/src/source/filesystem/s3/s3_dir.rs	0.00% <0.00%> (ø)
... and 130 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[lmatz]

Do we want to add this setting into the e2e tests in CI?

[CAJan93]

This looks really amazing! Thank you for this work!

[skyzh]

Do we want to add this setting into the e2e tests in CI?

It doesn’t work at all for now

[CAJan93]

It doesn’t work at all for now

What exactly does not work for you?

[lmatz]

It doesn’t work at all for now

What exactly does not work for you?

My understanding is that there are indeed multiple meta nodes by triggering this setting, but they just exist and only one node has actual connection with the rest of the components in the cluster.

[CAJan93]

My understanding is that there are indeed multiple meta nodes by triggering this setting, but they just exist and only one node has actual connection with the rest of the components in the cluster.

That is correct. In the current setup one node will become leader and the followers will idle forever, even if the leader fails. The relevant code that halts execution can be found here:

risingwave/src/meta/src/rpc/server.rs

Line 269 in 3699a78

futures::future::pending::<()>().await;

Status of HA setup

We are currently working on the HA setup for meta (see tracking issue). The first draft that does failover handling can be found here #6466

The current failover handling "works": A new leader gets elected and starts its services. However, because of #6534 this node immediately crashes.