Tracking: high availability for Meta service

[yezizp2012]

To support Meta's high availability, we need to do some refactor in Meta service:

Leader election

• feat: introduce leader and standby status for meta #6752
• feat: Failover meta node from follower to leader #6936
• feat: apply leader election with ETCD official APIs #6753
• feat: Meta HA fencing mechanism  #6786
• feat(meta): use serve_with_shutdown to gracefully shutdown services #6771
• Meta failover crashes Hummock  #6534

Service discovery

• feat: introduce meta cluster member service in meta #6754
• feat: support service discovery in meta client #6755
• feat: Meta clients can handle meta failover #6787 (part of 6755)
• Meta failover crashes RPC StreamingClient #6845 (To fix)

State persistence (DONE natually)

Test

• feat(meta): HA setup meta failover tests #6884
• feat(test): Change sim tests to match meta HA setup #7533

Others

• Export meta cluster information to grafana dashboard #7576
• Enable meta HA setup in operator
• Crate Meta K8s Service in cloud setup

[CAJan93]

Some helpful Resources. Please let me know if I forgot something.

• Design Docs
• ETCD Leader Election API Rust
• ETCD Concurrency API Reference
• Current leader registration in Meta
• Randomise leader election issue
• Alternative leadership election in K8s using sidecars
• Slack channel
• Flink HA setup

[CAJan93]

Looking forward to work on this task. However I am still working on a different kernel task and will be at KubeCon next week. I will start with this the earliest at the beginning of November.

[skyzh]

added a new task "support connecting to multiple meta-node on compute-node"

[CAJan93]

First draft version ready: Failover works, but Hummock crashes: #6534

[CAJan93]

support connecting to multiple meta-node on compute-node

@yezizp2012 I do not understand this part: My understanding is that the meta HA setup is a single-leader-system (see design docs). Therefore we only ever connect to one instance.
Is my understanding correct @fuyufjh ?

[yezizp2012]

support connecting to multiple meta-node on compute-node

@yezizp2012 I do not understand this part: My understanding is that the meta HA setup is a single-leader-system (see design docs). Therefore we only ever connect to one instance. Is my understanding correct @fuyufjh ?

IIUC, this should refers to the refactoring of the meta client: meta client should be able to connect and retrieve leader information from multi-meta nodes, which requires a get_leader_addr interface in meta.

[skyzh]

IIUC, this should refers to the refactoring of the meta client: meta client should be able to connect and retrieve leader information from multi-meta nodes, which requires a get_leader_addr interface in meta.

Yes. After that we can modify risedev to provide multiple meta addresses to compute nodes and frontend nodes. Currently it only picks the first one (see risedev warning)

[CAJan93]

Thank you very much for the clarifying. I would suggest that the change in the client and the change in the meta enabling a follower service which supports get_leader_addr will be done in separate PRs.

[CAJan93]

First version of the failover handling is done. Please have a look: #6466

There are a few things in the PR missing:

• e2e tests. I do not know how best to approach that. Please let me know if you have suggestions
• Currently we do busy-wait to do elections. ETCD also provides an update API afaik. I guess that this will require quite a reqwrite, which is why I would like to do this as future work.
• Meta crashes on failover because of Meta failover crashes Hummock  #6534. Will be addressed in future PR

[CAJan93]

added a new task "support connecting to multiple meta-node on compute-node"

Is this still up to date? I do not see the subtask @skyzh

[yezizp2012]

added a new task "support connecting to multiple meta-node on compute-node"

Is this still up to date? I do not see the subtask @skyzh

That should be a part of #6755 .

[CAJan93]

Tests are still running, but my guess is that #6771 is ready for review. This is a minor tasks overall, but merging it would still simplify further steps. Let me know if you have any objections or suggestions to the PR.

[CAJan93]

I currently have 2 PRs that are ready for review.

• feat(meta): use serve_with_shutdown to gracefully shutdown services #6771: Introduces graceful shutdown of services. This is only indirectly related to the HA setup. It will simplify further development.

• feat(meta): start follower svc if follower node #6792: Introduces the election mechanism, but does not introduce any failover (see features and limitations here). My expectation is that this does not affect our current setup, because we only ever deploy one meta node, which hopefully always becomes leader. PR reduces code coverage, because I delete a unittest which is no longer needed in a HA setup (also see #3398 )

CC @arkbriar

[CAJan93]

#6937 is ready for review. Please view the issue for features and limitations.

[CAJan93]

The fencing PR is ready for review. The CI pipeline is green. Also see #6786.

Maybe we can get this merged before New Years :)

[CAJan93]

Fencing is merged. Thank you very much for your guidance and approval @yezizp2012

[CAJan93]

feat(meta): Client failover is ready for review.

• Client is able to failover to new meta node
• Failover may require a few seconds. Meta nodes initially report stale leader information until election is finished