Grafana does not show all nodes in Dashboard

[CAJan93]

Describe the bug

I started the setup with multiple ETCD and meta nodes (see this PR). the Node Count does not list all the node. It omits the ETCD and the meta nodes. Is this on purpose? Could we change this please?

Screenshot from Grafana. I started a new cluster using the multi-node setup at 14:40.

Expected behavior

Node Count should list all nodes in the cluster.

[Gun9niR]

Will #5943 modify ClusterManager to track MetaNode?

[CAJan93]

Will #5943 modify ClusterManager to track MetaNode?

I don't think that that is in the scope. Maybe we should keep this as its own backlog item. What do you think?

[Gun9niR]

Ok. As long as each meta node can be notified of other meta nodes' existence, we can track MetaNode in Grafana. But for third-party dependencies such as etcd, I don't think it's the developer dashboard's job to track them, because:

• Not all third-party components offer a way to monitor the number of instances. Even if they do, the methods might differ a lot, requiring distinct code logic for each.
• The developer dashboard is mainly for profiling and debugging RisingWave's own nodes, i.e. compute/frontend/compactor/meta node. Normally, we shouldn't expect failures from third-party components, so tracking them would be unnecessary.

[CAJan93]

Why does a meta node need to be informed about the other nodes existence?

[skyzh]

Third party nodes are not included. We only count RisingWave components here.

[fuyufjh]

Before HA Meta we only have 1 meta node.

[CAJan93]

Third party nodes are not included. We only count RisingWave components here.

Meta counts as third party? Is this dashboard only about "kernel"? Should we not also keep track of meta nodes? They are vital for our service.

[yezizp2012]

Third party nodes are not included. We only count RisingWave components here.

Meta counts as third party? Is this dashboard only about "kernel"? Should we not also keep track of meta nodes? They are vital for our service.

Meta should need to be taken into account, ETCD does not. I believe after HA Meta we should track meta nodes as well in the dashboard. All meta nodes should be queryable from any meta nodes and managed by ClusterManager. I think this should be naturally supported in HA Meta.

[Gun9niR]

Meta should need to be taken into account, ETCD does not.

The kernel includes compute/compactor/meta/frontend nodes.

I believe after HA Meta we should track meta nodes as well in the dashboard. All meta nodes should be queryable from any meta nodes and managed by ClusterManager. I think this should be naturally supported in HA Meta.

This is what I mean by

As long as each meta node can be notified of other meta nodes' existence, we can track MetaNode in Grafana.

[yezizp2012]

Will be handled in #7576 .