metrics: grafana dashboards support multiple cluster (#1340)

[ti-srebot]

cherry-pick #1340 to release-4.0
You can switch your code base to this Pull Request by using git-extras:

# In ticdc repo:
git pr https://github.com/pingcap/ticdc/pull/1350

After apply modifications, you can push your change to this PR via:

git push [email protected]:ti-srebot/ticdc.git pr/1350:release-4.0-5e38a512fff6

---

What problem does this PR solve?

Many users(come from DBaaS、operator、tiup) monitor multiple TiDB clusters in one Grafana, this PR make Grafana dashboards support multiple clusters and not affect single cluster usage. Cluster variable is hidden by default.

What is changed and how it works?

• add a tidb_cluster label in all expr
• add cluster variable in Grafana templating

How it Works:
load it to your Grafana :))

Single Cluster:
No change

Multiple Cluster:
Construct a label that can uniquely identify the cluster. in tidb_operator, you can use {namespace}-{cluster_name} as your {tidb_cluster} variable.
add this configuration to your prometheus config

relabel_configs:
  - source_labels:
      - namespace
      - name
    separator: "-"
    target_label: tidb_cluster

Check List

Tests

• No code

Side effects

• No effects

Related changes

• metrics: grafana dashboards support multiple clusters tidb#22503

Release note

• metrics: grafana dashboards support multiple clusters

[ti-srebot]

/run-all-tests

[zier-one]

/merge

[amyangfei]

/merge

[ti-srebot]

Your auto merge job has been accepted, waiting for:

• 1351

[ti-srebot]

/run-all-tests

[ti-srebot]

@ti-srebot merge failed.

[amyangfei]

/merge

[ti-srebot]

/run-all-tests