cluster: fix health metrics (#2987)

[ti-srebot]

cherry-pick #2987 to release-3.0
You can switch your code base to this Pull Request by using git-extras:

# In pd repo:
git pr https://github.com/tikv/pd/pull/3369

After apply modifications, you can push your change to this PR via:

git push [email protected]:ti-srebot/pd.git pr/3369:release-3.0-66c26169b813

---

Signed-off-by: disksing [email protected]

What problem does this PR solve?

• Fix collectHealthStatus() in server/cluster/cluster.go doesn't handle removed member #2964
• Fix the issue that healthy value is 0 and unhealthy value is 1 (should be 1 and 0)

What is changed and how it works?

• add metrics reset (after removing a member, you will still need to restart PD or transfer leader to reset metrics)
• fix wrong value

Check List

Tests

• Manual test (add detailed scripts or steps below)

1. start 3pd+1tikv cluster
2. check /metrics output

pd_cluster_health_status{name="pd1"} 1
pd_cluster_health_status{name="pd2"} 1
pd_cluster_health_status{name="pd3"} 1

3. delete member pd3
4. transfer leader
5. check /metrics output again, removed member is not displayed

pd_cluster_health_status{name="pd1"} 1
pd_cluster_health_status{name="pd2"} 1

Release note

• Fix the issue that member health metrics not correct

[ti-srebot]

@disksing you're already a collaborator in bot's repo.

[HunDunDM]

/merge

[ti-chi-bot]

@HunDunDM: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[HunDunDM]

/merge cancel

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• disksing

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by writing /lgtm in a comment.
Reviewer can cancel approval by writing /lgtm cancel in a comment.

[nolouch]

no critical, close.