cluster: fix health metrics (#2987)

[ti-srebot]

cherry-pick #2987 to release-4.0
You can switch your code base to this Pull Request by using git-extras:

# In pd repo:
git pr https://github.com/tikv/pd/pull/3368

After apply modifications, you can push your change to this PR via:

git push [email protected]:ti-srebot/pd.git pr/3368:release-4.0-66c26169b813

---

Signed-off-by: disksing [email protected]

What problem does this PR solve?

• Fix collectHealthStatus() in server/cluster/cluster.go doesn't handle removed member #2964
• Fix the issue that healthy value is 0 and unhealthy value is 1 (should be 1 and 0)

What is changed and how it works?

• add metrics reset (after removing a member, you will still need to restart PD or transfer leader to reset metrics)
• fix wrong value

Check List

Tests

• Manual test (add detailed scripts or steps below)

1. start 3pd+1tikv cluster
2. check /metrics output

pd_cluster_health_status{name="pd1"} 1
pd_cluster_health_status{name="pd2"} 1
pd_cluster_health_status{name="pd3"} 1

3. delete member pd3
4. transfer leader
5. check /metrics output again, removed member is not displayed

pd_cluster_health_status{name="pd1"} 1
pd_cluster_health_status{name="pd2"} 1

Release note

• Fix the issue that member health metrics not correct

[ti-srebot]

@disksing you're already a collaborator in bot's repo.

[lhy1024]

plz fix conflict

[nolouch]

/run-all-tests

[rleungx]

/run-all-tests

[codecov]

Codecov Report

Merging #3368 (879155e) into release-4.0 (d00a83a) will increase coverage by 0.04%.
The diff coverage is 28.57%.

@@               Coverage Diff               @@
##           release-4.0    #3368      +/-   ##
===============================================
+ Coverage        74.71%   74.76%   +0.04%     
===============================================
  Files              209      209              
  Lines            19681    19683       +2     
===============================================
+ Hits             14705    14716      +11     
+ Misses            3644     3636       -8     
+ Partials          1332     1331       -1     

Flag	Coverage Δ
unittests	74.76% <28.57%> (+0.04%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
server/cluster/cluster.go	79.62% <28.57%> (+0.05%)	⬆️
pkg/dashboard/adapter/manager.go	81.52% <0.00%> (-3.27%)	⬇️
pkg/etcdutil/etcdutil.go	76.47% <0.00%> (-1.48%)	⬇️
client/client.go	69.03% <0.00%> (-0.51%)	⬇️
server/member/leader.go	68.34% <0.00%> (-0.46%)	⬇️
server/grpc_service.go	59.47% <0.00%> (-0.41%)	⬇️
server/server.go	74.12% <0.00%> (+0.49%)	⬆️
pkg/btree/btree.go	86.26% <0.00%> (+0.96%)	⬆️
pkg/mock/mockhbstream/mockhbstream.go	92.59% <0.00%> (+3.70%)	⬆️
server/id/id.go	80.95% <0.00%> (+4.76%)	⬆️
... and 1 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d00a83a...879155e. Read the comment docs.

[rleungx]

/merge

[ti-chi-bot]

@rleungx: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: dcd6804

[ti-chi-bot]

@ti-srebot: Your PR has out-of-dated, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.