add metrics for recreate/update resource event when sync work status

[chaosi-zju]

What type of PR is this?

/kind feature

What this PR does / why we need it:

add metrics for recreate/update resource event when sync work status

detail backgroud:

When I create a sa and a sa type secret in control plane, yet, I only propagate the secret to member clusters, then the secret will be deleted immediately by sa-controller of the member cluster. In turn the work_status_controller will watch the deletion event and recreate the resource to cluster, this cycle goes on and never stops.

Normally, there shouldn't be a large number of recreate events, so, I want to record a metric of the recreate event which can monitor the abnormal situation.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Test report:

# HELP recreate_resource_to_cluster Number of recreating operation of the resource to a target member cluster. By the result, 'error' means a resource recreated failed. Otherwise 'success'. Cluster means the target member cluster.
# TYPE recreate_resource_to_cluster counter
recreate_resource_to_cluster{apigroup="v1",cluster="member1",kind="Secret",result="success"} 810
recreate_resource_to_cluster{apigroup="v1",cluster="member2",kind="Secret",result="success"} 809
# HELP update_resource_to_cluster Number of updating operation of the resource to a target member cluster. By the result, 'error' means a resource updated failed. Otherwise 'success'. Cluster means the target member cluster.
# TYPE update_resource_to_cluster counter
update_resource_to_cluster{apigroup="rbac.authorization.k8s.io/v1",cluster="member1",kind="ClusterRole",result="success"} 2
update_resource_to_cluster{apigroup="rbac.authorization.k8s.io/v1",cluster="member2",kind="ClusterRole",result="success"} 1

Does this PR introduce a user-facing change?:

karmada-controller-manager: add metrics for recreate/update resource event when sync work status

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 0% with 12 lines in your changes missing coverage. Please review.

Project coverage is 28.26%. Comparing base (bc1c96e) to head (32850c4).
Report is 10 commits behind head on master.

Files	Patch %	Lines
pkg/controllers/status/work_status_controller.go	0.00%	6 Missing ⚠️
pkg/metrics/resource.go	0.00%	6 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5247      +/-   ##
==========================================
- Coverage   28.26%   28.26%   -0.01%     
==========================================
  Files         632      632              
  Lines       43732    43741       +9     
==========================================
+ Hits        12360    12362       +2     
- Misses      30470    30479       +9     
+ Partials      902      900       -2     

Flag	Coverage Δ
unittests	28.26% <0.00%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[XiShanYongYe-Chang]

Hi @chaunceyjiang @whitewindmills would you like to have a look?
/cc @chaunceyjiang @whitewindmills

[XiShanYongYe-Chang]

How about updating the release note like this:

karmada-controller-manager/karmada-agent: add metrics for recreate/update resource event when sync work status

[XiShanYongYe-Chang]

Thanks~
/lgtm

[chaosi-zju]

/retest

[chaosi-zju]

hi @whitewindmills, do you have any further comments?

[whitewindmills]

/lgtm

[XiShanYongYe-Chang]

/approve

[karmada-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: XiShanYongYe-Chang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [XiShanYongYe-Chang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[XiShanYongYe-Chang]

Can you help add those new metrics in the website?