[bugfix] workqueue metrics are mising

[RainbowMango]

What happened:
We received reports that the controller's work queue metrics can not be emitted. (See #5945 for more details, thanks to @CharlesQQ for reporting this).

This issue occurred before and was fixed in release 1.0 in #945, and then we made some subsequent impairments in #2899 and #3012. (Thanks to @Garrybest for doing this)

However after #2998 and #4706, this issue was introduced again in release-1.10.

This issue is used to track any relevant fix to ensure the problem is fully resolved without introducing any potential risk.

Root Cause:
TBD

Iteration tasks:

• Revist #2998, fix potential risk or twist solutions.
• Clarify the usage of Service Reference in the ResourceInterpreterWebhookConfiguration API.
• Clarify the usage documentation of Service Reference in the ResourceInterpreterWebhookConfiguration. (#5969, @XiShanYongYe-Chang)
• Minimum fix on master. (Fix the issue of missing workqueue metrics in the Karmada controller #5972, @XiShanYongYe-Chang )
• Backport to release branches. (release-1.10~release-1.12) (Cherrypick #5972: fix the issue of missing workqueue metrics in the Karmada controller #5979)
• Cleanup workaround made on release-1.0. (remove workaround code for resolving workqueue metrics loss by modifying import order #5945, @chaosi-zju)
• Introduce tests to cover the metrics. (introduce tests to check whether workqueue metrics exist #6003, @chaosi-zju)
• Expose metrics of karmada-metrics-adapter. (expose metrics of karmada-metrics-adapter #6013, @chaosi-zju)

Other Potential risks

• karmada-webhook panic in case of invalid configuration. (#5960, @XiShanYongYe-Chang)
• Cannot access karmada-interpreter-webhook by configuring Service in ResourceInterpreterWebhookConfiguration #5958

References:

• From Go 1.21, the program initialization order was specified more precisely. See Go 1.21 release notes for more details.

Echo from the release notes:

Package initialization order is now specified more precisely. The new algorithm is:

• Sort all packages by import path.
• Repeat until the list of packages is empty:
  • Find the first package in the list for which all imports are already initialized.
  • Initialize that package and remove it from the list.

• Go Program initialization
• Issue in controller-runtime: Add a new metrics to indicates the the current queue length. kubernetes-sigs/controller-runtime#1969
• Issue in Kubernetes: There is a conflict between the metrics of controller-runtime and component-base, the metric workqueue_depth of controller-runtime repository not take effect kubernetes/kubernetes#128326

[chaosi-zju]

the PR #6003 works for the task Introduce tests to cover the metrics

[RainbowMango]

Got it.

[chaosi-zju]

I found another sub issue:

https://github.com/karmada-io/karmada/blob/master/pkg/estimator/server/metrics/metrics.go

you can refer to this file and find that karmada-schedule-estimator has some custom metrics, but they are also missed.

---

so does scheduler's custom metrics:

https://github.com/karmada-io/karmada/blob/master/pkg/estimator/server/metrics/metrics.go

[chaosi-zju]

you can refer to this file and find that karmada-schedule-estimator has some custom metrics, but they are also missed.

The actual situation is that these custom metrics are not available right after installing the two components (not even zero values).

They can only be viewed once actually resources are scheduled.

Besides, ginkgo does not guarantee order of specs. in fact it prefers to enable randomized spec orders to prevent spec pollution, refer to onsi/ginkgo#1165 (comment)

[RainbowMango]

The actual situation is that these custom metrics are not available right after installing the two components (not even zero values).
They can only be viewed once actually resources are scheduled.

Just a guess, this is due to the metrics needing some labels to be initiated. Like result and type in estimating_request_total.

Show an example of the metrics?

[chaosi-zju]

Show an example of the metrics?

# HELP karmada_scheduler_schedule_attempts_total Number of attempts to schedule a ResourceBinding or ClusterResourceBinding
# TYPE karmada_scheduler_schedule_attempts_total counter
karmada_scheduler_schedule_attempts_total{result="scheduled",schedule_type="ReconcileSchedule"} 5
# HELP karmada_scheduler_scheduling_algorithm_duration_seconds Scheduling algorithm latency in seconds(exclude scale scheduler)
# TYPE karmada_scheduler_scheduling_algorithm_duration_seconds histogram
karmada_scheduler_scheduling_algorithm_duration_seconds_bucket{schedule_step="AssignReplicas",le="0.001"} 5
karmada_scheduler_scheduling_algorithm_duration_seconds_bucket{schedule_step="AssignReplicas",le="0.002"} 5
karmada_scheduler_scheduling_algorithm_duration_seconds_bucket{schedule_step="AssignReplicas",le="0.004"} 5
karmada_scheduler_scheduling_algorithm_duration_seconds_bucket{schedule_step="AssignReplicas",le="0.008"} 5
...

[RainbowMango]

Thanks. That's the reason.

[stulzq]

v1.9.0 can not find metric workqueue_adds_total

[XiShanYongYe-Chang]

Hi @stulzq, thanks for your feedback~

[chaosi-zju]

v1.9.0 can not find metric workqueue_adds_total

Hi @stulzq , from previous investigation:
In Karmada v1.9.9, we still use go v1.20, the Karmada controller didn't miss workqueue metrics.
From Karmada v1.10.0, we use go v1.21, start introducing this problem, metrics missed.

Can you please tell me how you installed karmada?

[chaosi-zju]

v1.9.0 can not find metric workqueue_adds_total

There is another situation, if you use karmada v1.9.0, but your go version is go v1.21 or higher, you will also have problems.

[husnialhamdani]

/assign