Performance degradation of NFS after upgrading Kubernetes from 1.20 to 1.22

[hyponet]

After upgrading our Kubernetes cluster from version 1.20 to 1.22, we have observed a version discrepancy, which is manifested in the form of a decrease in NFS performance. After some investigation, we suspect it might be related to this change:
#2787

To provide a brief description of our environment, our cluster has nearly 10,000 Pods, all of which are using the same NAS service via PV. On a single node, we can have more than 400 nfs mountpoints. Currently, the metrics collection includes NFS mount points, leading to a large number of statfs operations and consequently, slow NFS metadata operations.

Would it be possible to adjust the metrics collection for mountpoints so that data is only collected once for the same device?

[fusida]

cc @bobbypage @gjkim42

[mowangdk]

related issue & pr:

kubernetes/kubernetes#98009

#2787

@pacoxu PTAL

[pacoxu]

• kube 1.20.0 is using cadvisor v0.38.5. (v0.38.8 for kube v1.20.15)
• kube 1.22.0 is using cadvisor v0.39.2. (v0.39.4 for kube v1.22.17)

https://github.com/kubernetes/kubernetes/blob/v1.20.0/go.mod#L56C29-L56C29
https://github.com/kubernetes/kubernetes/blob/v1.22.0/go.mod#L49

diff in v0.38.5...v0.39.4

[chenk008]

For one nfs device, the information about used and available in different directory should be unified. For the same nfs device, cadvisor can stat nfs device once and reuse it's info in GetFsInfoForPath.