SVM Latency numbers differ significantly on Harvest 1.6 vs Harvest 2.0

[rahulguptajss]

Discussed in #940

^{Originally posted by jmg011 April 1, 2022}
We have some users using both harvest 1.6 dashboards and Harvest 2.0 dashboards. Some of the metrics I have set for Harvest 2.0 shows high latency at SVM level compared to harvest 1.6 metrics.

For example:

In Harvest 1.6 query, the SVM level latency is using vol_summary avg_latency -> This is 0.1 ms for an svm in my test case. This matches with the "statistics vserver latency show" command on the filer

_[netapp].[perf].[$Group].[$Cluster].[svm].[$SVM].[vol_summary].[avg_latency]_

In Harvest 2.0 query, for SVM level latency, I'm using below query. This query is reporting 25 ms latency for the same SVM. If I remove CIFS latency, it still reports almost 24 ms latency for my svm test case during the same time. This is not matching with the "statistics vserver latency show" or "statistics volume latency show" commands on the filer.

avg(svm_nfs_latency{svm=~"$SVM"}) + avg(svm_cifs_latency{svm=~"$SVM"})

I'm wondering why these two numbers differ so significantly?

In Harvest 2.0, when I calculate latency of all volumes on svm and sum it, it is still higher like 17ms for this test case SVM.

sum by (svm)(volume_avg_latency{svm="testcase_svm"})/1000

Are there different objects being used for both harvest 1.6 vs harvest 2.0 SVM latency calculations?

[rahulguptajss]

verified in release/22.05.0