Fix success/error rate metrics

[linkous8]

• Add "sum by(kubernetes_pod_name)" to success and error rate metrics so
  they are averaged by the number of pods instead of averaged by the
  number of pods times the number of "envoy_response_code_class"es

[linear]

ENG-549 Opsani Dev/Prometheus; success_rate and error_rate queries have incorrect averaging logic

Currently, the Opsani Dev connector configures success_rate and error_rate queries like such:

avg(rate(envoy_cluster_upstream_rq_xx{opsani_role!="tuning", envoy_response_code_class=~"2|3"}[1m]))
avg(rate(envoy_cluster_upstream_rq_xx{opsani_role!="tuning", envoy_response_code_class=~"4|5"}[1m]))

The expected behavior is that the avg function will provide us with the success/error rate averaged by the number of pods. However, running the query without avg demonstrates why that is not the case:

The above screenshot shows that the returned rate values are bucketed by kubernetes_pod_name AND by envoy_response_code_class which doubles the divisor used by the avg resulting in erroneous success/error rate counts when compared to total_request_rate.

The proposed solution is to wrap the rate query with sum by (kubernetes_pod_name)(…) so that the avg divisor is the number of pods as expected

Relevant metric configuration:

servox/servo/connectors/opsani_dev.py

Lines 172 to 194 in 44b1dc2

	servo.connectors.prometheus.PrometheusMetric(
	"main_success_rate",
	servo.types.Unit.requests_per_second,
	query='avg(rate(envoy_cluster_upstream_rq_xx{opsani_role!="tuning", envoy_response_code_class=~"2|3"}[1m]))',
	),
	servo.connectors.prometheus.PrometheusMetric(
	"tuning_success_rate",
	servo.types.Unit.requests_per_second,
	query='avg(rate(envoy_cluster_upstream_rq_xx{opsani_role="tuning", envoy_response_code_class=~"2|3"}[1m]))',
	absent=servo.connectors.prometheus.AbsentMetricPolicy.zero
	),
	servo.connectors.prometheus.PrometheusMetric(
	"main_error_rate",
	servo.types.Unit.requests_per_second,
	query='avg(rate(envoy_cluster_upstream_rq_xx{opsani_role!="tuning", envoy_response_code_class=~"4|5"}[1m]))',
	absent=servo.connectors.prometheus.AbsentMetricPolicy.zero
	),
	servo.connectors.prometheus.PrometheusMetric(
	"tuning_error_rate",
	servo.types.Unit.requests_per_second,
	query='avg(rate(envoy_cluster_upstream_rq_xx{opsani_role="tuning", envoy_response_code_class=~"4|5"}[1m]))',
	absent=servo.connectors.prometheus.AbsentMetricPolicy.zero
	),

[rstarmer]

This is great, and was likley much needed!