Skip to content

Commit

Permalink
Merge pull request #353 from grafana/add-per-route-p99-latency-in-rul…
Browse files Browse the repository at this point in the history
…er-config-api

Added 'Per route p99 latency' to ruler configuration API
  • Loading branch information
pracucci authored Jul 5, 2021
2 parents 27078c6 + 6f612e0 commit 2d4fd4e
Show file tree
Hide file tree
Showing 2 changed files with 12 additions and 2 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
* [ENHANCEMENT] Dashboards: defined container functions for common resources panels: containerDiskWritesPanel, containerDiskReadsPanel, containerDiskSpaceUtilization. #331
* [ENHANCEMENT] cortex-mixin: Added `alert_excluded_routes` config to exclude specific routes from alerts. #338
* [ENHANCEMENT] Added `CortexMemcachedRequestErrors` alert. #346
* [ENHANCEMENT] Ruler dashboard: added "Per route p99 latency" panel in the "Configuration API" row. #353
* [BUGFIX] Fixed `CortexIngesterHasNotShippedBlocks` alert false positive in case an ingester instance had ingested samples in the past, then no traffic was received for a long period and then it started receiving samples again. #308
* [BUGFIX] Alertmanager: fixed `--alertmanager.cluster.peers` CLI flag passed to alertmanager when HA is enabled. #329
* [BUGFIX] Fixed `CortexInconsistentRuntimeConfig` metric. #335
Expand Down
13 changes: 11 additions & 2 deletions cortex-mixin/dashboards/ruler.libsonnet
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
local utils = import 'mixin-utils/utils.libsonnet';

(import 'dashboard-utils.libsonnet') {
local ruler_config_api_routes_re = 'api_prom_rules.*|api_prom_api_v1_(rules|alerts)',

rulerQueries+:: {
ruleEvaluations: {
Expand Down Expand Up @@ -106,11 +107,19 @@ local utils = import 'mixin-utils/utils.libsonnet';
$.row('Configuration API (gateway)')
.addPanel(
$.panel('QPS') +
$.qpsPanel('cortex_request_duration_seconds_count{%s, route=~"api_prom_rules.*|api_prom_api_v1_(rules|alerts)"}' % $.jobMatcher($._config.job_names.gateway))
$.qpsPanel('cortex_request_duration_seconds_count{%s, route=~"%s"}' % [$.jobMatcher($._config.job_names.gateway), ruler_config_api_routes_re])
)
.addPanel(
$.panel('Latency') +
utils.latencyRecordingRulePanel('cortex_request_duration_seconds', $.jobSelector($._config.job_names.gateway) + [utils.selector.re('route', 'api_prom_rules.*|api_prom_api_v1_(rules|alerts)')])
utils.latencyRecordingRulePanel('cortex_request_duration_seconds', $.jobSelector($._config.job_names.gateway) + [utils.selector.re('route', ruler_config_api_routes_re)])
)
.addPanel(
$.panel('Per route p99 Latency') +
$.queryPanel(
'histogram_quantile(0.99, sum by (route, le) (cluster_job_route:cortex_request_duration_seconds_bucket:sum_rate{%s, route=~"%s"}))' % [$.jobMatcher($._config.job_names.gateway), ruler_config_api_routes_re],
'{{ route }}'
) +
{ yaxes: $.yaxes('s') }
)
)
.addRow(
Expand Down

0 comments on commit 2d4fd4e

Please sign in to comment.