Merge pull request #230 from grafana/add-avg-and-max-blocks-per-tenant

Added avg and max blocks / tenant

CHANGELOG.md

@@ -7,12 +7,12 @@
 * [ENHANCEMENT] Introduce a resources dashboard for the Alertmanager. #219
 * [ENHANCEMENT] Improves query visibility in the Ruler Dashboard for both chunks and blocks storage. #226
 * [ENHANCEMENT] Add query-scheduler to dashboards. Add alert for queries stuck in scheduler. #228
-* [ENHANCEMENT] Improved blocks storage observability: #224
+* [ENHANCEMENT] Improved blocks storage observability: #224 #230
   - Cortex / Writes: added current number of tenants in the cluster
   - Cortex / Writes Resources: added ingester disk read/writes/utilisation
   - Cortex / Reads Resources: added store-gateway disk read/writes/utilisation
   - Cortex / Queries: added "Lazy loaded index-headers" and "Index-header lazy load duration"
-  - Cortex / Compactor: added "Tenants compaction progress"
+  - Cortex / Compactor: added "Tenants compaction progress", "Average blocks / tenant" and "Tenants with largest number of blocks"
   - Alerts: added "CortexMemoryMapAreasTooHigh"
 * [BUGFIX] Fixed workingset memory panel while rolling out a StatefulSet. #229
 

cortex-mixin/dashboards/compactor.libsonnet

@@ -5,7 +5,7 @@ local utils = import 'mixin-utils/utils.libsonnet';
     ($.dashboard('Cortex / Compactor') + { uid: '9c408e1d55681ecb8a22c9fab46875cc' })
     .addClusterSelectorTemplates()
     .addRow(
-      $.row('Compactions')
+      $.row('Summary')
       .addPanel(
         $.textPanel('', |||
           - **Per-instance runs**: number of times a compactor instance triggers a compaction across all tenants its shard manage.
@@ -52,6 +52,23 @@ local utils = import 'mixin-utils/utils.libsonnet';
         $.latencyPanel('prometheus_tsdb_compaction_duration_seconds', '{%s}' % $.jobMatcher('compactor'))
       )
     )
+    .addRow(
+      $.row('')
+      .addPanel(
+        $.textPanel('', |||
+          - **Average blocks / tenant**: the average number of blocks per tenant.
+          - **Tenants with largest number of blocks**: the 10 tenants with the largest number of blocks.
+        |||),
+      )
+      .addPanel(
+        $.panel('Average blocks / tenant') +
+        $.queryPanel('avg(max by(user) (cortex_bucket_blocks_count{%s}))' % $.jobMatcher('compactor'), 'avg'),
+      )
+      .addPanel(
+        $.panel('Tenants with largest number of blocks') +
+        $.queryPanel('topk(10, max by(user) (cortex_bucket_blocks_count{%s}))' % $.jobMatcher('compactor'), '{{user}}'),
+      )
+    )
     .addRow(
       $.row('Garbage Collector')
       .addPanel(