Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dashboards: defined container functions for common resources panels #331

Merged
merged 3 commits into from
Jun 22, 2021
Merged

Dashboards: defined container functions for common resources panels #331

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e5a38db
refactor: resources dashboard comtainer functions
darrenjaneczek Jun 16, 2021
87e9af1
revert: matching spacing format of main
darrenjaneczek Jun 22, 2021
150e5f3
lint: white noise
darrenjaneczek Jun 22, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
* [CHANGE] Increased `CortexBadRuntimeConfig` alert severity to `critical` and removed support for `cortex_overrides_last_reload_successful` metric (was removed in Cortex 1.3.0). #335
* [ENHANCEMENT] cortex-mixin: Make `cluster_namespace_deployment:kube_pod_container_resource_requests_{cpu_cores,memory_bytes}:sum` backwards compatible with `kube-state-metrics` v2.0.0. #317
* [ENHANCEMENT] Added documentation text panels and descriptions to reads and writes dashboards. #324
* [ENHANCEMENT] Dashboards: defined container functions for common resources panels: containerDiskWritesPanel, containerDiskReadsPanel, containerDiskSpaceUtilization. #331
* [BUGFIX] Fixed `CortexIngesterHasNotShippedBlocks` alert false positive in case an ingester instance had ingested samples in the past, then no traffic was received for a long period and then it started receiving samples again. #308
* [BUGFIX] Alertmanager: fixed `--alertmanager.cluster.peers` CLI flag passed to alertmanager when HA is enabled. #329
* [BUGFIX] Fixed `CortexInconsistentRuntimeConfig` metric. #335
Expand Down
20 changes: 3 additions & 17 deletions cortex-mixin/dashboards/alertmanager-resources.libsonnet
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,30 +52,16 @@ local utils = import 'mixin-utils/utils.libsonnet';
.addRow(
$.row('Disk')
.addPanel(
$.panel('Writes') +
$.queryPanel(
'sum by(%s, %s, device) (rate(node_disk_written_bytes_total[$__rate_interval])) + %s' % [$._config.per_node_label, $._config.per_instance_label, $.filterNodeDiskContainer('alertmanager')],
'{{%s}} - {{device}}' % $._config.per_instance_label
) +
$.stack +
{ yaxes: $.yaxes('Bps') },
$.containerDiskWritesPanel('Writes', 'alertmanager'),
)
.addPanel(
$.panel('Reads') +
$.queryPanel(
'sum by(%s, %s, device) (rate(node_disk_read_bytes_total[$__rate_interval])) + %s' % [$._config.per_node_label, $._config.per_instance_label, $.filterNodeDiskContainer('alertmanager')],
'{{%s}} - {{device}}' % $._config.per_instance_label
) +
$.stack +
{ yaxes: $.yaxes('Bps') },
$.containerDiskReadsPanel('Reads', 'alertmanager'),
)
)
.addRow(
$.row('')
.addPanel(
$.panel('Disk Space Utilization') +
$.queryPanel('max by(persistentvolumeclaim) (kubelet_volume_stats_used_bytes{%s} / kubelet_volume_stats_capacity_bytes{%s}) and count by(persistentvolumeclaim) (kube_persistentvolumeclaim_labels{%s,label_name="alertmanager"})' % [$.namespaceMatcher(), $.namespaceMatcher(), $.namespaceMatcher()], '{{persistentvolumeclaim}}') +
{ yaxes: $.yaxes('percentunit') },
$.containerDiskSpaceUtilization('Disk Space Utilization', 'alertmanager'),
)
),
}
28 changes: 7 additions & 21 deletions cortex-mixin/dashboards/compactor-resources.libsonnet
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,27 +28,13 @@ local utils = import 'mixin-utils/utils.libsonnet';
.addRow(
$.row('Disk')
.addPanel(
$.panel('Disk Writes') +
$.queryPanel(
'sum by(%s, %s, device) (rate(node_disk_written_bytes_total[$__rate_interval])) + %s' % [$._config.per_node_label, $._config.per_instance_label, $.filterNodeDiskContainer('compactor')],
'{{%s}} - {{device}}' % $._config.per_instance_label
) +
$.stack +
{ yaxes: $.yaxes('Bps') },
)
.addPanel(
$.panel('Disk Reads') +
$.queryPanel(
'sum by(%s, %s, device) (rate(node_disk_read_bytes_total[$__rate_interval])) + %s' % [$._config.per_node_label, $._config.per_instance_label, $.filterNodeDiskContainer('compactor')],
'{{%s}} - {{device}}' % $._config.per_instance_label
) +
$.stack +
{ yaxes: $.yaxes('Bps') },
)
.addPanel(
$.panel('Disk Space Utilization') +
$.queryPanel('max by(persistentvolumeclaim) (kubelet_volume_stats_used_bytes{%s} / kubelet_volume_stats_capacity_bytes{%s}) and count by(persistentvolumeclaim) (kube_persistentvolumeclaim_labels{%s,label_name="compactor"})' % [$.namespaceMatcher(), $.namespaceMatcher(), $.namespaceMatcher()], '{{persistentvolumeclaim}}') +
{ yaxes: $.yaxes('percentunit') },
$.containerDiskWritesPanel('Disk Writes', 'compactor'),
)
.addPanel(
$.containerDiskReadsPanel('Disk Reads', 'compactor'),
)
.addPanel(
$.containerDiskSpaceUtilization('Disk Space Utilization', 'compactor'),
)
) + {
templating+: {
Expand Down
67 changes: 67 additions & 0 deletions cortex-mixin/dashboards/dashboard-utils.libsonnet
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,73 @@ local utils = import 'mixin-utils/utils.libsonnet';
containerNetworkTransmitBytesPanel(instanceName)::
$.containerNetworkPanel('Transmit Bandwidth', 'container_network_transmit_bytes_total', instanceName),

containerDiskWritesPanel(title, containerName)::
$.panel(title) +
$.queryPanel(
|||
sum by(%s, %s, device) (
rate(
node_disk_written_bytes_total[$__rate_interval]
)
)
+
%s
||| % [
$._config.per_node_label,
$._config.per_instance_label,
$.filterNodeDiskContainer(containerName),
],
'{{%s}} - {{device}}' % $._config.per_instance_label
) +
$.stack +
{ yaxes: $.yaxes('Bps') },

containerDiskReadsPanel(title, containerName)::
$.panel(title) +
$.queryPanel(
|||
sum by(%s, %s, device) (
rate(
node_disk_read_bytes_total[$__rate_interval]
)
) + %s
||| % [
$._config.per_node_label,
$._config.per_instance_label,
$.filterNodeDiskContainer(containerName),
],
'{{%s}} - {{device}}' % $._config.per_instance_label
) +
$.stack +
{ yaxes: $.yaxes('Bps') },

containerDiskSpaceUtilization(title, containerName)::
$.panel(title) +
$.queryPanel(
|||
max by(persistentvolumeclaim) (
kubelet_volume_stats_used_bytes{%(namespace)s} /
kubelet_volume_stats_capacity_bytes{%(namespace)s}
)
and
count by(persistentvolumeclaim) (
kube_persistentvolumeclaim_labels{
%(namespace)s,
%(label)s
}
)
||| % {
namespace: $.namespaceMatcher(),
label: $.containerLabelMatcher(containerName),
}, '{{persistentvolumeclaim}}'
) +
{ yaxes: $.yaxes('percentunit') },

containerLabelMatcher(containerName)::
if containerName == 'ingester'
then 'label_name=~"ingester.*"'
else 'label_name="%s"' % containerName,

goHeapInUsePanel(title, jobName)::
$.panel(title) +
$.queryPanel(
Expand Down
20 changes: 3 additions & 17 deletions cortex-mixin/dashboards/reads-resources.libsonnet
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,27 +103,13 @@ local utils = import 'mixin-utils/utils.libsonnet';
std.member($._config.storage_engine, 'blocks'),
$.row('')
.addPanel(
$.panel('Disk Writes') +
$.queryPanel(
'sum by(%s, %s, device) (rate(node_disk_written_bytes_total[$__rate_interval])) + %s' % [$._config.per_node_label, $._config.per_instance_label, $.filterNodeDiskContainer('store-gateway')],
'{{%s}} - {{device}}' % $._config.per_instance_label
) +
$.stack +
{ yaxes: $.yaxes('Bps') },
$.containerDiskWritesPanel('Disk Writes', 'store-gateway'),
)
.addPanel(
$.panel('Disk Reads') +
$.queryPanel(
'sum by(%s, %s, device) (rate(node_disk_read_bytes_total[$__rate_interval])) + %s' % [$._config.per_node_label, $._config.per_instance_label, $.filterNodeDiskContainer('store-gateway')],
'{{%s}} - {{device}}' % $._config.per_instance_label
) +
$.stack +
{ yaxes: $.yaxes('Bps') },
$.containerDiskReadsPanel('Disk Reads', 'store-gateway'),
)
.addPanel(
$.panel('Disk Space Utilization') +
$.queryPanel('max by(persistentvolumeclaim) (kubelet_volume_stats_used_bytes{%s} / kubelet_volume_stats_capacity_bytes{%s}) and count by(persistentvolumeclaim) (kube_persistentvolumeclaim_labels{%s,label_name="store-gateway"})' % [$.namespaceMatcher(), $.namespaceMatcher(), $.namespaceMatcher()], '{{persistentvolumeclaim}}') +
{ yaxes: $.yaxes('percentunit') },
$.containerDiskSpaceUtilization('Disk Space Utilization', 'store-gateway'),
)
) + {
templating+: {
Expand Down
20 changes: 3 additions & 17 deletions cortex-mixin/dashboards/writes-resources.libsonnet
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,27 +56,13 @@ local utils = import 'mixin-utils/utils.libsonnet';
.addRow(
$.row('')
.addPanel(
$.panel('Disk Writes') +
$.queryPanel(
'sum by(%s, %s, device) (rate(node_disk_written_bytes_total[$__rate_interval])) + %s' % [$._config.per_node_label, $._config.per_instance_label, $.filterNodeDiskContainer('ingester')],
'{{%s}} - {{device}}' % $._config.per_instance_label
) +
$.stack +
{ yaxes: $.yaxes('Bps') },
$.containerDiskWritesPanel('Disk Writes', 'ingester')
)
.addPanel(
$.panel('Disk Reads') +
$.queryPanel(
'sum by(%s, %s, device) (rate(node_disk_read_bytes_total[$__rate_interval])) + %s' % [$._config.per_node_label, $._config.per_instance_label, $.filterNodeDiskContainer('ingester')],
'{{%s}} - {{device}}' % $._config.per_instance_label
) +
$.stack +
{ yaxes: $.yaxes('Bps') },
$.containerDiskReadsPanel('Disk Reads', 'ingester')
)
.addPanel(
$.panel('Disk Space Utilization') +
$.queryPanel('max by(persistentvolumeclaim) (kubelet_volume_stats_used_bytes{%s} / kubelet_volume_stats_capacity_bytes{%s}) and count by(persistentvolumeclaim) (kube_persistentvolumeclaim_labels{%s,label_name=~"ingester.*"})' % [$.namespaceMatcher(), $.namespaceMatcher(), $.namespaceMatcher()], '{{persistentvolumeclaim}}') +
{ yaxes: $.yaxes('percentunit') },
$.containerDiskSpaceUtilization('Disk Space Utilization', 'ingester'),
)
)
+ {
Expand Down