Alertmanager: Skip starting the Alertmanager for Grafana tenants unless they have a promoted, non-default configuration #10491
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
santihernandezc
force-pushed
the
santihernandezc/dont_run_alertmanager_default_config
branch
from
5f2c772 to
c2e7145
Compare
santihernandezc
changed the title
santihernandezc
force-pushed
the
santihernandezc/dont_run_alertmanager_default_config
branch
from
a32f18d to
27219a7
Compare
santihernandezc
force-pushed
the
santihernandezc/dont_run_alertmanager_default_config
branch
from
27219a7 to
21c4fc9
Compare
santihernandezc
force-pushed
the
santihernandezc/dont_run_alertmanager_default_config
branch
from
ffbc07f to
1da54f9
Compare
santihernandezc
changed the title
|
💻 Deploy preview deleted. |
santihernandezc
commented
Jan 22, 2025
Comment on lines
+703
to
+704
| _, initSkipped := amInitSkipped[userID] | ||
| if !exists || initSkipped { |
There was a problem hiding this comment.
If the skipped Alertmanager was already running, I'm deleting its data and adding it to the map of Alertmanagers to stop.
Comment on lines
+731
to
+736
| // If the Grafana configuration is either default, not promoted, or empty, use the Mimir configuration. | ||
| if !cfgs.Grafana.Promoted || cfgs.Grafana.Default || cfgs.Grafana.RawConfig == "" { | ||
| level.Debug(am.logger).Log("msg", "using mimir config", "user", cfgs.Mimir.User) | ||
| isGrafanaTenant := am.cfg.GrafanaAlertmanagerTenantSuffix != "" && strings.HasSuffix(cfgs.Mimir.User, am.cfg.GrafanaAlertmanagerTenantSuffix) | ||
| return cfg, !isGrafanaTenant, nil | ||
| } |
There was a problem hiding this comment.
I decided to remove the granular debug log lines to shorten the function.
titolins
reviewed
Jan 23, 2025
cmd/mimir/help-all.txt.tmpl
Outdated
| @@ -239,6 +239,8 @@ Usage of ./cmd/mimir/mimir: | |||
| Enables periodic cleanup of alertmanager stateful data (notification logs and silences) from object storage. When enabled, data is removed for any tenant that does not have a configuration. (default true) | |||
| -alertmanager.grafana-alertmanager-compatibility-enabled | |||
| [experimental] Enable routes to support the migration and operation of the Grafana Alertmanager. | |||
| -alertmanager.grafana-tenant-suffix string | |||
There was a problem hiding this comment.
Just a super nitpick, but the flag name is bit generic IMO. Feels like it's missing an action there - e.g. skip-init-for-grafana-tenant-suffix. WDYT?
tacole02
approved these changes
Jan 23, 2025
santihernandezc
deleted the
santihernandezc/dont_run_alertmanager_default_config
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
We currently run an Alertmanager for each tenant in the database, no matter whether their configurations are usable or not. This results in many unused Alertmanagers consuming resources.
This PR adds the option to skip initializing Alertmanagers for Grafana Alertmanager tenants without a usable Grafana Alertmanager configuration.
The new
-alertmanager.grafana-alertmanager-conditionally-skip-tenant-suffixargument takes a string that will be compared against tenants' IDs. If it matches, the Alertmanager will only be initialized if there's a promoted, non-default, non-empty Grafana Alertmanager configuration for that tenant.Testing locally
I tested this by spinning up two Alertmanagers:
mimir-1built frommainmimir-1-newbuild from this branchI added 600 tenants to each one:
Both Alertmanagers owned 600 tenants, but only 200 of them were active in the Alertmanager build from this branch.
Tenants owned per AM.
Tenant config reloads per Alertmanager.
The memory footprint was reduced by roughly half in the Alertmanager filtering out the Grafana Alertmanager tenants without a usable configuration.
avg_over_time(go_memstats_heap_inuse_bytes{job=~"mimir-1.*"}[1m])CPU time in the new Alertmanager was decreased by roughly ~45%, goroutines by ~55%.
sum by (job) (rate(process_cpu_seconds_total{job=~"mimir-1.*"}[5m]))go_goroutines{job=~"mimir-1.*"}