Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Alertmanager: Add grafana config size limit #9402

Merged
merged 14 commits into from
Sep 27, 2024

Conversation

titolins
Copy link
Contributor

@titolins titolins commented Sep 25, 2024

Remote AM already has limits in place for it's own configuration. However, when using grafana compat mode, it doesn't have any. This aims to add a new config limit for grafana configuration.

Screenshot 2024-09-25 at 12 21 06

@titolins titolins requested review from a team and tacole02 as code owners September 25, 2024 10:11
Copy link
Contributor

@stevesg stevesg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good, couple of nits.

if err != nil {
maxBytesErr := &http.MaxBytesError{}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need for this to be a pointer (but see comment below, I don't think we need to keep it)

Suggested change
maxBytesErr := &http.MaxBytesError{}
var maxBytesErr http.MaxBytesError

Copy link
Contributor Author

@titolins titolins Sep 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does need to be a pointer actually. errors.As takes a pointer to the target which implements error. Since http.MaxBytesError has a pointer receiver on Error, the double indirection is required. I've learned this today TBH 😅
Removing the pointer and running the tests panics:
Screenshot 2024-09-25 at 13 57 35

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow, TIL. That's.... odd.

var input io.Reader
maxConfigSize := am.limits.AlertmanagerMaxGrafanaConfigSize(userID)
if maxConfigSize > 0 {
input = http.MaxBytesReader(w, r.Body, int64(maxConfigSize))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any practical difference between this and io.LimitReader which we use in api.go?

Copy link
Contributor Author

@titolins titolins Sep 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I saw the io.LimitReader example but I found that a bit confusing TBH - i.e. the reader returns an EOF (with nil error), so we have to explicitly check for the size. http.MaxReader will return an error of type MaxBytesError - IMO this has a better readability.
But I can switch to the LimitReader for consistency if that's preferred 👍

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm happy as-is, maybe we switch the LimitReader usage over in a follow up.

pkg/alertmanager/api_grafana.go Outdated Show resolved Hide resolved
@titolins titolins requested a review from stevesg September 25, 2024 13:28
@@ -3666,6 +3666,11 @@ The `limits` block configures default and per-tenant limits imposed by component
# CLI flag: -alertmanager.notification-rate-limit-per-integration
[alertmanager_notification_rate_limit_per_integration: <map of string to float64> | default = {}]

# Maximum size of the grafana configuration file for Alertmanager that tenant
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Maximum size of the grafana configuration file for Alertmanager that tenant
# Maximum size of the Grafana configuration file for Alertmanager that a tenant

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@@ -3666,6 +3666,11 @@ The `limits` block configures default and per-tenant limits imposed by component
# CLI flag: -alertmanager.notification-rate-limit-per-integration
[alertmanager_notification_rate_limit_per_integration: <map of string to float64> | default = {}]

# Maximum size of the grafana configuration file for Alertmanager that tenant
# can upload via Alertmanager API. 0 = no limit.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# can upload via Alertmanager API. 0 = no limit.
# can upload via the Alertmanager API. 0 = no limit.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done 👍

CHANGELOG.md Outdated
@@ -3236,6 +3236,8 @@ _Changes since `grafana/cortex-jsonnet` `1.9.0`._
}
```
* [FEATURE] Added multi-zone ingesters and store-gateways support. #1352 #1552
* [FEATURE] The following alertmanager limit was added: #9402
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to go at the top of the file (and not in the jsonnet section)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lol... Tks, done now 👍

@titolins titolins force-pushed the titolins/implement_config_size_limit branch from 9841950 to acf749e Compare September 26, 2024 08:03
@titolins titolins requested a review from stevesg September 26, 2024 08:04
@narqo
Copy link
Contributor

narqo commented Sep 27, 2024

The CHANGELOG has just been cut to prepare for the next release. Please rebase main and eventually move the CHANGELOG entry added / updated in this PR to the top of the CHANGELOG.md document. Thanks!

@titolins titolins force-pushed the titolins/implement_config_size_limit branch from acf749e to 9d9e501 Compare September 27, 2024 08:19
@titolins titolins force-pushed the titolins/implement_config_size_limit branch from 53c259a to ee1d491 Compare September 27, 2024 08:21
@titolins
Copy link
Contributor Author

The CHANGELOG has just been cut to prepare for the next release. Please rebase main and eventually move the CHANGELOG entry added / updated in this PR to the top of the CHANGELOG.md document. Thanks!

Awesome, tks! Done now 👍

Co-authored-by: Oleg Zaytsev <[email protected]>
@titolins titolins requested a review from colega September 27, 2024 08:33
CHANGELOG.md Outdated Show resolved Hide resolved
Copy link
Contributor

@stevesg stevesg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just a changelog nit.

Co-authored-by: Steve Simpson <[email protected]>
@stevesg stevesg merged commit ca7814e into main Sep 27, 2024
29 checks passed
@stevesg stevesg deleted the titolins/implement_config_size_limit branch September 27, 2024 18:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants