Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

validation: allow more labels for _info metrics by default #10028

Merged
merged 9 commits into from
Nov 27, 2024

Conversation

krajorama
Copy link
Contributor

What this PR does

The info metrics are generally low cardinality even though there are many informational labels. This is to support conversion from OTLP attributes easier.

Which issue(s) this PR fixes or relates to

Related to prometheus/prometheus#15448

Checklist

  • Tests updated.
  • Documentation added.
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
  • N/A about-versioning.md updated with experimental features.

The info metrics are generally low cardinality even though there are
many informational labels. This is to support conversion from OTLP
attributes easier.

Signed-off-by: György Krajcsovits <[email protected]>
Signed-off-by: György Krajcsovits <[email protected]>
Signed-off-by: György Krajcsovits <[email protected]>
@krajorama krajorama marked this pull request as ready for review November 26, 2024 18:06
@krajorama krajorama requested review from tacole02 and a team as code owners November 26, 2024 18:06
Copy link
Contributor

@aknuds1 aknuds1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please see comments.

CHANGELOG.md Outdated Show resolved Hide resolved
docs/sources/mimir/manage/mimir-runbooks/_index.md Outdated Show resolved Hide resolved

This non-critical error occurs when Mimir receives a write request that contains an info series with a number of labels that exceed the configured limit.
An info series is a series where the metric name ends in `_info`.
The limit protects the system’s stability from potential abuse or mistakes. To configure the limit on a per-tenant basis, use the `-validation.max-label-names-per-info-series` option.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To configure per tenant, you need YAML don't you rather than the flag:

Suggested change
The limit protects the system’s stability from potential abuse or mistakes. To configure the limit on a per-tenant basis, use the `-validation.max-label-names-per-info-series` option.
The limit protects the system’s stability from potential abuse or mistakes. To configure the limit on a per-tenant basis, use the `validation.max_label_names_per_info_series` option.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ack, I'll update this and where I copy pasted from above as well

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also it becomes limits.max_label_names_per_info_series I think

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually there's 10 places where we use -validation... flag and zero where we write yaml option. Do you want to change all?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I certainly wouldn't change existing code as part of this PR. I just don't think it makes sense to refer to a flag, when you have to use YAML per-tenant.

pkg/distributor/validate.go Show resolved Hide resolved
Copy link
Contributor

@tacole02 tacole02 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work, @krajorama . I left a few minor comments and suggestions.

### err-mimir-max-label-names-per-info-series

This non-critical error occurs when Mimir receives a write request that contains an info series with a number of labels that exceed the configured limit.
An info series is a series where the metric name ends in `_info`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is info series an abbreviation for information series or is it really just called an info series? If it's an abbreviation, we should spell it out except when using a code-formatted label.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not an abbreviation, these are called info series/metrics everywhere


This non-critical error occurs when Mimir receives a write request that contains an info series with a number of labels that exceed the configured limit.
An info series is a series where the metric name ends in `_info`.
The limit protects the system’s stability from potential abuse or mistakes. To configure the limit on a per-tenant basis, use the `-validation.max-label-names-per-info-series` option.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: I think misuse has a more positive connotation than abuse.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The doc uses abuse 21 times, and misuse zero times. Let's change in a different PR if we want to.

docs/sources/mimir/manage/mimir-runbooks/_index.md Outdated Show resolved Hide resolved
krajorama and others added 4 commits November 27, 2024 08:59
@aknuds1
Copy link
Contributor

aknuds1 commented Nov 27, 2024

Could you rebase on main?

Copy link
Contributor

@aknuds1 aknuds1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you're referencing the wrong flag in the error.

pkg/distributor/validate.go Outdated Show resolved Hide resolved
Copy link
Contributor

@aknuds1 aknuds1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks

@krajorama krajorama merged commit 444fe55 into main Nov 27, 2024
29 checks passed
@krajorama krajorama deleted the krajo/allow-more-labels-on-info-metrics branch November 27, 2024 17:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants