Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add metric name to metadata error #4363

Merged

Conversation

gouthamve
Copy link
Contributor

Currently our errors look like this:

level=warn ts=2021-07-14T09:33:02.743961441Z caller=ingester.go:681 org_id=XXXX traceID=3224439dfe4ee37c msg="failed to ingest some metadata" err="per-metric metadata limit of 10 exceeded, please contact administrator to raise it (local limit: 10 global limit: 0 actual local limit: 10)"
level=warn ts=2021-07-14T09:33:02.743862184Z caller=ingester.go:681 org_id=XXXX traceID=3224439dfe4ee37c msg="failed to ingest some metadata" err="per-metric metadata limit of 10 exceeded, please contact administrator to raise it (local limit: 10 global limit: 0 actual local limit: 10)"
level=warn ts=2021-07-14T09:33:02.743728652Z caller=ingester.go:681 org_id=XXXX traceID=3224439dfe4ee37c msg="failed to ingest some metadata" err="per-metric metadata limit of 10 exceeded, please contact administrator to raise it (local limit: 10 global limit: 0 actual local limit: 10)"
level=warn ts=2021-07-14T09:33:02.743496095Z caller=ingester.go:681 org_id=XXXX traceID=3224439dfe4ee37c msg="failed to ingest some metadata" err="per-metric metadata limit of 10 exceeded, please contact administrator to raise it (local limit: 10 global limit: 0 actual local limit: 10)"

Knowing the series name would be helpful.

@gouthamve gouthamve force-pushed the add-name-to-metadata-error branch from 2ace109 to 5c82e62 Compare July 14, 2021 12:38
Copy link
Contributor

@pracucci pracucci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. It's not the most optimized way to do it (for sample ingestion errors we did a large work to optimize the error path) but metadata ingestion errors didn't show to be a bottleneck so far. I'm also suprised there's no unit test error to change, but CI will eventually show it in case.

Signed-off-by: Goutham Veeramachaneni <[email protected]>
@gouthamve gouthamve force-pushed the add-name-to-metadata-error branch from 5c82e62 to 02d6f64 Compare July 14, 2021 12:54
@gouthamve gouthamve merged commit 728cfbe into cortexproject:master Jul 14, 2021
@gouthamve gouthamve deleted the add-name-to-metadata-error branch July 14, 2021 16:36
alvinlin123 pushed a commit to ac1214/cortex that referenced this pull request Jan 14, 2022
Signed-off-by: Goutham Veeramachaneni <[email protected]>
Signed-off-by: Alvin Lin <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants