Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs: add -frontend.enabled-ruler-query-stats to docs #6530

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

SungJin1212
Copy link
Contributor

This PR adds contents of #6504 to the Rule evaluations via query frontend guide docs (https://cortexmetrics.io/docs/guides/rule-evalutions-via-query-frontend).

Which issue(s) this PR fixes:
Fixes #

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@SungJin1212 SungJin1212 force-pushed the Improve-rule-evalutions-via-query-frontend branch 3 times, most recently from 34a9309 to d30c10c Compare January 22, 2025 03:19
@@ -29,6 +29,10 @@ In addition, you can configure gRPC client (Ruler -> Query Frontend) config, ple
You can configure the query response format via `-ruler.query-response-format`. It is used to retrieve query results from the Query Frontend.
The supported values are `protobuf` and `json`. We recommend using `protobuf`(default) because the retrieved query results containing native histograms are only supported on `protobuf`.

## Configure query stat log on the Query Frontend
You can enable the query stats log at the Query Frontend for queries coming from the Ruler via `-frontend.enabled-ruler-query-stats`.
It is disabled by default since the Ruler already leaves rule evaluation logs.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does it work in Ruler evaluation log now. It used to work because Ruler embeds its own querier to evaluate rules so it gathers query stats and pass via context.
But now since evaluation is done at QFE does Ruler evaluation log still provide the same information as of before?

Copy link
Contributor Author

@SungJin1212 SungJin1212 Jan 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh.. I just checked it, and the query stats logs values are 0 like:

cortex_ruler_query_seconds_total=0.003627083 query_wall_time_seconds=0s query_storage_wall_time_seconds=0s fetched_series_count=0 fetched_chunks_count=0 fetched_samples_count=0 fetched_chunks_bytes=0 fetched_data_bytes=0

And the Ruler query stats metrics tracked by queryStats also 0:

cortex_ruler_fetched_series_total, cortex_ruler_samples_total, cortex_ruler_fetched_chunks_bytes_total ...

I think we should fix it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added How to track query stat log and metric contents instead.

@SungJin1212 SungJin1212 force-pushed the Improve-rule-evalutions-via-query-frontend branch from d30c10c to 4c16499 Compare January 23, 2025 07:10
@SungJin1212 SungJin1212 force-pushed the Improve-rule-evalutions-via-query-frontend branch from 4c16499 to d043ebf Compare January 23, 2025 07:11
@SungJin1212 SungJin1212 force-pushed the Improve-rule-evalutions-via-query-frontend branch from d043ebf to 58edf03 Compare January 23, 2025 07:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants