frontend: Add option to "spin off" subqueries as actual range queries

[julienduchesne]

Issue: #10023

Description:

The feature was developed by inspiring myself from the query sharding feature. The queries are mapped into either downstream queries or subqueries. Both types of queries are run and the results are fed back into prometheus' engine and the result is calculated in the frontend.

The way it optimizes subqueries, is that range queries are passed back to the frontend and all of the range queries optimizations are applied (query sharding, splitting + caching)

Configuration:

This is a new feature that is completely isolated within a new middleware so it shouldn't affect current functionality of the frontend.
For safety, it requires two configurations to be enabled:

• --query-frontend.spin-off-instant-subqueries-to-url=<url> on the frontend. This should be set to the URL of the frontend for optimal performance. The range queries are load balanced across frontends
• instant_queries_with_subquery_spin_off in tenant configs. These are regexp patterns that allow us to match individual queries (or all of them). This will allow us to opt-in queries to enable the feature gradually

Performance impact:

The AST mapper only selects queries that are susceptible to be improved, others are just passed on to the next middleware.
For the queries that are improved, results can be up to 50x faster. When a query is selected, the worst cases I've seen are ~equal or a bit better in performance to unmodified queries.
Further tests will be done and the mapper may be improved to detect cases that aren't optimal

PromQL results impact:

None detected from all the tests I've done. The results were always identical.

[github-actions]

💻 Deploy preview available: https://deploy-preview-mimir-10460-zb444pucvq-vp.a.run.app/docs/mimir/latest/

[chencs]

Overall, I think this makes sense. I think it is a first for Mimir to see Queryable implemented in the frontend, so it's worth getting another person to comment on that pattern.

[julienduchesne]

Overall, I think this makes sense. I think it is a first for Mimir to see Queryable implemented in the frontend, so it's worth getting another person to comment on that pattern.

Query sharding does the same

[charleskorn]

Rather than a list of expressions, what if we only allowed a single expression? Anyone who wants to allow multiple expressions can chain them together in the expression with something like (...|...|...).

Also, would it be possible to parse and compile the expression once, rather than doing this for every query request?

[julienduchesne]

I'd like to leave it as-is because it might otherwise get complicated. I want to gradually add more queries to it, without modifying the current inclusions, the slice works better for that.

For the regexp matcher, I use prometheus' FastRegexMatcher (https://github.com/prometheus/prometheus/blob/7bc11dcb0664/pkg/labels/regexp.go#L29), just like the blocked queries feature. It has a cache: grafana/mimir-prometheus#465

[charleskorn]

Will these requests run concurrently? If not, seems like it might be worthwhile to do so.

[julienduchesne]

Reason: I don't want to unexpectedly blow up the query path

The case where the steps will go over 11000 is when querying far into the past, since even with a 30s resolution, 11000 steps is a 92h range, so most queries are supported with a single range query

If we always have only one inflight query, there's no risk of killing the query path, because if someone does an aggregation for a whole year, then it could potentially spawn dozens of range queries. By doing one at a time, we'll eventually time out the query instead of killing Mimir.