Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

POC: Cost Attribution Proposal 2 #9733

Closed
wants to merge 42 commits into from
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
4e75934
Poc: cost attribution proposal 1.2
ying-jeanne Oct 24, 2024
0456d94
test update
ying-jeanne Oct 24, 2024
f4ebc07
address review comments and fix lint and test
ying-jeanne Oct 25, 2024
cb99a3f
fix lint and ci
ying-jeanne Oct 25, 2024
b0d3f0a
change max-cost-attribution-cardinality-per-user to 10k
ying-jeanne Oct 25, 2024
ebd6105
change custom registry path
ying-jeanne Oct 25, 2024
a438331
Add license for lint
ying-jeanne Oct 25, 2024
13a0b2c
add reset logics to handle overflow and recovery from overflow
ying-jeanne Oct 27, 2024
7f0b372
remove noop implementation
ying-jeanne Nov 5, 2024
3397578
Merge remote-tracking branch 'origin/main' into poc-cost-attribution-2
ying-jeanne Nov 5, 2024
50fa0ec
add new discarded sample metrics
ying-jeanne Nov 5, 2024
abdd0cc
fix test
ying-jeanne Nov 5, 2024
698a5c6
address comment to combine 2 config compare
ying-jeanne Nov 6, 2024
da6b00b
add logic for overflow
ying-jeanne Nov 7, 2024
2b5e3ff
improve tests for cost attribution service
ying-jeanne Nov 7, 2024
cb2a2b6
Don't hold labels from store-gateways in two forms, and don't convert…
grafanabot Nov 18, 2024
078e689
add per tenant cost attribution label limit
ying-jeanne Nov 18, 2024
3131a2f
Merge remote-tracking branch 'origin/main' into poc-cost-attribution-2
ying-jeanne Nov 18, 2024
4bf418a
update doc
ying-jeanne Nov 18, 2024
5e9e1c1
fix unittest
ying-jeanne Nov 18, 2024
bd3e112
fix ci
ying-jeanne Nov 18, 2024
cf16611
fix ci
ying-jeanne Nov 18, 2024
3c1f886
remove unrelated changes
ying-jeanne Nov 18, 2024
d0cb1f3
update purge logics
ying-jeanne Nov 18, 2024
5d4a2c4
fix ci
ying-jeanne Nov 18, 2024
6091493
fix ci
ying-jeanne Nov 18, 2024
7324a1d
update logic for overflow, purge other metrics than overflow
ying-jeanne Nov 18, 2024
5af48e4
add distributor benchmark test for push
ying-jeanne Nov 19, 2024
203689a
Improve logging at ha_tracker sync operation (#9958) (#9961)
grafanabot Nov 20, 2024
a2f009b
add benchmark in ingester
ying-jeanne Nov 20, 2024
791a75d
Merge remote-tracking branch 'origin/main' into poc-cost-attribution-2
ying-jeanne Nov 20, 2024
00d2092
refactory benchmark tests
ying-jeanne Nov 22, 2024
9fba531
MQE: fix issue where subqueries could return series with no points (#…
charleskorn Nov 25, 2024
12d7d79
fix service dependencies
ying-jeanne Nov 25, 2024
e0525e1
fix the distributor crashloop
ying-jeanne Nov 27, 2024
6f1baa7
Merge remote-tracking branch 'origin/r317' into poc-cost-attribution-2
ying-jeanne Nov 27, 2024
4fa3da0
Merge remote-tracking branch 'origin/main' into poc-cost-attribution-2
ying-jeanne Dec 2, 2024
dbd8224
clean up
ying-jeanne Dec 2, 2024
cff98ab
rephrase config description
ying-jeanne Dec 2, 2024
b0a3b57
improve doc
ying-jeanne Dec 2, 2024
ac55ac8
update benchmark
ying-jeanne Dec 3, 2024
51e7a64
remove auth
ying-jeanne Dec 3, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
MQE: fix issue where subqueries could return series with no points (#…
…9998) (#10003)

* MQE: fix issue where subqueries could return series with no points

* Add changelog entry

* Add more test cases

(cherry picked from commit 1afa9f6)

# Conflicts:
#	CHANGELOG.md
  • Loading branch information
charleskorn authored Nov 25, 2024
commit 9fba531aaba0ea74b8420ee30ca215e6cef8dcf5
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
* [CHANGE] Ingester: Change `-initial-delay` for circuit breakers to begin when the first request is received, rather than at breaker activation. #9842
* [CHANGE] Query-frontend: apply query pruning before query sharding instead of after. #9913
* [CHANGE] Ingester: remove experimental flags `-ingest-storage.kafka.ongoing-records-per-fetch` and `-ingest-storage.kafka.startup-records-per-fetch`. They are removed in favour of `-ingest-storage.kafka.max-buffered-bytes`. #9906
* [FEATURE] Querier: add experimental streaming PromQL engine, enabled with `-querier.query-engine=mimir`. #9367 #9368 #9398 #9399 #9403 #9417 #9418 #9419 #9420 #9482 #9504 #9505 #9507 #9518 #9531 #9532 #9533 #9553 #9558 #9588 #9589 #9639 #9641 #9642 #9651 #9664 #9681 #9717 #9719 #9724 #9874
* [FEATURE] Querier: add experimental streaming PromQL engine, enabled with `-querier.query-engine=mimir`. #9367 #9368 #9398 #9399 #9403 #9417 #9418 #9419 #9420 #9482 #9504 #9505 #9507 #9518 #9531 #9532 #9533 #9553 #9558 #9588 #9589 #9639 #9641 #9642 #9651 #9664 #9681 #9717 #9719 #9724 #9874 #9998
* [FEATURE] Distributor: Add support for `lz4` OTLP compression. #9763
* [FEATURE] Query-frontend: added experimental configuration options `query-frontend.cache-errors` and `query-frontend.results-cache-ttl-for-errors` to allow non-transient responses to be cached. When set to `true` error responses from hitting limits or bad data are cached for a short TTL. #9028
* [FEATURE] Query-frontend: add middleware to control access to specific PromQL experimental functions on a per-tenant basis. #9798
Expand Down
16 changes: 16 additions & 0 deletions pkg/streamingpromql/engine_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -904,6 +904,22 @@ func TestSubqueries(t *testing.T) {
},
Start: time.Unix(10, 0),
},
{
// A query where SeriesMetadata returns some series but evaluates to no samples should not return anything.
Query: `(metric > Inf)[20s:10s]`,
Start: time.Unix(30, 0),
Result: promql.Result{
Value: promql.Matrix{},
},
},
{
// A nested subquery with the same properties as above.
Query: `last_over_time((metric > Inf)[20s:10s])[30s:5s]`,
Start: time.Unix(30, 0),
Result: promql.Result{
Value: promql.Matrix{},
},
},
{
Query: "metric[20s:5s]",
Result: promql.Result{
Expand Down
6 changes: 6 additions & 0 deletions pkg/streamingpromql/query.go
Original file line number Diff line number Diff line change
Expand Up @@ -740,6 +740,12 @@ func (q *Query) populateMatrixFromRangeVectorOperator(ctx context.Context, o typ
return nil, err
}

if len(floats) == 0 && len(histograms) == 0 {
types.FPointSlicePool.Put(floats, q.memoryConsumptionTracker)
types.HPointSlicePool.Put(histograms, q.memoryConsumptionTracker)
continue
}

m = append(m, promql.Series{
Metric: s.Labels,
Floats: floats,
Expand Down
6 changes: 6 additions & 0 deletions pkg/streamingpromql/testdata/ours/subqueries.test
Original file line number Diff line number Diff line change
Expand Up @@ -127,3 +127,9 @@ eval range from 0 to 4m step 20s sum_over_time(sum_over_time(metric[2m:30s])[3m:

eval range from 0 to 4m step 3m sum_over_time(sum_over_time(sum_over_time(metric[2m:30s])[3m:15s])[4m:20s])
{} 0 86

eval range from 0 to 4m step 15s last_over_time((metric > Inf)[20s:10s])
# Should produce no results.

eval instant at 3m last_over_time((metric > Inf)[20s:10s])
# Should produce no results.
Loading