Skip to content

Commit

Permalink
Improve PostingsForMatchers cache observability (#10525)
Browse files Browse the repository at this point in the history 
* Improve PostingsForMatchers cache observability

Signed-off-by: Marco Pracucci <[email protected]>

* Fixed unit tests

Signed-off-by: Marco Pracucci <[email protected]>

---------

Signed-off-by: Marco Pracucci <[email protected]>
  • Loading branch information
@pracucci
pracucci authored Jan 29, 2025
1 parent 266a393 commit 1712177
Show file tree
Hide file tree
Showing 6 changed files with 128 additions and 21 deletions.
4 changes: 3 additions & 1 deletion CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,15 +27,17 @@
* [ENHANCEMENT] Ingester: Hide tokens in ingester ring status page when ingest storage is enabled #10399
* [ENHANCEMENT] Ingester: add `active_series_additional_custom_trackers` configuration, in addition to the already existing `active_series_custom_trackers`. The `active_series_additional_custom_trackers` configuration allows you to configure additional custom trackers that get merged with `active_series_custom_trackers` at runtime. #10428
* [ENHANCEMENT] Query-frontend: Allow blocking raw http requests with the `blocked_requests` configuration. Requests can be blocked based on their path, method or query parameters #10484
* [ENHANCEMENT] Ingester: Added the following metrics exported by `PostingsForMatchers` cache: #10500
* [ENHANCEMENT] Ingester: Added the following metrics exported by `PostingsForMatchers` cache: #10500 #10525
* `cortex_ingester_tsdb_head_postings_for_matchers_cache_hits_total`
* `cortex_ingester_tsdb_head_postings_for_matchers_cache_misses_total`
* `cortex_ingester_tsdb_head_postings_for_matchers_cache_requests_total`
* `cortex_ingester_tsdb_head_postings_for_matchers_cache_skips_total`
* `cortex_ingester_tsdb_head_postings_for_matchers_cache_evictions_total`
* `cortex_ingester_tsdb_block_postings_for_matchers_cache_hits_total`
* `cortex_ingester_tsdb_block_postings_for_matchers_cache_misses_total`
* `cortex_ingester_tsdb_block_postings_for_matchers_cache_requests_total`
* `cortex_ingester_tsdb_block_postings_for_matchers_cache_skips_total`
* `cortex_ingester_tsdb_block_postings_for_matchers_cache_evictions_total`
* [ENHANCEMENT] Compactor: Shuffle users' order in `BlocksCleaner`. Prevents bucket indexes from going an extended period without cleanup during compactor restarts. #10513
* [BUGFIX] Distributor: Use a boolean to track changes while merging the ReplicaDesc components, rather than comparing the objects directly. #10185
* [BUGFIX] Querier: fix timeout responding to query-frontend when response size is very close to `-querier.frontend-client.grpc-max-send-msg-size`. #10154
Expand Down
2 changes: 1 addition & 1 deletion go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -288,7 +288,7 @@ require (
sigs.k8s.io/yaml v1.4.0 // indirect
)

replace github.com/prometheus/prometheus => github.com/grafana/mimir-prometheus v0.0.0-20250123075837-0cc2978b5013
replace github.com/prometheus/prometheus => github.com/grafana/mimir-prometheus v0.0.0-20250128175949-8d210000df3a

// Replace memberlist with our fork which includes some fixes that haven't been
// merged upstream yet:
Expand Down
4 changes: 2 additions & 2 deletions go.sum
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1283,8 +1283,8 @@ github.com/grafana/gomemcache v0.0.0-20241016125027-0a5bcc5aef40 h1:1TeKhyS+pvzO
github.com/grafana/gomemcache v0.0.0-20241016125027-0a5bcc5aef40/go.mod h1:IGRj8oOoxwJbHBYl1+OhS9UjQR0dv6SQOep7HqmtyFU=
github.com/grafana/memberlist v0.3.1-0.20220714140823-09ffed8adbbe h1:yIXAAbLswn7VNWBIvM71O2QsgfgW9fRXZNR0DXe6pDU=
github.com/grafana/memberlist v0.3.1-0.20220714140823-09ffed8adbbe/go.mod h1:MS2lj3INKhZjWNqd3N0m3J+Jxf3DAOnAH9VT3Sh9MUE=
github.com/grafana/mimir-prometheus v0.0.0-20250123075837-0cc2978b5013 h1:70NFJ8OVRMCPc89vN520cTJd0vo/elnaXoF7q0I6c2M=
github.com/grafana/mimir-prometheus v0.0.0-20250123075837-0cc2978b5013/go.mod h1:KfyZCeyGxf5gvl6VZbrQsd400nJjGw+ygMEtDVZKIT4=
github.com/grafana/mimir-prometheus v0.0.0-20250128175949-8d210000df3a h1:4dVgKpv9kmX/HHYBVayiZBvN4ibLVtetZWVY1GCMhL8=
github.com/grafana/mimir-prometheus v0.0.0-20250128175949-8d210000df3a/go.mod h1:KfyZCeyGxf5gvl6VZbrQsd400nJjGw+ygMEtDVZKIT4=
github.com/grafana/opentracing-contrib-go-stdlib v0.0.0-20230509071955-f410e79da956 h1:em1oddjXL8c1tL0iFdtVtPloq2hRPen2MJQKoAWpxu0=
github.com/grafana/opentracing-contrib-go-stdlib v0.0.0-20230509071955-f410e79da956/go.mod h1:qtI1ogk+2JhVPIXVc6q+NHziSmy2W5GbdQZFUHADCBU=
github.com/grafana/prometheus-alertmanager v0.25.1-0.20240930132144-b5e64e81e8d3 h1:6D2gGAwyQBElSrp3E+9lSr7k8gLuP3Aiy20rweLWeBw=
Expand Down
28 changes: 28 additions & 0 deletions pkg/ingester/metrics_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -258,6 +258,13 @@ func TestTSDBMetrics(t *testing.T) {
cortex_ingester_tsdb_head_postings_for_matchers_cache_skips_total{reason="ineligible"} 0
cortex_ingester_tsdb_head_postings_for_matchers_cache_skips_total{reason="stale-cached-entry"} 0
# HELP cortex_ingester_tsdb_head_postings_for_matchers_cache_evictions_total Total number of evictions from the PostingsForMatchers cache.
# TYPE cortex_ingester_tsdb_head_postings_for_matchers_cache_evictions_total counter
cortex_ingester_tsdb_head_postings_for_matchers_cache_evictions_total{reason="max-bytes-reached"} 0
cortex_ingester_tsdb_head_postings_for_matchers_cache_evictions_total{reason="max-items-reached"} 0
cortex_ingester_tsdb_head_postings_for_matchers_cache_evictions_total{reason="ttl-expired"} 0
cortex_ingester_tsdb_head_postings_for_matchers_cache_evictions_total{reason="unknown"} 0
# HELP cortex_ingester_tsdb_block_postings_for_matchers_cache_hits_total Total number of postings lists returned from the PostingsForMatchers cache.
# TYPE cortex_ingester_tsdb_block_postings_for_matchers_cache_hits_total counter
cortex_ingester_tsdb_block_postings_for_matchers_cache_hits_total 0
Expand All @@ -275,6 +282,13 @@ func TestTSDBMetrics(t *testing.T) {
cortex_ingester_tsdb_block_postings_for_matchers_cache_skips_total{reason="canceled-cached-entry"} 0
cortex_ingester_tsdb_block_postings_for_matchers_cache_skips_total{reason="ineligible"} 0
cortex_ingester_tsdb_block_postings_for_matchers_cache_skips_total{reason="stale-cached-entry"} 0
# HELP cortex_ingester_tsdb_block_postings_for_matchers_cache_evictions_total Total number of evictions from the PostingsForMatchers cache.
# TYPE cortex_ingester_tsdb_block_postings_for_matchers_cache_evictions_total counter
cortex_ingester_tsdb_block_postings_for_matchers_cache_evictions_total{reason="max-bytes-reached"} 0
cortex_ingester_tsdb_block_postings_for_matchers_cache_evictions_total{reason="max-items-reached"} 0
cortex_ingester_tsdb_block_postings_for_matchers_cache_evictions_total{reason="ttl-expired"} 0
cortex_ingester_tsdb_block_postings_for_matchers_cache_evictions_total{reason="unknown"} 0
`))
require.NoError(t, err)
}
Expand Down Expand Up @@ -512,6 +526,13 @@ func TestTSDBMetricsWithRemoval(t *testing.T) {
cortex_ingester_tsdb_head_postings_for_matchers_cache_skips_total{reason="ineligible"} 0
cortex_ingester_tsdb_head_postings_for_matchers_cache_skips_total{reason="stale-cached-entry"} 0
# HELP cortex_ingester_tsdb_head_postings_for_matchers_cache_evictions_total Total number of evictions from the PostingsForMatchers cache.
# TYPE cortex_ingester_tsdb_head_postings_for_matchers_cache_evictions_total counter
cortex_ingester_tsdb_head_postings_for_matchers_cache_evictions_total{reason="max-bytes-reached"} 0
cortex_ingester_tsdb_head_postings_for_matchers_cache_evictions_total{reason="max-items-reached"} 0
cortex_ingester_tsdb_head_postings_for_matchers_cache_evictions_total{reason="ttl-expired"} 0
cortex_ingester_tsdb_head_postings_for_matchers_cache_evictions_total{reason="unknown"} 0
# HELP cortex_ingester_tsdb_block_postings_for_matchers_cache_hits_total Total number of postings lists returned from the PostingsForMatchers cache.
# TYPE cortex_ingester_tsdb_block_postings_for_matchers_cache_hits_total counter
cortex_ingester_tsdb_block_postings_for_matchers_cache_hits_total 0
Expand All @@ -529,6 +550,13 @@ func TestTSDBMetricsWithRemoval(t *testing.T) {
cortex_ingester_tsdb_block_postings_for_matchers_cache_skips_total{reason="canceled-cached-entry"} 0
cortex_ingester_tsdb_block_postings_for_matchers_cache_skips_total{reason="ineligible"} 0
cortex_ingester_tsdb_block_postings_for_matchers_cache_skips_total{reason="stale-cached-entry"} 0
# HELP cortex_ingester_tsdb_block_postings_for_matchers_cache_evictions_total Total number of evictions from the PostingsForMatchers cache.
# TYPE cortex_ingester_tsdb_block_postings_for_matchers_cache_evictions_total counter
cortex_ingester_tsdb_block_postings_for_matchers_cache_evictions_total{reason="max-bytes-reached"} 0
cortex_ingester_tsdb_block_postings_for_matchers_cache_evictions_total{reason="max-items-reached"} 0
cortex_ingester_tsdb_block_postings_for_matchers_cache_evictions_total{reason="ttl-expired"} 0
cortex_ingester_tsdb_block_postings_for_matchers_cache_evictions_total{reason="unknown"} 0
`))
require.NoError(t, err)
}
Expand Down
107 changes: 92 additions & 15 deletions vendor/github.com/prometheus/prometheus/tsdb/postings_for_matchers_cache.go
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions vendor/modules.txt
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

0 comments on commit 1712177

Please sign in to comment.