Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve PostingsForMatchers cache observability #10525

Merged
merged 2 commits into from
Jan 29, 2025
Merged

Improve PostingsForMatchers cache observability #10525

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
954fc9f
Improve PostingsForMatchers cache observability
pracucci Jan 28, 2025
7f5e26d
Fixed unit tests
pracucci Jan 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,15 +27,17 @@
* [ENHANCEMENT] Ingester: Hide tokens in ingester ring status page when ingest storage is enabled #10399
* [ENHANCEMENT] Ingester: add `active_series_additional_custom_trackers` configuration, in addition to the already existing `active_series_custom_trackers`. The `active_series_additional_custom_trackers` configuration allows you to configure additional custom trackers that get merged with `active_series_custom_trackers` at runtime. #10428
* [ENHANCEMENT] Query-frontend: Allow blocking raw http requests with the `blocked_requests` configuration. Requests can be blocked based on their path, method or query parameters #10484
* [ENHANCEMENT] Ingester: Added the following metrics exported by `PostingsForMatchers` cache: #10500
* [ENHANCEMENT] Ingester: Added the following metrics exported by `PostingsForMatchers` cache: #10500 #10525
* `cortex_ingester_tsdb_head_postings_for_matchers_cache_hits_total`
* `cortex_ingester_tsdb_head_postings_for_matchers_cache_misses_total`
* `cortex_ingester_tsdb_head_postings_for_matchers_cache_requests_total`
* `cortex_ingester_tsdb_head_postings_for_matchers_cache_skips_total`
* `cortex_ingester_tsdb_head_postings_for_matchers_cache_evictions_total`
* `cortex_ingester_tsdb_block_postings_for_matchers_cache_hits_total`
* `cortex_ingester_tsdb_block_postings_for_matchers_cache_misses_total`
* `cortex_ingester_tsdb_block_postings_for_matchers_cache_requests_total`
* `cortex_ingester_tsdb_block_postings_for_matchers_cache_skips_total`
* `cortex_ingester_tsdb_block_postings_for_matchers_cache_evictions_total`
* [ENHANCEMENT] Compactor: Shuffle users' order in `BlocksCleaner`. Prevents bucket indexes from going an extended period without cleanup during compactor restarts. #10513
* [BUGFIX] Distributor: Use a boolean to track changes while merging the ReplicaDesc components, rather than comparing the objects directly. #10185
* [BUGFIX] Querier: fix timeout responding to query-frontend when response size is very close to `-querier.frontend-client.grpc-max-send-msg-size`. #10154
Expand Down
2 changes: 1 addition & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -288,7 +288,7 @@ require (
sigs.k8s.io/yaml v1.4.0 // indirect
)

replace github.com/prometheus/prometheus => github.com/grafana/mimir-prometheus v0.0.0-20250123075837-0cc2978b5013
replace github.com/prometheus/prometheus => github.com/grafana/mimir-prometheus v0.0.0-20250128175949-8d210000df3a

// Replace memberlist with our fork which includes some fixes that haven't been
// merged upstream yet:
Expand Down
4 changes: 2 additions & 2 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1283,8 +1283,8 @@ github.com/grafana/gomemcache v0.0.0-20241016125027-0a5bcc5aef40 h1:1TeKhyS+pvzO
github.com/grafana/gomemcache v0.0.0-20241016125027-0a5bcc5aef40/go.mod h1:IGRj8oOoxwJbHBYl1+OhS9UjQR0dv6SQOep7HqmtyFU=
github.com/grafana/memberlist v0.3.1-0.20220714140823-09ffed8adbbe h1:yIXAAbLswn7VNWBIvM71O2QsgfgW9fRXZNR0DXe6pDU=
github.com/grafana/memberlist v0.3.1-0.20220714140823-09ffed8adbbe/go.mod h1:MS2lj3INKhZjWNqd3N0m3J+Jxf3DAOnAH9VT3Sh9MUE=
github.com/grafana/mimir-prometheus v0.0.0-20250123075837-0cc2978b5013 h1:70NFJ8OVRMCPc89vN520cTJd0vo/elnaXoF7q0I6c2M=
github.com/grafana/mimir-prometheus v0.0.0-20250123075837-0cc2978b5013/go.mod h1:KfyZCeyGxf5gvl6VZbrQsd400nJjGw+ygMEtDVZKIT4=
github.com/grafana/mimir-prometheus v0.0.0-20250128175949-8d210000df3a h1:4dVgKpv9kmX/HHYBVayiZBvN4ibLVtetZWVY1GCMhL8=
github.com/grafana/mimir-prometheus v0.0.0-20250128175949-8d210000df3a/go.mod h1:KfyZCeyGxf5gvl6VZbrQsd400nJjGw+ygMEtDVZKIT4=
github.com/grafana/opentracing-contrib-go-stdlib v0.0.0-20230509071955-f410e79da956 h1:em1oddjXL8c1tL0iFdtVtPloq2hRPen2MJQKoAWpxu0=
github.com/grafana/opentracing-contrib-go-stdlib v0.0.0-20230509071955-f410e79da956/go.mod h1:qtI1ogk+2JhVPIXVc6q+NHziSmy2W5GbdQZFUHADCBU=
github.com/grafana/prometheus-alertmanager v0.25.1-0.20240930132144-b5e64e81e8d3 h1:6D2gGAwyQBElSrp3E+9lSr7k8gLuP3Aiy20rweLWeBw=
Expand Down
28 changes: 28 additions & 0 deletions pkg/ingester/metrics_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -258,6 +258,13 @@ func TestTSDBMetrics(t *testing.T) {
cortex_ingester_tsdb_head_postings_for_matchers_cache_skips_total{reason="ineligible"} 0
cortex_ingester_tsdb_head_postings_for_matchers_cache_skips_total{reason="stale-cached-entry"} 0

# HELP cortex_ingester_tsdb_head_postings_for_matchers_cache_evictions_total Total number of evictions from the PostingsForMatchers cache.
# TYPE cortex_ingester_tsdb_head_postings_for_matchers_cache_evictions_total counter
cortex_ingester_tsdb_head_postings_for_matchers_cache_evictions_total{reason="max-bytes-reached"} 0
cortex_ingester_tsdb_head_postings_for_matchers_cache_evictions_total{reason="max-items-reached"} 0
cortex_ingester_tsdb_head_postings_for_matchers_cache_evictions_total{reason="ttl-expired"} 0
cortex_ingester_tsdb_head_postings_for_matchers_cache_evictions_total{reason="unknown"} 0

# HELP cortex_ingester_tsdb_block_postings_for_matchers_cache_hits_total Total number of postings lists returned from the PostingsForMatchers cache.
# TYPE cortex_ingester_tsdb_block_postings_for_matchers_cache_hits_total counter
cortex_ingester_tsdb_block_postings_for_matchers_cache_hits_total 0
Expand All @@ -275,6 +282,13 @@ func TestTSDBMetrics(t *testing.T) {
cortex_ingester_tsdb_block_postings_for_matchers_cache_skips_total{reason="canceled-cached-entry"} 0
cortex_ingester_tsdb_block_postings_for_matchers_cache_skips_total{reason="ineligible"} 0
cortex_ingester_tsdb_block_postings_for_matchers_cache_skips_total{reason="stale-cached-entry"} 0

# HELP cortex_ingester_tsdb_block_postings_for_matchers_cache_evictions_total Total number of evictions from the PostingsForMatchers cache.
# TYPE cortex_ingester_tsdb_block_postings_for_matchers_cache_evictions_total counter
cortex_ingester_tsdb_block_postings_for_matchers_cache_evictions_total{reason="max-bytes-reached"} 0
cortex_ingester_tsdb_block_postings_for_matchers_cache_evictions_total{reason="max-items-reached"} 0
cortex_ingester_tsdb_block_postings_for_matchers_cache_evictions_total{reason="ttl-expired"} 0
cortex_ingester_tsdb_block_postings_for_matchers_cache_evictions_total{reason="unknown"} 0
`))
require.NoError(t, err)
}
Expand Down Expand Up @@ -512,6 +526,13 @@ func TestTSDBMetricsWithRemoval(t *testing.T) {
cortex_ingester_tsdb_head_postings_for_matchers_cache_skips_total{reason="ineligible"} 0
cortex_ingester_tsdb_head_postings_for_matchers_cache_skips_total{reason="stale-cached-entry"} 0

# HELP cortex_ingester_tsdb_head_postings_for_matchers_cache_evictions_total Total number of evictions from the PostingsForMatchers cache.
# TYPE cortex_ingester_tsdb_head_postings_for_matchers_cache_evictions_total counter
cortex_ingester_tsdb_head_postings_for_matchers_cache_evictions_total{reason="max-bytes-reached"} 0
cortex_ingester_tsdb_head_postings_for_matchers_cache_evictions_total{reason="max-items-reached"} 0
cortex_ingester_tsdb_head_postings_for_matchers_cache_evictions_total{reason="ttl-expired"} 0
cortex_ingester_tsdb_head_postings_for_matchers_cache_evictions_total{reason="unknown"} 0

# HELP cortex_ingester_tsdb_block_postings_for_matchers_cache_hits_total Total number of postings lists returned from the PostingsForMatchers cache.
# TYPE cortex_ingester_tsdb_block_postings_for_matchers_cache_hits_total counter
cortex_ingester_tsdb_block_postings_for_matchers_cache_hits_total 0
Expand All @@ -529,6 +550,13 @@ func TestTSDBMetricsWithRemoval(t *testing.T) {
cortex_ingester_tsdb_block_postings_for_matchers_cache_skips_total{reason="canceled-cached-entry"} 0
cortex_ingester_tsdb_block_postings_for_matchers_cache_skips_total{reason="ineligible"} 0
cortex_ingester_tsdb_block_postings_for_matchers_cache_skips_total{reason="stale-cached-entry"} 0

# HELP cortex_ingester_tsdb_block_postings_for_matchers_cache_evictions_total Total number of evictions from the PostingsForMatchers cache.
# TYPE cortex_ingester_tsdb_block_postings_for_matchers_cache_evictions_total counter
cortex_ingester_tsdb_block_postings_for_matchers_cache_evictions_total{reason="max-bytes-reached"} 0
cortex_ingester_tsdb_block_postings_for_matchers_cache_evictions_total{reason="max-items-reached"} 0
cortex_ingester_tsdb_block_postings_for_matchers_cache_evictions_total{reason="ttl-expired"} 0
cortex_ingester_tsdb_block_postings_for_matchers_cache_evictions_total{reason="unknown"} 0
`))
require.NoError(t, err)
}
Expand Down
107 changes: 92 additions & 15 deletions vendor/github.com/prometheus/prometheus/tsdb/postings_for_matchers_cache.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions vendor/modules.txt
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.