Merge pull request #15223 from jan--f/release-3.0.0-rc.0

CHANGELOG.md

@@ -3,10 +3,16 @@
 ## unreleased
 
 * [CHANGE] Scraping: Remove implicit fallback to the Prometheus text format in case of invalid/missing Content-Type and fail the scrape instead. Add ability to specify a `fallback_scrape_protocol` in the scrape config. #15136
-* [CHANGE] Remote-write: default enable_http2 to false.
-* [ENHANCEMENT] Scraping, rules: handle targets reappearing, or rules moving group, when out-of-order is enabled. #14710 
-- [BUGFIX] PromQL: Fix stddev+stdvar aggregations to always ignore native histograms. #14941
-- [BUGFIX] PromQL: Fix stddev+stdvar aggregations to treat Infinity consistently. #14941
+* [CHANGE] Remote-write: default enable_http2 to false. #15219
+* [CHANGE] Scraping: normalize "le" and "quantile" label values upon ingestion. #15164
+* [CHANGE] Scraping: config `scrape_classic_histograms` was renamed to `always_scrape_classic_histograms`. #15178
+* [CHANGE] Config: remove expand-external-labels flag, expand external labels env vars by default. #14657
+* [CHANGE] Disallow configuring AM with the v1 api. #13883
+* [ENHANCEMENT] Scraping, rules: handle targets reappearing, or rules moving group, when out-of-order is enabled. #14710
+* [ENHANCEMENT] Tools: add debug printouts to promtool rules unit testing #15196
+* [ENHANCEMENT] Scraping: support Created-Timestamp feature on native histograms. #14694
+* [BUGFIX] PromQL: Fix stddev+stdvar aggregations to always ignore native histograms. #14941
+* [BUGFIX] PromQL: Fix stddev+stdvar aggregations to treat Infinity consistently. #14941
 
 ## 3.0.0-beta.1 / 2024-10-09
 
@@ -42,10 +48,6 @@ Release 3.0.0-beta.0 includes new features such as a brand new UI and UTF-8 supp
 
 As is traditional with a beta release, we do **not** recommend users install 3.0.0-beta on critical production systems, but we do want everyone to test it out and find bugs.
 
-<<<<<<< HEAD
-<<<<<<< HEAD
-=======
->>>>>>> b10c3696c (Revert "updated changelog")
 * [CHANGE] UI: The old web UI has been replaced by a completely new one that is less cluttered and adds a few new features (PromLens-style tree view, better metrics explorer, "Explain" tab). However, it is still missing some features of the old UI (notably, exemplar display and heatmaps). To switch back to the old UI, you can use the feature flag `--enable-feature=old-ui` for the time being. #14872
 * [CHANGE] PromQL: Range selectors and the lookback delta are now left-open, i.e. a sample coinciding with the lower time limit is excluded rather than included. #13904
 * [CHANGE] Kubernetes SD: Remove support for `discovery.k8s.io/v1beta1` API version of EndpointSlice. This version is no longer served as of Kubernetes v1.25. #14365
@@ -57,12 +59,12 @@ As is traditional with a beta release, we do **not** recommend users install 3.0
 * [CHANGE] Remove deprecated `remote-write-receiver`,`promql-at-modifier`, and `promql-negative-offset` feature flags. #13456, #14526
 * [CHANGE] Remove deprecated `storage.tsdb.allow-overlapping-blocks`, `alertmanager.timeout`, and `storage.tsdb.retention` flags. #14640, #14643
 * [ENHANCEMENT] Move AM discovery page from "Monitoring status" to "Server status". #14875
-<<<<<<< HEAD
 * [FEATURE] Support config reload automatically - feature flag `auto-reload-config`. #14769
 * [BUGFIX] Scrape: Do not override target parameter labels with config params. #11029
 
-## 2.55.0-rc.0 / 2024-09-20
+## 2.55.0 / 2024-10-22
 
+* [FEATURE] PromQL: Add experimental `info` function. #14495
 * [FEATURE] Support UTF-8 characters in label names - feature flag `utf8-names`. #14482, #14880, #14736, #14727
 * [FEATURE] Scraping: Add the ability to set custom `http_headers` in config. #14817
 * [FEATURE] Scraping: Support feature flag `created-timestamp-zero-ingestion` in OpenMetrics. #14356, #14815
@@ -71,6 +73,7 @@ As is traditional with a beta release, we do **not** recommend users install 3.0
 * [FEATURE] Remote-Write: Support Google Cloud Monitoring authorization. #14346
 * [FEATURE] Promtool: `tsdb create-blocks` new option to add labels. #14403
 * [FEATURE] Promtool: `promtool test` adds `--junit` flag to format results. #14506
+* [FEATURE] TSDB: Add `delayed-compaction` feature flag, for people running many Prometheus to randomize timing. #12532
 * [ENHANCEMENT] OTLP receiver: Warn on exponential histograms with zero count and non-zero sum. #14706
 * [ENHANCEMENT] OTLP receiver: Interrupt translation on context cancellation/timeout. #14612
 * [ENHANCEMENT] Remote Read client: Enable streaming remote read if the server supports it. #11379
@@ -82,96 +85,16 @@ As is traditional with a beta release, we do **not** recommend users install 3.0
 * [ENHANCEMENT] Tracing: Improve PromQL tracing, including showing the operation performed for aggregates, operators, and calls. #14816
 * [ENHANCEMENT] API: Support multiple listening addresses. #14665
 * [ENHANCEMENT] TSDB: Backward compatibility with upcoming index v3. #14934
-* [PERF] TSDB: Query in-order and out-of-order series together. #14354, #14693, #14714, #14831, #14874, #14948
+* [PERF] TSDB: Query in-order and out-of-order series together. #14354, #14693, #14714, #14831, #14874, #14948, #15120
 * [PERF] TSDB: Streamline reading of overlapping out-of-order head chunks. #14729
+* [BUGFIX] PromQL: make sort_by_label stable. #14985
 * [BUGFIX] SD: Fix dropping targets (with feature flag `new-service-discovery-manager`). #13147
 * [BUGFIX] SD: Stop storing stale targets (with feature flag `new-service-discovery-manager`). #13622
 * [BUGFIX] Scraping: exemplars could be dropped in protobuf scraping. #14810
 * [BUGFIX] Remote-Write: fix metadata sending for experimental Remote-Write V2. #14766
 * [BUGFIX] Remote-Write: Return 4xx not 5xx when timeseries has duplicate label. #14716
 * [BUGFIX] Experimental Native Histograms: many fixes for incorrect results, panics, warnings. #14513, #14575, #14598, #14609, #14611, #14771, #14821
 * [BUGFIX] TSDB: Only count unknown record types in `record_decode_failures_total` metric. #14042
-=======
-- [CHANGE] UI: The old web UI has been replaced by a completely new one that is less cluttered and adds a few new features (PromLens-style tree view, better metrics explorer, "Explain" tab). However, it is still missing some features of the old UI (notably, exemplar display and heatmaps). To switch back to the old UI, you can use the feature flag `--enable-feature=old-ui` for the time being. #14872
-- [CHANGE] PromQL: Range selectors and the lookback delta are now left-open, i.e. a sample coinciding with the lower time limit is excluded rather than included. #13904
-- [CHANGE] Kubernetes SD: Remove support for `discovery.k8s.io/v1beta1` API version of EndpointSlice. This version is no longer served as of Kubernetes v1.25. #14365
-- [CHANGE] Kubernetes SD: Remove support for `networking.k8s.io/v1beta1` API version of Ingress. This version is no longer served as of Kubernetes v1.22. #14365
-- [CHANGE] UTF-8: Enable UTF-8 support by default. Prometheus now allows all UTF-8 characters in metric and label names. The corresponding `utf8-name` feature flag has been removed. #14705
-- [CHANGE] Console: Remove example files for the console feature. Users can continue using the console feature by supplying their own JavaScript and templates. #14807
-- [CHANGE] SD: Enable the new service discovery manager by default. This SD manager does not restart unchanged discoveries upon reloading. This makes reloads faster and reduces pressure on service discoveries' sources. The corresponding `new-service-discovery-manager` feature flag has been removed. #14770
-- [CHANGE] Agent mode has been promoted to stable. The feature flag `agent` has been removed. To run Prometheus in Agent mode, use the new `--agent` cmdline arg instead. #14747
-- [CHANGE] Remove deprecated `remote-write-receiver`,`promql-at-modifier`, and `promql-negative-offset` feature flags. #13456, #14526
-- [CHANGE] Remove deprecated `storage.tsdb.allow-overlapping-blocks`, `alertmanager.timeout`, and `storage.tsdb.retention` flags. #14640, #14643
-- [ENHANCEMENT] Move AM discovery page from "Monitoring status" to "Server status". #14875
-- [BUGFIX] Scrape: Do not override target parameter labels with config params. #11029
-
-## 2.55.0-rc.0 / 2024-09-20
-
-- [FEATURE] Support UTF-8 characters in label names - feature flag `utf8-names`. #14482, #14880, #14736, #14727
-- [FEATURE] Support config reload automatically - feature flag `auto-reload-config`. #14769
-- [FEATURE] Scraping: Add the ability to set custom `http_headers` in config. #14817
-- [FEATURE] Scraping: Support feature flag `created-timestamp-zero-ingestion` in OpenMetrics. #14356, #14815
-- [FEATURE] Scraping: `scrape_failure_log_file` option to log failures to a file. #14734
-- [FEATURE] OTLP receiver: Optional promotion of resource attributes to series labels. #14200
-- [FEATURE] Remote-Write: Support Google Cloud Monitoring authorization. #14346
-- [FEATURE] Promtool: `tsdb create-blocks` new option to add labels. #14403
-- [FEATURE] Promtool: `promtool test` adds `--junit` flag to format results. #14506
-- [ENHANCEMENT] OTLP receiver: Warn on exponential histograms with zero count and non-zero sum. #14706
-- [ENHANCEMENT] OTLP receiver: Interrupt translation on context cancellation/timeout. #14612
-- [ENHANCEMENT] Remote Read client: Enable streaming remote read if the server supports it. #11379
-- [ENHANCEMENT] Remote-Write: Don't reshard if we haven't successfully sent a sample since last update. #14450
-- [ENHANCEMENT] PromQL: Delay deletion of `__name__` label to the end of the query evaluation. This is **experimental** and enabled under the feature-flag `promql-delayed-name-removal`. #14477
-- [ENHANCEMENT] PromQL: Experimental `sort_by_label` and `sort_by_label_desc` sort by all labels when label is equal. #14655
-- [ENHANCEMENT] PromQL: Clarify error message logged when Go runtime panic occurs during query evaluation. #14621
-- [ENHANCEMENT] PromQL: Use Kahan summation for better accuracy in `avg` and `avg_over_time`. #14413
-- [ENHANCEMENT] Tracing: Improve PromQL tracing, including showing the operation performed for aggregates, operators, and calls. #14816
-- [ENHANCEMENT] API: Support multiple listening addresses. #14665
-- [ENHANCEMENT] TSDB: Backward compatibility with upcoming index v3. #14934
-- [PERF] TSDB: Query in-order and out-of-order series together. #14354, #14693, #14714, #14831, #14874, #14948
-- [PERF] TSDB: Streamline reading of overlapping out-of-order head chunks. #14729
-- [BUGFIX] SD: Fix dropping targets (with feature flag `new-service-discovery-manager`). #13147
-- [BUGFIX] SD: Stop storing stale targets (with feature flag `new-service-discovery-manager`). #13622
-- [BUGFIX] Scraping: exemplars could be dropped in protobuf scraping. #14810
-- [BUGFIX] Remote-Write: fix metadata sending for experimental Remote-Write V2. #14766
-- [BUGFIX] Remote-Write: Return 4xx not 5xx when timeseries has duplicate label. #14716
-- [BUGFIX] Experimental Native Histograms: many fixes for incorrect results, panics, warnings. #14513, #14575, #14598, #14609, #14611, #14771, #14821
-- [BUGFIX] TSDB: Only count unknown record types in `record_decode_failures_total` metric. #14042
->>>>>>> 58173ab1e (updated changelog)
-=======
-* [BUGFIX] Scrape: Do not override target parameter labels with config params. #11029
-
-## 2.55.0-rc.0 / 2024-09-20
-
-* [FEATURE] Support UTF-8 characters in label names - feature flag `utf8-names`. #14482, #14880, #14736, #14727
-* [FEATURE] Support config reload automatically - feature flag `auto-reload-config`. #14769
-* [FEATURE] Scraping: Add the ability to set custom `http_headers` in config. #14817
-* [FEATURE] Scraping: Support feature flag `created-timestamp-zero-ingestion` in OpenMetrics. #14356, #14815
-* [FEATURE] Scraping: `scrape_failure_log_file` option to log failures to a file. #14734
-* [FEATURE] OTLP receiver: Optional promotion of resource attributes to series labels. #14200
-* [FEATURE] Remote-Write: Support Google Cloud Monitoring authorization. #14346
-* [FEATURE] Promtool: `tsdb create-blocks` new option to add labels. #14403
-* [FEATURE] Promtool: `promtool test` adds `--junit` flag to format results. #14506
-* [ENHANCEMENT] OTLP receiver: Warn on exponential histograms with zero count and non-zero sum. #14706
-* [ENHANCEMENT] OTLP receiver: Interrupt translation on context cancellation/timeout. #14612
-* [ENHANCEMENT] Remote Read client: Enable streaming remote read if the server supports it. #11379
-* [ENHANCEMENT] Remote-Write: Don't reshard if we haven't successfully sent a sample since last update. #14450
-* [ENHANCEMENT] PromQL: Delay deletion of `__name__` label to the end of the query evaluation. This is **experimental** and enabled under the feature-flag `promql-delayed-name-removal`. #14477
-* [ENHANCEMENT] PromQL: Experimental `sort_by_label` and `sort_by_label_desc` sort by all labels when label is equal. #14655
-* [ENHANCEMENT] PromQL: Clarify error message logged when Go runtime panic occurs during query evaluation. #14621
-* [ENHANCEMENT] PromQL: Use Kahan summation for better accuracy in `avg` and `avg_over_time`. #14413
-* [ENHANCEMENT] Tracing: Improve PromQL tracing, including showing the operation performed for aggregates, operators, and calls. #14816
-* [ENHANCEMENT] API: Support multiple listening addresses. #14665
-* [ENHANCEMENT] TSDB: Backward compatibility with upcoming index v3. #14934
-* [PERF] TSDB: Query in-order and out-of-order series together. #14354, #14693, #14714, #14831, #14874, #14948
-* [PERF] TSDB: Streamline reading of overlapping out-of-order head chunks. #14729
-* [BUGFIX] SD: Fix dropping targets (with feature flag `new-service-discovery-manager`). #13147
-* [BUGFIX] SD: Stop storing stale targets (with feature flag `new-service-discovery-manager`). #13622
-* [BUGFIX] Scraping: exemplars could be dropped in protobuf scraping. #14810
-* [BUGFIX] Remote-Write: fix metadata sending for experimental Remote-Write V2. #14766
-* [BUGFIX] Remote-Write: Return 4xx not 5xx when timeseries has duplicate label. #14716
-* [BUGFIX] Experimental Native Histograms: many fixes for incorrect results, panics, warnings. #14513, #14575, #14598, #14609, #14611, #14771, #14821
-* [BUGFIX] TSDB: Only count unknown record types in `record_decode_failures_total` metric. #14042
->>>>>>> b10c3696c (Revert "updated changelog")
 
 ## 2.54.1 / 2024-08-27
 

docs/querying/functions.md

@@ -464,6 +464,97 @@ by the number of seconds under the specified time range window, and should be
 used primarily for human readability.  Use `rate` in recording rules so that
 increases are tracked consistently on a per-second basis.
 
+## `info()` (experimental)
+
+_The `info` function is an experiment to improve UX
+around including labels from [info metrics](https://grafana.com/blog/2021/08/04/how-to-use-promql-joins-for-more-effective-queries-of-prometheus-metrics-at-scale/#info-metrics).
+The behavior of this function may change in future versions of Prometheus,
+including its removal from PromQL. `info` has to be enabled via the 
+[feature flag](../feature_flags.md#experimental-promql-functions) `--enable-feature=promql-experimental-functions`._
+
+`info(v instant-vector, [data-label-selector instant-vector])` finds, for each time 
+series in `v`, all info series with matching _identifying_ labels (more on
+this later), and adds the union of their _data_ (i.e., non-identifying) labels
+to the time series. The second argument `data-label-selector` is optional.
+It is not a real instant vector, but uses a subset of its syntax.
+It must start and end with curly braces (`{ ... }`) and may only contain label matchers.
+The label matchers are used to constrain which info series to consider
+and which data labels to add to `v`.
+
+Identifying labels of an info series are the subset of labels that uniquely
+identify the info series. The remaining labels are considered
+_data labels_ (also called non-identifying). (Note that Prometheus's concept
+of time series identity always includes _all_ the labels. For the sake of the `info`
+function, we “logically” define info series identity in a different way than
+in the conventional Prometheus view.) The identifying labels of an info series
+are used to join it to regular (non-info) series, i.e. those series that have
+the same labels as the identifying labels of the info series. The data labels, which are
+the ones added to the regular series by the `info` function, effectively encode 
+metadata key value pairs. (This implies that a change in the data labels 
+in the conventional Prometheus view constitutes the end of one info series and
+the beginning of a new info series, while the “logical” view of the `info` function is
+that the same info series continues to exist, just with different “data”.)
+
+The conventional approach of adding data labels is sometimes called a “join query”,
+as illustrated by the following example:
+
+```
+  rate(http_server_request_duration_seconds_count[2m])
+* on (job, instance) group_left (k8s_cluster_name)
+  target_info
+```
+
+The core of the query is the expression `rate(http_server_request_duration_seconds_count[2m])`.
+But to add data labels from an info metric, the user has to use elaborate
+(and not very obvious) syntax to specify which info metric to use (`target_info`), what the
+identifying labels are (`on (job, instance)`), and which data labels to add
+(`group_left (k8s_cluster_name)`).
+
+This query is not only verbose and hard to write, it might also run into an “identity crisis”:
+If any of the data labels of `target_info` changes, Prometheus sees that as a change of series
+(as alluded to above, Prometheus just has no native concept of non-identifying labels).
+If the old `target_info` series is not properly marked as stale (which can happen with certain ingestion paths),
+the query above will fail for up to 5m (the lookback delta) because it will find a conflicting 
+match with both the old and the new version of `target_info`.
+
+The `info` function not only resolves this conflict in favor of the newer series, it also simplifies the syntax
+because it knows about the available info series and what their identifying labels are. The example query
+looks like this with the `info` function:
+
+```
+info(
+  rate(http_server_request_duration_seconds_count[2m]),
+  {k8s_cluster_name=~".+"}
+)
+```
+
+The common case of adding _all_ data labels can be achieved by
+omitting the 2nd argument of the `info` function entirely, simplifying
+the example even more:
+
+```
+info(rate(http_server_request_duration_seconds_count[2m]))
+```
+
+While `info` normally automatically finds all matching info series, it's possible to
+restrict them by providing a `__name__` label matcher, e.g.
+`{__name__="target_info"}`.
+
+### Limitations
+
+In its current iteration, `info` defaults to considering only info series with
+the name `target_info`. It also assumes that the identifying info series labels are
+`instance` and `job`. `info` does support other info series names however, through
+`__name__` label matchers. E.g., one can explicitly say to consider both 
+`target_info` and `build_info` as follows:
+`{__name__=~"(target|build)_info"}`. However, the identifying labels always
+have to be `instance` and `job`.
+
+These limitations are partially defeating the purpose of the `info` function.
+At the current stage, this is an experiment to find out how useful the approach
+turns out to be in practice. A final version of the `info` function will indeed
+consider all matching info series and with their appropriate identifying labels.
+
 ## `irate()`
 
 `irate(v range-vector)` calculates the per-second instant rate of increase of