Skip to content

Commit

Permalink
Update based on changes from code review
Browse files Browse the repository at this point in the history
  • Loading branch information
zenador committed Jun 23, 2022
1 parent 1b164da commit db27a98
Show file tree
Hide file tree
Showing 7 changed files with 31 additions and 35 deletions.
2 changes: 1 addition & 1 deletion cmd/mimir/config-descriptor.json
Original file line number Diff line number Diff line change
Expand Up @@ -5032,7 +5032,7 @@
"kind": "field",
"name": "max_stale_period",
"required": false,
"desc": "The maximum allowed age of a bucket index (last updated) before queries start failing because the bucket index is too old. The bucket index is periodically updated by the compactor, while this check is enforced in the querier (at query time).",
"desc": "The maximum allowed age of a bucket index (last updated) before queries start failing because the bucket index is too old. The bucket index is periodically updated by the compactor, and this check is enforced in the querier (at query time).",
"fieldValue": null,
"fieldDefaultValue": 3600000000000,
"fieldFlag": "blocks-storage.bucket-store.bucket-index.max-stale-period",
Expand Down
2 changes: 1 addition & 1 deletion cmd/mimir/help-all.txt.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -269,7 +269,7 @@ Usage of ./cmd/mimir/mimir:
-blocks-storage.bucket-store.bucket-index.idle-timeout duration
How long a unused bucket index should be cached. Once this timeout expires, the unused bucket index is removed from the in-memory cache. This option is used only by querier. (default 1h0m0s)
-blocks-storage.bucket-store.bucket-index.max-stale-period duration
The maximum allowed age of a bucket index (last updated) before queries start failing because the bucket index is too old. The bucket index is periodically updated by the compactor, while this check is enforced in the querier (at query time). (default 1h0m0s)
The maximum allowed age of a bucket index (last updated) before queries start failing because the bucket index is too old. The bucket index is periodically updated by the compactor, and this check is enforced in the querier (at query time). (default 1h0m0s)
-blocks-storage.bucket-store.bucket-index.update-on-error-interval duration
How frequently a bucket index, which previously failed to load, should be tried to load again. This option is used only by querier. (default 1m0s)
-blocks-storage.bucket-store.chunk-pool-max-bucket-size-bytes int
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,14 +12,14 @@ weight: 130
You can configure Grafana Mimir by using a YAML file or via command-line flags
that represent configuration parameters.
To specify the YAML file, use the `-config.file` command-line option.
If you specify both the command-line flags and YAML configuration parameters,
If you specify both the command-line flags and YAML configuration parameters,
the command-line flags take precedence over values in a YAML file.

To see the current configuration of any component,
go to the `/config` HTTP API endpoint.
Passwords are filtered out of this endpoint.

Parameters are
Parameters are
written in [YAML format](https://en.wikipedia.org/wiki/YAML), and
brackets indicate that a parameter is optional.

Expand All @@ -34,35 +34,35 @@ brackets indicate that a parameter is optional.
- `<prefix>`: a CLI flag prefix based on the context (look at the parent configuration block to see which CLI flags prefix should be used)
- `<relabel_config>`: a [Prometheus relabeling configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config)
- `<time>`: a timestamp, with available formats:
- `2006-01-20` (midnight, local timezone)
- `2006-01-20T15:04` (local timezone)
- RFC 3339 formats: `2006-01-20T15:04:05Z` (UTC) or `2006-01-20T15:04:05+07:00` (explicit timezone)
- `2006-01-20` (midnight, local timezone)
- `2006-01-20T15:04` (local timezone)
- RFC 3339 formats: `2006-01-20T15:04:05Z` (UTC) or `2006-01-20T15:04:05+07:00` (explicit timezone)

## Parameter categories

In order to simplify Mimir configuration, we categorize parameters by
maturity level and intended use:

- **Basic** parameters are ones that we expect the majority of users to modify.
Examples of basic parameters are object store credentials and
other dependency connection information.
These parameters will generally remain stable for long periods of time,
and they focus on user goals.
In this reference, any parameters that are not marked as `(advanced)` or
`(experimental)` are in the basic category.
Examples of basic parameters are object store credentials and
other dependency connection information.
These parameters will generally remain stable for long periods of time,
and they focus on user goals.
In this reference, any parameters that are not marked as `(advanced)` or
`(experimental)` are in the basic category.

- **Advanced** parameters are ones that few users will change
from their default values.
There are strong use cases for altered values.
In this reference, advanced-category parameters include `(advanced)` at
the beginning of their description.
from their default values.
There are strong use cases for altered values.
In this reference, advanced-category parameters include `(advanced)` at
the beginning of their description.

- **Experimental** parameters are for new and experimental features.
These parameters permit
early adopters and Mimir developers to gain confidence with new
features.
In this reference, experimental-category parameters include `(experimental)` at
the beginning of their description.
These parameters permit
early adopters and Mimir developers to gain confidence with new
features.
In this reference, experimental-category parameters include `(experimental)` at
the beginning of their description.

### Parameter lifecycle

Expand Down Expand Up @@ -1581,7 +1581,7 @@ gcs:
# JSON either from a Google Developers Console client_credentials.json file,
# or a Google Developers service account key. Needs to be valid JSON, not a
# filesystem path. If empty, fallback to Google default logic:
# filesystem path. If empty, fallback to Google default logic:
# 1. A JSON file whose path is specified by the GOOGLE_APPLICATION_CREDENTIALS
# environment variable. For workload identity federation, refer to
# https://cloud.google.com/iam/docs/how-to#using-workload-identity-federation
Expand Down Expand Up @@ -2031,7 +2031,7 @@ gcs:
# JSON either from a Google Developers Console client_credentials.json file,
# or a Google Developers service account key. Needs to be valid JSON, not a
# filesystem path. If empty, fallback to Google default logic:
# filesystem path. If empty, fallback to Google default logic:
# 1. A JSON file whose path is specified by the GOOGLE_APPLICATION_CREDENTIALS
# environment variable. For workload identity federation, refer to
# https://cloud.google.com/iam/docs/how-to#using-workload-identity-federation
Expand Down Expand Up @@ -3039,7 +3039,7 @@ gcs:
# JSON either from a Google Developers Console client_credentials.json file,
# or a Google Developers service account key. Needs to be valid JSON, not a
# filesystem path. If empty, fallback to Google default logic:
# filesystem path. If empty, fallback to Google default logic:
# 1. A JSON file whose path is specified by the GOOGLE_APPLICATION_CREDENTIALS
# environment variable. For workload identity federation, refer to
# https://cloud.google.com/iam/docs/how-to#using-workload-identity-federation
Expand Down Expand Up @@ -3351,8 +3351,8 @@ bucket_store:
# (advanced) The maximum allowed age of a bucket index (last updated) before
# queries start failing because the bucket index is too old. The bucket
# index is periodically updated by the compactor, while this check is
# enforced in the querier (at query time).
# index is periodically updated by the compactor, and this check is enforced
# in the querier (at query time).
# CLI flag: -blocks-storage.bucket-store.bucket-index.max-stale-period
[max_stale_period: <duration> | default = 1h]
Expand Down
3 changes: 1 addition & 2 deletions pkg/querier/blocks_finder_bucket_index.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@ import (
"github.com/thanos-io/thanos/pkg/objstore"

"github.com/grafana/mimir/pkg/storage/bucket"
"github.com/grafana/mimir/pkg/storage/tsdb"
"github.com/grafana/mimir/pkg/storage/tsdb/bucketindex"
"github.com/grafana/mimir/pkg/util/globalerror"
)
Expand Down Expand Up @@ -116,5 +115,5 @@ func (f *BucketIndexBlocksFinder) GetBlocks(ctx context.Context, userID string,
}

func newBucketIndexTooOldError(updatedAt time.Time, maxStalePeriod time.Duration) error {
return errors.New(globalerror.BucketIndexTooOld.MessageWithLimitConfig(fmt.Sprintf("the bucket index is too old. It was last updated at %s, which exceeds the maximum allowed staleness period of %v", updatedAt.UTC().Format(time.RFC3339Nano), maxStalePeriod), tsdb.BucketIndexConfigPrefix+tsdb.MaxStalePeriodFlag))
return errors.New(globalerror.BucketIndexTooOld.Message(fmt.Sprintf("the bucket index is too old. It was last updated at %s, which exceeds the maximum allowed staleness period of %v", updatedAt.UTC().Format(time.RFC3339Nano), maxStalePeriod)))
}
2 changes: 1 addition & 1 deletion pkg/querier/blocks_finder_bucket_index_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -233,7 +233,7 @@ func TestBlocksFinderBucketIndexErrMsgs(t *testing.T) {
}{
"newBucketIndexTooOldError": {
err: newBucketIndexTooOldError(time.Unix(1000000000, 0), time.Hour),
msg: `the bucket index is too old. It was last updated at 2001-09-09T01:46:40Z, which exceeds the maximum allowed staleness period of 1h0m0s (err-mimir-bucket-index-too-old). You can adjust the related per-tenant limit by configuring -blocks-storage.bucket-store.bucket-index.max-stale-period, or by contacting your service administrator.`,
msg: `the bucket index is too old. It was last updated at 2001-09-09T01:46:40Z, which exceeds the maximum allowed staleness period of 1h0m0s (err-mimir-bucket-index-too-old)`,
},
}

Expand Down
2 changes: 1 addition & 1 deletion pkg/querier/blocks_store_queryable_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -2017,7 +2017,7 @@ func TestBlocksStoreQueryableErrMsgs(t *testing.T) {
}{
"newStoreConsistencyCheckFailedError": {
err: newStoreConsistencyCheckFailedError([]ulid.ULID{ulid.MustNew(1, nil)}),
msg: `the consistency check failed because some blocks were not queried (err-mimir-store-consistency-check-failed). The blocks are: 00000000010000000000000000`,
msg: `the consistency check failed because some blocks were not queried (err-mimir-store-consistency-check-failed). The non-queried blocks are: 00000000010000000000000000`,
},
}

Expand Down
5 changes: 1 addition & 4 deletions pkg/storage/tsdb/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -64,9 +64,6 @@ const (
// DefaultPartitionerMaxGapSize is the default max size - in bytes - of a gap for which the store-gateway
// partitioner aggregates together two bucket GET object requests.
DefaultPartitionerMaxGapSize = uint64(512 * 1024)

BucketIndexConfigPrefix = "blocks-storage.bucket-store.bucket-index."
MaxStalePeriodFlag = "max-stale-period"
)

// Validation errors
Expand Down Expand Up @@ -359,5 +356,5 @@ func (cfg *BucketIndexConfig) RegisterFlagsWithPrefix(f *flag.FlagSet, prefix st
f.BoolVar(&cfg.Enabled, prefix+"enabled", true, "If enabled, queriers and store-gateways discover blocks by reading a bucket index (created and updated by the compactor) instead of periodically scanning the bucket.")
f.DurationVar(&cfg.UpdateOnErrorInterval, prefix+"update-on-error-interval", time.Minute, "How frequently a bucket index, which previously failed to load, should be tried to load again. This option is used only by querier.")
f.DurationVar(&cfg.IdleTimeout, prefix+"idle-timeout", time.Hour, "How long a unused bucket index should be cached. Once this timeout expires, the unused bucket index is removed from the in-memory cache. This option is used only by querier.")
f.DurationVar(&cfg.MaxStalePeriod, prefix+MaxStalePeriodFlag, time.Hour, "The maximum allowed age of a bucket index (last updated) before queries start failing because the bucket index is too old. The bucket index is periodically updated by the compactor, and this check is enforced in the querier (at query time).")
f.DurationVar(&cfg.MaxStalePeriod, prefix+"max-stale-period", time.Hour, "The maximum allowed age of a bucket index (last updated) before queries start failing because the bucket index is too old. The bucket index is periodically updated by the compactor, and this check is enforced in the querier (at query time).")
}

0 comments on commit db27a98

Please sign in to comment.