Skip to content

Commit

Permalink
Increase default timeout for index-queries cache requests (#6786)
Browse files Browse the repository at this point in the history
As use of the index-queries cache has increased in the store-gateway, we
see more and more timeouts making requests to it. Timeouts making a request
mean the connection is closed and reopened since it assumed to be "bad". This
can cause a huge number of connections on nodes to end up in `TIME_WAIT`.

Increase the timeout used by the index-queries cache to 450ms from the default
of 200ms in jsonnet and helm.

Signed-off-by: Nick Pillitteri <[email protected]>
  • Loading branch information
56quarters authored Nov 30, 2023
1 parent 7f92003 commit 7d7920f
Show file tree
Hide file tree
Showing 55 changed files with 73 additions and 0 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,7 @@
* [CHANGE] Changed default `_config.cluster_domain` from `cluster.local` to `cluster.local.` to reduce the number of DNS lookups made by Mimir. #6389
* [CHANGE] Query-frontend: changed default `_config.autoscaling_query_frontend_cpu_target_utilization` from `1` to `0.75`. #6395
* [CHANGE] Distributor: Increase HPA scale down period such that distributors are slower to scale down after autoscaling up. #6589
* [CHANGE] Store-gateway: Change the default timeout used for index-queries caches from `200ms` to `450ms`. #6786
* [FEATURE] Store-gateway: Allow automated zone-by-zone downscaling, that can be enabled via the `store_gateway_automated_downscale_enabled` flag. It is disabled by default. #6149
* [FEATURE] Ingester: Allow to configure TSDB Head early compaction using the following `_config` parameters: #6181
* `ingester_tsdb_head_early_compaction_enabled` (disabled by default)
Expand Down
1 change: 1 addition & 0 deletions operations/helm/charts/mimir-distributed/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ Entries should include a reference to the Pull Request that introduced the chang
* [CHANGE] Remove deprecated configuration parameter `blocks_storage.bucket_store.max_chunk_pool_bytes`. #6673
* [CHANGE] Reduce `-server.grpc-max-concurrent-streams` from 1000 to 500 for ingester and to 100 for all components. #5666
* [CHANGE] Changed default `clusterDomain` from `cluster.local` to `cluster.local.` to reduce the number of DNS lookups made by Mimir. #6389
* [CHANGE] Change the default timeout used for index-queries caches from `200ms` to `450ms`. #6786
* [ENHANCEMENT] Update the `rollout-operator` subchart to `0.10.0`. #6022 #6110 #6558 #6681
* [ENHANCEMENT] Add support for not setting replicas for distributor, querier, and query-frontend. #6373
* [BUGFIX] Let the unified gatway/nginx config listen on IPv6 as well. Followup to #5948. #6204
Expand Down
1 change: 1 addition & 0 deletions operations/helm/charts/mimir-distributed/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -192,6 +192,7 @@ mimir:
memcached:
addresses: {{ include "mimir.indexCacheAddress" . }}
max_item_size: {{ mul (index .Values "index-cache").maxItemMemory 1024 1024 }}
timeout: 450ms
max_idle_connections: 150
{{- end }}
{{- if index .Values "metadata-cache" "enabled" }}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ data:
addresses: dns+enterprise-https-values-mimir-index-cache.citestns.svc:11211
max_idle_connections: 150
max_item_size: 5242880
timeout: 450ms
metadata_cache:
backend: memcached
memcached:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ data:
addresses: dns+large-values-mimir-index-cache.citestns.svc:11211
max_idle_connections: 150
max_item_size: 5242880
timeout: 450ms
metadata_cache:
backend: memcached
memcached:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ data:
addresses: dns+small-values-mimir-index-cache.citestns.svc:11211
max_idle_connections: 150
max_item_size: 5242880
timeout: 450ms
metadata_cache:
backend: memcached
memcached:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ data:
addresses: dns+test-oss-k8s-1.25-values-mimir-index-cache.citestns.svc:11211
max_idle_connections: 150
max_item_size: 5242880
timeout: 450ms
metadata_cache:
backend: memcached
memcached:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ data:
addresses: dns+test-oss-values-mimir-index-cache.citestns.svc:11211
max_idle_connections: 150
max_item_size: 5242880
timeout: 450ms
metadata_cache:
backend: memcached
memcached:
Expand Down
1 change: 1 addition & 0 deletions operations/mimir-tests/test-all-components-generated.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -1443,6 +1443,7 @@ spec:
- -blocks-storage.bucket-store.index-cache.memcached.max-get-multi-concurrency=100
- -blocks-storage.bucket-store.index-cache.memcached.max-idle-connections=150
- -blocks-storage.bucket-store.index-cache.memcached.max-item-size=5242880
- -blocks-storage.bucket-store.index-cache.memcached.timeout=450ms
- -blocks-storage.bucket-store.index-header.lazy-loading-enabled=true
- -blocks-storage.bucket-store.index-header.lazy-loading-idle-timeout=60m
- -blocks-storage.bucket-store.metadata-cache.backend=memcached
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1445,6 +1445,7 @@ spec:
- -blocks-storage.bucket-store.index-cache.memcached.max-get-multi-concurrency=100
- -blocks-storage.bucket-store.index-cache.memcached.max-idle-connections=150
- -blocks-storage.bucket-store.index-cache.memcached.max-item-size=5242880
- -blocks-storage.bucket-store.index-cache.memcached.timeout=450ms
- -blocks-storage.bucket-store.index-header.lazy-loading-enabled=true
- -blocks-storage.bucket-store.index-header.lazy-loading-idle-timeout=60m
- -blocks-storage.bucket-store.metadata-cache.backend=memcached
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1444,6 +1444,7 @@ spec:
- -blocks-storage.bucket-store.index-cache.memcached.max-get-multi-concurrency=100
- -blocks-storage.bucket-store.index-cache.memcached.max-idle-connections=150
- -blocks-storage.bucket-store.index-cache.memcached.max-item-size=5242880
- -blocks-storage.bucket-store.index-cache.memcached.timeout=450ms
- -blocks-storage.bucket-store.index-header.lazy-loading-enabled=true
- -blocks-storage.bucket-store.index-header.lazy-loading-idle-timeout=60m
- -blocks-storage.bucket-store.metadata-cache.backend=memcached
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1789,6 +1789,7 @@ spec:
- -blocks-storage.bucket-store.index-cache.memcached.max-get-multi-concurrency=100
- -blocks-storage.bucket-store.index-cache.memcached.max-idle-connections=150
- -blocks-storage.bucket-store.index-cache.memcached.max-item-size=5242880
- -blocks-storage.bucket-store.index-cache.memcached.timeout=450ms
- -blocks-storage.bucket-store.index-header.lazy-loading-enabled=true
- -blocks-storage.bucket-store.index-header.lazy-loading-idle-timeout=60m
- -blocks-storage.bucket-store.metadata-cache.backend=memcached
Expand Down
1 change: 1 addition & 0 deletions operations/mimir-tests/test-autoscaling-generated.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -1789,6 +1789,7 @@ spec:
- -blocks-storage.bucket-store.index-cache.memcached.max-get-multi-concurrency=100
- -blocks-storage.bucket-store.index-cache.memcached.max-idle-connections=150
- -blocks-storage.bucket-store.index-cache.memcached.max-item-size=5242880
- -blocks-storage.bucket-store.index-cache.memcached.timeout=450ms
- -blocks-storage.bucket-store.index-header.lazy-loading-enabled=true
- -blocks-storage.bucket-store.index-header.lazy-loading-idle-timeout=60m
- -blocks-storage.bucket-store.metadata-cache.backend=memcached
Expand Down
1 change: 1 addition & 0 deletions operations/mimir-tests/test-consul-generated.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -1813,6 +1813,7 @@ spec:
- -blocks-storage.bucket-store.index-cache.memcached.max-get-multi-concurrency=100
- -blocks-storage.bucket-store.index-cache.memcached.max-idle-connections=150
- -blocks-storage.bucket-store.index-cache.memcached.max-item-size=5242880
- -blocks-storage.bucket-store.index-cache.memcached.timeout=450ms
- -blocks-storage.bucket-store.index-header.lazy-loading-enabled=true
- -blocks-storage.bucket-store.index-header.lazy-loading-idle-timeout=60m
- -blocks-storage.bucket-store.metadata-cache.backend=memcached
Expand Down
3 changes: 3 additions & 0 deletions operations/mimir-tests/test-consul-multi-zone-generated.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2279,6 +2279,7 @@ spec:
- -blocks-storage.bucket-store.index-cache.memcached.max-get-multi-concurrency=100
- -blocks-storage.bucket-store.index-cache.memcached.max-idle-connections=150
- -blocks-storage.bucket-store.index-cache.memcached.max-item-size=5242880
- -blocks-storage.bucket-store.index-cache.memcached.timeout=450ms
- -blocks-storage.bucket-store.index-header.lazy-loading-enabled=true
- -blocks-storage.bucket-store.index-header.lazy-loading-idle-timeout=60m
- -blocks-storage.bucket-store.metadata-cache.backend=memcached
Expand Down Expand Up @@ -2410,6 +2411,7 @@ spec:
- -blocks-storage.bucket-store.index-cache.memcached.max-get-multi-concurrency=100
- -blocks-storage.bucket-store.index-cache.memcached.max-idle-connections=150
- -blocks-storage.bucket-store.index-cache.memcached.max-item-size=5242880
- -blocks-storage.bucket-store.index-cache.memcached.timeout=450ms
- -blocks-storage.bucket-store.index-header.lazy-loading-enabled=true
- -blocks-storage.bucket-store.index-header.lazy-loading-idle-timeout=60m
- -blocks-storage.bucket-store.metadata-cache.backend=memcached
Expand Down Expand Up @@ -2541,6 +2543,7 @@ spec:
- -blocks-storage.bucket-store.index-cache.memcached.max-get-multi-concurrency=100
- -blocks-storage.bucket-store.index-cache.memcached.max-idle-connections=150
- -blocks-storage.bucket-store.index-cache.memcached.max-item-size=5242880
- -blocks-storage.bucket-store.index-cache.memcached.timeout=450ms
- -blocks-storage.bucket-store.index-header.lazy-loading-enabled=true
- -blocks-storage.bucket-store.index-header.lazy-loading-idle-timeout=60m
- -blocks-storage.bucket-store.metadata-cache.backend=memcached
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1686,6 +1686,7 @@ spec:
- -blocks-storage.bucket-store.index-cache.memcached.max-get-multi-concurrency=100
- -blocks-storage.bucket-store.index-cache.memcached.max-idle-connections=150
- -blocks-storage.bucket-store.index-cache.memcached.max-item-size=5242880
- -blocks-storage.bucket-store.index-cache.memcached.timeout=450ms
- -blocks-storage.bucket-store.index-header.lazy-loading-enabled=true
- -blocks-storage.bucket-store.index-header.lazy-loading-idle-timeout=60m
- -blocks-storage.bucket-store.metadata-cache.backend=memcached
Expand Down
1 change: 1 addition & 0 deletions operations/mimir-tests/test-continuous-test-generated.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -1247,6 +1247,7 @@ spec:
- -blocks-storage.bucket-store.index-cache.memcached.max-get-multi-concurrency=100
- -blocks-storage.bucket-store.index-cache.memcached.max-idle-connections=150
- -blocks-storage.bucket-store.index-cache.memcached.max-item-size=5242880
- -blocks-storage.bucket-store.index-cache.memcached.timeout=450ms
- -blocks-storage.bucket-store.index-header.lazy-loading-enabled=true
- -blocks-storage.bucket-store.index-header.lazy-loading-idle-timeout=60m
- -blocks-storage.bucket-store.metadata-cache.backend=memcached
Expand Down
1 change: 1 addition & 0 deletions operations/mimir-tests/test-defaults-generated.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -1193,6 +1193,7 @@ spec:
- -blocks-storage.bucket-store.index-cache.memcached.max-get-multi-concurrency=100
- -blocks-storage.bucket-store.index-cache.memcached.max-idle-connections=150
- -blocks-storage.bucket-store.index-cache.memcached.max-item-size=5242880
- -blocks-storage.bucket-store.index-cache.memcached.timeout=450ms
- -blocks-storage.bucket-store.index-header.lazy-loading-enabled=true
- -blocks-storage.bucket-store.index-header.lazy-loading-idle-timeout=60m
- -blocks-storage.bucket-store.metadata-cache.backend=memcached
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2332,6 +2332,7 @@ spec:
- -blocks-storage.bucket-store.index-cache.memcached.max-get-multi-concurrency=100
- -blocks-storage.bucket-store.index-cache.memcached.max-idle-connections=150
- -blocks-storage.bucket-store.index-cache.memcached.max-item-size=5242880
- -blocks-storage.bucket-store.index-cache.memcached.timeout=450ms
- -blocks-storage.bucket-store.index-header.lazy-loading-enabled=true
- -blocks-storage.bucket-store.index-header.lazy-loading-idle-timeout=60m
- -blocks-storage.bucket-store.metadata-cache.backend=memcached
Expand Down Expand Up @@ -2516,6 +2517,7 @@ spec:
- -blocks-storage.bucket-store.index-cache.memcached.max-get-multi-concurrency=100
- -blocks-storage.bucket-store.index-cache.memcached.max-idle-connections=150
- -blocks-storage.bucket-store.index-cache.memcached.max-item-size=5242880
- -blocks-storage.bucket-store.index-cache.memcached.timeout=450ms
- -blocks-storage.bucket-store.index-header.lazy-loading-enabled=true
- -blocks-storage.bucket-store.index-header.lazy-loading-idle-timeout=60m
- -blocks-storage.bucket-store.metadata-cache.backend=memcached
Expand Down Expand Up @@ -2700,6 +2702,7 @@ spec:
- -blocks-storage.bucket-store.index-cache.memcached.max-get-multi-concurrency=100
- -blocks-storage.bucket-store.index-cache.memcached.max-idle-connections=150
- -blocks-storage.bucket-store.index-cache.memcached.max-item-size=5242880
- -blocks-storage.bucket-store.index-cache.memcached.timeout=450ms
- -blocks-storage.bucket-store.index-header.lazy-loading-enabled=true
- -blocks-storage.bucket-store.index-header.lazy-loading-idle-timeout=60m
- -blocks-storage.bucket-store.metadata-cache.backend=memcached
Expand Down Expand Up @@ -3269,6 +3272,7 @@ spec:
- -blocks-storage.bucket-store.index-cache.memcached.max-get-multi-concurrency=100
- -blocks-storage.bucket-store.index-cache.memcached.max-idle-connections=150
- -blocks-storage.bucket-store.index-cache.memcached.max-item-size=5242880
- -blocks-storage.bucket-store.index-cache.memcached.timeout=450ms
- -blocks-storage.bucket-store.index-header.lazy-loading-enabled=true
- -blocks-storage.bucket-store.index-header.lazy-loading-idle-timeout=60m
- -blocks-storage.bucket-store.metadata-cache.backend=memcached
Expand Down Expand Up @@ -3404,6 +3408,7 @@ spec:
- -blocks-storage.bucket-store.index-cache.memcached.max-get-multi-concurrency=100
- -blocks-storage.bucket-store.index-cache.memcached.max-idle-connections=150
- -blocks-storage.bucket-store.index-cache.memcached.max-item-size=5242880
- -blocks-storage.bucket-store.index-cache.memcached.timeout=450ms
- -blocks-storage.bucket-store.index-header.lazy-loading-enabled=true
- -blocks-storage.bucket-store.index-header.lazy-loading-idle-timeout=60m
- -blocks-storage.bucket-store.metadata-cache.backend=memcached
Expand Down Expand Up @@ -3539,6 +3544,7 @@ spec:
- -blocks-storage.bucket-store.index-cache.memcached.max-get-multi-concurrency=100
- -blocks-storage.bucket-store.index-cache.memcached.max-idle-connections=150
- -blocks-storage.bucket-store.index-cache.memcached.max-item-size=5242880
- -blocks-storage.bucket-store.index-cache.memcached.timeout=450ms
- -blocks-storage.bucket-store.index-header.lazy-loading-enabled=true
- -blocks-storage.bucket-store.index-header.lazy-loading-idle-timeout=60m
- -blocks-storage.bucket-store.metadata-cache.backend=memcached
Expand Down
1 change: 1 addition & 0 deletions operations/mimir-tests/test-env-vars-generated.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -1448,6 +1448,7 @@ spec:
- -blocks-storage.bucket-store.index-cache.memcached.max-get-multi-concurrency=100
- -blocks-storage.bucket-store.index-cache.memcached.max-idle-connections=150
- -blocks-storage.bucket-store.index-cache.memcached.max-item-size=5242880
- -blocks-storage.bucket-store.index-cache.memcached.timeout=450ms
- -blocks-storage.bucket-store.index-header.lazy-loading-enabled=true
- -blocks-storage.bucket-store.index-header.lazy-loading-idle-timeout=60m
- -blocks-storage.bucket-store.metadata-cache.backend=memcached
Expand Down
1 change: 1 addition & 0 deletions operations/mimir-tests/test-etcd-replicas-generated.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -1193,6 +1193,7 @@ spec:
- -blocks-storage.bucket-store.index-cache.memcached.max-get-multi-concurrency=100
- -blocks-storage.bucket-store.index-cache.memcached.max-idle-connections=150
- -blocks-storage.bucket-store.index-cache.memcached.max-item-size=5242880
- -blocks-storage.bucket-store.index-cache.memcached.timeout=450ms
- -blocks-storage.bucket-store.index-header.lazy-loading-enabled=true
- -blocks-storage.bucket-store.index-header.lazy-loading-idle-timeout=60m
- -blocks-storage.bucket-store.metadata-cache.backend=memcached
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1491,6 +1491,7 @@ spec:
- -blocks-storage.bucket-store.index-cache.memcached.max-get-multi-concurrency=100
- -blocks-storage.bucket-store.index-cache.memcached.max-idle-connections=150
- -blocks-storage.bucket-store.index-cache.memcached.max-item-size=5242880
- -blocks-storage.bucket-store.index-cache.memcached.timeout=450ms
- -blocks-storage.bucket-store.index-header.lazy-loading-enabled=true
- -blocks-storage.bucket-store.index-header.lazy-loading-idle-timeout=60m
- -blocks-storage.bucket-store.metadata-cache.backend=memcached
Expand Down
1 change: 1 addition & 0 deletions operations/mimir-tests/test-helm-parity-generated.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -1550,6 +1550,7 @@ spec:
- -blocks-storage.bucket-store.index-cache.memcached.max-get-multi-concurrency=100
- -blocks-storage.bucket-store.index-cache.memcached.max-idle-connections=150
- -blocks-storage.bucket-store.index-cache.memcached.max-item-size=5242880
- -blocks-storage.bucket-store.index-cache.memcached.timeout=450ms
- -blocks-storage.bucket-store.index-header.lazy-loading-enabled=true
- -blocks-storage.bucket-store.index-header.lazy-loading-idle-timeout=60m
- -blocks-storage.bucket-store.metadata-cache.backend=memcached
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1443,6 +1443,7 @@ spec:
- -blocks-storage.bucket-store.index-cache.memcached.max-get-multi-concurrency=100
- -blocks-storage.bucket-store.index-cache.memcached.max-idle-connections=150
- -blocks-storage.bucket-store.index-cache.memcached.max-item-size=5242880
- -blocks-storage.bucket-store.index-cache.memcached.timeout=450ms
- -blocks-storage.bucket-store.index-header.lazy-loading-enabled=true
- -blocks-storage.bucket-store.index-header.lazy-loading-idle-timeout=60m
- -blocks-storage.bucket-store.metadata-cache.backend=memcached
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1449,6 +1449,7 @@ spec:
- -blocks-storage.bucket-store.index-cache.memcached.max-get-multi-concurrency=100
- -blocks-storage.bucket-store.index-cache.memcached.max-idle-connections=150
- -blocks-storage.bucket-store.index-cache.memcached.max-item-size=5242880
- -blocks-storage.bucket-store.index-cache.memcached.timeout=450ms
- -blocks-storage.bucket-store.index-header.lazy-loading-enabled=true
- -blocks-storage.bucket-store.index-header.lazy-loading-idle-timeout=60m
- -blocks-storage.bucket-store.metadata-cache.backend=memcached
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1455,6 +1455,7 @@ spec:
- -blocks-storage.bucket-store.index-cache.memcached.max-get-multi-concurrency=100
- -blocks-storage.bucket-store.index-cache.memcached.max-idle-connections=150
- -blocks-storage.bucket-store.index-cache.memcached.max-item-size=5242880
- -blocks-storage.bucket-store.index-cache.memcached.timeout=450ms
- -blocks-storage.bucket-store.index-header.lazy-loading-enabled=true
- -blocks-storage.bucket-store.index-header.lazy-loading-idle-timeout=60m
- -blocks-storage.bucket-store.metadata-cache.backend=memcached
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1449,6 +1449,7 @@ spec:
- -blocks-storage.bucket-store.index-cache.memcached.max-get-multi-concurrency=100
- -blocks-storage.bucket-store.index-cache.memcached.max-idle-connections=150
- -blocks-storage.bucket-store.index-cache.memcached.max-item-size=5242880
- -blocks-storage.bucket-store.index-cache.memcached.timeout=450ms
- -blocks-storage.bucket-store.index-header.lazy-loading-enabled=true
- -blocks-storage.bucket-store.index-header.lazy-loading-idle-timeout=60m
- -blocks-storage.bucket-store.metadata-cache.backend=memcached
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1813,6 +1813,7 @@ spec:
- -blocks-storage.bucket-store.index-cache.memcached.max-get-multi-concurrency=100
- -blocks-storage.bucket-store.index-cache.memcached.max-idle-connections=150
- -blocks-storage.bucket-store.index-cache.memcached.max-item-size=5242880
- -blocks-storage.bucket-store.index-cache.memcached.timeout=450ms
- -blocks-storage.bucket-store.index-header.lazy-loading-enabled=true
- -blocks-storage.bucket-store.index-header.lazy-loading-idle-timeout=60m
- -blocks-storage.bucket-store.metadata-cache.backend=memcached
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1899,6 +1899,7 @@ spec:
- -blocks-storage.bucket-store.index-cache.memcached.max-get-multi-concurrency=100
- -blocks-storage.bucket-store.index-cache.memcached.max-idle-connections=150
- -blocks-storage.bucket-store.index-cache.memcached.max-item-size=5242880
- -blocks-storage.bucket-store.index-cache.memcached.timeout=450ms
- -blocks-storage.bucket-store.index-header.lazy-loading-enabled=true
- -blocks-storage.bucket-store.index-header.lazy-loading-idle-timeout=60m
- -blocks-storage.bucket-store.metadata-cache.backend=memcached
Expand Down
Loading

0 comments on commit 7d7920f

Please sign in to comment.