GCS error handling, example, and documentation #397
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… gcs backend. Add gcs example using fake-gcs-server and add necessary settings. Add documentation page for GCS configuration. Update gcs client library reference for header parsing bug fix
mdisibio
requested review from
achatterjee-grafana,
annanay25 and
joe-elliott
as code owners
joe-elliott
requested changes
Dec 7, 2020
achatterjee-grafana
suggested changes
Dec 8, 2020
Comment on lines
10
to
20
| trace: | ||
| backend: gcs # store traces in gcs | ||
| s3: | ||
| bucket_name: tempo # store traces in this bucket | ||
| chunk_buffer_size: 10485760 # optional. buffer size for reads. default = 10MiB | ||
| endpoint: https://storage.googleapis.com/storage/v1/ # optional. api endpoint override | ||
| insecure: false # optional. Set to true to disable authentication | ||
| # and certificate checks. | ||
| project_id: project # optional if bucket already exists. Project ID | ||
| # to use when creating bucket on first run. | ||
| ``` |
There was a problem hiding this comment.
Suggested change
| trace: | |
| backend: gcs # store traces in gcs | |
| s3: | |
| bucket_name: tempo # store traces in this bucket | |
| chunk_buffer_size: 10485760 # optional. buffer size for reads. default = 10MiB | |
| endpoint: https://storage.googleapis.com/storage/v1/ # optional. api endpoint override | |
| insecure: false # optional. Set to true to disable authentication | |
| # and certificate checks. | |
| project_id: project # optional if bucket already exists. Project ID | |
| # to use when creating bucket on first run. | |
| ``` | |
| trace: | |
| backend: gcs # Store traces in gcs. | |
| s3: | |
| bucket_name: tempo # Store traces in this bucket. | |
| chunk_buffer_size: 10485760 # Optional, buffer size for reads. Default = 10MiB | |
| endpoint: https://storage.googleapis.com/storage/v1/ # Optional, API endpoint override. | |
| insecure: false # Optional, set to true to disable authentication | |
| # and certificate checks. | |
| project_id: project # Optional, if bucket already exists. Project ID | |
| # to use when creating a bucket on the first run. |
| http_listen_port: 3100 | ||
|
|
||
| distributor: | ||
| receivers: # this configuration will listen on all ports and protocols that tempo is capable of. |
There was a problem hiding this comment.
Suggested change
| receivers: # this configuration will listen on all ports and protocols that tempo is capable of. | |
| receivers: # This configuration will listen on all ports and protocols that tempo is capable of listening to. |
Comment on lines
+8
to
+12
| jaeger: # the receives all come from the OpenTelemetry collector. more configuration information can | ||
| protocols: # be found there: https://github.com/open-telemetry/opentelemetry-collector/tree/master/receiver | ||
| thrift_http: # | ||
| grpc: # for a production deployment you should only enable the receivers you need! | ||
| thrift_binary: |
There was a problem hiding this comment.
Suggested change
| jaeger: # the receives all come from the OpenTelemetry collector. more configuration information can | |
| protocols: # be found there: https://github.com/open-telemetry/opentelemetry-collector/tree/master/receiver | |
| thrift_http: # | |
| grpc: # for a production deployment you should only enable the receivers you need! | |
| thrift_binary: | |
| jaeger: # Receives all traces? coming from the OpenTelemetry collector. More configuration information can | |
| protocols: # be found here: https://github.com/open-telemetry/opentelemetry-collector/tree/master/receiver | |
| thrift_http: # | |
| grpc: # For a production deployment you should only enable the receivers you need! | |
| thrift_binary: |
|
|
||
| storage: | ||
| trace: | ||
| backend: gcs # backend configuration to use |
There was a problem hiding this comment.
Suggested change
| backend: gcs # backend configuration to use | |
| backend: gcs # The backend configuration to use. |
| trace: | ||
| backend: gcs # backend configuration to use | ||
| wal: | ||
| path: /tmp/tempo/wal # where to store the the wal locally |
There was a problem hiding this comment.
Suggested change
| path: /tmp/tempo/wal # where to store the the wal locally | |
| path: /tmp/tempo/wal # Path to store the wal locally. |
| wal: | ||
| path: /tmp/tempo/wal # where to store the the wal locally | ||
| bloom_filter_false_positive: .05 # bloom filter false positive rate. lower values create larger filters but fewer false positives | ||
| index_downsample: 10 # number of traces per index record |
There was a problem hiding this comment.
Suggested change
| index_downsample: 10 # number of traces per index record | |
| index_downsample: 10 # Number of traces per index record. |
| insecure: true | ||
| project_id: test | ||
| pool: | ||
| max_workers: 100 # the worker pool mainly drives querying, but is also used for polling the blocklist |
There was a problem hiding this comment.
Suggested change
| max_workers: 100 # the worker pool mainly drives querying, but is also used for polling the blocklist | |
| max_workers: 100 # The worker pool mainly drives querying but it is also used for polling the blocklist. |
Comment on lines
+39
to
+40
| bloom_filter_false_positive: .05 # bloom filter false positive rate. lower values create larger filters but fewer false positives | ||
| index_downsample: 10 # number of traces per index record |
There was a problem hiding this comment.
Suggested change
| bloom_filter_false_positive: .05 # bloom filter false positive rate. lower values create larger filters but fewer false positives | |
| index_downsample: 10 # number of traces per index record | |
| bloom_filter_false_positive: .05 # The bloom filter false positive rate. Lower values create larger filters but fewer false positives. | |
| index_downsample: 10 # Number of traces per index record. |
|
For the yaml comments I personally prefer lower case and brevity to grammatically correct sentences. my .02 |
Co-authored-by: achatterjee-grafana <[email protected]>
joe-elliott
approved these changes
Dec 8, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does:
This PR addresses the silent errors encountered here: #379 The root cause was the consumption of the iterator in getting tenant and block list for GCS backend. An error state with permissions could cause the loop to run infinitely and polling the blocklist would never complete. This behavior is cautioned about here: https://github.com/googleapis/google-cloud-go/blob/master/storage/bucket.go#L1183
This PR includes many updates, which will be explained here:
endpoint:stringto redirect api traffic.insecure:boolto disable authentication for the api usingoption.WithoutAuthentication(), which allows the storage client to run without error.STORAGE_EMULATOR_HOSTenvironment variable and when present configures the api to speak to the host using http. However the storage client then loses the api prefix configured in the endpoint ("/storage/v1/") upon the first file write, breaking subsequent api calls. See code here: https://github.com/googleapis/google-cloud-go/blob/master/storage/writer.go#L129Other implementation notes:
http.DefaultTransportwhich is then editedWhich issue(s) this PR fixes:
Fixes #379
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]