You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardexpand all lines: CHANGELOG.md
+32-1
Original file line number
Diff line number
Diff line change
@@ -9,11 +9,20 @@ NOTE: As semantic versioning states all 0.y.z releases can contain breaking chan
9
9
We use *breaking :warning:* to mark changes that are not backward compatible (relates only to v0.y.z releases.)
10
10
11
11
## Unreleased
12
+
-[#38](https://github.com/thanos-io/objstore/pull/38) GCS: Upgrade cloud.google.com/go/storage version to `v1.43.0`.
13
+
-[#145](https://github.com/thanos-io/objstore/pull/145) Include content length in the response of Get and GetRange.
12
14
13
15
### Fixed
16
+
-[#153](https://github.com/thanos-io/objstore/pull/153) Metrics: Fix `objstore_bucket_operation_duration_seconds_*` for `get` and `get_range` operations.
17
+
-[#117](https://github.com/thanos-io/objstore/pull/117) Metrics: Fix `objstore_bucket_operation_failures_total` incorrectly incremented if context is cancelled while reading object contents.
18
+
-[#115](https://github.com/thanos-io/objstore/pull/115) GCS: Fix creation of bucket with GRPC connections. Also update storage client to `v1.40.0`.
19
+
-[#102](https://github.com/thanos-io/objstore/pull/102) Azure: bump azblob sdk to get concurrency fixes.
14
20
-[#33](https://github.com/thanos-io/objstore/pull/33) Tracing: Add `ContextWithTracer()` to inject the tracer into the context.
15
21
-[#34](https://github.com/thanos-io/objstore/pull/34) Fix ignored options when creating shared credential Azure client.
16
22
-[#62](https://github.com/thanos-io/objstore/pull/62) S3: Fix ignored context cancellation in `Iter` method.
23
+
-[#77](https://github.com/thanos-io/objstore/pull/77) Fix buckets wrapped with metrics from being unable to determine object sizes in `Upload`.
24
+
-[#78](https://github.com/thanos-io/objstore/pull/78) S3: Fix possible concurrent modification of the PutUserMetadata map.
25
+
-[#79](https://github.com/thanos-io/objstore/pull/79) Metrics: Fix `objstore_bucket_operation_duration_seconds` for `iter` operations.
> This also changes the behaviour of `client.NewBucket`. Now it returns, uninstrumented and untraced bucket.
29
38
You can combine `objstore.WrapWithMetrics` and `tracing/{opentelemetry,opentracing}.WrapWithTraces` to have old behavior.
39
+
-[#69](https://github.com/thanos-io/objstore/pull/69)[#66](https://github.com/thanos-io/objstore/pull/66) Add `objstore_bucket_operation_transferred_bytes` that counts the number of total bytes read from the bucket operation Get/GetRange and also counts the number of total bytes written to the bucket operation Upload.
-[#73](https://github.com/thanos-io/objstore/pull/73) Аdded file path to erros from DownloadFile
42
+
-[#51](https://github.com/thanos-io/objstore/pull/51) Azure: Support using connection string authentication.
43
+
-[#76](https://github.com/thanos-io/objstore/pull/76) GCS: Query for object names only in `Iter` to possibly improve performance when listing objects.
44
+
-[#85](https://github.com/thanos-io/objstore/pull/85) S3: Allow checksum algorithm to be configured
45
+
-[#92](https://github.com/thanos-io/objstore/pull/92) GCS: Allow using a gRPC client.
46
+
-[#94](https://github.com/thanos-io/objstore/pull/94) Allow timingReadCloser to be seeker
47
+
-[#96](https://github.com/thanos-io/objstore/pull/96) Allow nopCloserWithObjectSize to be seeker
48
+
-[#86](https://github.com/thanos-io/objstore/pull/86) GCS: Add HTTP Config to GCS
-[#116](https://github.com/thanos-io/objstore/pull/116) Azure: Add new storage_create_container configuration property
54
+
-[#128](https://github.com/thanos-io/objstore/pull/128) GCS: Add support for `ChunkSize` for writer.
55
+
-[#130](https://github.com/thanos-io/objstore/pull/130) feat: Decouple creating bucket metrics from instrumenting the bucket
56
+
-[#150](https://github.com/thanos-io/objstore/pull/150) Add support for roundtripper wrapper.
30
57
31
58
### Changed
32
59
-[#38](https://github.com/thanos-io/objstore/pull/38)*: Upgrade minio-go version to `v7.0.45`.
33
60
-[#39](https://github.com/thanos-io/objstore/pull/39) COS: Upgrade cos sdk version to `v0.7.40`.
34
61
-[#35](https://github.com/thanos-io/objstore/pull/35) Azure: Update Azure SDK and fix breaking changes.
35
-
62
+
-[#65](https://github.com/thanos-io/objstore/pull/65)*: Upgrade minio-go version to `v7.0.61`.
63
+
-[#70](https://github.com/thanos-io/objstore/pull/70) GCS: Update cloud.google.com/go/storage version to `v1.27.0`.
64
+
-[#71](https://github.com/thanos-io/objstore/pull/71) Replace method `IsCustomerManagedKeyError` for a more generic `IsAccessDeniedErr` on the bucket interface.
65
+
-[#89](https://github.com/thanos-io/objstore/pull/89) GCS: Upgrade cloud.google.com/go/storage version to `v1.35.1`.
66
+
-[#123](https://github.com/thanos-io/objstore/pull/123)*: Upgrade minio-go version to `v7.0.71`.
Copy file name to clipboardexpand all lines: README.md
+83-7
Original file line number
Diff line number
Diff line change
@@ -48,7 +48,7 @@ See [MAINTAINERS.md](https://github.com/thanos-io/thanos/blob/main/MAINTAINERS.m
48
48
49
49
The core this module is the [`Bucket` interface](objstore.go):
50
50
51
-
```go mdox-exec="sed -n '37,50p' objstore.go"
51
+
```go mdox-exec="sed -n '39,55p' objstore.go"
52
52
// Bucket provides read and write access to an object storage bucket.
53
53
// NOTE: We assume strong consistency for write-read flow.
54
54
typeBucketinterface {
@@ -63,18 +63,31 @@ type Bucket interface {
63
63
// If object does not exist in the moment of deletion, Delete should throw error.
64
64
Delete(ctx context.Context, name string) error
65
65
66
+
// Name returns the bucket name for the provider.
67
+
Name() string
68
+
}
66
69
```
67
70
68
71
All [provider implementations](providers) have to implement `Bucket` interface that allows common read and write operations that all supported by all object providers. If you want to limit the code that will do bucket operation to only read access (smart idea, allowing to limit access permissions), you can use the [`BucketReader` interface](objstore.go):
69
72
70
-
```go mdox-exec="sed -n '68,88p' objstore.go"
71
-
73
+
```go mdox-exec="sed -n '71,106p' objstore.go"
72
74
// BucketReader provides read access to an object storage bucket.
73
75
typeBucketReaderinterface {
74
76
// Iter calls f for each entry in the given directory (not recursive.). The argument to f is the full
75
77
// object name including the prefix of the inspected directory.
78
+
76
79
// Entries are passed to function in sorted order.
77
-
Iter(ctx context.Context, dir string, f func(string) error, options ...IterOption) error
80
+
Iter(ctx context.Context, dir string, f func(name string) error, options ...IterOption) error
81
+
82
+
// IterWithAttributes calls f for each entry in the given directory similar to Iter.
83
+
// In addition to Name, it also includes requested object attributes in the argument to f.
84
+
//
85
+
// Attributes can be requested using IterOption.
86
+
// Not all IterOptions are supported by all providers, requesting for an unsupported option will fail with ErrOptionNotSupported.
87
+
IterWithAttributes(ctx context.Context, dir string, f func(attrs IterObjectAttributes) error, options ...IterOption) error
88
+
89
+
// SupportedIterOptions returns a list of supported IterOptions by the underlying provider.
90
+
SupportedIterOptions() []IterOptionType
78
91
79
92
// Get returns a reader for the given object name.
80
93
Get(ctx context.Context, name string) (io.ReadCloser, error)
@@ -88,7 +101,12 @@ type BucketReader interface {
88
101
// IsObjNotFoundErr returns true if error means that object is not found. Relevant to Get operations.
89
102
IsObjNotFoundErr(err error) bool
90
103
91
-
// IsCustomerManagedKeyError returns true if the permissions for key used to encrypt the object was revoked.
104
+
// IsAccessDeniedErr returns true if access to object is denied.
105
+
IsAccessDeniedErr(err error) bool
106
+
107
+
// Attributes returns information about the specified object.
108
+
Attributes(ctx context.Context, name string) (ObjectAttributes, error)
109
+
}
92
110
```
93
111
94
112
Those interfaces represent the object storage operations your code can use from `objstore` clients.
@@ -140,14 +158,15 @@ Thanos uses the [minio client](https://github.com/minio/minio-go) library to upl
140
158
141
159
> NOTE: S3 client was designed for AWS S3, but it can be configured against other S3-compatible object storages e.g Ceph
142
160
143
-
The S# object storage yaml configuration definition:
161
+
The S3 object storage yaml configuration definition:
144
162
145
163
```yaml mdox-exec="go run scripts/cfggen/main.go --name=s3.Config"
146
164
type: S3
147
165
config:
148
166
bucket: ""
149
167
endpoint: ""
150
168
region: ""
169
+
disable_dualstack: false
151
170
aws_sdk_auth: false
152
171
access_key: ""
153
172
insecure: false
@@ -175,6 +194,8 @@ config:
175
194
enable: false
176
195
list_objects_version: ""
177
196
bucket_lookup_type: auto
197
+
send_content_md5: true
198
+
disable_multipart: false
178
199
part_size: 67108864
179
200
sse_config:
180
201
type: ""
@@ -193,6 +214,8 @@ The field `prefix` can be used to transparently use prefixes in your S3 bucket.
193
214
194
215
The AWS region to endpoint mapping can be found in this [link](https://docs.aws.amazon.com/general/latest/gr/s3.html).
195
216
217
+
By default, the library prefers using [dual-stack endpoints](https://docs.aws.amazon.com/AmazonS3/latest/userguide/dual-stack-endpoints.html). You can explicitly disable this behaviour by setting `disable_dualstack: true`.
218
+
196
219
Make sure you use a correct signature version. Currently AWS requires signature v4, so it needs `signature_version2: false`. If you don't specify it, you will get an `Access Denied` error. On the other hand, several S3 compatible APIs use `signature_version2: true`.
197
220
198
221
You can configure the timeout settings for the HTTP client by setting the `http_config.idle_conn_timeout` and `http_config.response_header_timeout` keys. As a rule of thumb, if you are seeing errors like `timeout awaiting response headers` in your logs, you may want to increase the value of `http_config.response_header_timeout`.
@@ -346,6 +369,25 @@ type: GCS
346
369
config:
347
370
bucket: ""
348
371
service_account: ""
372
+
use_grpc: false
373
+
grpc_conn_pool_size: 0
374
+
http_config:
375
+
idle_conn_timeout: 0s
376
+
response_header_timeout: 0s
377
+
insecure_skip_verify: false
378
+
tls_handshake_timeout: 0s
379
+
expect_continue_timeout: 0s
380
+
max_idle_conns: 0
381
+
max_idle_conns_per_host: 0
382
+
max_conns_per_host: 0
383
+
tls_config:
384
+
ca_file: ""
385
+
cert_file: ""
386
+
key_file: ""
387
+
server_name: ""
388
+
insecure_skip_verify: false
389
+
disable_compression: false
390
+
chunk_size_bytes: 0
349
391
prefix: ""
350
392
```
351
393
@@ -418,6 +460,8 @@ type: AZURE
418
460
config:
419
461
storage_account: ""
420
462
storage_account_key: ""
463
+
storage_connection_string: ""
464
+
storage_create_container: false
421
465
container: ""
422
466
endpoint: ""
423
467
user_assigned_id: ""
@@ -453,6 +497,8 @@ If `msi_resource` is used, authentication is done via system-assigned managed id
453
497
454
498
If `user_assigned_id` is used, authentication is done via user-assigned managed identity. When using `user_assigned_id` the `msi_resource` defaults to `https://<storage_account>.<endpoint>`
455
499
500
+
If `storage_connection_string` is set, the values of `storage_account` and `endpoint` values will not be used. Use this method over `storage_account_key` if you need to authenticate via a SAS token.
501
+
456
502
The generic `max_retries` will be used as value for the `pipeline_config`'s `max_tries` and `reader_config`'s `max_retry_requests`. For more control, `max_retries` could be ignored (0) and one could set specific retry values.
457
503
458
504
##### OpenStack Swift
@@ -490,6 +536,22 @@ config:
490
536
connect_timeout: 10s
491
537
timeout: 5m
492
538
use_dynamic_large_objects: false
539
+
http_config:
540
+
idle_conn_timeout: 1m30s
541
+
response_header_timeout: 2m
542
+
insecure_skip_verify: false
543
+
tls_handshake_timeout: 10s
544
+
expect_continue_timeout: 1s
545
+
max_idle_conns: 100
546
+
max_idle_conns_per_host: 100
547
+
max_conns_per_host: 0
548
+
tls_config:
549
+
ca_file: ""
550
+
cert_file: ""
551
+
key_file: ""
552
+
server_name: ""
553
+
insecure_skip_verify: false
554
+
disable_compression: false
493
555
prefix: ""
494
556
```
495
557
@@ -578,7 +640,7 @@ prefix: ""
578
640
579
641
### Oracle Cloud Infrastructure Object Storage
580
642
581
-
To configure Oracle Cloud Infrastructure (OCI) Object Storage as Thanos Object Store, you need to provide appropriate authentication credentials to your OCI tenancy. The OCI object storage client implementation for Thanos supports either the default keypair or instance principal authentication.
643
+
To configure Oracle Cloud Infrastructure (OCI) Object Storage as a Thanos Object Store, you need to provide appropriate authentication credentials to your OCI tenancy. The OCI object storage client implementation for Thanos supports default keypair, instance principal, and OKE workload identity authentication.
582
644
583
645
#### API Signing Key
584
646
@@ -642,6 +704,20 @@ config:
642
704
643
705
You can also include any of the optional configuration just like the example in `Default Provider`.
644
706
707
+
#### OKE Workload Identity Provider
708
+
709
+
For Example:
710
+
711
+
```yaml
712
+
type: OCI
713
+
config:
714
+
provider: "oke-workload-identity"
715
+
bucket: ""
716
+
region: ""
717
+
```
718
+
719
+
The `bucket` and `region` fields are required. The `region` field identifies the bucket region.
720
+
645
721
##### HuaweiCloud OBS
646
722
647
723
To use HuaweiCloud OBS as an object store, you should apply for a HuaweiCloud Account to create an object storage bucket at first. More details: [HuaweiCloud OBS](https://support.huaweicloud.com/obs/index.html)
0 commit comments