-
Notifications
You must be signed in to change notification settings - Fork 2.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add s3 storage class to the AWS S3 exporter #35574
Conversation
im still waiting :D |
"I’m really looking forward to this feature release as soon as possible. Please review it soon, @atoulme." |
This PR was marked stale due to lack of activity. It will be closed in 14 days. |
@atoulme anything else required for merge this? |
This PR was marked stale due to lack of activity. It will be closed in 14 days. |
exporter/awss3exporter/README.md
Outdated
@@ -32,6 +32,7 @@ The following exporter configuration parameters are supported. | |||
| `encoding_file_extension` | file format extension suffix when using the `encoding` configuration option. May be left empty for no suffix to be appended. | | | |||
| `endpoint` | (REST API endpoint) overrides the endpoint used by the exporter instead of constructing it from `region` and `s3_bucket` | | | |||
| `s3_force_path_style` | [set this to `true` to force the request to use path-style addressing](http://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html) | false | | |||
| `storage_class` | S3 storageclass | STANDARD |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What are the allowed values for this config key? Is it a known enumeration? Is there a link to docs? Any validation we can perform?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The issue (#35173) mentions STANDARD_IA, is STANDARD valid?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added documentation about the storage class valid APIs and add the all possible options
Please rebase off latest main. Please add a changelog entry. |
@atoulme which changelog ? |
This PR was marked stale due to lack of activity. It will be closed in 14 days. |
Closed as inactive. Feel free to reopen if this PR is still being worked on. |
Please reopen 😭 |
80386bb
to
968e5dd
Compare
@atoulme are QueueSettings defaults in place ? After rebasing with master have I seen this. currently:
Specifically:
I fixed initialising This means any test must initialise |
yes your expected default config should have the default queue settings values. |
Fixed the test I left them blank , thanks to the lintern I catch it , sorry for the noise , first PR here. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a couple nits
exporter/awss3exporter/config.go
Outdated
@@ -68,6 +70,10 @@ func (c *Config) Validate() error { | |||
if c.S3Uploader.S3Bucket == "" && c.S3Uploader.Endpoint == "" { | |||
errs = multierr.Append(errs, errors.New("bucket or endpoint is required")) | |||
} | |||
if c.S3Uploader.StorageClass == "" { | |||
c.S3Uploader.StorageClass = "STANDARD" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The validate method should not modify the config. This should be set in the factory's CreateDefaultConfig
.
We seem to be missing actual validation here too. What if the user enters an invalid value?
exporter/awss3exporter/README.md
Outdated
| `s3_force_path_style` | [set this to `true` to force the request to use path-style addressing](http://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html) | false | | ||
| `storage_class` | [S3 storageclass](https://docs.aws.amazon.com/AmazonS3/latest/userguide/sc-howtoset.html) | STANDARD |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| `storage_class` | [S3 storageclass](https://docs.aws.amazon.com/AmazonS3/latest/userguide/sc-howtoset.html) | STANDARD |
Duplicate
bucket string | ||
builder *PartitionKeyBuilder | ||
uploader *manager.Uploader | ||
storageClass string |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
storageClass string | |
storageClass s3types.StorageClass |
@@ -26,6 +26,7 @@ func TestNewS3Manager(t *testing.T) { | |||
"my-bucket", | |||
&PartitionKeyBuilder{}, | |||
s3.New(s3.Options{}), | |||
"STANDARD", // <-- Add the new argument here |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"STANDARD", // <-- Add the new argument here | |
"STANDARD", |
@@ -139,6 +153,7 @@ func TestS3ManagerUpload(t *testing.T) { | |||
BaseEndpoint: aws.String(s.URL), | |||
Region: "local", | |||
}), | |||
"STANDARD_IA", // <-- Pass your desired storage class here |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"STANDARD_IA", // <-- Pass your desired storage class here | |
"STANDARD_IA", |
Thanks I will try to add your feedback between this Friday and Monday |
@atoulme and @djaglowski should I remove this lintern |
I'm not familiar with the lint error but I think we just need the lint check to pass so feel free to experiment. |
**Description:** Add s3 storage class <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> **Link to tracking Issue:** open-telemetry#35173 **Testing:** Unit for the config interface **Documentation:** Added storage class for the S3
Description: Add s3 storage class
Link to tracking Issue: #35173
Testing: Unit for the config interface
Documentation: Added storage class for the S3