Skip to content

Commit

Permalink
Address review feedback.
Browse files Browse the repository at this point in the history
Remove exporter-specific settings and use confighttp.HTTPClientSettings and exporterhelper.RetrySettings

Update README to reflect the above.

Signed-off-by: Max Ksyunz <[email protected]>
  • Loading branch information
MaxKsyunz committed Jul 19, 2023
1 parent 9575754 commit 4d7d124
Show file tree
Hide file tree
Showing 7 changed files with 188 additions and 154 deletions.
57 changes: 21 additions & 36 deletions exporter/opensearchexporter/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,54 +6,39 @@
| Supported pipeline types | traces |
| Distributions | [contrib] |

This exporter supports sending OpenTelemetry signals as documents to [OpenSearch](https://www.opensearch.org).
OpenSearch exporter supports sending OpenTelemetry signals as documents to [OpenSearch](https://www.opensearch.org).

The documents are sent using [observability catalog](https://github.com/opensearch-project/opensearch-catalog/tree/main/schema/observability) schema.

## Configuration options
### Indexing Options
- `dataset` (default=`default`) a user-provided label to classify source of telemetry. It is used to construct the name of the destination index or data stream.
- `namespace` (default=`namespace`) a user-provided label to group telemetry. It is used to construct the name of the destination index or data stream.

- `endpoints`: List of OpenSearch URLs. If endpoints is missing, the
OPENSEARCH_URL environment variable will be used.
- `num_workers` (optional): Number of workers publishing bulk requests concurrently. Defaults to `runtime.NumCPU()`, which returns the number of logical CPUs usable by the current process.
- `dataset` (default=`default`) a user-provided label. It is used to construct the name of the destination index or data stream.
- `namespace` (default=`namespace`)a user-provided label. It is used to construct the name of the destination index or data stream.
- `flush`: Event bulk buffer flush settings
- `bytes` (default=5242880): Write buffer flush limit.
- `interval` (default=30s): Write buffer time limit.
- `retry`: Event retry settings
- `enabled` (default=true): Enable/Disable event retry on error. Retry
support is enabled by default.
- `max_requests` (default=3): Number of HTTP request retries.
- `initial_interval` (default=100ms): Initial waiting time if a HTTP request failed.
- `max_interval` (default=1m): Max waiting time if a HTTP request failed.

### HTTP settings

- `read_buffer_size` (default=0): Read buffer size.
- `write_buffer_size` (default=0): Write buffer size used when.
- `timeout` (default=90s): HTTP request time limit.
- `headers` (optional): Headers to be send with each HTTP request.

### Security and Authentication settings

- `user` (optional): Username used for HTTP Basic Authentication.
- `password` (optional): Password used for HTTP Basic Authentication.
### HTTP Connection Options
OpenSearch export supports standard (HTTP client settings](https://github.com/open-telemetry/opentelemetry-collector/tree/main/config/confighttp#client-configuration).
- `endpoint` (required) `<url>:<port>` of OpenSearch node to send data to.

### TLS settings
- `ca_file` (optional): Root Certificate Authority (CA) certificate, for
verifying the server's identity, if TLS is enabled.
- `cert_file` (optional): Client TLS certificate.
- `key_file` (optional): Client TLS key.
- `insecure` (optional): In gRPC when set to true, this is used to disable the client transport security. In HTTP, this disables verifying the server's certificate chain and host name.
- `insecure_skip_verify` (optional): Will enable TLS but not verify the certificate.
is enabled.
Supports standard TLS settings as part of HTTP settings. See [TLS Configuration/Client Settings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md#client-configuration).

### Retry Options
- `retry_on_failure`: See [retry_on_failure](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md)

## Example

```yaml
extensions:
basicauth/client:
client_auth:
username: username
password: password

exporters:
opensearch/trace:
endpoints: [https://opensearch.example.com:9200]
endpoint: https://opensearch.example.com:9200
auth:
authenticator: basicauth/client
# ······
service:
pipelines:
Expand All @@ -63,4 +48,4 @@ service:
processors: [batch]
```
[devel]:https://github.com/open-telemetry/opentelemetry-collector#development
[contrib]:https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
[contrib]:https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
110 changes: 9 additions & 101 deletions exporter/opensearchexporter/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,118 +5,26 @@ package opensearchexporter // import "github.com/open-telemetry/opentelemetry-co

import (
"errors"
"os"
"time"

"go.opentelemetry.io/collector/config/configopaque"
"go.opentelemetry.io/collector/config/configtls"
)

const (
defaultOpenSearchEnvName = "OPENSEARCH_URL"
"go.opentelemetry.io/collector/config/confighttp"
"go.opentelemetry.io/collector/exporter/exporterhelper"
)

// Config defines configuration for OpenSearch exporter.
type Config struct {

// Endpoints holds the OpenSearch URLs the exporter should send events to.
//
// OPENSEARCH_URL environment variable is not set.
Endpoints []string `mapstructure:"endpoints"`

// NumWorkers configures the number of workers publishing bulk requests.
NumWorkers int `mapstructure:"num_workers"`

HTTPClientSettings `mapstructure:",squash"`
Retry RetrySettings `mapstructure:"retry"`
Flush FlushSettings `mapstructure:"flush"`
Namespace string `mapstructure:"namespace"`
Dataset string `mapstructure:"dataset"`
}

type HTTPClientSettings struct {
Authentication AuthenticationSettings `mapstructure:",squash"`

// ReadBufferSize for HTTP client. See http.Transport.ReadBufferSize.
ReadBufferSize int `mapstructure:"read_buffer_size"`

// WriteBufferSize for HTTP client. See http.Transport.WriteBufferSize.
WriteBufferSize int `mapstructure:"write_buffer_size"`

// Timeout configures the HTTP request timeout.
Timeout time.Duration `mapstructure:"timeout"`

// Headers allows users to configure optional HTTP headers that
// will be sent with each HTTP request.
Headers map[string]string `mapstructure:"headers,omitempty"`

configtls.TLSClientSetting `mapstructure:"tls,omitempty"`
}

// AuthenticationSettings defines user authentication related settings.
type AuthenticationSettings struct {
// User is used to configure HTTP Basic Authentication.
User string `mapstructure:"user"`

// Password is used to configure HTTP Basic Authentication.
Password configopaque.String `mapstructure:"password"`
}

// FlushSettings defines settings for configuring the write buffer flushing
// policy in the OpenSearch exporter. The exporter sends a bulk request with
// all events already serialized into the send-buffer.
type FlushSettings struct {
// Bytes sets the send buffer flushing limit.
Bytes int `mapstructure:"bytes"`

// Interval configures the max age of a document in the send buffer.
Interval time.Duration `mapstructure:"interval"`
}

// RetrySettings defines settings for the HTTP request retries in the OpenSearch exporter.
// Failed sends are retried with exponential backoff.
type RetrySettings struct {
// Enabled allows users to disable retry without having to comment out all settings.
Enabled bool `mapstructure:"enabled"`

// MaxRequests configures how often an HTTP request is retried before it is assumed to be failed.
MaxRequests int `mapstructure:"max_requests"`

// InitialInterval configures the initial waiting time if a request failed.
InitialInterval time.Duration `mapstructure:"initial_interval"`

// MaxInterval configures the max waiting time if consecutive requests failed.
MaxInterval time.Duration `mapstructure:"max_interval"`
}

type MappingsSettings struct {
// Mode configures the field mappings.
Mode string `mapstructure:"mode"`

// Try to find and remove duplicate fields
Dedup bool `mapstructure:"dedup"`

Dedot bool `mapstructure:"dedot"`
confighttp.HTTPClientSettings `mapstructure:",squash"`
exporterhelper.RetrySettings `mapstructure:"retry_on_failure"`
Namespace string `mapstructure:"namespace"`
Dataset string `mapstructure:"dataset"`
}

var (
errConfigNoEndpoint = errors.New("endpoints must be specified")
errConfigEmptyEndpoint = errors.New("endpoints must not include empty entries")
errConfigNoEndpoint = errors.New("endpoint must be specified")
)

// Validate validates the opensearch server configuration.
func (cfg *Config) Validate() error {
if len(cfg.Endpoints) == 0 {
if os.Getenv(defaultOpenSearchEnvName) == "" {
return errConfigNoEndpoint
}
if len(cfg.Endpoint) == 0 {
return errConfigNoEndpoint
}

for _, endpoint := range cfg.Endpoints {
if endpoint == "" {
return errConfigEmptyEndpoint
}
}

return nil
}
83 changes: 83 additions & 0 deletions exporter/opensearchexporter/config_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
// Copyright The OpenTelemetry Authors
// SPDX-License-Identifier: Apache-2.0

package opensearchexporter

import (
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"go.opentelemetry.io/collector/component"
"go.opentelemetry.io/collector/config/configauth"
"go.opentelemetry.io/collector/config/confighttp"
"go.opentelemetry.io/collector/config/configopaque"
"go.opentelemetry.io/collector/confmap/confmaptest"
"go.opentelemetry.io/collector/exporter/exporterhelper"
"path/filepath"
"testing"
"time"
)

func TestLoadConfig(t *testing.T) {
t.Parallel()

cm, err := confmaptest.LoadConf(filepath.Join("testdata", "config.yaml"))
require.NoError(t, err)

defaultCfg := newDefaultConfig()
defaultCfg.(*Config).Endpoint = "https://opensearch.example.com:9200"
maxIdleConns := 100
idleConnTimeout := 90 * time.Second

tests := []struct {
id component.ID
expected component.Config
configValidateAssert assert.ErrorAssertionFunc
}{
{
id: component.NewIDWithName(typeStr, ""),
expected: defaultCfg,
configValidateAssert: assert.NoError,
},
{
id: component.NewIDWithName(typeStr, "trace"),
expected: &Config{
Dataset: "ngnix",
Namespace: "eu",
HTTPClientSettings: confighttp.HTTPClientSettings{
Endpoint: "https://opensearch.example.com:9200",
Timeout: 2 * time.Minute,
Headers: map[string]configopaque.String{
"myheader": "test",
},
MaxIdleConns: &maxIdleConns,
IdleConnTimeout: &idleConnTimeout,
Auth: &configauth.Authentication{AuthenticatorID: component.NewID("sample_basic_auth")},
},
RetrySettings: exporterhelper.RetrySettings{
Enabled: true,
InitialInterval: 100 * time.Millisecond,
MaxInterval: 30 * time.Second,
MaxElapsedTime: 5 * time.Minute,
Multiplier: 1.5,
RandomizationFactor: 0.5,
},
},
configValidateAssert: assert.NoError,
},
}

for _, tt := range tests {
t.Run(tt.id.String(), func(t *testing.T) {
factory := NewFactory()
cfg := factory.CreateDefaultConfig()

sub, err := cm.Sub(tt.id.String())
require.NoError(t, err)
require.NoError(t, component.UnmarshalConfig(sub, cfg))

vv := component.ValidateConfig(cfg)
tt.configValidateAssert(t, vv)
assert.Equal(t, tt.expected, cfg)
})
}
}
22 changes: 7 additions & 15 deletions exporter/opensearchexporter/factory.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,8 @@ package opensearchexporter // import "github.com/open-telemetry/opentelemetry-co

import (
"context"
"time"

"go.opentelemetry.io/collector/component"
"go.opentelemetry.io/collector/config/confighttp"
"go.opentelemetry.io/collector/exporter"
"go.opentelemetry.io/collector/exporter/exporterhelper"
"go.opentelemetry.io/collector/pdata/ptrace"
Expand All @@ -24,24 +23,17 @@ const (
func NewFactory() exporter.Factory {
return exporter.NewFactory(
typeStr,
createDefaultConfig,
newDefaultConfig,
exporter.WithTraces(createTracesExporter, stability),
)
}

func createDefaultConfig() component.Config {
func newDefaultConfig() component.Config {
return &Config{
HTTPClientSettings: HTTPClientSettings{
Timeout: 90 * time.Second,
},
Namespace: "namespace",
Dataset: "default",
Retry: RetrySettings{
Enabled: true,
MaxRequests: 3,
InitialInterval: 100 * time.Millisecond,
MaxInterval: 1 * time.Minute,
},
HTTPClientSettings: confighttp.NewDefaultHTTPClientSettings(),
Namespace: "namespace",
Dataset: "default",
RetrySettings: exporterhelper.NewDefaultRetrySettings(),
}
}

Expand Down
20 changes: 18 additions & 2 deletions exporter/opensearchexporter/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -3,35 +3,50 @@ module github.com/open-telemetry/opentelemetry-collector-contrib/exporter/opense
go 1.19

require (
github.com/stretchr/testify v1.8.4
go.opentelemetry.io/collector/component v0.81.0
go.opentelemetry.io/collector/config/confighttp v0.81.0
go.opentelemetry.io/collector/config/configopaque v0.81.0
go.opentelemetry.io/collector/config/configtls v0.81.0
go.opentelemetry.io/collector/confmap v0.81.0
go.opentelemetry.io/collector/exporter v0.81.0
go.opentelemetry.io/collector/pdata v1.0.0-rcv0013
)

require (
github.com/cenkalti/backoff/v4 v4.2.1 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/felixge/httpsnoop v1.0.3 // indirect
github.com/fsnotify/fsnotify v1.6.0 // indirect
github.com/go-logr/logr v1.2.4 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/protobuf v1.5.3 // indirect
github.com/golang/snappy v0.0.4 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/klauspost/compress v1.16.6 // indirect
github.com/knadh/koanf v1.5.0 // indirect
github.com/knadh/koanf/v2 v2.0.1 // indirect
github.com/mitchellh/copystructure v1.2.0 // indirect
github.com/mitchellh/mapstructure v1.5.1-0.20220423185008-bf980b35cac4 // indirect
github.com/mitchellh/reflectwalk v1.0.2 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/rs/cors v1.9.0 // indirect
go.opencensus.io v0.24.0 // indirect
go.opentelemetry.io/collector v0.81.0 // indirect
go.opentelemetry.io/collector/config/configauth v0.81.0 // indirect
go.opentelemetry.io/collector/config/configcompression v0.81.0 // indirect
go.opentelemetry.io/collector/config/configtelemetry v0.81.0 // indirect
go.opentelemetry.io/collector/confmap v0.81.0 // indirect
go.opentelemetry.io/collector/config/configtls v0.81.0 // indirect
go.opentelemetry.io/collector/config/internal v0.81.0 // indirect
go.opentelemetry.io/collector/consumer v0.81.0 // indirect
go.opentelemetry.io/collector/extension v0.81.0 // indirect
go.opentelemetry.io/collector/extension/auth v0.81.0 // indirect
go.opentelemetry.io/collector/featuregate v1.0.0-rcv0013 // indirect
go.opentelemetry.io/collector/processor v0.81.0 // indirect
go.opentelemetry.io/collector/receiver v0.81.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.42.0 // indirect
go.opentelemetry.io/otel v1.16.0 // indirect
go.opentelemetry.io/otel/metric v1.16.0 // indirect
go.opentelemetry.io/otel/trace v1.16.0 // indirect
Expand All @@ -44,4 +59,5 @@ require (
google.golang.org/genproto/googleapis/rpc v0.0.0-20230530153820-e85fd2cbaebc // indirect
google.golang.org/grpc v1.56.1 // indirect
google.golang.org/protobuf v1.31.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)
Loading

0 comments on commit 4d7d124

Please sign in to comment.