Add logging support to the Elasticsearch exporter #1800
Comments
|
Thanks Janaa. I'll clarify by midweek how we are moving forward on on adding logs support to the Elastic exporter. |
|
Currently working on a logs Elasticsearch exporter for the C++ repo, it'd be useful to see how the collector defines the mapping between OTEL log fields to Elasticsearch so that we can stay consistent. |
|
Thanks for your patience @rakyll and @MarkSeufert. Elastic plans to work on the OpenTelemetry Collector exporter toward Elasticsearch right now and it should not be a long task. @rakyll we would like to verify with you one point please: the PR #1801 you initiated connects to Elastic APM Server rather than connecting to Elasticsearch (see the line of code here).
@rakyll we guess that you also have in mind the addition of a direct exporter toward Elasticsearch, is this correct? If we are aligned on this vision, we can start implementing with no delay. Moreover, @MarkSeufert highlighted an important point: a clear and documented mapping between OpenTelemetry Semantic Conventions and the Elasticsearch data model: Elastic Common Schema. The specification work started a while ago on https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/logs/data-model.md#elastic-common-schema . We have designed proof of concept to have clarity on this mapping, we propose to use the Pull Requests of this Elasticsearch exporter to validate with the OpenTelemetry community the mapping decisions. Would this work for you? |
|
@cyrille-leclerc would you be interested in contributing an exporter. We can help you to validate the scheme mapping. Sorry for opening #1801. I was just looking into Elastic for the first time and wanted to draft a change to note I was looking into it but didn't have time. I'd be happy if you can take over. |
|
As has been pointed out by @cyrille-leclerc, the current elastic exporter does not ingest data right into Elasticsearch, but forwards metrics and traces to the Elastic APM Server. The APM server applies some additional processing before indexing traces and metrics as documents in Elasticsearch. In addition to forwarding APM-Server, we plan to add a second "mode" to the elastic exporter. For now we will support indexing only logs directly, but it would be nice to add metrics as a follow up later. We will introduce new settings like We expect events to be published to follow the OpenTelemetry Semantics. The OTel Semantics to ECS mapping hasn't been maintained for some time. In the process we will look for required updates and add support for users to configure the mapping. I hope to have an initial PR ready soon. |
|
cc @alolita |
This is the first step in adding the Elasticsearch exporter. Initially we will only support the Logs exporter interface, and potentially will add metrics in the future as well. This change only provides some boilerplate initializing the exporter. But the exporter is not yet usable (or part of) any opentelemetry collector distribution. The elasticsearch exporter is based on the official [go-elasticsearch](https://github.com/elastic/go-elasticsearch) client. We will use the BulkIndexer provided by the client for event publishing. The client and BulkIndexer provide some support for retrying already. The Elasticsearch Bulk API can report errors at the HTTP level, but uses selective ACKs for individual events. This allows us to retry only failed events and/or reject events that can not be indexed (e.g. due to an mapping error). The 429 error code might even inidcate that we should backoff a little before retrying. **Link to tracking Issue:** open-telemetry#1800 **Testing:** Only configuration loading and validation tests have been added so far. The exporter currently panics when trying to publish events. More unit and integration tests will be added in the future. **Documentation:** All settings that will be available initially are documented in the README.md file.
* Init Elasticsearch exporter This is the first step in adding the Elasticsearch exporter. Initially we will only support the Logs exporter interface, and potentially will add metrics in the future as well. This change only provides some boilerplate initializing the exporter. But the exporter is not yet usable (or part of) any opentelemetry collector distribution. The elasticsearch exporter is based on the official [go-elasticsearch](https://github.com/elastic/go-elasticsearch) client. We will use the BulkIndexer provided by the client for event publishing. The client and BulkIndexer provide some support for retrying already. The Elasticsearch Bulk API can report errors at the HTTP level, but uses selective ACKs for individual events. This allows us to retry only failed events and/or reject events that can not be indexed (e.g. due to an mapping error). The 429 error code might even inidcate that we should backoff a little before retrying. **Link to tracking Issue:** #1800 **Testing:** Only configuration loading and validation tests have been added so far. The exporter currently panics when trying to publish events. More unit and integration tests will be added in the future. **Documentation:** All settings that will be available initially are documented in the README.md file. * Rename urls setting to endpoints * fix lint * Add factory and exporter initialization tests * Lint checks * Error lint * Fix import order * fix typo * Do not "shadow" err in a deferred func * review * fix typo in exporter_test.go * use const for environmant variable name in tests * fix format after gorename * typo
* Init Elasticsearch exporter This is the first step in adding the Elasticsearch exporter. Initially we will only support the Logs exporter interface, and potentially will add metrics in the future as well. This change only provides some boilerplate initializing the exporter. But the exporter is not yet usable (or part of) any opentelemetry collector distribution. The elasticsearch exporter is based on the official [go-elasticsearch](https://github.com/elastic/go-elasticsearch) client. We will use the BulkIndexer provided by the client for event publishing. The client and BulkIndexer provide some support for retrying already. The Elasticsearch Bulk API can report errors at the HTTP level, but uses selective ACKs for individual events. This allows us to retry only failed events and/or reject events that can not be indexed (e.g. due to an mapping error). The 429 error code might even inidcate that we should backoff a little before retrying. **Link to tracking Issue:** #1800 **Testing:** Only configuration loading and validation tests have been added so far. The exporter currently panics when trying to publish events. More unit and integration tests will be added in the future. **Documentation:** All settings that will be available initially are documented in the README.md file. * Rename urls setting to endpoints * fix lint * Add factory and exporter initialization tests * Lint checks * Error lint * Fix import order * fix typo * Do not "shadow" err in a deferred func * review * fix typo in exporter_test.go * use const for environmant variable name in tests * fix format after gorename * typo
**Description:** The change adds support for encoding OpenTelemetry log records to JSON. The encoder tries to remove duplicate entries in case the attribute map (which is an array if key value pairs) contains duplicates. Mixed style attributes, with key names having dots and other fields having attribute maps as value will be normalized, such that the JSON encoding will be either completely flat, or values are properly merged into a single JSON object (when dedot is enabled). The normalization helps with deduplication, and (not yet implemented) dedot support will allow us to present a well formated JSON event if Ingest Node is used (The dedotting in Elasticsearch does happen after Ingest Node). Next: - Dedotting support - Custom (configurable) field mapping - `publishLogs` unit testing - Integration tests **Link to tracking Issue:** #1800 **Testing:** The internal document type with field deduplication is fully tested (89%) via unit tests. The change also hooks up `publishLogs`, but this functionality is not covered by tests yet, as the PR has already grown quite a bit (I tested locally with a custom otel collector distribution). I would like to add additional tests in a separate PR, to keep focused on the JSON encoding only here.
|
Hey guys, what's the status of the exporter? Couldn't determine if it's "released" yet. Didn't find any documentation. |
|
Hello All! |
Add dedot support to Elasticsearch exporter. With all fields being sorted we iterate over the keys and dedot while serializing the document to JSON. **Link to tracking Issue:** #1800 **Testing:** Unit tests for dedot and flat serialization have been added. **Documentation:** The dedot setting has already been documented. This PR adds the missing feature.
|
I have tried to add the component locally, logs and traces seem to work but the thing that I need which is metrics is still not there. Any update on the timelines and if I can contribute? |
|
|
* Changes stdout to expose the actual tracerProvider This enables flushing and stopping the exporter. * updated changelog * Update CHANGELOG.md Co-authored-by: Tyler Yahn <[email protected]> Co-authored-by: Tyler Yahn <[email protected]>
|
The logs exporter is currently shipped as part of the contrib distribution: https://github.com/open-telemetry/opentelemetry-collector-releases/blob/main/distributions/otelcol-contrib/manifest.yaml Closing the original issue. If there are additional enhancement requests, please open separate items for them. |
This issue is moved here from #1630 as a tracking bug.
@cyrille-leclerc Let me know if anyone is already working on this. This future is on our TODO list and we can help if no one is actively working on it.
The text was updated successfully, but these errors were encountered: