From 32cbf2a257466551d39625ab6981eabfd3daf6ac Mon Sep 17 00:00:00 2001 From: Mauri de Souza Meneguzzo Date: Thu, 30 Jan 2025 06:04:20 -0300 Subject: [PATCH] [exporter/elasticsearch] [chore] add docs for logs_dynamic_id (#37472) #### Description Adds docs for setting the document id dynamically. #### Link to tracking issue Related https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/36882 --- exporter/elasticsearchexporter/README.md | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/exporter/elasticsearchexporter/README.md b/exporter/elasticsearchexporter/README.md index d78d5cba8421..0c5b4b0dbd52 100644 --- a/exporter/elasticsearchexporter/README.md +++ b/exporter/elasticsearchexporter/README.md @@ -143,7 +143,7 @@ This can be customised through the following settings: - `date_format`(default=`%Y.%m.%d`): Time format (based on strftime) to generate the second part of the Index name. - `logs_dynamic_id` (optional): Dynamically determines the document ID to be used in Elasticsearch based on a log record attribute. - - `enabled`(default=false): Enable/Disable dynamic ID for log records. If `elasticsearch.document_id` exists and is not an empty string in the log record attributes, it will be used as the document ID. Otherwise, the document ID will be generated by Elasticsearch. The attribute `elasticsearch.document_id` is removed from the final document. + - `enabled`(default=false): Enable/Disable dynamic ID for log records. If `elasticsearch.document_id` exists and is not an empty string in the log record attributes, it will be used as the document ID. Otherwise, the document ID will be generated by Elasticsearch. The attribute `elasticsearch.document_id` is removed from the final document. See [Setting a document id dynamically](#setting-a-document-id-dynamically). ### Elasticsearch document mapping @@ -349,6 +349,26 @@ Otherwise, it is mapped to an empty string (""). In case the record contains `timestamp`, this value is used. Otherwise, the `observed timestamp` is used. +## Setting a document id dynamically + +The `logs_dynamic_id` setting allows users to set the document ID dynamically based on a log record attribute. +Besides the ability to control the document ID, this setting also works as a deduplication mechanism, as Elasticsearch will refuse to index a document with the same ID. + +The log record attribute `elasticsearch.document_id` can be set explicitly by a processor based on the log record. + +As an example, the `transform` processor can create this attribute dynamically: + +```yaml +processors: + transform/es-doc-id: + error_mode: ignore + log_statements: + - context: log + condition: attributes["event_name"] != null && attributes["event_creation_time"] != null + statements: + - set(attributes["elasticsearch.document_id"], Concat(["log", attributes["event_name"], attributes["event_creation_time"], "-")) +``` + ## Known issues ### version_conflict_engine_exception