Skip to content

Commit

Permalink
[exporter/elasticsearch] [chore] add docs for logs_dynamic_id (#37472)
Browse files Browse the repository at this point in the history
#### Description

Adds docs for setting the document id dynamically.

#### Link to tracking issue

Related
#36882
  • Loading branch information
mauri870 authored Jan 30, 2025
1 parent 79a2527 commit 32cbf2a
Showing 1 changed file with 21 additions and 1 deletion.
22 changes: 21 additions & 1 deletion exporter/elasticsearchexporter/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,7 @@ This can be customised through the following settings:
- `date_format`(default=`%Y.%m.%d`): Time format (based on strftime) to generate the second part of the Index name.

- `logs_dynamic_id` (optional): Dynamically determines the document ID to be used in Elasticsearch based on a log record attribute.
- `enabled`(default=false): Enable/Disable dynamic ID for log records. If `elasticsearch.document_id` exists and is not an empty string in the log record attributes, it will be used as the document ID. Otherwise, the document ID will be generated by Elasticsearch. The attribute `elasticsearch.document_id` is removed from the final document.
- `enabled`(default=false): Enable/Disable dynamic ID for log records. If `elasticsearch.document_id` exists and is not an empty string in the log record attributes, it will be used as the document ID. Otherwise, the document ID will be generated by Elasticsearch. The attribute `elasticsearch.document_id` is removed from the final document. See [Setting a document id dynamically](#setting-a-document-id-dynamically).

### Elasticsearch document mapping

Expand Down Expand Up @@ -349,6 +349,26 @@ Otherwise, it is mapped to an empty string ("").

In case the record contains `timestamp`, this value is used. Otherwise, the `observed timestamp` is used.

## Setting a document id dynamically

The `logs_dynamic_id` setting allows users to set the document ID dynamically based on a log record attribute.
Besides the ability to control the document ID, this setting also works as a deduplication mechanism, as Elasticsearch will refuse to index a document with the same ID.

The log record attribute `elasticsearch.document_id` can be set explicitly by a processor based on the log record.

As an example, the `transform` processor can create this attribute dynamically:

```yaml
processors:
transform/es-doc-id:
error_mode: ignore
log_statements:
- context: log
condition: attributes["event_name"] != null && attributes["event_creation_time"] != null
statements:
- set(attributes["elasticsearch.document_id"], Concat(["log", attributes["event_name"], attributes["event_creation_time"], "-"))
```

## Known issues

### version_conflict_engine_exception
Expand Down

0 comments on commit 32cbf2a

Please sign in to comment.