ndjson parser doesn't expand keys if target is set

[mjaepel]

Hi,

it seems that there is the same issue with the ndjson parser like in the decode_json_fields processor some time ago: #32010

dotted fields aren't expanded if the target option in ndjson parser is set. All works fine if the target option is missing.
Seems to be this code position: https://github.com/elastic/beats/blob/main/libbeat/reader/readjson/json.go#L187

Example Filebeat config:

- type: filestream
  id: my_dummy_log
  paths:
    - /tmp/my_dummy_log
  parsers:
    - ndjson:
        target: 'dummy'
        expand_keys: true

Example data:

{ "my.test": 1 }

Expected event:

{ "dummy":  { "my": { "test": 1 } } }

But got event:

{ "dummy":  { "my.test": 1 } }

Tried with

- type: filestream
  id: my_dummy_log
  paths:
    - /tmp/my_dummy_log
  parsers:
    - ndjson:
        expand_keys: true

Then I get:

{ "my": { "test": 1 } }

It would be nice to get the old processor bug fix also for the ndjson parser.

Regards,
Marcel

For confirmed bugs, please report:

• Version: tested on 8.6.2, but code on github assumes that also current releases are affected
• Operating System: Ubuntu 20.04
• Discuss Forum URL: https://discuss.elastic.co/t/ndjson-parser-doesnt-expand-keys-if-target-is-set/334799

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[belimawr]

I can confirm this is still an issue in our latest release, v8.15.2 and main branch.

[belimawr]

The current workaround is to use the decode_json_fields processor, ex config:

- type: filestream
  id: my-filestream-id
  enabled: true
  paths:
    - /tmp/ex.ndjson
  processors:
    - decode_json_fields:
        fields:
          - message
        target: dummy
        expand_keys: true