Plugin drops all logs but first when placed directly after an in_forward input

[osela]

I'm using this plugin in a fluentd forwarder/aggregator structure, where a fluentd forwarder runs as a DaemonSet on all nodes, forwarding all logs to fluentd aggregator pods running in a deployment. All filtering (including the kubernetes-metadata filter) is done on the aggregator.

When placing this plugin directly after the in_forward input plugin, only the first event in each event stream ("batch") is processed and continues to the next filter in the chain. All other logs get thrown away.

After further investigation, it seems that the call to first on this line causes problems when called on an instance of MessagePackEventStream, like the one returned from the in_forward plugin. When first is called before each on this instance, each returns only the first item.

Opened fluent/fluentd#2099 in fluentd regarding the weird behaviour of MessagePackEventStream.

[richm]

In the meantime, you might be able to workaround the problem by using use_journal true and adding a filter to create CONTAINER_NAME and CONTAINER_ID_FULL fields - see https://github.com/openshift/origin-aggregated-logging/blob/master/fluentd/configs.d/input-post-forward-mux.conf#L30

[osela]

Thanks @richm. As a temporary workaround I actually added a custom filter plugin that does nothing before the kubernetes-metadata filter.