Multiline Parser: Custom parser cannot replicate built-in CRI parser?

[pranavmarla]

Bug Report

Describe the bug
As per this multiline parsing doc, Fluent Bit now comes with some built-in multiline parsers and also gives us the ability to create our own custom multiline parsers.

Unfortunately, it looks like the custom parser functionality is not as powerful as the built-in parsers, which means that, for some of the built-in multiline parsers (eg. CRI), it is not currently possible to create a custom multiline parser that replicates/reproduces its functionality.

This means that, if we want to modify the built-in CRI multiline parser by creating our own custom CRI multiline parser, that is not currently possible.

If this is a known limitation, which I believe it is (see below for details), is there an ETA for when this will be fixed?

If I am mistaken, and it is possible right now, could you please provide the config for a custom multiline parser that does the same thing as the built-in CRI multiline parser?

Proof
1.
Reading through the documentation on regex states, needed to create a custom multiline parser, it appears that it can only catch multiline logs where the first line is formatted differently from the subsequent lines:
i.e.

A
B
B
B
...

But, as mentioned in #1316, CRI multiline logs follow the opposite format:

A
A
A
...
B

So, based off of my interpretation, it doesn't seem possible to create a custom regex multiline parser that can match CRI multiline logs.

2.
In the custom multiline parser doc, I see this explanation of one of the parameters:

type: Set the multiline mode, for now, we support the type regex.

Initially, this didn't convey anything to me, since there's no mention of how many other types there are, or what those other types are.
However, I later found some types mentioned here:

/* Types available */
#define FLB_ML_REGEX     1    /* pattern is a regular expression    */
#define FLB_ML_ENDSWITH  2    /* record key/content ends with 'abc' */
#define FLB_ML_EQ        3    /* record key/content equaks 'abc'    */

Assuming the documentation is referring to these types, then it implies that we can only create custom multiline parsers that replicate built-in parsers of the first type (regex) -- CRI by contrast, is the last type (equals).

Based on this, this appears to be a known limitation.

Your Environment

• Version used: 1.8.4

Additional context
As mentioned above, this is an issue primarily because it means we cannot make any modifications to the built-in CRI multi-line parsers (by building our own custom parsers).
By contrast, if we consider the regular (single-line) parsers, it is possible to make our own custom parser that can replicate any of the built-in parsers (including CRI).

[pranavmarla]

If this is a known limitation, which I believe it is (see below for details), is there an ETA for when this will be fixed?

FYI @edsiper: When you have time, would you mind providing an ETA for when the custom multiline parser functionality will be on par with the built-in parsers?

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.

[pmalek-sumo]

I believe this is not stale.

[gnom7]

@pranavmarla @edsiper any guidance please?