API Gateway Layer 7 logs to ECS schema

[Randy-312]

We currently have filebeat sending in Layer7 logs to our ELK stack.
However, we're no longer 'parsing' it out, due to some directional changes to our pre-processing layer, and everything is simply unparsed.

We're going to align to ECS Schema, but need some guidance on Layer7, and if Anyone ELSE is working through this, they may wish to contribute as well.

We can start with Traffic Logging, which is what we have coming in today, and I have solid regex for as well.

Here are my notes so far..

CA has documented some work for what they would do to send to elk.

Here are their fields

And CA's instructions on setup

[Randy-312]

If someone else is able to take up this work, let us know. We won't be able to take this up anytime soon.

[Randy-312]

There may be an easier way to accomplish this now.
With the 9.4 CR1 release (documented in 10.0), Layer7 is able to output log in JSON format.

I would think about using filebeat to create new fields (ECS compliant) where the ones they provide aren't quite aligned to ECS. Alternatively, renaming them to match our own format where we use record.

[webmat]

We created meta-issue #938 to add proxy support in ECS. Closing in favor of the meta issue. Stay tuned :-)