Compatibility with ModSecurity version 2.9.1

[zimmerle]

Hi Guys,

I am following the project for a while, congrats!

Yesterday we have released ModSecurity version 2.9.1. There was a modification in the logs that might affect you. The issue owasp-modsecurity/ModSecurity#840 contains more details about it.

Also, in v2.9.1 there is this possibility to save the audit logs in JSON format, not sure if the format is structured in the shape that you need for elasticsearch, but may be useful.

[bitsofinfo]

Does this affect the audit logs?

[bitsofinfo]

I just don't have the time to look at this now @zimmerie, thanks for bringing it to the projects attention though! Perhaps yourself or some prior contributors @MWilkinson @breml could take a look?

[breml]

I quickly scrolled through owasp-modsecurity/ModSecurity#840 and as far as I understand, this affects mainly error.log and not audit.log format. @zimmerle can you confirm?

[zimmerle]

Hi @breml,

The only change that seems relevant to you is the fact that now we have this new line:

Apache-Error: [file "apache2_util.c"] [line 271] [level 3] [client %s] ModSecurity: %s%s [uri "%s"]%s

This happens to appear on the "section H" of the audit logs.

[Alex-Kw]

I believe I'm seeing this issue. I'm noticing this issue with section H parsing above 2.9.0. Field names like auditLogTrailer.Action are normal but there is a field name called "
auditLogTrailer. [line 271] [level 3] [client $IPADDR] ModSecurity" with data starting as the log continues, "Access denied with code ....."

This results in a bunch of new unique field names, which logstash/elastic didn't like. for now I've edited my elastic template to allow more field names as I look at the regex, but I figure you would be able to fix this much easier than I could. If I figure it out I will update.

Edit, it appears this is happening due to the "ModSecurity:" section of the Apache-Error line having a colon. The section H is split on value_split => ":" so that section of the audit log trailer seems to create the unwanted behavior with the parsing thereof.

[bitsofinfo]

Please submit a PR if you have a patch

[Alex-Kw]

If I figure out a good way to handle it, I will. Thank you.

[Alex-Kw]

I've submitted a pull request that works well enough for me. I will not be offended if you do not like it :) I opted to go with dropping that line from the raw section H, prior to splitting that section. I did this because the valuable data in that one line, as far as I could tell, was already extracted otherwise.

I haven't been able to find a better way to contact you, but wanted to thank you for this project. I use it with mlogc and logstash http input plugin (behind nginx https) to ship logs to an elk stack from several remote sensors. It's been an awesome replacement to auditconsole so far. Cheers.

[bitsofinfo]

Should be addressed by 1.2.2