Unable to troubleshoot mod_security blocked requests

[mattops]

As discussed in this thread https://mojdt.slack.com/archives/C57UPMZLY/p1611057689075100

It seems there was a bug introduced into the the newer version of mod_security which was rolled out with nginx-ingress updates.

helm chart versions:
2.13.0 -> 3.6.0

Includes:
nginx-ingress
0.35.0 -> 0.40.2

Currently it is not possible to troubleshoot why a particular request was blocked. This is potentially a big problem for teams who want to use mod_security. There are hundred of rules and almost limitless ways in which to trigger them.

There is a service we would like to enable mod_security on, however it is unlikely we can practically do it without being able to diagnose any potential blocked requests.

Raising this ticket for visibility in the CP team - and maybe a work around can be found.

[razvan-moj]

waiting for upstream release 1.0.2; release 1.0.0 seems too buggy to revert to
https://github.com/SpiderLabs/ModSecurity-nginx/blob/master/CHANGES#L12

[AntonyBishop]

Still waiting in release 1.0.2

[petergphillips]

Would it be possible to investigate workarounds for this? There haven't been any commits to the repo in the last month or so, nor is there any indication that the issues have been fixed, let alone that a release is imminent. That release note comment was added to the repository 14 months ago - https://github.com/SpiderLabs/ModSecurity-nginx/blame/master/CHANGES#L12.

We are left in the situation at present where, for security reasons, we need modsecurity, but it isn't fit for purpose in that we can't add it to any new projects because we can't then tailor the allowlist etc. to the application.

[AntonyBishop]

Hi @petergphillips will raise a discussion with the team to see what the alternatives might be.

[AntonyBishop]

#2787

[mattops]

Just to reiterate what @petergphillips said - we really need a work around for this. Essentially, we need to somehow be able to see what has been blocked and by which rule. Thanks.

[sldblog]

Our (interventions) current approach is to go live with our service on the public internet with the WAF in front of the service. +1 for us, it'd be good if we could investigate potential reports.

For context, we expect some of our users to be from large external organisations, so any visibility on who's getting blocked and why would be great. (It will be a commercial relationship, so will be sensitive if they cannot work.)

[razvan-moj]

kubernetes/ingress-nginx#7230

[razvan-moj]

latest nginx release pulls a modsec release from Sep 2020, and does not include the fix we need: owasp-modsecurity/ModSecurity-nginx@e028ca4
will keep checking

[vijay-veeranki]

Default ingress controller and modsec ingress controller is upgraded to the latest helm chart version "3.34.0"

@mattops can you please check if you can troubleshoot any better now.

[razvan-moj]

^ the above upgrade means we are now at this ModSec commit: https://github.com/kubernetes/ingress-nginx/search?q=22e53aba4e3ae8c7d59a3672d6727e49246afe96

dated 17 Sep 2020

[petergphillips]

Can't see that the release has made any difference to the logging. Searching kibana for modsecurity shows that the request is blocked, but provides no information as to why it is blocked.
I've also tried upping the debug level in the ingress i.e.

                                                                  SecDebugLog /var/log/nginx/error.log
                                                                  SecDebugLogLevel 3

but that hasn't made any difference. Setting it to 4 gives lots and lots of information, but again doesn't provide the crucial detail as to a request gets blocked.

More information:
First search for modsecurity blocked requests in kibana. This then gives a unique id e.g. 162504239077.853828. Searching on that unique id should then give the reason why the request was blocked but doesn't show any more information.

[razvan-moj]

@petergphillips from https://github.com/fastly/waf_testbed/blob/master/templates/default/REQUEST-949-BLOCKING-EVALUATION.conf.erb#L44 the ModSec ID you get (949110) means several rules were hit before by this request and their total score is >5
A first hint would be the file:/// parameter, but more details should be available at level 4

[petergphillips]

The file:/// was just an easy way for me to trigger the rule. From the slack conversation we used to just be able to search for modsecurity and warning and that would show us the reason - unfortunately the kibana data from 2020 has now disappeared from the logs.
Setting the SecDebugLogLevel to 3 or even 1 should still give us the reason though.
You can see I did try setting it to 4 earlier. That added 1,121 lines to the logging, but I can't see any information in those log lines that shows why the request was blocked.

[razvan-moj]

@petergphillips got it. We'll keep an eye on https://github.com/kubernetes/ingress-nginx/releases and test as soon as a fix is available.

[razvan-moj]

Still wqiting for owasp-modsecurity/ModSecurity-nginx#170

[vijay-veeranki]

Once the user moved to 1.0 module.
User can use the new mod-sec class to see the logs

[AntonyBishop]

Solution addressed in new ingress controllers. Teams are switching to new controllers so issue can be closed.