Delay watchdog checks while any other nsm_sensor_ps script runs #21
Conversation
…hecks This check pauses the --if-stale checks run by the watchdog cron job if it detects any other nsm_sensor_ps script running. This prevents race conditions such as when rule_update or daily_restart is starting/stopping processes, and watchdog interferes, leaving an extra process running.
|
Hi @petiepooo , Thanks for the pull request and detailed notes! I'll review as time allows. |
|
To followup, I'm currently running this patch on my systems. Every couple of days, always at 12:04:01 UTC, in /var/log/nsm/watchdog.log*, I now see this:
And once in a while, I believe when someone is manually restarting the services just when watchdog is triggere, I get this:
When that happens, I also get an email (since I have exim4-daemon-light installed) like:
I no longer get duplicate snort or pcap_agent processes like I used to, which tells me it's working as planned. I like the email as a failsafe to ensure the patch isn't completely breaking the watchdog. It would not be generated unless a mailer like exim4 is installed, so base installs would never even notice. If it's not needed, the echo call above the exit in the patch could be removed. Alternatively, the loop count could be increased to more than 30 (but less than 240 or so lest the watchdog calls start overlapping) if you feel it should wait longer before giving up. I'm really surprised this isn't a huge, oft-reported issue, as I've been experiencing it since I started using SecurityOnion three+ years ago. Yet, searching through https://groups.google.com/forum/#!forum/security-onion, I can't find anyone else talking about this. The biggest effect is when it happens to snort, and there are duplicate snort processes using the same config file and vying for access to the same alert file. That breaks things badly, as it starts dropping alerts. |
This check pauses the --if-stale checks run by the watchdog cron job if it
detects any other nsm_sensor_ps script running. This prevents race conditions
such as when rule_update or daily_restart is starting/stopping processes, and
watchdog interferes, leaving an extra process running.
History:
I would often find an extra pcap_agent (and occasionally a snort alert process) running after the automated daily restarts and/or rule-update processes run. The most recent showed these symptoms:
(relevant portion of watchdog.log)
Note that the timestamp in the logfile matches the time that the duplicate processes were created. This is also four minutes after the sensor-newday cron job runs, but also right when the nsm-watchdog cron job runs.
Solution:
In the lib-nsm-common-utils library in the function that handles restarts of stale processes (when run with the --if-stale flag by the watchdog cron job), check to see if any other of the nsm_sensor_ps-* scripts are running, and if so, pause, checking every second up to 30 times. If other scripts end, continue with the --if-stale checks. If they're still running after 30 checks, abort.
Note that it reports the error using echo_msg_end, but it also echos to stderr, which generates an email from cron to root if you have a mailer configured on the system. This is to alert on timeouts rather than just hide in the watchdog.log file. At most, it will generate an email every 5 minutes, so there should not be an uncontrollable flood, and emails only ever go out if a mailer such as exim4 is installed and configured on the sensor.
Testing:
The patch author's recommended testing method: