Security harden casues service failure

[balliet75]

Description:

In the newer 1.9.4 version of the package, the additional parameters added to the service file cause the Service to no longer start if the sysctl parameter "user.max_user_namespaces" is set to "user.max_user_namespaces = 0".

Problem:

Upgrade RHEL 9.4 irqbalance from version 1.9.2 to 1.9.4 causes the Service to fail with the following error if the parameter user.max_user_namespaces = 0 is set:

× irqbalance.service - irqbalance daemon
     Loaded: loaded (/usr/lib/systemd/system/irqbalance. Service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Thu 2024-11-14 22:40:01 UTC; 16s ago
   Duration: 20ms
       Docs: man:irqbalance(1)
             https://github.com/Irqbalance/irqbalance
    Process: 83726 ExecStart=/usr/sbin/irqbalance $IRQBALANCE_ARGS (code=exited, status=217/USER)
   Main PID: 83726 (code=exited, status=217/USER)
        CPU: 18ms

Nov 14 22:40:01 dev10 systemd[1]: Started irqbalance daemon.
Nov 14 22:40:01 dev10 systemd[83726]: irqbalance.service: Failed to set up user namespacing: No space left on device
Nov 14 22:40:01 dev10 systemd[83726]: irqbalance.service: Failed at step USER spawning /usr/bin/irqbalance: No space left on device
Nov 14 22:40:01 dev10 systemd[1]: irqbalance.service: Main process exited, code=exited, status=217/USER
Nov 14 22:40:01 dev10 systemd[1]: irqbalance.service: Failed with result 'exit-code'.

Cause:

Change of service file from

[Unit]
Description=irqbalance daemon
Documentation=man:irqbalance(1)
Documentation=https://github.com/Irqbalance/irqbalance
ConditionVirtualization=!container
ConditionCPUs=>1

[Service]
EnvironmentFile=-/etc/sysconfig/irqbalance
ExecStart=/usr/sbin/irqbalance --foreground $IRQBALANCE_ARGS
ReadOnlyPaths=/
ReadWritePaths=/proc/irq
RestrictAddressFamilies=AF_UNIX AF_NETLINK
RuntimeDirectory=irqbalance/

[Install]
WantedBy=multi-user.target

To the following in 1.9.4

[Unit]
Description=irqbalance daemon
Documentation=man:irqbalance(1)
Documentation=https://github.com/Irqbalance/irqbalance
ConditionVirtualization=!container
ConditionCPUs=>1

[Service]
EnvironmentFile=-/etc/sysconfig/irqbalance
ExecStart=/usr/sbin/irqbalance $IRQBALANCE_ARGS
CapabilityBoundingSet=
NoNewPrivileges=yes
ProtectSystem=strict
ReadOnlyPaths=/
ReadWritePaths=/proc/irq
RestrictAddressFamilies=AF_UNIX AF_NETLINK
RuntimeDirectory=irqbalance/
IPAddressDeny=any
ProtectHome=true
PrivateTmp=yes
PrivateNetwork=yes
PrivateUsers=true
ProtectHostname=yes
ProtectClock=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes
RestrictNamespaces=yes
LockPersonality=yes
MemoryDenyWriteExecute=yes
RestrictRealtime=yes
RestrictSUIDSGID=yes
RemoveIPC=yes
PrivateMounts=yes
SystemCallFilter=@cpu-emulation @privileged @system-service
SystemCallFilter=~@clock @module @mount @obsolete @raw-io @reboot @resources @swap
SystemCallErrorNumber=EPERM
SystemCallArchitectures=native

[Install]
WantedBy=multi-user.target

Resolve:

Remove the security settings in sysctl.conf for "user.max_user_namespaces = 0" and allow default "user.max_user_namespaces = 30578"

or

Modify /etc/systemd/system/multi-user.target.wants/irqbalance. The service is false instead of valid for the PrivateUsers setting.

PrivateUsers=false

Then reload systemd daemon and restart the irqbalance.service

[nhorman]

Thats....working as designed. The unit file expects to be able create a user namespace, and your systems is set to by default (or through admin intervention) disallow user namespace (the user.max_user_namespaces = 0 parameter)

I'm not sure why a system would default to that value, but its a security precaution to run irqbalance in its own user namespace to prevent privilege escalation.

so you need to make a choice, either disallow PrivateUsers (as you have done), or alter your system settings to set user.max_user_namespaces to a non-zero value.

What we can't do is just change the setting for everyone, because different admins may make different choices.

[tannermsmith1]

To answer "why a system would default to that value": Security hardening practices for those organizations that follow US DoD standards.

RHEL 9 must disable the use of user namespaces.

[nhorman]

My point above still remains. Some users may not need to conform to DoD standards, and they have a choice for what to do here, they can disallow PrivateUsers in the service file

[tannermsmith1]

I agree! Just pointing out where the issue might lie for some folks as they upgrade from RHEL 9.4 to 9.5. They packaged the latest irqbalance release in that upgrade.

[balliet75]

While the code works as designed, I disagree with the idea that changing the defaults set in the service file is not a problem. The change, implemented in version 1.9.4, breaks users' usage models. The service functioned with this setting in place prior to the change. It now requires action from administrators, which was not required previously when an essential action like sudo dnf update is run.

The security practice has been in use for years. As pointed out by the article that Tanner linked, it is a practice in security hardening for government and telco companies worldwide. Essentially, you take these users from working to non-working every time they upgrade the irqbalance.

It's possible to work around the issue, but finding a resolution to a failure caused by a simple update isn't enjoyable. Resolving this took a day of research, diagnostics, and testing, which delayed a patch release that was not related to this code but affected OS functionality. Thus, I shared this on Git Hub with others who will experience the same failure.

[nhorman]

Heres my take:

1. Red Hat conforms to a DoD specification (The referenced text appears to be common criteria related I think)
2. Red Hat builds and ships irqbalance as a rpm package, meaning they take the upstream source, modify it however they see fit, and ship it to end users
3. Irqbalance (this upstream project) does not undertake any certification for common criteria or any other DoD process
4. Irqbalance added those systemd settings in the provided service unit file in response to Additional systemd options #294, in which we decided to undertake various hardening settings to further sandbox irqbalance, independently and without consideration for common criteria requirements (which again, the upstream project doesn't pay attention to, because no one working independently on an upstream project goes through common criteria certification without a large corporation behind them)

So from my read, Red Hat did two things here that caused this problem:
a) They (correctly) followed common criteria requirements and decided to disable user namespaces
b) They built and shipped irqbalance without making a corresponding modification to the provided systemd unit file for irqbalance to disable PrivateUsers (I'm basing this on the observation that the Fedora rpm for irqbalance from which RHEL9 was forked just blindly copies the service file from the misc directory to the built rpm)

Thats on Red Hat to fix, not upstream. What you're asking for here is for the upstream project to conform to whatever Red Hats specific use case is. The second I do that, someone else is going to come along and and ask "hey, I'm using irqbalance in my little embedded project and you reduced my security by not running irqbalance in a private user namespace, fix that please"

The bottom line is that these defaults aren't going to address everyones needs, and yes, sometimes they change, requiring downstream adjustments.

If you are using irqbalance built by Red Hat in a common criteria certified environment, then the "right" solution isn't to ask the upstream project to conform to do what RHEL needs, its to open a bug with Red Hat and say "hey" your irqbalance unit file isn't written in line with those criteria, for exactly the reasons we've discussed above. That way you get the behavior you expect in the environment you're running in without impacting other users who may be relying on it.

[balliet75]

Those are excellent points. It does need to be logged as a Redhat bug, and then they can decide to fix it for the customers that this affects, like ourselves and our customers.

Thank you very much for the detailed response!

[tannermsmith1]

Thanks for the additional insight, @nhorman. Our approach will follow the same as @balliet75 to submit a ticket to RH on the issue. If it generates a Bugzilla report, happy to post that back up here as you can anticipate more people starting to search for this issue as they upgrade (if it doesn't get resolved).

[nhorman]

Please, if you post a link to it here, I'll follow along and respond if they have any additional feedback

[balliet75]

I logged a case with RedHat and was informed of the following open JIRA (Bugzilla) bug ticket https://issues.redhat.com/browse/RHEL-40120