Handle container creation when cgroups have already been mounted in another location

[craigfurman]

As described in #1367, if a non-cpuset subsystem has been been mounted earlier in the mount table at a disjoint location to the rest of the hierarchy (e.g. /sys/fs/cgroup/{cpuset,cpu,memory...}) then runc create will fail with:

container_linux.go:259: starting container process caused "process_linux.go:283: applying cgroup configuration for process caused \"open /sys/fs/cpuset.cpus: no such file or directory\""

To replicate this behaviour, using Ubuntu 14.04 with a 4.4 kernel (linux-generic-lts-xenial package), create a mount table that looks like this:

cgroup /cgroup cgroup rw,relatime,cpu 0 0
none /sys/fs/cgroup tmpfs rw,relatime 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/cpu cgroup rw,relatime,cpu 0 0
cgroup /sys/fs/cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,relatime,blkio 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,relatime,memory 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,relatime,devices 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,relatime,freezer 0 0
cgroup /sys/fs/cgroup/net_cls cgroup rw,relatime,net_cls 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/net_prio cgroup rw,relatime,net_prio 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,relatime,hugetlb 0 0
cgroup /sys/fs/cgroup/pids cgroup rw,relatime,pids 0 0

Note that /cgroup can't have subsystem cpuset, otherwise everything will be fine.

On runc master:

→ runc create test
container_linux.go:259: starting container process caused "process_linux.go:283: applying cgroup configuration for process caused \"open /sys/fs/cpuset.cpus: no such file or directory\""

Using this PR, creation completes successfully. It succeeds because we no longer use the "cgroup root" (the dir that is assumed to contain all cgroup subsystem mountpoints) to work out the highest cpuset subsystem dir. Instead we look for the closest mountpoint ancestor of the cgroup dir we are operating on.

The following shell functions may come in handy for setting up a mount table that looks like this:

function cmount() { mount -t cgroup -o cpu cgroup /cgroup && mount -t tmpfs none /sys/fs/cgroup && cgroups-mount; }

function cumount { cgroups-umount && umount /sys/fs/cgroup && umount /cgroup; }

We weren't sure how to integration test this, as providing a mount table like above seems to break the integration tests in docker, precisely due to this bug but if anyone has any ideas we are happy to update.

We think it would be ideal if runc took a "cgroup root" as configuration (it could be optional), but didn't want to change too much in this PR.

Cheers,
@williammartin and Craig

[rh-atomic-bot]

257/265 passed on RHEL - Failed.
254/263 passed on CentOS - Failed.
263/264 passed on Fedora - Failed.
Log - https://aos-ci.s3.amazonaws.com/opencontainers/runc/runc-integration-tests-prs/294/fullresults.xml

[williammartin]

That failed Travis test looks to be a flake since I've seen on other Travis builds. I tried the !rebuild runc#1372 but didn't look like I was allowed.

Can someone kick the tests off again?

[cyphar]

Wait, is the only cgroup that we have problems with cpuset? That doesn't sound right.

[craigfurman]

@cyphar it's because cpuset subsystem impl gets the cgroup root again

[craigfurman]

Hi @cyphar , I know you're busy, but did you get a chance to check out the explanation above?

[cyphar]

LGTM. Sorry for the delay.

[williammartin]

No problem. Hmm, need one more LGTM on this, any idea who would be good to review?

[crosbymichael]

LGTM