cgroup-path

[ansd]

TLDR:

1. prepend bpm to the cgroup path
2. do not base32 encode the container id

Details:

In a previous story, the decision was to base32 encode the job name for the runc container-id which becomes the cgroup-path in the file /proc/[pid]/cgroup:

hierarchy-ID:controller-list:cgroup-path

I would like to write a sysdig-bosh release which will be colocated on VMs and which should ideally also be able to monitor bpm containers. Sysdig parses the cgroup-path in order to classify the container type (see here for LXC).
For example, the cgroup-path for an LXC container looks like /lxc/my-container:

cat /proc/<pid of lxc container>/cgroup
11:hugetlb:/lxc/my-container

It's the default for LXC (see here):

lxc.cgroup.pattern
Format string used to generate the cgroup path (e.g. lxc/%n).

According to the sysdig code, the same applies to mesos, libvirt-lxc and docker.
Only the bpm container has this base32 encoded cgroup-path like

12:pids:/MJWG6YTTORXXEZI-

for the blobstore for example.

Having prepended something like bpm. to the cgroup-path would allow to parse it easily in sysdig such that the filter csysdig -pc container.type=bpm could match BPM containers.

But even though if the cgroup-path looks like bpm.MJWG6YTTORXXEZI-, the output for above filter would show a container column with the name MJWG6YTTORXXEZI- rather than blobstore.

Therefore, it would also be better to not base32 encode the whole container name, but encode maybe only those chars which do not match the regex ^[\w+-\.]+$ (see code). This would probably require to have a special delimiter char for the decoding process in order to identify the encoded chars.

See also Slack discussion for more context.

If the above two requirements are okay with you, I could also create a PR.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/161874197

The labels on this github issue will be updated when the story is started.

[xoebus]

Just merged a new encoding format for BPM job IDs. It should go out in the next release. Let us know if it's missing anything.

Thanks for the issue!

[ansd]

@xoebus many thanks for changing the container-id.

As a side note:
After testing your change I realise that the cgroup-path is not just the container-id anymore:

bosh/0:~# cat /proc/<blobstore pid>/cgroup
12:devices:/system.slice/runc-bpm-blobstore.scope

So, the cgroup-path is /system.slice/runc-bpm-blobstore.scope rather than just bpm-blobstore.
This was introduced by pass --systemd-cgroup when on a systemd machine.
/system.slice/runc- comes from here and .scope comes from here.

(The parent (system.slice), ScopePrefix (runc), and the appended scope (vs slice) are configurable.)

It makes parsing the cgroup-path in sysdig less straightforward.
However, I think that parsing the cgroup-path for bpm- on systemd and non-systemd machines still identifies the BPM container type. And the container-id (from sysdig perspective) is then the string between bpm- and either .scope or the end of the string.

[xoebus]

Changing the prefix will change behavior as it's changing the cgroup hierarchy. We're going to probably need to change that soon as we start doing CPU share limiting which is very dependent on the hierarchy. I don't know if we can make any guarantees about what that path will be.

[ansd]

@xoebus hmm I'm not that deep into how CPU share limiting works and how the cgroup hierachy gets affected, and I don't know whether the following suggestion makes sense but wouldn't it be possible that the cgroup-path stays somewhat stable and BPM identifiable by introducing a bpm root cgroup like Mesos is doing it for example (see https://mesos.readthedocs.io/en/latest/configuration/):

--cgroups_root=VALUE	Name of the root cgroup. (default: mesos)

LXC seems to also have this root cgroup (see https://linuxcontainers.org/lxc/manpages/man5/lxc.system.conf.5.html):

lxc.cgroup.pattern
Format string used to generate the cgroup path (e.g. lxc/%n).

That way you could introduce any other hierarchy (including CPU share limiting) below this root cgroup.