Filtering by cgroups doesn't work

[aborniak]

I want to run execsnoop with --cgroupmap option for tracing new processes inside target container.

Environment:

[root@fc tools]# bpftrace -V
bpftrace v0.10.0-172-g127c
[root@fc tools]# bpftool -V
bpftool v5.6.7
[root@fc tools]#

[root@fc tools]# uname -a
Linux fc.my-clouds.net 5.6.13-200.fc31.x86_64 #1 SMP Thu May 14 23:26:14 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
[root@fc tools]# cat /etc/redhat-release
Fedora release 31 (Thirty One)
[root@fc tools]#

Target container:

[root@fc tools]# podman inspect -f='{{.State.Pid}}' f28b90006a73
1058296
[root@fc tools]# podman ps
CONTAINER ID  IMAGE                                       COMMAND         CREATED       STATUS           PORTS  NAMES
edab363096a1  docker.io/library/redis:latest              redis-server    25 hours ago  Up 25 hours ago         redis-server
ed39f00bbb63  docker.io/library/percona:latest            mysqld          25 hours ago  Up 25 hours ago         percona-server
f28b90006a73  localhost/local/6_2_0-188rubysource:latest  /usr/sbin/init  25 hours ago  Up 25 hours ago         admiring_heisenberg

[root@fc tools]# ps aux | grep [1]058296
root     1058296  0.0  0.0  93804 10148 ?        Ss   Jun15   0:00 /usr/sbin/init
[root@fc tools]#

Create MAP and find out namespace ID HEX for container f28b90006a73:

[root@fc tools]# stat -Lc '%i' /proc/1058296/ns/mnt
4026532792
[root@fc tools]#
[root@fc containers]# NS_ID_HEX="$(printf '%016x' $(stat -Lc '%i' /proc/1058296/ns/mnt) | sed 's/.\{2\}/&\n/g' | tac)"
[root@fc containers]# echo $NS_ID_HEX
b8 03 00 f0 00 00 00 00
[root@fc containers]# bpftool map update pinned /sys/fs/bpf/test5 key hex $NS_ID_HEX value hex 00 00 00 00 any
[root@fc containers]#

Test:

[root@fc ~]# podman exec -it f28b90006a73 bash
[root@onapp /]# w
 14:05:17 up 13 days, 23:37,  0 users,  load average: 0.05, 0.08, 0.02
USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
[root@onapp /]# w
 14:05:17 up 13 days, 23:37,  0 users,  load average: 0.05, 0.08, 0.02
USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
[root@onapp /]# w
 14:05:19 up 13 days, 23:37,  0 users,  load average: 0.04, 0.07, 0.02
USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
[root@onapp /]#

Result: execsnoop doesn't show any activity

[root@fc tools]# ./execsnoop --cgroupmap /sys/fs/bpf/test5
PCOMM            PID    PPID   RET ARGS

Expected result: see some activity inside container f28b90006a73

@alban @mauriciovasquezbernal Could you advise , might be I did something wrong or misunderstood https://github.com/iovisor/bcc/blob/master/docs/special_filtering.md .

[alban]

@aborniak You can either filter by cgroup with --cgroupmap and using the cgroup id, or you can filter by mount namespace with --mntnsmap and using the inode number of the mnt namespace. But in your report, you do a mix of both. I suggest you replace your last command by:

./execsnoop --mntnsmap /sys/fs/bpf/test5

[aborniak]

@alban It looks like my "execsnoop" version is a little bit outdated. (Installed from default fedora repository. )

It doesn't have "mntnsmap" option.

[root@fc tools]# ./execsnoop --help  | grep mntnsmap
[root@fc tools]#

[root@fc tools]# rpm -qa | grep bcc
bcc-tools-0.14.0-1.fc31.x86_64
python3-bcc-0.14.0-1.fc31.noarch
pcp-pmda-bcc-5.1.1-1.fc31.x86_64
bcc-devel-0.14.0-1.fc31.x86_64
bcc-0.14.0-1.fc31.x86_64
[root@fc tools]#

How I can find out "cgroup id" for using "--cgroupmap" option ?

I have tried to get cgroup id in the following way, but I guess it is also wrong , because it also didn't work.

[root@fc containers]# CGROUPID=$(stat /proc/1058296/ns/pid -c "%N" | cut -d[ -f2 | cut -d] -f1)
[root@fc containers]# echo $CGROUPID
4026532793
[root@fc containers]# CGROUPID_HEX=$(printf %016x $CGROUPID | sed -e 's/\(..\)/0x\1\n/g' | tac)
[root@fc containers]# echo $CGROUPID_HEX
0xb9 0x03 0x00 0xf0 0x00 0x00 0x00 0x00

[root@fc containers]# bpftool map create /sys/fs/bpf/test2 type hash key 8 value 4 entries 128 name test1 flags 0
[root@fc containers]# bpftool map update pinned /sys/fs/bpf/test2 key hex $CGROUPID_HEX value hex 00 00 00 00 any

So ,I used "/proc/1058296/ns/pid" instead "/proc/1058296/ns/mnt"

[mauriciovasquezbernal]

Hi @aborniak, you can get the cgroup id manually by:

$ cat /proc/$PID_OF_CONTAINER/cgroup

# pick the path in the line starting with 0::
# If there is no any line starting with 0:: it means podman is not using cgroup v2, 
# you'll have to use the mountns option.

# The prefix could be /sys/fs/cgroup/unified/ or /sys/fs/cgroup/ depending on the configuration of your system.
$ docker run --rm --privileged -v /sys/fs/cgroup:/sys/fs/cgroup docker.io/kinvolk/cgroupid:latest cgroupid hex /sys/fs/cgroup/unified/$PATH_FROM_ABOVE

# update map using bpftool ...

Please notice that there is an issue in runc and it's possible that the command that you execute with podman exec is in the wrong cgroup v2 path: opencontainers/runc#2087.

Let me know if you find further issues.

[aborniak]

@mauriciovasquezbernal Thank you for explanation.

I am also affected with opencontainers/runc#2087 , so I used "podman run -it localhost/local/6_2_0-188rubysource /bin/bash" to confirm that filter works.

Have a nice day!