Unable to launch on ubuntu 22.04 with cgroupv2 #7671

rlex · 2022-06-01T15:24:41Z

Description

Recently moved to another provider and upon creation of gvisor-backed pod i see following error:

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: failed to create shim: OCI runtime create failed: creating container: write /sys/fs/cgroup/kubepods/besteffort/pod66933fac-4dba-4c07-a80a-f3f01253cb7c/cgroup.procs: device or resource busy: unknown

Configs are same - deployed from same ansible role.
Major difference is ubuntu 22.04 lts vs 20.04 (was working on 20.04), maybe kernel issue?
20.04 was running with

Linux master-15.11.0-43-generic #47~20.04.2-Ubuntu SMP Mon Dec 13 11:06:56 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Steps to reproduce

k3s 1.23.6, standard runtime class from docs.
containerd://1.5.11-k3s2
Test manifest:

apiVersion: v1
kind: Pod
metadata:
  name: gvisor
spec:
  runtimeClassName: gvisor
  containers:
    - name: nginx
      image: nginx

runsc version

runsc version release-20220510.0
spec: 1.0.2-dev

docker version (if using docker)

No response

uname

Linux master-2 5.15.0-33-generic #34-Ubuntu SMP Wed May 18 13:34:26 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

kubectl (if using Kubernetes)

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.5", GitCommit:"5c99e2ac2ff9a3c549d9ca665e7bc05a3e18f07e", GitTreeState:"clean", BuildDate:"2021-12-16T08:38:33Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"darwin/arm64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.6+k3s1", GitCommit:"418c3fa858b69b12b9cefbcff0526f666a6236b9", GitTreeState:"clean", BuildDate:"2022-04-28T22:16:18Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}



### repo state (if built from source)

_No response_

### runsc debug logs (if available)

_No response_

The text was updated successfully, but these errors were encountered:

fvoznika · 2022-06-03T18:43:27Z

/cc @manninglucas

I've seen this error before when runsc didn't support systemd cgroupsv2. But support has been added on 2022-03-17, and you are on a newer release (20220510.0). Ubuntu 22.04 uses cgroupsv2, but not sure if it changed from 20.04. What cgroup driver are you using in K8s? Please enable debug logging and attach the logs to this issue.

rlex · 2022-06-28T23:50:18Z

sorry for long reply...
Create:

root@node-1:/tmp# cat runsc-7e2e24d677eb222bf1dc964589055e2a4943b36bd8d88b8f3d30798c32e649e7-gvisor.create.log
I0629 02:48:16.498632  793356 main.go:214] ***************************
I0629 02:48:16.498710  793356 main.go:215] Args: [runsc --root=/run/containerd/runsc/k8s.io --log=/run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/7e2e24d677eb222bf1dc964589055e2a4943b36bd8d88b8f3d30798c32e649e7/log.json --log-format=json --debug=true --debug-log=/tmp/runsc-7e2e24d677eb222bf1dc964589055e2a4943b36bd8d88b8f3d30798c32e649e7-gvisor.%COMMAND%.log create --bundle /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/7e2e24d677eb222bf1dc964589055e2a4943b36bd8d88b8f3d30798c32e649e7 --pid-file /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/7e2e24d677eb222bf1dc964589055e2a4943b36bd8d88b8f3d30798c32e649e7/init.pid --user-log /var/log/pods/default_gvisor_a61c0ca3-d7de-4e49-84ee-081d1d3b793e/gvisor.log 7e2e24d677eb222bf1dc964589055e2a4943b36bd8d88b8f3d30798c32e649e7]
I0629 02:48:16.498753  793356 main.go:216] Version release-20220606.0
I0629 02:48:16.498765  793356 main.go:217] GOOS: linux
I0629 02:48:16.498777  793356 main.go:218] GOARCH: amd64
I0629 02:48:16.498790  793356 main.go:219] PID: 793356
I0629 02:48:16.498803  793356 main.go:220] UID: 0, GID: 0
I0629 02:48:16.498815  793356 main.go:221] Configuration:
I0629 02:48:16.498826  793356 main.go:222] 		RootDir: /run/containerd/runsc/k8s.io
I0629 02:48:16.498838  793356 main.go:223] 		Platform: ptrace
I0629 02:48:16.498850  793356 main.go:224] 		FileAccess: exclusive, overlay: false
I0629 02:48:16.498865  793356 main.go:225] 		Network: sandbox, logging: false
I0629 02:48:16.498879  793356 main.go:226] 		Strace: false, max size: 1024, syscalls:
I0629 02:48:16.498891  793356 main.go:227] 		LISAFS: false
I0629 02:48:16.498903  793356 main.go:228] 		Debug: true
I0629 02:48:16.498914  793356 main.go:229] 		Systemd: false
I0629 02:48:16.498926  793356 main.go:230] ***************************
D0629 02:48:16.500810  793356 specutils.go:75] Spec:
{
  "ociVersion": "1.0.2-dev",
  "process": {
    "user": {
      "uid": 0,
      "gid": 0
    },
    "args": [
      "/pause"
    ],
    "env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ],
    "cwd": "/",
    "noNewPrivileges": true,
    "oomScoreAdj": -998
  },
  "root": {
    "path": "/run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/7e2e24d677eb222bf1dc964589055e2a4943b36bd8d88b8f3d30798c32e649e7/rootfs",
    "readonly": true
  },
  "hostname": "gvisor",
  "mounts": [
    {
      "destination": "/proc",
      "type": "proc",
      "source": "/run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/7e2e24d677eb222bf1dc964589055e2a4943b36bd8d88b8f3d30798c32e649e7/proc",
      "options": [
        "nosuid",
        "noexec",
        "nodev"
      ]
    },
    {
      "destination": "/dev",
      "type": "tmpfs",
      "source": "/run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/7e2e24d677eb222bf1dc964589055e2a4943b36bd8d88b8f3d30798c32e649e7/tmpfs",
      "options": [
        "nosuid",
        "strictatime",
        "mode=755",
        "size=65536k"
      ]
    },
    {
      "destination": "/dev/pts",
      "type": "devpts",
      "source": "/run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/7e2e24d677eb222bf1dc964589055e2a4943b36bd8d88b8f3d30798c32e649e7/devpts",
      "options": [
        "nosuid",
        "noexec",
        "newinstance",
        "ptmxmode=0666",
        "mode=0620",
        "gid=5"
      ]
    },
    {
      "destination": "/dev/mqueue",
      "type": "mqueue",
      "source": "/run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/7e2e24d677eb222bf1dc964589055e2a4943b36bd8d88b8f3d30798c32e649e7/mqueue",
      "options": [
        "nosuid",
        "noexec",
        "nodev"
      ]
    },
    {
      "destination": "/sys",
      "type": "sysfs",
      "source": "/run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/7e2e24d677eb222bf1dc964589055e2a4943b36bd8d88b8f3d30798c32e649e7/sysfs",
      "options": [
        "nosuid",
        "noexec",
        "nodev",
        "ro"
      ]
    },
    {
      "destination": "/dev/shm",
      "type": "tmpfs",
      "source": "/run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/7e2e24d677eb222bf1dc964589055e2a4943b36bd8d88b8f3d30798c32e649e7/shm",
      "options": [
        "ro"
      ]
    },
    {
      "destination": "/etc/resolv.conf",
      "type": "bind",
      "source": "/var/lib/rancher/k3s/agent/containerd/io.containerd.grpc.v1.cri/sandboxes/7e2e24d677eb222bf1dc964589055e2a4943b36bd8d88b8f3d30798c32e649e7/resolv.conf",
      "options": [
        "rbind",
        "ro"
      ]
    }
  ],
  "annotations": {
    "dev.gvisor.spec.cgroup-parent": "/kubepods/besteffort/poda61c0ca3-d7de-4e49-84ee-081d1d3b793e",
    "dev.gvisor.spec.mount.gvisorinternaldevshm.options": "rw",
    "dev.gvisor.spec.mount.gvisorinternaldevshm.share": "pod",
    "dev.gvisor.spec.mount.gvisorinternaldevshm.source": "/run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/7e2e24d677eb222bf1dc964589055e2a4943b36bd8d88b8f3d30798c32e649e7/shm",
    "dev.gvisor.spec.mount.gvisorinternaldevshm.type": "tmpfs",
    "io.kubernetes.cri.container-type": "sandbox",
    "io.kubernetes.cri.sandbox-id": "7e2e24d677eb222bf1dc964589055e2a4943b36bd8d88b8f3d30798c32e649e7",
    "io.kubernetes.cri.sandbox-log-directory": "/var/log/pods/default_gvisor_a61c0ca3-d7de-4e49-84ee-081d1d3b793e",
    "io.kubernetes.cri.sandbox-name": "gvisor",
    "io.kubernetes.cri.sandbox-namespace": "default"
  },
  "linux": {
    "resources": {
      "cpu": {
        "shares": 2
      }
    },
    "cgroupsPath": "/kubepods/besteffort/poda61c0ca3-d7de-4e49-84ee-081d1d3b793e/7e2e24d677eb222bf1dc964589055e2a4943b36bd8d88b8f3d30798c32e649e7",
    "namespaces": [
      {
        "type": "pid"
      },
      {
        "type": "ipc"
      },
      {
        "type": "uts"
      },
      {
        "type": "mount"
      },
      {
        "type": "network",
        "path": "/var/run/netns/cni-f9c13dea-aef5-aa37-07e3-9a5e1a7b2ba1"
      }
    ]
  }
}
D0629 02:48:16.501414  793356 container.go:180] Create container, cid: 7e2e24d677eb222bf1dc964589055e2a4943b36bd8d88b8f3d30798c32e649e7, rootDir: "/run/containerd/runsc/k8s.io"
D0629 02:48:16.501636  793356 container.go:238] Creating new sandbox for container, cid: 7e2e24d677eb222bf1dc964589055e2a4943b36bd8d88b8f3d30798c32e649e7
D0629 02:48:16.501730  793356 cgroup.go:405] New cgroup for pid: self, &{Mountpoint:/sys/fs/cgroup Path:/kubepods/besteffort/poda61c0ca3-d7de-4e49-84ee-081d1d3b793e Controllers:[cpuset cpu io memory hugetlb pids rdma misc] Own:[]}
D0629 02:48:16.501826  793356 cgroup_v2.go:130] Installing cgroup path "/sys/fs/cgroup/kubepods/besteffort/poda61c0ca3-d7de-4e49-84ee-081d1d3b793e"
D0629 02:48:16.503271  793356 cgroup.go:405] New cgroup for pid: self, &{Mountpoint:/sys/fs/cgroup Path:/kubepods/besteffort/poda61c0ca3-d7de-4e49-84ee-081d1d3b793e/7e2e24d677eb222bf1dc964589055e2a4943b36bd8d88b8f3d30798c32e649e7 Controllers:[cpuset cpu io memory hugetlb pids rdma misc] Own:[]}
D0629 02:48:16.503334  793356 cgroup_v2.go:130] Installing cgroup path "/sys/fs/cgroup/kubepods/besteffort/poda61c0ca3-d7de-4e49-84ee-081d1d3b793e/7e2e24d677eb222bf1dc964589055e2a4943b36bd8d88b8f3d30798c32e649e7"
D0629 02:48:16.541505  793356 cgroup.go:111] Setting "/sys/fs/cgroup/kubepods/besteffort/poda61c0ca3-d7de-4e49-84ee-081d1d3b793e/cgroup.procs" to "0"
D0629 02:48:16.541672  793356 cgroup_v2.go:216] Restoring cgroup "/sys/fs/cgroup/system.slice/k3s.service"
D0629 02:48:16.541709  793356 cgroup.go:111] Setting "/sys/fs/cgroup/system.slice/k3s.service/cgroup.procs" to "0"
D0629 02:48:16.541861  793356 container.go:729] Destroy container, cid: 7e2e24d677eb222bf1dc964589055e2a4943b36bd8d88b8f3d30798c32e649e7
D0629 02:48:16.541930  793356 cgroup_v2.go:175] Deleting cgroup "/sys/fs/cgroup/kubepods/besteffort/poda61c0ca3-d7de-4e49-84ee-081d1d3b793e/7e2e24d677eb222bf1dc964589055e2a4943b36bd8d88b8f3d30798c32e649e7"
D0629 02:48:16.541997  793356 cgroup_v2.go:186] Removing cgroup for path="/sys/fs/cgroup/kubepods/besteffort/poda61c0ca3-d7de-4e49-84ee-081d1d3b793e/7e2e24d677eb222bf1dc964589055e2a4943b36bd8d88b8f3d30798c32e649e7"
W0629 02:48:16.542321  793356 util.go:49] FATAL ERROR: creating container: write /sys/fs/cgroup/kubepods/besteffort/poda61c0ca3-d7de-4e49-84ee-081d1d3b793e/cgroup.procs: device or resource busy
W0629 02:48:16.542626  793356 main.go:255] Failure to execute command, err: 1

Shim log is empty

rlex · 2022-06-29T00:31:35Z

problem solved by adding

[plugins.cri.containerd.runtimes.runc.options]
  SystemdCgroup = true

to containerd options

rlex · 2022-06-29T02:46:11Z

Well, nope. False alarm.

Trying with

[plugins.cri.containerd.runtimes.runsc]
  runtime_type = "io.containerd.runsc.v1"
[plugins.cri.containerd.runtimes.runsc.options]
  TypeUrl = "io.containerd.runsc.v1.options"
  ConfigPath = "/var/lib/rancher/k3s/agent/etc/containerd/runsc.toml"

i'm still seeing same error.

Not sure what triggered running it while i was debugging completely different problem.

Ubuntu 22.04 have cgroupv2 mounted by default to /sys/fs/cgroup

cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
none on /run/cilium/cgroupv2 type cgroup2 (rw,relatime)

Ubuntu 20.04 had cgroupv1

avagin · 2022-06-29T23:07:04Z

SystemdCgroup

Well, nope. False alarm.

Trying with

[plugins.cri.containerd.runtimes.runsc]
  runtime_type = "io.containerd.runsc.v1"
[plugins.cri.containerd.runtimes.runsc.options]
  TypeUrl = "io.containerd.runsc.v1.options"
  ConfigPath = "/var/lib/rancher/k3s/agent/etc/containerd/runsc.toml"

Could you try to add SystemdCgroup = true in plugins.cri.containerd.runtimes.runsc.options?

rlex · 2022-06-29T23:09:11Z

Yeah, i tried, nothing happens. If i enable systemdcgroups in runsc.toml, it then throws me error about invalid cgroup format.

rlex · 2022-07-07T16:22:42Z

Interesting fact: it fails only on control-plane nodes. My control-plane is not tainted (default in k3s), but as soon as i used nodeselector to point to some worker node it works. Configs are the same everywhere.

rlex · 2022-07-14T01:29:39Z

If i explicitely set systemd-cgroup to true in runsc_config, i'm getting following message:

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: creating container: cannot set up cgroup for root: expected cgroupsPath to be of format "slice:prefix:name" for systemd cgroups, got "/kubepods/besteffort/pod152d5fd3-cb35-458c-88aa-de207aa5386f" instead: unknown

What's really interesting is that k3s on control-plane generates following containerd config, where it doesn't launch, either with device/resource busy or format error (in case we enforce systemd-cgroup)

#Template needs to be re-synced with each containerd update
#Current version:
#https://github.com/k3s-io/k3s/blob/v1.24.2%2Bk3s1/pkg/agent/templates/templates_linux.go

[plugins.opt]
  path = "/var/lib/rancher/k3s/agent/containerd"
[plugins.cri]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  enable_selinux = false
  enable_unprivileged_ports = true
  enable_unprivileged_icmp = true
  sandbox_image = "rancher/mirrored-pause:3.6"
[plugins.cri.containerd]
  snapshotter = "overlayfs"
  disable_snapshot_annotations = true
[plugins.cri.containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"
[plugins.cri.containerd.runtimes.runc.options]
	SystemdCgroup = false
[plugins.cri.containerd.runtimes.runsc]
  runtime_type = "io.containerd.runsc.v1"
[plugins.cri.containerd.runtimes.runsc.options]
  SystemdCgroup = false
  TypeUrl = "io.containerd.runsc.v1.options"
  ConfigPath = "/var/lib/rancher/k3s/agent/etc/containerd/runsc.toml"

and here is one from worker nodes, where gvisor works perfectly:

#Template needs to be re-synced with each containerd update
#Current version:
#https://github.com/k3s-io/k3s/blob/v1.24.2%2Bk3s1/pkg/agent/templates/templates_linux.go

[plugins.opt]
  path = "/var/lib/rancher/k3s/agent/containerd"
[plugins.cri]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  enable_selinux = false
  enable_unprivileged_ports = true
  enable_unprivileged_icmp = true
  sandbox_image = "rancher/mirrored-pause:3.6"
[plugins.cri.containerd]
  snapshotter = "overlayfs"
  disable_snapshot_annotations = true
[plugins.cri.containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"
[plugins.cri.containerd.runtimes.runc.options]
	SystemdCgroup = true
[plugins.cri.containerd.runtimes.runsc]
  runtime_type = "io.containerd.runsc.v1"
[plugins.cri.containerd.runtimes.runsc.options]
  SystemdCgroup = true
  TypeUrl = "io.containerd.runsc.v1.options"
  ConfigPath = "/var/lib/rancher/k3s/agent/etc/containerd/runsc.toml"

There is no major differences between those k3s installs: same OSes, same kernels, same k3s versions, and containerd config is generated from exact same template (you can see it here: https://github.com/k3s-io/k3s/blob/v1.24.2%2Bk3s1/pkg/agent/templates/templates_linux.go).

But, strangely enough, there is difference between generated configs: one (worker) is having

SystemdCgroup = true

In runc options (and also in runsc, since i also added it to template)

and another (master) don't:

SystemdCgroup = false

Should ask k3s why it's like that, but still, any ideas on how to get gvisor working on all nodes?

rlex · 2022-07-14T01:57:17Z

looks like regression in k3s. I'll wait for next release where this will be fixed and will report if it will help.

rlex · 2022-07-22T03:27:16Z

In the end, it was k3s regression: k3s-io/k3s#5851

Built version with fix and k3s on masters started to work properly. Sorry for bothering!

rlex added the type: bug Something isn't working label Jun 1, 2022

rlex closed this as completed Jun 29, 2022

rlex reopened this Jun 29, 2022

rlex changed the title ~~Unable to launch~~ Unable to launch on ubuntu 22.04 with cgroupv2 Jun 29, 2022

avagin assigned manninglucas Jun 29, 2022

rlex closed this as completed Jul 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to launch on ubuntu 22.04 with cgroupv2 #7671

Unable to launch on ubuntu 22.04 with cgroupv2 #7671

rlex commented Jun 1, 2022 •

edited

Loading

fvoznika commented Jun 3, 2022

rlex commented Jun 28, 2022

rlex commented Jun 29, 2022

rlex commented Jun 29, 2022 •

edited

Loading

avagin commented Jun 29, 2022

rlex commented Jun 29, 2022 •

edited

Loading

rlex commented Jul 7, 2022 •

edited

Loading

rlex commented Jul 14, 2022

rlex commented Jul 14, 2022

rlex commented Jul 22, 2022 •

edited

Loading

Unable to launch on ubuntu 22.04 with cgroupv2 #7671

Unable to launch on ubuntu 22.04 with cgroupv2 #7671

Comments

rlex commented Jun 1, 2022 • edited Loading

Description

Steps to reproduce

runsc version

docker version (if using docker)

uname

kubectl (if using Kubernetes)

fvoznika commented Jun 3, 2022

rlex commented Jun 28, 2022

rlex commented Jun 29, 2022

rlex commented Jun 29, 2022 • edited Loading

avagin commented Jun 29, 2022

rlex commented Jun 29, 2022 • edited Loading

rlex commented Jul 7, 2022 • edited Loading

rlex commented Jul 14, 2022

rlex commented Jul 14, 2022

rlex commented Jul 22, 2022 • edited Loading

rlex commented Jun 1, 2022 •

edited

Loading

rlex commented Jun 29, 2022 •

edited

Loading

rlex commented Jun 29, 2022 •

edited

Loading

rlex commented Jul 7, 2022 •

edited

Loading

rlex commented Jul 22, 2022 •

edited

Loading