Hitting Container Kernel Memory Limit causes OOM on Docker Host #1001
Comments
|
I see you're running the ubuntu packaged build of Docker (not sure what versions you installed on the other distributions); are you seeing the same problem with the official packages of docker and containerd? I just tried to reproduce the issue on a Ubuntu 18.04 droplet on DigitalOcean, and wasn't able to reproduce the issue; curl -fsSL https://get.docker.com | sh
# ...
docker version
Client: Docker Engine - Community
Version: 19.03.8
API version: 1.40
Go version: go1.12.17
Git commit: afacb8b7f0
Built: Wed Mar 11 01:25:46 2020
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.8
API version: 1.40 (minimum version 1.12)
Go version: go1.12.17
Git commit: afacb8b7f0
Built: Wed Mar 11 01:24:19 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.13
GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683
root@ubuntu-s-1vcpu-2gb-ams3-01:~# docker info
Client:
Debug Mode: false
docker info
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 19.03.8
Storage Driver: overlay2
Backing Filesystem: <unknown>
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-66-generic
Operating System: Ubuntu 18.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.947GiB
Name: ubuntu-s-1vcpu-2gb-ams3-01
ID: 24J2:2M3F:5RKT:FJNS:GDJK:K6Y7:XJ6M:4XQI:ARBT:5FR2:TCRP:YEU6
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
uname -a
Linux ubuntu-s-1vcpu-2gb-ams3-01 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linuxdocker build -t bomb https://github.com/lmm-git/docker-fork-bomb.git
docker run -it --rm --memory=128M --kernel-memory=16M bomb
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
Traceback (most recent call last):KilledAnd without a tty attached; docker run --memory=128M --kernel-memory=16M bomb
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
Traceback (most recent call last):
File "/bomb.py", line 6, in <module>
os.fork()
OSError: [Errno 12] Cannot allocate memory
KilledNote that to protect against a fork-bomb, you may also be interested in setting |
|
Trying on a machine with kernel 5.x now |
|
Hm, after running a couple of times on the first machine, it looks like containerd is killed, and the docker daemon with it. But didn't manage to repeat that situation |
|
Managed to reproduce that situation (containerd gone, and docker daemon being shutdown with it) on ubuntu 19.10; Syslog; |
|
Ok, and trying again, without a TTY attached to the container, and I managed to get my SSH session being killed And a second time (this time with (also managed with |
|
Thanks for investigating! Here are some Docker versions we tested:
In our experience, reproducing the issue involves some try and error. It might be worth to try with more or less other programs to get a higher chance for the OOM killer to kill an innocent program. |
|
@thaJeztah Do you have any news or additional insights about this issue? |
|
It seems the similar issue as we have met. |
|
@congcongke We're still facing this issue. A workaround we have been using since May (and which hasn't led to this issue so far) is to set the kernel memory limit greater than or equal to the memory limit. |
|
+1 We have the same issue where a process in one container gets killed then causing a cascading chain of events where Details |
|
To prevent containerd being OOM killed by the kernel, you can add an systemd drop-in file to adjust the OOM memory score for it. A change for that was merged in docker/containerd-packaging#186, but is not yet in a released version of the package. |
|
Thanks for that @thaJeztah - ack on the PR, we'll manually change that for the time being until the next release! Thanks again for all the work that you put into maintaining this! 🎉 |
4 tasks
|
Any update on this issue? Some container on my machine triggered oom-killer continuously and caused hour-long outage for my services. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Expected behavior
If a kernel memory limit has been set for a specific container and is hit during program execution, only the container is affected by the kernel Out Of Memory (OOM) killer. Any process within the container might be affected and as a worst-case scenario, the container might quit.
Further, we expected the documentation regarding the
--kernel-memoryoption to be correct. For our case, with a kernel memory limit below the memory limit (both set), it states:Actual behavior
If the kernel memory limit of the container is hit and more kernel memory is required, the Docker host (!, not the container) experiences a system-wide OOM (even though the host has more than enough free memory available). As a result, arbitrary applications and system services might be killed. In our experience, system-critical services such as the file system driver for ext4 were affected, causing the file system to get corrupted and further services to crash.
In our testings, the documentation proved to be incorrect: The host and other containers were affected by OOM killer.
Shell (shortend, see this repo for full log):
Syslog (shortend, see this repo for full log):
Steps to reproduce the behavior
docker run -it --rm --memory=128M --kernel-memory=16M lmmdock/fork-bombSource Code / Dockerfile / Log output:
lmm-git/docker-fork-bomb(incl. full Syslog)lmmdock/fork-bombOutput of
docker version:Output of
docker info:Additional environment details (AWS, VirtualBox, physical, etc.)
→ All showed similar behavior.
The text was updated successfully, but these errors were encountered: