[BUG] Docker compose service causes a kernel panic due to resource starvation

[varishtsg]

Description

This is actually a problem on low resource devices like raspberry pi 4. I have a 8GB raspberry pi, and need to run 4 docker compose files on boot. On boot when the docker service starts, all the services compete for resources, although I have enough ram, the containers get starved for CPU resources which results in a kernel panic and raspberry pi crashes.

Here clearly if I can start the compose files in a staggered manner, it would totally prevent this.

When inspecting the running services using top, I saw that dockerd was using a lot of CPU and load averages were around 6.0 - 8.0

Steps To Reproduce

On raspberry pi 4 8GB running Ubuntu mate 22.04, I have 4 docker compose files that need to be run.

The compose files are for self hosted services like pihole, bitwarden, memos etc.

When starting the system, all the services run in parallel and this leads to a CPU bottleneck, since most services require a decent amount of compute on startup, but once they are running the utilization minimizes.

If the resource requirements are high for a long time, after a while it causes a kernel panic and the raspberry pi crashes.

Compose Version

Docker Compose version v2.17.3

Docker Environment

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.4
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.17.3
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 8
  Running: 7
  Paused: 0
  Stopped: 1
 Images: 10
 Server Version: 23.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-1028-raspi
 Operating System: Ubuntu 22.04.2 LTS
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 7.624GiB
 Name: raspberrypi
 ID: 12c34dd7-3371-4e61-a7bb-946ce4637d6f
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

This is similar to issue #8849, #5422

[ndeloof]

have you seen my comment #8849 (comment) ?

[ndeloof]

Reading your issue more carefully I notice you have "4 docker compose files", so I assume you run 4 docker compose up commands. As you can easily guess, if those running concurrently, Compose can't help limit the amount of consumed resources. Can't you get them ran sequentially? Or group them in a single compose file so that Compose can limit parallelism?

[varishtsg]

Running all the containers in one single compose is not possible, as I want to maintain some level of isolation using stacks.

Here are the 4 stacks:

1. PiHole, Dnscrypt, Portainer
2. JDownloader, qBittorrent
3. Bitwarden and its dependencies
4. Memos and other self hosted utilities.

I did check your comment regarding the variable, COMPOSE_PARALLEL_LIMIT

Where can I set it up? I installed docker using the install script from here: https://github.com/docker/docker-install

EDIT: I did set the variable COMPOSE_PARALLEL_LIMIT=8 under /etc/environment but I do end up with the same issue. When I restart the system, a lot of resources get used up by the dockerd. My load averages shoot up to 6.0 while starting all the services. However, once they are up, they do normalize to around 1.2 - 1.7

[ndeloof]

COMPOSE_PARALLEL_LIMIT applies to a single run of the docker compose command, so won't have any effect if you run those 4 commands isolated. I can't see any other option but you to add some delay between commands at boot time to reduce dockerd load starting services

[ndeloof]

AFAICT this issue is not related to compose, but to startup management : same would apply if you just were using plain docker run xxx on startup. So I'm closing this issue as "not planned", i.e. docker compose only can help you when you give it total control on the lifecycle. Running independent commands with distinct compose files makes it impossible for it to manage concurrency.