Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker CE 18.09.0 on Ubuntu 18.04.1 doesn't create any containers #485

Closed
manishtomar opened this issue Nov 13, 2018 · 25 comments
Closed

Docker CE 18.09.0 on Ubuntu 18.04.1 doesn't create any containers #485

manishtomar opened this issue Nov 13, 2018 · 25 comments

Comments

@manishtomar
Copy link

@ata-sql commented on Fri Nov 09 2018

hello,
after dist-upgrade of Ubuntu to Ubuntu 18.04.1 LTS we can't create any containers

docker info:
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 2
Server Version: 18.09.0
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: c4446665cb9c30056f4998ed953e6d4ff22c7c39
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.15.0-33-generic
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.852GiB
Name: ata-test-ubuntu-bionic
ID: NOKD:6JTX:G2NS:FM3F:LCTW:B4TO:IEAP:XY7S:DCSI:4WX7:QK6B:KU2V
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
HTTP Proxy: http://proxy.efinity.local:3128
HTTPS Proxy: http://proxy.efinity.local:3128
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Registry Mirrors:
 https://docker.arti.efinity.local/
Live Restore Enabled: true
Product License: Community Engine
WARNING: No swap limit support

docker run --network none --rm hello-world ends with:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"/hello\": stat /hello: no such file or directory": unknown.

In syslog we have then:

Nov  9 10:04:13 ata-test-ubuntu-bionic containerd[741]: time="2018-11-09T10:04:13.488624898+01:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/890df89ef5abaab97a7a853410e21ae8b8deac30b4ef86f7c051c7cd1c1ed4b2/shim.sock" debug=false pid=4072
Nov  9 10:04:13 ata-test-ubuntu-bionic containerd[741]: time="2018-11-09T10:04:13.835464827+01:00" level=info msg="shim reaped" id=890df89ef5abaab97a7a853410e21ae8b8deac30b4ef86f7c051c7cd1c1ed4b2
Nov  9 10:04:13 ata-test-ubuntu-bionic dockerd[794]: time="2018-11-09T10:04:13.846098066+01:00" level=error msg="stream copy error: reading from a closed fifo"
Nov  9 10:04:13 ata-test-ubuntu-bionic dockerd[794]: time="2018-11-09T10:04:13.847104823+01:00" level=error msg="stream copy error: reading from a closed fifo"
Nov  9 10:04:13 ata-test-ubuntu-bionic dockerd[794]: time="2018-11-09T10:04:13.985087191+01:00" level=error msg="890df89ef5abaab97a7a853410e21ae8b8deac30b4ef86f7c051c7cd1c1ed4b2 cleanup: failed to delete container from containerd: no such container"

Downgrading docker to 18.06.1-ce (only that package, rest of system remains unchanged) fixes this problem.

@thaJeztah
Copy link
Member

@ata-sql could you provide more information?

  • what version of docker was installed before upgrading? and; was it running without issues?
  • what version of Ubuntu was installed before upgrading? (14.04? 16.04?)

I see that you have only 2 images, and no containers on that machine;

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 2

Did you have existing containers before doing the dist-upgrade? If so; did you remove those after upgrading? Or did you perhaps switch to a different storage driver?

@ghost
Copy link

ghost commented Nov 13, 2018

a) 18.06.1, without any problems on ~100 PCs
b) Ubuntu 18.04.1 LTS
I removed images and containers. I couldn't run new container, so I purged containers and images - to check if problem occurs on fresh system also. Nothing in docker configuration was changed - it is managed by saltstack.

@thaJeztah
Copy link
Member

It only occurs on hosts that were upgraded, or do you see the same on a completely fresh host as well?

@ghost
Copy link

ghost commented Nov 13, 2018

What do you mean by that? We didn't upgrade OS distribution from 16.04 to 18.04 - it was 18.04 for some time (month or two) - and it wasn't upgrade of running system, but fresh reinstall of 18.04.

@thaJeztah
Copy link
Member

Apologies, I think I misunderstood your mention of "after dist-upgrade of Ubuntu to Ubuntu 18.04.1 LTS", so I thought you updated an existing machine to 18.04.1 and docker stopped working.

So; IIUC;

  • Fresh machine with Ubuntu 18.04.1 LTS
  • Fresh install of Docker 18.09.0
  • Any container fails to run

I'm not very familiar with Saltstack (and of course don't know how its configured in your situation);

  • I see you're using a proxy; are there any other configuration changes made to the docker and containerd services?
  • Are there any other entries in the log (e.g. during startup of the dockerd and/or containerd services) that may provide more information?
  • If you run the check-config.sh script (latest version in the moby repository) - does anything stand out?

I just tried an install on a fresh Ubuntu 18.04.1 machine, and was not able to reproduce the issue, so anything you can provide to narrow down what's causing the problem for you would be helpful.

Here's the output of the machine I just installed ;

Client:
 Version:           18.09.0
 API version:       1.39
 Go version:        go1.10.4
 Git commit:        4d60db4
 Built:             Wed Nov  7 00:49:01 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.0
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.4
  Git commit:       4d60db4
  Built:            Wed Nov  7 00:16:44 2018
  OS/Arch:          linux/amd64
  Experimental:     false


Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 18.09.0
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: c4446665cb9c30056f4998ed953e6d4ff22c7c39
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-36-generic
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.947GiB
Name: ubuntu-test
ID: D52A:72HU:AJTE:SBVS:S6ZG:UZVR:EBBZ:R4PO:C3Q6:MGDF:I33K:T43G
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

WARNING: No swap limit support

@ghost
Copy link

ghost commented Nov 13, 2018

daemon.json:
{"dns": ["192.168.53.5", "192.168.53.6"], "dns-search": ["our.local", "aws.our.com", "our.desktop"], "live-restore": true, "registry-mirrors": ["https://docker.arti.our.local"]}

check-config.sh output:

warning: /proc/config.gz does not exist, searching other paths for kernel config ...
info: reading kernel config from /boot/config-4.15.0-33-generic ...

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_NF_NAT_IPV4: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_NF_NAT_NEEDED: enabled
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: enabled
- CONFIG_MEMCG_SWAP_ENABLED: missing
    (cgroup swap accounting is currently not enabled, you can enable it by setting boot option "swapaccount=1")
- CONFIG_LEGACY_VSYSCALL_EMULATE: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_IOSCHED_CFQ: enabled
- CONFIG_CFQ_GROUP_IOSCHED: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: enabled (as module)
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: missing
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled (as module)
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled
      - CONFIG_CRYPTO_SEQIV: enabled
      - CONFIG_CRYPTO_GHASH: enabled
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled (as module)
      - CONFIG_XFRM_ALGO: enabled (as module)
      - CONFIG_INET_ESP: enabled (as module)
      - CONFIG_INET_XFRM_MODE_TRANSPORT: enabled (as module)
  - "ipvlan":
    - CONFIG_IPVLAN: enabled (as module)
  - "macvlan":
    - CONFIG_MACVLAN: enabled (as module)
    - CONFIG_DUMMY: enabled (as module)
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_FTP: enabled (as module)
    - CONFIG_NF_NAT_TFTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_TFTP: enabled (as module)
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: enabled (as module)
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled (as module)
    - CONFIG_BTRFS_FS_POSIX_ACL: enabled
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled
    - CONFIG_DM_THIN_PROVISIONING: enabled (as module)
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

/var/log/systlog:

Nov 13 11:44:26 ata-test-ubuntu-bionic systemd-udevd[21555]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Nov 13 11:44:26 ata-test-ubuntu-bionic systemd-udevd[21555]: Could not generate persistent MAC address for veth92a16bf: No such file or directory
Nov 13 11:44:26 ata-test-ubuntu-bionic systemd-udevd[21556]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Nov 13 11:44:26 ata-test-ubuntu-bionic systemd-udevd[21556]: Could not generate persistent MAC address for veth24dee86: No such file or directory
Nov 13 11:44:26 ata-test-ubuntu-bionic kernel: [  513.113948] docker0: port 1(veth24dee86) entered blocking state
Nov 13 11:44:26 ata-test-ubuntu-bionic kernel: [  513.113951] docker0: port 1(veth24dee86) entered disabled state
Nov 13 11:44:26 ata-test-ubuntu-bionic kernel: [  513.114020] device veth24dee86 entered promiscuous mode
Nov 13 11:44:26 ata-test-ubuntu-bionic kernel: [  513.114197] IPv6: ADDRCONF(NETDEV_UP): veth24dee86: link is not ready
Nov 13 11:44:26 ata-test-ubuntu-bionic NetworkManager[878]: <info>  [1542105866.9859] manager: (veth92a16bf): new Veth device (/org/freedesktop/NetworkManager/Devices/10)
Nov 13 11:44:26 ata-test-ubuntu-bionic NetworkManager[878]: <info>  [1542105866.9874] manager: (veth24dee86): new Veth device (/org/freedesktop/NetworkManager/Devices/11)
Nov 13 11:44:27 ata-test-ubuntu-bionic NetworkManager[878]: <info>  [1542105867.0004] devices added (path: /sys/devices/virtual/net/veth92a16bf, iface: veth92a16bf)
Nov 13 11:44:27 ata-test-ubuntu-bionic NetworkManager[878]: <info>  [1542105867.0004] device added (path: /sys/devices/virtual/net/veth92a16bf, iface: veth92a16bf): no ifupdown configuration found.
Nov 13 11:44:27 ata-test-ubuntu-bionic NetworkManager[878]: <info>  [1542105867.0037] devices added (path: /sys/devices/virtual/net/veth24dee86, iface: veth24dee86)
Nov 13 11:44:27 ata-test-ubuntu-bionic NetworkManager[878]: <info>  [1542105867.0038] device added (path: /sys/devices/virtual/net/veth24dee86, iface: veth24dee86): no ifupdown configuration found.
Nov 13 11:44:27 ata-test-ubuntu-bionic containerd[1133]: time="2018-11-13T11:44:27.014992142+01:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/2736815116a4e66bcb3654ae5201bb62726d9220314fbc00f86084cf4b5efbf9/shim.sock" debug=false pid=21568
Nov 13 11:44:27 ata-test-ubuntu-bionic kernel: [  513.365353] eth0: renamed from veth92a16bf
Nov 13 11:44:27 ata-test-ubuntu-bionic NetworkManager[878]: <info>  [1542105867.2511] devices removed (path: /sys/devices/virtual/net/veth92a16bf, iface: veth92a16bf)
Nov 13 11:44:27 ata-test-ubuntu-bionic NetworkManager[878]: <info>  [1542105867.2518] device (veth24dee86): carrier: link connected
Nov 13 11:44:27 ata-test-ubuntu-bionic NetworkManager[878]: <info>  [1542105867.2519] device (docker0): carrier: link connected
Nov 13 11:44:27 ata-test-ubuntu-bionic kernel: [  513.381353] IPv6: ADDRCONF(NETDEV_CHANGE): veth24dee86: link becomes ready
Nov 13 11:44:27 ata-test-ubuntu-bionic kernel: [  513.381396] docker0: port 1(veth24dee86) entered blocking state
Nov 13 11:44:27 ata-test-ubuntu-bionic kernel: [  513.381398] docker0: port 1(veth24dee86) entered forwarding state
Nov 13 11:44:27 ata-test-ubuntu-bionic gnome-shell[3476]: Removing a network device that was not added
Nov 13 11:44:27 ata-test-ubuntu-bionic containerd[1133]: time="2018-11-13T11:44:27.399559367+01:00" level=info msg="shim reaped" id=2736815116a4e66bcb3654ae5201bb62726d9220314fbc00f86084cf4b5efbf9
Nov 13 11:44:27 ata-test-ubuntu-bionic dockerd[1359]: time="2018-11-13T11:44:27.409843856+01:00" level=error msg="stream copy error: reading from a closed fifo"
Nov 13 11:44:27 ata-test-ubuntu-bionic dockerd[1359]: time="2018-11-13T11:44:27.409843612+01:00" level=error msg="stream copy error: reading from a closed fifo"
Nov 13 11:44:27 ata-test-ubuntu-bionic kernel: [  513.571369] docker0: port 1(veth24dee86) entered disabled state
Nov 13 11:44:27 ata-test-ubuntu-bionic kernel: [  513.571429] veth92a16bf: renamed from eth0
Nov 13 11:44:27 ata-test-ubuntu-bionic NetworkManager[878]: <info>  [1542105867.4930] manager: (veth92a16bf): new Veth device (/org/freedesktop/NetworkManager/Devices/12)
Nov 13 11:44:27 ata-test-ubuntu-bionic systemd-udevd[21639]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Nov 13 11:44:27 ata-test-ubuntu-bionic kernel: [  513.635654] docker0: port 1(veth24dee86) entered disabled state
Nov 13 11:44:27 ata-test-ubuntu-bionic kernel: [  513.638060] device veth24dee86 left promiscuous mode
Nov 13 11:44:27 ata-test-ubuntu-bionic kernel: [  513.638066] docker0: port 1(veth24dee86) entered disabled state
Nov 13 11:44:27 ata-test-ubuntu-bionic NetworkManager[878]: <info>  [1542105867.5271] devices added (path: /sys/devices/virtual/net/veth92a16bf, iface: veth92a16bf)
Nov 13 11:44:27 ata-test-ubuntu-bionic NetworkManager[878]: <info>  [1542105867.5272] device added (path: /sys/devices/virtual/net/veth92a16bf, iface: veth92a16bf): no ifupdown configuration found.
Nov 13 11:44:27 ata-test-ubuntu-bionic NetworkManager[878]: <info>  [1542105867.5273] devices removed (path: /sys/devices/virtual/net/veth92a16bf, iface: veth92a16bf)
Nov 13 11:44:27 ata-test-ubuntu-bionic NetworkManager[878]: <info>  [1542105867.5292] device (veth24dee86): released from master device docker0
Nov 13 11:44:27 ata-test-ubuntu-bionic gnome-shell[3476]: Removing a network device that was not added
Nov 13 11:44:27 ata-test-ubuntu-bionic gnome-shell[3476]: Removing a network device that was not added
Nov 13 11:44:27 ata-test-ubuntu-bionic NetworkManager[878]: <info>  [1542105867.5371] devices removed (path: /sys/devices/virtual/net/veth24dee86, iface: veth24dee86)
Nov 13 11:44:27 ata-test-ubuntu-bionic dockerd[1359]: time="2018-11-13T11:44:27.677288749+01:00" level=error msg="2736815116a4e66bcb3654ae5201bb62726d9220314fbc00f86084cf4b5efbf9 cleanup: failed to delete container from containerd: no such container"

@thaJeztah
Copy link
Member

daemon.json:
{"dns": ["192.168.53.5", "192.168.53.6"], "dns-search": ["our.local", "aws.our.com", "our.desktop"], "live-restore": true, "registry-mirrors": ["https://docker.arti.our.local"]}

Not sure how this would relate to starting new containers, but I overlooked you're using live-restore; did you stop all containers before upgrading the docker daemon?

live-restore cannot be used when doing major version upgrades (e.g. 18.06 -> 18.09), and can only be used for installing patch releases (e.g. 18.06.0 -> 18.06.1). For major updates, it's important to stop containers first.

I don't see anything standing out in the output of check-config.sh (all looks normal)

I do see some messages about NetworkManager in the logs; NetworkManager is known to cause some issues in certain situations (trying to manage virtual interfaces that containers create); what happens if you try to start the container in host network mode?

docker run --rm --network=host hello-world

@ghost
Copy link

ghost commented Nov 13, 2018

For sure there were no running containers when I did it in test environment.
I tried to do it with no network and same result.
net=host

atalarek@ata-test-ubuntu-bionic:~$ docker run --network host --rm hello-world
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:301: running exec setns process for init caused \"exit status 22\"": unknown.
Nov 13 12:55:21 ata-test-ubuntu-bionic containerd[1071]: time="2018-11-13T12:55:21.956100414+01:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/11a549a67ff4d616024cfd5d3bb5f8fc5140c5c5da00245f13aca56c8fb4bbf6/shim.sock" debug=false pid=3171
Nov 13 12:55:22 ata-test-ubuntu-bionic containerd[1071]: time="2018-11-13T12:55:22.006538198+01:00" level=info msg="shim reaped" id=11a549a67ff4d616024cfd5d3bb5f8fc5140c5c5da00245f13aca56c8fb4bbf6
Nov 13 12:55:22 ata-test-ubuntu-bionic dockerd[1327]: time="2018-11-13T12:55:22.016949331+01:00" level=error msg="stream copy error: reading from a closed fifo"
Nov 13 12:55:22 ata-test-ubuntu-bionic dockerd[1327]: time="2018-11-13T12:55:22.017002349+01:00" level=error msg="stream copy error: reading from a closed fifo"
Nov 13 12:55:22 ata-test-ubuntu-bionic dockerd[1327]: time="2018-11-13T12:55:22.114964671+01:00" level=error msg="11a549a67ff4d616024cfd5d3bb5f8fc5140c5c5da00245f13aca56c8fb4bbf6 cleanup: failed to delete container from containerd: no such container"
Nov 13 12:55:22 ata-test-ubuntu-bionic dockerd[1327]: time="2018-11-13T12:55:22.137635125+01:00" level=error msg="Handler for POST /v1.39/containers/11a549a67ff4d616024cfd5d3bb5f8fc5140c5c5da00245f13aca56c8fb4bbf6/start returned error: OCI runtime create failed: container_linux.go:348: starting container process caused \"process_linux.go:301: running exec setns process for init caused \\\"exit status 22\\\"\": unknown"

net=none

atalarek@ata-test-ubuntu-bionic:~$ docker run --network none --rm hello-world
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"/hello\": stat /hello: no such file or directory": unknown.
Nov 13 12:57:00 ata-test-ubuntu-bionic containerd[1071]: time="2018-11-13T12:57:00.908117245+01:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/d8860060862463bb4722277bd3ff8f824e8d2393a1562975734c8aba8b3ab62c/shim.sock" debug=false pid=3238
Nov 13 12:57:01 ata-test-ubuntu-bionic containerd[1071]: time="2018-11-13T12:57:01.260087623+01:00" level=info msg="shim reaped" id=d8860060862463bb4722277bd3ff8f824e8d2393a1562975734c8aba8b3ab62c
Nov 13 12:57:01 ata-test-ubuntu-bionic dockerd[1327]: time="2018-11-13T12:57:01.270676024+01:00" level=error msg="stream copy error: reading from a closed fifo"
Nov 13 12:57:01 ata-test-ubuntu-bionic dockerd[1327]: time="2018-11-13T12:57:01.270768054+01:00" level=error msg="stream copy error: reading from a closed fifo"
Nov 13 12:57:01 ata-test-ubuntu-bionic dockerd[1327]: time="2018-11-13T12:57:01.410038429+01:00" level=error msg="d8860060862463bb4722277bd3ff8f824e8d2393a1562975734c8aba8b3ab62c cleanup: failed to delete container from containerd: no such container"

@thaJeztah
Copy link
Member

No clue yet. Did a quick search for these errors; moby/moby#36084 mentions a similar error, and looks to be related to a specific kernel version.

Other possible suspects would be the CONFIG_OVERLAY_FS_REDIRECT_DIR=y configuration (see moby/moby#34342), but a patch was applied quite a while back to take that into account 🤔 (also see moby/moby#33733 (comment))

@ghost
Copy link

ghost commented Nov 14, 2018

Adding "storage-opts": [ "overlay2.override_kernel_check=1" ] option to docker didn't change anything.
I don't think it's kernel issue, because after upgrading to 4.17 problem still occurs.

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 18.09.0
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: c4446665cb9c30056f4998ed953e6d4ff22c7c39
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.17.0-041700-generic
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.852GiB
Name: ata-test-ubuntu-bionic
ID: NOKD:6JTX:G2NS:FM3F:LCTW:B4TO:IEAP:XY7S:DCSI:4WX7:QK6B:KU2V
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
HTTP Proxy: http://proxy.efinity.local:3128
HTTPS Proxy: http://proxy.efinity.local:3128
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Registry Mirrors:
 https://docker.arti.efinity.local/
Live Restore Enabled: true
Product License: Community Engine

WARNING: No swap limit support

and still

Status: Downloaded newer image for hello-world:latest
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"/hello\": stat /hello: no such file or directory": unknown.

@thaJeztah
Copy link
Member

It's not the override_kernel_check, because that was only used for Red Hat (3.10) kernels that had support for overlay back ported.

@rmoriz
Copy link

rmoriz commented Nov 15, 2018

Having the same issue on Debian 9.6 after upgrading from 18.03 or 18.06 to 18.09 (5:18.09.0~3-0~debian-stretch) release. Downgrade to 18.06 (18.06.1~ce~3-0~debian) fixed the issue. No other packages were updated. In our case it turned out that customized systemd units (docker.socket and docker.service) in /etc/systemd/system caused the issue because 18.09 introduced the new containerd.service unit and got rid of docker.socket. After deletion of our old units and daemon-reloading it works again.

@pisymbol
Copy link

Same issue on 18.04 for me. OCI runtime errors with latest docker package. Needed to downgrade to 18.06.1ce3-0-ubuntu for it to work again.

The latest docker pkg is definitely borked somehow.

@thaJeztah
Copy link
Member

@pisymbol did you get any error logs during startup of the dockerd (and/or containerd) service?

@pisymbol
Copy link

pisymbol commented Nov 20, 2018 via email

@thaJeztah
Copy link
Member

There was an issue with packaging due to the containerd service doing a modprobe in the ExecPreStart of the service (see containerd/containerd#2776, containerd/containerd#2772 and #475) but the service should not continue to start in that case.

The OCI runtime error is generated in runc; https://github.com/opencontainers/runc/blob/v1.0.0-rc5/libcontainer/container_linux.go#L348, but not sure what the cause is. I'll ask around in the runtime team if they have seen this issue before.

@jamshid
Copy link

jamshid commented Nov 24, 2018

I'm having this same problem since upgrading my ubuntu 16.04 server from 18.06 to 18.09.
Hoping I don't have to reset this environment (delete /var/lib/docker?) but maybe that's fastest way.

# docker run --rm --network=host hello-world
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:301: running exec setns process for init caused \"exit status 22\"": unknown.
# docker info
Containers: 3
 Running: 0
 Paused: 0
 Stopped: 3
Images: 1
Server Version: 18.09.0
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: pending
 NodeID: 4pn2yghnqc4klx588exfq5efq
 Is Manager: false
 Node Address: 172.30.15.10
 Manager Addresses:
  172.30.15.3:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: c4446665cb9c30056f4998ed953e6d4ff22c7c39
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-139-generic
Operating System: Ubuntu 16.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.66GiB
Name: mb1-10.tx.example.com
ID: YNPQ:NGJV:B3HW:2W4W:3EDA:I34V:2Q57:SWFR:JRS6:VLBU:HBVY:DULZ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
 provider=generic
Experimental: false
Insecure Registries:
 docker.tx.example.com:5000
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

WARNING: No swap limit support

@edwardwu
Copy link

Fixed it by removing old version of runc - using whereis runc.

I had the same issue after migration to 18.04.1 and docker-ce version is

docker version
Client:
Version: 18.09.0
API version: 1.39
Go version: go1.10.4
Git commit: 4d60db4
Built: Wed Nov 7 00:49:01 2018
OS/Arch: linux/amd64
Experimental: false

The error is
"OCI runtime create failed: target os mismatch with current os linux: unknown"

@pisymbol
Copy link

Is there any update to this? I don't have an old version of runc (I do have a custom script though called runc that calls docker run etc.).

@ghost
Copy link

ghost commented Jan 31, 2019

Are you planning to fix that issue? It's more than 2 months now and issue seems to be quite serious.

@thaJeztah
Copy link
Member

@ata-sql do you happen to have a customised systemd unit file for the docker service? I just recalled docker/docs#8001 (related to moby/moby#38506), which could be related.

Fixing the issue will be quite difficult if it's there's no consistent reproducer, or cause identified.

@pisymbol
Copy link

So for me my fix was simply to rename my 'runc' script to something else and that fixed it.

@ghost
Copy link

ghost commented Feb 1, 2019

Hello,
yes, we have customized systemd unit. I've just tested uninstalling "our" docker and installing it again without our customization - it worked fine.
Thanks for clues - I will make some research in that direction.

@ghost
Copy link

ghost commented Feb 1, 2019

For others:
remove MountFlags=slave from service file.
Close this issue please.

@thaJeztah
Copy link
Member

Good to hear @ata-sql! I'll close this issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants