Error response from daemon / Docker engine panic: leave after shutdown

[gesellix]

Expected behavior

Docker engine should not panic

Actual behavior

Docker engine is not reachable. Client prints Error response from daemon: Bad response from Docker engine, Engine logs show panic: leave after shutdown

Information

Using the latest Docker for Mac beta:

Version 1.12.3-rc1-beta29 (13397)
Channel: Beta
2b9098935b

Output of docker info:

Containers: 7
 Running: 2
 Paused: 0
 Stopped: 5
Images: 16
Server Version: 1.12.3-rc1
Storage Driver: overlay2
 Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: overlay bridge host null
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.4.27-moby
Operating System: Alpine Linux v3.4
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.951 GiB
Name: moby
ID: GWRQ:KUJC:2BHK:DECZ:XRV5:27FI:K2ER:TDEP:GGOT:YG6A:NWWE:YYZI
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 25
 Goroutines: 47
 System Time: 2016-10-26T09:52:21.299394691Z
 EventsListeners: 1
No Proxy: *.local, 169.254/16
Username: gesellix
Registry: https://index.docker.io/v1/
WARNING: No kernel memory limit support
Experimental: true
Insecure Registries:
 example.com:5000
 127.0.0.0/8

Diagnostic ID: 823B859D-B25B-4842-9FCD-BD4C4DCB4BF2

The last entries in /var/log/docker.log are:

time="2016-10-26T08:34:47.711172083Z" level=debug msg="Calling POST /swarm/init" 
time="2016-10-26T08:34:47.711666887Z" level=debug msg="form data: {\"ForceNewCluster\":false,\"ListenAddr\":\"0.0.0.0:4554\",\"Spec\":{\"AcceptancePolicy\":{\"Policies\":[{\"Autoaccept\":true,\"Role\":\"MANAGER\"},{\"Autoaccept\":true,\"Role\":\"WORKER\"}]},\"CAConfig\":{},\"Dispatcher\":{},\"Orchestration\":{},\"Raft\":{}}}" 
time="2016-10-26T08:34:47.737327728Z" level=error msg="failed to send node leave: timed out broadcasting node event" 
panic: leave after shutdown

goroutine 3571 [running]:
panic(0x17c5e60, 0xc82146b260)
    /usr/local/go/src/runtime/panic.go:481 +0x3e6
github.com/hashicorp/memberlist.(*Memberlist).Leave(0xc8208b8ee0, 0x3b9aca00, 0x0, 0x0)
    /go/src/github.com/docker/docker/vendor/src/github.com/hashicorp/memberlist/memberlist.go:578 +0xcd
github.com/docker/libnetwork/networkdb.(*NetworkDB).clusterLeave(0xc820427380, 0x0, 0x0)
    /go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/networkdb/cluster.go:206 +0x14c
github.com/docker/libnetwork/networkdb.(*NetworkDB).Close(0xc820427380)
    /go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/networkdb/networkdb.go:194 +0x2b
github.com/docker/libnetwork.(*controller).agentClose(0xc8204b73b0)
    /go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/agent.go:342 +0x161
github.com/docker/libnetwork.(*controller).clusterAgentInit(0xc8204b73b0)
    /go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/controller.go:323 +0x2dc
created by github.com/docker/libnetwork.(*controller).SetClusterProvider
    /go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/controller.go:238 +0xc4

• A reproducible case if this is a bug, Dockerfiles FTW

Dockerfile to reproduce the error, an automated build is available as gesellix/for-mac-issue-849:

FROM java:8
RUN mkdir -p /issue-849 && cd /issue-849 && git clone https://github.com/gesellix/docker-client.git
WORKDIR /issue-849/docker-client
RUN ./gradlew compileTestGroovy
CMD [ './gradlew' ]

Steps to reproduce the behavior

docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock gesellix/for-mac-issue-849 ./gradlew integrationTest --tests "*DockerSwarmIntegrationSpec"

[gesellix]

The integration test mentioned above (DockerSwarmIntegrationSpec) to reproduce the error makes a bunch of swarm init, and swarm leave --force. Maybe it doesn't use a valid config or there's some timing issue due to some missing waits? See the source code at DockerSwarmIntegrationSpec for details, I'm happy to help if you have any questions!

[gesellix]

After further investigation it seems like a fast and repeated docker swarm init; docker swarm leave --force; docker swarm init; ... sequence can trigger the error.
Those commands sequence is obviously not the real life use case, but integration tests are part of a developer's life ;-) and I also didn't have any problems in older releases (pre 1.12.3-rc1-beta29 (13397)).
Is this something which might be fixed easily?

[mchiang0610]

@gesellix Thanks for sending this. I am able to reproduce this using your method

docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock gesellix/for-mac-issue-849 ./gradlew integrationTest --tests "*DockerSwarmIntegrationSpec"

Docker for Mac Version 1.12.3-beta29.2 (13499). Channel: Beta (902414df0c)
Operating System: MacOS 10.12.1 Beta (16B2548a).

Thank you for all the help. We will look into a potential fix for this.

[samoht]

A fix has been pushed upstream moby/libnetwork#1533 and will be integrated in Docker 1.13. We will ship it in Docker for Mac as soon it is released. Thanks for your report!

[gesellix]

Thanks!

[gesellix]

I tested the issue against Docker 1.13.0-rc3 (Git commit 4d92237) and can confirm this issue to be fixed.

[balrajsingh]

Thanks for the report and thanks for confirming that it's fixed. Closing.

[docker-robott]

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked