[bug] Cannot build with DOCKER_BUILDKIT=0

[Niek]

• I have tried with the latest version of Docker Desktop
• I have tried disabling enabled experimental features
• I have uploaded Diagnostics
• Diagnostics ID: 34762F73-A587-449C-9D39-39DCCE54882E/20230524163419

Expected behavior

It should be possible to build a simple Dockerfile with DOCKER_BUILDKIT=0

Actual behavior

Docker fails with Error response from daemon: Bad response from Docker engine

Information

• macOS Version: 13.4
• Intel chip or Apple chip: M1
• Docker Desktop Version: 4.19.0

Output of /Applications/Docker.app/Contents/MacOS/com.docker.diagnose check

❯ /Applications/Docker.app/Contents/MacOS/com.docker.diagnose check
[2023-05-24T16:36:52.615394000Z][com.docker.diagnose][I] set path configuration to OnHost
Starting diagnostics

[PASS] DD0027: is there available disk space on the host?
[PASS] DD0028: is there available VM disk space?
[PASS] DD0018: does the host support virtualization?
[PASS] DD0001: is the application running?
[PASS] DD0017: can a VM be started?
[PASS] DD0016: is the LinuxKit VM running?
[PASS] DD0011: are the LinuxKit services running?
[PASS] DD0004: is the Docker engine running?
[PASS] DD0015: are the binary symlinks installed?
[PASS] DD0031: does the Docker API work?
[PASS] DD0013: is the $PATH ok?
[PASS] DD0003: is the Docker CLI working?
[PASS] DD0038: is the connection to Docker working?
[FAIL] DD0014: are the backend processes running? 1 error occurred:
	* com.docker.vpnkit is not running


[PASS] DD0007: is the backend responding?
[PASS] DD0008: is the native API responding?
[FAIL] DD0009: is the vpnkit API responding? dial unix vpnkit.diag.sock: connect: connection refused
[PASS] DD0010: is the Docker API proxy responding?
[SKIP] DD0030: is the image access management authorized?
[PASS] DD0033: does the host have Internet access?
[PASS] DD0018: does the host support virtualization?
[PASS] DD0001: is the application running?
[PASS] DD0017: can a VM be started?
[PASS] DD0016: is the LinuxKit VM running?
[PASS] DD0011: are the LinuxKit services running?
[PASS] DD0004: is the Docker engine running?
[PASS] DD0015: are the binary symlinks installed?
[PASS] DD0031: does the Docker API work?
[PASS] DD0032: do Docker networks overlap with host IPs?

Please investigate the following 1 issue:

1 : The test: are the backend processes running?
    Failed with: 1 error occurred:
	* com.docker.vpnkit is not running



Not all of the backend processes are running.

Steps to reproduce the behavior

❯ ls
Dockerfile
❯ cat Dockerfile
FROM scratch
ENTRYPOINT ["FAIL"]
❯ DOCKER_BUILDKIT=0 docker build .
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  2.048kB
Error response from daemon: Bad response from Docker engine

The following info is in the log file ~/Library/Containers/com.docker.docker/Data/log/vm/dockerd.log with debug logging enabled:

[2023-05-24T16:38:10.074557054Z][dockerd][I] time="2023-05-24T16:38:10.074469262Z" level=debug msg="Calling POST /v1.42/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&shmsize=0&target=&ulimits=null&version=1"
[2023-05-24T16:38:10.077051679Z][dockerd][I] 2023/05/24 16:38:10 http: panic serving @: not implemented
[2023-05-24T16:38:10.077059845Z][dockerd][I] goroutine 4608 [running]:
[2023-05-24T16:38:10.077067179Z][dockerd][I] net/http.(*conn).serve.func1()
[2023-05-24T16:38:10.077067387Z][dockerd][I] 	/usr/local/go/src/net/http/server.go:1854 +0xb0
[2023-05-24T16:38:10.077070512Z][dockerd][I] panic({0x1ce2320, 0x24df2d0})
[2023-05-24T16:38:10.077070637Z][dockerd][I] 	/usr/local/go/src/runtime/panic.go:890 +0x248
[2023-05-24T16:38:10.077072970Z][dockerd][I] github.com/docker/docker/daemon/containerd.(*ImageService).MakeImageCache(0x4001754c08?, {0x1232348?, 0x30?}, {0x1ef3460?, 0x4001754c01?, 0x51c890?})
[2023-05-24T16:38:10.077074429Z][dockerd][I] 	/go/src/github.com/docker/docker/daemon/containerd/cache.go:11 +0x2c
[2023-05-24T16:38:10.077088804Z][dockerd][I] github.com/docker/docker/builder/dockerfile.newImageProber.func1(...)
[2023-05-24T16:38:10.077089137Z][dockerd][I] 	/go/src/github.com/docker/docker/builder/dockerfile/imageprobe.go:32
[2023-05-24T16:38:10.077101845Z][dockerd][I] github.com/docker/docker/builder/dockerfile.newImageProber({0x2504e98, 0x40015f6280}, {0xffff58288018?, 0x400000f830}, {0x37f6ce8?, 0x0, 0x0}, 0x0?)
[2023-05-24T16:38:10.077102012Z][dockerd][I] 	/go/src/github.com/docker/docker/builder/dockerfile/imageprobe.go:35 +0xe4
[2023-05-24T16:38:10.077104554Z][dockerd][I] github.com/docker/docker/builder/dockerfile.newBuilder({0x2504e98, 0x40015f6280}, {0x40007b4fc0, {0x25161b0, 0x400000f830}, {{0x24e10c0, 0x400000ea98}, {0x24e1d40, 0x400000eab0}, {0x24e1d40, ...}, ...}, ...})
[2023-05-24T16:38:10.077116679Z][dockerd][I] 	/go/src/github.com/docker/docker/builder/dockerfile/builder.go:138 +0x88
[2023-05-24T16:38:10.077119720Z][dockerd][I] github.com/docker/docker/builder/dockerfile.(*BuildManager).Build(0x40008097c0, {0x2504f40, 0x40015f8300}, {{0x24ee0d8, 0x40015f8420}, {{0x24e10c0, 0x400000ea98}, {0x24e1d40, 0x400000eab0}, {0x24e1d40, ...}, ...}, ...})
[2023-05-24T16:38:10.077120179Z][dockerd][I] 	/go/src/github.com/docker/docker/builder/dockerfile/builder.go:94 +0x214
[2023-05-24T16:38:10.077123095Z][dockerd][I] github.com/docker/docker/api/server/backend/build.(*Backend).Build(0x4000f01710, {0x2504f40, 0x40015f8300}, {{0x24ee0d8, 0x40015f8420}, {{0x24e10c0, 0x400000ea98}, {0x24e1d40, 0x400000eab0}, {0x24e1d40, ...}, ...}, ...})
[2023-05-24T16:38:10.077123304Z][dockerd][I] 	/go/src/github.com/docker/docker/api/server/backend/build/backend.go:69 +0x154
[2023-05-24T16:38:10.077126095Z][dockerd][I] github.com/docker/docker/api/server/router/build.(*buildRouter).postBuild(0x4000f79a00, {0x2504f40, 0x40015f8300}, {0x2503820, 0x40011d0000}, 0x4000814700, 0xffff594bbf01?)
[2023-05-24T16:38:10.077126429Z][dockerd][I] 	/go/src/github.com/docker/docker/api/server/router/build/build_routes.go:280 +0x5d0
[2023-05-24T16:38:10.077128887Z][dockerd][I] github.com/docker/docker/api/server/middleware.ExperimentalMiddleware.WrapHandler.func1({0x2504f40, 0x40015f8300}, {0x2503820?, 0x40011d0000?}, 0x1ce2320?, 0x4000aee190?)
[2023-05-24T16:38:10.077129054Z][dockerd][I] 	/go/src/github.com/docker/docker/api/server/middleware/experimental.go:26 +0x128
[2023-05-24T16:38:10.077131887Z][dockerd][I] github.com/docker/docker/api/server/middleware.VersionMiddleware.WrapHandler.func1({0x2504f40, 0x40015f8240}, {0x2503820, 0x40011d0000}, 0xffff5af948a0?, 0xffff83b48a68?)
[2023-05-24T16:38:10.077132054Z][dockerd][I] 	/go/src/github.com/docker/docker/api/server/middleware/version.go:62 +0x3e8
[2023-05-24T16:38:10.077141637Z][dockerd][I] github.com/docker/docker/pkg/authorization.(*Middleware).WrapHandler.func1({0x2504f40, 0x40015f8240}, {0x2503820?, 0x40011d0000?}, 0x4000814700, 0x2?)
[2023-05-24T16:38:10.077141887Z][dockerd][I] 	/go/src/github.com/docker/docker/pkg/authorization/middleware.go:59 +0x490
[2023-05-24T16:38:10.077144387Z][dockerd][I] github.com/docker/docker/api/server/middleware.DebugRequestMiddleware.func1({0x2504f40, 0x40015f8240}, {0x2503820, 0x40011d0000}, 0x4000814700, 0x4000aee040?)
[2023-05-24T16:38:10.077144554Z][dockerd][I] 	/go/src/github.com/docker/docker/api/server/middleware/debug.go:25 +0x4f8
[2023-05-24T16:38:10.077147595Z][dockerd][I] github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1({0x2503820, 0x40011d0000}, 0x4000814600)
[2023-05-24T16:38:10.077147762Z][dockerd][I] 	/go/src/github.com/docker/docker/api/server/server.go:123 +0x180
[2023-05-24T16:38:10.077152262Z][dockerd][I] net/http.HandlerFunc.ServeHTTP(0x4000814500?, {0x2503820?, 0x40011d0000?}, 0x800?)
[2023-05-24T16:38:10.077152387Z][dockerd][I] 	/usr/local/go/src/net/http/server.go:2122 +0x38
[2023-05-24T16:38:10.077155012Z][dockerd][I] github.com/docker/docker/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0x40001a7d40, {0x2503820, 0x40011d0000}, 0x4000814400)
[2023-05-24T16:38:10.077155387Z][dockerd][I] 	/go/src/github.com/docker/docker/vendor/github.com/gorilla/mux/mux.go:210 +0x19c
[2023-05-24T16:38:10.077158220Z][dockerd][I] net/http.serverHandler.ServeHTTP({0x24f4370?}, {0x2503820, 0x40011d0000}, 0x4000814400)
[2023-05-24T16:38:10.077158429Z][dockerd][I] 	/usr/local/go/src/net/http/server.go:2936 +0x2c0
[2023-05-24T16:38:10.077161345Z][dockerd][I] net/http.(*conn).serve(0x40016a8000, {0x2504f40, 0x4000b3df50})
[2023-05-24T16:38:10.077162137Z][dockerd][I] 	/usr/local/go/src/net/http/server.go:1995 +0x518
[2023-05-24T16:38:10.077171054Z][dockerd][I] created by net/http.(*Server).Serve
[2023-05-24T16:38:10.077171470Z][dockerd][I] 	/usr/local/go/src/net/http/server.go:3089 +0x4e8

[nicks]

you currently have the beta containerd integration turned on. does turning it off fix the issue?
https://docs.docker.com/desktop/containerd/

[Niek]

Yes, I can confirm that turning off containerd fixes the issue.

[rumpl]

@Niek note that in the next version of Docker Desktop the classic builder should work with containerd, we would love it if you would give it a try once it’s out

[thaJeztah]

@Niek 👋 out of curiosity; is there specific reason you need to use the classic builder? I know the plan is to sunset the classic builder at some point, so interested to learn what the main reason is for using it.

[Niek]

@rumpl Definitely interested in testing if the next version fixes this.

@thaJeztah That's a long story... It originates from this issue: httptoolkit/httptoolkit#423
See also: https://github.com/httptoolkit/httptoolkit-website/blob/main/src/docs/guides/docker.md#docker-buildkit-builds-cant-be-intercepted

[thaJeztah]

@Niek thanks for the extra info; I personally hadn't heard of that issue before (but I'm also not on the team maintaining BuildKit 😅)

Let me /cc @tonistiigi @crazy-max in case they're interested

[pimterry]

@rumpl @thaJeztah The above suggests this is resolved, but I've updated to the latest version (4.21.1) and it still fails when building the example here with the containerd beta enabled.

The error has now changed though, so the above repro now prints:

> DOCKER_BUILDKIT=0 docker build .
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM scratch
 --->
Step 2/2 : ENTRYPOINT ["fail"]
invalid reference format

That's as opposed to the Error response from daemon: Bad response from Docker engine message printed previously, as above.

The exact same build works fine with the containerd beta disabled.

Any ideas?

[pimterry]

out of curiosity; is there specific reason you need to use the classic builder?

Oh, and just for context: I maintain the software used by @Niek which integrates with Docker, and currently doesn't work with the containerd beta enabled due to this issue.

The main reason I'm using the classic builder in that integration is because BuildKit isn't supported by Dockerode (see apocas/dockerode#601, which blames docker/for-linux#1136 for this) so I can't use BuildKit even if I want to. That won't be unique to me though - it's a very widely used package for Docker integration.

[thaJeztah]

Ah, I guess there's an issue with special-casing scratch. I suspect it's trying to look up scratch as an actual image, and probably failing because of that.

Here's witch FROM scratch;

echo -e 'FROM scratch\nENTRYPOINT ["fail"]' | DOCKER_BUILDKIT=0 docker build -
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM scratch
 --->
Step 2/2 : ENTRYPOINT ["fail"]
invalid reference format

And here's FROM alpine (or any other image);

echo -e 'FROM alpine\nENTRYPOINT ["fail"]' | DOCKER_BUILDKIT=0 docker build -
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM alpine
7144f7bab3d4: Already exists
b312e4b0e2c6: Download complete
f6648c04cd6c: Download complete
 ---> 7144f7bab3d4
Step 2/2 : ENTRYPOINT ["fail"]
 ---> Running in d40d15a0a60c
Removing intermediate container d40d15a0a60c
 ---> b185bc886f29
Successfully built b185bc886f29

[thaJeztah]

I opened a tracking ticket in the Moby repository;

• containerd-integration: classic builder fails when building FROM scratch moby/moby#46248

[pimterry]

Ah, that makes sense. Perfect, thanks!

[dm17]

@Niek 👋 out of curiosity; is there specific reason you need to use the classic builder? I know the plan is to sunset the classic builder at some point, so interested to learn what the main reason is for using it.

I just updated docker compose and hit this too! Yes, for a very good reason: docker/buildx#175

In the docs it says To disable the containerd image store, clear the Use containerd for pulling and storing images checkbox. But how does one disable it if they're not using Docker Desktop?

If there isn't an easy workaround (like keeping a DOCKER_BUILDKIT=0 env var), then do you know the latest version of compose can I downgrade to in order to keep things working?

EDIT: OK, think there's some regression around DOCKER_BUILDKIT=0 being recognized from .env - but I was able to get it working by specifying it before docker compose on the cli.