DOCKER_BUILDKIT=1 prevents custom networks using docker build --network

[jhrabi]

Not sure if this is a Docker CLI or BuildKit issue, but when using BuildKit by specifying DOCKER_BUILDKIT=1 as part of a Docker Build in Docker v18.09, a custom Docker network is no longer recognized.

So for example, a command of "docker build --network demo-net ." will now return:

Error response from daemon: network mode "demo-net" not supported by buildkit

Seemingly this means that it only supports specific network modes are being supported, and not custom Docker networks.

[AkihiroSuda]

cc @kunalkushwaha

[antifuchs]

I have just stumbled across this too (running an apt_cache container locally to support my builds); believe this is intentional: this conditional pretty much says that no network modes other than host, none, or the default are supported /:

[MarcosMorelli]

im also looking at this one. I want to build and add to a custom network along with mongodb to run integration tests;

[crevao]

I'm also very interested in this!

[MarcosMorelli]

any updates at this one?

[snakenstein]

I faced this issue too when running build in Swarm from CI server. We attach to network where container with private Nuget server resides to pull required packages.
And because of this limitation we can't use buildkit for such builds.

[barrelmaker97]

Have run into this issue as well trying to build an image that uses a Redis instance for testing during the build process. Any updates?

[ghost]

I have just stumbled across this too (running an apt_cache container locally to support my builds); believe this is intentional: this conditional pretty much says that no network modes other than host, none, or the default are supported /:

I suspect there just wasn't time/focus to implement the feature, it's marked as a roadblock item in the comment on moby/moby#40379 - if it's removed it would be a definite functional regression.

[SuperSandro2000]

This is a real blocker for me cause I cannot access my DNS server in another container while building.

[kanduvisla]

Any updates or workaround on this one?

[AkihiroSuda]

workaround: export DOCKER_BUILDKIT=0

[kanduvisla]

@AkihiroSuda that's not really a workaround since then you cannot use the features provided by docker buildkit. Like secrets, in my case.

[winseros]

This affects us as we use lots of auxiliary containers (SQL Server, Kafka, Redis) to run integration tests during builds.

Without --network, we will have no ways to use the BuildKit, and we'll have to stick to the old build model.

[RytisLT]

Since BuildKit is now enabled in docker by default we need a solution for this issue, can't use BuildKit without it :(

[MT-Jacobs]

@RytisLT is that true? I don't see anything indicating that, and moby/moby#40379 seems to say that BuildKit still isn't the default.

[RytisLT]

@RytisLT yes and no. It's already the default for new installations (or after factory reset) on windows. Source: docker.com

[lucastheisen]

Another workaround, while not ideal, is to --publish any services you needed from the --network and use the host.docker.internal URL from your build to access it:

docker build --tag artifacts - <<EOF
FROM busybox
RUN set -e; \
    mkdir --parents /opt/artifact; \
    echo "hello world" > /opt/artifact/hello;
EXPOSE 8080
ENTRYPOINT ["httpd", "-f", "-h", "/opt/artifact", "-p", "8080"]
EOF

docker run --detach --name artifact-repo --rm --publish-all artifacts
port="$(
  docker inspect \
    --format '{{(index (index .NetworkSettings.Ports "8080/tcp") 0).HostPort}}' \
    artifact-repo)"

docker build --tag tester - <<EOF
FROM busybox
RUN wget --output-document - http://host.docker.internal:${port}/hello
EOF

docker kill artifact-repo

Gives

Error response from daemon: Cannot kill container: artifact-repo: No such container: artifact-repo
Sending build context to Docker daemon  2.048kB
Step 1/4 : FROM busybox
 ---> 219ee5171f80
Step 2/4 : RUN set -e;     mkdir --parents /opt/artifact;     echo "hello world" > /opt/artifact/hello;
 ---> Using cache
 ---> 965724e107bd
Step 3/4 : EXPOSE 8080
 ---> Using cache
 ---> e39c7128ebca
Step 4/4 : ENTRYPOINT ["httpd", "-f", "-h", "/opt/artifact", "-p", "8080"]
 ---> Using cache
 ---> 7102e8ab2186
Successfully built 7102e8ab2186
Successfully tagged artifacts:latest
4696cbdd65268a55cb34ae450a8ec1e51c91bf1eaa5511ac0f4f4e515b34a958
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM busybox
 ---> 219ee5171f80
Step 2/2 : RUN wget --output-document - http://host.docker.internal:32774/hello
 ---> Running in 20c2229c7de7
Connecting to host.docker.internal:32774 (192.168.65.2:32774)
hello world
writing to stdout
-                    100% |********************************|    12  0:00:00 ETA
written to stdout
Removing intermediate container 20c2229c7de7
 ---> 96c004a8abab
Successfully built 96c004a8abab
Successfully tagged tester:latest
artifact-repo

[Chealer]

@RytisLT yes and no. It's already the default for new installations (or after factory reset) on windows. Source: docker.com

To clarify, that comment refers to the following Docker for Windows 2.4.0.0 change:

Docker Desktop now enables BuildKit by default after a reset to factory defaults. To revert to the old docker build experience, go to Preferences > Docker Engine and then disable the BuildKit feature.

However, as of version 3.2, Preferences is called Settings and this setting has no real GUI. Basically, one needs to add the following to the configuration field:

  "features": {
    "buildkit": false
  }

[Chealer]

Considering that DOCKER_BUILDKIT=1 is now default as indicated above, it may be best to retitle this ticket to just:

Buildkit prevents custom networks using docker build --network

Or, as this tracker is specific to Buildkit, just:

Prevents custom networks using docker build --network

While we're at it, I do not use docker build and still experience this (using docker-compose and a docker-compose.yml file which specifies bridge as a network). So this could be even more generic, like:

Does not support bridge and more network types

It would also mitigate to improve the error message pointing to this ticket or at least clarifying what BuildKit is, or that BuildKit is optional.

[tonistiigi]

If you want to use custom docker network with buildkit use docker buildx create with --driver-opt network=customname that will create a buildkit instance in docker container that is part of that network. https://github.com/docker/buildx/blob/master/docs/reference/buildx_create.md

Buildkit does not depend on libnetwork, and can't see these overlay networks that are internal to docker. Instead deploy Buildkit to the same network with docker to have this behavior.

[dm17]

I don't understand this... It seems to work fine on most systems, and as I understand it we aren't using any custom "network mode." It is calling the name of our external network (in the docker-compose.yml) a "network mode" - but is it really a network mode? Setting DOCKER_BUILDKIT=0 in .env doesn't have an effect. Running DOCKER_BUILDKIT=0 docker-compose up -d does fix it. I couldn't reproduce this "network mode" error with buildkit on any Linux systems - only on Mac. This made me notice the only other thread I can find on this issue is by a Mac user: https://giters.com/ibm-messaging/mq-container/issues/473

My guess is this is due to the build-time networking this needs. So does this mean no build-time networking is allowed with buildkit, or I'm just not allowed to have a custom name for such a network?

@tonistiigi I'm not clear on how that could be done in a docker-compose.yml - can it?

[sudo-bmitch]

@dm17 it's not clear your issue is related to building an image with a custom network. Can you provide an example, commands, and output of the failing build?

[TBBle]

It was closed because BuildKit itself doesn't deal with the networking, per #978 (comment) and docker/buildx#175 (comment) you create a buildx instance attached to the desired custom network, and the BuildKit running in that will see your custom network.

[dm17]

It was closed because BuildKit itself doesn't deal with the networking, per #978 (comment) and [docker/buildx#175 (comment)]

So it is moby that needs to support the networking that allows for this, not buildkit?

(docker/buildx#175 (comment)) you create a buildx instance attached to the desired custom network, and the BuildKit running in that will see your custom network.

That doesn't seem to work for me. After creating the builder with the --driver-opt option and --use'ing it (required for the work around you state), buildx bake still says ERROR: network mode "custom" not supported by buildkit.
I stated this in the buildx repo as well: docker/buildx#175

[TBBle]

So it is moby that needs to support the networking that allows for this, not buildkit?

Yes. BuildKit is just a daemon, and the networking is supplied by the container it runs in. buildx create creates such a container, and can hence be attached to a custom network like any container running in Docker. BuildKit itself doesn't know or care about the details of the network configuration beyond of the container it runs in.

That doesn't seem to work for me. After creating the builder with the --driver-opt option and --use'ing it (required for the work around you state), buildx bake still says ERROR: network mode "custom" not supported by buildkit.

I just followed up on buildx ticket where you mentioned this; I suggest a more-detailed repro, because buildx bake should not actually be seeing a network mode at all, it should be driving the build though buildx instances; but there might be a bug to be fixed.

[dm17]

So it is moby that needs to support the networking that allows for this, not buildkit?

Yes. BuildKit is just a daemon, and the networking is supplied by the container it runs in.

Sweet, so this limitation of "no custom networks during buildtime now that docker uses buildkit" isn't necessary? :)

[TBBle]

The limitation is docker buildx build doesn't create instances, so you can't specify the networking configuration at that time. The error message from buildx could possibly be clearer, as it doesn't really direct you towards what to do about it, which I recall being an issue for a few other "You need to create a custom instance" behaviour-changes.

[Hronom]

Relates docker/buildx#175

[B3nsten]

workaround: export DOCKER_BUILDKIT=0

This solution is 4 years old and still the only way I get my build process connected to my postgres container without exposing my database in production.

[TBBle]

I described a flow in docker/buildx#175 (comment) which should work (it worked at the time, anyway), and due to the way buildx works, any other solution to build inside a temporary custom network is going to be of roughly the same shape: docker buildx create then docker buildx build then docker buildx rm.

[dm17]

and due to the way buildx works

Which mean one cannot use docker compose files - which means the code that runs and builds the containers will diverge. It seems like this is worth moving to k3s (or similar) over.

[TBBle]

I'm not sure what you mean by not being able to use docker compose files. If this is the combination of Docker compose and docker buildx bake, then there's a working example at docker/buildx#175 (comment) but at this time I haven't seen a working example of how it works without buildx, so I don't know if I was solving the correct problem.

Anyway, this is still not a BuildKit issue, but a buildx issue (since we're talking about behaviours of docker buildx build and docker buildx bake, and docker/buildx#175 is the right place for that.

[dm17]

I'm not sure what you mean by not being able to use docker compose files

Can't use docker compose files without buildx using buildkit. The point is that they caused all this divergence in our code paths for reasons they will not explain. It isn't good engineering in my opinion. If there's a reason buildkit musn't ever have built-time networking, then explain why exactly. If it can, then we can go back to the breadth of functionality we enjoyed with the classic builder and forego all of this complexity and divergence.

[TBBle]

Can't use docker compose files

To do what? Please show an example of what used to work for you, i.e. Compose file etc. and the commands you run to "use it".

Right now I'm merely guessing what you're actually trying to do.

Since my last guess was able to use custom networks without issues, but you're still having problems, clearly my guess was wrong and I need more details on your use-case before I can try and resolve it.

---

If there's a reason buildkit musn't ever have built-time networking

Buildkit doesn't ever have built-in networking because it has no connection to the networking. It executes containers for builds with the networking configuration that is provided to it, either by buildx/Docker, or by some other container where buildkitd is running.

This has been explained multiple times in this ticket, and in docker/buildx#175, by both project leads and also me.

Buildkit does not depend on libnetwork, and can't see these overlay networks that are internal to docker.

BuildKit is just a daemon, and the networking is supplied by the container it runs in. buildx create creates such a container, and can hence be attached to a custom network like any container running in Docker. BuildKit itself doesn't know or care about the details of the network configuration beyond of the container it runs in.

The limitation is docker buildx build doesn't create instances, so you can't specify the networking configuration at that time.

Also, see docker/buildx#175 (comment) and in particular the final part where the setup of the in-Docker integrated version of buildkit (i.e. when it's not a daemon) is described, and how it differs from the builder which was just a direct part of Docker.

[B3nsten]

Well, what I completely overlooked: While the builder may be running in the custom network, the build process itself still does not. For my case, I was only missing the --network=host on the buildx build. docker/buildx#2306 (comment) explicitly hinted that, although it is present in docker/buildx#175 (comment)