Dockerfile should use cross-compile when building for a different arch when possible

[kaovilai]

What do you want to happen?

Dockerfile should use BUILDPLATFORM to cross build to reduce build time.

https://docs.docker.com/build/building/multi-platform/#cross-compilation

This is already seen used in docker-buildx target. We should refactor so that BUILDPLATFORM is used on all build Make targets, even for a single arch build.

Extra Labels

No response

[kaovilai]

This would inadvertently fix #4544 for me in a "benefits all default scaffold users" way without introducing maintenance overhead of introducing condition for a customized scaffold.

[camilamacedo86]

Hi @kaovilai,

Thank you very much for raising this issue and for your suggestions.

As we discussed in GitHub Issue #4544 and in Slack, Kubebuilder provides two separate build targets to meet different needs:

• docker-build → Designed for single-architecture builds (default).
• docker-buildx → Designed for multi-platform builds using buildx.

I do not think we can cannot accept your request to add BUILDPLATFORM to the default Dockerfile. Following are some reasons:

BUILDPLATFORM Is Only Required for Multi-Platform Builds (buildx)

• Standard docker build does not require BUILDPLATFORM because it builds for the local architecture by default.
• Adding BUILDPLATFORM would force cross-compilation unnecessarily, even when most users only need a local build.

The Default docker-build Target Is Designed for Simplicity

• The current make docker-build works out of the box without requiring BuildKit or buildx.

• Users can build their images without additional configuration:

  Makefile docker-build target

• Adding BUILDPLATFORM would require BuildKit, which is not always enabled by default.

BUILDPLATFORM Requires BuildKit and buildx, Which Are Not Always Available

• BUILDPLATFORM only works with buildx, which is not enabled by default in all Docker environments.
• Some setups do not support buildx out of the box, such as:
  • Legacy Docker environments without BuildKit.
  • CI/CD pipelines using standard docker build commands.
  • Older versions of Docker.

Multi-Platform Builds Are Already Supported Without Modifying docker-build or Dockerfile

• The docker-buildx target already correctly handles multi-platform builds:

  • It creates a temporary Dockerfile.cross with BUILDPLATFORM dynamically injected.
  • It ensures that BuildKit and buildx are enabled before using BUILDPLATFORM.
  • It provides flexibility—users who need multi-platform support can opt-in by running make docker-buildx.

  Makefile docker-buildx target

BUILDPLATFORM Can Actually Slow Down Native Builds

• Using BUILDPLATFORM in the default Dockerfile forces cross-compilation unnecessarily.
• When BUILDPLATFORM is set, Docker may attempt to use QEMU-based emulation, which adds overhead and slows down the build on certain systems.
• Example:
  • Native build on linux/amd64 (without BUILDPLATFORM) → Fast
  • Forcing BUILDPLATFORM=linux/amd64 on a linux/amd64 machine → Slower due to additional processing overhead
• Keeping BUILDPLATFORM optional avoids unnecessary slowdowns for users who only need native builds.

Enforcing BUILDPLATFORM Would Add Unnecessary Complexity to CI/CD Pipelines

• Most CI/CD environments use standard Docker setups without BuildKit enabled.
• Pipelines would require extra setup to enable buildx and BuildKit, increasing maintenance overhead.
• For example, enforcing BUILDPLATFORM would break our own E2E tests, which rely on native builds and do not require cross-compilation.

Cross-Compilation with BUILDPLATFORM Is Not Always a Best Practice

While BUILDPLATFORM can be useful in some cases, it is not always the best practice due to the following reasons:

• Single-Architecture Builds Do Not Need It
  If an application is only running on linux/amd64, adding BUILDPLATFORM adds unnecessary complexity. A simpler Dockerfile without it is preferred.

• Some Languages and Dependencies Do Not Support Cross-Compilation Well

  • Eexample: C/C++ libraries have architecture-specific depend

• It Adds Unnecessary Complexity for Local Development

  • For most developers, cross-compilation is not needed locally. A simple docker build without --platform is faster and more predictable.

By keeping BUILDPLATFORM optional, we allow users to choose whether they want to enable it while maintaining a simple and reliable default configuration. If your project requires BUILDPLATFORM, you are free to modify your Dockerfile to include it. However, for the community it does not seems the best approach.

Let us know if you have any further questions and if you agree that we can close this one as not accepted.

[kaovilai]

When BUILDPLATFORM is set, Docker may attempt to use QEMU-based emulation, which adds overhead and slows down the build on certain systems.

That's not what I've seen in practice with podman alongside my colleagues.
openshift/oadp-operator#1489

We have seen build speedup across the board.

When Dockerfile lacks BUILDPLATFORM, Docker attempts to use QEMU-based emulation for a different arch build for every stages.
When Dockerfile has BUILDPLATFORM, Docker only has to emulate the final stage when we copy binaries over from build stage which is fast because emulation is avoided.

[kaovilai]

My teammates uses podman everyday and do not have issues with make docker-build with BUILDPLATFORM.

[kaovilai]

BUILDPLATFORM Requires BuildKit and buildx, Which Are Not Always Available

podman also works here. :)

What is the support scope of kubebuilder? How far back of older systems does kubebuilder needs to support?

[kaovilai]

Eexample: C/C++ libraries have architecture-specific depend

does projects using kubebuilder uses C/C++ libs?

[camilamacedo86]

Hi @kaovilai,

If we adopt your request, wouldn’t that require all users to use buildx?
We can't enforce buildx for everyone, which is why we have a specific target for that.

• Users who need cross-compilation can use make docker-buildx.
• Those who don’t can simply run docker-build.

Could you please let me know if https://github.com/kubernetes-sigs/kubebuilder/pull/4545/files will be accepted? Would it address your need, or would you still be looking to change the Dockerfile?

Let me know your thoughts!

[kaovilai]

require all users to use buildx?

Not in my experience, at least if you use podman, you will not require buildx. I have not validated for legacy docker for non native arch build.

[kaovilai]

Could you please let me know if https://github.com/kubernetes-sigs/kubebuilder/pull/4545/files will be accepted? Would it address your need, or would you still be looking to change the Dockerfile?

I guess #4545 would be ok for me, but to your point it's specific to our customization.

At least from my team we would want to use BUILDPLATFORM everywhere including single arch because we spend a lot of time doing one of following

• dev on arm64: building images for amd64 cluster
• dev on amd64: building images for arm64 cluster

and when we give images to customers that's when we build for all arches.

From our experience even for a single arch build, we do not experience deficiencies in performance when BUILDPLATFORM is included.

And I have at least use this trick to fix an issue with CGO in replicatedhq/local-volume-provider#66
Although that may have more to do with their use of CGO_ENABLED=0 when go build along with inability for qemu emulation to build with said libraries.

replicatedhq/local-volume-provider#65 (comment)

92.91 internal/bytealg: /usr/local/go/pkg/tool/linux_amd64/asm: signal: segmentation fault (core dumped)
------
Dockerfile:23
--------------------
  21 |     COPY pkg ./pkg
  22 |     ARG VERSION=main
  23 | >>> RUN CGO_ENABLED=0 go build -ldflags=" -X github.com/replicatedhq/local-volume-provider/pkg/version.version=$VERSION " -o /go/bin/local-volume-fileserver ./cmd/local-volume-fileserver
  24 |     
  25 |     FROM debian:bullseye-slim
--------------------
ERROR: failed to solve: process "/dev/.buildkit_qemu_emulator /bin/sh -c CGO_ENABLED=0 go build -ldflags=\" -X github.com/replicatedhq/local-volume-provider/pkg/version.version=$VERSION \" -o /go/bin/local-volume-fileserver ./cmd/local-volume-fileserver" did not complete successfully: exit code: 1

When BUILDPLATFORM is added, the segmentation fault goes away.

[kaovilai]

I assume if kubebuilder adopts BUILDPLATFORM everywhere would be my preference but it is understandable if the expectation is to support C/C++ projects as part of kubebuilder default scaffold.

[kaovilai]

Here is a quote from my testing back in August (RH slack)

before adding BUILDPLATFORM

❯ docker system prune --all -f &&  time docker build -f Dockerfile.ubi . --platform=linux/amd64 
docker build -f Dockerfile.ubi . --platform=linux/amd64  1.89s user 3.26s system 2% cpu 3:21.93 total

after adding BUILDPLATFORM

❯ docker system prune --all -f &&  time docker build -f Dockerfile.ubi . --platform=linux/amd64 
docker build -f Dockerfile.ubi . --platform=linux/amd64  1.73s user 3.23s system 5% cpu 1:38.16 total

and I’m doing amd64 here cause my build system is arm.

My team saved a considerable amount of time building for single arch with BUILDPLATFORM.

So to conclude, if the preference is for code to be applicable to everyone, I would push for kubebuilder to use BUILDPLATFORM by default everywhere for multi-stage build unless support for other requirements are blockers.

I would agree to only keep these changes in our customized scaffolding otherwise.

[kaovilai]

If we adopt your request, wouldn’t that require all users to use buildx?

Legacy builder supports --platform without buildx
https://docs.docker.com/reference/cli/docker/build-legacy/

[kaovilai]

When BUILDPLATFORM is set, Docker may attempt to use QEMU-based emulation, which adds overhead and slows down the build on certain systems.

contrary to this statement, in docker docs,

the FROM instruction is pinned to the native platform of the builder (using the --platform=$BUILDPLATFORM option) to prevent emulation from kicking in.

https://docs.docker.com/build/building/multi-platform/#:~:text=the%20following%20example%2C-,the,option)%20to%20prevent%20emulation%20from%20kicking%20in.,-Then%20the%20pre