Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot do cross-platform builds from Dockerfiles with RUN commands #13924

Closed
capnajax opened this issue Apr 20, 2022 · 22 comments
Closed

Cannot do cross-platform builds from Dockerfiles with RUN commands #13924

capnajax opened this issue Apr 20, 2022 · 22 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. macos MacOS (OSX) related remote Problem is in podman-remote stale-issue

Comments

@capnajax
Copy link

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I am using an arm-based Mac (M1 Ultra) to build an image for an Intel-based server. The Dockerfile has a RUN command in it. The build always fails when it reaches the RUN command.

Steps to reproduce the issue:

  1. Have a Dockerfile with a RUN command in it.
  2. From an M1-based Mac, build your image.
podman build --platform=linux/amd64 -t my_tag .
  1. Note failure at RUN step
STEP 6/20: RUN npm install -g npm
exec container process `/bin/sh`: Exec format error
Error: error building at STEP "RUN npm install -g npm": error while running runtime: exit status 1

Describe the results you received:

The image failed to build with an Exec format error, the same error I would expect if I deployed an arm container on an amd64 machine.

Describe the results you expected:

I should be able to build an image on my M1 machine.

Additional information you deem important (e.g. issue happens only occasionally):

This happens consistently

Output of podman version:

Client:       Podman Engine
Version:      4.0.3
API Version:  4.0.3
Go Version:   go1.18
Built:        Fri Apr  1 10:28:59 2022
OS/Arch:      darwin/arm64

Server:       Podman Engine
Version:      4.0.2
API Version:  4.0.2
Go Version:   go1.16.14
Built:        Thu Mar  3 08:58:50 2022
OS/Arch:      linux/arm64

Output of podman info --debug:

host:
  arch: arm64
  buildahVersion: 1.24.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.0-2.fc35.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: '
  cpus: 1
  distribution:
    distribution: fedora
    variant: coreos
    version: "35"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 501
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 5.15.18-200.fc35.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 1566859264
  memTotal: 2048237568
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.4.2-1.fc35.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.4.2
      commit: f6fbc8f840df1a414f31a60953ae514fa497c748
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/501/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc35.aarch64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 9m 23.19s
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 6
  runRoot: /run/user/501/containers
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 4.0.2
  Built: 1646319530
  BuiltTime: Thu Mar  3 08:58:50 2022
  GitCommit: ""
  GoVersion: go1.16.14
  OsArch: linux/arm64
  Version: 4.0.2

Package info (e.g. output of rpm -q podman or apt list podman):

n/a

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

No

Additional environment details (AWS, VirtualBox, physical, etc.):

@openshift-ci openshift-ci bot added the kind/bug Categorizes issue or PR as related to a bug. label Apr 20, 2022
@github-actions github-actions bot added macos MacOS (OSX) related remote Problem is in podman-remote labels Apr 20, 2022
@flouthoc
Copy link
Collaborator

Hi @capnajax , Thanks for creating the issue. Adding --platform=linux/amd64 in build command makes sure that your base image will be pulled for amd64 and your host arch is arm so binaries will get Exec format error. In order to resolve this you need to setup qemu_user_static and binfmt_misc see https://github.com/multiarch/qemu-user-static

In case you are only interested in pulling base image for arm but tagging final build as amd64 you can use inline --platform to override pull arch see examples here containers/buildah#3757 but I think you need the first one and i can tell this better if you can tell what is the intention of your build file.

@capnajax
Copy link
Author

capnajax commented Apr 20, 2022

Hi, @flouthoc thank you for responding so quickly and telling me about the qemu_user_static option. I will try that as soon as I can.

Yes, I am intentionally pulling an amd64 base image because I am targeting an Intel-based Kubernetes cluster, but I am working on an M1-based Mac Studio. The same command and Dockerfile, but using Docker, works without issue.

docker build --platform=linux/amd64 -t my_tag .
# works

I am assuming this has to do with the virtualization engine behind Docker/Podman as the only layers that should have to execute a command inside the container at build-time are all RUN commands. Perhaps one supports Rosetta or has a compatibility layer, and the other doesn't?

@afbjorklund
Copy link
Contributor

@rhatdan
Copy link
Member

rhatdan commented Apr 20, 2022

The problem is we don't have qemu-user-static installed on the fedora coreos machines by default. We need to make this happen automatically.

@github-actions
Copy link

A friendly reminder that this issue had no activity for 30 days.

@github-actions
Copy link

A friendly reminder that this issue had no activity for 30 days.

@rhatdan
Copy link
Member

rhatdan commented Jun 27, 2022

qemu-user-static is now partially installed on fcoreos, so this should work at least for building X86 from an M1 or Arm from an X86 machine.

@rhatdan rhatdan closed this as completed Jun 27, 2022
@credativ-dar
Copy link

Wow podman actually supports cross builds, nice work, it just needs some polishing. (I actually planned on ditching it and installing docker instead when I saw this error -.-)

Could podman just recognize that the qemu-user-static static packages are missing and hint users to install them? Or, alternatively, has a (soft?) dependency on this package been considered?

@rhatdan
Copy link
Member

rhatdan commented Jan 24, 2023

I could see us adding it as a suggests, but do you think this would help people discover it?

@credativ-dar
Copy link

credativ-dar commented Jan 25, 2023

It would need to be a recommends, a suggests wouldn't do a thing, if I understand this correctly. "Recommends" would install qemu-user-static like a "Requires" dependency by default but still allows users to ignore/uninstall it if they wish to save space. I personally prefer a "all batteries included" approach when I install packages, I have enough bandwidth and storage to spare for the occasional unused dependency. On the other hand, this may not be true for all users and many will probably only build images for their local machine and never need qemu-user-static.

Considering this I think the best approach would be to detect the missing package when it is needed and inform the user about this. sudo dnf install qemu-user-static is a quick fix once you know it, getting the above error message is misleading and frustrating.

@rhatdan
Copy link
Member

rhatdan commented Jan 25, 2023

qemu-user-static is a huge package, we (I) recently broke it in to multiple different packages.

qemu-user-static.i686 : QEMU user mode emulation of qemu targets static build
qemu-user-static.x86_64 : QEMU user mode emulation of qemu targets static build
======================== Name Matched: qemu-user-static ========================
qemu-user-static-aarch64.x86_64 : QEMU user mode emulation of aarch64 qemu
                                : targets static build
qemu-user-static-alpha.x86_64 : QEMU user mode emulation of alpha qemu targets
                              : static build
qemu-user-static-arm.x86_64 : QEMU user mode emulation of arm qemu targets
                            : static build
qemu-user-static-cris.x86_64 : QEMU user mode emulation of cris qemu targets
                             : static build
qemu-user-static-hexagon.x86_64 : QEMU user mode emulation of hexagon qemu
                                : targets static build
qemu-user-static-hppa.x86_64 : QEMU user mode emulation of hppa qemu targets
                             : static build
qemu-user-static-loongarch64.x86_64 : QEMU user mode emulation of loongarch64
                                    : qemu targets static build
qemu-user-static-m68k.x86_64 : QEMU user mode emulation of m68k qemu targets
                             : static build
qemu-user-static-microblaze.x86_64 : QEMU user mode emulation of microblaze qemu
                                   : targets static build
qemu-user-static-mips.x86_64 : QEMU user mode emulation of mips qemu targets
                             : static build
qemu-user-static-nios2.x86_64 : QEMU user mode emulation of nios2 qemu targets
                              : static build
qemu-user-static-or1k.x86_64 : QEMU user mode emulation of or1k qemu targets
                             : static build
qemu-user-static-ppc.x86_64 : QEMU user mode emulation of ppc qemu targets
                            : static build
qemu-user-static-riscv.x86_64 : QEMU user mode emulation of riscv qemu targets
                              : static build
qemu-user-static-s390x.x86_64 : QEMU user mode emulation of s390x qemu targets
                              : static build
qemu-user-static-sh4.x86_64 : QEMU user mode emulation of sh4 qemu targets
                            : static build
qemu-user-static-sparc.x86_64 : QEMU user mode emulation of sparc qemu targets
                              : static build
qemu-user-static-supported.x86_64 : QEMU user mode emulation of supported qemu
                                  : targets static build
qemu-user-static-x86.x86_64 : QEMU user mode emulation of x86 qemu targets
                            : static build
qemu-user-static-xtensa.x86_64 : QEMU user mode emulation of xtensa qemu targets
                               : static build

Perhaps we could recommends x86_64 and arch64 and then suggests the rest.

Pulling them all in by default seems way overkill.

@rhatdan
Copy link
Member

rhatdan commented Jan 25, 2023

@lsm5 WDYT?

@lsm5
Copy link
Member

lsm5 commented Jan 26, 2023

@rhatdan fedora-server (and I think also fedora-cloud) will likely ignore Recommends and Suggests. If it exists already on fcos, then we could have it as a Recommends on desktop and Requires on server.

I'm guessing Recommending (on desktop) / Requiring (on server) for all arches supported by fedora and suggests for the other arches, but I'm also fine with recommending only x86_64 and aarch64 and suggesting the rest.

@rhatdan
Copy link
Member

rhatdan commented Jan 26, 2023

Yes understood that fcos will ignore both, we have requested that they be added.

If we can differentiate desktop from server, that would be fine, although I am not sure the complexity is worth it.

@lsm5
Copy link
Member

lsm5 commented Jan 27, 2023

we already do it in containers-common, so should be ok I guess.

Recommends: crun
Requires: (crun if fedora-release-identity-server)
Recommends: netavark
Requires: (netavark if fedora-release-identity-server)
Recommends: slirp4netns
Requires: (slirp4netns if fedora-release-identity-server)

@rhatdan
Copy link
Member

rhatdan commented Jan 27, 2023

Nice.
Recommends qemu-user-static for workstation
Recommmends qemu-user-static-ARM and X86_64 for Server.
Suggests qemu-user-static for Server.

@lsm5
Copy link
Member

lsm5 commented Jan 31, 2023

Recommends and suggests will be ignored for server by default, so they have to be Requires, if at all.

This is what I'm thinking for now:

diff --git a/containers-common.spec b/containers-common.spec
index 40f1b46..52434d1 100644
--- a/containers-common.spec
+++ b/containers-common.spec
@@ -75,7 +75,10 @@ Requires: (slirp4netns if fedora-release-identity-server)
 Requires: iptables
 Requires: nftables
 Suggests: containernetworking-plugins >= 0.9.1-1
-Suggests: qemu-user-static
+Recommends: qemu-user-static
+Requires: (qemu-user-static-aarch64 if fedora-release-identity-server)
+Requires: (qemu-user-static-arm if fedora-release-identity-server)
+Requires: (qemu-user-static-x86 if fedora-release-identity-server)

@lsm5
Copy link
Member

lsm5 commented Jan 31, 2023

@rhatdan btw does qemu-user-static-x86 include x86_64? I don't see a separate package for x86_64.

@lsm5
Copy link
Member

lsm5 commented Jan 31, 2023

PTAL:
f37: https://bodhi.fedoraproject.org/updates/FEDORA-2023-20624cd618
f36: https://bodhi.fedoraproject.org/updates/FEDORA-2023-041092cd8f

@rhatdan
Copy link
Member

rhatdan commented Jan 31, 2023

I have no idea, but I would figure.

@credativ-dar
Copy link

Just tried it on Fedora 37 workstation: DNF shows all the arches als Depencencies and only the meta packages as weak dependency. Probably a bug in DNF showing weak dependency's dependencies in the wrong category. (Shall we report that?)

$ sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-20624cd618
[...]
Dependencies resolved.
==============================================================================================================================
 Package                                  Architecture        Version                      Repository                    Size
==============================================================================================================================
Upgrading:
 containers-common                        noarch              4:1-78.fc37                  updates-testing               86 k
 containers-common-extra                  noarch              4:1-78.fc37                  updates-testing              7.2 k
Installing dependencies:
 qemu-user-static-aarch64                 x86_64              2:7.0.0-13.fc37              updates                      4.2 M
 qemu-user-static-alpha                   x86_64              2:7.0.0-13.fc37              updates                      1.4 M
 qemu-user-static-arm                     x86_64              2:7.0.0-13.fc37              updates                      2.1 M
 qemu-user-static-cris                    x86_64              2:7.0.0-13.fc37              updates                      1.4 M
 qemu-user-static-hexagon                 x86_64              2:7.0.0-13.fc37              updates                      1.7 M
 qemu-user-static-hppa                    x86_64              2:7.0.0-13.fc37              updates                      1.4 M
 qemu-user-static-m68k                    x86_64              2:7.0.0-13.fc37              updates                      1.5 M
 qemu-user-static-microblaze              x86_64              2:7.0.0-13.fc37              updates                      1.7 M
 qemu-user-static-mips                    x86_64              2:7.0.0-13.fc37              updates                      3.8 M
 qemu-user-static-nios2                   x86_64              2:7.0.0-13.fc37              updates                      1.4 M
 qemu-user-static-or1k                    x86_64              2:7.0.0-13.fc37              updates                      1.4 M
 qemu-user-static-ppc                     x86_64              2:7.0.0-13.fc37              updates                      2.6 M
 qemu-user-static-riscv                   x86_64              2:7.0.0-13.fc37              updates                      2.0 M
 qemu-user-static-s390x                   x86_64              2:7.0.0-13.fc37              updates                      1.5 M
 qemu-user-static-sh4                     x86_64              2:7.0.0-13.fc37              updates                      1.7 M
 qemu-user-static-sparc                   x86_64              2:7.0.0-13.fc37              updates                      2.1 M
 qemu-user-static-x86                     x86_64              2:7.0.0-13.fc37              updates                      2.0 M
 qemu-user-static-xtensa                  x86_64              2:7.0.0-13.fc37              updates                      3.8 M
Installing weak dependencies:
 qemu-user-static                         x86_64              2:7.0.0-13.fc37              updates                       26 k

Transaction Summary
==============================================================================================================================
Install  19 Packages
Upgrade   2 Packages

I'm allowed to remove those, keeping only aarch, x86 and arm:

$ sudo dnf remove qemu-user-static
Dependencies resolved.
==============================================================================================================================
 Package                                   Architecture         Version                          Repository              Size
==============================================================================================================================
Removing:
 qemu-user-static                          x86_64               2:7.0.0-13.fc37                  @updates                45 k
Removing unused dependencies:
 qemu-user-static-alpha                    x86_64               2:7.0.0-13.fc37                  @updates               5.0 M
 qemu-user-static-cris                     x86_64               2:7.0.0-13.fc37                  @updates               5.0 M
 qemu-user-static-hexagon                  x86_64               2:7.0.0-13.fc37                  @updates               7.8 M
 qemu-user-static-hppa                     x86_64               2:7.0.0-13.fc37                  @updates               5.1 M
 qemu-user-static-m68k                     x86_64               2:7.0.0-13.fc37                  @updates               5.4 M
 qemu-user-static-microblaze               x86_64               2:7.0.0-13.fc37                  @updates                10 M
 qemu-user-static-mips                     x86_64               2:7.0.0-13.fc37                  @updates                36 M
 qemu-user-static-nios2                    x86_64               2:7.0.0-13.fc37                  @updates               5.1 M
 qemu-user-static-or1k                     x86_64               2:7.0.0-13.fc37                  @updates               5.1 M
 qemu-user-static-ppc                      x86_64               2:7.0.0-13.fc37                  @updates                20 M
 qemu-user-static-riscv                    x86_64               2:7.0.0-13.fc37                  @updates                12 M
 qemu-user-static-s390x                    x86_64               2:7.0.0-13.fc37                  @updates               5.7 M
 qemu-user-static-sh4                      x86_64               2:7.0.0-13.fc37                  @updates                10 M
 qemu-user-static-sparc                    x86_64               2:7.0.0-13.fc37                  @updates                16 M
 qemu-user-static-xtensa                   x86_64               2:7.0.0-13.fc37                  @updates                22 M

Transaction Summary
==============================================================================================================================
Remove  16 Packages

Freed space: 171 M

Great stuff, many thanks to both of you! ❤️

@lsm5
Copy link
Member

lsm5 commented Feb 1, 2023

@credativ-dar ack, thanks for testing and good to know it's working for you. Looks like hard dependencies of weak dependencies get listed as hard dependencies, I see the same on my side. I do agree those should get listed under weak dependencies. Anyway, we'll keep this dependency setup unless and until people complain :) .

@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 2, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 2, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. macos MacOS (OSX) related remote Problem is in podman-remote stale-issue
Projects
None yet
Development

No branches or pull requests

6 participants