Issue with latest Docker hub arm64 image #9901

manalejandro · 2023-05-29T20:20:10Z

Checklist

This is a bug report, not a question. Ask questions on discuss.ipfs.io.
I have searched on the issue tracker for my bug.
I am running the latest kubo version or have an issue updating.

Installation method

ipfs-desktop

Version

No response

Config

https://hub.docker.com/layers/ipfs/kubo/latest/images/sha256-0b53c782c90a6486ced5411f29085b4aa784723fc75a21ac7223aab0206e1964

Description

Could be one architecture setting?

ipfs | exec /entrypoint.sh: exec format error

The text was updated successfully, but these errors were encountered:

lidel · 2023-06-02T14:18:05Z

@manalejandro what is the platform you are using for running this?
Are you able to test an image published before May 10 and see if the problem is still there?

manalejandro · 2023-06-03T02:17:26Z

@manalejandro what is the platform you are using for running this? Are you able to test an image published before May 10 and see if the problem is still there?

I have it deployed on one Pi4, something strange happened, i download this image from the hub and it doesn't exist.

REPOSITORY                             TAG             IMAGE ID                                                                  CREATED        SIZE
ipfs/kubo                              master-latest   sha256:2f93a1c01b4cb8aab338fde8cbcd04e216b48a2d1dcc33d1065e9d5d05df736b   9 hours ago    76.6MB

manalejandro · 2023-06-03T21:43:19Z

Downloading the correct image the same error occurs when executing the IPFS command, it smells to me that it is the architecture in the CI.

$ docker pull ipfs/kubo@sha256:d20c43407cd520f01bbe5b692f18d3287a8264a875ac76d542d23330a14b913e
$ docker tag ipfs/kubo@sha256:d20c43407cd520f01bbe5b692f18d3287a8264a875ac76d542d23330a14b913e ipfs/kubo:master-latest

ipfs  | exec /sbin/tini: exec format error
ipfs exited with code 0

lidel · 2023-06-05T13:46:24Z

@manalejandro Are you able to test an image for v0.19.2 and see if the problem is still there?

manalejandro · 2023-06-06T02:33:20Z

@manalejandro Are you able to test an image for v0.19.2 and see if the problem is still there?

that image works well.

ipfs  | ipfs version 0.19.2
ipfs  | Found IPFS fs-repo at /data/ipfs
ipfs  | Initializing daemon...

manalejandro · 2023-06-14T08:50:29Z

If I build the docker image with the latest version it works perfectly, could be some parameter that affects the architecture when building it for image deployment.

ipfs  | ipfs version 0.21.0-dev
ipfs  | Found IPFS fs-repo at /data/ipfs
ipfs  | Initializing daemon...
ipfs  | Kubo version: 0.21.0-dev-61f8c73
ipfs  | Repo version: 13
ipfs  | System version: arm64/linux
ipfs  | Golang version: go1.19.1

lidel · 2023-06-29T14:09:49Z

@manalejandro interesting, how are you building the image? docker build -t kubo-test . ?
Did you docker pull golang:1.19-buster and docker pull busybox:1.31.1-glibc beforehand, just to be sure you use same tags as CI?

manalejandro · 2023-06-29T21:12:37Z

@manalejandro interesting, how are you building the image? docker build -t kubo-test . ? Did you docker pull golang:1.19-buster and docker pull busybox:1.31.1-glibc beforehand, just to be sure you use same tags as CI?

Hi, I cross compile the last commit with docker from a x64 machine for the Pi4 arm64 image:

$ docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

$ git clone https://github.com/ipfs/kubo && cd kubo

$ docker buildx build --platform arm64 -t ipfs .

The last commit I tried was 61f8c73 but now it doesn't work with the latest code from github.

manalejandro · 2023-07-02T02:51:06Z

@manalejandro interesting, how are you building the image? docker build -t kubo-test . ? Did you docker pull golang:1.19-buster and docker pull busybox:1.31.1-glibc beforehand, just to be sure you use same tags as CI?

Hi, I cross compile the last commit with docker from a x64 machine for the Pi4 arm64 image:
$ docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

$ git clone https://github.com/ipfs/kubo && cd kubo

$ docker buildx build --platform arm64 -t ipfs .
The last commit I tried was 61f8c73 but now it doesn't work with the latest code from github.

Sorry I'm very rude and didn't look at the Dockerfile arguments, now the last commit works.

$ docker buildx build --build-arg "BUILDPLATFORM=linux/arm64" --build-arg "TARGETPLATFORM=linux/arm64" --build-arg "TARGETOS=linux" --build-arg "TARGETARCH=arm64" --platform arm64 -t 192.168.1.124:5000/ipfs .

Kubo version: 0.22.0-dev-f797e9e

lidel · 2023-07-04T13:44:12Z

@manalejandro ack. Just want to confirm: you are building your own image because the upstream one does not work, right?

manalejandro · 2023-07-04T13:47:45Z

@manalejandro ack. Just want to confirm: you are building your own image because the upstream one does not work, right?

Yes, that's the point of my issue.

pablomendezroyo · 2023-07-13T11:36:21Z

same issue

Marketen · 2023-07-13T11:45:50Z

With docker versions v0.20.0 & v0.21.0 I have the same issue. When building an image on an arm64 device with these versions, a docker inspect <image-tag> returns "Architecture": "arm64", the expected architecture.

However, when executing docker run --rm --entrypoint="" <image-tag> uname -m to get current architecture, the docker container built with these images retuns x86_64.

This does not happen on versions v0.19.0 and lower. where the same command docker run --rm --entrypoint="" <image-tag> uname -m returns the expected architecture on the container build with arm64: aarch64

kallisti5 · 2023-07-13T14:42:04Z

Same issue here. 0.19.1, 0.19.2 work ok. 0.20.0 and 0.21.0 both error out like you packed your arm64 build in a x86 container.

{"msg":"exec container process `/sbin/tini`: Exec format error","level":"error","time":"2023-07-13T14:38:22.392159Z"}

See jesec/flood#451 (comment) for a similar issue.

More data:

pi:~# uname -a
Linux pi 5.15.93-0-rpi4 #1-Alpine SMP PREEMPT Thu Feb 9 23:56:17 UTC 2023 aarch64 Linux

pi:~# podman run -it --entrypoint /bin/sh docker.io/ipfs/kubo:v0.20.0
{"msg":"exec container process `/bin/sh`: Exec format error","level":"error","time":"2023-07-13T14:46:12.600165Z"}

pi:~# podman run -it --entrypoint ipfs docker.io/ipfs/kubo:v0.20.0
USAGE
  ipfs  - Global p2p merkle-dag filesystem.

  ipfs [--config=<config> | -c] [--debug | -D] [--help] [-h] [--api=<api>] [--offline] [--cid-base=<base>] [--upgrade-cidv0-in-output] [--encoding=<encoding> | --enc] [--timeout=<timeout>] <command> ...

SUBCOMMANDS
  BASIC COMMANDS
    init          Initialize local IPFS configuration
    add <path>    Add a file to IPFS
    cat <ref>     Show IPFS object data
    get <ref>     Download IPFS objects
    ls <ref>      List links from an object
    refs <ref>    List hashes of links from an object

The above confirms that the chroot is all x86_64 or something... and the ipfs binary itself is arm64. If you're doing some multi-stage built, your final step is using an x86_64/amd64 container to install the arm64 kubo into.

kallisti5 · 2023-07-13T14:52:25Z

suspect: 6f08cde

Maybe your caching image layers means it's caching x86 layers and not figuring out the arch is different? 3 minutes seems way to quick to pull multiple base images of differing architectures.

Also of note:
6f08cde#diff-dd2c0eb6ea5cfc6c4bd4eac30934e2d5746747af48fef6da689e85b752f39557R54

Maybe the target image is getting the default arm64 because BUILDPLATFORM is unset?

samsapti · 2023-07-15T21:58:36Z

Also of note:
6f08cde#diff-dd2c0eb6ea5cfc6c4bd4eac30934e2d5746747af48fef6da689e85b752f39557R54

Maybe the target image is getting the default arm64 because BUILDPLATFORM is unset?

@kallisti5 you're on to something, but it's not the caching that's causing the problem. It's this:

--platform=${BUILDPLATFORM:-linux/amd64}

It's not because BUILDPLATFORM is unset, it's set. It's because it's the wrong variable that's used. BUILDPLATFORM is the platform that the image is built ON, while TARGETPLATFORM is the platform the image is built FOR. So BUILDPLATFORM is always set to linux/amd64, and the variable that should be used in the final stage is actually TARGETPLATFORM, like so:

FROM --platform=${TARGETPLATFORM:-linux/amd64} busybox:1.31.1-glibc

kallisti5 · 2023-07-18T13:55:42Z

AAH. Yup. I missed that lol. You're compiling on BUILDPLATFORM for TARGETPLATFORM.. so the installed container needs to be TARGETPLATFORM instead of BUILDPLATFORM

deavmi · 2023-07-27T17:51:32Z

With docker versions v0.20.0 & v0.21.0 I have the same issue. When building an image on an arm64 device with these versions, a docker inspect <image-tag> returns "Architecture": "arm64", the expected architecture.

However, when executing docker run --rm --entrypoint="" <image-tag> uname -m to get current architecture, the docker container built with these images retuns x86_64.

This does not happen on versions v0.19.0 and lower. where the same command docker run --rm --entrypoint="" <image-tag> uname -m returns the expected architecture on the container build with arm64: aarch64

God help me, I am testing this now because I have the same issue on latest

lidel · 2023-07-27T18:06:56Z

Upstream image was fixed by #10021 today.
Once we release Kubo 0.22, the upstream latest image will be fixed too.

Your options here:

(A) build image yourself until Kubo 0.22 is released
(B) use the master- dev images until kubo 0.22 is released
- starting with ipfs/kubo:master-2023-07-27-c5868a8, it works fine on my rPI (arm64):

deavmi · 2023-07-27T18:08:39Z

Upstream image was fixed by #10021 today. Once we release Kubo 0.22, the upstream latest image will be fixed too.

For now, you can use the master-latest, starting with ipfs/kubo:master-2023-07-27-c5868a8, it works fine on my rPI (arm64):

Thank you, this seems to work as well!

(A little off tangent question, where's the ipfs config file created within this image?)

lidel · 2023-07-27T18:12:39Z

@deavmi IPFS_HOME is at /data/ipfs but you may want to customize your instance via scripts in /container-init.d instead – see https://docs.ipfs.tech/install/run-ipfs-inside-docker/#customizing-your-node
(In the future, ask support questions like this at https://discuss.ipfs.tech/c/help/13 🙏 )

manalejandro added kind/bug A bug in existing code (including security flaws) need/triage Needs initial labeling and prioritization labels May 29, 2023

lidel added this to IPFS Shipyard Team Jun 2, 2023

lidel added the need/author-input Needs input from the original author label Jun 2, 2023

lidel self-assigned this Jun 12, 2023

lidel mentioned this issue Jun 29, 2023

ci: fix platform of the target docker image #9861

Closed

aschmahmann added P1 High: Likely tackled by core team if no one steps up and removed need/triage Needs initial labeling and prioritization need/author-input Needs input from the original author labels Jul 10, 2023

Marketen mentioned this issue Jul 13, 2023

Add migration depending on architecture dappnode/DNP_IPFS#105

Closed

lidel mentioned this issue Jul 27, 2023

ci: simplify Dockerfile and add docker image testing #10021

Merged

1 task

lidel closed this as completed Jul 27, 2023

github-project-automation bot moved this to 🎉 Done in IPFS Shipyard Team Jul 27, 2023

TheColorman mentioned this issue Aug 4, 2023

Exec format error when building from source from docker image #10038

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue with latest Docker hub arm64 image #9901

Issue with latest Docker hub arm64 image #9901

manalejandro commented May 29, 2023

lidel commented Jun 2, 2023 •

edited

Loading

manalejandro commented Jun 3, 2023

manalejandro commented Jun 3, 2023

lidel commented Jun 5, 2023

manalejandro commented Jun 6, 2023

manalejandro commented Jun 14, 2023

lidel commented Jun 29, 2023

manalejandro commented Jun 29, 2023

manalejandro commented Jul 2, 2023

lidel commented Jul 4, 2023

manalejandro commented Jul 4, 2023

pablomendezroyo commented Jul 13, 2023

Marketen commented Jul 13, 2023 •

edited

Loading

kallisti5 commented Jul 13, 2023 •

edited

Loading

kallisti5 commented Jul 13, 2023 •

edited

Loading

samsapti commented Jul 15, 2023 •

edited

Loading

kallisti5 commented Jul 18, 2023

deavmi commented Jul 27, 2023

lidel commented Jul 27, 2023 •

edited

Loading

deavmi commented Jul 27, 2023

lidel commented Jul 27, 2023

Issue with latest Docker hub arm64 image #9901

Issue with latest Docker hub arm64 image #9901

Comments

manalejandro commented May 29, 2023

Checklist

Installation method

Version

Config

Description

lidel commented Jun 2, 2023 • edited Loading

manalejandro commented Jun 3, 2023

manalejandro commented Jun 3, 2023

lidel commented Jun 5, 2023

manalejandro commented Jun 6, 2023

manalejandro commented Jun 14, 2023

lidel commented Jun 29, 2023

manalejandro commented Jun 29, 2023

manalejandro commented Jul 2, 2023

lidel commented Jul 4, 2023

manalejandro commented Jul 4, 2023

pablomendezroyo commented Jul 13, 2023

Marketen commented Jul 13, 2023 • edited Loading

kallisti5 commented Jul 13, 2023 • edited Loading

kallisti5 commented Jul 13, 2023 • edited Loading

samsapti commented Jul 15, 2023 • edited Loading

kallisti5 commented Jul 18, 2023

deavmi commented Jul 27, 2023

lidel commented Jul 27, 2023 • edited Loading

deavmi commented Jul 27, 2023

lidel commented Jul 27, 2023

lidel commented Jun 2, 2023 •

edited

Loading

Marketen commented Jul 13, 2023 •

edited

Loading

kallisti5 commented Jul 13, 2023 •

edited

Loading

kallisti5 commented Jul 13, 2023 •

edited

Loading

samsapti commented Jul 15, 2023 •

edited

Loading

lidel commented Jul 27, 2023 •

edited

Loading