Build failure on aarch64-unkown-linux-musl cross compile

[recmo]

Version

#14 157.5 └── ethers v0.5.4
#14 157.5     ├── ethers-contract v0.5.4
#14 157.5     │   ├── ethers-contract-abigen v0.5.4
#14 157.5     │   │   ├── ethers-core v0.5.5
#14 157.5     │   ├── ethers-contract-derive v0.5.4 (proc-macro)
#14 157.5     │   │   ├── ethers-contract-abigen v0.5.4 (*)
#14 157.5     │   │   ├── ethers-core v0.5.5
#14 157.5     │   ├── ethers-core v0.5.5 (*)
#14 157.5     │   ├── ethers-providers v0.5.5
#14 157.5     │   │   ├── ethers-core v0.5.5 (*)
#14 157.5     ├── ethers-core v0.5.5 (*)
#14 157.5     ├── ethers-middleware v0.5.4
#14 157.5     │   ├── ethers-contract v0.5.4 (*)
#14 157.5     │   ├── ethers-core v0.5.5 (*)
#14 157.5     │   ├── ethers-providers v0.5.5 (*)
#14 157.5     │   ├── ethers-signers v0.5.4
#14 157.5     │   │   ├── ethers-core v0.5.5 (*)
#14 157.5     ├── ethers-providers v0.5.5 (*)
#14 157.5     └── ethers-signers v0.5.4 (*)

Platform
Docker linux/arm64 rust:latest cross compile to aarch64-unkown-linux-musl

uname -a: Linux buildkitsandbox 5.10.47-linuxkit #1 SMP PREEMPT Sat Jul 3 21:50:16 UTC 2021 aarch64 GNU/Linux

Description

Dockerfile

FROM rust
WORKDIR /src

RUN apt-get update &&\
    apt-get install -y texinfo &&\
    apt-get clean && rm -rf /var/lib/apt/lists/* &&\
    cargo install cargo-tree &&\
    rustup target add $(uname -m)-unknown-linux-musl

# Build {x86_64,aarch64}-linux-musl toolchain
# This is required for some build.rs scripts
ARG MUSL_CROSS_VERSION=0.9.9
RUN curl -fL "https://github.com/richfelker/musl-cross-make/archive/v${MUSL_CROSS_VERSION}.tar.gz"\
    | tar xz && cd musl-cross-make-${MUSL_CROSS_VERSION} &&\
    make install TARGET=$(uname -m)-linux-musl OUTPUT=/usr/local/musl &&\
    rm -r /src/musl-cross-make-${MUSL_CROSS_VERSION}
ENV CC_x86_64_unknown_linux_musl=/usr/local/musl/bin/x86_64-linux-musl-gcc
ENV CC_aarch64_unknown_linux_musl=/usr/local/musl/bin/aarch64-linux-musl-gcc

# Build zlib
ARG ZLIB_VERSION=1.2.11
RUN curl -fL "http://zlib.net/zlib-$ZLIB_VERSION.tar.gz" | tar xz && cd "zlib-$ZLIB_VERSION" &&\
    export CC=/usr/local/musl/bin/$(uname -m)-linux-musl-gcc &&\
    ./configure --static --prefix=/usr/local/musl && make && make install &&\
    rm -r "/src/zlib-$ZLIB_VERSION"

# Build OpenSSL
ARG OPENSSL_VERSION=1.1.1l
RUN curl -fL "https://www.openssl.org/source/openssl-$OPENSSL_VERSION.tar.gz" | tar xz &&\
    cd "openssl-$OPENSSL_VERSION" &&\
    export CC=/usr/local/musl/bin/$(uname -m)-linux-musl-gcc &&\
    ./Configure no-shared --prefix=/usr/local/musl linux-$(uname -m) &&\
    make install_sw &&\
    rm -r "/src/openssl-$OPENSSL_VERSION"
ENV OPENSSL_DIR=/usr/local/musl
ENV OPENSSL_STATIC=1

RUN cargo init
RUN echo 'ethers = "0.5"' >> ./Cargo.toml
RUN uname -a
RUN cargo tree | grep ethers
RUN cargo run --target $(uname -m)-unknown-linux-musl

docker build --platform linux/arm64 --progress plain .

#13 87.85    Compiling hyper-tls v0.5.0
#13 88.04    Compiling reqwest v0.11.6
#13 88.72    Compiling ethers-signers v0.5.4
#13 89.44    Compiling ethers-providers v0.5.5
#13 89.49    Compiling ethers-contract-abigen v0.5.4
#13 93.30    Compiling ethers-contract-derive v0.5.4
#13 97.54    Compiling ethers-contract v0.5.4
#13 97.84 error: /src/target/debug/deps/libethers_contract_derive-d00aabe3fae08b23.so: undefined symbol: sigsetjmp
#13 97.84   --> /usr/local/cargo/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/ethers-contract-0.5.4/src/lib.rs:53:9
#13 97.84    |
#13 97.84 53 | pub use ethers_contract_derive::{abigen, EthAbiType, EthEvent};
#13 97.84    |         ^^^^^^^^^^^^^^^^^^^^^^
#13 97.84 
#13 97.87 error: could not compile `ethers-contract` due to previous error
#13 ERROR: executor failed running [/bin/sh -c cargo run --target $(uname -m)-unknown-linux-musl]: exit code: 101
------
 > [9/9] RUN cargo run --target $(uname -m)-unknown-linux-musl:
------
executor failed running [/bin/sh -c cargo run --target $(uname -m)-unknown-linux-musl]: exit code: 101

It's probably a dependency with a broken build.rs, I'll dig into the (large!) dependency tree and see what is the root cause. It's pretty funny to see sigsetjmp pop up here though :)

The x86_64 equivalent builds fine. (docker build --platform linux/amd64 --progress plain .)

[recmo]

So far I've narrowed it down to the reqwest dependency in ethers-contract-abigen which is compiled as a proc-macro.

reqwest has a ton of transitive dependencies, including C libs like openssl. For proc-macro usage they need to be compiled native and not cross. Likely something going wrong there. I'll dig a bit more, but one solution would be to make reqwest an optional dependency behind a feature flag. Making http calls from proc-macros is a bit sketchy anyway.

[recmo]

Yes, the issue was with openssl-sys being compiled both for the host (for the proc-macro) and for the target. Thes require different installations, but both used the one specified in the OPENSSL_DIR env (which is only for target). Fortunately, openssl-sys allows target-prefixed env variables. So change

ENV OPENSSL_DIR=/usr/local/musl
ENV OPENSSL_STATIC=1

into

ENV X86_64_UNKNOWN_LINUX_MUSL_OPENSSL_DIR=/usr/local/musl
ENV X86_64_UNKNOWN_LINUX_MUSL_OPENSSL_STATIC=1
ENV AARCH64_UNKNOWN_LINUX_MUSL_OPENSSL_DIR=/usr/local/musl
ENV AARCH64_UNKNOWN_LINUX_MUSL_OPENSSL_STATIC=1

and the Dockerfile builds correctly.

[recmo]

Feel free to close this since my issue was solved and the root-cause was not ethers-rs. I'm leaving it open for the feature request to make reqwest optional in proc-macros, which causes issues in cross-compiles. I'm not able to change the 'bug' label into 'enhancement'.

[mattsse]

great insight, thanks.
I think feature gating reqwest is reasonable, should be configurable

[gakonst]

@recmo closing, lmk if #580 fixed the issue, else we can re-open