when pre-building custom docker image inside docker container cross uses weird path

[DoumanAsh]

Checklist

• I've looked through the issues and pull requests for similar reports

Describe your issue

When building with CROSS_CONTAINER_IN_CONTAINER=true pre-building uses weird path e.g.:

docker build --label 'org.cross-rs.for-cross-target=aarch64-unknown-linux-gnu' --label 'org.cross-rs.workspace_root=/mount/core' --tag cross-custom-core:aarch64-unknown-linux-gnu-2ed18-pre-build --build-arg 'CROSS_CMD=dpkg --add-architecture arm64 && apt-get update && apt-get install -y libc6:arm64 openssl libudev-dev:arm64 pkg-config-aarch64-linux-gnu && alias pkg-config=aarch64-linux-gnu-pkg-config' --build-arg 'CROSS_DEB_ARCH=arm64' --file /mount/core/target/aarch64-unknown-linux-gnu/Dockerfile.aarch64-unknown-linux-gnu-custom /run/desktop/mnt/host/d/repos/work/core`

While CROSS_CONTAINER_IN_CONTAINER=false would avoid this problem using file system inside docker as it is, in my case it is /mount/core, but it makes me unable to build due to missing cargo, not sure why exactly:

sh: 1: cargo: not found

The way I start docker image:

docker run --rm --network host --name docker-desktop -v <my repo>:/mount -v //var/run/docker.sock:/var/run/docker.sock --privileged -it <image name>

It seems it only happens because I mounted this directory, but this is honestly not good idea.
It should use mounted folder, not trying to pick source by inspecting docker contaier and checking the mount

It is probably not going to be a problem inside github actions where I intend to use it, but this is basically unusable when I do local test

What target(s) are you cross-compiling for?

aarch64-unknown-linux-gnu

Which operating system is the host (e.g computer cross is on) running?

• macOS
• Windows
• Linux / BSD
• other OS (specify in description)

What architecture is the host?

• x86_64 / AMD64
• arm32
• arm64 (including Mac M1)

What container engine is cross using?

• docker
• podman
• other container engine (specify in description)

cross version

cross 0.2.4 (4645d93 2022-07-10)

Example

No response

Additional information / notes

No response

[Emilgardis]

Hi!

Are you sure you need container in container? Sounds to me like you don't need it.

The container that cross is invoked in with CROSS_CONTAINER_IN_CONTAINER needs to have rust installed via rustup. It won't use the hosts rust installation. So, the solution would be to install rust in your docker image.

This could be a XY problem, what are you trying to do?

[DoumanAsh]

So I build custom docker image containing both rust and flutter.
Due to cross-compilation need, we use cross to build for arm64.

So I thought I have to use CROSS_CONTAINER_IN_CONTAINER=true because I invoke cross from within running container.
But it seems I'm wrong?

So my docker image includes:

1. rustup with image's host toolchain installed
2. flutter
3. cross binary

I have to build rust part using cross with following cross config:

16:52 $ cat .\core\Cross.toml
[build.env]
passthrough = [
    "PAYBOX_CLIENT_VERSION",
    "RUSTFLAGS"
]

[target.aarch64-unknown-linux-gnu]
pre-build = ["dpkg --add-architecture arm64 && apt-get update && apt-get install -y libc6:arm64 openssl libudev-dev:arm64 pkg-config-aarch64-linux-gnu && alias pkg-config=aarch64-linux-gnu-pkg-config"]

[target.aarch64-unknown-linux-gnu.env]
passthrough = [
    "PAYBOX_CLIENT_VERSION",
    "RUSTFLAGS"
]
volumes = [
    "DARTFILES_DIR",
]

[Emilgardis]

are the rust binaries added to PATH in the image?

i.e, can you yourself run cargo -V inside the image?

[DoumanAsh]

Yes I can

[DoumanAsh]

Btw, I noticed I had added --force-non-host because I used musl toolchain previously, but could it be a problem?

[Emilgardis]

you only need to rustup install with --force-non-host if the host (in this case the container/image) is not the same architecture (i.e, x86_64-linux in most cases). cross will automatically install the needed toolchain (which is x86_64-unknown-linux-gnu unless you use newer features in current main, image.toolchain )

I think I know the issue though (for the path thing), we're not correctly identifying/pathing the target folder, should be an easy fix

unsure why cargo is not found

[DoumanAsh]

Here is docker image that I use:

FROM ubuntu:22.04 as build

USER root

ARG FLUTTER_VERSION="3.0.5"
ENV CARGO_HOME=/usr/local/cargo
ENV FLUTTER_HOME=/usr/local/flutter
ENV RUSTUP_HOME=/usr/local/rustup \
    FLUTTER_SDK_HOME=$FLUTTER_HOME \
    PATH=$CARGO_HOME/bin:$FLUTTER_HOME/bin:$PATH \
    RUST_VERSION=stable \
    CROSS_CONTAINER_IN_CONTAINER=true \
    DEBIAN_FRONTEND=noninteractive \
    DEBCONF_NONINTERACTIVE_SEEN=true \
    TZ=Asia/Tokyo 

# Keep this as single RUN to reduce image size
RUN set -eux; \
    echo ">>>Prepare time zone settings" ;\
    ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone ;\
    echo ">>>Install system packages" ;\
    apt-get update ;\
    apt-get install -y make curl file git unzip xz-utils zip wget docker tar binutils ;\
    wget -q https://download.docker.com/linux/ubuntu/dists/jammy/pool/stable/amd64/docker-ce-cli_20.10.17~3-0~ubuntu-jammy_amd64.deb ;\
    dpkg -i docker-ce-cli_20.10.17~3-0~ubuntu-jammy_amd64.deb && rm docker-ce-cli_20.10.17~3-0~ubuntu-jammy_amd64.deb ;\
    apt-get clean; \
    rm -rf /var/lib/apt/lists/*; \
    echo ">>>Install Rust" ;\
    curl https://sh.rustup.rs -sSf | sh -s -- -y --no-modify-path --profile minimal --default-toolchain $RUST_VERSION ;\
    wget -q https://github.com/cross-rs/cross/releases/download/v0.2.4/cross-x86_64-unknown-linux-musl.tar.gz ;\
    tar -xf cross-x86_64-unknown-linux-musl.tar.gz && mv cross $CARGO_HOME/bin && rm -rf cross-*;\
    wget -q https://raw.githubusercontent.com/ardera/flutter-engine-binaries-for-arm/37f9852034729e0ab0f52bf913294e833ff1ec55/arm64/gen_snapshot_linux_x64_release ;\
    mv gen_snapshot_linux_x64_release $CARGO_HOME/bin ;\
    chmod -R a+w+x $RUSTUP_HOME $CARGO_HOME; \
    rm -rf $CARGO_HOME/registry ;\
    echo ">>>Install flutter" ;\
    wget -q https://storage.googleapis.com/flutter_infra_release/releases/stable/linux/flutter_linux_$FLUTTER_VERSION-stable.tar.xz ;\
    tar xf flutter_linux_$FLUTTER_VERSION-stable.tar.xz ;\
    mv flutter $FLUTTER_HOME && rm flutter_linux_$FLUTTER_VERSION-stable.tar.xz ;\
    git config --global --add safe.directory $FLUTTER_HOME ;\
    $FLUTTER_HOME/bin/dart --disable-analytics ;\
    $FLUTTER_HOME/bin/flutter config --no-analytics --enable-linux-desktop ;\
    $FLUTTER_HOME/bin/flutter precache --universal --linux ;\
    find $RUSTUP_HOME -type f -executable -exec strip --strip-unneeded {} \; && find $RUSTUP_HOME -name *.so -exec strip --strip-unneeded {} \; && find $RUSTUP_HOME -name *.rlib -exec strip -d {} \; && find $RUSTUP_HOME -name *.a -exec strip -d {} \;

WORKDIR /
CMD [ "/bin/bash" ]

It is a bit of mess, but you can see that I copy cross into $CARGO_HOME/bin so they are both in the same place

[DoumanAsh]

unsure why cargo is not found

so what should be a correct way to invoke cross when running within container?

I think I used it fine without CROSS_CONTAINER_IN_CONTAINER=true in github actions CI, but unfortunately locally I seem unable to test it

[DoumanAsh]

Enabling verbose I noticed how it runs image:

/usr/bin/docker run --userns host <options...> -t ghcr.io/cross-rs/aarch64-unknown-linux-gnu:main sh -c 'PATH=$PATH:/rust/bin cargo --verbose build --target=aarch64-unknown-linux-gnu --release'

PATH=$PATH:/rust/bin cargo --verbose build --target=aarch64-unknown-linux-gnu --release

It seems to execute cargo within ghcr.io/cross-rs/aarch64-unknown-linux-gnu
But it doesn't actually have cargo

[Emilgardis]

Yes, because we mount rust and family inside this container. That's how cross works basically. In the omitted <options...> there should be a -v /path/to/.cargo:/cargo etc

[DoumanAsh]

yes, it does mount -v /usr/local/cargo:/cargo:Z

Which on my docker image is:

root@docker-desktop:/mount# ls /usr/local/cargo
bin  env  git  registry

but it doesn't use /cargo/bin in PATH=$PATH:/rust/bin
So I looked up how it mounts /rust: -v /usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu:/rust:Z,ro

its content:

root@docker-desktop:/mount# ls /usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu
bin  etc  lib  libexec  share
root@docker-desktop:/mount# ls /usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin
cargo  rust-gdb  rust-gdbgui  rust-lldb  rustc  rustdoc

Notice how ro is specified, which means it should be readonly.
Would docker give it executable rights?

P.s. I cannot really test it because on windows docker permissions are broken
P.s.s opening created container without sh command I see that /rust directory is empty:

root@f59b80a0656e:/mount/core# ls /rust
root@f59b80a0656e:/mount/core#

In fact all folders are mounted without any files... So I guess it is docker bug?

[Emilgardis]

can you try using this attached binary for cross

cross-x86_64-unknown-linux-gnu.zip
it's from the artifacts here: https://github.com/cross-rs/cross/actions/runs/2831450427#artifacts on pr #994

make sure to chmod +x the binary

[DoumanAsh]

@Emilgardis with your PR and CROSS_CONTAINER_IN_CONTAINER=true it starts building

Just for reference it correctly mounts directories

/var/lib/docker/overlay2/88787e049217240ee1b0abf7cff0167980100780f22664eaa4f35fd087450f69/merged/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin

[Emilgardis]

awesome!

[DoumanAsh]

I assume we can disregard case of CROSS_CONTAINER_IN_CONTAINER=false not working since it is not intended for use within docker container?

[Emilgardis]

Yes, I think so, it could be a real issue but for now it should be fine.

[DoumanAsh]

Ok, thank you.
I hope you can release it soon as I'd prefer to use proper cross version 😄

[Emilgardis]

the cargo not found issue is #728, we're not able to fix it easily

[DoumanAsh]

Hm in my case I don't run docker within WSL, but I do use WSL as backend though so maybe it is related