try to workaround qemu-ppc64le brokeness

[japaric]

On ubuntu 16.04, this crashes:

$ sudo apt-get install -qq binfmt-support qemu-user-static gcc-powerpc64le-linux-gnu libc6-dev-ppc64le-cross

$ echo 'int main() {}' > a.c

$ powerpc64-linux-gnu-gcc a.c

$ export QEMU_LD_PREFIX=/usr/powerpc64le-linux-gnu/

$ qemu-ppc64 ./a.out
Invalid instruction
NIP 000000400082e0bc   LR 0000004000806ea4 CTR 000000000000000a XER 0000000000000000 CPU#0
MSR 8000000002806001 HID0 0000000000000000  HF 0000000002806001 idx 0
TB 00014118 60637766692038
GPR00 00000040008023b8 00000040007ff540 0000004000858e00 00000040007ff5a0
GPR04 0000000000000000 0000000000000280 0000000000000000 0000000000000000
GPR08 000000000000000a 00000040007ff5a0 0000000000000000 0000000000000000
GPR12 00000040007ff5a0 0000000000000000 0000000000000000 0000000000000000
GPR16 0000000000000000 0000000000000000 0000000000000000 0000000000000000
GPR20 0000000000000000 0000000000000000 0000000000000000 0000000000000000
GPR24 0000000000000000 0000000000000000 0000000000000000 0000000000000000
GPR28 0000000000000000 00000040007ffaf0 0000000000000000 0000000000000000
CR 40000002  [ G  -  -  -  -  -  -  E  ]             RES ffffffffffffffff
FPR00 0000000000000000 0000000000000000 0000000000000000 0000000000000000
FPR04 0000000000000000 0000000000000000 0000000000000000 0000000000000000
FPR08 0000000000000000 0000000000000000 0000000000000000 0000000000000000
FPR12 0000000000000000 0000000000000000 0000000000000000 0000000000000000
FPR16 0000000000000000 0000000000000000 0000000000000000 0000000000000000
FPR20 0000000000000000 0000000000000000 0000000000000000 0000000000000000
FPR24 0000000000000000 0000000000000000 0000000000000000 0000000000000000
FPR28 0000000000000000 0000000000000000 0000000000000000 0000000000000000
FPSCR 0000000000000000
qemu: uncaught target signal 4 (Illegal instruction) - core dumped
Illegal instruction (core dumped)

And the same happens if you try a Rust program cross compiled for powerpc64le-unknown-linux-gnu.

IIRC, this is caused by QEMU and gcc not agreeing on their default arch level (-march) or cpu features they use. So, it may be possible to workaround this by building qemu from source but configuring it to support the same instruction set that gcc emits. Alternatively, tweaking gcc via CFLAGS (or by flat out recompiling with different configure options) may also work.

[amboar]

@japaric I've tried to reproduce your issue. As a note, you're compiling BE with powerpc64-* but invoking qemu-ppc64 (BE) with a QEMU_LD_PREFIX pointing to an LE environment. Trying to reproduce that, I get:

$ cat a.c
int main() {}
$ powerpc64-linux-gnu-gcc a.c
$ QEMU_LD_PREFIX=/usr/powerpc64le-linux-gnu qemu-ppc64 a.out
/lib64/ld64.so.1: No such file or directory

Running with QEMU_LD_PREFIX pointing to a BE environment, I get:

$ QEMU_LD_PREFIX=/usr/powerpc64-linux-gnu qemu-ppc64 a.out
$ echo $?
0

This is on:

$ fgrep -i -e name -e version /etc/os-release 
NAME="Ubuntu"
VERSION="16.04.1 LTS (Xenial Xerus)"
...
$ qemu-ppc64 -version
qemu-ppc64 version 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.5), Copyright (c) 2003-2008 Fabrice Bellard

I came across your issue when I ran into the similar result with running a host-cross-compiled llvm-config:

$ QEMU_LD_PREFIX=/usr/powerpc64le-linux-gnu qemu-ppc64le powerpc64le-unknown-linux-gnu/llvm/bin/llvm-config --ldflags              
Invalid instruction
NIP 000000400082e0bc   LR 0000004000806ea4 CTR 000000000000000a XER 0000000000000000 CPU#0
MSR 8000000002806001 HID0 0000000000000000  HF 0000000002806001 idx 0
TB 00009829 42219014253107
GPR00 00000040008023b8 00000040007ff9d0 0000004000858e00 00000040007ffa30
GPR04 0000000000000000 0000000000000280 0000000000000000 0000000000000000
GPR08 000000000000000a 00000040007ffa30 0000000000000000 0000000000000000
GPR12 00000040007ffa30 0000000000000000 0000000000000000 0000000000000000
GPR16 0000000000000000 0000000000000000 0000000000000000 0000000000000000
GPR20 0000000000000000 0000000000000000 0000000000000000 0000000000000000
GPR24 0000000000000000 0000000000000000 0000000000000000 0000000000000000
GPR28 0000000000000000 00000040007fff80 0000000000000000 0000000000000000
CR 40000002  [ G  -  -  -  -  -  -  E  ]             RES ffffffffffffffff
FPR00 0000000000000000 0000000000000000 0000000000000000 0000000000000000
FPR04 0000000000000000 0000000000000000 0000000000000000 0000000000000000
FPR08 0000000000000000 0000000000000000 0000000000000000 0000000000000000
FPR12 0000000000000000 0000000000000000 0000000000000000 0000000000000000
FPR16 0000000000000000 0000000000000000 0000000000000000 0000000000000000
FPR20 0000000000000000 0000000000000000 0000000000000000 0000000000000000
FPR24 0000000000000000 0000000000000000 0000000000000000 0000000000000000
FPR28 0000000000000000 0000000000000000 0000000000000000 0000000000000000
FPSCR 0000000000000000
qemu: uncaught target signal 4 (Illegal instruction) - core dumped
Illegal instruction (core dumped)

Your mismatch suggestion triggered the solution to my issue, which is to invoke qemu-ppc64le with a usable CPU type:

$ QEMU_LD_PREFIX=/usr/powerpc64le-linux-gnu qemu-ppc64le -cpu POWER8 powerpc64le-unknown-linux-gnu/llvm/bin/llvm-config --ldflags
-L/home/andrew/src/rust-lang/rust/powerpc64le-unknown-linux-gnu/llvm/lib 

Maybe that will help?

Edit: qemu-pp64le also looks at QEMU_CPU in the environment. Setting this allows you to work with binfmt_misc without a wrapper.