Cross-compiling from MacOS to x86_64-linux-musl

[dandxy89]

Hello @rust-or/owners ,

Have you of you attempted to create an AWS Lambda using the library?

[lovasoa]

Hello !
No I haven't. I'm not familiar with lambdas, why would it cause a problem ?

[lovasoa]

If it requires a single statically-linked binary, than I recommend using the HiGHS solver, which is statically linked by default (in highs-sys).

[dandxy89]

I've pushed a demo here - if you have time would appreciate your thoughts.

The current error is this when using the Docker:

error: linker `x86_64-linux-musl-gcc` not found
  |
  = note: No such file or directory (os error 2)

error: aborting due to previous error

error: could not compile `hello_lp`

To learn more, run the command again with --verbose.  

And also:

rustup target add x86_64-unknown-linux-musl

CROSS_COMPILE=x86_64-linux-musl- cargo build --release --target x86_64-unknown-linux-musl

gives the following error:

error: linking with `x86_64-linux-musl-gcc` failed: exit code: 1
  |
  = note: "x86_64-linux-musl-gcc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-Wl,--eh-frame-hdr" "-nostartfiles" "/Users/dixeda/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-unknown-linux-musl/lib/self-contained/rcrt1.o" "/Users/dixeda/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-unknown-linux-musl/lib/self-contained/crti.o" "-L" "/Users/dixeda/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-unknown-linux-musl/lib" "-L" "/Users/dixeda/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-unknown-linux-musl/lib/self-contained" "/Users/dixeda/Rust/hello_lp/target/x86_64-unknown-linux-musl/release/deps/hello_lp-a570d8da80d2e6c2.hello_lp.3rkesbqq-cgu.0.rcgu.o" "-o" "/Users/dixeda/Rust/hello_lp/target/x86_64-unknown-linux-musl/release/deps/hello_lp-a570d8da80d2e6c2" "-Wl,--gc-sections" "-static-pie" "-Wl,-zrelro" "-Wl,-znow" "-Wl,-O1" "-nodefaultlibs" "-L" "/Users/dixeda/Rust/hello_lp/target/x86_64-unknown-linux-musl/release/deps" "-L" "/Users/dixeda/Rust/hello_lp/target/release/deps" "-L" "/Users/dixeda/Rust/hello_lp/target/x86_64-unknown-linux-musl/release/build/highs-sys-075817a4a76bacfc/out/lib" "-L" "/Users/dixeda/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-unknown-linux-musl/lib" "-Wl,-Bstatic" "/var/folders/hs/w_c46xx12dv250lktkxj_nqh37j9hh/T/rustcZiZbnM/libhighs_sys-b1a4f44102fee618.rlib" "-Wl,--start-group" "/var/folders/hs/w_c46xx12dv250lktkxj_nqh37j9hh/T/rustcZiZbnM/libunwind-75081e895819d734.rlib" "/var/folders/hs/w_c46xx12dv250lktkxj_nqh37j9hh/T/rustcZiZbnM/liblibc-d3a472b733fbfd21.rlib" "-Wl,--end-group" "/Users/dixeda/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-unknown-linux-musl/lib/libcompiler_builtins-f4df6b79d8e73f02.rlib" "-Wl,-Bdynamic" "-lstdc++" "-lgomp" "/Users/dixeda/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-unknown-linux-musl/lib/self-contained/crtn.o"
  = note: /usr/local/Cellar/musl-cross/0.9.9/libexec/bin/../lib/gcc/x86_64-linux-musl/9.2.0/../../../../x86_64-linux-musl/bin/ld: /var/folders/hs/w_c46xx12dv250lktkxj_nqh37j9hh/T/rustcZiZbnM/libhighs_sys-b1a4f44102fee618.rlib(highs_c_api.cpp.o): in function `__static_initialization_and_destruction_0(int, int) [clone .constprop.0]':
          highs_c_api.cpp:(.text.startup._Z41__static_initialization_and_destruction_0ii.constprop.0+0x2d): undefined reference to `__dso_handle'
          /usr/local/Cellar/musl-cross/0.9.9/libexec/bin/../lib/gcc/x86_64-linux-musl/9.2.0/../../../../x86_64-linux-musl/bin/ld: highs_c_api.cpp:(.text.startup._Z41__static_initialization_and_destruction_0ii.constprop.0+0x56): undefined reference to `__dso_handle'
          /usr/local/Cellar/musl-cross/0.9.9/libexec/bin/../lib/gcc/x86_64-linux-musl/9.2.0/../../../../x86_64-linux-musl/bin/ld: highs_c_api.cpp:(.text.startup._Z41__static_initialization_and_destruction_0ii.constprop.0+0x7f): undefined reference to `__dso_handle'
          /usr/local/Cellar/musl-cross/0.9.9/libexec/bin/../lib/gcc/x86_64-linux-musl/9.2.0/../../../../x86_64-linux-musl/bin/ld: highs_c_api.cpp:(.text.startup._Z41__static_initialization_and_destruction_0ii.constprop.0+0xa8): undefined reference to `__dso_handle'
          /usr/local/Cellar/musl-cross/0.9.9/libexec/bin/../lib/gcc/x86_64-linux-musl/9.2.0/../../../../x86_64-linux-musl/bin/ld: highs_c_api.cpp:(.text.startup._Z41__static_initialization_and_destruction_0ii.constprop.0+0xd1): undefined reference to `__dso_handle'
          /usr/local/Cellar/musl-cross/0.9.9/libexec/bin/../lib/gcc/x86_64-linux-musl/9.2.0/../../../../x86_64-linux-musl/bin/ld: /var/folders/hs/w_c46xx12dv250lktkxj_nqh37j9hh/T/rustcZiZbnM/libhighs_sys-b1a4f44102fee618.rlib(highs_c_api.cpp.o):highs_c_api.cpp:(.text.startup._Z41__static_initialization_and_destruction_0ii.constprop.0+0xf7): more undefined references to `__dso_handle' follow
          /usr/local/Cellar/musl-cross/0.9.9/libexec/bin/../lib/gcc/x86_64-linux-musl/9.2.0/../../../../x86_64-linux-musl/bin/ld: /Users/dixeda/Rust/hello_lp/target/x86_64-unknown-linux-musl/release/deps/hello_lp-a570d8da80d2e6c2: hidden symbol `__dso_handle' isn't defined
          /usr/local/Cellar/musl-cross/0.9.9/libexec/bin/../lib/gcc/x86_64-linux-musl/9.2.0/../../../../x86_64-linux-musl/bin/ld: final link failed: bad value
          collect2: error: ld returned 1 exit status

No luck with either

[dandxy89]

The error seems to point vaguely to highs cpp

[lovasoa]

Ok, I transferred the issue to the highs-sys repository.

I think the problem is that highs-sys links highs itself dynamically, but it links libstdc++ dynamically, and your target doesn't have a libstdc++

https://github.com/rust-or/highs-sys/blob/master/build.rs#L55

[lovasoa]

Are you interested in working on this issue ? Could you make a PR that detects whether libstdc++ should be statically linked and sets the right values in build.rs ? You could also update the CI to add a linux musl target.

[dandxy89]

I can certainly give it a go.

Thanks for the quick investigation - wish I had flagged it several days ago now to be fair.

Many thanks @lovasoa

[lovasoa]

I'm not an expert with C++ and CMake, but it looks like it has a useful option to add standard library linker flags:
https://cmake.org/cmake/help/latest/variable/CMAKE_LANG_STANDARD_LIBRARIES.html

So you should be able to add

        .define("CMAKE_CXX_STANDARD_LIBRARIES", "-static-libgcc -static-libstdc++")

to the build script.

Of course you will need to have a libstdc++.a for static linking to succeed.

[lovasoa]

If you have a linux system lying around, I'd advise you try to get it working without cross-compilation first (just creating a fully statically linked binary from linux), then check if it cross-compiles correctly.

[lovasoa]

libstdc++ should indeed probably be statically linked, but contrarily to what I said your initial issue may not be with that.

See rust-lang/rust#36710

and the suggested workaround, using docker: rust-rocksdb/rust-rocksdb#440 (comment)

[dandxy89]

Hey @lovasoa

Managed to get hold of a Linux machine but unable to build.

~/highs-sys(master) » cargo build                                                                                                                                101 ↵ dan@dan-Galago-UltraPro
   Compiling highs-sys v0.1.4 (/home/dan/highs-sys)
error: failed to run custom build command for `highs-sys v0.1.4 (/home/dan/highs-sys)`

Caused by:
  process didn't exit successfully: `/home/dan/highs-sys/target/debug/build/highs-sys-064c0f4b45f3275d/build-script-build` (exit code: 101)
  --- stdout
  running: "cmake" "/home/dan/highs-sys/HiGHS" "-DFAST_BUILD=ON" "-DSHARED=OFF" "-DCMAKE_MSVC_RUNTIME_LIBRARY=MultiThreadedDLL" "-DCMAKE_CXX_STANDARD_LIBRARIES=-static-libgcc -static-libstdc++" "-DCMAKE_INSTALL_PREFIX=/home/dan/highs-sys/target/debug/build/highs-sys-8fd57449c0e17be3/out" "-DCMAKE_C_FLAGS= -ffunction-sections -fdata-sections -fPIC -m64" "-DCMAKE_C_COMPILER=/usr/bin/cc" "-DCMAKE_CXX_FLAGS= -ffunction-sections -fdata-sections -fPIC -m64" "-DCMAKE_CXX_COMPILER=/usr/bin/c++" "-DCMAKE_ASM_FLAGS= -ffunction-sections -fdata-sections -fPIC -m64" "-DCMAKE_ASM_COMPILER=/usr/bin/cc" "-DCMAKE_BUILD_TYPE=Debug"
  -- Linking with OpenMP... 
  -- FAST_BUILD set to on.
   Note: The HiGHS team is preparing for our first official release. If you
         experience any issues please let us know via email or on GitHub.
  -- Configuring done
  -- Generating done
  -- Build files have been written to: /home/dan/highs-sys/target/debug/build/highs-sys-8fd57449c0e17be3/out/build
  running: "cmake" "--build" "." "--target" "install" "--config" "Debug" "--"
  Consolidate compiler generated dependencies of target libhighs
  [ 80%] Built target libhighs
  Consolidate compiler generated dependencies of target highs
  [ 81%] Building CXX object app/CMakeFiles/highs.dir/__/src/mip/HighsDynamicRowMatrix.cpp.o
  app/CMakeFiles/highs.dir/build.make:131: recipe for target 'app/CMakeFiles/highs.dir/__/src/mip/HighsDynamicRowMatrix.cpp.o' failed
  CMakeFiles/Makefile2:141: recipe for target 'app/CMakeFiles/highs.dir/all' failed
  Makefile:145: recipe for target 'all' failed

  --- stderr
  CMake Warning (dev) in CMakeLists.txt:
    Policy CMP0110 is not set: add_test() supports arbitrary characters in test
    names.  Run "cmake --help-policy CMP0110" for policy details.  Use the
    cmake_policy command to set the policy and suppress this warning.

    The following name given to add_test() is invalid if CMP0110 is not set or
    set to OLD:

      `25fv47 ´

  This warning is for project developers.  Use -Wno-dev to suppress it.

  CMake Warning (dev) in CMakeLists.txt:
    Policy CMP0110 is not set: add_test() supports arbitrary characters in test
    names.  Run "cmake --help-policy CMP0110" for policy details.  Use the
    cmake_policy command to set the policy and suppress this warning.

    The following name given to add_test() is invalid if CMP0110 is not set or
    set to OLD:

      `80bau3b ´

  This warning is for project developers.  Use -Wno-dev to suppress it.

  CMake Warning (dev) in CMakeLists.txt:
    Policy CMP0110 is not set: add_test() supports arbitrary characters in test
    names.  Run "cmake --help-policy CMP0110" for policy details.  Use the
    cmake_policy command to set the policy and suppress this warning.

    The following name given to add_test() is invalid if CMP0110 is not set or
    set to OLD:

      `greenbea ´

  This warning is for project developers.  Use -Wno-dev to suppress it.

  CMake Warning (dev) in CMakeLists.txt:
    Policy CMP0110 is not set: add_test() supports arbitrary characters in test
    names.  Run "cmake --help-policy CMP0110" for policy details.  Use the
    cmake_policy command to set the policy and suppress this warning.

    The following name given to add_test() is invalid if CMP0110 is not set or
    set to OLD:

      `bgetam ´

  This warning is for project developers.  Use -Wno-dev to suppress it.

  CMake Warning (dev) in CMakeLists.txt:
    Policy CMP0110 is not set: add_test() supports arbitrary characters in test
    names.  Run "cmake --help-policy CMP0110" for policy details.  Use the
    cmake_policy command to set the policy and suppress this warning.

    The following name given to add_test() is invalid if CMP0110 is not set or
    set to OLD:

      `gas11 ´

  This warning is for project developers.  Use -Wno-dev to suppress it.

  In file included from /home/dan/highs-sys/HiGHS/src/mip/HighsDynamicRowMatrix.cpp:10:0:
  /home/dan/highs-sys/HiGHS/src/./mip/HighsDynamicRowMatrix.h:56:3: error: ‘size_t’ does not name a type
     size_t nonzeroCapacity() const { return ARvalue_.size(); }
     ^
  cc1plus: warning: unrecognized command line option ‘-Wno-unused-const-variable’
  cc1plus: warning: unrecognized command line option ‘-Wno-defaulted-function-deleted’
  make[2]: *** [app/CMakeFiles/highs.dir/__/src/mip/HighsDynamicRowMatrix.cpp.o] Error 1
  make[1]: *** [app/CMakeFiles/highs.dir/all] Error 2
  make: *** [all] Error 2
  thread 'main' panicked at '
  command did not execute successfully, got: exit code: 2

Are you able to advise / take a look at this at all?