cargo test --doc not load same shared libraries as cargo test --lib #8531

adamcrume · 2020-07-23T04:47:04Z

Problem
cargo test --doc doesn't appear to be loading the same shared libraries as cargo test --lib in 1.45

Steps

Clone https://github.com/tensorflow/rust
Set Rust to 1.44
Note that both cargo test --doc and cargo test --lib pass
Set Rust to 1.45
Note that cargo test --lib passes but cargo test --doc fails with error while loading shared libraries: libtensorflow.so.1: cannot open shared object file: No such file or directory

Notes

Output of cargo version:
cargo 1.44.0 (05d080f 2020-05-06) or cargo 1.45.0 (744bd1f 2020-06-15)

Running on Ubuntu 20.04 LTS.

The text was updated successfully, but these errors were encountered:

thoren-d · 2020-10-03T23:06:56Z

For what it's worth, I'm seeing the same issue running doctests against my library that uses SDL2 (https://github.com/Rust-SDL2/rust-sdl2). cargo +1.44.0 test --doc works.

raskr · 2021-01-16T08:02:28Z

Same problem with any OS + rust1.48+, but not with rust1.42.
cargo test --doc with shared lib seems to be broken.

adamcrume · 2021-04-04T18:12:07Z

Using git bisect, I narrowed down to 3fd2814 being the breaking commit.

pipehappy1 · 2022-03-04T09:38:47Z

This is still broken on 1.58

Fixes rust-lang#8531. This issue was introduced by 3fd2814. Prior to that commit, the `rustdoc_process` function took other branch of the if statement being modified by this commit. The flag was previously called `is_host`, and `rustdoc` was run reporting `false`. In that commit, the flag was changed to `is_rustc_tool`, and rustdoc began reporting `true`, which resulted in the native directories added by build scripts no longer being appended to LD_LIBRARY_PATH when running doctests. This commit changes the behavior so that we are always appending the same set of paths, and the only variance is if we are cross compiling or not. An alternative would be to change rustdoc to always pass `kind: CompileKind::Host, is_rustc_tool: false`, but as rustdoc is in fact a rustc tool, that feels wrong. The commit which introduced the bug did not include context on why the flag was changed to `is_rustc_tool`, so I don't have a sense for if we should just change that flag to mean something else. While the test suite previously had an explicit test for the library path behavior of `cargo run`, it did not test this for various test forms. This commit modifies the test to ensure the behavior is consistent across `run`, normal tests, and doctests.

Trolldemorted · 2022-05-10T16:34:50Z

This issue is known for almost 3 years, could you give us a timeline when a fix will come? Not knowing about this I just wasted hours trying to find out why my doc tests fail :(

justsmth · 2022-06-15T20:47:38Z

Is there a workaround for this issue?

Update:
I found that I can workaround this by prefixing the command with LD_LIBRARY_PATH indicating the location of the shared libraries: LD_LIBRARY_PATH=... cargo test --doc

Trolldemorted · 2022-09-13T20:36:21Z

@adamcrume do you have any clue what might have caused this bug? Is @alexcrichton still around? Iirc he was stepping down from several Rust projects, but not cargo.

adamcrume · 2022-09-14T04:11:10Z

I don't know what specifically in 3fd2814 caused this. I'm not familiar with the cargo codebase.

weihanglo · 2022-09-14T06:27:53Z

We appreciate Alex efforts on Cargo and Alex has already retired from the Cargo Team.¹

There was a previous attempt #10469 and concerns were listed #10469 (comment). Feel free to pick it up and continue.

https://blog.rust-lang.org/inside-rust/2022/03/31/cargo-team-changes.html ↩

Trolldemorted · 2023-06-25T11:07:18Z

@weihanglo with all due respect, the concerns you linked do not point towards a, obvious, clear path forward how this issue should be remediated. I do understand you rely on others to help you fix bugs, but a community can do that with reasonable efficacy only if it is crystal clear how things should work. The concerns you listed are referenced and thought through, but they don't enable outsiders to grasp how you'd want things to be done.

This bug is a problem for the Rust ecosystem. I'd love to recommend Rust more strongly in industrial projects, but this is one of the reasons why I cannot. The cargo team has broken the test coverage for all crates that include native dylibs, and some crates have responded by disabling their doctests with no_run, other crates have stopped efforts to link against included native libraries with a guaranteed matching version, and instead rely on package managers installing a more or less matching version, thus causing undefined behavior. Cargo should not ignore bugs that incentivize maintainers to reduce test coverage or cause undefined behaviour for years.

@ehuss I hate to be nagging, but at which point in time will the cargo team fix this issue, if no volunteers are able to address it in a way you'd accept it? How do you select which bug fixes are scheduled for a new cargo release?

In some system, `cargo test --doc` may fail due to the bug: rust-lang/cargo#8531

tompscanlan · 2023-09-18T18:23:01Z

Looks as though no one is looking at this. I'm new to the code, but this seems approachable to me. I'll take a wack if it's really open.

@echo

Update CI diff --git a/.github/workflows/sys-bindings-generator.yml b/.github/workflows/sys-bindings-generator.yml index 3fba893d80..dc226f9f1b 100644 --- a/.github/workflows/sys-bindings-generator.yml +++ b/.github/workflows/sys-bindings-generator.yml @@ -270,3 +270,36 @@ jobs: run: ./scripts/build/collect_build_src.sh -t ${{ matrix.target }} - name: Commit & Push changes run: ./scripts/ci/ci_add_commit_rebase_push.sh "Collected source files for ${{ matrix.target }}" + collect-nasm-and-commit: + needs: generate-windows-bindings-and-commit + if: github.repository == 'aws/aws-lc-rs' + runs-on: windows-latest + steps: + - uses: actions/checkout@v4 + with: + submodules: 'recursive' + ref: ${{ github.ref_name }} + - uses: dtolnay/rust-toolchain@master + id: toolchain + with: + toolchain: stable + targets: "x86_64-pc-windows-msvc,x86_64-pc-windows-gnu" + - uses: ilammy/setup-nasm@v1 + - name: Build aws-lc-sys + shell: bash + run: AWS_LC_SYS_PREBUILT_NASM=0 cargo build -p aws-lc-sys --release --target x86_64-pc-windows-msvc + - name: Collect NASM object files + shell: bash + run: ./scripts/build/collect_nasm_obj.sh + - name: Clean build + shell: bash + run: cargo clean + - name: Test aws-lc-rs for x86_64-pc-windows-msvc + shell: bash + run: AWS_LC_SYS_PREBUILT_NASM=1 cargo build -p aws-lc-sys --target x86_64-pc-windows-msvc + - name: Test aws-lc-sys for x86_64-pc-windows-gnu + shell: bash + run: AWS_LC_SYS_PREBUILT_NASM=1 cargo build -p aws-lc-sys --target x86_64-pc-windows-gnu + - name: Commit & Push changes + shell: bash + run: ./scripts/ci/ci_add_commit_rebase_push.sh "Collected NASM files" diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index a0da3f48b8..71716a64c0 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -37,7 +37,7 @@ jobs: - --no-default-features --features non-fips,ring-sig-verify,unstable - --no-default-features --features non-fips,alloc,unstable steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@master @@ -66,7 +66,7 @@ jobs: - --no-default-features --features aws-lc-sys,bindgen,unstable - --release --all-targets --features bindgen,unstable steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - if: ${{ matrix.os == 'macos-13-xlarge' }} @@ -106,9 +106,10 @@ jobs: - --no-default-features --features non-fips,ring-io,unstable - --no-default-features --features non-fips,ring-sig-verify,unstable - --no-default-features --features non-fips,alloc,unstable + env: + AWS_LC_SYS_PREBUILT_NASM: 1 steps: - - uses: ilammy/setup-nasm@v1 - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@master @@ -125,7 +126,7 @@ jobs: name: aws-ls-rs coverage runs-on: ubuntu-latest steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' lfs: true @@ -167,7 +168,7 @@ jobs: - --no-default-features --features fips,asan runs-on: ubuntu-latest steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@master @@ -196,7 +197,7 @@ jobs: os: [ ubuntu-latest, macos-12, macos-13-xlarge ] static: [ 0, 1 ] steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@stable @@ -205,6 +206,40 @@ jobs: # See: rust-lang/cargo#8531 run: cargo test -p aws-lc-rs --tests + build-env-nasm-test: + if: github.repository_owner == 'aws' + name: prebuilt NASM verification + runs-on: windows-latest + strategy: + fail-fast: false + matrix: + target: + - 'x86_64-pc-windows-msvc' + - 'x86_64-pc-windows-gnu' + steps: + - uses: actions/checkout@v4 + with: + submodules: 'recursive' + - uses: dtolnay/rust-toolchain@stable + - name: Install NASM + uses: ilammy/setup-nasm@v1 + - name: Remove NASM artifacts + shell: bash + run: | + cargo clean + rm ./aws-lc-sys/builder/prebuilt-nasm/* + - name: Run cargo test + shell: bash + run: AWS_LC_SYS_PREBUILT_NASM=0 cargo test --tests -p aws-lc-rs --release --no-default-features --features aws-lc-sys + - name: Collect NASM outputs + shell: bash + run: ./scripts/build/collect_nasm_obj.sh + - name: Flag any NASM changes + shell: bash + run: | + git add . + git diff --cached --exit-code HEAD -- aws-lc-sys/builder/prebuilt-nasm/*.txt + build-env-external-bindgen-test: if: github.repository_owner == 'aws' name: aws-lc-rs FIPS - External bindgen test @@ -216,14 +251,14 @@ jobs: matrix: os: [ ubuntu-latest, macos-12, macos-13-xlarge, windows-latest ] steps: - - if: ${{ matrix.os == 'windows-latest' }} - uses: ilammy/setup-nasm@v1 - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@stable - - name: Install ninja-build tool - uses: seanmiddleditch/gha-setup-ninja@v4 + - if: ${{ matrix.os == 'windows-latest' }} + uses: ilammy/setup-nasm@v1 + - if: ${{ matrix.os == 'windows-latest' }} + uses: seanmiddleditch/gha-setup-ninja@v5 - name: Install bindgen-cli run: cargo install --locked bindgen-cli - name: Remove bindings @@ -245,12 +280,10 @@ jobs: os: [ ubuntu-latest, macos-12, macos-13-xlarge ] static: [ 0, 1 ] steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@stable - - name: Install ninja-build tool - uses: seanmiddleditch/gha-setup-ninja@v4 - uses: actions/setup-go@v4 with: go-version: '>=1.18' @@ -271,7 +304,7 @@ jobs: matrix: os: [ ubuntu-latest, macos-12, macos-13-xlarge, windows-latest ] steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@stable @@ -306,15 +339,14 @@ jobs: matrix: os: [ ubuntu-latest, macos-12, macos-13-xlarge, windows-latest ] steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@stable - - name: Install ninja-build tool - uses: seanmiddleditch/gha-setup-ninja@v4 - uses: actions/setup-go@v4 with: go-version: '>=1.18' + - uses: seanmiddleditch/gha-setup-ninja@v5 - name: Run cargo test run: cargo test -p aws-lc-rs --tests --no-default-features --features fips - name: Release build @@ -346,14 +378,14 @@ jobs: matrix: os: [ ubuntu-latest, macos-12, macos-13-xlarge, windows-latest ] steps: - - if: ${{ matrix.os == 'windows-latest' }} - uses: ilammy/setup-nasm@v1 - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' + - if: ${{ matrix.os == 'windows-latest' }} + uses: ilammy/setup-nasm@v1 + - if: ${{ matrix.os == 'windows-latest' }} + uses: seanmiddleditch/gha-setup-ninja@v5 - uses: dtolnay/rust-toolchain@stable - - name: Install ninja-build tool - uses: seanmiddleditch/gha-setup-ninja@v4 - uses: actions/setup-go@v4 with: go-version: '>=1.18' @@ -378,7 +410,7 @@ jobs: - macos-12 - macos-13-xlarge steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' lfs: true diff --git a/aws-lc-sys/builder/cmake_builder.rs b/aws-lc-sys/builder/cmake_builder.rs index e25cecb443..c5069c5341 100644 --- a/aws-lc-sys/builder/cmake_builder.rs +++ b/aws-lc-sys/builder/cmake_builder.rs @@ -3,9 +3,9 @@ use crate::OutputLib::{Crypto, RustWrapper, Ssl}; use crate::{ - cargo_env, emit_warning, execute_command, is_crt_static, is_no_asm, option_env, target, - target_arch, target_env, target_family, target_os, target_underscored, target_vendor, - OutputLibType, + allow_prebuilt_nasm, cargo_env, emit_warning, execute_command, is_crt_static, is_no_asm, + option_env, target, target_arch, target_env, target_family, target_os, target_underscored, + target_vendor, test_nasm_command, OutputLibType, }; use std::env; use std::ffi::OsString; @@ -22,10 +22,6 @@ fn test_clang_cl_command() -> bool { execute_command("clang-cl".as_ref(), &["--version".as_ref()]).status } -fn test_nasm_command() -> bool { - execute_command("nasm".as_ref(), &["-version".as_ref()]).status -} - fn find_cmake_command() -> Option<OsString> { if let Some(cmake) = option_env("CMAKE") { emit_warning(&format!( @@ -162,7 +158,7 @@ impl CmakeBuilder { // See issue: aws#453 if target_os() == "windows" { - Self::configure_windows(&mut cmake_cfg); + self.configure_windows(&mut cmake_cfg); } // If the build environment vendor is Apple @@ -213,7 +209,7 @@ impl CmakeBuilder { cmake_cfg } - fn configure_windows(cmake_cfg: &mut cmake::Config) { + fn configure_windows(&self, cmake_cfg: &mut cmake::Config) { match (target_env().as_str(), target_arch().as_str()) { ("msvc", "aarch64") => { cmake_cfg.generator_toolset(format!( @@ -243,6 +239,22 @@ impl CmakeBuilder { } _ => {} } + if target_arch() == "x86_64" && Some(true) == allow_prebuilt_nasm() { + emit_warning("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"); + emit_warning("!!! Using pre-built NASM binaries !!!"); + emit_warning("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"); + + let script_path = self + .manifest_dir + .join("builder") + .join("prebuilt-nasm.bat") + .display() + .to_string(); + let script_path = script_path.replace('\\', "/"); + + cmake_cfg.define("CMAKE_ASM_NASM_COMPILER", script_path.as_str()); + cmake_cfg.define("CMAKE_VERBOSE_MAKEFILE", "1"); + } } fn configure_open_harmony(cmake_cfg: &mut cmake::Config) { @@ -298,10 +310,14 @@ impl crate::Builder for CmakeBuilder { let mut missing_dependency = false; if target_os() == "windows" { - if target_arch() == "x86_64" && !test_nasm_command() && !is_no_asm() { + if target_arch() == "x86_64" + && !is_no_asm() + && !test_nasm_command() + && Some(true) != allow_prebuilt_nasm() + { eprintln!( - "Consider setting `AWS_LC_SYS_NO_ASM` in the environment for development builds.\ - See User Guide about the limitations: https://aws.github.io/aws-lc-rs/index.html" + "Consider setting `AWS_LC_SYS_PREBUILT_NASM` in the build environment.\ + See User Guide: https://aws.github.io/aws-lc-rs/index.html" ); eprintln!("Missing dependency: nasm"); missing_dependency = true; diff --git a/aws-lc-sys/builder/main.rs b/aws-lc-sys/builder/main.rs index 23f0214991..987e1038ca 100644 --- a/aws-lc-sys/builder/main.rs +++ b/aws-lc-sys/builder/main.rs @@ -319,6 +319,7 @@ static mut AWS_LC_SYS_NO_PREFIX: bool = false; static mut AWS_LC_SYS_INTERNAL_BINDGEN: bool = false; static mut AWS_LC_SYS_EXTERNAL_BINDGEN: bool = false; static mut AWS_LC_SYS_NO_ASM: bool = false; +static mut AWS_LC_SYS_PREBUILT_NASM: Option<bool> = None; fn initialize() { unsafe { @@ -328,6 +329,7 @@ fn initialize() { AWS_LC_SYS_EXTERNAL_BINDGEN = env_var_to_bool("AWS_LC_SYS_EXTERNAL_BINDGEN").unwrap_or(false); AWS_LC_SYS_NO_ASM = env_var_to_bool("AWS_LC_SYS_NO_ASM").unwrap_or(false); + AWS_LC_SYS_PREBUILT_NASM = env_var_to_bool("AWS_LC_SYS_PREBUILT_NASM"); } if !is_external_bindgen() && (is_internal_bindgen() || !has_bindgen_feature()) { @@ -363,6 +365,7 @@ fn is_bindgen_required() -> bool { || !has_pregenerated() } +#[allow(dead_code)] fn internal_bindgen_supported() -> bool { // TODO: internal bindgen creates invalid bindings on FreeBSD // See: aws#476 @@ -385,6 +388,10 @@ fn is_no_asm() -> bool { unsafe { AWS_LC_SYS_NO_ASM } } +fn allow_prebuilt_nasm() -> Option<bool> { + unsafe { AWS_LC_SYS_PREBUILT_NASM } +} + fn has_bindgen_feature() -> bool { cfg!(feature = "bindgen") } @@ -393,6 +400,10 @@ fn has_pregenerated() -> bool { unsafe { PREGENERATED } } +fn test_nasm_command() -> bool { + execute_command("nasm".as_ref(), &["-version".as_ref()]).status +} + fn prepare_cargo_cfg() { // This is supported in Rust >= 1.77.0 // Also remove `#![allow(unexpected_cfgs)]` from src/lib.rs diff --git a/aws-lc-sys/builder/prebuilt-nasm.bat b/aws-lc-sys/builder/prebuilt-nasm.bat new file mode 100644 index 0000000000..9c761db12a --- /dev/null +++ b/aws-lc-sys/builder/prebuilt-nasm.bat @@ -0,0 +1,21 @@ +@echo off +set "ScriptDir=%~dp0" +set "ScriptDir=%ScriptDir:~0,-1%" +:loop +set "arg1=%~1" +if "%arg1%"=="-o" goto end +if "%arg1%"=="" goto failure +shift +goto loop +:end +shift +set "path=%~1" +for %%f in ("%path%") do set "filename=%%~nxf" +copy "%ScriptDir%\prebuilt-nasm\%filename%" "%path%" +exit 0 + +:failure +echo PATH: %path% 1>&2 +echo FILENAME: %filename% 1>&2 +echo ScriptDir: %ScriptDir% 1>&2 +exit 1 \ No newline at end of file diff --git a/scripts/build/collect_nasm_obj.sh b/scripts/build/collect_nasm_obj.sh new file mode 100644 index 0000000000..c425e9236f --- /dev/null +++ b/scripts/build/collect_nasm_obj.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 OR ISC + +set -ex +set -o pipefail + +if [[ ${BASH_VERSINFO[0]} -lt 4 ]]; then + echo Must use bash 4 or later: ${BASH_VERSION} + exit 1 +fi + +SCRIPT_DIR=$(cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd) +REPO_ROOT=$(git rev-parse --show-toplevel) +SYS_CRATE_DIR="${REPO_ROOT}/aws-lc-sys" +PREBUILT_NASM_DIR="${SYS_CRATE_DIR}/builder/prebuilt-nasm" +mkdir -p "${PREBUILT_NASM_DIR}" +rm -f "${PREBUILT_NASM_DIR}"/* + +DUMPBIN="$(find /c/Program\ Files/Microsoft\ Visual\ Studio/ -path "*/Hostx64/x64/*" -name "dumpbin.exe" -print -quit)" + +for nasm_file in `find aws-lc-sys/aws-lc/generated-src/win-x86_64/ -name "*.asm"`; do + OBJNAME=$(basename "${nasm_file}"); + NASM_OBJ=$(find target/ -name "${OBJNAME/.asm/.obj}"); + cp "${NASM_OBJ}" "${PREBUILT_NASM_DIR}" + # We remove the '.debug$S' value, which indicates the size of the debug section. This value can change across builds + # because it typically contains full source file paths that vary by build environment + "${DUMPBIN}" //DISASM "${PREBUILT_NASM_DIR}"/"${OBJNAME/.asm/.obj}" | grep -v '.debug$S' | sed -e "s/^Dump of file.*/Dump of file ${OBJNAME/.asm/.obj}/" > "${PREBUILT_NASM_DIR}"/"${OBJNAME/.asm/}"-disasm.txt +done

@echo

Update CI diff --git a/.github/workflows/sys-bindings-generator.yml b/.github/workflows/sys-bindings-generator.yml index 3fba893d80..dc226f9f1b 100644 --- a/.github/workflows/sys-bindings-generator.yml +++ b/.github/workflows/sys-bindings-generator.yml @@ -270,3 +270,36 @@ jobs: run: ./scripts/build/collect_build_src.sh -t ${{ matrix.target }} - name: Commit & Push changes run: ./scripts/ci/ci_add_commit_rebase_push.sh "Collected source files for ${{ matrix.target }}" + collect-nasm-and-commit: + needs: generate-windows-bindings-and-commit + if: github.repository == 'aws/aws-lc-rs' + runs-on: windows-latest + steps: + - uses: actions/checkout@v4 + with: + submodules: 'recursive' + ref: ${{ github.ref_name }} + - uses: dtolnay/rust-toolchain@master + id: toolchain + with: + toolchain: stable + targets: "x86_64-pc-windows-msvc,x86_64-pc-windows-gnu" + - uses: ilammy/setup-nasm@v1 + - name: Build aws-lc-sys + shell: bash + run: AWS_LC_SYS_PREBUILT_NASM=0 cargo build -p aws-lc-sys --release --target x86_64-pc-windows-msvc + - name: Collect NASM object files + shell: bash + run: ./scripts/build/collect_nasm_obj.sh + - name: Clean build + shell: bash + run: cargo clean + - name: Test aws-lc-rs for x86_64-pc-windows-msvc + shell: bash + run: AWS_LC_SYS_PREBUILT_NASM=1 cargo build -p aws-lc-sys --target x86_64-pc-windows-msvc + - name: Test aws-lc-sys for x86_64-pc-windows-gnu + shell: bash + run: AWS_LC_SYS_PREBUILT_NASM=1 cargo build -p aws-lc-sys --target x86_64-pc-windows-gnu + - name: Commit & Push changes + shell: bash + run: ./scripts/ci/ci_add_commit_rebase_push.sh "Collected NASM files" diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index a0da3f48b8..71716a64c0 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -37,7 +37,7 @@ jobs: - --no-default-features --features non-fips,ring-sig-verify,unstable - --no-default-features --features non-fips,alloc,unstable steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@master @@ -66,7 +66,7 @@ jobs: - --no-default-features --features aws-lc-sys,bindgen,unstable - --release --all-targets --features bindgen,unstable steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - if: ${{ matrix.os == 'macos-13-xlarge' }} @@ -106,9 +106,10 @@ jobs: - --no-default-features --features non-fips,ring-io,unstable - --no-default-features --features non-fips,ring-sig-verify,unstable - --no-default-features --features non-fips,alloc,unstable + env: + AWS_LC_SYS_PREBUILT_NASM: 1 steps: - - uses: ilammy/setup-nasm@v1 - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@master @@ -125,7 +126,7 @@ jobs: name: aws-ls-rs coverage runs-on: ubuntu-latest steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' lfs: true @@ -167,7 +168,7 @@ jobs: - --no-default-features --features fips,asan runs-on: ubuntu-latest steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@master @@ -196,7 +197,7 @@ jobs: os: [ ubuntu-latest, macos-12, macos-13-xlarge ] static: [ 0, 1 ] steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@stable @@ -205,6 +206,40 @@ jobs: # See: rust-lang/cargo#8531 run: cargo test -p aws-lc-rs --tests + build-env-nasm-test: + if: github.repository_owner == 'aws' + name: prebuilt NASM verification + runs-on: windows-latest + strategy: + fail-fast: false + matrix: + target: + - 'x86_64-pc-windows-msvc' + - 'x86_64-pc-windows-gnu' + steps: + - uses: actions/checkout@v4 + with: + submodules: 'recursive' + - uses: dtolnay/rust-toolchain@stable + - name: Install NASM + uses: ilammy/setup-nasm@v1 + - name: Remove NASM artifacts + shell: bash + run: | + cargo clean + rm ./aws-lc-sys/builder/prebuilt-nasm/* + - name: Run cargo test + shell: bash + run: AWS_LC_SYS_PREBUILT_NASM=0 cargo test --tests -p aws-lc-rs --release --no-default-features --features aws-lc-sys + - name: Collect NASM outputs + shell: bash + run: ./scripts/build/collect_nasm_obj.sh + - name: Flag any NASM changes + shell: bash + run: | + git add . + git diff --cached --exit-code HEAD -- aws-lc-sys/builder/prebuilt-nasm/*.txt + build-env-external-bindgen-test: if: github.repository_owner == 'aws' name: aws-lc-rs FIPS - External bindgen test @@ -216,14 +251,14 @@ jobs: matrix: os: [ ubuntu-latest, macos-12, macos-13-xlarge, windows-latest ] steps: - - if: ${{ matrix.os == 'windows-latest' }} - uses: ilammy/setup-nasm@v1 - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@stable - - name: Install ninja-build tool - uses: seanmiddleditch/gha-setup-ninja@v4 + - if: ${{ matrix.os == 'windows-latest' }} + uses: ilammy/setup-nasm@v1 + - if: ${{ matrix.os == 'windows-latest' }} + uses: seanmiddleditch/gha-setup-ninja@v5 - name: Install bindgen-cli run: cargo install --locked bindgen-cli - name: Remove bindings @@ -245,12 +280,10 @@ jobs: os: [ ubuntu-latest, macos-12, macos-13-xlarge ] static: [ 0, 1 ] steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@stable - - name: Install ninja-build tool - uses: seanmiddleditch/gha-setup-ninja@v4 - uses: actions/setup-go@v4 with: go-version: '>=1.18' @@ -271,7 +304,7 @@ jobs: matrix: os: [ ubuntu-latest, macos-12, macos-13-xlarge, windows-latest ] steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@stable @@ -306,15 +339,14 @@ jobs: matrix: os: [ ubuntu-latest, macos-12, macos-13-xlarge, windows-latest ] steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@stable - - name: Install ninja-build tool - uses: seanmiddleditch/gha-setup-ninja@v4 - uses: actions/setup-go@v4 with: go-version: '>=1.18' + - uses: seanmiddleditch/gha-setup-ninja@v5 - name: Run cargo test run: cargo test -p aws-lc-rs --tests --no-default-features --features fips - name: Release build @@ -346,14 +378,14 @@ jobs: matrix: os: [ ubuntu-latest, macos-12, macos-13-xlarge, windows-latest ] steps: - - if: ${{ matrix.os == 'windows-latest' }} - uses: ilammy/setup-nasm@v1 - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' + - if: ${{ matrix.os == 'windows-latest' }} + uses: ilammy/setup-nasm@v1 + - if: ${{ matrix.os == 'windows-latest' }} + uses: seanmiddleditch/gha-setup-ninja@v5 - uses: dtolnay/rust-toolchain@stable - - name: Install ninja-build tool - uses: seanmiddleditch/gha-setup-ninja@v4 - uses: actions/setup-go@v4 with: go-version: '>=1.18' @@ -378,7 +410,7 @@ jobs: - macos-12 - macos-13-xlarge steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' lfs: true diff --git a/aws-lc-sys/builder/cmake_builder.rs b/aws-lc-sys/builder/cmake_builder.rs index e25cecb443..c5069c5341 100644 --- a/aws-lc-sys/builder/cmake_builder.rs +++ b/aws-lc-sys/builder/cmake_builder.rs @@ -3,9 +3,9 @@ use crate::OutputLib::{Crypto, RustWrapper, Ssl}; use crate::{ - cargo_env, emit_warning, execute_command, is_crt_static, is_no_asm, option_env, target, - target_arch, target_env, target_family, target_os, target_underscored, target_vendor, - OutputLibType, + allow_prebuilt_nasm, cargo_env, emit_warning, execute_command, is_crt_static, is_no_asm, + option_env, target, target_arch, target_env, target_family, target_os, target_underscored, + target_vendor, test_nasm_command, OutputLibType, }; use std::env; use std::ffi::OsString; @@ -22,10 +22,6 @@ fn test_clang_cl_command() -> bool { execute_command("clang-cl".as_ref(), &["--version".as_ref()]).status } -fn test_nasm_command() -> bool { - execute_command("nasm".as_ref(), &["-version".as_ref()]).status -} - fn find_cmake_command() -> Option<OsString> { if let Some(cmake) = option_env("CMAKE") { emit_warning(&format!( @@ -162,7 +158,7 @@ impl CmakeBuilder { // See issue: #453 if target_os() == "windows" { - Self::configure_windows(&mut cmake_cfg); + self.configure_windows(&mut cmake_cfg); } // If the build environment vendor is Apple @@ -213,7 +209,7 @@ impl CmakeBuilder { cmake_cfg } - fn configure_windows(cmake_cfg: &mut cmake::Config) { + fn configure_windows(&self, cmake_cfg: &mut cmake::Config) { match (target_env().as_str(), target_arch().as_str()) { ("msvc", "aarch64") => { cmake_cfg.generator_toolset(format!( @@ -243,6 +239,22 @@ impl CmakeBuilder { } _ => {} } + if target_arch() == "x86_64" && Some(true) == allow_prebuilt_nasm() { + emit_warning("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"); + emit_warning("!!! Using pre-built NASM binaries !!!"); + emit_warning("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"); + + let script_path = self + .manifest_dir + .join("builder") + .join("prebuilt-nasm.bat") + .display() + .to_string(); + let script_path = script_path.replace('\\', "/"); + + cmake_cfg.define("CMAKE_ASM_NASM_COMPILER", script_path.as_str()); + cmake_cfg.define("CMAKE_VERBOSE_MAKEFILE", "1"); + } } fn configure_open_harmony(cmake_cfg: &mut cmake::Config) { @@ -298,10 +310,14 @@ impl crate::Builder for CmakeBuilder { let mut missing_dependency = false; if target_os() == "windows" { - if target_arch() == "x86_64" && !test_nasm_command() && !is_no_asm() { + if target_arch() == "x86_64" + && !is_no_asm() + && !test_nasm_command() + && Some(true) != allow_prebuilt_nasm() + { eprintln!( - "Consider setting `AWS_LC_SYS_NO_ASM` in the environment for development builds.\ - See User Guide about the limitations: https://aws.github.io/aws-lc-rs/index.html" + "Consider setting `AWS_LC_SYS_PREBUILT_NASM` in the build environment.\ + See User Guide: https://aws.github.io/aws-lc-rs/index.html" ); eprintln!("Missing dependency: nasm"); missing_dependency = true; diff --git a/aws-lc-sys/builder/main.rs b/aws-lc-sys/builder/main.rs index 23f0214991..987e1038ca 100644 --- a/aws-lc-sys/builder/main.rs +++ b/aws-lc-sys/builder/main.rs @@ -319,6 +319,7 @@ static mut AWS_LC_SYS_NO_PREFIX: bool = false; static mut AWS_LC_SYS_INTERNAL_BINDGEN: bool = false; static mut AWS_LC_SYS_EXTERNAL_BINDGEN: bool = false; static mut AWS_LC_SYS_NO_ASM: bool = false; +static mut AWS_LC_SYS_PREBUILT_NASM: Option<bool> = None; fn initialize() { unsafe { @@ -328,6 +329,7 @@ fn initialize() { AWS_LC_SYS_EXTERNAL_BINDGEN = env_var_to_bool("AWS_LC_SYS_EXTERNAL_BINDGEN").unwrap_or(false); AWS_LC_SYS_NO_ASM = env_var_to_bool("AWS_LC_SYS_NO_ASM").unwrap_or(false); + AWS_LC_SYS_PREBUILT_NASM = env_var_to_bool("AWS_LC_SYS_PREBUILT_NASM"); } if !is_external_bindgen() && (is_internal_bindgen() || !has_bindgen_feature()) { @@ -363,6 +365,7 @@ fn is_bindgen_required() -> bool { || !has_pregenerated() } +#[allow(dead_code)] fn internal_bindgen_supported() -> bool { // TODO: internal bindgen creates invalid bindings on FreeBSD // See: #476 @@ -385,6 +388,10 @@ fn is_no_asm() -> bool { unsafe { AWS_LC_SYS_NO_ASM } } +fn allow_prebuilt_nasm() -> Option<bool> { + unsafe { AWS_LC_SYS_PREBUILT_NASM } +} + fn has_bindgen_feature() -> bool { cfg!(feature = "bindgen") } @@ -393,6 +400,10 @@ fn has_pregenerated() -> bool { unsafe { PREGENERATED } } +fn test_nasm_command() -> bool { + execute_command("nasm".as_ref(), &["-version".as_ref()]).status +} + fn prepare_cargo_cfg() { // This is supported in Rust >= 1.77.0 // Also remove `#![allow(unexpected_cfgs)]` from src/lib.rs diff --git a/aws-lc-sys/builder/prebuilt-nasm.bat b/aws-lc-sys/builder/prebuilt-nasm.bat new file mode 100644 index 0000000000..9c761db12a --- /dev/null +++ b/aws-lc-sys/builder/prebuilt-nasm.bat @@ -0,0 +1,21 @@ +@echo off +set "ScriptDir=%~dp0" +set "ScriptDir=%ScriptDir:~0,-1%" +:loop +set "arg1=%~1" +if "%arg1%"=="-o" goto end +if "%arg1%"=="" goto failure +shift +goto loop +:end +shift +set "path=%~1" +for %%f in ("%path%") do set "filename=%%~nxf" +copy "%ScriptDir%\prebuilt-nasm\%filename%" "%path%" +exit 0 + +:failure +echo PATH: %path% 1>&2 +echo FILENAME: %filename% 1>&2 +echo ScriptDir: %ScriptDir% 1>&2 +exit 1 \ No newline at end of file diff --git a/scripts/build/collect_nasm_obj.sh b/scripts/build/collect_nasm_obj.sh new file mode 100644 index 0000000000..c425e9236f --- /dev/null +++ b/scripts/build/collect_nasm_obj.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 OR ISC + +set -ex +set -o pipefail + +if [[ ${BASH_VERSINFO[0]} -lt 4 ]]; then + echo Must use bash 4 or later: ${BASH_VERSION} + exit 1 +fi + +SCRIPT_DIR=$(cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd) +REPO_ROOT=$(git rev-parse --show-toplevel) +SYS_CRATE_DIR="${REPO_ROOT}/aws-lc-sys" +PREBUILT_NASM_DIR="${SYS_CRATE_DIR}/builder/prebuilt-nasm" +mkdir -p "${PREBUILT_NASM_DIR}" +rm -f "${PREBUILT_NASM_DIR}"/* + +DUMPBIN="$(find /c/Program\ Files/Microsoft\ Visual\ Studio/ -path "*/Hostx64/x64/*" -name "dumpbin.exe" -print -quit)" + +for nasm_file in `find aws-lc-sys/aws-lc/generated-src/win-x86_64/ -name "*.asm"`; do + OBJNAME=$(basename "${nasm_file}"); + NASM_OBJ=$(find target/ -name "${OBJNAME/.asm/.obj}"); + cp "${NASM_OBJ}" "${PREBUILT_NASM_DIR}" + # We remove the '.debug$S' value, which indicates the size of the debug section. This value can change across builds + # because it typically contains full source file paths that vary by build environment + "${DUMPBIN}" //DISASM "${PREBUILT_NASM_DIR}"/"${OBJNAME/.asm/.obj}" | grep -v '.debug$S' | sed -e "s/^Dump of file.*/Dump of file ${OBJNAME/.asm/.obj}/" > "${PREBUILT_NASM_DIR}"/"${OBJNAME/.asm/}"-disasm.txt +done

@echo

Update CI diff --git a/.github/workflows/sys-bindings-generator.yml b/.github/workflows/sys-bindings-generator.yml index 3fba893d80..dc226f9f1b 100644 --- a/.github/workflows/sys-bindings-generator.yml +++ b/.github/workflows/sys-bindings-generator.yml @@ -270,3 +270,36 @@ jobs: run: ./scripts/build/collect_build_src.sh -t ${{ matrix.target }} - name: Commit & Push changes run: ./scripts/ci/ci_add_commit_rebase_push.sh "Collected source files for ${{ matrix.target }}" + collect-nasm-and-commit: + needs: generate-windows-bindings-and-commit + if: github.repository == 'aws/aws-lc-rs' + runs-on: windows-latest + steps: + - uses: actions/checkout@v4 + with: + submodules: 'recursive' + ref: ${{ github.ref_name }} + - uses: dtolnay/rust-toolchain@master + id: toolchain + with: + toolchain: stable + targets: "x86_64-pc-windows-msvc,x86_64-pc-windows-gnu" + - uses: ilammy/setup-nasm@v1 + - name: Build aws-lc-sys + shell: bash + run: AWS_LC_SYS_PREBUILT_NASM=0 cargo build -p aws-lc-sys --release --target x86_64-pc-windows-msvc + - name: Collect NASM object files + shell: bash + run: ./scripts/build/collect_nasm_obj.sh + - name: Clean build + shell: bash + run: cargo clean + - name: Test aws-lc-rs for x86_64-pc-windows-msvc + shell: bash + run: AWS_LC_SYS_PREBUILT_NASM=1 cargo build -p aws-lc-sys --target x86_64-pc-windows-msvc + - name: Test aws-lc-sys for x86_64-pc-windows-gnu + shell: bash + run: AWS_LC_SYS_PREBUILT_NASM=1 cargo build -p aws-lc-sys --target x86_64-pc-windows-gnu + - name: Commit & Push changes + shell: bash + run: ./scripts/ci/ci_add_commit_rebase_push.sh "Collected NASM files" diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index a0da3f48b8..71716a64c0 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -37,7 +37,7 @@ jobs: - --no-default-features --features non-fips,ring-sig-verify,unstable - --no-default-features --features non-fips,alloc,unstable steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@master @@ -66,7 +66,7 @@ jobs: - --no-default-features --features aws-lc-sys,bindgen,unstable - --release --all-targets --features bindgen,unstable steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - if: ${{ matrix.os == 'macos-13-xlarge' }} @@ -106,9 +106,10 @@ jobs: - --no-default-features --features non-fips,ring-io,unstable - --no-default-features --features non-fips,ring-sig-verify,unstable - --no-default-features --features non-fips,alloc,unstable + env: + AWS_LC_SYS_PREBUILT_NASM: 1 steps: - - uses: ilammy/setup-nasm@v1 - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@master @@ -125,7 +126,7 @@ jobs: name: aws-ls-rs coverage runs-on: ubuntu-latest steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' lfs: true @@ -167,7 +168,7 @@ jobs: - --no-default-features --features fips,asan runs-on: ubuntu-latest steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@master @@ -196,7 +197,7 @@ jobs: os: [ ubuntu-latest, macos-12, macos-13-xlarge ] static: [ 0, 1 ] steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@stable @@ -205,6 +206,40 @@ jobs: # See: rust-lang/cargo#8531 run: cargo test -p aws-lc-rs --tests + build-env-nasm-test: + if: github.repository_owner == 'aws' + name: prebuilt NASM verification + runs-on: windows-latest + strategy: + fail-fast: false + matrix: + target: + - 'x86_64-pc-windows-msvc' + - 'x86_64-pc-windows-gnu' + steps: + - uses: actions/checkout@v4 + with: + submodules: 'recursive' + - uses: dtolnay/rust-toolchain@stable + - name: Install NASM + uses: ilammy/setup-nasm@v1 + - name: Remove NASM artifacts + shell: bash + run: | + cargo clean + rm ./aws-lc-sys/builder/prebuilt-nasm/* + - name: Run cargo test + shell: bash + run: AWS_LC_SYS_PREBUILT_NASM=0 cargo test --tests -p aws-lc-rs --release --no-default-features --features aws-lc-sys + - name: Collect NASM outputs + shell: bash + run: ./scripts/build/collect_nasm_obj.sh + - name: Flag any NASM changes + shell: bash + run: | + git add . + git diff --cached --exit-code HEAD -- aws-lc-sys/builder/prebuilt-nasm/*.txt + build-env-external-bindgen-test: if: github.repository_owner == 'aws' name: aws-lc-rs FIPS - External bindgen test @@ -216,14 +251,14 @@ jobs: matrix: os: [ ubuntu-latest, macos-12, macos-13-xlarge, windows-latest ] steps: - - if: ${{ matrix.os == 'windows-latest' }} - uses: ilammy/setup-nasm@v1 - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@stable - - name: Install ninja-build tool - uses: seanmiddleditch/gha-setup-ninja@v4 + - if: ${{ matrix.os == 'windows-latest' }} + uses: ilammy/setup-nasm@v1 + - if: ${{ matrix.os == 'windows-latest' }} + uses: seanmiddleditch/gha-setup-ninja@v5 - name: Install bindgen-cli run: cargo install --locked bindgen-cli - name: Remove bindings @@ -245,12 +280,10 @@ jobs: os: [ ubuntu-latest, macos-12, macos-13-xlarge ] static: [ 0, 1 ] steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@stable - - name: Install ninja-build tool - uses: seanmiddleditch/gha-setup-ninja@v4 - uses: actions/setup-go@v4 with: go-version: '>=1.18' @@ -271,7 +304,7 @@ jobs: matrix: os: [ ubuntu-latest, macos-12, macos-13-xlarge, windows-latest ] steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@stable @@ -306,15 +339,14 @@ jobs: matrix: os: [ ubuntu-latest, macos-12, macos-13-xlarge, windows-latest ] steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@stable - - name: Install ninja-build tool - uses: seanmiddleditch/gha-setup-ninja@v4 - uses: actions/setup-go@v4 with: go-version: '>=1.18' + - uses: seanmiddleditch/gha-setup-ninja@v5 - name: Run cargo test run: cargo test -p aws-lc-rs --tests --no-default-features --features fips - name: Release build @@ -346,14 +378,14 @@ jobs: matrix: os: [ ubuntu-latest, macos-12, macos-13-xlarge, windows-latest ] steps: - - if: ${{ matrix.os == 'windows-latest' }} - uses: ilammy/setup-nasm@v1 - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' + - if: ${{ matrix.os == 'windows-latest' }} + uses: ilammy/setup-nasm@v1 + - if: ${{ matrix.os == 'windows-latest' }} + uses: seanmiddleditch/gha-setup-ninja@v5 - uses: dtolnay/rust-toolchain@stable - - name: Install ninja-build tool - uses: seanmiddleditch/gha-setup-ninja@v4 - uses: actions/setup-go@v4 with: go-version: '>=1.18' @@ -378,7 +410,7 @@ jobs: - macos-12 - macos-13-xlarge steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' lfs: true diff --git a/aws-lc-sys/builder/cmake_builder.rs b/aws-lc-sys/builder/cmake_builder.rs index e25cecb443..c5069c5341 100644 --- a/aws-lc-sys/builder/cmake_builder.rs +++ b/aws-lc-sys/builder/cmake_builder.rs @@ -3,9 +3,9 @@ use crate::OutputLib::{Crypto, RustWrapper, Ssl}; use crate::{ - cargo_env, emit_warning, execute_command, is_crt_static, is_no_asm, option_env, target, - target_arch, target_env, target_family, target_os, target_underscored, target_vendor, - OutputLibType, + allow_prebuilt_nasm, cargo_env, emit_warning, execute_command, is_crt_static, is_no_asm, + option_env, target, target_arch, target_env, target_family, target_os, target_underscored, + target_vendor, test_nasm_command, OutputLibType, }; use std::env; use std::ffi::OsString; @@ -22,10 +22,6 @@ fn test_clang_cl_command() -> bool { execute_command("clang-cl".as_ref(), &["--version".as_ref()]).status } -fn test_nasm_command() -> bool { - execute_command("nasm".as_ref(), &["-version".as_ref()]).status -} - fn find_cmake_command() -> Option<OsString> { if let Some(cmake) = option_env("CMAKE") { emit_warning(&format!( @@ -162,7 +158,7 @@ impl CmakeBuilder { // See issue: aws#453 if target_os() == "windows" { - Self::configure_windows(&mut cmake_cfg); + self.configure_windows(&mut cmake_cfg); } // If the build environment vendor is Apple @@ -213,7 +209,7 @@ impl CmakeBuilder { cmake_cfg } - fn configure_windows(cmake_cfg: &mut cmake::Config) { + fn configure_windows(&self, cmake_cfg: &mut cmake::Config) { match (target_env().as_str(), target_arch().as_str()) { ("msvc", "aarch64") => { cmake_cfg.generator_toolset(format!( @@ -243,6 +239,22 @@ impl CmakeBuilder { } _ => {} } + if target_arch() == "x86_64" && Some(true) == allow_prebuilt_nasm() { + emit_warning("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"); + emit_warning("!!! Using pre-built NASM binaries !!!"); + emit_warning("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"); + + let script_path = self + .manifest_dir + .join("builder") + .join("prebuilt-nasm.bat") + .display() + .to_string(); + let script_path = script_path.replace('\\', "/"); + + cmake_cfg.define("CMAKE_ASM_NASM_COMPILER", script_path.as_str()); + cmake_cfg.define("CMAKE_VERBOSE_MAKEFILE", "1"); + } } fn configure_open_harmony(cmake_cfg: &mut cmake::Config) { @@ -298,10 +310,14 @@ impl crate::Builder for CmakeBuilder { let mut missing_dependency = false; if target_os() == "windows" { - if target_arch() == "x86_64" && !test_nasm_command() && !is_no_asm() { + if target_arch() == "x86_64" + && !is_no_asm() + && !test_nasm_command() + && Some(true) != allow_prebuilt_nasm() + { eprintln!( - "Consider setting `AWS_LC_SYS_NO_ASM` in the environment for development builds.\ - See User Guide about the limitations: https://aws.github.io/aws-lc-rs/index.html" + "Consider setting `AWS_LC_SYS_PREBUILT_NASM` in the build environment.\ + See User Guide: https://aws.github.io/aws-lc-rs/index.html" ); eprintln!("Missing dependency: nasm"); missing_dependency = true; diff --git a/aws-lc-sys/builder/main.rs b/aws-lc-sys/builder/main.rs index 23f0214991..987e1038ca 100644 --- a/aws-lc-sys/builder/main.rs +++ b/aws-lc-sys/builder/main.rs @@ -319,6 +319,7 @@ static mut AWS_LC_SYS_NO_PREFIX: bool = false; static mut AWS_LC_SYS_INTERNAL_BINDGEN: bool = false; static mut AWS_LC_SYS_EXTERNAL_BINDGEN: bool = false; static mut AWS_LC_SYS_NO_ASM: bool = false; +static mut AWS_LC_SYS_PREBUILT_NASM: Option<bool> = None; fn initialize() { unsafe { @@ -328,6 +329,7 @@ fn initialize() { AWS_LC_SYS_EXTERNAL_BINDGEN = env_var_to_bool("AWS_LC_SYS_EXTERNAL_BINDGEN").unwrap_or(false); AWS_LC_SYS_NO_ASM = env_var_to_bool("AWS_LC_SYS_NO_ASM").unwrap_or(false); + AWS_LC_SYS_PREBUILT_NASM = env_var_to_bool("AWS_LC_SYS_PREBUILT_NASM"); } if !is_external_bindgen() && (is_internal_bindgen() || !has_bindgen_feature()) { @@ -363,6 +365,7 @@ fn is_bindgen_required() -> bool { || !has_pregenerated() } +#[allow(dead_code)] fn internal_bindgen_supported() -> bool { // TODO: internal bindgen creates invalid bindings on FreeBSD // See: aws#476 @@ -385,6 +388,10 @@ fn is_no_asm() -> bool { unsafe { AWS_LC_SYS_NO_ASM } } +fn allow_prebuilt_nasm() -> Option<bool> { + unsafe { AWS_LC_SYS_PREBUILT_NASM } +} + fn has_bindgen_feature() -> bool { cfg!(feature = "bindgen") } @@ -393,6 +400,10 @@ fn has_pregenerated() -> bool { unsafe { PREGENERATED } } +fn test_nasm_command() -> bool { + execute_command("nasm".as_ref(), &["-version".as_ref()]).status +} + fn prepare_cargo_cfg() { // This is supported in Rust >= 1.77.0 // Also remove `#![allow(unexpected_cfgs)]` from src/lib.rs diff --git a/aws-lc-sys/builder/prebuilt-nasm.bat b/aws-lc-sys/builder/prebuilt-nasm.bat new file mode 100644 index 0000000000..9c761db12a --- /dev/null +++ b/aws-lc-sys/builder/prebuilt-nasm.bat @@ -0,0 +1,21 @@ +@echo off +set "ScriptDir=%~dp0" +set "ScriptDir=%ScriptDir:~0,-1%" +:loop +set "arg1=%~1" +if "%arg1%"=="-o" goto end +if "%arg1%"=="" goto failure +shift +goto loop +:end +shift +set "path=%~1" +for %%f in ("%path%") do set "filename=%%~nxf" +copy "%ScriptDir%\prebuilt-nasm\%filename%" "%path%" +exit 0 + +:failure +echo PATH: %path% 1>&2 +echo FILENAME: %filename% 1>&2 +echo ScriptDir: %ScriptDir% 1>&2 +exit 1 \ No newline at end of file diff --git a/scripts/build/collect_nasm_obj.sh b/scripts/build/collect_nasm_obj.sh new file mode 100644 index 0000000000..c425e9236f --- /dev/null +++ b/scripts/build/collect_nasm_obj.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 OR ISC + +set -ex +set -o pipefail + +if [[ ${BASH_VERSINFO[0]} -lt 4 ]]; then + echo Must use bash 4 or later: ${BASH_VERSION} + exit 1 +fi + +SCRIPT_DIR=$(cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd) +REPO_ROOT=$(git rev-parse --show-toplevel) +SYS_CRATE_DIR="${REPO_ROOT}/aws-lc-sys" +PREBUILT_NASM_DIR="${SYS_CRATE_DIR}/builder/prebuilt-nasm" +mkdir -p "${PREBUILT_NASM_DIR}" +rm -f "${PREBUILT_NASM_DIR}"/* + +DUMPBIN="$(find /c/Program\ Files/Microsoft\ Visual\ Studio/ -path "*/Hostx64/x64/*" -name "dumpbin.exe" -print -quit)" + +for nasm_file in `find aws-lc-sys/aws-lc/generated-src/win-x86_64/ -name "*.asm"`; do + OBJNAME=$(basename "${nasm_file}"); + NASM_OBJ=$(find target/ -name "${OBJNAME/.asm/.obj}"); + cp "${NASM_OBJ}" "${PREBUILT_NASM_DIR}" + # We remove the '.debug$S' value, which indicates the size of the debug section. This value can change across builds + # because it typically contains full source file paths that vary by build environment + "${DUMPBIN}" //DISASM "${PREBUILT_NASM_DIR}"/"${OBJNAME/.asm/.obj}" | grep -v '.debug$S' | sed -e "s/^Dump of file.*/Dump of file ${OBJNAME/.asm/.obj}/" > "${PREBUILT_NASM_DIR}"/"${OBJNAME/.asm/}"-disasm.txt +done

adamcrume added the C-bug Category: bug label Jul 23, 2020

ehuss added the A-doctests Area: rustdoc --test label Jul 23, 2020

adamcrume mentioned this issue Oct 15, 2020

External protos tensorflow/rust#278

Closed

scott-wilson mentioned this issue May 15, 2021

Wrap the imf_version module vfx-rs/openexr-bind#20

Merged

22 tasks

scott-wilson mentioned this issue Jun 16, 2021

Tracking "cargo test --doc not load same shared libraries as cargo test --lib" vfx-rs/openexr-bind#33

Open

jackgerrits added a commit to jackgerrits/vowpalwabbit-rs that referenced this issue Aug 5, 2021

Turn off doctest due to bug in tooling rust-lang/cargo#8531

1c3c339

sgrif mentioned this issue Mar 9, 2022

Properly set dylib search path for doctests #10469

Closed

Trolldemorted mentioned this issue May 10, 2022

Use precompiled z3 libs by default prove-rs/z3.rs#193

Closed

weihanglo added S-needs-mentor Status: Issue or feature is accepted, but needs a team member to commit to helping and reviewing. A-environment-variables Area: environment variables E-medium Experience: Medium and removed A-environment-variables Area: environment variables labels May 14, 2023

justsmth mentioned this issue Jun 28, 2023

Dynamic build for FIPS on non-linux platforms aws/aws-lc-rs#173

Merged

noel2004 mentioned this issue Aug 16, 2023

Doc-test in busmapping has to be disabled scroll-tech/zkevm-circuits#782

Open

yasuo-ozu added a commit to QunaSys/z3.rs that referenced this issue Aug 23, 2023

Implement 'download-z3' flag

57a2477

In some system, `cargo test --doc` may fail due to the bug: rust-lang/cargo#8531

tompscanlan mentioned this issue Sep 20, 2023

test for doctest path problem #8531, and solution #12710

Closed

epage added this to testing-devex backlog Oct 11, 2023

justsmth mentioned this issue Feb 1, 2024

Add 'fips' cargo feature to enable to the aws-lc-rs crypto provider with the AWS-LC FIPS implementations. rustls/rustls#1732

Merged

weihanglo added the I-nominated-to-discuss To be discussed during issue triage on the next Cargo team meeting label Feb 8, 2024

weihanglo mentioned this issue Feb 8, 2024

doctests rust-lang/testing-devex-team#5

Open

ehuss added this to Cargo status tracker Feb 10, 2024

ehuss moved this to Nominated in Cargo status tracker Feb 10, 2024

weihanglo mentioned this issue Feb 26, 2024

fix(doctest): search native libs in build script outputs #13490

Merged

bors closed this as completed in #13490 Feb 26, 2024

github-project-automation bot moved this to Done in testing-devex backlog Feb 26, 2024

ehuss mentioned this issue Jun 6, 2024

rust stable 1.78 using 'x86_64-pc-windows-gnu' host tool to build 'thumbv7em-none-eabihf' arm bare metal target got rust-lld failed rust-lang/rust#125809

Closed

goatgoose mentioned this issue Jun 24, 2024

feat(bindings): Add hyper compatibility crate aws/s2n-tls#4617

Merged

Luthaf mentioned this issue Nov 22, 2024

target/debug/deps folder not propagated in LD_LIBRARY_PATH xd009642/tarpaulin#1297

Closed

brody4hire mentioned this issue Nov 26, 2024

CI: enable doc tests on all CI hosts (and update CI doc test comments) rustls/rustls#2242

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cargo test --doc not load same shared libraries as cargo test --lib #8531

cargo test --doc not load same shared libraries as cargo test --lib #8531

adamcrume commented Jul 23, 2020

thoren-d commented Oct 3, 2020

raskr commented Jan 16, 2021 •

edited

Loading

adamcrume commented Apr 4, 2021

pipehappy1 commented Mar 4, 2022

Trolldemorted commented May 10, 2022 •

edited

Loading

justsmth commented Jun 15, 2022 •

edited

Loading

Trolldemorted commented Sep 13, 2022

adamcrume commented Sep 14, 2022

weihanglo commented Sep 14, 2022

Trolldemorted commented Jun 25, 2023

tompscanlan commented Sep 18, 2023

cargo test --doc not load same shared libraries as cargo test --lib #8531

cargo test --doc not load same shared libraries as cargo test --lib #8531

Comments

adamcrume commented Jul 23, 2020

thoren-d commented Oct 3, 2020

raskr commented Jan 16, 2021 • edited Loading

adamcrume commented Apr 4, 2021

pipehappy1 commented Mar 4, 2022

Trolldemorted commented May 10, 2022 • edited Loading

justsmth commented Jun 15, 2022 • edited Loading

Trolldemorted commented Sep 13, 2022

adamcrume commented Sep 14, 2022

weihanglo commented Sep 14, 2022

Footnotes

Trolldemorted commented Jun 25, 2023

tompscanlan commented Sep 18, 2023

raskr commented Jan 16, 2021 •

edited

Loading

Trolldemorted commented May 10, 2022 •

edited

Loading

justsmth commented Jun 15, 2022 •

edited

Loading