Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix android build of pqcrypto-internals #74

Closed
wants to merge 1 commit into from
Closed

fix android build of pqcrypto-internals #74

wants to merge 1 commit into from

Conversation

vaf-hub
Copy link

@vaf-hub vaf-hub commented Feb 3, 2025

Fix the android build. Clang does not like the sha3 feature to be present in the arch flag when compiling for android.

The android build broke with e3b6f66

See this error.

error occurred in cc-rs: Command LC_ALL="C" "/usr/local/lib/android/sdk/ndk/26.1.10909125/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android26-clang" "-O0" "-DANDROID" "-ffunction-sections" "-fdata-sections" "-fPIC" "-gdwarf-2" "-fno-omit-frame-pointer" "-Wall" "-Wextra" "-march=armv8-a+sha3" "-o" "/home/runner/work/tutanota/tutanota/tuta-sdk/rust/target/aarch64-linux-android/debug/build/pqcrypto-internals-99d56cd060658fdc/out/9cad35ea665b40e6-feat.o" "-c" "cfiles/keccak2x/feat.S" with args aarch64-linux-android26-clang did not execute successfully (status code exit status: 1)

Here's a quick fix, that does the job for us: See: tutao/tutanota#8424
However, I am not sure which other targets require a similar fix.

What do you think @thomwiggers @charlag ?

@vaf-hub vaf-hub mentioned this pull request Feb 3, 2025
Merged
@thomwiggers
Copy link
Member

Could you maybe point me to the error text that Clang gives for feat.S instead of just the error code returned?

@charlag
Copy link
Contributor

charlag commented Feb 4, 2025

Here's an excerpt: 
warning: [email protected]: <instantiation>:1:22: error: unexpected token in argument list
warning: [email protected]: ; Execute theta, but without xoring into the state yet.
warning: [email protected]:                      ^
warning: [email protected]: cfiles/keccak2x/feat.S:147:5: note: while in macro instantiation
warning: [email protected]:     round
warning: [email protected]:     ^
warning: [email protected]: <instantiation>:2:24: error: unexpected token in argument list
warning: [email protected]:     ; Compute parities p[i] = a[i] ^ a[5+i] ^ ... ^ a[20+i].
warning: [email protected]:                        ^
warning: [email protected]: cfiles/keccak2x/feat.S:147:5: note: while in macro instantiation
warning: [email protected]:     round
warning: [email protected]:     ^
warning: [email protected]: <instantiation>:15:34: error: unexpected token in argument list
warning: [email protected]:     rax1.2d v30, v29, v26 ; d[0] = rotl(p[1], 1) ^ p[4]
warning: [email protected]:                                  ^
warning: [email protected]: cfiles/keccak2x/feat.S:147:5: note: while in macro instantiation
warning: [email protected]:     round
warning: [email protected]:     ^
warning: [email protected]: <instantiation>:16:34: error: unexpected token in argument list
warning: [email protected]:     rax1.2d v29, v27, v29 ; d[3] = rotl(p[4], 1) ^ p[2]
warning: [email protected]:                                  ^
warning: [email protected]: cfiles/keccak2x/feat.S:147:5: note: while in macro instantiation
warning: [email protected]:     round
warning: [email protected]:     ^
warning: [email protected]: <instantiation>:17:34: error: unexpected token in argument list
warning: [email protected]:     rax1.2d v27, v25, v27 ; d[1] = rotl(p[2], 1) ^ p[0]
warning: [email protected]:                                  ^
warning: [email protected]: cfiles/keccak2x/feat.S:147:5: note: while in macro instantiation
warning: [email protected]:     round
warning: [email protected]:     ^
warning: [email protected]: <instantiation>:18:34: error: unexpected token in argument list
warning: [email protected]:     rax1.2d v25, v28, v25 ; d[4] = rotl(p[0], 1) ^ p[3]
warning: [email protected]:                                  ^
warning: [email protected]: cfiles/keccak2x/feat.S:147:5: note: while in macro instantiation
warning: [email protected]:     round
warning: [email protected]:     ^
warning: [email protected]: <instantiation>:19:34: error: unexpected token in argument list
warning: [email protected]:     rax1.2d v28, v26, v28 ; d[2] = rotl(p[3], 1) ^ p[1]
warning: [email protected]:                                  ^
warning: [email protected]: cfiles/keccak2x/feat.S:147:5: note: while in macro instantiation
warning: [email protected]:     round
warning: [email protected]:     ^
warning: [email protected]: <instantiation>:21:20: error: unexpected token in argument list
warning: [email protected]:     ; Xor parities from step theta into the state at the same time
warning: [email protected]:                    ^
warning: [email protected]: cfiles/keccak2x/feat.S:147:5: note: while in macro instantiation
warning: [email protected]:     round
warning: [email protected]:     ^
warning: [email protected]: <instantiation>:22:20: error: unexpected token in argument list
warning: [email protected]:     ; as executing rho and pi.
warning: [email protected]:                    ^
warning: [email protected]: cfiles/keccak2x/feat.S:147:5: note: while in macro instantiation
warning: [email protected]:     round
warning: [email protected]:     ^
warning: [email protected]: <instantiation>:50:7: error: unrecognized instruction mnemonic, did you mean: cmhi?
warning: [email protected]:     ; Chi
warning: [email protected]:       ^
warning: [email protected]: cfiles/keccak2x/feat.S:147:5: note: while in macro instantiation
warning: [email protected]:     round
warning: [email protected]:     ^
warning: [email protected]: <instantiation>:91:7: error: unrecognized instruction mnemonic
warning: [email protected]:     ; iota
warning: [email protected]:       ^
warning: [email protected]: cfiles/keccak2x/feat.S:147:5: note: while in macro instantiation
warning: [email protected]:     round
warning: [email protected]:     ^
warning: [email protected]: ToolExecError: Command LC_ALL="C" "/usr/local/lib/android/sdk/ndk/26.1.10909125/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android26-clang" "-O0" "-DANDROID" "-ffunction-sections" "-fdata-sections" "-fPIC" "-gdwarf-2" "-fno-omit-frame-pointer" "-Wall" "-Wextra" "-march=armv8-a+sha3" "-o" "/home/runner/work/tutanota/tutanota/tuta-sdk/rust/target/aarch64-linux-android/debug/build/pqcrypto-internals-99d56cd060658fdc/out/9cad35ea665b40e6-feat.o" "-c" "cfiles/keccak2x/feat.S" with args aarch64-linux-android26-clang did not execute successfully (status code exit status: 1).
error: failed to run custom build command for `pqcrypto-internals v0.2.7 (https://github.com/rustpq/pqcrypto?rev=b39937410b149156de81fc8e3150753aff925aa4#b3993741)`

Caused by:
  process didn't exit successfully: `/home/runner/work/tutanota/tutanota/tuta-sdk/rust/target/debug/build/pqcrypto-internals-85b124472deb57ba/build-script-build` (exit status: 1)
  --- stdout
  cargo:includepath=/home/runner/.cargo/git/checkouts/pqcrypto-e9887127992f2d13/b399374/pqcrypto-internals/include
  OUT_DIR = Some(/home/runner/work/tutanota/tutanota/tuta-sdk/rust/target/aarch64-linux-android/debug/build/pqcrypto-internals-99d56cd060658fdc/out)
  OPT_LEVEL = Some(0)
  TARGET = Some(aarch64-linux-android)
  HOST = Some(x86_64-unknown-linux-gnu)
  cargo:rerun-if-env-changed=CC_aarch64-linux-android
  CC_aarch64-linux-android = Some(/usr/local/lib/android/sdk/ndk/26.1.10909125/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android26-clang)
  RUSTC_WRAPPER = None
  cargo:rerun-if-env-changed=CC_ENABLE_DEBUG_OUTPUT
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some(true)
  cargo:rerun-if-env-changed=CFLAGS_aarch64-linux-android
  CFLAGS_aarch64-linux-android = None
  cargo:rerun-if-env-changed=CFLAGS_aarch64_linux_android
  CFLAGS_aarch64_linux_android = None
  cargo:rerun-if-env-changed=TARGET_CFLAGS
  TARGET_CFLAGS = None
  cargo:rerun-if-env-changed=CFLAGS
  CFLAGS = None
  CARGO_ENCODED_RUSTFLAGS = Some()
  exit status: 0
  exit status: 0
  exit status: 0
  exit status: 0
  exit status: 0
  cargo:rerun-if-env-changed=AR_aarch64-linux-android
  AR_aarch64-linux-android = Some(/usr/local/lib/android/sdk/ndk/26.1.10909125/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-ar)
  cargo:rerun-if-env-changed=ARFLAGS_aarch64-linux-android
  ARFLAGS_aarch64-linux-android = None
  cargo:rerun-if-env-changed=ARFLAGS_aarch64_linux_android
  ARFLAGS_aarch64_linux_android = None
  cargo:rerun-if-env-changed=TARGET_ARFLAGS
  TARGET_ARFLAGS = None
  cargo:rerun-if-env-changed=ARFLAGS
  ARFLAGS = None
  cargo:rustc-link-lib=static=pqclean_common
  cargo:rustc-link-search=native=/home/runner/work/tutanota/tutanota/tuta-sdk/rust/target/aarch64-linux-android/debug/build/pqcrypto-internals-99d56cd060658fdc/out
  cargo:rustc-link-lib=pqclean_common
  OUT_DIR = Some(/home/runner/work/tutanota/tutanota/tuta-sdk/rust/target/aarch64-linux-android/debug/build/pqcrypto-internals-99d56cd060658fdc/out)
  OPT_LEVEL = Some(0)
  TARGET = Some(aarch64-linux-android)
  HOST = Some(x86_64-unknown-linux-gnu)
  cargo:rerun-if-env-changed=CC_aarch64-linux-android
  CC_aarch64-linux-android = Some(/usr/local/lib/android/sdk/ndk/26.1.10909125/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android26-clang)
  RUSTC_WRAPPER = None
  cargo:rerun-if-env-changed=CC_ENABLE_DEBUG_OUTPUT
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some(true)
  cargo:rerun-if-env-changed=CFLAGS_aarch64-linux-android
  CFLAGS_aarch64-linux-android = None
  cargo:rerun-if-env-changed=CFLAGS_aarch64_linux_android
  CFLAGS_aarch64_linux_android = None
  cargo:rerun-if-env-changed=TARGET_CFLAGS
  TARGET_CFLAGS = None
  cargo:rerun-if-env-changed=CFLAGS
  CFLAGS = None
  CARGO_ENCODED_RUSTFLAGS = Some()
  cargo:warning=<instantiation>:1:22: error: unexpected token in argument list
  cargo:warning=; Execute theta, but without xoring into the state yet.
  cargo:warning=                     ^
  cargo:warning=cfiles/keccak2x/feat.S:147:5: note: while in macro instantiation
  cargo:warning=    round
  cargo:warning=    ^
  cargo:warning=<instantiation>:2:24: error: unexpected token in argument list
  cargo:warning=    ; Compute parities p[i] = a[i] ^ a[5+i] ^ ... ^ a[20+i].
  cargo:warning=                       ^
  cargo:warning=cfiles/keccak2x/feat.S:147:5: note: while in macro instantiation
  cargo:warning=    round
  cargo:warning=    ^
  cargo:warning=<instantiation>:15:34: error: unexpected token in argument list
  cargo:warning=    rax1.2d v30, v29, v26 ; d[0] = rotl(p[1], 1) ^ p[4]
  cargo:warning=                                 ^
  cargo:warning=cfiles/keccak2x/feat.S:147:5: note: while in macro instantiation
  cargo:warning=    round
  cargo:warning=    ^
  cargo:warning=<instantiation>:16:34: error: unexpected token in argument list
  cargo:warning=    rax1.2d v29, v27, v29 ; d[3] = rotl(p[4], 1) ^ p[2]
  cargo:warning=                                 ^
  cargo:warning=cfiles/keccak2x/feat.S:147:5: note: while in macro instantiation
  cargo:warning=    round
  cargo:warning=    ^
  cargo:warning=<instantiation>:17:34: error: unexpected token in argument list
  cargo:warning=    rax1.2d v27, v25, v27 ; d[1] = rotl(p[2], 1) ^ p[0]
  cargo:warning=                                 ^
  cargo:warning=cfiles/keccak2x/feat.S:147:5: note: while in macro instantiation
  cargo:warning=    round
  cargo:warning=    ^
  cargo:warning=<instantiation>:18:34: error: unexpected token in argument list
  cargo:warning=    rax1.2d v25, v28, v25 ; d[4] = rotl(p[0], 1) ^ p[3]
  cargo:warning=                                 ^
  cargo:warning=cfiles/keccak2x/feat.S:147:5: note: while in macro instantiation
  cargo:warning=    round
  cargo:warning=    ^
  cargo:warning=<instantiation>:19:34: error: unexpected token in argument list
  cargo:warning=    rax1.2d v28, v26, v28 ; d[2] = rotl(p[3], 1) ^ p[1]
  cargo:warning=                                 ^
  cargo:warning=cfiles/keccak2x/feat.S:147:5: note: while in macro instantiation
  cargo:warning=    round
  cargo:warning=    ^
  cargo:warning=<instantiation>:21:20: error: unexpected token in argument list
  cargo:warning=    ; Xor parities from step theta into the state at the same time
  cargo:warning=                   ^
  cargo:warning=cfiles/keccak2x/feat.S:147:5: note: while in macro instantiation
  cargo:warning=    round
  cargo:warning=    ^
  cargo:warning=<instantiation>:22:20: error: unexpected token in argument list
  cargo:warning=    ; as executing rho and pi.
  cargo:warning=                   ^
  cargo:warning=cfiles/keccak2x/feat.S:147:5: note: while in macro instantiation
  cargo:warning=    round
  cargo:warning=    ^
  cargo:warning=<instantiation>:50:7: error: unrecognized instruction mnemonic, did you mean: cmhi?
  cargo:warning=    ; Chi
  cargo:warning=      ^
  cargo:warning=cfiles/keccak2x/feat.S:147:5: note: while in macro instantiation
  cargo:warning=    round
  cargo:warning=    ^
  cargo:warning=<instantiation>:91:7: error: unrecognized instruction mnemonic
  cargo:warning=    ; iota
  cargo:warning=      ^
  cargo:warning=cfiles/keccak2x/feat.S:147:5: note: while in macro instantiation
  cargo:warning=    round
  cargo:warning=    ^
  exit status: 0
  exit status: 1
  cargo:warning=ToolExecError: Command LC_ALL="C" "/usr/local/lib/android/sdk/ndk/26.1.10909125/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android26-clang" "-O0" "-DANDROID" "-ffunction-sections" "-fdata-sections" "-fPIC" "-gdwarf-2" "-fno-omit-frame-pointer" "-Wall" "-Wextra" "-march=armv8-a+sha3" "-o" "/home/runner/work/tutanota/tutanota/tuta-sdk/rust/target/aarch64-linux-android/debug/build/pqcrypto-internals-99d56cd060658fdc/out/9cad35ea665b40e6-feat.o" "-c" "cfiles/keccak2x/feat.S" with args aarch64-linux-android26-clang did not execute successfully (status code exit status: 1).

  --- stderr


  error occurred in cc-rs: Command LC_ALL="C" "/usr/local/lib/android/sdk/ndk/26.1.10909125/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android26-clang" "-O0" "-DANDROID" "-ffunction-sections" "-fdata-sections" "-fPIC" "-gdwarf-2" "-fno-omit-frame-pointer" "-Wall" "-Wextra" "-march=armv8-a+sha3" "-o" "/home/runner/work/tutanota/tutanota/tuta-sdk/rust/target/aarch64-linux-android/debug/build/pqcrypto-internals-99d56cd060658fdc/out/9cad35ea665b40e6-feat.o" "-c" "cfiles/keccak2x/feat.S" with args aarch64-linux-android26-clang did not execute successfully (status code exit status: 1).


warning: build failed, waiting for other jobs to finish...

FAILURE: Build failed with an exception.

@thomwiggers
Copy link
Member

Ah I see now, building without SHA3 support should lead to a broken binary due to https://github.com/rustpq/pqcrypto/blob/main/pqcrypto-internals/cfiles/keccak2x/feat.S being functionally empty. This is not what we want.

But it appears that the assembler syntax is somehow incompatible with the Android toolchain, so we need to fix that somehow.

@thomwiggers thomwiggers mentioned this pull request Feb 4, 2025
Merged
@vaf-hub
Copy link
Author

vaf-hub commented Feb 5, 2025

Thanks for fixing the underlying problem and releasing it!
I do not think the resulting binary was broken. At least we were able to use ML-KEM (which requires SHA3) successfully with it...

@thomwiggers
Copy link
Member

The problem should have been that if the runtime detection finds SHA3 hardware support, it switches over to the assembly implementation instead of the portable C implementation. That code was commented out (though I'm not sure why it was linking in the first place).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@vaf-hub @thomwiggers @charlag