Add internal APIs for ML-DSA #1999

jakemas · 2024-11-18T20:07:38Z

Issues:

Resolves #CryptoAlg-2723

Description of changes:

This PR adds the internal functions from FIPS 204: Module-Lattice-Based Digital Signature Standard. We base this implementation on the upstream reference implementation of ML-DSA pq-crystals/dilithium@444cdcc. However, the upstream commit only includes implementation of ML-DSA.Sign_internal and ML-DSA.Verify_internal, so we also include ML-DSA.KeyGen_internal to complete the implementation.

Changes:

Implementation of external ML-DSA algorithm functions:
- FIPS 204 Algorithm 1 ML-DSA.KeyGen as crypto_sign_keypair
- FIPS 204 Algorithm 2 ML-DSA.Sign as crypto_sign_signature
- FIPS 204 Algorithm 3 ML-DSA.Verify as crypto_sign_verify
Implementation of internal ML-DSA algorithm functions:
- FIPS 204 Algorithm 6 ML-DSA.KeyGen_internal as crypto_sign_keypair_internal
- FIPS 204 Algorithm 7 ML-DSA.Sign_internal as crypto_sign_signature_internal
- FIPS 204 Algorithm 8 ML-DSA.Verify_internal as crypto_sign_verify_internal
Addition of internal functions to the pqdsa API:
- ml_dsa_65_keypair_internal
- ml_dsa_65_sign_internal
- ml_dsa_65_verify_internal
Addition of KATs from seeds (KATs from https://github.com/post-quantum-cryptography/KAT/tree/main)
- New testing framework to run KATs from seeds (as in ACVP)
Removal of pq_custom_randombytes.{c/h}

Call-outs:

We can remove the testing mechanism for the KATs pq_custom_randombytes as we now support KATs that use the internal functions that provide randomness via an input seed.

Testing:

The KATs have been migrated to use the internal functions.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and the ISC license.

codecov-commenter · 2024-11-18T20:31:05Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 78.93%. Comparing base (ab8953b) to head (019e09d).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1999      +/-   ##
==========================================
+ Coverage   78.89%   78.93%   +0.04%     
==========================================
  Files         595      594       -1     
  Lines      102451   102432      -19     
  Branches    14527    14524       -3     
==========================================
+ Hits        80827    80853      +26     
+ Misses      20976    20930      -46     
- Partials      648      649       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚨 Try these New Features:

Flaky Tests Detection - Detect and resolve failed and flaky tests

crypto/dilithium/ml_dsa.h

crypto/dilithium/pqcrystals_dilithium_ref_common/README.md

crypto/dilithium/pqcrystals_dilithium_ref_common/sign.c

crypto/dilithium/p_pqdsa_test.cc

crypto/dilithium/pqcrystals_dilithium_ref_common/sign.c

crypto/dilithium/ml_dsa.c

crypto/dilithium/p_pqdsa_test.cc

crypto/dilithium/pqcrystals_dilithium_ref_common/sign.h

crypto/dilithium/kat/mldsa65.txt

crypto/dilithium/p_pqdsa_test.cc

crypto/dilithium/ml_dsa.h

crypto/dilithium/kat/mldsa65.txt

crypto/dilithium/p_pqdsa_test.cc

crypto/dilithium/pqcrystals_dilithium_ref_common/sign.c

dkostic · 2024-11-20T23:13:56Z

crypto/dilithium/pqcrystals_dilithium_ref_common/sign.c

-  /* Compute mu = CRH(tr, 0, ctxlen, ctx, msg) */
-  mu[0] = 0;
-  mu[1] = ctxlen;
+  /* Compute mu = CRH(tr, pre, msg) */


can you add references to lines in FIPS 204, Algorithm 7 please (for verify_internal and keygen_internal as well)? For example, I believe this comment refers to line 6 that says: 𝜇 ← H(BytesToBits(𝑡𝑟)||𝑀 ′, 64) ?

Yes, I'm staging a PR that adds a LOT of documentation and call out to each of the algorithms in FIPS 204 and their implementation here.

However, I want to get the larger code integrations complete before I start touching every single file in the dilithium repo to add comments. This will prevent me from re-base/conflicts pain down the line.

I'll add specific call outs to the lines in sign/verify in that forthcoming PR.

sure, but the point of documentation is to make the review easier, once we merge the code it's not as useful :)

Ahh I understand now, added these callouts to lines in the spec in 8c3d53e. I plan on adding a lot more documentation like this.

crypto/dilithium/pqcrystals_dilithium_ref_common/sign.c

dkostic · 2024-11-21T19:10:29Z

crypto/dilithium/pqcrystals_dilithium_ref_common/sign.c

    goto rej;
+  }



Suggested change

/* FIPS 204: line 26 Compute signer's hint */

crypto/dilithium/pqcrystals_dilithium_ref_common/sign.c

This change adds ML-DSA-44 and ML-DSA-87 to AWS-LC. As we already have support for ML-DSA-65 through the PQDSA signature API (see #1963) and already support internal functions and KATs that use these internal APIs (see #1999), this change consists of: - The addition of the 6 `pqdsa` APIs for ML-DSA-44 and ML-DSA-87: `ml_dsa_{44/87}_keypair`, `ml_dsa_{44/87}_keypair_internal`, `ml_dsa_{44/87}_sign`, `ml_dsa_{44/87}_sign_internal`, `ml_dsa_{44/87}_verify`, and`ml_dsa_{44/87}_verify_internal` - The function methods: `sig_ml_dsa_44_method` and `sig_ml_dsa_87_method` - The algorithm data structs: `sig_ml_dsa_44` and `sig_ml_dsa_87` Integrating ML-DSA-44/87 into X.509 will be in a follow-up commit. Benchmarking results: ``` Did 33000 MLDSA44 keygen operations in 1000554us (32981.7 ops/sec) Did 8541 MLDSA44 signing operations in 1061153us (8048.8 ops/sec) Did 32000 MLDSA44 verify operations in 1016751us (31472.8 ops/sec) Did 17000 MLDSA65 keygen operations in 1021812us (16637.1 ops/sec) Did 6000 MLDSA65 signing operations in 1148331us (5225.0 ops/sec) Did 20000 MLDSA65 verify operations in 1019696us (19613.7 ops/sec) Did 12000 MLDSA87 keygen operations in 1011438us (11864.3 ops/sec) Did 4344 MLDSA87 signing operations in 1043422us (4163.2 ops/sec) Did 12000 MLDSA87 verify operations in 1027267us (11681.5 ops/sec) ``` Testing: `ML-DSA-44/87` are added to the test harness. This requires the inclusion of test harness raw public keys `mldsa{44/87}kPublicKey` and encoded public keys `mldsa{44/87}kPublicKeySPKI`. The lengths of the encodings are well defined by https://datatracker.ietf.org/doc/draft-ietf-lamps-dilithium-certificates/

### Issues: Resolves #CryptoAlg-2889 ### Description of changes: The industry has decided upon a few conventions to assist with the usability of ML-DSA key sizes, one of which is to decrease the amount of bytes transferred when encoding ML-DSA private keys. As ML-DSA keys can be deterministically generated from a provided 32-byte seed, many standards bodies have selected to use private keys in seed format. For example, the IETF draft that standardizes the use of ML-DSA in X.509. https://datatracker.ietf.org/doc/draft-ietf-lamps-dilithium-certificates/ This provides: - Compliance with IETF spec https://datatracker.ietf.org/doc/draft-ietf-lamps-dilithium-certificates/ - Will allow interoperability with BC to allow customers to import ML-DSA keys The functionality that provides key generation from seeds for ML-DSA was added in my previous PR #1999. This PR exposes the internal key generation functions within the `PQDSA` struct, so that a new function named `PQDSA_KEY_set_raw_keypair_from_seed` can directly call the method. The function `PQDSA_KEY_set_raw_keypair_from_seed` simply performs: - a check that the provided `seed` is of the correct length - allocation of public/private key buffers - key generation from seed using `pqdsa_keygen_internal` As such, `pqdsa_priv_decode` has been modified to call: - `PQDSA_KEY_set_raw_private_key` when the provided length of the key is `pqdsa->private_key_len` - `PQDSA_KEY_set_raw_keypair_from_seed` when the provided length of the key is `pqdsa->keygen_seed_len` We implement this at the EVP layer by modifying `EVP_PKEY_pqdsa_new_raw_private_key` to: - call `PQDSA_KEY_set_raw_private_key` when the `len` provided is `private_key_len` - call `PQDSA_KEY_set_raw_keypair_from_seed` when the `len` provided is `keygen_seed_len` ### Call-outs: - The API `EVP_PKEY_pqdsa_new_raw_private_key` will now populate both public and private keys when provided a seed. - The API `EVP_PKEY_pqdsa_new_raw_public_key` only sets the public key - The API `PQDSA_KEY_set_raw_private_key` will only set private key - The API `PQDSA_KEY_set_raw_public_key` will only set public key - The API `EVP_parse_public_key` sets the public key from DER encoded of public key - The API `EVP_parse_private_key` can accept DER encodings of either the full private key representation, or the 32-byte seed. - The API `EVP_marshal_public_key` sets the DER encoded public key from a EVP PKEY - The API `EVP_marshal_private_key` sets the DER encoded full private key representation (it cannot marshal the seed, as this cannot be derived from the full private key representation. ### Testing: To provide KAT for keys from seeds I have included the `Appendix C. Examples` from https://datatracker.ietf.org/doc/draft-ietf-lamps-dilithium-certificates/. For each provided ML-DSA parameter set, we generate the example private key provided from seed, and check that the corresponding public key matches that in the specification. These tests are in `PQDSAParameterTest` namely, `ParsePrivateKey`. These examples show how much shorter ML-DSA-87 private keys are in seed format: ``` -----BEGIN PRIVATE KEY----- MDICAQAwCwYJYIZIAWUDBAMTBCAAAQIDBAUGBwgJCgsMDQ4PEBESExQVFhcYGRob HB0eHw== -----END PRIVATE KEY----- ``` By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and the ISC license.

added internal APIs for ml-dsa

33dee33

jakemas requested a review from a team as a code owner November 18, 2024 20:07

jakemas mentioned this pull request Nov 18, 2024

Adding internal functions for ML-DSA #1977

Closed

andrewhop reviewed Nov 19, 2024

View reviewed changes

jakemas added 2 commits November 19, 2024 10:04

CR fixes

54ca109

documentation update

1bbbb24

jakemas mentioned this pull request Nov 19, 2024

Switch ML-DSA to use AWS-LC SHA3 #2001

Merged

size to uint8

dc471ec

dkostic reviewed Nov 20, 2024

View reviewed changes

crypto/dilithium/ml_dsa.c Show resolved Hide resolved

crypto/dilithium/p_pqdsa_test.cc Outdated Show resolved Hide resolved

crypto/dilithium/pqcrystals_dilithium_ref_common/sign.h Outdated Show resolved Hide resolved

crypto/dilithium/kat/mldsa65.txt Outdated Show resolved Hide resolved

removed deterministic mode, edit pqdsa test

aada875

andrewhop reviewed Nov 20, 2024

View reviewed changes

crypto/dilithium/p_pqdsa_test.cc Show resolved Hide resolved

crypto/dilithium/p_pqdsa_test.cc Show resolved Hide resolved

andrewhop previously approved these changes Nov 20, 2024

View reviewed changes

dkostic reviewed Nov 20, 2024

View reviewed changes

CR fixes

061ebce

jakemas dismissed andrewhop’s stale review via 061ebce November 21, 2024 00:14

jakemas added 2 commits November 21, 2024 08:59

documentation

7e78803

added documentation to keygen sign verify

8c3d53e

dkostic reviewed Nov 21, 2024

View reviewed changes

jakemas added 3 commits November 21, 2024 11:50

documentation additions

dfd1601

more clarity with fips204

9261040

chr to tr

ea29c49

dkostic previously approved these changes Nov 21, 2024

View reviewed changes

removed test name lookup

019e09d

jakemas dismissed dkostic’s stale review via 019e09d November 21, 2024 23:38

dkostic approved these changes Nov 21, 2024

View reviewed changes

dkostic enabled auto-merge (squash) November 21, 2024 23:42

andrewhop approved these changes Nov 22, 2024

View reviewed changes

dkostic merged commit c48572a into aws:main Nov 22, 2024
115 of 117 checks passed

jakemas mentioned this pull request Nov 22, 2024

Add ML-DSA-44 and ML-DSA-87 to PQDSA API #2009

Merged

jakemas mentioned this pull request Feb 3, 2025

ML-DSA private keys from seeds #2157

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add internal APIs for ML-DSA #1999

Add internal APIs for ML-DSA #1999

jakemas commented Nov 18, 2024 •

edited

Loading

codecov-commenter commented Nov 18, 2024 •

edited

Loading

dkostic Nov 20, 2024

jakemas Nov 21, 2024

dkostic Nov 21, 2024

jakemas Nov 21, 2024

dkostic Nov 21, 2024

jakemas Nov 21, 2024

Add internal APIs for ML-DSA #1999

Add internal APIs for ML-DSA #1999

Conversation

jakemas commented Nov 18, 2024 • edited Loading

Issues:

Description of changes:

Call-outs:

Testing:

codecov-commenter commented Nov 18, 2024 • edited Loading

Codecov Report

dkostic Nov 20, 2024

Choose a reason for hiding this comment

jakemas Nov 21, 2024

Choose a reason for hiding this comment

dkostic Nov 21, 2024

Choose a reason for hiding this comment

jakemas Nov 21, 2024

Choose a reason for hiding this comment

dkostic Nov 21, 2024

Choose a reason for hiding this comment

jakemas Nov 21, 2024

Choose a reason for hiding this comment

jakemas commented Nov 18, 2024 •

edited

Loading

codecov-commenter commented Nov 18, 2024 •

edited

Loading