SHA3 and SHAKE - New API Design (#2098)

### Issues:
Resolves #CryptoAlg-2810

### Description of changes: 
AWS-LC supports SHA3 and SHAKE algorithms though low level SHA3_Init,
SHA3_Update, SHA3_Final and SHAKE_init, SHAKE_Final APIs. Currently,
there are two issues with the implementation and usage of SHA3 and
SHAKE:
- There is no support for SHAKE_Update function. SHAKE is implemented by
calling SHAKE_Init, SHA3_Update and SHAKE_Final.
- SHAKE_Final allows multiple consecutive calls to enable incremental
XOF output generation.

This PR addresses both of them as follows:
- Introduce new API layers - FIPS202, SHA3 and SHAKE. 
- _Keccak1600_ layer (#2097)
implements KeccakF1600 Absorb and Squeeze functions; Keccak1600 layer
does _not_ manage internal input/output buffers.
- _FIPS202_ layer implements Reset, Init, Update, and Finalize
functionalities; FIPS202 layer manages the internal input/output
buffers, allowing incremental requests (not necessarily multiple of
block size) to Update (Absorb) and Squeeze for input/output processing.
(Other functionalities, such as zero-ing of bitstate, block size checks,
etc. are also handled by FIPS202 API layer).
- _FIPS202_ layer implements all common behavior between SHA3 and SHAKE
algorithms.
- _FIPS202_ layer checks/updates the |ctx->state| flag when handling a
common behavior between SHA3 and SHAKE algorithms. |ctx->state| is
updated in the higher level SHA3_ SHAKE_ API layer when the behavior of
both algorithms diverges (SHAKE _can_ allow incremental squeezes).
- _SHA3_ layer implements Init, Update, and Final functionalities; SHA3
layer only implements SHA3 algorithm, thus, offers a single-call
SHA3_Final function. SHA3_Final will update internal |ctx->state| flag
to prevent any sequential calls.
- _SHAKE_ layer implements XOF SHAKE algorithm, therefore, offers Init,
Absorb, Squeeze, and Final functionalities;
- _SHAKE_ layer implements Init, and Absorb, Squeeze with incremental
call support for absorb (byte-wise) and squeeze (block-wise).
- _SHAKE_ layer implements a single-call SHAKE_Final function that
generates an arbitrary length output and finalizes SHAKE. Incremental
XOF output generation is handled by |SHAKE_Squeeze|. |SHAKE_Squeeze| can
be called multiple times. SHAKE_Final should be called only once.

- KECCAK600_CTX struct updates:
   - Remove |padded| field
   - Introduce |state| field
- |state| can be |KECCAK1600_STATE_ABSORB|, |KECCAK1600_STATE_SQUEEZE|,
|KECCAK1600_STATE_FINAL|
- |KECCAK1600_STATE_ABSORB| - allows incremental absorbs until the state
is changed
- |KECCAK1600_STATE_SQUEEZE| - allows incremental squeezes for
|SHAKE_Squeeze|
- |KECCAK1600_STATE_Final| - prevents from incremental squeezes via
|SHAKE_Final| and prevents from consecutive calls to |SHA3_Final| (Final
functions are single-shot functions).

SHA3 vs SHAKE algorithms (APIs usage):
>- SHA3 digest generation: SHA3_Init; SHA3_Update; SHA3_Final;
>- SHAKE (single-shot-output) output generation: SHAKE_Init;
SHAKE_Absorb; SHAKE_Final;
>- SHAKE (incremental) output generation: SHAKE_Init; SHAKE_Absorb;
SHAKE_Squeeze<sup>+</sup>;

### Call-outs:
Service indicator is updated: 
- Inside SHA3 and SHAKE single shot APIs (as previously in AWS-LC);
- Inside SHA3_Final (as previously in AWS-LC);
- Inside SHAKE_Final (Single-Shot XOF Final output generation as
previously in AWS-LC);
- Inside SHAKE_Squeeze (Streaming XOF Squeezes output generation updates
the service indicator after each extendable output update);

All other algorithms that use SHA3/SHAKE APIs are updated:
- ML-KEM (SHA3/SHAKE calls will be inlined later)
- ML-DSA (SHAKE_Squeeze (incremental XOF output functionality) inside
ML-DSA is never invoked with the KAT test vectors and gtests)
### Testing:
_./crypto/crypto_test --gtest_filter="KeccakInternalTest.*"_
_./crypto/crypto_test --gtest_filter="SHA3Test.*"_
_./crypto/crypto_test --gtest_filter="SHAKETest.*"_

_./crypto/crypto_test --gtest_filter="All/PerKEMTest.*"_
_./crypto/crypto_test --gtest_filter="All/PQDSAParameterTest.*"_

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license and the ISC license.

---------

Co-authored-by: Jake Massimo <[email protected]>
Co-authored-by: Will Childs-Klein <[email protected]>
Co-authored-by: Justin W Smith <[email protected]>
Co-authored-by: Shubham Mittal <[email protected]>
Co-authored-by: Samuel Chiang <[email protected]>
Co-authored-by: David Benjamin <[email protected]>
Co-authored-by: Theo Buehler <[email protected]>
Co-authored-by: Adam Langley <[email protected]>
Co-authored-by: Brian Ledger <[email protected]>
Co-authored-by: Nick Harper <[email protected]>
Co-authored-by: Andrew Hopkins <[email protected]>
Co-authored-by: torben-hansen <[email protected]>
Co-authored-by: Sean McGrail <[email protected]>
Co-authored-by: olivergillespie <[email protected]>

crypto/fipsmodule/digest/digests.c

@@ -430,7 +430,7 @@ static void shake128_init(EVP_MD_CTX *ctx) {
 }
 
 static void shake128_update(EVP_MD_CTX *ctx, const void *data, size_t count) {
-  CHECK(SHA3_Update(ctx->md_data, data, count));
+  CHECK(SHAKE_Absorb(ctx->md_data, data, count));
 }
 
 static void shake128_final(EVP_MD_CTX *ctx, uint8_t *md, size_t len) {
@@ -455,7 +455,7 @@ static void shake256_init(EVP_MD_CTX *ctx) {
 }
 
 static void shake256_update(EVP_MD_CTX *ctx, const void *data, size_t count) {
-  CHECK(SHA3_Update(ctx->md_data, data, count));
+  CHECK(SHAKE_Absorb(ctx->md_data, data, count));
 }
 
 static void shake256_finalXOF(EVP_MD_CTX *ctx, uint8_t *md, size_t len) {

crypto/fipsmodule/ml_kem/ml_kem_ref/symmetric-shake.c

@@ -29,8 +29,8 @@ void kyber_shake128_absorb(KECCAK1600_CTX *ctx,
   // SHAKE_Init always returns 1 when called with correct block size value
   SHAKE_Init(ctx, SHAKE128_BLOCKSIZE);
 
-  // SHA3_Update always returns 1 on first call of sizeof(extseed) (34 bytes)
-  SHA3_Update(ctx, extseed, sizeof(extseed));
+  // SHAKE_Absorb always returns 1 on first call of sizeof(extseed) (34 bytes)
+  SHAKE_Absorb(ctx, extseed, sizeof(extseed));
 }
 
 /*************************************************
@@ -48,8 +48,9 @@ void kyber_shake128_absorb(KECCAK1600_CTX *ctx,
 void kyber_shake128_squeeze(KECCAK1600_CTX *ctx, uint8_t *out, int nblocks)
 {
   // Return code checks can be omitted
-  // SHAKE_Final always returns 1
-  SHAKE_Final(out, ctx, nblocks * SHAKE128_BLOCKSIZE);
+  // SHAKE_Squeeze always returns 1 when |ctx->state| flag is different 
+  // from |KECCAK1600_STATE_FINAL|
+  SHAKE_Squeeze(out, ctx, nblocks * SHAKE128_BLOCKSIZE);
 }
 
 /*************************************************
@@ -94,12 +95,13 @@ void kyber_shake256_rkprf(ml_kem_params *params, uint8_t out[KYBER_SSBYTES], con
   // SHAKE_Init always returns 1 when called with correct block size value
   SHAKE_Init(&ctx, SHAKE256_BLOCKSIZE);
 
-  // SHA3_Update always returns 1 on first call of KYBER_SYMBYTES (32 bytes)
-  SHA3_Update(&ctx, key, KYBER_SYMBYTES);
+  // SHAKE_Absorb always returns 1 on first call of KYBER_SYMBYTES (32 bytes)
+  SHAKE_Absorb(&ctx, key, KYBER_SYMBYTES);
 
-  // SHA3_Update always returns 1 processing all data blocks that don't need pad
-  SHA3_Update(&ctx, input, params->ciphertext_bytes);
+  // SHAKE_Absorb always returns 1 processing all data blocks that don't need pad
+  SHAKE_Absorb(&ctx, input, params->ciphertext_bytes);
 
-  // SHAKE_Final always returns 1
+  // SHAKE_Final always returns 1 when |ctx->state| flag is set to  
+  // |KECCAK1600_STATE_ABSORB| (no previous calls to SHAKE_Final)
   SHAKE_Final(out, &ctx, KYBER_SSBYTES);
 }

crypto/fipsmodule/sha/internal.h

@@ -71,6 +71,16 @@ extern "C" {
 // SHAKE128 has the maximum block size among the SHA3/SHAKE algorithms.
 #define SHA3_MAX_BLOCKSIZE SHAKE128_BLOCKSIZE
 
+// Define state flag values for Keccak-based functions
+#define KECCAK1600_STATE_ABSORB     0 
+// KECCAK1600_STATE_SQUEEZE is set when |SHAKE_Squeeze| is called.
+// It remains set while |SHAKE_Squeeze| is called repeatedly to output 
+// chunks of the XOF output.
+#define KECCAK1600_STATE_SQUEEZE    1  
+// KECCAK1600_STATE_FINAL is set once |SHAKE_Final| is called 
+// so that |SHAKE_Squeeze| cannot be called anymore.
+#define KECCAK1600_STATE_FINAL      2 
+
 typedef struct keccak_st KECCAK1600_CTX;
 
 // The data buffer should have at least the maximum number of
@@ -82,7 +92,7 @@ struct keccak_st {
   size_t buf_load;                                 // used bytes in below buffer
   uint8_t buf[SHA3_MAX_BLOCKSIZE];                 // should have at least the max data block size bytes
   uint8_t pad;                                     // padding character
-  uint8_t padded;                                  // denotes if padding has been performed
+  uint8_t state;                                  // denotes the keccak phase (absorb, squeeze, final)
 };
 
 // Define SHA{n}[_{variant}]_ASM if sha{n}_block_data_order[_{variant}] is
@@ -396,32 +406,43 @@ OPENSSL_EXPORT uint8_t *SHAKE128(const uint8_t *data, const size_t in_len,
 OPENSSL_EXPORT uint8_t *SHAKE256(const uint8_t *data, const size_t in_len,
                                  uint8_t *out, size_t out_len);
 
-// SHAKE_Init initializes |ctx| with specified |block_size|, returns 1 on
-// success and 0 on failure. Calls SHA3_Init under the hood.
-int SHAKE_Init(KECCAK1600_CTX *ctx, size_t block_size);
-
-// SHAKE_Final writes |len| bytes of finalized digest to |md|, returns 1 on
-// success and 0 on failure. Calls SHA3_Final under the hood.
-int SHAKE_Final(uint8_t *md, KECCAK1600_CTX *ctx, size_t len);
-
-// SHA3_Reset zeros the bitstate and the amount of processed input.
-void SHA3_Reset(KECCAK1600_CTX *ctx);
-
-// SHA3_Init initialises |ctx| fields and returns 1 on success and 0 on failure.
+// SHA3_Init initialises |ctx| fields through |FIPS202_Init| and 
+// returns 1 on success and 0 on failure.
 OPENSSL_EXPORT int SHA3_Init(KECCAK1600_CTX *ctx, size_t bitlen);
 
-// SHA3_Update processes all data blocks that don't need pad through
-// |Keccak1600_Absorb| and returns 1 and 0 on failure.
+ // SHA3_Update check |ctx| pointer and |len| value, calls |FIPS202_Update| 
+ // and returns 1 on success and 0 on failure.
 int SHA3_Update(KECCAK1600_CTX *ctx, const void *data, size_t len);
 
-// SHA3_Final pads the last data block and processes it through |Keccak1600_Absorb|.
-// It processes the data through |Keccak1600_Squeeze| and returns 1 and 0 on failure.
+// SHA3_Final pads the last data block and absorbs it through |FIPS202_Finalize|.
+// It then calls |Keccak1600_Squeeze| and returns 1 on success 
+// and 0 on failure.
 int SHA3_Final(uint8_t *md, KECCAK1600_CTX *ctx);
 
-// Keccak1600_Absorb processes the largest multiple of |r| out of |len| bytes and
-// returns the remaining number of bytes.
+// SHAKE_Init initialises |ctx| fields through |FIPS202_Init| and 
+// returns 1 on success and 0 on failure.
+int SHAKE_Init(KECCAK1600_CTX *ctx, size_t block_size);
+
+// SHAKE_Absorb checks |ctx| pointer and |len| values. It updates and absorbs 
+// input blocks via |FIPS202_Update|.
+int SHAKE_Absorb(KECCAK1600_CTX *ctx, const void *data,
+                               size_t len);
+
+// SHAKE_Squeeze pads the last data block and absorbs it through 
+// |FIPS202_Finalize| on first call. It writes |len| bytes of incremental 
+// XOF output to |md| and returns 1 on success and 0 on failure. It can be 
+// called multiple times.
+int SHAKE_Squeeze(uint8_t *md, KECCAK1600_CTX *ctx, size_t len);
+
+// SHAKE_Final writes |len| bytes of finalized extendible output to |md|, returns 1 on
+// success and 0 on failure. It should be called once to finalize absorb and
+// squeeze phases. Incremental XOF output should be generated via |SHAKE_Squeeze|.
+int SHAKE_Final(uint8_t *md, KECCAK1600_CTX *ctx, size_t len);
+
+// Keccak1600_Absorb processes the largest multiple of |r| (block size) out of
+// |len| bytes and returns the remaining number of bytes.
 size_t Keccak1600_Absorb(uint64_t A[KECCAK1600_ROWS][KECCAK1600_ROWS],
-                         const uint8_t *data, size_t len, size_t r);
+                                  const uint8_t *data, size_t len, size_t r);
 
 // Keccak1600_Squeeze generates |out| value of |len| bytes (per call). It can be called
 // multiple times when used as eXtendable Output Function. |padded| indicates