feat: wonky rollups

[MirandaWood]

Wonky Rollups

See full write up here.
This PR eliminates the need to pad empty transactions in a rollup, so we can run the fewest rollup circuits required for a block. Main changes are:

• Implementing a greedy-filled wonky tree in sol (computeUnbalancedRoot) and ts (unbalanced_tree.ts, and getTxsEffectHash) - thanks to @alexghr for the idea and help on this!
• Using this tree to calculate txsEffectHash, outHash, and gather membership paths for consuming messages in the outHash tree
• ContentCommitment.txTreeHeight -> .numTxs as we have variable height (numTxs may also not be required, as we gather the number from the txs effect calculation anyway)
• Merge rollups can now take in one base and one merge rollup as input
• orchestrator.ts now forms a wonky tree (proving-state.ts -> findMergeLevel) instead of a balanced tree with padding when constructing the rollup circuits and enqueuing proofs
• We only pad blocks if we have less than 2 transactions, since a root rollup still needs 2 inputs

---

The tree greedy fills txs from L to R in a rollup structure e.g. 5 txs looks like:

  //                  root
  //             /            \
  //          merge           base
  //      /          \
  //   merge        merge
  //  /     \      /     \
  // base  base  base   base

...and 7 txs looks like:

  //                     root
  //             /                  \
  //         merge3                merge5
  //      /          \             /    \
  //   merge1       merge2      merge4  base
  //  /     \      /     \      /    \
  // base  base  base   base  base  base

Eliminates the need to use padding txs and circuits. E.g. previously, 5txs would have:

  //
  //                        root
  //              /                     \
  //          merge                    merge
  //      /          \             /           \
  //   merge        merge        merge        merge
  //  /     \      /     \      /    \       /     \
  // base  base  base   base  base  <pad>  <pad>  <pad>

Processing 3 extra txs and => base circuits, and 3 extra merge circuits.

To recalulate a wonky tree from the number of txs in a block, simply decompose it into powers of 2 to find the subtree sizes. Calculate the roots of these subtrees and hash from right to left. A full example in the write up.

[github-actions]

Changes to circuit sizes

Generated at commit: 5569537965321e3608b4cfc061c042da4ace2392, compared to commit: c07cf2cf2b004dba46a3138a1f64f207b6ee537f

🧾 Summary (100% most significant diffs)

Program	ACIR opcodes (+/-)	%	Circuit size (+/-)	%
rollup_merge	+24 ❌	+1.83%	+405 ❌	+0.07%
rollup_root	+23 ❌	+1.07%	+405 ❌	+0.04%

---

Full diff report 👇

Program	ACIR opcodes (+/-)	%	Circuit size (+/-)	%
rollup_merge	1,339 (+24)	+1.83%	602,067 (+405)	+0.07%
rollup_root	2,168 (+23)	+1.07%	911,186 (+405)	+0.04%

[AztecBot]

Benchmark results

Metrics with a significant change:

• avm_simulation_time_ms (GasToken:_increase_public_balance): 70.9 (-37%)
• avm_simulation_time_ms (Token:assert_minter_and_mint): 60.9 (-35%)
• app_circuit_proving_time_in_ms (AuthRegistry:set_authorized (avm)): 1,634 (+18%)
• app_circuit_proving_time_in_ms (FPC:prepare_fee (avm)): 4,821 (-16%)
• app_circuit_proving_time_in_ms (FPC:pay_refund (avm)): 16,666 (-29%)
• protocol_circuit_proving_time_in_ms (private-kernel-tail-to-public): 84,832 (-17%)
• protocol_circuit_proving_time_in_ms (base-rollup): 72,462 (-18%)
• protocol_circuit_proving_time_in_ms (root-parity): 33,955 (-17%)
• protocol_circuit_proving_time_in_ms (public-kernel-setup): 39,028 (-16%)
• protocol_circuit_proving_time_in_ms (root-rollup): 19,660 (-19%)

Detailed results

All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.

This benchmark source data is available in JSON format on S3 here.

Proof generation

Each column represents the number of threads used in proof generation.

Metric	1 threads	4 threads	16 threads	32 threads	64 threads
proof_construction_time_sha256_30_ms	11,426 (-1%)	3,065 (-1%)	1,448 (-1%)	1,463 (-12%)	1,609 (+5%)
proof_construction_time_sha256_100_ms	43,928 (-1%)	11,760 (-1%)	5,476	5,431	5,385
proof_construction_time_poseidon_hash_ms	78.0 (-1%)	34.0	34.0	57.0 (-2%)	88.0 (+1%)
proof_construction_time_poseidon_hash_30_ms	1,519 (-1%)	416 (-1%)	200	232 (+3%)	265 (+1%)
proof_construction_time_poseidon_hash_100_ms	5,755 (-1%)	1,565 (-1%)	719 (-1%)	770 (-2%)	790 (-1%)

L2 block published to L1

Each column represents the number of txs on an L2 block published to L1.

Metric	4 txs	8 txs	16 txs
l1_rollup_calldata_size_in_bytes	1,412	1,412	1,412
l1_rollup_calldata_gas	9,476	9,460	9,452
l1_rollup_execution_gas	611,215	611,350	611,493
l2_block_processing_time_in_ms	757 (-1%)	1,414 (-1%)	2,712
l2_block_building_time_in_ms	25,907 (+1%)	51,573	101,547 (+1%)
l2_block_rollup_simulation_time_in_ms	25,906 (+1%)	51,572	101,547 (+1%)
l2_block_public_tx_process_time_in_ms	22,278 (+1%)	47,735	97,705 (+1%)

L2 chain processing

Each column represents the number of blocks on the L2 chain where each block has 8 txs.

Metric	3 blocks	5 blocks
node_history_sync_time_in_ms	7,048 (-1%)	9,888 (-1%)
node_database_size_in_bytes	12,197,968	16,076,880
pxe_database_size_in_bytes	16,254	26,813

Circuits stats

Stats on running time and I/O sizes collected for every kernel circuit run across all benchmarks.

Circuit	simulation_time_in_ms	witness_generation_time_in_ms	proving_time_in_ms	input_size_in_bytes	output_size_in_bytes	proof_size_in_bytes	num_public_inputs	size_in_gates
private-kernel-init	122	489 (+5%)	11,983 (-11%)	20,634	67,190	92,352	2,819	524,288
private-kernel-inner	384	1,072 (+11%)	44,477 (-15%)	94,902	67,190	92,352	2,819	2,097,152
private-kernel-tail	312 (-1%)	1,917 (+9%)	46,321 (-14%)	99,121	71,733	14,912	399	2,097,152
base-parity	6.70 (+1%)	1,976 (+2%)	2,554 (-8%)	128	64.0	2,208	2.00	131,072
root-parity	49.7 (-4%)	57.9 (-23%)	⚠️ 33,955 (-17%)	27,100	64.0	2,720	18.0	2,097,152
base-rollup	8,071 (+1%)	5,090 (+3%)	⚠️ 72,462 (-18%)	170,330	728 (-4%)	3,648	47.0	4,194,304
root-rollup	112 (+2%)	73.5 (-18%)	⚠️ 19,660 (-19%)	25,253	620	3,456	41.0	1,048,576
public-kernel-setup	723 (-8%)	3,703 (-2%)	⚠️ 39,028 (-16%)	116,905	93,334	125,344	3,850	2,097,152
public-kernel-app-logic	639 (+1%)	4,824 (+1%)	39,805 (-14%)	116,905	93,334	125,344	3,850	2,097,152
public-kernel-tail	1,420 (+1%)	32,933 (-11%)	165,786 (-14%)	511,910	10,014	14,912	399	8,388,608
private-kernel-reset-small	562	2,085 (-2%)	41,377 (-14%)	123,313	67,190	92,352	2,819	2,097,152
public-kernel-teardown	633 (-1%)	4,869 (+3%)	39,903 (-12%)	116,905	93,334	125,344	3,850	2,097,152
merge-rollup	29.4 (+1%)	N/A	N/A	16,486	728 (-4%)	N/A	N/A	N/A
private-kernel-tail-to-public	N/A	7,320 (+1%)	⚠️ 84,832 (-17%)	N/A	N/A	125,344	3,850	4,194,304

Stats on running time collected for app circuits

Function	input_size_in_bytes	output_size_in_bytes	witness_generation_time_in_ms	proof_size_in_bytes	proving_time_in_ms	size_in_gates	num_public_inputs
ContractClassRegisterer:register	1,344	9,944	426	N/A	N/A	N/A	N/A
ContractInstanceDeployer:deploy	1,408	9,944	40.8 (+1%)	N/A	N/A	N/A	N/A
MultiCallEntrypoint:entrypoint	1,920	9,944	1,817 (+2%)	N/A	N/A	N/A	N/A
GasToken:deploy	1,376	9,944	1,001 (+1%)	N/A	N/A	N/A	N/A
SchnorrAccount:constructor	1,312	9,944	1,457 (+4%)	N/A	N/A	N/A	N/A
SchnorrAccount:entrypoint	2,304	9,944	2,849 (-1%)	16,768	51,112 (-11%)	2,097,152	457
Token:privately_mint_private_note	1,280	9,944	1,705 (-8%)	N/A	N/A	N/A	N/A
FPC:fee_entrypoint_public	1,344	9,944	379 (+4%)	16,768	10,216 (-12%)	524,288	457
Token:transfer	1,312	9,944	4,576 (-7%)	16,768	41,262 (-15%)	2,097,152	457
AuthRegistry:set_authorized (avm)	21,043	N/A	N/A	87,552	⚠️ 1,634 (+18%)	N/A	N/A
FPC:prepare_fee (avm)	28,495	N/A	N/A	88,448	⚠️ 4,821 (-16%)	N/A	N/A
Token:transfer_public (avm)	44,885	N/A	N/A	88,170	3,536 (-13%)	N/A	N/A
AuthRegistry:consume (avm)	34,973	N/A	N/A	87,968	2,789 (-9%)	N/A	N/A
FPC:pay_refund (avm)	41,394	N/A	N/A	89,248	⚠️ 16,666 (-29%)	N/A	N/A
Benchmarking:create_note	1,344	9,944	1,409 (+1%)	N/A	N/A	N/A	N/A
SchnorrAccount:verify_private_authwit	1,280	9,944	74.5 (+1%)	N/A	N/A	N/A	N/A
Token:unshield	1,376	9,944	3,768 (-11%)	N/A	N/A	N/A	N/A
FPC:fee_entrypoint_private	1,376	9,944	4,824 (-11%)	N/A	N/A	N/A	N/A

AVM Simulation

Time to simulate various public functions in the AVM.

Function	time_ms	bytecode_size_in_bytes
GasToken:_increase_public_balance	⚠️ 70.9 (-37%)	13,873
GasToken:set_portal	16.9 (-4%)	3,495
Token:constructor	95.7 (+4%)	24,207
FPC:constructor	64.1 (-1%)	13,893
GasToken:mint_public	53.1	10,241
Token:mint_public	638 (+1%)	19,216
Token:assert_minter_and_mint	⚠️ 60.9 (-35%)	13,034
AuthRegistry:set_authorized	31.7 (-10%)	7,869
FPC:prepare_fee	178 (+9%)	15,129
Token:transfer_public	35.2 (+5%)	31,425
FPC:pay_refund	174 (-21%)	28,061
Benchmarking:increment_balance	2,735	15,407
Token:_increase_public_balance	56.3 (-20%)	15,089
FPC:pay_refund_with_shielded_rebate	200 (+2%)	29,148

Public DB Access

Time to access various public DBs.

Function	time_ms
get-nullifier-index	0.155 (-4%)

Tree insertion stats

The duration to insert a fixed batch of leaves into each tree type.

Metric	1 leaves	16 leaves	64 leaves	128 leaves	256 leaves	512 leaves	1024 leaves
batch_insert_into_append_only_tree_16_depth_ms	10.5 (+1%)	16.8	N/A	N/A	N/A	N/A	N/A
batch_insert_into_append_only_tree_16_depth_hash_count	16.8	31.7	N/A	N/A	N/A	N/A	N/A
batch_insert_into_append_only_tree_16_depth_hash_ms	0.608 (+1%)	0.518	N/A	N/A	N/A	N/A	N/A
batch_insert_into_append_only_tree_32_depth_ms	N/A	N/A	48.9	76.0	132 (+1%)	244 (-1%)	472
batch_insert_into_append_only_tree_32_depth_hash_count	N/A	N/A	95.9	159	287	543	1,055
batch_insert_into_append_only_tree_32_depth_hash_ms	N/A	N/A	0.500	0.466	0.452	0.443 (-1%)	0.441
batch_insert_into_indexed_tree_20_depth_ms	N/A	N/A	60.0	112 (-1%)	183	352 (-1%)	698
batch_insert_into_indexed_tree_20_depth_hash_count	N/A	N/A	109	207	355	691	1,363
batch_insert_into_indexed_tree_20_depth_hash_ms	N/A	N/A	0.506	0.503	0.485	0.477 (-1%)	0.479 (+1%)
batch_insert_into_indexed_tree_40_depth_ms	N/A	N/A	73.2	N/A	N/A	N/A	N/A
batch_insert_into_indexed_tree_40_depth_hash_count	N/A	N/A	133	N/A	N/A	N/A	N/A
batch_insert_into_indexed_tree_40_depth_hash_ms	N/A	N/A	0.521	N/A	N/A	N/A	N/A

Miscellaneous

Transaction sizes based on how many contract classes are registered in the tx.

Metric	0 registered classes	1 registered classes
tx_size_in_bytes	85,672	670,983

Transaction size based on fee payment method

| Metric | |
| - | |

[MirandaWood]

Moved to MerkleLib.sol as it's useful for other contracts as well.

[MirandaWood]

Moved this section to wonky-tree.md as we don't need it for logs any more, but may be useful in future.

[AztecBot]

Docs Preview

Hey there! 👋 You can check your preview at https://667f04a6c64fca3893727aeb--aztec-docs-dev.netlify.app

[PhilWindle]

The non-circuit stuff looks fine to me.

It makes me wonder if we could get rid of the need to tell the orchestrator up front the size of the block. Currently ProvingState holds a heap of MergeRollupInputData objects sized accordingly based on knowing upfront the number of merge levels. But if we changed that structure such that there was no fixed 'end' to the proving tree then we might be able to get it to work. So the orchestrator just receives txs and pairs them up greedily. When it's told the block is completed it knows the final level it needs to build merges for before completing the root.

Anyway, a job for another PR. Thanks for this!

[MirandaWood]

The non-circuit stuff looks fine to me.

It makes me wonder if we could get rid of the need to tell the orchestrator up front the size of the block. Currently ProvingState holds a heap of MergeRollupInputData objects sized accordingly based on knowing upfront the number of merge levels. But if we changed that structure such that there was no fixed 'end' to the proving tree then we might be able to get it to work. So the orchestrator just receives txs and pairs them up greedily. When it's told the block is completed it knows the final level it needs to build merges for before completing the root.

Anyway, a job for another PR. Thanks for this!

Thanks Phil! Yes I agree, I think we could omit starting a block with the number of transactions. It wouldn't be too difficult but would require changing how the proving-state is stored. It currently goes from bottom to top and assigns the bottom level as the highest number required and the root level as 0. We can't calculate the levels required or their width without the number of txs, so I guess we'd have to reverse this and store mergeInputData based on level and width, then recalibrate once we do know the max number.

As you say, based on this recalibration we can work out where the root lives and build merges accordingly, so definitely doable! Happy to take this on in another PR.

[LeilaWang]

Can it be u32?

[LeilaWang]

Looks like the subtreeSizes are simply the non-zero bits of the binary form of numTxs.
Consider doing the following so that we don't have to allocate some memory for the sizes:

uint256 currentSubtreeSize = 1;
uint256 numTxs = _leaves.length;
while (currentSubtreeSize <= numTxs) {
  if (!(currentSubtreeSize & numTxs)) {
    continue;
  }
  // Do the thing.
  currentSubtreeSize <<= 1;
}

(☝️ Might contain bug and incorrect syntax)

[MirandaWood]

Oh of course!! Thanks, great catch. Will also remove the subtree size fn and refactor min/max path length (the latter will disappear once we move to a balanced out_hash tree, so I won't spend too much time on it)

[LeilaWang]

Why is the min not 1?

[MirandaWood]

The RHS subtree is itself an unbalanced tree - not a node of 1, as we greedy fill left to right.
Like for 7, filling left to right means even though we do have a subtree of 1, the min path length is 2. As we connect it to the subtree of 2, which is then connected to the subtree of 4:

  //                     root
  //             /                  \
  //         merge3                merge5
  //      /          \             /    \
  //   merge1       merge2      merge4  base
  //  /     \      /     \      /    \
  // base  base  base   base  base  base

[LeilaWang]

Can we move the unbalanced tree up a level so that it's not in the standard tree?

[LeilaWang]

Can throw an error when a method is not supported or does not make sense.

[LeilaWang]

Don't think it will work when called repeatedly? But we don't really need to call this more than once! So maybe throw an error if this.size is not zero.

[MirandaWood]

Good point, I'd missed that! Added an error

[LeilaWang]

Brilliant work!! 🎉