feat: Add side effect counter to logs

[MirandaWood]

Closes #5017

This PR adds side effect counters to logs, tracks them throughout the kernels, sorts them (only in private kernel tail, public kernel tail is todo), and hashes them to one value in the tail.

Private path

Encrypted and unencrypted logs flow via a private call:

• A private fn calling emit_(un)encrypted_log now actually pushes the hash of the log to the PrivateContext as a SideEffect
• These logs hashes are maintained through PrivateCircuitPublicInputs
• They are collected in init and inner kernels
• Like note hashes, we now provide index hints in the tail and sort the logs hashes
• In the final step of tail, we either run to_public, keeping individual logs hashes, or finish_tail (converting .end to CombinedAccumulatedData), where logs are hashed to a single value
• As before, the single value is rolled up and recalculated in L1

Public path

Unencrypted logs flow via a public call:

• A private fn calling emit_unencrypted_log now actually pushes the hash of the log to the PublicContext as a SideEffect
• These logs hashes are maintained through PublicCircuitPublicInputs
• They are collected in init and inner kernels
• Like note hashes, we currently don't sort in the public tail (I've added a ts hack like the note hash one)
• In the final step of tail, we hash logs to a single value
• As before, the single value is rolled up and recalculated in L1

TODOs

• Since individual logs are not hashed by the circuits (this is part of Hash logs inside Noir #1165 which should be done once Remove chopped notes from logs #1641 is finished), instead a hash is passed via the oracles, the context circuits can't calculate the length of each log. So the length is still passed in.
• In the kernels, logs currently sit as SideEffects but should be some special counted logs struct. It makes sense to me to first separate logs which are linked to note preimages (no issue, but part of kernel epic), then assign each type of logs its own struct.
• I added some hacks/qs (see comments below).

[AztecBot]

Benchmark results

Metrics with a significant change:

• batch_insert_into_append_only_tree_32_depth_ms (4096): 1,959 (-30%)
• batch_insert_into_append_only_tree_32_depth_hash_ms (4096): 0.469 (-30%)
• batch_insert_into_indexed_tree_20_depth_ms (4096): 2,925 (-24%)
• l2_block_processing_time_in_ms (64): 10,367 (-16%)
• note_trial_decrypting_time_in_ms (32): 30.9 (+93%)

Detailed results

All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.

This benchmark source data is available in JSON format on S3 here.

L2 block published to L1

Each column represents the number of txs on an L2 block published to L1.

Metric	8 txs	32 txs	64 txs
l1_rollup_calldata_size_in_bytes	772	772	772
l1_rollup_calldata_gas	6,832	6,832	6,844
l1_rollup_execution_gas	587,313	587,313	587,325
l2_block_processing_time_in_ms	1,453 (-1%)	5,330 (+2%)	⚠️ 10,367 (-16%)
note_successful_decrypting_time_in_ms	159 (-4%)	619 (+2%)	1,013
note_trial_decrypting_time_in_ms	61.7 (-7%)	⚠️ 30.9 (+93%)	37.3 (+30%)
l2_block_building_time_in_ms	19,009 (+8%)	73,698 (+3%)	149,274 (+4%)
l2_block_rollup_simulation_time_in_ms	18,773 (+8%)	72,841 (+3%)	147,593 (+5%)
l2_block_public_tx_process_time_in_ms	8,957 (+12%)	34,119 (+6%)	69,601 (+8%)

L2 chain processing

Each column represents the number of blocks on the L2 chain where each block has 16 txs.

Metric	5 blocks	10 blocks
node_history_sync_time_in_ms	16,032 (+2%)	30,091 (+1%)
note_history_successful_decrypting_time_in_ms	1,340 (+14%)	2,352 (+2%)
note_history_trial_decrypting_time_in_ms	73.8 (+64%)	87.0 (-40%)
node_database_size_in_bytes	19,116,112 (+2%)	35,848,272 (+3%)
pxe_database_size_in_bytes	29,859	59,414

Circuits stats

Stats on running time and I/O sizes collected for every circuit run across all benchmarks.

Circuit	circuit_simulation_time_in_ms	circuit_input_size_in_bytes	circuit_output_size_in_bytes
private-kernel-init	153 (-1%)	45,173 (+1%)	27,384 (+4%)
private-kernel-inner	169 (+3%)	73,608 (+2%)	27,384 (+4%)
private-kernel-ordering	125 (+10%)	53,138 (+4%)	41,123 (+5%)
base-parity	47.2 (+12%)	128	311
root-parity	7.29	1,244	311
base-rollup	19,507 (+2%)	110,928	957
root-rollup	47.5	4,551	821
public-kernel-app-logic	101 (+11%)	50,319 (+4%)	42,853 (+5%)
public-kernel-tail	387 (+5%)	159,352 (+1%)	7,493
merge-rollup	8.28 (+7%)	2,760	957
public-kernel-teardown	89.5 (+5%)	50,319 (+4%)	42,853 (+5%)
public-kernel-setup	88.3 (+5%)	50,319 (+4%)	42,853 (+5%)

Tree insertion stats

The duration to insert a fixed batch of leaves into each tree type.

Metric	1 leaves	16 leaves	64 leaves	128 leaves	512 leaves	1024 leaves	2048 leaves	4096 leaves	32 leaves
batch_insert_into_append_only_tree_16_depth_ms	11.1	18.4 (+3%)	N/A	N/A	N/A	N/A	N/A	N/A	N/A
batch_insert_into_append_only_tree_16_depth_hash_count	16.8	31.6	N/A	N/A	N/A	N/A	N/A	N/A	N/A
batch_insert_into_append_only_tree_16_depth_hash_ms	0.644	0.567 (+2%)	N/A	N/A	N/A	N/A	N/A	N/A	N/A
batch_insert_into_append_only_tree_32_depth_ms	N/A	N/A	51.9	81.0 (+1%)	261 (+1%)	506	981	⚠️ 1,959 (-30%)	N/A
batch_insert_into_append_only_tree_32_depth_hash_count	N/A	N/A	96.0	159	543	1,055	2,079	4,127	N/A
batch_insert_into_append_only_tree_32_depth_hash_ms	N/A	N/A	0.532	0.499	0.476 (+1%)	0.472	0.466	⚠️ 0.469 (-30%)	N/A
batch_insert_into_indexed_tree_20_depth_ms	N/A	N/A	61.3 (+1%)	120 (+1%)	379 (+1%)	746 (+1%)	1,467	⚠️ 2,925 (-24%)	N/A
batch_insert_into_indexed_tree_20_depth_hash_count	N/A	N/A	105	207	691	1,363	2,707	5,395	N/A
batch_insert_into_indexed_tree_20_depth_hash_ms	N/A	N/A	0.540	0.539	0.514 (+1%)	0.510	0.509	0.509 (-23%)	N/A
batch_insert_into_indexed_tree_40_depth_ms	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	67.9
batch_insert_into_indexed_tree_40_depth_hash_count	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	109
batch_insert_into_indexed_tree_40_depth_hash_ms	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	0.590

Miscellaneous

Transaction sizes based on how many contract classes are registered in the tx.

Metric	0 registered classes	1 registered classes
tx_size_in_bytes	42,886 (+5%)	535,757 (+1%)

Transaction size based on fee payment method

Metric	native fee payment method	fpc_public fee payment method	fpc_private fee payment method
tx_with_fee_size_in_bytes	897 (-1%)	1,145 (-1%)	1,345 (-2%)

Transaction processing duration by data writes.

Metric	0 new note hashes	1 new note hashes	2 new note hashes
tx_pxe_processing_time_ms	1,290 (+6%)	863 (+6%)	3,509 (+2%)

Metric	1 public data writes	2 public data writes	3 public data writes	4 public data writes	5 public data writes	8 public data writes
tx_sequencer_processing_time_ms	835 (+6%)	682 (+4%)	1,301 (+5%)	886 (+9%)	1,996 (+4%)	1,148 (+9%)

[fcarreiro]

Ok I think I understand. LMK once things are stable/final so that I refactor them :)

Thanks!

[LeilaWang]

Can we check that public_inputs.end.(un)encrypted_logs_hashes contains the hashes. And maybe can add more than one log to the private call data.

[MirandaWood]

Done :)

[LeilaWang]

Should also propagate the logs in end_non_revertible. Oh and I just realized the propagate_new_l2_to_l1_messages in update_public_end_non_revertible_values should be propagate_new_l2_to_l1_messages_non_revertible. We really need to find a way to reduce the duplicated code here...

[MirandaWood]

Would it be best to have a flag for each propagate function to tell it whether to push to end or end_non_revertible? Rather than having 2 of each function!
So in public setup and teardown, when calling update_public_end_non_revertible_values, we should push the logs to .end_non_revertible rather than .end? (Previously, the single hash was just accumulated to end in all cases)

[MirandaWood]

For now I added a new function, mimicing the rest of the code, just to avoid a lot of changes. I'd be happy to take on the task of reducing the code here, just in another PR?

[LeilaWang]

Now the logs have counters we should be able to split them!

[MirandaWood]

True but I'm not sure how to handle the preimages length! Since we don't hash inside the circuit, the circuit doesn't 'know' the length of each log yet, so we can't split the length up. We do always recombine and sum the lengths anyway in CombinedAccumulatedData, so maybe that's fine for now?

[MirandaWood]

For now I split the logs, and added a TODO to do the same for the lengths once 1165 is complete. It doesn't seem to break anything, since lengths are re-summed in combine anyway.

[LeilaWang]

That's a good idea!

[LeilaWang]

Seems to me it's not a hack but a necessary thing to do! Maybe we can provide an instance when running the simulator, and every time a log is emitted it's pushed to an array in this instance. Then at the end of the simulation we can just retrieve the logs from it that are already "sorted".

[LeilaWang]

🚀