feat: Capture broadcasted private fns in node #5318

spalladino · 2024-03-19T17:58:35Z

Captures individual private functions broadcasted from the class registerer contract in the aztec node archiver and store them for future use.

AztecBot · 2024-03-20T19:11:24Z

Benchmark results

No metrics with a significant change found.

Detailed results

All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.

This benchmark source data is available in JSON format on S3 here.

Values are compared against data from master at commit 5d669b93 and shown if the difference exceeds 1%.

L2 block published to L1

Each column represents the number of txs on an L2 block published to L1.

Metric	8 txs	32 txs	64 txs
l1_rollup_calldata_size_in_bytes	676	676	676
l1_rollup_calldata_gas	6,400	6,424	6,424
l1_rollup_execution_gas	585,733	585,757	585,757
l2_block_processing_time_in_ms	1,321 (+2%)	4,655 (-2%)	9,116 (+1%)
note_successful_decrypting_time_in_ms	189 (+2%)	507 (-3%)	965 (+1%)
note_trial_decrypting_time_in_ms	94.2 (+2%)	25.8 (-30%)	95.7 (+15%)
l2_block_building_time_in_ms	17,271	65,301	129,780 (-1%)
l2_block_rollup_simulation_time_in_ms	7,928 (-1%)	28,199	55,519 (-3%)
l2_block_public_tx_process_time_in_ms	9,324 (+1%)	37,047 (+1%)	74,167 (+1%)

L2 chain processing

Each column represents the number of blocks on the L2 chain where each block has 16 txs.

Metric	5 blocks	10 blocks
node_history_sync_time_in_ms	14,538 (+3%)	26,726 (+1%)
note_history_successful_decrypting_time_in_ms	1,325 (+6%)	2,471 (+2%)
note_history_trial_decrypting_time_in_ms	101 (+7%)	194 (+58%)
node_database_size_in_bytes	18,673,744	35,065,936
pxe_database_size_in_bytes	29,859	59,414

Circuits stats

Stats on running time and I/O sizes collected for every circuit run across all benchmarks.

Circuit	circuit_simulation_time_in_ms	circuit_input_size_in_bytes	circuit_output_size_in_bytes
private-kernel-init	247 (+3%)	44,344	28,181
private-kernel-ordering	214	52,847	14,263
base-parity	4,467 (+1%)	128	311
root-parity	1,624 (-6%)	1,244	311
base-rollup	18,025 (-3%)	165,727	861
root-rollup	50.1	4,359	725
private-kernel-inner	598 (+1%)	73,728	28,181
public-kernel-app-logic	402 (+1%)	35,185	28,151
public-kernel-tail	173 (+1%)	40,862	28,151
merge-rollup	8.18 (-1%)	2,568	861

Tree insertion stats

The duration to insert a fixed batch of leaves into each tree type.

Metric	1 leaves	16 leaves	64 leaves	128 leaves	512 leaves	1024 leaves	2048 leaves	4096 leaves	32 leaves
batch_insert_into_append_only_tree_16_depth_ms	9.99	16.0	N/A	N/A	N/A	N/A	N/A	N/A	N/A
batch_insert_into_append_only_tree_16_depth_hash_count	16.8	31.6	N/A	N/A	N/A	N/A	N/A	N/A	N/A
batch_insert_into_append_only_tree_16_depth_hash_ms	0.583	0.496 (+1%)	N/A	N/A	N/A	N/A	N/A	N/A	N/A
batch_insert_into_append_only_tree_32_depth_ms	N/A	N/A	45.8	72.4	231 (-2%)	451 (+1%)	866	1,712	N/A
batch_insert_into_append_only_tree_32_depth_hash_count	N/A	N/A	96.0	159	543	1,055	2,079	4,127	N/A
batch_insert_into_append_only_tree_32_depth_hash_ms	N/A	N/A	0.470	0.447	0.422 (-2%)	0.420 (+1%)	0.413	0.411	N/A
batch_insert_into_indexed_tree_20_depth_ms	N/A	N/A	53.6	107	341 (+2%)	661	1,304	2,607 (-1%)	N/A
batch_insert_into_indexed_tree_20_depth_hash_count	N/A	N/A	104	207	691	1,363	2,707	5,395	N/A
batch_insert_into_indexed_tree_20_depth_hash_ms	N/A	N/A	0.478	0.482	0.462 (+1%)	0.454	0.453	0.453 (-1%)	N/A
batch_insert_into_indexed_tree_40_depth_ms	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	61.5
batch_insert_into_indexed_tree_40_depth_hash_count	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	109
batch_insert_into_indexed_tree_40_depth_hash_ms	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	0.538

Miscellaneous

Transaction sizes based on how many contract classes are registered in the tx.

Metric	0 registered classes
tx_size_in_bytes	21,991

Transaction processing duration by data writes.

Metric	0 new note hashes	1 new note hashes
tx_pxe_processing_time_ms	3,162 (+3%)	1,650

Metric	0 public data writes	1 public data writes
tx_sequencer_processing_time_ms	11.1 (+3%)	1,152 (+1%)

spalladino · 2024-03-21T19:34:24Z

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @spalladino and the rest of your teammates on Graphite

just-mitch · 2024-03-21T23:12:05Z

Looks really good. Awaiting the changes that come from making the tests pass.

spalladino · 2024-03-22T01:27:13Z

Tests were passing until I rebased. Oh well...

PhilWindle

Just some nits

PhilWindle · 2024-03-22T11:33:58Z

yarn-project/archiver/src/archiver/kv_archiver_store/contract_class_store.ts

    this.#contractClasses = db.openMap('archiver_contract_classes');
  }

-  addContractClass(contractClass: ContractClassPublic): Promise<boolean> {
-    return this.#contractClasses.set(contractClass.id.toString(), serializeContractClassPublic(contractClass));
+  async addContractClass(contractClass: ContractClassPublic): Promise<void> {


Is it not still worth returning the result?

Just changed it to return the result, even though it gets discarded by the callers. Still, I dug a bit deeper to figure out where that boolean is coming from: it's from lmdb's put method that may return false if the insertion doesn't happen due to a versioning setting, which we're not using. So I think we should just update all those booleans to just voids in the Map interface. @alexghr thoughts, given you had worked on it extensively?

Sounds good to me!

PhilWindle · 2024-03-22T11:37:55Z

yarn-project/archiver/src/archiver/kv_archiver_store/kv_archiver_store.ts

+    privateFunctions: ExecutablePrivateFunctionWithMembershipProof[],
+  ): Promise<boolean> {
+    await this.#contractClassStore.addPrivateFunctions(contractClassId, privateFunctions);
+    return Promise.resolve(true);


Should this return value mean something? Maybe the function we forward to should return something. Otherwise we can presumably just return void.

PhilWindle · 2024-03-22T11:38:21Z

yarn-project/archiver/src/archiver/memory_archiver_store/memory_archiver_store.ts

+      ...newPrivateFunctions.filter(newFn => !privateFunctions.find(f => f.selector.equals(newFn.selector))),
+    ];
+    this.privateFunctions.set(contractClassId.toString(), updatedPrivateFunctions);
+    return Promise.resolve(true);


As before, should this mean something?

PhilWindle · 2024-03-22T11:47:28Z

Looks good!

socket-security · 2024-03-22T13:02:33Z

No dependency changes detected. Learn more about Socket for GitHub ↗︎

👍 No dependency changes detected in pull request

Captures individual private and unconstrained functions broadcasted from the class registerer contract in the aztec node archiver and store them for future use.

spalladino · 2024-03-22T17:06:57Z

Closing in favor of merging everything in one go in #5353

Includes: - #5351 - #5350 - #5349 - #5318 - And deletes isInternal flag from functions in contract classes since it's no longer required by kernel.

LMDB operations return a false boolean if an insert or delete operation fails due to a failed version check, which we are not using. So we just turn every return value to void. Context [here](#5318 (comment)).

spalladino force-pushed the palla/capture-broadcasted-fns branch 6 times, most recently from c5edfc8 to 9cf17ca Compare March 20, 2024 18:58

spalladino force-pushed the palla/capture-broadcasted-fns branch from 9cf17ca to 98bbd5e Compare March 20, 2024 19:32

spalladino marked this pull request as ready for review March 20, 2024 19:36

spalladino changed the title ~~feat: Capture broadcasted fns in node~~ feat: Capture broadcasted private fns in node Mar 20, 2024

spalladino force-pushed the palla/capture-broadcasted-fns branch from 98bbd5e to 516576f Compare March 20, 2024 19:56

spalladino mentioned this pull request Mar 20, 2024

feat: Capture broadcasted unconstrained functions in node #5349

Closed

spalladino enabled auto-merge (squash) March 20, 2024 21:39

spalladino disabled auto-merge March 20, 2024 21:39

This was referenced Mar 20, 2024

docs(yellow-paper): Update function broadcasting functions #5350

Closed

chore: Bytecode is buffer not string #5351

Closed

feat: Capture broadcasted functions in node #5353

Merged

spalladino force-pushed the palla/capture-broadcasted-fns branch 2 times, most recently from f6e6772 to 6717d5a Compare March 21, 2024 19:34

PhilWindle approved these changes Mar 22, 2024

View reviewed changes

spalladino force-pushed the palla/capture-broadcasted-fns branch 2 times, most recently from c130827 to c24f9f2 Compare March 22, 2024 12:59

spalladino force-pushed the palla/capture-broadcasted-fns branch 3 times, most recently from 34eb446 to 445abea Compare March 22, 2024 15:39

feat: Capture broadcasted fns in node

5e1c241

Captures individual private and unconstrained functions broadcasted from the class registerer contract in the aztec node archiver and store them for future use.

spalladino force-pushed the palla/capture-broadcasted-fns branch from 445abea to 5e1c241 Compare March 22, 2024 16:58

spalladino closed this Mar 22, 2024

spalladino added a commit that referenced this pull request Mar 22, 2024

feat: Capture broadcasted functions in node (#5353)

bc05db2

Includes: - #5351 - #5350 - #5349 - #5318 - And deletes isInternal flag from functions in contract classes since it's no longer required by kernel.

spalladino mentioned this pull request Mar 26, 2024

chore: Remove unused boolean return values from stores #5470

Merged

ludamad deleted the palla/capture-broadcasted-fns branch August 22, 2024 15:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Capture broadcasted private fns in node #5318

feat: Capture broadcasted private fns in node #5318

spalladino commented Mar 19, 2024 •

edited

Loading

AztecBot commented Mar 20, 2024 •

edited

Loading

L2 block published to L1

L2 chain processing

Circuits stats

Tree insertion stats

Miscellaneous

spalladino commented Mar 21, 2024

just-mitch commented Mar 21, 2024

spalladino commented Mar 22, 2024

PhilWindle left a comment

PhilWindle Mar 22, 2024

spalladino Mar 22, 2024

alexghr Mar 22, 2024

PhilWindle Mar 22, 2024

PhilWindle Mar 22, 2024

PhilWindle commented Mar 22, 2024 •

edited

Loading

socket-security bot commented Mar 22, 2024 •

edited

Loading

spalladino commented Mar 22, 2024

feat: Capture broadcasted private fns in node #5318

feat: Capture broadcasted private fns in node #5318

Conversation

spalladino commented Mar 19, 2024 • edited Loading

AztecBot commented Mar 20, 2024 • edited Loading

Benchmark results

L2 block published to L1

L2 chain processing

Circuits stats

Tree insertion stats

Miscellaneous

spalladino commented Mar 21, 2024

just-mitch commented Mar 21, 2024

spalladino commented Mar 22, 2024

PhilWindle left a comment

Choose a reason for hiding this comment

PhilWindle Mar 22, 2024

Choose a reason for hiding this comment

spalladino Mar 22, 2024

Choose a reason for hiding this comment

alexghr Mar 22, 2024

Choose a reason for hiding this comment

PhilWindle Mar 22, 2024

Choose a reason for hiding this comment

PhilWindle Mar 22, 2024

Choose a reason for hiding this comment

PhilWindle commented Mar 22, 2024 • edited Loading

socket-security bot commented Mar 22, 2024 • edited Loading

spalladino commented Mar 22, 2024

spalladino commented Mar 19, 2024 •

edited

Loading

AztecBot commented Mar 20, 2024 •

edited

Loading

PhilWindle commented Mar 22, 2024 •

edited

Loading

socket-security bot commented Mar 22, 2024 •

edited

Loading