Add BlockEpilogue (replaces StateCheckpoint), with BlockEndInfo #11298

igor-aptos · 2023-12-12T01:58:46Z

We need to add additional block-end-info onchain, so things like gas estimation can correctly know if blocks are full

Introducing new transaction type - BlockEpilogue, which will be added at the end of the block, instead of StateCheckpoint. BlockEpilogue has additional information - namely BlockEndInfo

Update gas estimator to use new methods

Description

Test Plan

trunk-io · 2023-12-12T01:58:49Z

⏱️ 42h 20m total CI duration on this PR

Job	Cumulative Duration	Recent Runs
windows-build	6h 57m	🟩 🟩 🟩 🟩 🟩 (+16 more)
rust-unit-tests	5h 39m	🟥 🟥 🟥 🟥 🟥 (+7 more)
test-replay / replay-verify (8)	2h	🟥
test-replay / replay-verify (9)	2h	🟥
test-replay / replay-verify (16)	2h	🟥
test-replay / replay-verify (10)	2h	🟥
test-replay / replay-verify (15)	1h 41m	🟩
test-replay / replay-verify (11)	1h 33m	🟩
test-replay / replay-verify (12)	1h 21m	🟩
test-replay / replay-verify (6)	1h 20m	🟩
test-replay / replay-verify (0)	1h 18m	🟩
test-replay / replay-verify (2)	1h 14m	🟩
test-replay / replay-verify (13)	1h 9m	🟩
test-replay / replay-verify (1)	1h 9m	🟩
rust-lints	1h 6m	🟥 🟥 🟥 🟥 🟩 (+7 more)
test-replay / replay-verify (4)	1h 1m	🟩
rust-images / rust-all	58m	🟩 🟩 🟥 🟩 🟩 (+1 more)
test-replay / replay-verify (14)	56m	🟩
run-tests-main-branch	55m	🟩 🟩 🟩 🟩 🟩 (+7 more)
test-replay / replay-verify (7)	52m	🟩
test-replay / replay-verify (5)	50m	🟩
check	50m	🟩 ⬜ 🟩 ⬜ 🟩 (+12 more)
test-replay / replay-verify (3)	50m	🟩
check-dynamic-deps	45m	🟩 🟩 🟩 🟩 🟩 (+17 more)
rust-move-unit-coverage	42m	🟩 🟥
general-lints	31m	🟩 🟩 🟩 🟩 🟩 (+7 more)
execution-performance / single-node-performance	24m	🟩
semgrep/ci	7m	🟩 🟩 🟩 🟩 🟩 (+16 more)
file_change_determinator	2m	🟩 ⬜ 🟩 ⬜ 🟩 (+12 more)
file_change_determinator	2m	🟩 🟩 🟩 🟩 🟩 (+6 more)
permission-check	2m	🟩 ⬜ 🟩 ⬜ 🟩 (+14 more)
file_change_determinator	1m	🟩 🟩 🟩 🟩 🟩 (+1 more)
permission-check	1m	🟩 🟩 🟩 🟩 🟩 (+16 more)
permission-check	49s	🟩 🟩 🟩 🟩 🟩 (+14 more)
docs-lint	41s	🟩
permission-check	26s	🟩 ⬜ 🟩 ⬜ 🟩 (+12 more)
permission-check	22s	🟩 🟩 ⬜ 🟩 🟩 (+2 more)
determine-docker-build-metadata	11s	🟩 🟩 🟩 🟩 🟩 (+1 more)
determine-test-metadata	6s	🟩

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

igor-aptos · 2023-12-12T01:59:03Z

Current dependencies on/for this PR:

main
- PR Refactor VMChangeSet to abstract size for gas charging #10879
  - PR Move AggV1 materialize from transaction execution to block executor materialization #11110
    - PR Improved block gas limit - output size limit, conflict aware, io/compute multipliers #10943

This stack of pull requests is managed by Graphite.

codecov · 2023-12-12T19:55:33Z

Codecov Report

Attention: Patch coverage is 34.84848% with 43 lines in your changes missing coverage. Please review.

Project coverage is 68.7%. Comparing base (a23e74a) to head (385a510).
Report is 2 commits behind head on main.

❗ Current head 385a510 differs from pull request most recent head ecfcf88

Please upload reports for the commit ecfcf88 to get more accurate results.

Files	Patch %	Lines
aptos-move/block-executor/src/limit_processor.rs	0.0%	19 Missing ⚠️
types/src/transaction/block_epilogue.rs	0.0%	10 Missing ⚠️
types/src/transaction/mod.rs	0.0%	7 Missing ⚠️
aptos-move/aptos-vm/src/aptos_vm.rs	0.0%	3 Missing ⚠️
aptos-move/block-executor/src/executor.rs	83.3%	3 Missing ⚠️
types/src/transaction/block_output.rs	83.3%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main   #11298       +/-   ##
===========================================
- Coverage    71.4%    68.7%     -2.7%     
===========================================
  Files        2319      782     -1537     
  Lines      456030   179189   -276841     
===========================================
- Hits       325836   123270   -202566     
+ Misses     130194    55919    -74275

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

making sure empty blocks respect new flag to create BlockEpilogue

github-actions · 2024-06-11T04:54:37Z

✅ Forge suite `compat` success on `a68e71c05caebf01504d4499110f3fba213fb53d` ==> `ecfcf887a0c0a079f2210dcb89ec5b63c6e433fa`

Compatibility test results for a68e71c05caebf01504d4499110f3fba213fb53d ==> ecfcf887a0c0a079f2210dcb89ec5b63c6e433fa (PR)
1. Check liveness of validators at old version: a68e71c05caebf01504d4499110f3fba213fb53d
compatibility::simple-validator-upgrade::liveness-check : committed: 10970.6306318661 txn/s, latency: 3089.5005766864874 ms, (p50: 2700 ms, p90: 6300 ms, p99: 8400 ms), latency samples: 390160
2. Upgrading first Validator to new version: ecfcf887a0c0a079f2210dcb89ec5b63c6e433fa
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 6933.88290152758 txn/s, latency: 4423.249379213945 ms, (p50: 4400 ms, p90: 5600 ms, p99: 6300 ms), latency samples: 243240
3. Upgrading rest of first batch to new version: ecfcf887a0c0a079f2210dcb89ec5b63c6e433fa
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 7042.510580149963 txn/s, latency: 4250.518437322054 ms, (p50: 4100 ms, p90: 6200 ms, p99: 6400 ms), latency samples: 245860
4. upgrading second batch to new version: ecfcf887a0c0a079f2210dcb89ec5b63c6e433fa
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 11760.865480047803 txn/s, latency: 2607.3250050849183 ms, (p50: 2400 ms, p90: 4300 ms, p99: 6000 ms), latency samples: 393320
5. check swarm health
Compatibility test for a68e71c05caebf01504d4499110f3fba213fb53d ==> ecfcf887a0c0a079f2210dcb89ec5b63c6e433fa passed
Test Ok

github-actions · 2024-06-11T04:56:03Z

✅ Forge suite `realistic_env_max_load` success on `ecfcf887a0c0a079f2210dcb89ec5b63c6e433fa`

two traffics test: inner traffic : committed: 9409.03548896873 txn/s, latency: 4180.667754166256 ms, (p50: 4200 ms, p90: 4500 ms, p99: 5700 ms), latency samples: 4056400
two traffics test : committed: 99.95572326103617 txn/s, latency: 1985.1450549450549 ms, (p50: 2000 ms, p90: 2100 ms, p99: 6000 ms), latency samples: 1820
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.234, avg: 0.216", "QsPosToProposal: max: 1.950, avg: 1.881", "ConsensusProposalToOrdered: max: 0.317, avg: 0.286", "ConsensusOrderedToCommit: max: 0.372, avg: 0.359", "ConsensusProposalToCommit: max: 0.659, avg: 0.645"]
Max round gap was 1 [limit 4] at version 1979756. Max no progress secs was 5.346714 [limit 15] at version 1979756.
Test Ok

github-actions · 2024-06-11T04:57:31Z

✅ Forge suite `framework_upgrade` success on `a68e71c05caebf01504d4499110f3fba213fb53d` ==> `ecfcf887a0c0a079f2210dcb89ec5b63c6e433fa`

Compatibility test results for a68e71c05caebf01504d4499110f3fba213fb53d ==> ecfcf887a0c0a079f2210dcb89ec5b63c6e433fa (PR)
Upgrade the nodes to version: ecfcf887a0c0a079f2210dcb89ec5b63c6e433fa
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1191.1660872023626 txn/s, submitted: 1192.798460462037 txn/s, failed submission: 1.6323732596743419 txn/s, expired: 1.6323732596743419 txn/s, latency: 2641.5092795614723 ms, (p50: 2100 ms, p90: 4500 ms, p99: 9600 ms), latency samples: 102160
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1261.168975477044 txn/s, submitted: 1262.9254782284047 txn/s, failed submission: 1.7565027513607854 txn/s, expired: 1.7565027513607854 txn/s, latency: 2685.824263828094 ms, (p50: 1900 ms, p90: 4900 ms, p99: 11100 ms), latency samples: 100520
5. check swarm health
Compatibility test for a68e71c05caebf01504d4499110f3fba213fb53d ==> ecfcf887a0c0a079f2210dcb89ec5b63c6e433fa passed
Upgrade the remaining nodes to version: ecfcf887a0c0a079f2210dcb89ec5b63c6e433fa
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1197.4807792319996 txn/s, submitted: 1200.0032397014656 txn/s, failed submission: 2.522460469466104 txn/s, expired: 2.522460469466104 txn/s, latency: 2763.781491765607 ms, (p50: 2100 ms, p90: 5100 ms, p99: 8500 ms), latency samples: 104440
Test Ok

igor-aptos mentioned this pull request Dec 12, 2023

Always add StateCheckpoint after execution #11287

Merged

igor-aptos force-pushed the igor/add_checkpoint_after_execution branch from b83339f to 6cbd9d3 Compare December 12, 2023 17:09

igor-aptos force-pushed the igor/add_block_limit_info_payload branch from 407d137 to e11d72f Compare December 12, 2023 17:09

igor-aptos force-pushed the igor/add_checkpoint_after_execution branch from 6cbd9d3 to 723a255 Compare December 12, 2023 19:35

igor-aptos force-pushed the igor/add_block_limit_info_payload branch from e11d72f to ef71323 Compare December 12, 2023 19:36

igor-aptos force-pushed the igor/add_checkpoint_after_execution branch from 723a255 to ced8219 Compare December 12, 2023 21:12

igor-aptos force-pushed the igor/add_block_limit_info_payload branch from ef71323 to 0f7296a Compare December 12, 2023 21:12

igor-aptos changed the title ~~Add BlockEndInfo into StateCheckpoint~~ Add BlockEpilogue (replaces StateCheckpoint), and add BlockEndInfo into inside Dec 12, 2023

igor-aptos changed the title ~~Add BlockEpilogue (replaces StateCheckpoint), and add BlockEndInfo into inside~~ Add BlockEpilogue (replaces StateCheckpoint), with BlockEndInfo Dec 12, 2023

igor-aptos marked this pull request as ready for review December 12, 2023 21:15

igor-aptos requested review from msmouse, lightmark, gregnazario, zekun000, sasha8, ibalajiarun, gelash, danielxiangzl, davidiw, wrwg, banool and 0xmaayan as code owners December 12, 2023 21:15

igor-aptos force-pushed the igor/add_block_limit_info_payload branch from a3f7f65 to 747567a Compare June 9, 2024 15:29

igor-aptos added 6 commits June 9, 2024 08:30

add block limit info payload

ae19ec2

review comments

0856965

fix is_non_reconfig_block_ending

fef8442

fixing tests

12ada9d

changing to V0 enums

2d497dc

making sure empty blocks respect new flag to create BlockEpilogue

goldens

9864934

igor-aptos force-pushed the igor/add_block_limit_info_payload branch from 747567a to 104c66f Compare June 9, 2024 15:36

This comment has been minimized.

Sign in to view

igor-aptos force-pushed the igor/add_block_limit_info_payload branch from 104c66f to dc7a72c Compare June 10, 2024 05:03

igor-aptos enabled auto-merge (squash) June 10, 2024 05:04

This comment has been minimized.

Sign in to view

added smoke test, and fixed state checkpoint check

ecfcf88

igor-aptos force-pushed the igor/add_block_limit_info_payload branch from dc7a72c to ecfcf88 Compare June 11, 2024 04:28

This comment has been minimized.

Sign in to view

igor-aptos merged commit 4709e89 into main Jun 11, 2024
42 of 43 checks passed

igor-aptos deleted the igor/add_block_limit_info_payload branch June 11, 2024 05:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add BlockEpilogue (replaces StateCheckpoint), with BlockEndInfo #11298

Add BlockEpilogue (replaces StateCheckpoint), with BlockEndInfo #11298

igor-aptos commented Dec 12, 2023 •

edited

Loading

trunk-io bot commented Dec 12, 2023 •

edited

Loading

igor-aptos commented Dec 12, 2023 •

edited

Loading

codecov bot commented Dec 12, 2023 •

edited

Loading

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Jun 11, 2024

github-actions bot commented Jun 11, 2024

github-actions bot commented Jun 11, 2024

Add BlockEpilogue (replaces StateCheckpoint), with BlockEndInfo #11298

Add BlockEpilogue (replaces StateCheckpoint), with BlockEndInfo #11298

Conversation

igor-aptos commented Dec 12, 2023 • edited Loading

Description

Test Plan

trunk-io bot commented Dec 12, 2023 • edited Loading

igor-aptos commented Dec 12, 2023 • edited Loading

codecov bot commented Dec 12, 2023 • edited Loading

Codecov Report

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Jun 11, 2024

✅ Forge suite compat success on a68e71c05caebf01504d4499110f3fba213fb53d ==> ecfcf887a0c0a079f2210dcb89ec5b63c6e433fa

github-actions bot commented Jun 11, 2024

✅ Forge suite realistic_env_max_load success on ecfcf887a0c0a079f2210dcb89ec5b63c6e433fa

github-actions bot commented Jun 11, 2024

✅ Forge suite framework_upgrade success on a68e71c05caebf01504d4499110f3fba213fb53d ==> ecfcf887a0c0a079f2210dcb89ec5b63c6e433fa

igor-aptos commented Dec 12, 2023 •

edited

Loading

trunk-io bot commented Dec 12, 2023 •

edited

Loading

igor-aptos commented Dec 12, 2023 •

edited

Loading

codecov bot commented Dec 12, 2023 •

edited

Loading

✅ Forge suite `compat` success on `a68e71c05caebf01504d4499110f3fba213fb53d` ==> `ecfcf887a0c0a079f2210dcb89ec5b63c6e433fa`

✅ Forge suite `realistic_env_max_load` success on `ecfcf887a0c0a079f2210dcb89ec5b63c6e433fa`

✅ Forge suite `framework_upgrade` success on `a68e71c05caebf01504d4499110f3fba213fb53d` ==> `ecfcf887a0c0a079f2210dcb89ec5b63c6e433fa`