Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Run spec tests v1.2.0-rc.1 #4206

Closed
wants to merge 1 commit into from
Closed

Run spec tests v1.2.0-rc.1 #4206

wants to merge 1 commit into from

Conversation

dapplion
Copy link
Contributor

Motivation

Start preparing to support v1.2.0.

Description

Lodestar is missing

  • fork-choice equivocations
  • handle large withdrawable epoch

@dapplion dapplion force-pushed the dapplion/spec-tests-v1.2.0-rc.1 branch from 0be34e2 to d25f545 Compare June 27, 2022 14:16
@dapplion dapplion force-pushed the dapplion/spec-tests-v1.2.0-rc.1 branch from d25f545 to c83f165 Compare June 27, 2022 14:36
@github-actions
Copy link
Contributor

github-actions bot commented Jun 27, 2022 •
edited
Loading

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: c6dc155 Previous: f15a8a9 Ratio
altair processAttestation - 250000 vs - 7PWei normalcase 3.8378 ms/op 3.9165 ms/op 0.98
altair processAttestation - 250000 vs - 7PWei worstcase 6.1013 ms/op 6.2333 ms/op 0.98
altair processAttestation - setStatus - 1/6 committees join 229.88 us/op 234.27 us/op 0.98
altair processAttestation - setStatus - 1/3 committees join 456.88 us/op 452.00 us/op 1.01
altair processAttestation - setStatus - 1/2 committees join 652.78 us/op 631.51 us/op 1.03
altair processAttestation - setStatus - 2/3 committees join 843.84 us/op 797.63 us/op 1.06
altair processAttestation - setStatus - 4/5 committees join 1.2270 ms/op 1.1105 ms/op 1.10
altair processAttestation - setStatus - 100% committees join 1.3664 ms/op 1.3544 ms/op 1.01
altair processBlock - 250000 vs - 7PWei normalcase 30.203 ms/op 32.468 ms/op 0.93
altair processBlock - 250000 vs - 7PWei normalcase hashState 38.539 ms/op 36.532 ms/op 1.05
altair processBlock - 250000 vs - 7PWei worstcase 91.938 ms/op 89.255 ms/op 1.03
altair processBlock - 250000 vs - 7PWei worstcase hashState 115.69 ms/op 119.69 ms/op 0.97
phase0 processBlock - 250000 vs - 7PWei normalcase 4.0496 ms/op 4.1237 ms/op 0.98
phase0 processBlock - 250000 vs - 7PWei worstcase 55.228 ms/op 53.074 ms/op 1.04
altair processEth1Data - 250000 vs - 7PWei normalcase 747.68 us/op 791.57 us/op 0.94
Tree 40 250000 create 864.26 ms/op 851.93 ms/op 1.01
Tree 40 250000 get(125000) 334.91 ns/op 329.00 ns/op 1.02
Tree 40 250000 set(125000) 2.9421 us/op 2.7907 us/op 1.05
Tree 40 250000 toArray() 36.682 ms/op 37.155 ms/op 0.99
Tree 40 250000 iterate all - toArray() + loop 37.373 ms/op 36.086 ms/op 1.04
Tree 40 250000 iterate all - get(i) 124.92 ms/op 129.66 ms/op 0.96
MutableVector 250000 create 20.859 ms/op 20.521 ms/op 1.02
MutableVector 250000 get(125000) 15.558 ns/op 15.825 ns/op 0.98
MutableVector 250000 set(125000) 705.99 ns/op 729.26 ns/op 0.97
MutableVector 250000 toArray() 7.9552 ms/op 8.4299 ms/op 0.94
MutableVector 250000 iterate all - toArray() + loop 8.5736 ms/op 8.8632 ms/op 0.97
MutableVector 250000 iterate all - get(i) 3.8008 ms/op 3.7434 ms/op 1.02
Array 250000 create 7.6212 ms/op 7.5938 ms/op 1.00
Array 250000 clone - spread 4.0546 ms/op 4.3780 ms/op 0.93
Array 250000 get(125000) 1.7470 ns/op 1.8460 ns/op 0.95
Array 250000 set(125000) 1.7330 ns/op 1.7530 ns/op 0.99
Array 250000 iterate all - loop 196.77 us/op 197.70 us/op 1.00
effectiveBalanceIncrements clone Uint8Array 300000 109.72 us/op 1.3570 ms/op 0.08
effectiveBalanceIncrements clone MutableVector 300000 1.5490 us/op 901.00 ns/op 1.72
effectiveBalanceIncrements rw all Uint8Array 300000 289.75 us/op 289.99 us/op 1.00
effectiveBalanceIncrements rw all MutableVector 300000 224.86 ms/op 180.51 ms/op 1.25
phase0 afterProcessEpoch - 250000 vs - 7PWei 217.67 ms/op 231.27 ms/op 0.94
phase0 beforeProcessEpoch - 250000 vs - 7PWei 99.149 ms/op 76.269 ms/op 1.30
altair processEpoch - mainnet_e81889 573.01 ms/op 640.22 ms/op 0.90
mainnet_e81889 - altair beforeProcessEpoch 166.94 ms/op 176.98 ms/op 0.94
mainnet_e81889 - altair processJustificationAndFinalization 43.354 us/op 37.777 us/op 1.15
mainnet_e81889 - altair processInactivityUpdates 13.219 ms/op 12.504 ms/op 1.06
mainnet_e81889 - altair processRewardsAndPenalties 155.86 ms/op 96.091 ms/op 1.62
mainnet_e81889 - altair processRegistryUpdates 8.5840 us/op 8.1820 us/op 1.05
mainnet_e81889 - altair processSlashings 1.5790 us/op 2.5210 us/op 0.63
mainnet_e81889 - altair processEth1DataReset 2.2830 us/op 2.4000 us/op 0.95
mainnet_e81889 - altair processEffectiveBalanceUpdates 2.9690 ms/op 2.8202 ms/op 1.05
mainnet_e81889 - altair processSlashingsReset 13.819 us/op 12.394 us/op 1.11
mainnet_e81889 - altair processRandaoMixesReset 12.959 us/op 11.485 us/op 1.13
mainnet_e81889 - altair processHistoricalRootsUpdate 2.3780 us/op 1.9530 us/op 1.22
mainnet_e81889 - altair processParticipationFlagUpdates 7.0770 us/op 6.1320 us/op 1.15
mainnet_e81889 - altair processSyncCommitteeUpdates 1.6210 us/op 1.9150 us/op 0.85
mainnet_e81889 - altair afterProcessEpoch 229.96 ms/op 219.63 ms/op 1.05
phase0 processEpoch - mainnet_e58758 578.44 ms/op 556.72 ms/op 1.04
mainnet_e58758 - phase0 beforeProcessEpoch 256.43 ms/op 204.55 ms/op 1.25
mainnet_e58758 - phase0 processJustificationAndFinalization 32.986 us/op 32.194 us/op 1.02
mainnet_e58758 - phase0 processRewardsAndPenalties 142.60 ms/op 79.440 ms/op 1.80
mainnet_e58758 - phase0 processRegistryUpdates 20.976 us/op 19.707 us/op 1.06
mainnet_e58758 - phase0 processSlashings 1.8210 us/op 2.0700 us/op 0.88
mainnet_e58758 - phase0 processEth1DataReset 1.9890 us/op 2.2100 us/op 0.90
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 2.3763 ms/op 2.3710 ms/op 1.00
mainnet_e58758 - phase0 processSlashingsReset 10.176 us/op 11.897 us/op 0.86
mainnet_e58758 - phase0 processRandaoMixesReset 13.495 us/op 14.361 us/op 0.94
mainnet_e58758 - phase0 processHistoricalRootsUpdate 2.4280 us/op 2.4100 us/op 1.01
mainnet_e58758 - phase0 processParticipationRecordUpdates 12.682 us/op 13.466 us/op 0.94
mainnet_e58758 - phase0 afterProcessEpoch 188.15 ms/op 187.97 ms/op 1.00
phase0 processEffectiveBalanceUpdates - 250000 normalcase 3.4971 ms/op 3.4517 ms/op 1.01
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 3.7114 ms/op 3.5898 ms/op 1.03
altair processInactivityUpdates - 250000 normalcase 28.958 ms/op 34.391 ms/op 0.84
altair processInactivityUpdates - 250000 worstcase 36.434 ms/op 33.781 ms/op 1.08
phase0 processRegistryUpdates - 250000 normalcase 16.225 us/op 15.179 us/op 1.07
phase0 processRegistryUpdates - 250000 badcase_full_deposits 523.84 us/op 461.96 us/op 1.13
phase0 processRegistryUpdates - 250000 worstcase 0.5 231.64 ms/op 236.59 ms/op 0.98
altair processRewardsAndPenalties - 250000 normalcase 97.561 ms/op 138.51 ms/op 0.70
altair processRewardsAndPenalties - 250000 worstcase 145.14 ms/op 111.27 ms/op 1.30
phase0 getAttestationDeltas - 250000 normalcase 14.561 ms/op 14.021 ms/op 1.04
phase0 getAttestationDeltas - 250000 worstcase 14.992 ms/op 14.676 ms/op 1.02
phase0 processSlashings - 250000 worstcase 6.3985 ms/op 6.3151 ms/op 1.01
altair processSyncCommitteeUpdates - 250000 329.62 ms/op 321.90 ms/op 1.02
BeaconState.hashTreeRoot - No change 701.00 ns/op 677.00 ns/op 1.04
BeaconState.hashTreeRoot - 1 full validator 74.952 us/op 73.426 us/op 1.02
BeaconState.hashTreeRoot - 32 full validator 794.29 us/op 735.54 us/op 1.08
BeaconState.hashTreeRoot - 512 full validator 6.8329 ms/op 7.0296 ms/op 0.97
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 89.073 us/op 147.37 us/op 0.60
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.3000 ms/op 1.2895 ms/op 1.01
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 16.725 ms/op 16.796 ms/op 1.00
BeaconState.hashTreeRoot - 1 balances 68.986 us/op 70.949 us/op 0.97
BeaconState.hashTreeRoot - 32 balances 719.15 us/op 665.91 us/op 1.08
BeaconState.hashTreeRoot - 512 balances 5.7239 ms/op 5.7819 ms/op 0.99
BeaconState.hashTreeRoot - 250000 balances 99.824 ms/op 98.458 ms/op 1.01
aggregationBits - 2048 els - zipIndexesInBitList 38.972 us/op 38.147 us/op 1.02
regular array get 100000 times 76.803 us/op 80.184 us/op 0.96
wrappedArray get 100000 times 76.610 us/op 79.596 us/op 0.96
arrayWithProxy get 100000 times 34.413 ms/op 34.456 ms/op 1.00
ssz.Root.equals 614.00 ns/op 572.00 ns/op 1.07
byteArrayEquals 587.00 ns/op 612.00 ns/op 0.96
shuffle list - 16384 els 13.115 ms/op 12.921 ms/op 1.01
shuffle list - 250000 els 191.35 ms/op 188.95 ms/op 1.01
processSlot - 1 slots 14.211 us/op 14.338 us/op 0.99
processSlot - 32 slots 2.1917 ms/op 2.2247 ms/op 0.99
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 441.34 us/op 444.12 us/op 0.99
getCommitteeAssignments - req 1 vs - 250000 vc 6.1102 ms/op 6.1436 ms/op 0.99
getCommitteeAssignments - req 100 vs - 250000 vc 8.3369 ms/op 8.6063 ms/op 0.97
getCommitteeAssignments - req 1000 vs - 250000 vc 9.1255 ms/op 8.8560 ms/op 1.03
computeProposers - vc 250000 20.742 ms/op 20.233 ms/op 1.03
computeEpochShuffling - vc 250000 190.60 ms/op 191.39 ms/op 1.00
getNextSyncCommittee - vc 250000 316.73 ms/op 307.85 ms/op 1.03
pass gossip attestations to forkchoice per slot 6.9002 ms/op 5.9999 ms/op 1.15
computeDeltas 3.8600 ms/op 3.7288 ms/op 1.04
computeProposerBoostScoreFromBalances 1.0230 ms/op 1.0644 ms/op 0.96
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 2.3787 ms/op 2.2771 ms/op 1.04
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 92.600 us/op 85.685 us/op 1.08
BLS verify - blst-native 2.0897 ms/op 2.1530 ms/op 0.97
BLS verifyMultipleSignatures 3 - blst-native 4.0457 ms/op 4.3684 ms/op 0.93
BLS verifyMultipleSignatures 8 - blst-native 8.8086 ms/op 9.4402 ms/op 0.93
BLS verifyMultipleSignatures 32 - blst-native 31.698 ms/op 34.038 ms/op 0.93
BLS aggregatePubkeys 32 - blst-native 44.349 us/op 46.258 us/op 0.96
BLS aggregatePubkeys 128 - blst-native 169.37 us/op 176.19 us/op 0.96
getAttestationsForBlock 78.244 ms/op 78.258 ms/op 1.00
isKnown best case - 1 super set check 505.00 ns/op 516.00 ns/op 0.98
isKnown normal case - 2 super set checks 498.00 ns/op 519.00 ns/op 0.96
isKnown worse case - 16 super set checks 508.00 ns/op 496.00 ns/op 1.02
CheckpointStateCache - add get delete 12.270 us/op 11.982 us/op 1.02
validate gossip signedAggregateAndProof - struct 4.8631 ms/op 4.8178 ms/op 1.01
validate gossip attestation - struct 2.2823 ms/op 2.2959 ms/op 0.99
altair verifyImport mainnet_s3766816:31 7.3774 s/op 7.2128 s/op 1.02
pickEth1Vote - no votes 2.2575 ms/op 2.3032 ms/op 0.98
pickEth1Vote - max votes 27.200 ms/op 26.698 ms/op 1.02
pickEth1Vote - Eth1Data hashTreeRoot value x2048 14.406 ms/op 12.779 ms/op 1.13
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 25.079 ms/op 23.152 ms/op 1.08
pickEth1Vote - Eth1Data fastSerialize value x2048 1.8104 ms/op 1.7051 ms/op 1.06
pickEth1Vote - Eth1Data fastSerialize tree x2048 20.695 ms/op 19.149 ms/op 1.08
bytes32 toHexString 1.3370 us/op 1.1830 us/op 1.13
bytes32 Buffer.toString(hex) 875.00 ns/op 749.00 ns/op 1.17
bytes32 Buffer.toString(hex) from Uint8Array 1.1280 us/op 1.0810 us/op 1.04
bytes32 Buffer.toString(hex) + 0x 807.00 ns/op 784.00 ns/op 1.03
Object access 1 prop 0.41400 ns/op 0.41100 ns/op 1.01
Map access 1 prop 0.34500 ns/op 0.35000 ns/op 0.99
Object get x1000 20.226 ns/op 20.191 ns/op 1.00
Map get x1000 1.2070 ns/op 1.1530 ns/op 1.05
Object set x1000 128.92 ns/op 132.55 ns/op 0.97
Map set x1000 78.984 ns/op 78.088 ns/op 1.01
Return object 10000 times 0.41330 ns/op 0.43370 ns/op 0.95
Throw Error 10000 times 6.5748 us/op 6.7524 us/op 0.97
enrSubnets - fastDeserialize 64 bits 3.1360 us/op 3.0520 us/op 1.03
enrSubnets - ssz BitVector 64 bits 897.00 ns/op 898.00 ns/op 1.00
enrSubnets - fastDeserialize 4 bits 459.00 ns/op 445.00 ns/op 1.03
enrSubnets - ssz BitVector 4 bits 825.00 ns/op 817.00 ns/op 1.01
prioritizePeers score -10:0 att 32-0.1 sync 2-0 118.64 us/op 101.92 us/op 1.16
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 140.06 us/op 135.19 us/op 1.04
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 242.81 us/op 242.51 us/op 1.00
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 511.45 us/op 502.08 us/op 1.02
prioritizePeers score 0:0 att 64-1 sync 4-1 505.03 us/op 503.19 us/op 1.00
RateTracker 1000000 limit, 1 obj count per request 213.72 ns/op 208.55 ns/op 1.02
RateTracker 1000000 limit, 2 obj count per request 157.68 ns/op 153.05 ns/op 1.03
RateTracker 1000000 limit, 4 obj count per request 128.29 ns/op 128.42 ns/op 1.00
RateTracker 1000000 limit, 8 obj count per request 119.10 ns/op 116.02 ns/op 1.03
RateTracker with prune 4.9350 us/op 4.6620 us/op 1.06
array of 16000 items push then shift 3.5227 us/op 3.6665 us/op 0.96
LinkedList of 16000 items push then shift 28.057 ns/op 25.685 ns/op 1.09
array of 16000 items push then pop 273.52 ns/op 265.06 ns/op 1.03
LinkedList of 16000 items push then pop 22.149 ns/op 22.793 ns/op 0.97
array of 24000 items push then shift 4.9744 us/op 5.1494 us/op 0.97
LinkedList of 24000 items push then shift 27.705 ns/op 26.283 ns/op 1.05
array of 24000 items push then pop 221.47 ns/op 227.06 ns/op 0.98
LinkedList of 24000 items push then pop 22.049 ns/op 23.281 ns/op 0.95
intersect bitArray bitLen 8 13.083 ns/op 13.357 ns/op 0.98
intersect array and set length 8 180.13 ns/op 183.17 ns/op 0.98
intersect bitArray bitLen 128 79.310 ns/op 81.043 ns/op 0.98
intersect array and set length 128 2.4886 us/op 2.4226 us/op 1.03

by benchmarkbot/action

@dapplion
Copy link
Contributor Author

@dapplion dapplion closed this Jul 12, 2022
@dapplion dapplion deleted the dapplion/spec-tests-v1.2.0-rc.1 branch August 6, 2022 08:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@dapplion