Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update libp2p to 0.41.0 #4717

Merged
merged 16 commits into from
Jan 10, 2023
Merged

Update libp2p to 0.41.0 #4717

merged 16 commits into from
Jan 10, 2023

Conversation

wemeetagain
Copy link
Member

@wemeetagain wemeetagain commented Nov 2, 2022 •
edited by dapplion
Loading

Motivation

We want upstream fixes and other goodies. Current version

lodestar/packages/beacon-node/package.json

Line 142 in 7d662d7

"libp2p": "0.39.2",

Description

Minimal edits to update to latest libp2p and networking dependencies

@wemeetagain wemeetagain requested a review from a team as a code owner November 2, 2022 18:26
@github-actions
Copy link
Contributor

github-actions bot commented Nov 2, 2022 •
edited
Loading

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: 9683b84 Previous: cdaf906 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 1.3152 ms/op 963.39 us/op 1.37
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 95.935 us/op 59.027 us/op 1.63
BLS verify - blst-native 2.5613 ms/op 2.1750 ms/op 1.18
BLS verifyMultipleSignatures 3 - blst-native 5.3461 ms/op 4.4828 ms/op 1.19
BLS verifyMultipleSignatures 8 - blst-native 11.934 ms/op 9.6939 ms/op 1.23
BLS verifyMultipleSignatures 32 - blst-native 42.432 ms/op 35.230 ms/op 1.20
BLS aggregatePubkeys 32 - blst-native 58.520 us/op 46.749 us/op 1.25
BLS aggregatePubkeys 128 - blst-native 229.94 us/op 183.71 us/op 1.25
getAttestationsForBlock 114.92 ms/op 75.259 ms/op 1.53
isKnown best case - 1 super set check 515.00 ns/op 444.00 ns/op 1.16
isKnown normal case - 2 super set checks 489.00 ns/op 431.00 ns/op 1.13
isKnown worse case - 16 super set checks 501.00 ns/op 436.00 ns/op 1.15
CheckpointStateCache - add get delete 10.295 us/op 8.2490 us/op 1.25
validate gossip signedAggregateAndProof - struct 6.1960 ms/op 5.0163 ms/op 1.24
validate gossip attestation - struct 2.9105 ms/op 2.3571 ms/op 1.23
pickEth1Vote - no votes 2.6292 ms/op 2.1445 ms/op 1.23
pickEth1Vote - max votes 20.686 ms/op 15.487 ms/op 1.34
pickEth1Vote - Eth1Data hashTreeRoot value x2048 15.676 ms/op 13.179 ms/op 1.19
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 25.243 ms/op 20.321 ms/op 1.24
pickEth1Vote - Eth1Data fastSerialize value x2048 1.5088 ms/op 1.0567 ms/op 1.43
pickEth1Vote - Eth1Data fastSerialize tree x2048 12.229 ms/op 9.6143 ms/op 1.27
bytes32 toHexString 1.2190 us/op 828.00 ns/op 1.47
bytes32 Buffer.toString(hex) 710.00 ns/op 606.00 ns/op 1.17
bytes32 Buffer.toString(hex) from Uint8Array 1.0840 us/op 971.00 ns/op 1.12
bytes32 Buffer.toString(hex) + 0x 694.00 ns/op 587.00 ns/op 1.18
Object access 1 prop 0.35300 ns/op 0.26500 ns/op 1.33
Map access 1 prop 0.32900 ns/op 0.24800 ns/op 1.33
Object get x1000 15.548 ns/op 10.957 ns/op 1.42
Map get x1000 1.0030 ns/op 0.90600 ns/op 1.11
Object set x1000 108.15 ns/op 67.827 ns/op 1.59
Map set x1000 75.953 ns/op 40.850 ns/op 1.86
Return object 10000 times 0.42170 ns/op 0.42840 ns/op 0.98
Throw Error 10000 times 7.5817 us/op 6.5882 us/op 1.15
fastMsgIdFn sha256 / 200 bytes 5.6390 us/op 4.8530 us/op 1.16
fastMsgIdFn h32 xxhash / 200 bytes 601.00 ns/op 448.00 ns/op 1.34
fastMsgIdFn h64 xxhash / 200 bytes 791.00 ns/op 594.00 ns/op 1.33
fastMsgIdFn sha256 / 1000 bytes 18.020 us/op 15.749 us/op 1.14
fastMsgIdFn h32 xxhash / 1000 bytes 823.00 ns/op 629.00 ns/op 1.31
fastMsgIdFn h64 xxhash / 1000 bytes 1.0360 us/op 711.00 ns/op 1.46
fastMsgIdFn sha256 / 10000 bytes 162.71 us/op 136.47 us/op 1.19
fastMsgIdFn h32 xxhash / 10000 bytes 2.7940 us/op 2.5170 us/op 1.11
fastMsgIdFn h64 xxhash / 10000 bytes 2.0000 us/op 1.7350 us/op 1.15
enrSubnets - fastDeserialize 64 bits 3.0680 us/op 2.1730 us/op 1.41
enrSubnets - ssz BitVector 64 bits 1.0550 us/op 763.00 ns/op 1.38
enrSubnets - fastDeserialize 4 bits 365.00 ns/op 269.00 ns/op 1.36
enrSubnets - ssz BitVector 4 bits 1.0520 us/op 757.00 ns/op 1.39
prioritizePeers score -10:0 att 32-0.1 sync 2-0 202.13 us/op 144.59 us/op 1.40
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 220.75 us/op 170.86 us/op 1.29
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 334.91 us/op 257.64 us/op 1.30
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 583.60 us/op 419.98 us/op 1.39
prioritizePeers score 0:0 att 64-1 sync 4-1 689.00 us/op 475.38 us/op 1.45
array of 16000 items push then shift 5.3194 us/op 52.679 us/op 0.10
LinkedList of 16000 items push then shift 18.477 ns/op 12.184 ns/op 1.52
array of 16000 items push then pop 236.33 ns/op 182.63 ns/op 1.29
LinkedList of 16000 items push then pop 17.559 ns/op 11.932 ns/op 1.47
array of 24000 items push then shift 8.0284 us/op 77.382 us/op 0.10
LinkedList of 24000 items push then shift 19.686 ns/op 12.393 ns/op 1.59
array of 24000 items push then pop 233.31 ns/op 189.17 ns/op 1.23
LinkedList of 24000 items push then pop 18.394 ns/op 11.891 ns/op 1.55
intersect bitArray bitLen 8 27.485 ns/op 21.698 ns/op 1.27
intersect array and set length 8 203.13 ns/op 123.11 ns/op 1.65
intersect bitArray bitLen 128 105.55 ns/op 70.580 ns/op 1.50
intersect array and set length 128 2.4310 us/op 1.6822 us/op 1.45
Buffer.concat 32 items 7.0800 us/op 4.4650 us/op 1.59
Uint8Array.set 32 items 4.6820 us/op 3.6640 us/op 1.28
pass gossip attestations to forkchoice per slot 4.7560 ms/op 3.7196 ms/op 1.28
computeDeltas 5.5113 ms/op 4.3452 ms/op 1.27
computeProposerBoostScoreFromBalances 2.8978 ms/op 2.4635 ms/op 1.18
altair processAttestation - 250000 vs - 7PWei normalcase 5.3394 ms/op 3.3213 ms/op 1.61
altair processAttestation - 250000 vs - 7PWei worstcase 7.4478 ms/op 6.0040 ms/op 1.24
altair processAttestation - setStatus - 1/6 committees join 248.60 us/op 177.44 us/op 1.40
altair processAttestation - setStatus - 1/3 committees join 440.11 us/op 353.42 us/op 1.25
altair processAttestation - setStatus - 1/2 committees join 657.18 us/op 517.09 us/op 1.27
altair processAttestation - setStatus - 2/3 committees join 873.43 us/op 679.61 us/op 1.29
altair processAttestation - setStatus - 4/5 committees join 1.1741 ms/op 937.59 us/op 1.25
altair processAttestation - setStatus - 100% committees join 1.4656 ms/op 1.1287 ms/op 1.30
altair processBlock - 250000 vs - 7PWei normalcase 31.399 ms/op 23.426 ms/op 1.34
altair processBlock - 250000 vs - 7PWei normalcase hashState 41.521 ms/op 37.870 ms/op 1.10
altair processBlock - 250000 vs - 7PWei worstcase 114.11 ms/op 80.566 ms/op 1.42
altair processBlock - 250000 vs - 7PWei worstcase hashState 128.45 ms/op 112.40 ms/op 1.14
phase0 processBlock - 250000 vs - 7PWei normalcase 4.2858 ms/op 3.1098 ms/op 1.38
phase0 processBlock - 250000 vs - 7PWei worstcase 63.964 ms/op 49.939 ms/op 1.28
altair processEth1Data - 250000 vs - 7PWei normalcase 1.0148 ms/op 616.23 us/op 1.65
vc - 250000 eb 1 eth1 1 we 0 wn 0 - smpl 15 20.175 us/op 8.6580 us/op 2.33
vc - 250000 eb 0.95 eth1 0.1 we 0.05 wn 0 - smpl 219 43.360 us/op 30.051 us/op 1.44
vc - 250000 eb 0.95 eth1 0.3 we 0.05 wn 0 - smpl 42 25.715 us/op 15.600 us/op 1.65
vc - 250000 eb 0.95 eth1 0.7 we 0.05 wn 0 - smpl 18 20.068 us/op 7.7490 us/op 2.59
vc - 250000 eb 0.1 eth1 0.1 we 0 wn 0 - smpl 1020 136.43 us/op 113.38 us/op 1.20
vc - 250000 eb 0.03 eth1 0.03 we 0 wn 0 - smpl 11777 1.2276 ms/op 1.0815 ms/op 1.14
vc - 250000 eb 0.01 eth1 0.01 we 0 wn 0 - smpl 16384 1.6497 ms/op 1.3330 ms/op 1.24
vc - 250000 eb 0 eth1 0 we 0 wn 0 - smpl 16384 1.6999 ms/op 1.2375 ms/op 1.37
vc - 250000 eb 0 eth1 0 we 0 wn 0 nocache - smpl 16384 5.6251 ms/op 4.1361 ms/op 1.36
vc - 250000 eb 0 eth1 1 we 0 wn 0 - smpl 16384 3.3348 ms/op 2.6958 ms/op 1.24
vc - 250000 eb 0 eth1 1 we 0 wn 0 nocache - smpl 16384 9.3383 ms/op 6.7442 ms/op 1.38
Tree 40 250000 create 986.92 ms/op 584.07 ms/op 1.69
Tree 40 250000 get(125000) 337.13 ns/op 235.08 ns/op 1.43
Tree 40 250000 set(125000) 3.5201 us/op 1.9102 us/op 1.84
Tree 40 250000 toArray() 36.020 ms/op 25.659 ms/op 1.40
Tree 40 250000 iterate all - toArray() + loop 34.698 ms/op 25.932 ms/op 1.34
Tree 40 250000 iterate all - get(i) 122.12 ms/op 104.21 ms/op 1.17
MutableVector 250000 create 15.179 ms/op 13.794 ms/op 1.10
MutableVector 250000 get(125000) 14.240 ns/op 10.543 ns/op 1.35
MutableVector 250000 set(125000) 828.05 ns/op 489.10 ns/op 1.69
MutableVector 250000 toArray() 7.0538 ms/op 5.4020 ms/op 1.31
MutableVector 250000 iterate all - toArray() + loop 7.3531 ms/op 5.9001 ms/op 1.25
MutableVector 250000 iterate all - get(i) 5.6180 ms/op 2.8309 ms/op 1.98
Array 250000 create 6.7901 ms/op 5.2225 ms/op 1.30
Array 250000 clone - spread 2.8701 ms/op 1.8693 ms/op 1.54
Array 250000 get(125000) 1.2950 ns/op 0.93000 ns/op 1.39
Array 250000 set(125000) 1.3710 ns/op 0.93600 ns/op 1.46
Array 250000 iterate all - loop 142.14 us/op 150.91 us/op 0.94
effectiveBalanceIncrements clone Uint8Array 300000 98.649 us/op 33.592 us/op 2.94
effectiveBalanceIncrements clone MutableVector 300000 918.00 ns/op 524.00 ns/op 1.75
effectiveBalanceIncrements rw all Uint8Array 300000 294.56 us/op 244.96 us/op 1.20
effectiveBalanceIncrements rw all MutableVector 300000 235.06 ms/op 132.81 ms/op 1.77
phase0 afterProcessEpoch - 250000 vs - 7PWei 208.14 ms/op 185.98 ms/op 1.12
phase0 beforeProcessEpoch - 250000 vs - 7PWei 120.05 ms/op 57.754 ms/op 2.08
altair processEpoch - mainnet_e81889 658.43 ms/op 476.16 ms/op 1.38
mainnet_e81889 - altair beforeProcessEpoch 156.81 ms/op 81.804 ms/op 1.92
mainnet_e81889 - altair processJustificationAndFinalization 61.456 us/op 17.394 us/op 3.53
mainnet_e81889 - altair processInactivityUpdates 12.076 ms/op 9.3023 ms/op 1.30
mainnet_e81889 - altair processRewardsAndPenalties 193.92 ms/op 76.778 ms/op 2.53
mainnet_e81889 - altair processRegistryUpdates 12.632 us/op 2.4730 us/op 5.11
mainnet_e81889 - altair processSlashings 2.4880 us/op 613.00 ns/op 4.06
mainnet_e81889 - altair processEth1DataReset 3.0660 us/op 628.00 ns/op 4.88
mainnet_e81889 - altair processEffectiveBalanceUpdates 2.6220 ms/op 1.9182 ms/op 1.37
mainnet_e81889 - altair processSlashingsReset 18.422 us/op 4.8590 us/op 3.79
mainnet_e81889 - altair processRandaoMixesReset 24.653 us/op 6.4430 us/op 3.83
mainnet_e81889 - altair processHistoricalRootsUpdate 3.3570 us/op 1.0260 us/op 3.27
mainnet_e81889 - altair processParticipationFlagUpdates 14.627 us/op 2.8500 us/op 5.13
mainnet_e81889 - altair processSyncCommitteeUpdates 3.0610 us/op 600.00 ns/op 5.10
mainnet_e81889 - altair afterProcessEpoch 212.77 ms/op 196.30 ms/op 1.08
phase0 processEpoch - mainnet_e58758 697.81 ms/op 479.47 ms/op 1.46
mainnet_e58758 - phase0 beforeProcessEpoch 299.35 ms/op 172.88 ms/op 1.73
mainnet_e58758 - phase0 processJustificationAndFinalization 59.491 us/op 19.144 us/op 3.11
mainnet_e58758 - phase0 processRewardsAndPenalties 134.07 ms/op 96.787 ms/op 1.39
mainnet_e58758 - phase0 processRegistryUpdates 30.811 us/op 7.7190 us/op 3.99
mainnet_e58758 - phase0 processSlashings 2.3680 us/op 558.00 ns/op 4.24
mainnet_e58758 - phase0 processEth1DataReset 2.9360 us/op 529.00 ns/op 5.55
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 2.2514 ms/op 1.6246 ms/op 1.39
mainnet_e58758 - phase0 processSlashingsReset 19.200 us/op 3.1390 us/op 6.12
mainnet_e58758 - phase0 processRandaoMixesReset 29.315 us/op 4.3360 us/op 6.76
mainnet_e58758 - phase0 processHistoricalRootsUpdate 3.9180 us/op 618.00 ns/op 6.34
mainnet_e58758 - phase0 processParticipationRecordUpdates 23.806 us/op 3.3680 us/op 7.07
mainnet_e58758 - phase0 afterProcessEpoch 169.99 ms/op 161.87 ms/op 1.05
phase0 processEffectiveBalanceUpdates - 250000 normalcase 2.3639 ms/op 1.9638 ms/op 1.20
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 2.5591 ms/op 2.2333 ms/op 1.15
altair processInactivityUpdates - 250000 normalcase 48.383 ms/op 39.687 ms/op 1.22
altair processInactivityUpdates - 250000 worstcase 58.671 ms/op 32.467 ms/op 1.81
phase0 processRegistryUpdates - 250000 normalcase 25.219 us/op 7.5390 us/op 3.35
phase0 processRegistryUpdates - 250000 badcase_full_deposits 508.05 us/op 376.19 us/op 1.35
phase0 processRegistryUpdates - 250000 worstcase 0.5 234.83 ms/op 191.19 ms/op 1.23
altair processRewardsAndPenalties - 250000 normalcase 140.52 ms/op 99.475 ms/op 1.41
altair processRewardsAndPenalties - 250000 worstcase 96.502 ms/op 72.162 ms/op 1.34
phase0 getAttestationDeltas - 250000 normalcase 14.762 ms/op 11.662 ms/op 1.27
phase0 getAttestationDeltas - 250000 worstcase 15.039 ms/op 12.425 ms/op 1.21
phase0 processSlashings - 250000 worstcase 6.3091 ms/op 5.2866 ms/op 1.19
altair processSyncCommitteeUpdates - 250000 325.28 ms/op 285.57 ms/op 1.14
BeaconState.hashTreeRoot - No change 536.00 ns/op 469.00 ns/op 1.14
BeaconState.hashTreeRoot - 1 full validator 79.363 us/op 70.978 us/op 1.12
BeaconState.hashTreeRoot - 32 full validator 1.0065 ms/op 765.89 us/op 1.31
BeaconState.hashTreeRoot - 512 full validator 8.1029 ms/op 7.7143 ms/op 1.05
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 104.63 us/op 89.752 us/op 1.17
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.5357 ms/op 1.2427 ms/op 1.24
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 20.716 ms/op 16.274 ms/op 1.27
BeaconState.hashTreeRoot - 1 balances 79.189 us/op 67.348 us/op 1.18
BeaconState.hashTreeRoot - 32 balances 724.38 us/op 655.85 us/op 1.10
BeaconState.hashTreeRoot - 512 balances 6.8306 ms/op 6.1179 ms/op 1.12
BeaconState.hashTreeRoot - 250000 balances 124.95 ms/op 97.861 ms/op 1.28
aggregationBits - 2048 els - zipIndexesInBitList 27.284 us/op 24.015 us/op 1.14
regular array get 100000 times 52.731 us/op 60.505 us/op 0.87
wrappedArray get 100000 times 55.092 us/op 60.504 us/op 0.91
arrayWithProxy get 100000 times 31.870 ms/op 29.275 ms/op 1.09
ssz.Root.equals 1.1640 us/op 908.00 ns/op 1.28
byteArrayEquals 1.1660 us/op 894.00 ns/op 1.30
shuffle list - 16384 els 12.289 ms/op 11.402 ms/op 1.08
shuffle list - 250000 els 179.91 ms/op 166.86 ms/op 1.08
processSlot - 1 slots 17.517 us/op 12.955 us/op 1.35
processSlot - 32 slots 2.4623 ms/op 1.9267 ms/op 1.28
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 372.53 us/op 313.37 us/op 1.19
getCommitteeAssignments - req 1 vs - 250000 vc 5.5120 ms/op 5.4327 ms/op 1.01
getCommitteeAssignments - req 100 vs - 250000 vc 7.8732 ms/op 7.9467 ms/op 0.99
getCommitteeAssignments - req 1000 vs - 250000 vc 8.5597 ms/op 8.5032 ms/op 1.01
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 9.6200 ns/op 7.3100 ns/op 1.32
state getBlockRootAtSlot - 250000 vs - 7PWei 1.4576 us/op 1.1473 us/op 1.27
computeProposers - vc 250000 20.385 ms/op 16.968 ms/op 1.20
computeEpochShuffling - vc 250000 189.85 ms/op 171.22 ms/op 1.11
getNextSyncCommittee - vc 250000 340.13 ms/op 282.89 ms/op 1.20

by benchmarkbot/action

This was referenced Nov 3, 2022
Closed
Closed
@wemeetagain wemeetagain marked this pull request as draft November 4, 2022 14:21
@philknows philknows added this to the v1.4.0 milestone Nov 8, 2022
@wemeetagain wemeetagain changed the title Update libp2p to 0.40.0 Update libp2p to 0.41.0 Dec 5, 2022
@twoeths twoeths mentioned this pull request Dec 14, 2022
Closed
@dapplion dapplion mentioned this pull request Dec 30, 2022
Closed
@wemeetagain
Copy link
Member Author

It seems there's a memory leak.

Screenshot shows high process memory (the spike back down is process being restarted)

Screenshot from 2023-01-04 11-27-54

@wemeetagain
Copy link
Member Author

This leak may also be present in the unstable branch. Current theory is its related to the rate of inbound peer connections.

@wemeetagain wemeetagain marked this pull request as ready for review January 9, 2023 16:15
@wemeetagain wemeetagain merged commit bd3d91a into unstable Jan 10, 2023
@wemeetagain wemeetagain deleted the cayman/update-libp2p branch January 10, 2023 03:51
@twoeths
Copy link
Contributor

twoeths commented Jan 11, 2023

This leak may also be present in the unstable branch. Current theory is its related to the rate of inbound peer connections.

yes, specifically it's due to ChainSafe/node-snappy-stream#16

@nflaig nflaig mentioned this pull request Jan 31, 2023
Merged
18 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dapplion dapplion dapplion approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v1.4.0
Development

Successfully merging this pull request may close these issues.
4 participants
@wemeetagain @twoeths @dapplion @philknows