Skip to content

Commit

Permalink
polygon/sync: message listener to preserve peer events ordering (#10032)
Browse files Browse the repository at this point in the history
Observed the following issue in a long running Astrid sync on
bor-mainnet:
```
[DBUG] [04-17|14:25:43.504] [p2p.peerEventObserver] received new peer event id=Disconnect peerId=51935aa1eeabdb73b70d36c7d5953a3bfdf5c84e88241c44a7d16d508b281d397bdd8504c934bfb45af146b86eb5899ccea85e590774f9823d056a424080b763
[DBUG] [04-17|14:25:43.504] [p2p.peerEventObserver] received new peer event id=Connect peerId=51935aa1eeabdb73b70d36c7d5953a3bfdf5c84e88241c44a7d16d508b281d397bdd8504c934bfb45af146b86eb5899ccea85e590774f9823d056a424080b763
```

Note the timestamps are the same on the millisecond level, however the
disconnect was processed before the connect which is wrong (connect
should always be first).

This then got the `PeerTracker` in a bad state - it kept on returning
peer
`51935aa1eeabdb73b70d36c7d5953a3bfdf5c84e88241c44a7d16d508b281d397bdd8504c934bfb45af146b86eb5899ccea85e590774f9823d056a424080b763`
as a valid peer to download from, which caused repeated `peer not found`
errors when sending messages to it.

Fix is to have the message listener wait for all observers to finish
processing peer event 1 before proceeding to notifying them about peer
event 2.
  • Loading branch information
taratorio authored Apr 23, 2024
1 parent 6d9a5fd commit 190cbfa
Showing 1 changed file with 12 additions and 1 deletion.
13 changes: 12 additions & 1 deletion polygon/p2p/message_listener.go
Original file line number Diff line number Diff line change
Expand Up @@ -170,7 +170,18 @@ func (ml *messageListener) notifyPeerEventObservers(peerEvent *sentry.PeerEvent)
ml.observersMu.Lock()
defer ml.observersMu.Unlock()

notifyObservers(ml.peerEventObservers, peerEvent)
// wait on all observers to finish processing the peer event before notifying them
// with subsequent events in order to preserve the ordering of the sentry messages
var wg sync.WaitGroup
for _, observer := range ml.peerEventObservers {
wg.Add(1)
go func(observer MessageObserver[*sentry.PeerEvent]) {
defer wg.Done()
observer(peerEvent)
}(observer)
}

wg.Wait()
return nil
}

Expand Down

0 comments on commit 190cbfa

Please sign in to comment.