[server]: Start Peers Asynchronously #1658
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cfromknecht
added
p2p
|
Running this on the faucet now! |
cfromknecht
force-pushed
the
async-peer-start
branch
from
c2b8d35 to
921112f
Compare
|
Added commit to bump peer write timeout to 50s, and rebased on master |
halseth
previously approved these changes
Aug 1, 2018
server.go
Outdated
| @@ -1719,10 +1719,10 @@ func (s *server) findPeerByPubStr(pubStr string) (*peer, error) { | |||
| // the cleanup routine to exit early. | |||
| // | |||
| // NOTE: This MUST be launched as a goroutine. | |||
| func (s *server) peerTerminationWatcher(p *peer) { | |||
| func (s *server) peerTerminationWatcher(p *peer, ready chan struct{}) { | |||
There was a problem hiding this comment.
ready should be explained in the godoc
|
|
||
| // Otherwise, signal to the peerTerminationWatcher that the peer startup | ||
| // was successful, and to begin watching the peer's wait group. | ||
| close(ready) |
There was a problem hiding this comment.
What's the reason for doing this here, and not as the last thing in this method?
There was a problem hiding this comment.
It could be done later, as at this point even a Disconnect would unblock the select in WaitForDisconnect. I chose to put it here for clarity, and to stop selecting as soon as possible to avoid futex inflation
cfromknecht
force-pushed
the
async-peer-start
branch
3 times, most recently
from
90f5fb2 to
b90365f
Compare
halseth
previously approved these changes
Aug 2, 2018
cfromknecht
added
ready to merge
cfromknecht
force-pushed
the
async-peer-start
branch
2 times, most recently
from
2965f67 to
329a69c
Compare
|
Can be rebased now that the two dependent PR's have been merged. |
This commit adds additional synchronization logic to WaitForDisconnect, such that it can be spawned before Start has been executed by the server. Without modification, the current version will return immediately since no goroutines will have been spawned. To solve this, we modify WaitForDisconnect to block until: 1) the peer is disconnected, 2) the peer is successfully started, before watching the waitgroup. In the first case, the waitgroup will block until all (if any) spawned goroutines have exited. Otherwise, if the Start is successful, we can switch to watching the waitgroup, knowing that waitgroup counter is positive.
This commit adds asynchronous starting of peers, in order to avoid potential DOS vectors. Currently, we block with the server's mutex while peers exchange Init messages and perform other setup. Thus, a remote peer that does not reply with an init message will cause server to block for 15s per attempt. We also modify the startup behavior to spawn peerTerminationWatchers before starting the peer itself, ensuring that a peer is properly cleaned up if the initialization fails. Currently, failing to start a peer does not execute the bulk of the teardown logic, since it is not spawned until after a successful Start occurs.
Sometimes when performing an initial sync, the remote node isn't able to pull messages off the wire because of long running tasks and queues are saturated. With a shorter write timeout, we will give up trying to send messages and teardown the connection, even though the peer is still active.
cfromknecht
force-pushed
the
async-peer-start
branch
from
329a69c to
d4d9097
Compare
|
rebased and 💚 |
Roasbeef
approved these changes
Aug 16, 2018
| // call to Start returns no error. Otherwise, if the peer fails to start, | ||
| // calling Disconnect will signal the quit channel and the method will not | ||
| // block, since no goroutines were spawned. | ||
| func (p *peer) WaitForDisconnect(ready chan struct{}) { |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds asynchronous starting of peers,
in order to avoid potential DOS vectors. Currently,
we block with the server's mutex while peers exchange
Init messages and perform other setup. Thus, a remote
peer that does not reply with an init message will
cause server to block for 15s per attempt.
We also modify the startup behavior to spawn
peerTerminationWatchers before starting the
peer itself, ensuring that a peer is properly
cleaned up if the initialization fails. Currently,
failing to start a peer does not execute the bulk
of the teardown logic, since it is not spawned
until after a successful Start occurs.
The final commit is purely just a code move to place
the relevant methods in closer proximity, and
organize them roughly in the expected execution
order.
Prerequesities: