server: Implement separate mutex for peer state. #3251
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
davecgh
force-pushed
the
server_peer_state_mtx
branch
from
866b840 to
7772437
Compare
dajohi
approved these changes
May 13, 2024
jholdstock
reviewed
May 14, 2024
jholdstock
approved these changes
May 16, 2024
davecgh
force-pushed
the
server_peer_state_mtx
branch
from
7772437 to
0d0fbe7
Compare
Currently, all operations related to adding, removing, and banning peers are implemented via a single goroutine and channels to protect concurrent access. The existing implementation has worked well for over a decade, however, it is really not ideal in a few ways: - It is difficult to improve parallel processing since everything is forced into a single goroutine - It requires a lot of plumbing to provide access to any related information - The use of multiple channels means the ordering of events is not entirely well defined. In regards to the final point about event ordering, one example is that it's possible for a peer to be removed before it was ever added. The surrounding code is aware of these possibilities and handles things gracefully, but it really is not ideal. In practice, channels are best suited for passing ownership of data, distributing units of work, and communicating async results while mutexes are better suited for caches and state. Converting all of the code the code related to updating and querying the server's peer state to synchronous code that makes use of a separate mutex to protect it will address the aforementioned concerns. Namely, it: - Improves the semantics in regards to the aforementioned ordering - Ultimately allows more code to run in parallel in the individual peer goroutines - Requires less plumbing for updating and querying the state - Makes the state available to calling code so it can make better decisions Thus, this is part of a series of commits that aims to make that conversion. This first commit introduces a separate mutex on the peer state and updates the code to protect all references to the relevant fields with that mutex. This will allow future commits to methodically refactor the various operations without introducing races.
This updates the server peer connection request field to be safe for concurrent access by making it an atomic pointer. It also reorders some of the server peer fields and adds some comments that call out the concurrency semantics while here. This is a part of the overall effort to convert all of the code related to updating and querying the server's peer state to synchronous code that makes use of a separate mutex to protect it.
This modifies the connectionsWithIP method in the server to make use of the forAllPeers iterator instead of repeating the same logic for each peer category.
This refactors the logic related to adding a connected peer to the server out of the peer handler since it is now protected by the newly added peer state mutex. This is a part of the overall effort to convert all of the code related to updating and querying the server's peer state to synchronous code that makes use of a separate mutex to protect it.
This refactors the logic related to removing a disconnected peer from the server out of the peer handler since it is now protected by the newly added peer state mutex. This is a part of the overall effort to convert all of the code related to updating and querying the server's peer state to synchronous code that makes use of a separate mutex to protect it.
This refactors the logic for querying the number of connected peers out of the peer handler since it is now protected by the newly added peer state mutex. This is a part of the overall effort to convert all of the code related to updating and querying the server's peer state to synchronous code that makes use of a separate mutex to protect it.
This refactors the logic for querying the outbound group counts out of the peer handler since it is now protected by the newly added peer state mutex. This is a part of the overall effort to convert all of the code related to updating and querying the server's peer state to synchronous code that makes use of a separate mutex to protect it.
This refactors the logic for manually connecting peers out of the peer handler since it is now protected by the newly added peer state mutex. This is a part of the overall effort to convert all of the code related to updating and querying the server's peer state to synchronous code that makes use of a separate mutex to protect it.
This refactors the logic for canceling a pending connection out of the peer handler since it no longer needs to be plumbed through the query channel. This is a part of the overall effort to convert all of the code related to updating and querying the server's peer state to synchronous code that makes use of a separate mutex to protect it.
This refactors the logic for manually removing persistent peers out of the peer handler since it no longer needs to be plumbed through the query channel. This is a part of the overall effort to convert all of the code related to updating and querying the server's peer state to synchronous code that makes use of a separate mutex to protect it.
This refactors the logic for querying the persistent peers out of the peer handler since it is now protected by the newly added peer state mutex and no longer needs to be plumbed through the query channel.. This is a part of the overall effort to convert all of the code related to updating and querying the server's peer state to synchronous code that makes use of a separate mutex to protect it.
This refactors the logic for manually disconnecting peers out of the peer handler since it no longer needs to be plumbed through the query channel. This is a part of the overall effort to convert all of the code related to updating and querying the server's peer state to synchronous code that makes use of a separate mutex to protect it.
This removes the query channel and associated handler and plumbing now that it is no longer used. This completes the overall effort to convert all of the code related to updating and querying the server's peer state to synchronous code that makes use of a separate mutex to protect it.
davecgh
force-pushed
the
server_peer_state_mtx
branch
from
0d0fbe7 to
78bf86b
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, all operations related to adding, removing, and banning peers are implemented via a single goroutine and channels to protect concurrent access.
The existing implementation has worked well for over a decade, however, it is really not ideal in a few ways:
In regards to the final point about event ordering, one example is that it's possible for a peer to be removed before it was ever added. The surrounding code is aware of these possibilities and handles things gracefully, but it really is not ideal.
In practice, channels are best suited for passing ownership of data, distributing units of work, and communicating async results while mutexes are better suited for caches and state.
Converting all of the code the code related to updating and querying the server's peer state to synchronous code that makes use of a separate mutex to protect it will address the aforementioned concerns. Namely, it:
This consists of a series of commits to help ease the review process. Each commit is intended to be a self-contained and logically easy to follow change such that the code continues to compile and the tests continue to pass at each step.
See the description of each commit for further details.
Fixes #2694.