Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Libp2p stream limits #1017

Merged
merged 14 commits into from
Dec 28, 2022
Merged

Libp2p stream limits #1017

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
6b3b68b
Remove setting yamux window update mode, the same value is already th…
nazar-pc Dec 12, 2022
7d1247f
Restore tracking of number of connected peers
nazar-pc Dec 12, 2022
bfac7a9
Implement libp2p stream limits with wrappers
nazar-pc Dec 12, 2022
d50c0a9
Remove now unnecessary batching restrictions
nazar-pc Dec 12, 2022
acebfaa
Remove unnecessary boxing and pinning in various places
nazar-pc Dec 12, 2022
5029bf8
Subtract Kademlia tasks concurrency from the rest, so that only one S…
nazar-pc Dec 13, 2022
b43bd6c
Write basic tests for `maintain_semaphore_permits_capacity`
nazar-pc Dec 13, 2022
6060dd6
Drop permit when query is finished and not earlier
nazar-pc Dec 14, 2022
fc80120
Support peers boosting threshold customization, set it to 5 for more …
nazar-pc Dec 14, 2022
6e989d3
Resizable semaphore (#1019)
rahulksnv Dec 27, 2022
ce20efc
Merge remote-tracking branch 'origin/main' into libp2p-stream-limits
nazar-pc Dec 27, 2022
47fb5db
Update to newer libp2p revision
nazar-pc Dec 27, 2022
e98c358
Update libp2p to fork with kademlia handler buffering
nazar-pc Dec 28, 2022
21e719c
Increase timeout and max interval for piece publishing to reduce prob…
nazar-pc Dec 28, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions crates/subspace-networking/src/node_runner.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ use std::collections::hash_map::Entry;
use std::collections::{HashMap, HashSet};
use std::fmt::Debug;
use std::pin::Pin;
use std::sync::atomic::Ordering;
use std::sync::Weak;
use std::time::Duration;
use tokio::time::Sleep;
Expand Down Expand Up @@ -293,9 +294,18 @@ where
endpoint,
..
} => {
let shared = match self.shared_weak.upgrade() {
Some(shared) => shared,
None => {
return;
}
};
Comment on lines +333 to +338
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a note. We can do this right now:

let Some(shared) = self.shared_weak.upgrade() else { return };

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is exactly what I wrote initially. Unfortunately rustfmt seems to be unable to format let .. else, which is why I'm not using it yet anywhere. I'll be checking with new Rust releases as we upgrade and will start using it once formatting works.


let is_reserved_peer = self.reserved_peers.contains_key(&peer_id);
debug!(%peer_id, %is_reserved_peer, "Connection established [{num_established} from peer]");

shared.connected_peers_count.fetch_add(1, Ordering::SeqCst);

let (in_connections_number, out_connections_number) = {
let network_info = self.swarm.network_info();
let connections = network_info.connection_counters();
Expand Down Expand Up @@ -346,7 +356,15 @@ where
num_established,
..
} => {
let shared = match self.shared_weak.upgrade() {
Some(shared) => shared,
None => {
return;
}
};
debug!("Connection closed with peer {peer_id} [{num_established} from peer]");

shared.connected_peers_count.fetch_sub(1, Ordering::SeqCst);
}
SwarmEvent::OutgoingConnectionError { peer_id, error } => match error {
DialError::Transport(ref addresses) => {
Expand Down
3 changes: 3 additions & 0 deletions crates/subspace-networking/src/shared.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ use libp2p::gossipsub::error::{PublishError, SubscriptionError};
use libp2p::gossipsub::Sha256Topic;
use libp2p::{Multiaddr, PeerId};
use parking_lot::Mutex;
use std::sync::atomic::AtomicUsize;
use std::sync::Arc;

#[derive(Debug)]
Expand Down Expand Up @@ -83,6 +84,7 @@ pub(crate) struct Shared {
pub(crate) id: PeerId,
/// Addresses on which node is listening for incoming requests.
pub(crate) listeners: Mutex<Vec<Multiaddr>>,
pub(crate) connected_peers_count: Arc<AtomicUsize>,
/// Sender end of the channel for sending commands to the swarm.
pub(crate) command_sender: mpsc::Sender<Command>,
}
Expand All @@ -93,6 +95,7 @@ impl Shared {
handlers: Handlers::default(),
id,
listeners: Mutex::default(),
connected_peers_count: Arc::new(AtomicUsize::new(0)),
command_sender,
}
}
Expand Down