Proposed DHTv2 Changes #473

aschmahmann · 2020-03-03T22:05:14Z

This branch contains the various changes to the DHT that will result in us bumping the DHT protocol version.

The changes include:

Query logic that more accurately matches Kademlia (taken from Query logic closely match Kademlia paper #436)
Allowing the DHT to switch between client and server modes Enable switching DHT between client and server modes #469
Configurable peer filtering Configurable Peer Filtering #471
Utilizing libp2p events to add peers to the routing table Utilize identify events to add peers to the routing table #472

routing.go

query.go

…h means do not abort the query early

…ient.

…we have heard about (in a given path) has been found

…ally. Concurrency option now sets alpha. DisjointPaths option now sets d. Default number of disjoint paths is now bucketSize/2.

* feat: consume identify events to evaluate routing table addition * fix: routing table no longer gets an update just because new messages have arrived or been sent * fix: add already connected peers into the routing table before listening to events Co-authored-by: Raúl Kripalani <[email protected]> Co-authored-by: Aarsh Shah <[email protected]>

1. Continue to best-effort provide, but still return an error when we fail to send provider records to the _best_ peers. 2. Continue returning the best peer's we've found in GetClosestPeers, but also return an error to indicate that we didn't find the closest ones. And fix the hang test.

Allows specifying peer filter functions on query and on adding peers to the routing table. This patch also includes some reasonable default functions for a public-only and private-only DHT. Co-authored-by: Will Scott <[email protected]> Co-authored-by: Steven Allen <[email protected]>

This should stop us from waiting on unresponsive peers. This only kicks in when we've already _sent_ a request so this: * Shouldn't be blocked on other requests we're planning on making. * Shouldn't be blocked on dialing.

* Add Kademlia async lookup implementation (replaces prior implementation) * Add lookup events that allow full lookup reconstruction from logs * Introduce lookup stopping condition, based on parameter "beta" Co-authored-by: Adin Schmahmann <[email protected]>

Previously, we'd only return the target peer if we were connected to them. However, this makes it difficult to impossible to find peers that are mostly disconnected from the network. This change also changes `p` to `from` in several places as `p` is _very_ confusing. We should probably switch away from using `p` everywhere.

* new RT management policy

* fix Low Peers and Cleanup

…rs (#536) * record valuable peers algorithm only takes into account latency of seed peers

…ns an error instead of hanging

Except deprecated warn calls

* Remove deprecated events. * Fix warning -> warn. * Use structured logging (some places).

The querying node _is_ allowed to reset and go away. fixes #431

When this option is passed, the DHT node will listen on and query _only_ using the old DHT protocol. Importantly, the node won't even pretend to be a new DHT node because it's routing table includes V1 peers.

We no longer need this now that we explicitly support v1 compatibility mode.

…nd refresh routing tables for orphaned nodes

Stebalien · 2020-04-06T04:44:00Z

aschmahmann mentioned this pull request Mar 3, 2020

Query logic closely match Kademlia paper #436

Closed

Stebalien force-pushed the cypress branch 4 times, most recently from 657cbbd to e505723 Compare March 6, 2020 04:02

aarshkshah1992 reviewed Mar 6, 2020

View reviewed changes

routing.go Outdated Show resolved Hide resolved

query.go Outdated Show resolved Hide resolved

aarshkshah1992 reviewed Mar 6, 2020

View reviewed changes

query.go Outdated Show resolved Hide resolved

aarshkshah1992 reviewed Mar 6, 2020

View reviewed changes

query.go Outdated Show resolved Hide resolved

Stebalien added this to the Working Kademlia milestone Mar 10, 2020

aschmahmann force-pushed the cypress branch from 20355c5 to 7ada018 Compare March 10, 2020 20:24

Stebalien force-pushed the cypress branch from 54a4c81 to e42ee94 Compare March 13, 2020 22:05

aschmahmann force-pushed the cypress branch 2 times, most recently from 82d847c to 314213d Compare March 26, 2020 23:25

aschmahmann and others added 17 commits April 3, 2020 12:50

upgrade deprecated query events

57a12ce

first pass at proper kad

6793dc1

fix Search/GetValue to be Kad compliant. Default quorum is now 0 whic…

564764f

…h means do not abort the query early

fix refreshing buckets during query. getting topK peers is more effic…

616b0e1

…ient.

Search/GetValues stops if using quorum

4aee8bb

error on queries to empty routing table

6dac71c

query switches from alpha mode to k mode if no peers closer than one …

85f88e1

…we have heard about (in a given path) has been found

SearchValue will only update k closest peers to have latest record

c7a8d0d

added todo comment about FindPeer and peer connectedness

cfb4b40

fix: use dht.bucketSize instead of KValue for queries

19e4d2e

test: switch tests to use dht.bucketSize instead of KValue for queries

11776e0

options: KValue and AlphaValue global variables no longer used intern…

e967aca

…ally. Concurrency option now sets alpha. DisjointPaths option now sets d. Default number of disjoint paths is now bucketSize/2.

try another peer sorting metric. fix bug in when findproviders returns

dd0bbb2

check for stopping the query in between dialing and querying

4c2cf46

cleanup unused code

41f47f7

Stebalien and others added 13 commits April 3, 2020 12:50

chore: run with race detector enabled and disabled

fa52ddd

bump the peerstore version

29bc71a

feat: reduce request read timeout (#527)

fad6c8a

This should stop us from waiting on unresponsive peers. This only kicks in when we've already _sent_ a request so this: * Shouldn't be blocked on other requests we're planning on making. * Shouldn't be blocked on dialing.

fix: cleanup context cancellations (#530)

39990c0

New RT management policy (#520)

8fa0438

* new RT management policy

spawn the correct number of simultaneous queries

d715495

Periodic Pinging and active fixLowPeers (#533)

2539114

* fix Low Peers and Cleanup

Record valuable peers will only take into account latency of seed pee…

6a36698

…rs (#536) * record valuable peers algorithm only takes into account latency of seed peers

chore: update deps

485c60a

refreshing the routing table after the dht context is cancelled retur…

a4eaa70

…ns an error instead of hanging

Stebalien force-pushed the cypress branch from f3318e8 to a4eaa70 Compare April 3, 2020 19:52

Stebalien and others added 7 commits April 3, 2020 15:19

chore: fix lints

5aad954

Except deprecated warn calls

feat: better logging

445472b

* Remove deprecated events. * Fix warning -> warn. * Use structured logging (some places).

ci: use IPFS circle configs

dfea791

fix(test): don't panic on read/write error

d520a1e

The querying node _is_ allowed to reset and go away. fixes #431

chore(ci): remove travis

deee8b7

remove deprecated global variables

5e318d3

use dht context during refreshes so that they can be logged

6f1df26

Stebalien mentioned this pull request Apr 5, 2020

ConnManager should be able to create connections libp2p/go-libp2p#870

Closed

Stebalien and others added 4 commits April 5, 2020 19:28

feat: add a v1 compatibility mode

9bab969

When this option is passed, the DHT node will listen on and query _only_ using the old DHT protocol. Importantly, the node won't even pretend to be a new DHT node because it's routing table includes V1 peers.

chore: remove unused testProtocols

8af12dd

We no longer need this now that we explicitly support v1 compatibility mode.

chore: update kbucket

7035c59

test: fix TestFindPeerQuery and TestFindPeerQueryMinimal to not try a…

a7fd35f

…nd refresh routing tables for orphaned nodes

Stebalien marked this pull request as ready for review April 6, 2020 04:27

Stebalien approved these changes Apr 6, 2020

View reviewed changes

aschmahmann merged commit 1ba0310 into master Apr 6, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Proposed DHTv2 Changes #473

Proposed DHTv2 Changes #473

aschmahmann commented Mar 3, 2020

Stebalien commented Apr 6, 2020

Proposed DHTv2 Changes #473

Proposed DHTv2 Changes #473

Conversation

aschmahmann commented Mar 3, 2020

Stebalien commented Apr 6, 2020