feat: async peerstore backed by datastores #1058
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We have a peerstore that keeps all data for all observed peers in memory with no eviction. This is fine when you don't discover many peers but when using the DHT you encounter a significant number of peers so our peer storage grows and grows over time. We have a persistent peer store, but it just periodically writes peers into the datastore to be read at startup, still keeping them in memory. This change refactors the peerstore to use a datastore by default, reading and writing peer info as it arrives. It can be configured with a MemoryDatastore if desired. It was necessary to change the peerstore and *book interfaces to be asynchronous since the datastore api is asynchronous. This will need a follow up PR to libp2p-interfaces and others.
achingbrain
force-pushed
the
feat/async-peerstore
branch
from
d028eab to
5e38d87
Compare
|
Heap usage after running this branch for 24 hours, just idling with the dht self-query running, seems healthy:
Running the same way with master and alanshaw/abortable-iterator#15 patched in:
Network conditions were different between the tests so absolute memory usage isn't comparable, but the trend is clear. |
achingbrain
added a commit
to libp2p/js-libp2p-interfaces
that referenced
this pull request
Dec 28, 2021
Refactors interfaces and classes used by `libp2p-interfaces` to use the async peer store from libp2p/js-libp2p#1058
achingbrain
added a commit
to libp2p/js-libp2p-kad-dht
that referenced
this pull request
Dec 29, 2021
Refactors interfaces and classes used by `libp2p-interfaces` to use the async peer store from libp2p/js-libp2p#1058 BREAKING CHANGE: peerstore methods are now all async
achingbrain
added a commit
to libp2p/js-libp2p-interfaces
that referenced
this pull request
Dec 30, 2021
Refactors interfaces and classes used by `libp2p-interfaces` to use the async peer store from libp2p/js-libp2p#1058 BREAKING CHANGE: peerstore methods are now all async
achingbrain
added a commit
to libp2p/js-libp2p-kad-dht
that referenced
this pull request
Dec 30, 2021
Refactors interfaces and classes used by `libp2p-interfaces` to use the async peer store from libp2p/js-libp2p#1058 BREAKING CHANGE: peerstore methods are now all async
achingbrain
added a commit
to libp2p/js-libp2p-floodsub
that referenced
this pull request
Dec 30, 2021
Refactors interfaces and classes used by `libp2p-interfaces` to use the async peer store from libp2p/js-libp2p#1058 BREAKING CHANGE: peerstore methods are now all async
achingbrain
added a commit
to libp2p/js-libp2p-floodsub
that referenced
this pull request
Dec 31, 2021
Refactors interfaces and classes used by `libp2p-interfaces` to use the async peer store from libp2p/js-libp2p#1058 BREAKING CHANGE: peerstore methods are now all async
achingbrain
added a commit
to achingbrain/gossipsub-js
that referenced
this pull request
Dec 31, 2021
Refactors interfaces and classes used by `libp2p-interfaces` to use the async peer store from libp2p/js-libp2p#1058 Fixes a memory leak where peer data (multiaddrs, protocols, etc) is never evicted from memory. BREAKING CHANGE: peerstore methods and pubsub start/stop are now all async
achingbrain
added a commit
to achingbrain/gossipsub-js
that referenced
this pull request
Dec 31, 2021
Refactors interfaces and classes used by `libp2p-interfaces` to use the async peer store from libp2p/js-libp2p#1058 Fixes a memory leak where peer data (multiaddrs, protocols, etc) is never evicted from memory. BREAKING CHANGE: peerstore methods and pubsub start/stop are now all async
achingbrain
added a commit
to achingbrain/gossipsub-js
that referenced
this pull request
Jan 15, 2022
Refactors interfaces and classes used by `libp2p-interfaces` to use the async peer store from libp2p/js-libp2p#1058 Fixes a memory leak where peer data (multiaddrs, protocols, etc) is never evicted from memory. BREAKING CHANGE: peerstore methods and pubsub start/stop are now all async
achingbrain
added a commit
to libp2p/js-libp2p-daemon
that referenced
this pull request
Jan 17, 2022
Refactors interfaces and classes used by `libp2p-interfaces` to use the async peer store from libp2p/js-libp2p#1058 BREAKING CHANGE: peerstore methods are now all async
achingbrain
added a commit
to ipfs/js-ipfs-repo
that referenced
this pull request
Jan 18, 2022
Adds repo migration to convert multiple datastore entries into the single entry that will be used by the next version of libp2p. Refs: libp2p/js-libp2p#1058 BREAKING CHANGE: updates repo version and migrates repo
achingbrain
added a commit
to ipfs/js-ipfs
that referenced
this pull request
Jan 18, 2022
Refactors interfaces and classes used by `libp2p-interfaces` to use the async peer store from libp2p/js-libp2p#1058 Fixes memory leak in libp2p, discussion in linked issue above. BREAKING CHANGE: peerstore methods are now all async
achingbrain
force-pushed
the
feat/async-peerstore
branch
from
33aac86 to
9f9a527
Compare
Merged
achingbrain
added a commit
that referenced
this pull request
Jan 21, 2022
achingbrain
pushed a commit
that referenced
this pull request
Jan 25, 2022
## [0.36.0](https://github.com/libp2p/js-libp2p/compare/v0.35.8...v0.36.0) (2022-01-25) ### ⚠ BREAKING CHANGES * abort-controller dep is gone from dependency tree * `libp2p.handle`, `libp2p.registrar.register` and the peerstore methods have become async ### Features * add fetch protocol ([#1036](https://github.com/libp2p/js-libp2p/issues/1036)) ([d8ceb0b](https://github.com/libp2p/js-libp2p/commit/d8ceb0bc66fe225d1335d3f05b9a3a30983c2a57)) * async peerstore backed by datastores ([#1058](https://github.com/libp2p/js-libp2p/issues/1058)) ([978eb36](https://github.com/libp2p/js-libp2p/commit/978eb3676fad5d5d50ddb28d1a7868f448cbb20b)) * connection gater ([#1142](https://github.com/libp2p/js-libp2p/issues/1142)) ([ff32eba](https://github.com/libp2p/js-libp2p/commit/ff32eba6a0fa222af1a7a46775d5e0346ad6ebdf)) ### Bug Fixes * cache build artefacts ([#1091](https://github.com/libp2p/js-libp2p/issues/1091)) ([5043cd5](https://github.com/libp2p/js-libp2p/commit/5043cd56435a264e83db4fb8388d33e9a0442fff)) * catch errors during identify ([#1138](https://github.com/libp2p/js-libp2p/issues/1138)) ([12f1bb0](https://github.com/libp2p/js-libp2p/commit/12f1bb0aeec4b639bd2af05807215f3b4284e379)) * import uint8arrays package in example ([#1083](https://github.com/libp2p/js-libp2p/issues/1083)) ([c3700f5](https://github.com/libp2p/js-libp2p/commit/c3700f55d5a0b62182d683ca37258887b24065b9)) * make tests more reliable ([#1139](https://github.com/libp2p/js-libp2p/issues/1139)) ([b7e8706](https://github.com/libp2p/js-libp2p/commit/b7e87066a69970f1adca4ba552c7fdf624916a7e)) * prevent auto-dialer from dialing self ([#1104](https://github.com/libp2p/js-libp2p/issues/1104)) ([9b22c6e](https://github.com/libp2p/js-libp2p/commit/9b22c6e2f987a20c6639cd07f31fe9c824e24923)) * remove abort-controller dep ([#1095](https://github.com/libp2p/js-libp2p/issues/1095)) ([0a4dc54](https://github.com/libp2p/js-libp2p/commit/0a4dc54d084c901df47cce1788bd5922090ee037)) * try all peer addresses when dialing a relay ([#1140](https://github.com/libp2p/js-libp2p/issues/1140)) ([63aa480](https://github.com/libp2p/js-libp2p/commit/63aa480800974515f44d3b7e013da9c8ccaae8ad)) * update any-signal and timeout-abort-controller ([#1128](https://github.com/libp2p/js-libp2p/issues/1128)) ([e0354b4](https://github.com/libp2p/js-libp2p/commit/e0354b4c6b95bb90656b868849182eb3efddf096)) * update multistream select ([#1136](https://github.com/libp2p/js-libp2p/issues/1136)) ([00e4959](https://github.com/libp2p/js-libp2p/commit/00e49592a356e39b20c889d5f40b9bb37d4bf293)) * update node-forge ([#1133](https://github.com/libp2p/js-libp2p/issues/1133)) ([a4bba35](https://github.com/libp2p/js-libp2p/commit/a4bba35948e1cd8dbe5147f2c8d6385b1fbb6fae)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
achingbrain
added a commit
to ipfs/js-ipfs
that referenced
this pull request
Jan 27, 2022
Refactors interfaces and classes used by `libp2p-interfaces` to use the async peer store from libp2p/js-libp2p#1058 Fixes memory leak in libp2p, discussion in linked issue above. BREAKING CHANGE: peerstore methods are now all async, the repo is migrated to v12
SgtPooki
pushed a commit
to ipfs/js-kubo-rpc-client
that referenced
this pull request
Aug 18, 2022
Refactors interfaces and classes used by `libp2p-interfaces` to use the async peer store from libp2p/js-libp2p#1058 Fixes memory leak in libp2p, discussion in linked issue above. BREAKING CHANGE: peerstore methods are now all async, the repo is migrated to v12
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We have a peerstore that keeps all data for all observed peers in
memory with no eviction.
This is fine when you don't discover many peers but when using the
DHT you encounter a significant number of peers so our peer storage
grows and grows over time.
We have a persistent peer store, but it just periodically writes
peers into the datastore to be read at startup, still keeping them
in memory.
It also means a restart doesn't give you any temporary reprieve from
the memory leak as the previously observed peer data is read into
memory at startup.
This change refactors the peerstore to use a datastore by default,
reading and writing peer info as it arrives. It can be configured
with a MemoryDatastore if desired.
It was necessary to change the peerstore and *book interfaces to
be asynchronous since the datastore api is asynchronous.
Depends on: