From 4ebe7e633d64fb7ca8d77716a10a6593f482646b Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 27 Apr 2020 15:12:04 -0700 Subject: [PATCH 1/4] doc: fixup config docs 1. Update go-ipfs-config to fix default datastore. 2. Update TOC. 3. Update profiles list and indicate that badger isn't experimental. --- docs/config.md | 86 +++++++++++++++++++++++++++++++++++--------------- go.mod | 2 +- go.sum | 4 +-- 3 files changed, 64 insertions(+), 28 deletions(-) diff --git a/docs/config.md b/docs/config.md index a01e3a274b4..73a8ab8576d 100644 --- a/docs/config.md +++ b/docs/config.md @@ -12,50 +12,82 @@ applied with `--profile` flag to `ipfs init` or with the `ipfs config profile apply` command. When a profile is applied a backup of the configuration file will be created in `$IPFS_PATH`. -Available profiles: +The available configuration profiles are listed below. You can also find them +documented in `ipfs config profile --help`. - `server` - Recommended for nodes with public IPv4 address (servers, VPSes, etc.), - disables host and content discovery in local networks. + Disables local host discovery, recommended when + running IPFS on machines with public IPv4 addresses. + +- `randomports` + + Use a random port number for swarm. + +- `default-datatore` + + Configures the node to use the default datastore (flatfs). + + Read the "flatfs" profile description for more information on this datastore. + + This profile may only be applied when first initializing the node. - `local-discovery` - Sets default values to fields affected by `server` profile, enables - discovery in local networks. + Sets default values to fields affected by the server + profile, enables discovery in local networks. - `test` - Reduces external interference, useful for running ipfs in test environments. - Note that with these settings node won't be able to talk to the rest of the - network without manual bootstrap. + Reduces external interference of IPFS daemon, this + is useful when using the daemon in test environments. - `default-networking` - Restores default network settings. Inverse profile of the `test` profile. + Restores default network settings. + Inverse profile of the test profile. -- `badgerds` +- `flatfs` - Replaces default datastore configuration with experimental badger datastore. - If you apply this profile after `ipfs init`, you will need to convert your - datastore to the new configuration. You can do this using - [ipfs-ds-convert](https://github.com/ipfs/ipfs-ds-convert) + Configures the node to use the flatfs datastore. - WARNING: badger datastore is experimental. Make sure to backup your data - frequently. + This is the most battle-tested and reliable datastore, but it's significantly + slower than the badger datastore. You should use this datastore if: -- `default-datastore` + - You need a very simple and very reliable datastore you and trust your + filesystem. This datastore stores each block as a separate file in the + underlying filesystem so it's unlikely to loose data unless there's an issue + with the underlying file system. + - You need to run garbage collection on a small (<= 10GiB) datastore. The + default datastore, badger, can leave several gigabytes of data behind when + garbage collecting. + - You're concerned about memory usage. In its default configuration, badger can + use up to several gigabytes of memory. - Restores default datastore configuration. + This profile may only be applied when first initializing the node. -- `lowpower` - Reduces daemon overhead on the system. May affect node functionality, - performance of content discovery and data fetching may be degraded. +- `badgerds` + + Configures the node to use the badger datastore. -- `randomports` + This is the fastest datastore. Use this datastore if performance, especially + when adding many gigabytes of files, is critical. However: + + - This datastore will not properly reclaim space when your datastore is + smaller than several gigabytes. If you run IPFS with '--enable-gc' (you have + enabled block-level garbage collection), you plan on storing very little data in + your IPFS node, and disk usage is more critical than performance, consider using + flatfs. + - This datastore uses up to several gigabytes of memory. + + This profile may only be applied when first initializing the node. + +- `lowpower` - Generate random port for swarm. + Reduces daemon overhead on the system. May affect node + functionality - performance of content discovery and data + fetching may be degraded. ## Table of Contents @@ -67,6 +99,12 @@ Available profiles: - [`Addresses.NoAnnounce`](#addressesnoannounce) - [`API`](#api) - [`API.HTTPHeaders`](#apihttpheaders) +- [`AutoNAT`](#autonat) + - [`AutoNAT.ServiceMode`](#autonatservicemode) + - [`AutoNAT.Throttle`](#autonatthrottle) + - [`AutoNAT.Throttle.GlobalLimit`](#autonatthrottlegloballimit) + - [`AutoNAT.Throttle.PeerLimit`](#autonatthrottlepeerlimit) + - [`AutoNAT.Throttle.Interval`](#autonatthrottleinterval) - [`Bootstrap`](#bootstrap) - [`Datastore`](#datastore) - [`Datastore.StorageMax`](#datastorestoragemax) @@ -89,7 +127,6 @@ Available profiles: - [`Gateway.Writable`](#gatewaywritable) - [`Gateway.PathPrefixes`](#gatewaypathprefixes) - [`Gateway.PublicGateways`](#gatewaypublicgateways) - - [`Gateway` recipes](#gateway-recipes) - [`Identity`](#identity) - [`Identity.PeerID`](#identitypeerid) - [`Identity.PrivKey`](#identityprivkey) @@ -111,7 +148,6 @@ Available profiles: - [`Swarm.DisableRelay`](#swarmdisablerelay) - [`Swarm.EnableRelayHop`](#swarmenablerelayhop) - [`Swarm.EnableAutoRelay`](#swarmenableautorelay) - - [`Swarm.EnableAutoNATService`](#swarmenableautonatservice) - [`Swarm.ConnMgr`](#swarmconnmgr) - [`Swarm.ConnMgr.Type`](#swarmconnmgrtype) - [`Swarm.ConnMgr.LowWater`](#swarmconnmgrlowwater) diff --git a/go.mod b/go.mod index 10cc8c347a0..a56897590aa 100644 --- a/go.mod +++ b/go.mod @@ -31,7 +31,7 @@ require ( github.com/ipfs/go-ipfs-blockstore v0.1.4 github.com/ipfs/go-ipfs-chunker v0.0.5 github.com/ipfs/go-ipfs-cmds v0.2.2 - github.com/ipfs/go-ipfs-config v0.5.2 + github.com/ipfs/go-ipfs-config v0.5.3 github.com/ipfs/go-ipfs-ds-help v0.1.1 github.com/ipfs/go-ipfs-exchange-interface v0.0.1 github.com/ipfs/go-ipfs-exchange-offline v0.0.1 diff --git a/go.sum b/go.sum index 6a50ffbe97f..ea887bf4034 100644 --- a/go.sum +++ b/go.sum @@ -291,8 +291,8 @@ github.com/ipfs/go-ipfs-chunker v0.0.5 h1:ojCf7HV/m+uS2vhUGWcogIIxiO5ubl5O57Q7Na github.com/ipfs/go-ipfs-chunker v0.0.5/go.mod h1:jhgdF8vxRHycr00k13FM8Y0E+6BoalYeobXmUyTreP8= github.com/ipfs/go-ipfs-cmds v0.2.2 h1:F2pro/Q3ifRUsdxEKIS8cg8lO4R6WiwAyERiaG8I9no= github.com/ipfs/go-ipfs-cmds v0.2.2/go.mod h1:kqlUrp6m2ceoaJe40cXpADCi5aS6NKRn0NIeuLp5CeM= -github.com/ipfs/go-ipfs-config v0.5.2 h1:SPWiMNo7IOW0k+meO3PIprggp/PbZGUiO57L7HQ/sOY= -github.com/ipfs/go-ipfs-config v0.5.2/go.mod h1:nSLCFtlaL+2rbl3F+9D4gQZQbT1LjRKx7TJg/IHz6oM= +github.com/ipfs/go-ipfs-config v0.5.3 h1:3GpI/xR9FoJNTjU6YvCMRbYyEi0dBVY5UtlUTcNRlSA= +github.com/ipfs/go-ipfs-config v0.5.3/go.mod h1:nSLCFtlaL+2rbl3F+9D4gQZQbT1LjRKx7TJg/IHz6oM= github.com/ipfs/go-ipfs-delay v0.0.0-20181109222059-70721b86a9a8/go.mod h1:8SP1YXK1M1kXuc4KJZINY3TQQ03J2rwBG9QfXmbRPrw= github.com/ipfs/go-ipfs-delay v0.0.1 h1:r/UXYyRcddO6thwOnhiznIAiSvxMECGgtv35Xs1IeRQ= github.com/ipfs/go-ipfs-delay v0.0.1/go.mod h1:8SP1YXK1M1kXuc4KJZINY3TQQ03J2rwBG9QfXmbRPrw= From fa3f4505e9325e34fcdd3d0ab11d52dd44327403 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 27 Apr 2020 15:19:49 -0700 Subject: [PATCH 2/4] docs(datastore): fixup some notes on sync safety --- docs/datastores.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/docs/datastores.md b/docs/datastores.md index 0bab1a7e2a9..e7310022b85 100644 --- a/docs/datastores.md +++ b/docs/datastores.md @@ -4,6 +4,7 @@ This document describes the different possible values for the `Datastore.Spec` field in the ipfs configuration file. ## flatfs + Stores each key value pair as a file on the filesystem. The shardFunc is prefixed with `/repo/flatfs/shard/v1` then followed by a descriptor of the sharding strategy. Some example values are: @@ -21,8 +22,7 @@ The shardFunc is prefixed with `/repo/flatfs/shard/v1` then followed by a descri } ``` -NOTE: flatfs should only be used as a block store (mounted at `/blocks`) as the -current implementation is not complete. +NOTE: flatfs must only be used as a block store (mounted at `/blocks`) as it only partially implements the datastore interface. You can mount flatfs for /blocks only using the mount datastore (described below). ## levelds Uses a leveldb database to store key value pairs. @@ -36,10 +36,11 @@ Uses a leveldb database to store key value pairs. ``` ## badgerds + Uses [badger](https://github.com/dgraph-io/badger) as a key value store. -* `syncWrites`: Flush every write to disk before continuing. Disabling this option may leave your datastore in an inconsistent state after a crash. -* `truncate`: Truncate the DB if a partially written sector is found (defaults to true). This only happens if a IPFS crashes half-way through a write so this option is usually safe to leave on. +* `syncWrites`: Flush every write to disk before continuing. Setting this to false is safe as go-ipfs will automatically flush writes to disk before and after performing critical operations like pinning. However, you can set this to true to be extra-safe (at the cost of a 2-3x slowdown when adding files). +* `truncate`: Truncate the DB if a partially written sector is found (defaults to true). There is no good reason to set this to false unless you want to manually recover partially written (and unpinned) blocks if go-ipfs crashes half-way through a adding a file. ```json { @@ -51,6 +52,7 @@ Uses [badger](https://github.com/dgraph-io/badger) as a key value store. ``` ## mount + Allows specified datastores to handle keys prefixed with a given path. The mountpoints are added as keys within the child datastore definitions. @@ -71,6 +73,7 @@ The mountpoints are added as keys within the child datastore definitions. ``` ## measure + This datastore is a wrapper that adds metrics tracking to any datastore. ```json From 6d7d3ef30ecfb892d599d5325947089fe67bfeea Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 27 Apr 2020 16:00:48 -0700 Subject: [PATCH 3/4] doc: expand on relay documentation --- docs/config.md | 37 +++++++++++++++++++++++++++---------- 1 file changed, 27 insertions(+), 10 deletions(-) diff --git a/docs/config.md b/docs/config.md index 73a8ab8576d..5513acf092e 100644 --- a/docs/config.md +++ b/docs/config.md @@ -756,24 +756,41 @@ go-ipfs node accessible from the public internet. ### `Swarm.DisableRelay` -Disables the p2p-circuit relay transport. +Disables the p2p-circuit relay transport. This will prevent this node from +connecting to nodes behind relays, or accepting connections from nodes behind +relays. ### `Swarm.EnableRelayHop` -Enables HOP relay for the node. +Configures this node to act as a relay "hop". A relay "hop" relays traffic for other peers. -If this is enabled, the node will act as an intermediate (Hop Relay) node in -relay circuits for connected peers. +WARNING: Do not enable this option unless you know what you're doing. Other +peers will randomly decide to use your node as a relay and consume _all_ +available bandwidth. There is _no_ rate-limiting. ### `Swarm.EnableAutoRelay` -Enables automatic relay for this node. +Enables "automatic relay" mode for this node. This option does two _very_ +different things based on the `Swarm.EnableRelayHop`. See +[#7228](https://github.com/ipfs/go-ipfs/issues/7228) for context. -If the node is a HOP relay (`EnableRelayHop` is true) then it will advertise -itself as a relay through the DHT. Otherwise, the node will test its own NAT -situation (dialability) using passively discovered AutoNAT services. If the node -is not publicly reachable, then it will seek HOP relays advertised through the -DHT and override its public address(es) with relay addresses. +#### Mode 1: `EnableRelayHop` is `false` + +If `Swarm.EnableAutoRelay` is enabled and `Swarm.EnableRelayHop` is disabled, +your node will automatically _use_ public relays from the network if it detects +that it cannot be reached from the public internet (e.g., it's behind a +firewall). This is likely the feature you're looking for. + +If you enable `EnableAutoRelay`, you should almost certainly disable +`EnableRelayHop`. + +#### Mode 2: `EnableRelayHop` is `true` + +If `EnableAutoRelay` is enabled and `EnableRelayHop` is enabled, your node will +_act_ as a public relay for the network. Furthermore, in addition to simply +relaying traffic, your node will advertise itself as a public relay. Unless you +have the bandwidth of a small ISP, do not enable both of these options at the +same time. ### `Swarm.EnableAutoNATService` From 4c663bcf242b76ee090b3138c0ae94da1740d284 Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Mon, 27 Apr 2020 16:01:27 -0700 Subject: [PATCH 4/4] feat: update experimental documentations for 0.5.0 release Mostly: 1. Remove old experiments. 2. Update reasons for features being experimental. 3. Comment on experiments that will be enabled in the near future. --- docs/experimental-features.md | 214 +++++++--------------------------- 1 file changed, 41 insertions(+), 173 deletions(-) diff --git a/docs/experimental-features.md b/docs/experimental-features.md index 415a308bb99..18cbd993fa3 100644 --- a/docs/experimental-features.md +++ b/docs/experimental-features.md @@ -38,7 +38,7 @@ the above issue. ### State -experimental, default-disabled. +Candidate, disabled by default but will be enabled by default in 0.6.0. ### In Version @@ -79,75 +79,40 @@ signed) by running: - [ ] Needs authenticating modes to be implemented - [ ] needs performance analyses to be done ---- - -## Client mode DHT routing +## Raw Leaves for unixfs files -Allows the dht to be run in a mode that doesn't serve requests to the network, -saving bandwidth. +Allows files to be added with no formatting in the leaf nodes of the graph. ### State -stable -### In Version - -0.5.0 - -### How to enable - -run your daemon with the `--routing=dhtclient` flag. - ---- - -## go-multiplex stream muxer -Adds support for using the go-multiplex stream muxer alongside (or instead of) -yamux and spdy. This multiplexer is far simpler, and uses less memory and -bandwidth than the others, but is lacking on congestion control and backpressure -logic. It is available to try out and experiment with. - -### State -Stable +Stable but not used by default. ### In Version + 0.4.5 ### How to enable -To make it the default stream muxer, set the environment variable -`LIBP2P_MUX_PREFS` as follows: -``` -export LIBP2P_MUX_PREFS="/mplex/6.7.0 /yamux/1.0.0 /spdy/3.1.0" -``` - ---- - -## Raw Leaves for unixfs files -Allows files to be added with no formatting in the leaf nodes of the graph. - -### State -experimental. - -### In Version -master, 0.4.5 - -### How to enable -Use `--raw-leaves` flag when calling `ipfs add`. +Use `--raw-leaves` flag when calling `ipfs add`. This will save some space when adding files. ### Road to being a real feature -- [ ] Needs more people to use and report on how well it works. ---- +Enabling this feature _by default_ will change the CIDs (hashes) of all newly imported files and will prevent newly imported files from deduplicating against previously imported files. While we do intend on enabling this by default, we plan on doing so once we have a large batch of "hash-changing" features we can enable all at once. ## ipfs filestore + Allows files to be added without duplicating the space they take up on disk. ### State -experimental. + +Experimental. ### In Version -master, 0.4.7 + +0.4.7 ### How to enable + Modify your ipfs config: ``` ipfs config --json Experimental.FilestoreEnabled true @@ -159,23 +124,26 @@ Finally, when adding files with ipfs add, pass the --nocopy flag to use the filestore instead of copying the files into your local IPFS repo. ### Road to being a real feature + - [ ] Needs more people to use and report on how well it works. - [ ] Need to address error states and failure conditions - [ ] Need to write docs on usage, advantages, disadvantages - [ ] Need to merge utility commands to aid in maintenance and repair of filestore ---- - ## ipfs urlstore + Allows ipfs to retrieve blocks contents via a URL instead of storing it in the datastore ### State -experimental. + +Experimental. ### In Version -master, v0.4.17 + +v0.4.17 ### How to enable + Modify your ipfs config: ``` ipfs config --json Experimental.UrlstoreEnabled true @@ -190,19 +158,20 @@ And then add a file at a specific URL using `ipfs urlstore add ` - [ ] Need to implement caching - [ ] Need to add metrics to monitor performance ---- - ## Private Networks It allows ipfs to only connect to other peers who have a shared secret key. ### State -Experimental + +Stable but not quite ready for prime-time. ### In Version -master, 0.4.7 + +0.4.7 ### How to enable + Generate a pre-shared-key using [ipfs-swarm-key-gen](https://github.com/Kubuxu/go-ipfs-swarm-key-gen)): ``` go get github.com/Kubuxu/go-ipfs-swarm-key-gen/ipfs-swarm-key-gen @@ -240,10 +209,10 @@ variable to `1` to force the usage of private networks. If no private network is configured, the daemon will fail to start. ### Road to being a real feature -- [ ] Needs more people to use and report on how well it works -- [ ] More documentation ---- +- [x] Needs more people to use and report on how well it works +- [ ] More documentation +- [ ] Needs better tooling/UX. ## ipfs p2p @@ -253,11 +222,11 @@ similar. ### State -Experimental +Experimental, will be stabilized in 0.6.0 ### In Version -master, 0.4.10 +0.4.10 ### How to enable @@ -351,11 +320,8 @@ with `ssh [user]@127.0.0.1 -p 2222`. ### Road to being a real feature -- [ ] Needs more people to use and report on how well it works / fits use cases -- [ ] More documentation -- [ ] Support other protocols (e.g, Unix domain sockets, WebSockets, etc.) ---- +- [ ] More documentation ## p2p http proxy @@ -367,7 +333,7 @@ Experimental ### In Version -master, 0.4.19 +0.4.19 ### How to enable @@ -429,62 +395,15 @@ the remote machine (which needs to be a http server!) with path `$FORWARDED_PATH You should now see the resulting HTTP response: IPFS rocks! ### Custom protocol names + We also support the use of protocol names of the form /x/$NAME/http where $NAME doesn't contain any "/"'s ### Road to being a real feature + - [ ] Needs p2p streams to graduate from experiments - [ ] Needs more people to use and report on how well it works / fits use cases - [ ] More documentation - ---- - -## Circuit Relay - -Allows peers to connect through an intermediate relay node when there -is no direct connectivity. - -### State -Experimental - -### In Version -master, 0.4.11 - -### How to enable - -The relay transport is enabled by default, which allows peers to dial through -a relay and listens for incoming relay connections. The transport can be disabled -by setting `Swarm.DisableRelay = true` in the configuration. - -By default, peers don't act as intermediate nodes (relays). This can be enabled -by setting `Swarm.EnableRelayHop = true` in the configuration. Note that the -option needs to be set before online services are started to have an effect; an -already online node would have to be restarted. - -### Basic Usage: - -To connect peers QmA and QmB through a relay node QmRelay: - -- Both peers should connect to the relay: -`ipfs swarm connect /transport/address/p2p/QmRelay` -- Peer QmA can then connect to peer QmB using the relay: -`ipfs swarm connect /p2p/QmRelay/p2p-circuit/p2p/QmB` - -Peers can also connect with an unspecific relay address, which will -try to dial through known relays: -`ipfs swarm connect /p2p-circuit/p2p/QmB` - -Peers can see their (unspecific) relay address in the output of -`ipfs swarm addrs listen` - -### Road to being a real feature - -- [ ] Needs more people to use it and report on how well it works. -- [ ] Advertise relay addresses to the DHT for NATed or otherwise unreachable - peers. -- [ ] Active relay discovery for specific relay address advertisement. We would - like advertised relay addresses to designate specific relays for efficient dialing. -- [ ] Dialing priorities for relay addresses; arguably, relay addresses should - have lower priority than direct dials. +- [ ] Need better integration with the subdomain gateway feature. ## Plugins @@ -502,41 +421,11 @@ See [Plugin docs](./plugins.md) ### Road to being a real feature -- [ ] Better support for platforms other than Linux -- [ ] More plugins and plugin types +- [x] More plugins and plugin types +- [ ] A way to reliably build and distribute plugins. +- [ ] Better support for platforms other than Linux & MacOS - [ ] Feedback on stability ---- - -## Badger datastore - -### In Version - -0.4.11 - -Badger-ds is new datastore implementation based on -https://github.com/dgraph-io/badger. - - -### Basic Usage - -``` -$ ipfs init --profile=badgerds -``` -or install https://github.com/ipfs/ipfs-ds-convert/ and -``` -[BACKUP ~/.ipfs] -$ ipfs config profile apply badgerds -$ ipfs-ds-convert convert -``` - -You can read more in the [datastore](./datastores.md#badgerds) documentation. - -### Road to being a real feature - -- [ ] Needs more testing -- [ ] Make sure there are no unknown major problems - ## Directory Sharding / HAMT ### In Version @@ -562,8 +451,6 @@ ipfs config --json Experimental.ShardingEnabled true - [ ] Make sure that objects that don't have to be sharded aren't - [ ] Generalize sharding and define a new layer between IPLD and IPFS ---- - ## IPNS pubsub ### In Version @@ -595,8 +482,6 @@ run your daemon with the `--enable-namesys-pubsub` flag; enables pubsub. so that we don't have to hit the DHT for the initial resolution. Alternatively, we could republish the last record periodically. ---- - ## QUIC ### In Version @@ -605,7 +490,7 @@ run your daemon with the `--enable-namesys-pubsub` flag; enables pubsub. ### State -Experiment, disabled by default +Candidate, disabled by default but it will be enabled by default in 0.6.0. ### How to enable @@ -643,31 +528,16 @@ Automatically discovers relays and advertises relay addresses when the node is b Modify your ipfs config: ``` +ipfs config --json Swarm.EnableRelayHop false ipfs config --json Swarm.EnableAutoRelay true ``` -Bootstrappers (and other public nodes) need to also enable the AutoNATService: -``` -ipfs config --json Swarm.EnableAutoNATService true -``` +NOTE: Ensuring `Swarm.EnableRelayHop` is _false_ is extremely important here. If you set it to true, you will _act_ as a public relay for the rest of the network instead of _using_ the public relays. ### Road to being a real feature - [ ] needs testing - -## TLS 1.3 as default handshake protocol - -### In Version - -0.5.0 - -### State - -Stable - ---- - ## Strategic Providing ### State @@ -694,8 +564,6 @@ ipfs config --json Experimental.StrategicProviding true - [ ] provide all - [ ] provide strategic ---- - ## GraphSync ### State