Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Badger2 integration #7705

Closed
wants to merge 2 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -430,6 +430,7 @@ Listing of the main packages used in the IPFS ecosystem. There are also three sp
| [`go-ds-measure`](//github.com/ipfs/go-ds-measure) | [![Travis CI](https://flat.badgen.net/travis/ipfs/go-ds-measure/master)](https://travis-ci.com/ipfs/go-ds-measure) | [![codecov](https://codecov.io/gh/ipfs/go-ds-measure/branch/master/graph/badge.svg?style=flat-square)](https://codecov.io/gh/ipfs/go-ds-measure) | a metric-collecting database adapter |
| [`go-ds-leveldb`](//github.com/ipfs/go-ds-leveldb) | [![Travis CI](https://flat.badgen.net/travis/ipfs/go-ds-leveldb/master)](https://travis-ci.com/ipfs/go-ds-leveldb) | [![codecov](https://codecov.io/gh/ipfs/go-ds-leveldb/branch/master/graph/badge.svg?style=flat-square)](https://codecov.io/gh/ipfs/go-ds-leveldb) | a leveldb based datastore |
| [`go-ds-badger`](//github.com/ipfs/go-ds-badger) | [![Travis CI](https://flat.badgen.net/travis/ipfs/go-ds-badger/master)](https://travis-ci.com/ipfs/go-ds-badger) | [![codecov](https://codecov.io/gh/ipfs/go-ds-badger/branch/master/graph/badge.svg?style=flat-square)](https://codecov.io/gh/ipfs/go-ds-badger) | a badgerdb based datastore |
| [`go-ds-badger2`](//github.com/ipfs/go-ds-badger2) | [![Travis CI](https://flat.badgen.net/travis/ipfs/go-ds-badger2/master)](https://travis-ci.com/ipfs/go-ds-badger2) | [![codecov](https://codecov.io/gh/ipfs/go-ds-badger2/branch/master/graph/badge.svg?style=flat-square)](https://codecov.io/gh/ipfs/go-ds-badger2) | a badger2db based datastore |
| **Namesys** |
| [`go-ipns`](//github.com/ipfs/go-ipns) | [![Travis CI](https://flat.badgen.net/travis/ipfs/go-ipns/master)](https://travis-ci.com/ipfs/go-ipns) | [![codecov](https://codecov.io/gh/ipfs/go-ipns/branch/master/graph/badge.svg?style=flat-square)](https://codecov.io/gh/ipfs/go-ipns) | IPNS datastructures and validation logic |
| **Repo** |
Expand Down
18 changes: 17 additions & 1 deletion docs/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ documented in `ipfs config profile --help`.
This is the most battle-tested and reliable datastore, but it's significantly
slower than the badger datastore. You should use this datastore if:

- You need a very simple and very reliable datastore you and trust your
- You need a very simple and very reliable datastore and you trust your
filesystem. This datastore stores each block as a separate file in the
underlying filesystem so it's unlikely to loose data unless there's an issue
with the underlying file system.
Expand All @@ -71,6 +71,22 @@ documented in `ipfs config profile --help`.

Configures the node to use the badger datastore.

This is a the fast datastore. Use this datastore if performance, especially
when adding many gigabytes of files, is critical. However:

- This datastore will not properly reclaim space when your datastore is
smaller than several gigabytes. If you run IPFS with '--enable-gc' (you have
enabled block-level garbage collection), you plan on storing very little data in
your IPFS node, and disk usage is more critical than performance, consider using
flatfs.
- This datastore uses up to several gigabytes of memory.

This profile may only be applied when first initializing the node.

- `badger2ds`

Configures the node to use the badger2 datastore.

This is the fastest datastore. Use this datastore if performance, especially
when adding many gigabytes of files, is critical. However:

Expand Down
24 changes: 24 additions & 0 deletions docs/datastores.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,13 +41,37 @@ Uses [badger](https://github.com/dgraph-io/badger) as a key value store.

* `syncWrites`: Flush every write to disk before continuing. Setting this to false is safe as go-ipfs will automatically flush writes to disk before and after performing critical operations like pinning. However, you can set this to true to be extra-safe (at the cost of a 2-3x slowdown when adding files).
* `truncate`: Truncate the DB if a partially written sector is found (defaults to true). There is no good reason to set this to false unless you want to manually recover partially written (and unpinned) blocks if go-ipfs crashes half-way through a adding a file.
* `vlogFileSize`: Sets the maximum size of a single value log file to the size specified, or to the default if unspecified.

```json
{
"type": "badgerds",
"path": "<location of badger inside repo>",
"syncWrites": true|false,
"truncate": true|false,
"vlogFileSize": <max size of a value log>
}
```

## badger2ds

Uses [badger2](https://github.com/dgraph-io/badger) as a key value store.

* `syncWrites`: Flush every write to disk before continuing. Setting this to false is safe as go-ipfs will automatically flush writes to disk before and after performing critical operations like pinning. However, you can set this to true to be extra-safe (at the cost of a 2-3x slowdown when adding files).
* `truncate`: Truncate the DB if a partially written sector is found (defaults to true). There is no good reason to set this to false unless you want to manually recover partially written (and unpinned) blocks if go-ipfs crashes half-way through a adding a file.
* `compression`: Configure compression. When compression is enabled, every block is compressed using the specified algorithm. This option doesn't affect existing tables. Only the newly created tables are compressed. Compression can be configured to use the snappy algorithm or ZSTD with level 1, 2 or 3. An unspecified value uses the default configuration.
* `blockCacheSize`: Specifies how much data cache should hold in memory. If compression is disabled, adding a cache leads to unnecessary overhead which may affect read performance. A value of 0 means no block cache, and an unspecified value uses the default configuration.
* `vlogFileSize`: Sets the maximum size of a single value log file to the size specified, or to the default if unspecified.

```json
{
aschmahmann marked this conversation as resolved.
Show resolved Hide resolved
"type": "badger2ds",
"path": "<location of badger2 inside repo>",
"syncWrites": true|false,
"truncate": true|false,
gammazero marked this conversation as resolved.
Show resolved Hide resolved
"compression": "none"|"snappy"|"zstd1"|"zstd2"|"zstd3",
"blockCacheSize": <size of block cache>,
"vlogFileSize": <max size of a value log>
}
```

Expand Down
14 changes: 8 additions & 6 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ require (
github.com/bren2010/proquint v0.0.0-20160323162903-38337c27106d
github.com/cheggaaa/pb v1.0.29
github.com/coreos/go-systemd/v22 v22.1.0
github.com/dgraph-io/badger/v2 v2.0.1-rc1.0.20201002071632-68fb85d55d4b
github.com/dustin/go-humanize v1.0.0
github.com/elgris/jsondiff v0.0.0-20160530203242-765b5c24c302
github.com/fsnotify/fsnotify v1.4.9
Expand All @@ -21,7 +22,8 @@ require (
github.com/ipfs/go-cidutil v0.0.2
github.com/ipfs/go-datastore v0.4.5
github.com/ipfs/go-detect-race v0.0.1
github.com/ipfs/go-ds-badger v0.2.5
github.com/ipfs/go-ds-badger v0.2.6
github.com/ipfs/go-ds-badger2 v0.1.1-0.20201009224942-a4499bb2f243
github.com/ipfs/go-ds-flatfs v0.4.5
github.com/ipfs/go-ds-leveldb v0.4.2
github.com/ipfs/go-ds-measure v0.1.0
Expand All @@ -31,7 +33,7 @@ require (
github.com/ipfs/go-ipfs-blockstore v0.1.4
github.com/ipfs/go-ipfs-chunker v0.0.5
github.com/ipfs/go-ipfs-cmds v0.4.0
github.com/ipfs/go-ipfs-config v0.9.0
github.com/ipfs/go-ipfs-config v0.10.0
github.com/ipfs/go-ipfs-ds-help v0.1.1
github.com/ipfs/go-ipfs-exchange-interface v0.0.1
github.com/ipfs/go-ipfs-exchange-offline v0.0.1
Expand Down Expand Up @@ -101,11 +103,11 @@ require (
github.com/whyrusleeping/multiaddr-filter v0.0.0-20160516205228-e903e4adabd7
github.com/whyrusleeping/tar-utils v0.0.0-20180509141711-8c6c8ba81d5c
go.uber.org/fx v1.13.1
go.uber.org/zap v1.15.0
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9
go.uber.org/zap v1.16.0
golang.org/x/crypto v0.0.0-20201002170205-7f63de1d35b0
golang.org/x/lint v0.0.0-20200302205851-738671d3881b // indirect
golang.org/x/sys v0.0.0-20200615200032-f1bc736245b1
golang.org/x/sys v0.0.0-20201009025420-dfb3f7c4e634
golang.org/x/tools v0.0.0-20200522201501-cb1345f3a375 // indirect
)

go 1.13
go 1.14
Loading