Provide a way to truncate a corrupted Badger DB

[schomatis]

Badger needs explicit consent from the user to truncate a corrupted DB when starting: passing a --truncate flag. That flag is not being provided as an option right now and the user gets errors like:

# IPFS_PATH=/ipfs/ipfs_master/repo ./ipfs daemon
Initializing daemon...
Error: Unable to replay value log: "/ipfs/ipfs_master/repo/badgerds/000017.vlog": Data corruption detected. Value log truncate required to run DB. This would result in data loss.
Received interrupt signal, shutting down...
(Hit ctrl-c again to force-shutdown the daemon.)

(and #5213 (comment)), which are not really useful.

Documentation should also be added as it's a normal scenario to have a (relatively small) portion of the DB corrupted after a crash.

[Stebalien]

Will this small amount include "synced" data? If so, and if there's no way for us to actually sync the data, we should move to a different datastore. If not, we should just truncate by default.

Explicitly, I'd expect the following semantics:

1. If Datastore.Put has returned, the data must be persisted. Passing the truncate option shouldn't remove this data.
2. If Datastore.Put has not returned, we don't care. Passing the truncate option may remove this data.

[schomatis]

Will this small amount include "synced" data?

It shouldn't if syncWrites is set (which by default it is), as everything will be directly persisted to the value log on disk, and from that point it can be recovered after a crash. However, if that is a deal breaker, let me double check that and get back to you.

[Stebalien]

We need some commit point. Some point where we can know that the data has been persisted.

[schomatis]

Sorry for the delay here, completing @bigs notes from the Dev meeting, if syncWrites is set and I commit the transaction the following should happen:

The Commit will guarantee that all the entries will passed to the valueLog structure (flushing pendingWrites). The valueLog will write a buffer filled with those entries in a Go file opened with the O_DSYNC flag. In case a system error occurs after committing the transaction, when opening the DB the system will be able to replay at least up to the last entry of that transaction in the SSTables while iterating the value log.

(@manishrjain Feel free to ignore this but if you can would you mind validating that what I'm saying in this last paragraph is correct?)

[manishrjain]

Yes, that understanding is correct. If it was applied successfully to the value log, it would be replayed back after crash.

[Stebalien]

So, it looks like, given the semantics we expect, we should always enable truncation if SyncWrites is enabled.

PR: ipfs/go-ds-badger#30

[Stebalien]

@schomatis do you have time to fix this as we discussed? That is:

1. Add a truncate config option.
2. Turn it on by default.

[schomatis]

Yes, I'll try to get to it next week.

Turn it on by default.

Could you point me to the part of the code that generates the config file please?

[Stebalien]

You'll need to add something to the profile here: https://github.com/ipfs/go-ipfs-config/blob/235aea7b2426fa745412285d1d2a4a129ed564a5/profile.go#L123

Document it in: https://github.com/ipfs/go-ipfs/blob/master/docs/datastores.md#badgerds

And handle the configuration here: https://github.com/ipfs/go-ipfs/blob/4fb2666a2463bf0f17e21d3e02f176a5eb917745/repo/fsrepo/datastores.go#L341-L412

[Stebalien]

@djdv or @schomatis can we try to move forward on this?

[schomatis]

Yes, sorry, totally forgot about this one.