etcd-dump-logs: Expand to allow diagnosing CRC corrupted problems in WAL log

[ptabor]

Before this PR diagnosing etcd not running due to crc wal log corruption landed with:

$ go build ./tools/etcd-dump-logs/... && ./etcd-dump-logs ../../Downloads/data
{"level":"warn","msg":"found unexpected non-snap file; skipping","path":"gke-c10ed2ca2a3e4fb8a45f-ffba-8587271_status.txt"}
{"level":"warn","msg":"found unexpected non-snap file; skipping","path":"gke-c10ed2ca2a3e4fb8a45f-ffba-7612155_status.txt"}
Snapshot:
term=32 index=38873895 nodes=[b71f75320dc06a6c] confstate={"voters":[13195394291058371180],"auto_leave":false}
Start dumping log entries from snapshot.
2022/12/23 16:25:28 Failed reading WAL: walpb: crc mismatch

So we just now CRC got corrupted ... somewhere...

With the PR:

$ go build ./tools/etcd-dump-logs/... && ./etcd-dump-logs --raw ../../Downloads/data > dump

2022/12/23 16:27:11 Failed reading WAL: walpb: crc mismatch: expected: 1d9658e9 computed: 37beb207: in file '0000000000000215-0000000002502c07.wal' at position: 37124224
p

In the dump file we see the exact record that got corrupted (index: 38856823):

{"level":"warn","msg":"Reading entry failed with CRC error","error":"walpb: crc mismatch: expected: 1d9658e9 computed: 37beb207: in file '0000000000000215-0000000002502c07.wal' at position: 37124224"}
Entry: Term:32 Index:38856823 Data:"..."
HardState: term:32 vote:13195394291058371180 commit:38856823

We see that there were snapshots post this corruption. Last one: Snapshot: index:38873895 term:32

Entry: Term:32 Index:38882697 Data:"..."

HardState: term:32 vote:13195394291058371180 commit:38882697

So there were 38882697-38856823= 25874 entries written after the corruption.

Ideally we should have a tool (based on the improved in this PR decoder) that allows to trim the WAL log file to contain just the last snapshot and all the follow up records (or just the entries that have 'index>=' consistent_index taken from bbolt.

[ahrtr]

Suggested change

	lg.Warn("Ignoring not .wal file", zap.String("filename", finfo.Name()))
	lg.Warn("Ignoring not .wal file", zap.String("filename", finfo.Name()))
	continue

[ahrtr]

This comment isn't resolved yet. Did you intentionally include files which do not suffixed with ".wal"?

[ptabor]

I'm sorry - missed this. I'm mimicking the logic of the ReadAll that do skips not .wal files.

[ahrtr]

I see that log, zap.Logger and fmt are used to output messages in this PR. Suggest to keep it consistent with the existing behavior: Use fmt to output normal messages and log to output error or warning messages.

[ptabor]

Fixed.

[ahrtr]

There are also some other places which need updated. Please search "lg." in main.go and raw.go.

[ptabor]

I thought log & lg is long term the same logging mechanism as we do in etcd.
Migrated to log for now.

[ahrtr]

LGTM

[ptabor]

Thank you for review.