Unable to restart lighthouse if altair fork is missed

[realbigsean]

Description

If you have lighthouse running v1.4 until after the altair fork occurs and then try to upgrade to v1.5, lighthouse won't start up. This is because we attempt to deserialize the head of the chain by first deserializing the slot of the head block, and comparing it to the fork schedule. But when we saved the block to the database, we used the pre-fork serialization (even though the slot of the block is post-fork).

Version

1.5.0-rc.1

Present Behaviour

Aug 19 18:24:29.427 INFO Starting beacon chain                   method: resume, service: beacon
Aug 19 18:24:44.083 CRIT Failed to start beacon node             reason: Failed to build beacon chain: DB error when reading head block: SszDecodeError(OffsetIntoFixedPortion(220)). Ensure the data directory is not initialized for a different network. The --purge-db flag can be used to permanently delete the existing data directory.
Aug 19 18:24:44.083 INFO Internal shutdown received              reason: Failed to start beacon node
Aug 19 18:24:44.083 INFO Shutting down..                         reason: Failure("Failed to start beacon node")

Expected Behaviour

I think we should automatically try to re-sync from before the fork in this scenario

Steps to resolve

Thinking we could do either or both of these:

• If we fail to deserialize the head of the beacon chain at startup, we could attempt to deserialize it based on the prior fork, and if that's successful it would suggest we're in this scenario. So we could start up lighthouse and automatically start re-syncing from right before the fork.
• We could add a flag --resync-from-fork altair to help recover from situations like this

[michaelsproul]

I'm going to have a go at this. It already looks like it might be a bit painful: all of our decoding methods assume they can determine the structure from the slot, so it might be messy to backtrack from the head of the minority chain to the pre-fork block while loading + deleting along the way.

[paulhauner]

I'll flag this as v1.5.0 for the time being, just so it stays on our radar before we release. If we decide to push it to a later release, that's totally fine 🙂

[paulhauner]

Resolved in #2529