feat: improve wallet recovery and scanning handling of reorgs

[philipr-za]

Description

@mikethetike Merging this into the weatherwax branch because I would like to see it tested on the Mobile apps sooner rather than later and not sure what the plan is for migrating them to the new Development

This PR updates the wallet UtxoScanner to better detect and handle reorgs. Previously only the last scanned block was check to see if it had been re-orged out and if it had then the whole chain was scanned from scratch. Reorgs of a few blocks are common so to reduce the amount of work done after a reorg this PR adds maintaining history of previously scanned block headers so that the depth of the reorg can be quickly determined and the rescan only be done for affected blocks.

This PR adds a number of updates:

• Add the sync_utxos_by_block method to the BaseNodeWalletService RPC service to facilitate a more appropriate UTXO sync strategy for the wallet
• Add the ScannedBlocks table to the wallet DB to keep a history of scanned block headers and the number and amount of outputs recovered from each block.
• Update the UTXO scanner to use this stored history to detect reorgs and only sync the required UTXOs
• Built out a test harness to test the scanner thoroughly in Rust integration tests.

How Has This Been Tested?

Added new Rust integration tests
Added back excluded Cucumber tests

[SWvheerden]

LGTM

[SWvheerden]

Suggested change

	} else if requested_epoch_time <= mid_header.timestamp.as_u64() {
	}
	if requested_epoch_time <= mid_header.timestamp.as_u64() {

[SWvheerden]

This can be made into a separate if. It does not have to be a single chained if-else statement.
We can even split it up into three if's but two is probably better.

[philipr-za]

Clippy begs to differ

[sdbondi]

Looks good. Good idea to add the call to the base node RPC service so that we connect to a single rpc service in the wallet. There were some bugs fixed in #3663 that were causing a problem with the indexes if include_pruned_utxos == false, or start index != a block boundary so index based syncing works fine after the fix (apologies, I had only tested with pruned sync and empty wallet which are not effected by the bugs). Though I like the simplification of the handler by using block boundary indexes in the wallet.

The batching of many utxos in a single message, with large blocks, is anecdotally worse (unless there is sensible evidence to the contrary, we must measure to be sure we understand this) because:

1. the longer it takes for a whole message to arrive (say because of its size) the longer the client side has to wait before beginning processing (a note on this, I suspect buffering can be easily optimised in the RPC client implementation for long streams so that any chunking on the client side is unnecessary/slower)
2. large responses are sent as chunk internally as necessary, so this diminishes any per-message RPC header byte saving (RPC header is almost always 5-10 bytes)

In this case, performance will be similar for small blocks and much worse for large blocks as observed in block sync. There is probably a happy middle ground for message size vs number of messages (block boundary indexes cannot be used). Though I would prefer the use of the same (bug free) code for both handlers, I think this can go in and we can make some observations.

[philipr-za]

The batching of many utxos in a single message, with large blocks, is anecdotally worse (unless there is sensible evidence to the contrary, we must measure to be sure we understand this) because:

Yes, I did consider this could be an issue. I went with the streaming by block boundary because it is a much simpler conceptual model but it wouldn't be hard to split a large block into multiple smaller paged messages. However, I decided it wasn't worth the effort at this stage because the pending on the filter byte will pretty much solve this potential issue I think. So I decided it will be something to look at once the filter byte has been implemented.