fix(en): gracefully shutdown en waiting for reorg detector

[montekki]

This PR changes the shutdown order of the external node to a more graceful one.

Why?

Currently sutting down the EN because of reorgs and inconsistency checks may be raceful and a shutting down consistency checker may bring down the reorg detector with it which would prevent the reorg detector from doing it's job in the legitimate cases of reorgs where the node actually needs to be rolled back to some previous state.

What happens currently?

The concerns have been focused around the logic of consistency checker that used to panic when an inconsistency was found. That is no longer the case (anyhow::bail! used to be panic):

zksync-era/core/lib/zksync_core/src/consistency_checker/mod.rs

Lines 181 to 183 in feb8a6c

	Ok(false) => {
	anyhow::bail!("Batch {} is inconsistent with L1", batch_number.0);
	}

however, this error would still unconditionally bring the node down because currently the handler for ConsistencyChecker is a part of wait_tasks that would resolve if any of the tasks has resolved:

zksync-era/core/bin/external_node/src/main.rs

Lines 401 to 404 in feb8a6c

	tokio::select! {
	_ = wait_for_tasks(task_handles, particular_crypto_alerts, graceful_shutdown, tasks_allowed_to_finish) => {},
	_ = sigint_receiver => {
	tracing::info!("Stop signal received, shutting down");

The proposed logic works as follows:

The handle to reorg detector is fused and on the node shutdown, after all the components are gracefully shutdown we manually check the result of the reorg detector. This logic handles both situations:

1. node is shutting down
2. reorg detector has found a reorg and is performing a rollback and shutdown.

[codecov]

Codecov Report

Attention: 3 lines in your changes are missing coverage. Please review.

Comparison is base (ec41692) 35.61% compared to head (a50031f) 35.62%.
Report is 9 commits behind head on main.

❗ Current head a50031f differs from pull request most recent head f8d670d. Consider uploading reports for the commit f8d670d to get more accurate results

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #270   +/-   ##
=======================================
  Coverage   35.61%   35.62%           
=======================================
  Files         520      520           
  Lines       28352    28348    -4     
=======================================
  Hits        10098    10098           
+ Misses      18254    18250    -4     

Files	Coverage Δ
core/bin/external_node/src/config/mod.rs	21.42% <ø> (ø)
core/bin/external_node/src/main.rs	0.00% <ø> (ø)
core/lib/zksync_core/src/reorg_detector/mod.rs	46.15% <0.00%> (ø)

... and 8 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[RomanBrodetski]

Taking a step back - why do we need such an (arguably) complicated logic? It's partly because we need to run the reorg detector in parallel. We have a similar case with Circuit Breaker on the main node. Is there any potential to extract some abstraction?

[popzxc]

@montekki I think it's a good solution for now. A proper task management entity is in the plans too.