Regenesis support (#1693)

Closes #1209



The change separates the initial chain state from the chain config and
stores them in separate files when generating a snapshot. The state
snapshot can be generated in a new format where parquet is used for
compression and indexing while postcard is used for encoding. This
enables importing in a stream like fashion which reduces memory
requirements. Json encoding is still supported to enable easy manual
setup. However, parquet is prefered for large state files.

   ### Snapshot command

The CLI was expanded to allow customizing the used encoding. Snapshots
are now generated along with a metadata file describing the encoding
used. The metadata file contains encoding details as well as the
location of additional files inside the snapshot directory containing
the actual data. The chain config is always generated in the JSON
format.

The snapshot command now has the '--output-directory' for specifying
where to save the snapshot.

   ### Run command

The run command now includes the 'db_prune' flag which when provided
will prune the existing db and start genesis from the provided snapshot
metadata file or the local testnet configuration.

The snapshot metadata file contains paths to the chain config file and
files containing chain state items (coins, messages, contracts, contract
states, and balances), which are loaded via streaming.

Each item group in the genesis process is handled by a separate worker,
allowing for parallel loading. Workers stream file contents in batches.

A database transaction is committed every time an item group is
succesfully loaded. Resumability is achieved by recording the last
loaded group index within the same db tx. If loading is aborted, the
remaining workers are shutdown. Upon restart, workers resume from the
last processed group.

   ### Contract States and Balances

Using uniform-sized batches may result in batches containing items from
multiple contracts. Optimal performance can presumably be achieved by
selecting a batch size that typically encompasses an entire contract's
state or balance, allowing for immediate initialization of relevant
Merkle trees.

---------

Co-authored-by: Salka1988 <[email protected]>
Co-authored-by: Brandon Vrooman <[email protected]>
Co-authored-by: segfault_magnet <[email protected]>
Co-authored-by: Ahmed Sagdati <[email protected]>
Co-authored-by: Brandon Kite <[email protected]>
Co-authored-by: Mitchell Turner <[email protected]>
Co-authored-by: Green Baneling <[email protected]>

CHANGELOG.md

@@ -10,6 +10,33 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
 
 - [#1731](https://github.com/FuelLabs/fuel-core/pull/1731): Expose `schema.sdl` from `fuel-core-client`.
 
+### Changed
+
+#### Breaking
+
+- [#1693](https://github.com/FuelLabs/fuel-core/pull/1693): The change separates the initial chain state from the chain config and stores them in separate files when generating a snapshot. The state snapshot can be generated in a new format where parquet is used for compression and indexing while postcard is used for encoding. This enables importing in a stream like fashion which reduces memory requirements. Json encoding is still supported to enable easy manual setup. However, parquet is prefered for large state files.
+
+  ### Snapshot command
+
+  The CLI was expanded to allow customizing the used encoding. Snapshots are now generated along with a metadata file describing the encoding used. The metadata file contains encoding details as well as the location of additional files inside the snapshot directory containing the actual data. The chain config is always generated in the JSON format.
+
+  The snapshot command now has the '--output-directory' for specifying where to save the snapshot.
+
+  ### Run command
+
+  The run command now includes the 'db_prune' flag which when provided will prune the existing db and start genesis from the provided snapshot metadata file or the local testnet configuration.
+
+  The snapshot metadata file contains paths to the chain config file and files containing chain state items (coins, messages, contracts, contract states, and balances), which are loaded via streaming.
+
+  Each item group in the genesis process is handled by a separate worker, allowing for parallel loading. Workers stream file contents in batches.
+
+  A database transaction is committed every time an item group is succesfully loaded. Resumability is achieved by recording the last loaded group index within the same db tx. If loading is aborted, the remaining workers are shutdown. Upon restart, workers resume from the last processed group.
+
+  ### Contract States and Balances
+
+  Using uniform-sized batches may result in batches containing items from multiple contracts. Optimal performance can presumably be achieved by selecting a batch size that typically encompasses an entire contract's state or balance, allowing for immediate initialization of relevant Merkle trees.
+
+
 ## [Version 0.23.0]
 
 ### Added