Optimized v1 to v2 DB upgrade

[AmineKhaldi]

Huge thanks to @neurosis69 for helping me test on his setups, especially the RPi4 one which allowed me to address the fact that our existing upgrade tool never finishes.

[arvidn]

it appears to work on a Linux, AMD. It's still pretty slow though:

there is very little free space on the volume where the output database will be written:
   /home/arvid/blockchain_v2_mainnet.sqlite
free space: 307.59 GiB expected about 309.59 GiB
-- Opening file for reading: /home/arvid/blockchain_v1_mainnet.sqlite
-- Opening file for writing: /home/arvid/blockchain_v2_mainnet.sqlite
-- Initializing v2 version
-- Initializing current_peak
-- Peak: 85afa3e553d069236d2807f419641a1720777d2a6fd6279039ab44e5ad545b94 Height: 5040549
-- DB v1 to v2 conversion started
-- [1/4] Converting full_blocks
-- [1/4] Converting full_blocks SUCCEEDED in 1760.91 seconds
-- [1/4] Creating full_blocks height index
-- [1/4] Creating full_blocks height index SUCCEEDED in 87.67 seconds
-- [1/4] Creating full_blocks is_fully_compactified index
-- [1/4] Creating full_blocks is_fully_compactified index SUCCEEDED in 19.68 seconds
-- [1/4] Creating full_blocks main_chain index
-- [1/4] Creating full_blocks main_chain index SUCCEEDED in 19.21 seconds
-- [2/4] Converting sub_epoch_segments_v3
-- [2/4] Converting sub_epoch_segments_v3 SUCCEEDED in 51.54 seconds
-- [3/4] Converting hints
-- [3/4] Converting hints SUCCEEDED in 784.96 seconds
-- [3/4] Creating hints hint_index index
-- [3/4] Creating hints hint_index index SUCCEEDED in 30.89 seconds
-- [4/4] Converting coin_record
-- [4/4] Converting coin_record SUCCEEDED in 5595.02 seconds
-- [4/4] Creating coin_record coin_confirmed_index index
-- [4/4] Creating coin_record coin_confirmed_index index SUCCEEDED in 355.17 seconds
-- [4/4] Creating coin_record coin_spent_index index
-- [4/4] Creating coin_record coin_spent_index index SUCCEEDED in 273.84 seconds
-- [4/4] Creating coin_record coin_puzzle_hash index
-- [4/4] Creating coin_record coin_puzzle_hash index SUCCEEDED in 586.16 seconds                             
-- [4/4] Creating coin_record coin_parent_index index
-- [4/4] Creating coin_record coin_parent_index index SUCCEEDED in 617.20 seconds                             


LEAVING PREVIOUS DB FILE UNTOUCHED /home/arvid/blockchain_v1_mainnet.sqlite

        Command being timed: "chia db upgrade --input /home/arvid/blockchain_v1_mainnet.sqlite --output /home/arvid/blockchain_v2_mainnet.sqlite"
        User time (seconds): 6424.17
        System time (seconds): 2723.58
        Percent of CPU this job got: 89%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 2:49:46
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 1118976
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 3
        Minor (reclaiming a frame) page faults: 4281147
        Voluntary context switches: 12717802
        Involuntary context switches: 99105
        Swaps: 0
        File system inputs: 1029512296
        File system outputs: 735527032
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

It seems our prediction of the target DB size is a little bit off (by 2x):

$ ls -lah ~/blockchain_v2_mainnet.sqlite 
-rw-r--r-- 1 arvid arvid 154G mar 29 00:42 /home/arvid/blockchain_v2_mainnet.sqlite

[arvidn]

diff coverage report:

chia/cmds/db_upgrade_func.py (3.7%): Missing lines 54,118,120,122,124,126-127,129,131-132,135-168,182-196,203-210,227-233,238-242,246-251,255-257,264-266,270-272,279-281,285-286,294-296,305-307,311-320,323-326,337-346,350-355,359-360,374-380,398-412,416-421,425-430,434-439,443-448,452

[arvidn]

Comparing this to main.

version	timing	resident size
main	4:32:01 (16320s) , 100%	2239048 kiB, 100%
This PR	2:49:46 (10186s), 62.4%	1118976 kiB, 50%

there is very little free space on the volume where the output database will be written:
   /home/arvid/blockchain_v2_mainnet.sqlite
free space: 305.64 GiB expected about 309.59 GiB
opening file for reading: /home/arvid/blockchain_v1_mainnet.sqlite
opening file for writing: /home/arvid/blockchain_v2_mainnet.sqlite
initializing v2 version
initializing v2 block store
peak: 85afa3e553d069236d2807f419641a1720777d2a6fd6279039ab44e5ad545b94 height: 5040549
[1/5] converting full_blocks
      1618.47 seconds
[2/5] converting sub_epoch_segments_v3
      51.57 seconds
[3/5] converting hint_store
      596.79 seconds
[4/5] converting coin_store
      12020.08 seconds                             
[5/5] build indices
      block store
      coin store
      hint store
      2033.23 seconds                             


LEAVING PREVIOUS DB FILE UNTOUCHED /home/arvid/blockchain_v1_mainnet.sqlite

        Command being timed: "chia db upgrade --input /home/arvid/blockchain_v1_mainnet.sqlite --output /home/arvid/blockchain_v2_mainnet.sqlite"
        User time (seconds): 6498.67
        System time (seconds): 8077.26
        Percent of CPU this job got: 89%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 4:32:01
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 2239048
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 3004808
        Voluntary context switches: 19096210
        Involuntary context switches: 79769
        Swaps: 0
        File system inputs: 1039312368
        File system outputs: 2842889992
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0