Track v0.17.0 changes

[doitian]

Features

• feat: Remove rules of special reserve blocks #1119: Remove rules of special reserve blocks (@doitian)

  This is a breaking change: b:consensus, b:database

  For block 1~11, the reward target is genesis block. Yes the reward for genesis is issued multiple times.

  Genesis block must have the lock serialized in the cellbase witness, which is set to bootstrap_lock.

• feat: Add get_header and get_header_by_number RPC methods #1125: Add get_header and get_header_by_number RPC methods (@TheWaWaR)

  Add two RPC methods

  • get_header
  • get_header_by_number

  Usage scenario:

  • ckb-cli get genesis block header to distinguish testnet and mainnet
  • SPV client just get headers to verify a transaction

• feat: Allow TransactionScriptsVerifier set custom debugger printer #1124: Allow TransactionScriptsVerifier set custom debugger printer (@TheWaWaR)

  Current Debugger only log the output and can not detected by outside.

  Here we pass a callback to process debug output.

• feat: Secondary miner issurance, split DAO as a separate contract #1094: Secondary miner issurance, split DAO as a separate contract (@xxuejie)

  This is a breaking change: b:consensus, b:database

  TODO:

  • tests

  This PR is still in WIP progress, it's lacking fixes for the tests so CI is doomed to fail. however it serves 2 purposes:

  • It provides a base so we can start writing tests for the system script PR: feat: Add NervosDAO as a CKB VM script ckb-system-scripts#21
  • Since it's a relatively large PR, publishing it earlier means we can get review process started early as well.

  Now all the tests have been fixed.

• feat: remove output for bootstrap lock in genesis block #1137: Remove output for bootstrap lock in genesis block (@doitian)

  The lock has already been written into the cellbase witness in the genesis block.

• feat: --ba-data requires --ba-arg or --ba-code-hash #1138: --ba-data requires --ba-arg or --ba-code-hash (@doitian)

Bug Fixes

• fix: random failure caused by dirty exit in RPC test #1092: Random failure caused by dirty exit in RPC test (@doitian)

  Close the server before exit RPC test.

• fix: ibd should remain false once returned false #1097: Ibd should remain false once returned false (@quake)

  This PR is trying to resolve ibd peer timeout slow switching issue for some special case, e.x. node resume after hibernation, it's a performance optimization also.

  bitcoin reference: https://github.com/bitcoin/bitcoin/blob/0b68fca700713e8e4c843c982b6047dc04410bc0/src/validation.cpp#L1027

• fix: resolve compact block switch fork issue #1100: Resolve compact block switch fork issue (@quake)

• fix: fix debug log state error #1101: Fix debug log state error (@driftluo)

  fix debug log state error

• fix: fix sync logic #1103: Fix sync logic (@driftluo)

  1. fix protect node count
  2. fix header sync eviction condition

• fix: potential error in alert version compare #1108: Potential error in alert version compare (@jjyr)

  As @keroro520 mentioned, there is a potential bug in case like: "0.10.0" < "0.9.10"

• fix: rpc test #1117: Rpc test (@jjyr)

• fix: Process orphan blocks when their parents were relayed #1127: Process orphan blocks when their parents were relayed (@keroro520)

  • test: Add an integration-test test_relay_parent_of_orphan_block
  • feat: Move orphan_block_pool into SyncSharedState
  • fix: Process orphan blocks when their parents were relayed. If relayer does not process the orphan_block_pool when inserting a new relaying block, it may leave some orphan blocks that be unable to re-process any further. This change makes synchronizer and relayer share the same one implementation of insert_new_block, so the above bug been fixed.

• fix: mark failed dialing as feeler #1109: Mark failed dialing as feeler (@jjyr)

• fix: Integration test broken by #1094 #1134: Integration test broken by feat: Secondary miner issurance, split DAO as a separate contract #1094 (@xxuejie)

• fix: total difficulty comparison should include hash #1135: Total difficulty comparison should include hash (@quake)

  resolve similar bug as fix: resolve compact block switch fork issue #1100 and refactor the previous fix to method.

• fix: resolve fresh proposal txs checking bug #1139: Resolve fresh proposal txs checking bug (@quake)

• fix: prof tps exclude cellbase #1144: Prof tps exclude cellbase (@jjyr)

Improvements

• perf: Reveal network errors and involver handle it #1072: Reveal network errors and involver handle it (@keroro520)

  • feat: Reveal network errors. Currently, when CKBProtocolContext receives an error from p2p, it only logs the error and doesn't return to the caller. I change to CKBProtocolContext return the error to the caller, and caller handles it.

  • perf: Short-circuiting break if occurs network error, Synchronizer responses Blocks and Transactions. This is the original intention of this PR. To achieve it, I have to make CKBProtocolContext reveals the network errors, which introduces most of the change code.

• refactor: Define a general Filter struct #1073: Define a general Filter struct (@keroro520)

  • refactor: Define a Filter struct

  • feat: Use LruCache to avoid memory leaking

    If we use native HashMap/HashSet as cache/filter, we have to prevent memory overflow by timely check or else. Replacing HashMap/HashSet with LruCache help us limit the memory usage, and reduce works around pruning staled items.

• perf: Avoid re-requesting blocks in orphan pool #1098: Avoid re-requesting blocks in orphan pool (@keroro520)

• refactor(storage): use flatbuffer instead of bincode in storage #1032 storage: Use flatbuffer instead of bincode in storage (@yangby-cryptape)

  This is a breaking change: b:database

  Associated Issues

  • Should not use same struct for both storage and core logic in all cases #892
  • Use different ways to serialize and deserialize #1031

  Description

  • I use flatbuffer, not cfb. We can do it later.

  • I copy some code from ckb-protocol, not just use it, because:

    • Since the data is from local storage, we can ensure the data format shouldn't be malformed.

      Do expect(..) is safety for storage, but not safety for network.

    • I try to split those proto-types into different schemas, classify them according to their usages.

    • If required, we can rewrite proto-types for storage, and don't have to modify proto-types for network.

    • I rewrite some of them.

  • I just remove bincode from ckb-store, but I keep the bincode for computing hashes. We can do it later.

  • I can't remove serde from ckb-core, because:

    • ckb/chain/src/chain.rs

      Lines 58 to 60 in 99dd90e

      	#[derive(Debug, Default, Serialize, Deserialize)]
      	pub struct ForkChanges {
      	// blocks attached to index after forks

    • ckb/util/instrument/src/import.rs

      Line 69 in 99dd90e

      let block: Arc<Block> = Arc::new(serde_json::from_str(&s)?);

    • A few log outputs.

    I think we should use jsonrpc-types for importing and exporting jsons and use Display for log outputs.

    Then we can remove serde from ckb-core.

  BREAKING CHANGE

  The data format in storage is changed.

• perf: skip stored block processing #1126: Skip stored block processing (@quake)

Misc

• More tests on ckb utils #1081: More tests on ckb utils (@jjyr)

Doc

• doc: add alerts example in RPC doc #1106: Add alerts example in RPC doc (@jjyr)