Dictionary V2 Ethereum

[jiqiang90]

Description

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

Fixes # (issue)

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Checklist

• I have tested locally
• I have performed a self review of my changes
• Updated any relevant documentation
• Linked to any relevant issues
• I have added tests relevant to my changes
• Any dependent changes have been merged and published in downstream modules
• My code is up to date with the base branch
• I have updated relevant changelogs. We suggest using chan

[github-actions]

Coverage report

Caution

Test run failed

St.❔	Category	Percentage	Covered / Total
🔴	Statements	49.67%	3311/6666
🟡	Branches	67.64%	349/516
🔴	Functions	47.04%	119/253
🔴	Lines	49.67%	3311/6666

Test suite run failed

Failed tests: 1/73. Failed suites: 2/14.

  ● rawBlockToEthBlock › successfully convers a block

    FetchError: request to http://localhost:3000/rpc/eth-mainnet failed, reason: connect ECONNREFUSED ::1:3000

      at ClientRequest.<anonymous> (node_modules/node-fetch/lib/index.js:1491:11)

---

  ● Test suite failed to run

    Jest worker encountered 4 child process exceptions, exceeding retry limit

      at ChildProcessWorker.initialize (node_modules/jest-worker/build/workers/ChildProcessWorker.js:181:21)

Report generated by 🧪jest coverage report action from 09973bd

[stwiname]

I think this is a good starting point but could evolve to be more flexible.

If we think about the future there are a few scenarios we could prepare for now:

• V2 Dictionaries may have a block range that isn't the whole network, so there might be more than one dictionary for a network.
• We can use this to build other data sources, e.g Space and Time or KYVE

Considering these future use cases i think this can be updated to support these cases:

• If there is an interface IDictionary that has a function getData to get block content/block numbers for a block range given the current datasources
• V1, V2 dictionaries can implement IDictionary
• The dictionary service can also implement IDictionary but control a collection of other IDictionary it can then route calls of getData to the appropriate IDictionary in the collection. This would behave similar to how we support multiple endpoints

Maybe we should look at renaming from Dictionary to DataProvider to better explain the purpose of this part of the code.

[stwiname]

Just be cautious about heightsBlocks being empty, enqueueBlocks could be called like so enqueueBlocks([])

[jiqiang90]

Good point, from line 93-95 we did an assertion for heightsBlocks.length, this ensure heights blocks at least have 1 block. I can made a change to enqueueBlocks input, ensure latestBufferHeight never to be undefined.

[stwiname]

I think this should be the project.network.dictionary instead. It doesn't take into account the manifest dictionary like it used to

[jiqiang90]

I will leave this with //TODO, this require update in type-core too, can do this when rebase with main

[stwiname]

This should have tests for this.

[stwiname]

I think this can be saved for another PR. But this service has nothing specific to ethereum, it would be good if this could now live purely in node-core. I think only substrate would need a specific version because of specVersions

[stwiname]

It would be good to have a lot more tests here to cover some various use cases similar to the v1 tests.

I'm particularly interested in handling contract creation filters where the to field is null

[stwiname]

This needs a test

[stwiname]

Suggested change

	filter.function
	filter.function !== undefined

The same changes i mentioned earlier for functions should be applied here.

Because this code is very similar for v1 and v2 it could probably be generalised

[stwiname]

This could be a large amount of data to log

[jiqiang90]

I removed ${JSON.stringify(data), I think from error msg from rawBlockToEthBlock is very detailed should be sufficent

[stwiname]

I don't think we need to worry about it for now because we don't have receipts. But we should apply the formatter changes from here https://github.com/subquery/subql-ethereum/blob/main/packages/node/src/ethereum/ethers/op/op-provider.ts#L19 to this formatter/ have a global formatter

[anhnhx131]

@stwiname
Hi, in the isFullBlock function of packages/node/src/ethereum/block.ethereum.ts, I saw logic

export function isFullBlock(block: BlockContent): block is EthereumBlock {
  // Light etherum block just contains transaction hashes for transactions.
  // If the block has no transactions then both types would be the same

  if (block.transactions.length && block.logs.length) {
    return typeof (block as EthereumBlock).transactions[0] !== 'string';
  }
  return false;
}

does this mean only writing index with blocks, which have transactions and logs?
What happens to the block containing the simple transaction? They will be ignored, right?