[ZK Stack] - How to integrate a data availbility layer in self deployed zkysnc hyperchain using validium as L1 batch commit?

[sachanayush47]

Team or Project

No response

Module Affected

• System contracts (era-system-contracts)
• Smart contracts (era-contracts)
• zkEVM (era-zk_evm)
• Server (zksync-era)
• Prover (era-boojum)
• Fee withdrawer (era-fee-withdrawer)
• Block Explorer (clock-explorer)
• Portal - Wallet + Bridge (dapp-portal)
• Other (please specify below)

Rust Version

1.80.1

Operating System

• Ubuntu
• MacOS
• Windows
• Other (Please specify below)

Issue Description

We are trying to integrate data availability layer particularly "Celestia" into self deployed zksync hyperchain using validium as L1 data batch commit
We were unable any doc/article related to integrating DAL into zksync hyperchain.

Thanks

Expected Behavior

NA

Current Behavior

NA

Repository Link (if applicable)

No response

Additional Details

No response

Prior Research

• Yes, I searched for similar issues and did not find any matching ones.

[gleenai[bot]]

To integrate a Data Availability Layer (DAL) like Celestia into your self-deployed ZKsync hyperchain using Validium for L1 batch commits, you can follow the approach outlined in the ZK Stack documentation. Here’s a summarized guide based on the information available:

1. Modify the Configuration File: In your ZK Stack configuration, change the setting from "Blob" to "Calldata" to accommodate the integration with Celestia. This is done by setting ETH_SENDER_SENDER_PUBDATA_SENDING_MODE=Calldata.

2. Commit Batch Changes: At the commit batch step, replace the pubdata with hash(pubdata) and submit the pubdata to Celestia or your chosen alternative Data Availability storage.

This approach is based on the general method of changing the Data Availability Layer in ZK Stack, which should be similar for integrating with Celestia. For more detailed guidance and community support, you can visit the ZKsync developers discussions on Github.

Your feedback helps us serve you better. If this fully addresses your question, please give it a thumbs up. If it doesn't, please give it a thumbs down. This will inform the community and team so we can improve our support.

[bxpana]

Celestia was just added and you can see how to add it in the PR, matter-labs/zksync-era#2983

We're hoping to have Eigen available soon as well, matter-labs/zksync-era#3155

[sachanayush47]

Celestia was just added and you can see how to add it in the PR, matter-labs/zksync-era#2983

We're hoping to have Eigen available soon as well, matter-labs/zksync-era#3155

Hi,
I tried the mentioned steps in the PR with the fresh deployment but this doesn't seems to work.
When I try to do a transaction via metamask, it never succeeds and stuck at "sending" step. Block explorer is also completely blank. Earlier this was not the case.

Logs L2 server:

This is logged only when I ran the zkstack server first time after that chain is not processing any new transaction.

2024-10-30T11:05:46.561571Z  INFO zksync_da_clients::celestia::sdk: broadcast blob transaction succeeded tx_hash=fd34451151e349c78c2d31c5c2e168a070f1295b8155d1d848a4b5f6815e6430

2024-10-30T11:22:00.426345Z  WARN zksync_base_token_adjuster::base_token_ratio_provider: No latest price found in the database. Using default ratio.
2024-10-30T11:22:01.326700Z DEBUG EthTxManager::loop_iteration: zksync_web3_decl::client: Request to request `eth_blockNumber` by component `eth_tx_manager` was rate-limited using policy 1 reqs/0ns: AcquireStats { was_waiting: true, total_sleep_time: 400ns } network="ethereum_17000" component="eth_tx_manager" origin=request `eth_blockNumber`
2024-10-30T11:22:02.426687Z  WARN zksync_base_token_adjuster::base_token_ratio_provider: No latest price found in the database. Using default ratio.
2024-10-30T11:22:03.856629Z  INFO zksync_node_api_server::web3::metrics: Observed error code 3 (origin: App) for method `eth_call`
2024-10-30T11:22:03.861860Z  INFO zksync_node_api_server::web3::metrics: Observed error code 3 (origin: App) for method `eth_call`
2024-10-30T11:22:03.867724Z  INFO zksync_node_api_server::web3::metrics: Observed error code 3 (origin: App) for method `eth_call`
2024-10-30T11:22:03.889634Z  INFO zksync_node_api_server::web3::metrics: Observed error code 3 (origin: App) for method `eth_call`
2024-10-30T11:22:03.895550Z  INFO zksync_node_api_server::web3::metrics: Observed error code 3 (origin: App) for method `eth_call`
2024-10-30T11:22:03.968689Z  INFO zksync_node_api_server::web3::metrics: Observed error code 3 (origin: App) for method `eth_call`
2024-10-30T11:22:03.975278Z  INFO zksync_node_api_server::web3::metrics: Observed error code 3 (origin: App) for method `eth_call`
2024-10-30T11:22:04.425981Z  WARN zksync_base_token_adjuster::base_token_ratio_provider: No latest price found in the database. Using default ratio.
2024-10-30T11:22:06.426909Z  WARN zksync_base_token_adjuster::base_token_ratio_provider: No latest price found in the database. Using default ratio.
2024-10-30T11:22:08.426792Z  WARN zksync_base_token_adjuster::base_token_ratio_provider: No latest price found in the database. Using default ratio.
2024-10-30T11:22:10.426094Z  WARN zksync_base_token_adjuster::base_token_ratio_provider: No latest price found in the database. Using default ratio.
2024-10-30T11:22:12.426692Z  WARN zksync_base_token_adjuster::base_token_ratio_provider: No latest price found in the database. Using default ratio.
2024-10-30T11:22:14.426214Z  WARN zksync_base_token_adjuster::base_token_ratio_provider: No latest price found in the database. Using default ratio.

Block explorer logs:

data-fetcher-1  | [DataFetcher] Info    30/10/2024 11:22:52.325 [TokenService] Cannot parse ERC20 contract. Might be a token of a different type. - {"contractAddress":"0x6d997a4d90461b0B175fFE6430862B53D15A5688"} +0ms
data-fetcher-1  | [DataFetcher] Info    30/10/2024 11:22:52.325 [TokenService] Cannot parse ERC20 contract. Might be a token of a different type. - {"contractAddress":"0x1c96C3d90B12A41f7b037D30a80bfA1DdAf050E4"} +0ms
data-fetcher-1  | [DataFetcher] Info    30/10/2024 11:22:52.325 [TokenService] Cannot parse ERC20 contract. Might be a token of a different type. - {"contractAddress":"0x5512e23457C4a7a6E3DdC497F0EBCB55199f1225"} +0ms
worker-1        | [Worker] Info 30/10/2024 11:22:52.358 [BlockProcessor] Adding block #0 - {"blockNumber":0} +175ms
worker-1        | [Worker] Info 30/10/2024 11:22:52.358 [BlockProcessor] Adding block #1 - {"blockNumber":1} +0ms
worker-1        | [Worker] Info 30/10/2024 11:22:52.358 [BlockProcessor] Adding block #2 - {"blockNumber":2} +0ms
worker-1        | [Worker] Error        30/10/2024 11:22:52.391 [UnitOfWork] Error while processing the transaction. Rolling back - {"stack":["QueryFailedError: null value in column \"root\" of relation \"transactionReceipts\" violates not-null constraint\n    at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:211:19)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async InsertQueryBuilder.execute (/usr/src/app/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33)\n    at async TransactionReceiptRepository.add (/usr/src/app/packages/worker/dist/repositories/base.repository.js:37:9)\n    at async TransactionProcessor.add (/usr/src/app/packages/worker/dist/transaction/transaction.processor.js:46:9)\n    at async Promise.all (index 0)\n    at async BlockProcessor.addBlock (/usr/src/app/packages/worker/dist/block/block.processor.js:142:13)\n    at async Promise.all (index 1)\n    at async /usr/src/app/packages/worker/dist/block/block.processor.js:108:17\n    at async /usr/src/app/packages/worker/dist/unitOfWork/unitOfWork.provider.js:81:17"]} +33ms
worker-1        | [Worker] Info 30/10/2024 11:22:52.399 [BatchProcessor] Getting batch from the blockchain - {"batchNumber":0,"currentBatchState":0} +8ms
worker-1        | [Worker] Info 30/10/2024 11:22:52.400 [BatchProcessor] Getting batch from the blockchain - {"batchNumber":0,"currentBatchState":2} +1ms
worker-1        | [Worker] Error        30/10/2024 11:22:52.402 [BlockService] Error on processing next block range, waiting 256000 ms to retry - {"stack":["QueryFailedError: null value in column \"root\" of relation \"transactionReceipts\" violates not-null constraint\n    at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:211:19)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async InsertQueryBuilder.execute (/usr/src/app/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33)\n    at async TransactionReceiptRepository.add (/usr/src/app/packages/worker/dist/repositories/base.repository.js:37:9)\n    at async TransactionProcessor.add (/usr/src/app/packages/worker/dist/transaction/transaction.processor.js:46:9)\n    at async Promise.all (index 0)\n    at async BlockProcessor.addBlock (/usr/src/app/packages/worker/dist/block/block.processor.js:142:13)\n    at async Promise.all (index 1)\n    at async /usr/src/app/packages/worker/dist/block/block.processor.js:108:17\n    at async /usr/src/app/packages/worker/dist/unitOfWork/unitOfWork.provider.js:81:17"]} +2ms
worker-1        | [Worker] Info 30/10/2024 11:22:52.412 [BatchProcessor] Getting batch from the blockchain - {"batchNumber":0,"currentBatchState":3} +10ms
worker-1        | [Worker] Info 30/10/2024 11:22:53.198 [BatchProcessor] Getting batch from the blockchain - {"batchNumber":0,"currentBatchState":1} +786ms
worker-1        | [Worker] Info 30/10/2024 11:22:53.417 [BatchProcessor] Getting batch from the blockchain - {"batchNumber":0,"currentBatchState":0} +219ms

[bxpana]

From the logs you provided it seems like things are healthy and the blob was broadcasted, but it looks like you are sending an invalid txn which is causing the issues

[sachanayush47]

From the logs you provided it seems like things are healthy and the blob was broadcasted, but it looks like you are sending an invalid txn which is causing the issues

The issue is with the block explorer only when using a external DAL, block explorer doesn't show any transactions.

[BlockService] Error on processing next block range, waiting 256000 ms to retry - {"stack":["QueryFailedError: null value in column \"root\" of relation \"transactionReceipts\" violates not-null constraint\n    at PostgresQueryRunner.query (/home/zksync/block-explorer/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:211:19)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async InsertQueryBuilder.execute (/home/zksync/block-explorer/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33)\n    at async TransactionReceiptRepository.add (/home/zksync/block-explorer/packages/worker/dist/repositories/base.repository.js:37:9)\n    at async TransactionProcessor.add (/home/zksync/block-explorer/packages/worker/dist/transaction/transaction.processor.js:46:9)\n    at async Promise.all (index 0)\n    at async BlockProcessor.addBlock (/home/zksync/block-explorer/packages/worker/dist/block/block.processor.js:142:13)\n    at async Promise.all (index 1)\n    at async /home/zksync/block-explorer/packages/worker/dist/block/block.processor.js:108:17\n    at async /home/zksync/block-explorer/packages/worker/dist/unitOfWork/unitOfWork.provider.js:81:17"]} +1ms