Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failed to apply inherents error #841

Closed
green-jay opened this issue Oct 21, 2024 · 9 comments · Fixed by #881
Closed

Failed to apply inherents error #841

green-jay opened this issue Oct 21, 2024 · 9 comments · Fixed by #881

Comments

@green-jay
Copy link

Recently we observed that Chopsticks fails randomly but frequently when trying to produce a new block on Asset Hub. The error is Failed to apply inherents

Running with command:

npx @acala-network/chopsticks@latest xcm --relaychain=configs/polkadot.yml --parachain=configs/acala.yml --parachain=confi
gs/hydration.yml --parachain=configs/polkadot-asset-hub.yml

Full log:

    app: "chopsticks"
/Users/jaygreen/.npm/_npx/81ad9c881cb83600/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/block-builder.js:214
            throw new Error('Failed to apply inherents');
                  ^

Error: Failed to apply inherents
    at initNewBlock (/Users/jaygreen/.npm/_npx/81ad9c881cb83600/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/block-builder.js:214:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async buildBlock (/Users/jaygreen/.npm/_npx/81ad9c881cb83600/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/block-builder.js:321:44)
    at async TxPool.buildBlock (/Users/jaygreen/.npm/_npx/81ad9c881cb83600/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/txpool.js:344:43)
    at async TxPool.buildBlockIfNeeded (/Users/jaygreen/.npm/_npx/81ad9c881cb83600/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/txpool.js:328:9)```
@xlc
Copy link
Member

xlc commented Oct 22, 2024

can you provide bit more details on how to reproduce this error? e.g. do you need to send some XCM to trigger it? or some transaction on AH? does it happens every time?

@green-jay
Copy link
Author

It doesn't fail every time but maybe every 3rd time or so when sending any TX on Asset Hub. In my case I was sending XCMs to Asset Hub from and to Hydration

@xlc
Copy link
Member

xlc commented Oct 23, 2024

can you provide the encoded tx on hydration so I can replay it? tried a few simple one and works for me

@green-jay
Copy link
Author

Probably not connected to asset hub, now i sent this transfer Hydration -> Bifrost transfer and same error occured:
0x01010489000b00000000e1f50500000000000000000000000003010200b91f0100d43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d00

[08:57:19.309] INFO (block-builder): Hydration building #6,263,381
    app: "chopsticks"
    number: 6263381
    extrinsics: [
      "0x990284d43593c715fdd31c61141abd04…2c8558854ccde39a5684e7a56da27d00"
    ]
    umpCount: 0
        chopsticks::executor  TRACE: Completed Core_initialize_block
        chopsticks::executor  TRACE: Calling BlockBuilder_apply_extrinsic
             runtime::system  TRACE: Used block weight: PerDispatchClass { normal: Weight { ref_time: 0, proof_size: 0 }, operational: Weight { ref_time: 0, proof_size: 0 }, mandatory: Weight { ref_time: 28481410650, proof_size: 160701 } }
             runtime::system  TRACE: Used block length: 11
        chopsticks::executor  TRACE: Completed BlockBuilder_apply_extrinsic
[08:57:33.786] WARN (block-builder): Failed to apply inherents {} Error: Block 0x0335844ec624daecf3ad8bbf44e77d50da71b937f0a075b701eb4d16867ee0a3 not found
    app: "chopsticks"
/Users/jaygreen/.npm/_npx/81ad9c881cb83600/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/block-builder.js:214
            throw new Error('Failed to apply inherents');
                  ^

Error: Failed to apply inherents
    at initNewBlock (/Users/jaygreen/.npm/_npx/81ad9c881cb83600/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/block-builder.js:214:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async buildBlock (/Users/jaygreen/.npm/_npx/81ad9c881cb83600/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/block-builder.js:321:44)
    at async TxPool.buildBlock (/Users/jaygreen/.npm/_npx/81ad9c881cb83600/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/txpool.js:344:43)
    at async TxPool.buildBlockIfNeeded (/Users/jaygreen/.npm/_npx/81ad9c881cb83600/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/txpool.js:328:9)

@green-jay green-jay changed the title Asset Hub kills Chopsticks Failed to apply inherents error Oct 24, 2024
@ermalkaleci
Copy link
Collaborator

Seems like you're using you custom configs, can you share those as well?

@green-jay
Copy link
Author

Nothing special, sometimes overriden wasm but mostly not even that.
configs.zip

@green-jay
Copy link
Author

Also another person from our team encountered the same problem many times with vanilla chopsticks

@x676f64
Copy link
Contributor

x676f64 commented Feb 9, 2025

I am also experiencing this issue with hydration. I'm pulling their config (https://github.com/galacticcouncil/hydration-node/blob/master/launch-configs/chopsticks/hydradx.yml) but without the wasm override. This was in a batch token transfer test

❯ npx @acala-network/chopsticks@latest xcm -r polkadot -p configs/hydradx.yml
        chopsticks::executor  TRACE: Calling Metadata_metadata
[23:17:42.066] INFO: Hydration RPC listening on http://[::]:8000 and ws://[::]:8000
    app: "chopsticks"
[23:17:42.067] INFO: Loading config file https://raw.githubusercontent.com/AcalaNetwork/chopsticks/master/configs/polkadot.yml
    app: "chopsticks"
        chopsticks::executor  TRACE: Calling Metadata_metadata
        chopsticks::executor  TRACE: Completed Metadata_metadata
        chopsticks::executor  TRACE: Calling AuraApi_slot_duration
        chopsticks::executor  TRACE: Completed AuraApi_slot_duration
        chopsticks::executor  TRACE: Calling Metadata_metadata
[23:17:44.988] INFO: Polkadot RPC listening on http://[::]:8001 and ws://[::]:8001
    app: "chopsticks"
[23:17:45.485] INFO (xcm): Connected relaychain 'Polkadot' with parachain 'Hydration'
    app: "chopsticks"
        chopsticks::executor  TRACE: Completed Metadata_metadata
        chopsticks::executor  TRACE: Calling Metadata_metadata_versions
        chopsticks::executor  TRACE: Completed Metadata_metadata_versions
        chopsticks::executor  TRACE: Calling Metadata_metadata_at_version
        chopsticks::executor  TRACE: Completed Metadata_metadata_at_version
        chopsticks::executor  TRACE: Calling AccountNonceApi_account_nonce
        chopsticks::executor  TRACE: Completed AccountNonceApi_account_nonce
        chopsticks::executor  TRACE: Calling TaggedTransactionQueue_validate_transaction
      runtime::metadata-hash  DEBUG: CheckMetadataHash::additional_signed => None
        chopsticks::executor  TRACE: Completed TaggedTransactionQueue_validate_transaction
[23:18:39.500] INFO (block-builder): Hydration building #6,928,754
    app: "chopsticks"
    number: 6928754
    extrinsics: [
      "0x711684da66f0b7718cc73651dc2db0ca…05f1847b7758f9420f00070010a5d4e8"
    ]
    umpCount: 0
        chopsticks::executor  TRACE: Calling Core_initialize_block
        chopsticks::executor  TRACE: Completed Core_initialize_block
        chopsticks::executor  TRACE: Calling BlockBuilder_apply_extrinsic
             runtime::system  TRACE: Used block weight: PerDispatchClass { normal: Weight { ref_time: 0, proof_size: 0 }, operational: Weight { ref_time: 0, proof_size: 0 }, mandatory: Weight { ref_time: 26960877099, proof_size: 90504 } }
             runtime::system  TRACE: Used block length: 11
        chopsticks::executor  TRACE: Completed BlockBuilder_apply_extrinsic
[23:19:10.451] WARN (block-builder): Failed to apply inherents {} Error: Block 0xe6fa312c2d35199fad4f8a0ba23b2e2bbe4eb240fb12e2752dd71821fceb983e not found
    app: "chopsticks"
/Users/null/.npm/_npx/81ad9c881cb83600/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/block-builder.js:213
            throw new Error('Failed to apply inherents');
                  ^

Error: Failed to apply inherents
    at initNewBlock (/Users/null/.npm/_npx/81ad9c881cb83600/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/block-builder.js:213:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async buildBlock (/Users/null/.npm/_npx/81ad9c881cb83600/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/block-builder.js:320:44)
    at async TxPool.buildBlock (/Users/null/.npm/_npx/81ad9c881cb83600/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/txpool.js:344:43)
    at async TxPool.buildBlockIfNeeded (/Users/null/.npm/_npx/81ad9c881cb83600/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/txpool.js:328:9)

Node.js v23.3.

I am unable to reliably reproduce it, a second attempt at running this script did not trigger the stack trace. This did happen to me a few hours earlier in the night with a previous version of my code and again wasn't reproducible on the second try.

@xlc
Copy link
Member

xlc commented Feb 9, 2025

My suspicion is it was trying to build a block on a fork, and during the building, the fork got purged, so it failed to fetch state for the purged bock. At least that will be one edge case that's not handled currently.

Maybe we can have Chopsticks default to use the finalized block, instead of best

@xlc xlc mentioned this issue Feb 9, 2025
Merged
@xlc xlc closed this as completed in #881 Feb 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

default to finalzied head AcalaNetwork/chopsticks
4 participants
@xlc @ermalkaleci @x676f64 @green-jay