Parachain stopped finalizing blocks after upgrade

[gflow33]

I am running a pendulum rpc node and after upgrading from https://github.com/pendulum-chain/pendulum/releases/tag/1.1.0 to https://github.com/pendulum-chain/pendulum/releases/tag/1.6.0 the node stopped finalizing blocks and produced the errors in the log below

The container in which it runs is on ubuntu 22.04 and run with the following flags

./pendulum-collator \
--name="\U0001F6E1 DWELLIR PENDULUM RPC \U0001F6E1" \
--chain=pendulum \
--pruning=archive \
--rpc-max-connections=1000 \
--rpc-external --rpc-port=9933 \
--rpc-cors=all \
--rpc-methods=unsafe \
--prometheus-external \
--runtime-cache-size=16 \
--max-runtime-instances=32 \
--execution wasm \
-- \
--chain=polkadot

Node logs

Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]: 2025-02-04 16:11:15 [Parachain] Evicting failed runtime instance error=Runtime panicked: Could not lookup information required to validate the transaction
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]: 2025-02-04 16:11:15 [Parachain] Block prepare storage changes error: Error at calling runtime api: Execution failed: Runtime panicked: Could not lookup information required to validate the transaction
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]: 2025-02-04 16:11:15 [Parachain] 💔 Error importing block 0x7f9123dfff8be6108cd59a5b3346dbcbb6e1f0a54f5c0bde7c140db14f9d2466: consensus error: Import failed: Error at calling runtime api: Execution failed: Runtime panicked: Could not lookup information required to validate the transaction
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]: ====================
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]: Version: 1.6.0-93c9dbe0542
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:    0: sp_panic_handler::set::{{closure}}
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:    1: std::panicking::rust_panic_with_hook
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:    2: std::panicking::begin_panic_handler::{{closure}}
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:    3: std::sys_common::backtrace::__rust_end_short_backtrace
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:    4: rust_begin_unwind
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:    5: core::panicking::panic_fmt
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:    6: frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::execute_extrinsics_with_book_keeping::{{closure}}::panic_cold_display
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:    7: frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::execute_extrinsics_with_book_keeping
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:    8: tracing::span::Span::in_scope
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:    9: frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::execute_block
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:   10: amplitude_runtime::api::dispatch
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:   11: environmental::using
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:   12: sc_executor::executor::WasmExecutor<H>::with_instance::{{closure}}
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:   13: sc_executor::wasm_runtime::RuntimeCache::with_instance
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:   14: <sc_executor::executor::NativeElseWasmExecutor<D> as sp_core::traits::CodeExecutor>::call
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:   15: sp_state_machine::execution::StateMachine<B,H,Exec>::execute
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:   16: <sc_service::client::call_executor::LocalCallExecutor<Block,B,E> as sc_client_api::call_executor::CallExecutor<Block>>::contextual_call
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:   17: <sc_service::client::client::Client<B,E,Block,RA> as sp_api::CallApiAt<Block>>::call_api_at
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:   18: <amplitude_runtime::RuntimeApiImpl<__SrApiBlock__,RuntimeApiImplCall> as sp_transaction_pool::runtime_api::TaggedTransactionQueue<__SrApiBlock__>>::__runtime_api_internal_call_api_at
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:   19: sp_api::Core::execute_block
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:   20: <&sc_service::client::client::Client<B,E,Block,RA> as sc_consensus::block_import::BlockImport<Block>>::import_block::{{closure}}
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:   21: <alloc::sync::Arc<T> as sc_consensus::block_import::BlockImport<B>>::import_block::{{closure}}
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:   22: <cumulus_client_consensus_common::ParachainBlockImport<Block,BI,BE> as sc_consensus::block_import::BlockImport<Block>>::import_block::{{closure}}
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:   23: <alloc::boxed::Box<dyn sc_consensus::block_import::BlockImport<B>+Error = sp_consensus::error::Error+core::marker::Send+core::marker::Sync> as sc_consensus::block_import::BlockImport<B>>::import_block::{{closure}}
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:   24: futures_util::future::future::FutureExt::poll_unpin
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:   25: sc_consensus::import_queue::basic_queue::BlockImportWorker<B>::new::{{closure}}
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:   26: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:   27: <sc_service::task_manager::prometheus_future::PrometheusFuture<T> as core::future::future::Future>::poll
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:   28: <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:   29: <tracing_futures::Instrumented<T> as core::future::future::Future>::poll
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:   30: tokio::runtime::park::CachedParkThread::block_on
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:   31: tokio::runtime::handle::Handle::block_on
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:   32: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:   33: tokio::runtime::task::core::Core<T,S>::poll
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:   34: tokio::runtime::task::harness::Harness<T,S>::poll
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:   35: tokio::runtime::blocking::pool::Inner::run
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:   36: std::sys_common::backtrace::__rust_begin_short_backtrace
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:   37: core::ops::function::FnOnce::call_once{{vtable.shim}}
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:   38: std::sys::pal::unix::thread::Thread::new::thread_start
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:   39: <unknown>
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:   40: <unknown>
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]: Thread 'tokio-runtime-worker' panicked at 'Could not lookup information required to validate the transaction', /home/gitlab-runner/.cargo/git/checkouts/polkadot-sdk-2a8e7ffb960df38b/0a31531/substrate/frame/executive/src/lib.rs:578
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]: This is a bug. Please report it at:
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]:         https://github.com/pendulum-chain/pendulum/issues/new
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]: 2025-02-04 16:11:15 [Parachain] Failed to check if the parachain block expired
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]: 2025-02-04 16:11:15 [Parachain] Evicting failed runtime instance error=Runtime panicked: Could not lookup information required to validate the transaction
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]: 2025-02-04 16:11:15 [Parachain] Block prepare storage changes error: Error at calling runtime api: Execution failed: Runtime panicked: Could not lookup information required to validate the transaction
Feb 04 16:11:15 juju-ab466c-0 polkadot[2284297]: 2025-02-04 16:11:15 [Parachain] 💔 Error importing block 0x7f9123dfff8be6108cd59a5b3346dbcbb6e1f0a54f5c0bde7c140db14f9d2466: consensus error: Import failed: Error at calling runtime api: Execution failed: Runtime panicked: Could not lookup information required to validate the transaction