Westend crashes with wasm error: out of bounds memory access at #9412261

[EclesioMeloJunior]

Describe the bug

The following stack trace was retrieved from a Gossamer staging node while syncing with Westend

September 28, 2023 at 17:28 (UTC-4:00) | 2023-09-28T21:28:27Z ERROR requesting max blocks from best block header: while handling workers results: while handling ready block: processing block data with header and body: handling block: failed to execute block 9412261: running runtime function: wasm error: out of bounds memory access 
September 28, 2023 at 17:28 (UTC-4:00) | wasm stack trace: 
September 28, 2023 at 17:28 (UTC-4:00) | .<memory_db::MemoryDB<H,KF,T,M> as core::default::Default>::default::h035979c933311a62(i32)
September 28, 2023 at 17:28 (UTC-4:00) | .pallet_session::historical::ProvingTrie<T>::generate_for::h1bf55405a66df38a(i32,i32) 
September 28, 2023 at 17:28 (UTC-4:00) | .<(TupleElement0,TupleElement1) as frame_support::traits::hooks::OnInitialize<BlockNumber>>::on_initialize::h8fe686fce2635624(i32) i64
September 28, 2023 at 17:28 (UTC-4:00) | .frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::initialize_block::h413dbea923e2629a(i32)
September 28, 2023 at 17:28 (UTC-4:00) | .Core_execute_block(i32,i32) i64 pkg=sync 
September 28, 2023 at 17:28 (UTC-4:00) | 2023-09-28T21:28:27Z INFO 🚣 currently syncing, 16 peers connected, 16 available workers, target block number 17655067, finalised block number 9412096 with hash 0x1e5b932b5aabb083b452b64a01ee7be1306b5a432a6b74bc9c5bdf8f074e3546 pkg=sync
September 28, 2023 at 17:28 (UTC-4:00) | 2023-09-28T21:28:20Z WARN failed to advertise in the DHT: failed to find any peer in table pkg=network 
September 28, 2023 at 17:28 (UTC-4:00) | 2023-09-28T21:28:04Z INFO 🔨 finalised block number 9412096 with hash 0x1e5b932b5aabb083b452b64a01ee7be1306b5a432a6b74bc9c5bdf8f074e3546 pkg=sync
September 28, 2023 at 17:28 (UTC-4:00)	2023-09-28T21:28:27Z ERROR requesting max blocks from best block header: while handling workers results: while handling ready block: processing block data with header and body: handling block: failed to execute block 9412261: running runtime function: wasm error: out of bounds memory access
September 28, 2023 at 17:28 (UTC-4:00) | wasm stack trace:
September 28, 2023 at 17:28 (UTC-4:00) | .<memory_db::MemoryDB<H,KF,T,M> as core::default::Default>::default::h035979c933311a62(i32)
September 28, 2023 at 17:28 (UTC-4:00) | .pallet_session::historical::ProvingTrie<T>::generate_for::h1bf55405a66df38a(i32,i32)
September 28, 2023 at 17:28 (UTC-4:00) | .<(TupleElement0,TupleElement1) as frame_support::traits::hooks::OnInitialize<BlockNumber>>::on_initialize::h8fe686fce2635624(i32) i64
September 28, 2023 at 17:28 (UTC-4:00) | .frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::initialize_block::h413dbea923e2629a(i32)
September 28, 2023 at 17:28 (UTC-4:00) | .Core_execute_block(i32,i32) i64 pkg=sync
September 28, 2023 at 17:28 (UTC-4:00) | 2023-09-28T21:28:27Z INFO 🚣 currently syncing, 16 peers connected, 16 available workers, target block number 17655067, finalised block number 9412096 with hash 0x1e5b932b5aabb083b452b64a01ee7be1306b5a432a6b74bc9c5bdf8f074e3546 pkg=sync
September 28, 2023 at 17:28 (UTC-4:00) | 2023-09-28T21:28:20Z WARN failed to advertise in the DHT: failed to find any peer in table pkg=network
September 28, 2023 at 17:28 (UTC-4:00) | 2023-09-28T21:28:04Z INFO 🔨 finalised block number 9412096 with hash 0x1e5b932b5aabb083b452b64a01ee7be1306b5a432a6b74bc9c5bdf8f074e3546 pkg=sync

[EclesioMeloJunior]

The problem related to block #9412261 (wasm error: out of bounds memory access) is not related to memory grow, in my tests I setup a wazero runtime with WithMemoryCapacityFromMax(true) and WithMemoryLimitPages(23) of 23 and I got the following stack trace

rtConfig := wazero.NewRuntimeConfig().
    WithMemoryCapacityFromMax(true).
    WithMemoryLimitPages(23)

rt := wazero.NewRuntimeWithConfig(ctx, rtConfig)
_, err = rt.NewHostModuleBuilder("env").
     ExportMemory("memory", 23).

Executing block #9412261...
    instance_test.go:482:
        	Error Trace:	/Users/eclesiojunior/go/src/github.com/ChainSafe/gossamer/lib/runtime/wazero/instance_test.go:482
        	Error:      	Received unexpected error:
        	            	running runtime function: allocating: allocator out of space; failed to grow heap; heap.Grow ignored (recovered by wazero)
        	            	wasm stack trace:
        	            		env.ext_storage_get_version_1(i64) i64
        	            		.sp_io::storage::extern_host_function_impls::get::hf3b4edf587f8e8f1(i32,i32,i32)
        	            		.pallet_staking::pallet::pallet::Pallet<T>::eras_stakers::hdfb9a8654716e6b5(i32,i32,i32)
        	            		.<(TupleElement0,TupleElement1) as frame_support::traits::hooks::OnInitialize<BlockNumber>>::on_initialize::h8fe686fce2635624(i32) i64
        	            		.frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::initialize_block::h413dbea923e2629a(i32)
        	            		.Core_execute_block(i32,i32) i64

The above stacktrace is similar to the one describe in the issue #3509, however the difference is in the error: running runtime function: allocating: allocator out of space; failed to grow heap; heap.Grow ignored (recovered by wazero) while in the issue is out of bounds memory acess. So as far as I can tell the looks like the wazero runtime is capable of handling the memory grow.

fyi @timwu20 @jimjbrettj

[EclesioMeloJunior]

Increasing the wasm page to 65535 which gives us a memory size of (4.294.901.760 bytes) we face a different problem

2023-11-12T00:39:53Z ERROR requesting max blocks from best block header: while handling workers results: while handling ready block: processing block data with header and body: handling block: failed to execute block 9412261: running runtime function: wasm error: invalid table access

wasm stack trace: | c48ea0385bf0461c9736705c26c5ecc1 | gossamer-westend
-- | -- | --
November 11, 2023 at 20:39 (UTC-4:00) | .<memory_db::MemoryDB<H,KF,T,M> as core::default::Default>::default::h035979c933311a62(i32) | c48ea0385bf0461c9736705c26c5ecc1 | gossamer-westend
November 11, 2023 at 20:39 (UTC-4:00) | .pallet_session::historical::ProvingTrie<T>::generate_for::h1bf55405a66df38a(i32,i32) | c48ea0385bf0461c9736705c26c5ecc1 | gossamer-westend
November 11, 2023 at 20:39 (UTC-4:00) | .<(TupleElement0,TupleElement1) as frame_support::traits::hooks::OnInitialize<BlockNumber>>::on_initialize::h8fe686fce2635624(i32) i64 | c48ea0385bf0461c9736705c26c5ecc1 | gossamer-westend
November 11, 2023 at 20:39 (UTC-4:00) | .frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::initialize_block::h413dbea923e2629a(i32) | c48ea0385bf0461c9736705c26c5ecc1 | gossamer-westend
November 11, 2023 at 20:39 (UTC-4:00) | .Core_execute_block(i32,i32) i64 pkg=sync

wasm stack trace:	[c48ea0385bf0461c9736705c26c5ecc1](https://us-east-2.console.aws.amazon.com/ecs/v2/clusters/gossamer-stg/services/gossamer-westend-svc/tasks/c48ea0385bf0461c9736705c26c5ecc1?region=us-east-2)	gossamer-westend
November 11, 2023 at 20:39 (UTC-4:00)	.<memory_db::MemoryDB<H,KF,T,M> as core::default::Default>::default::h035979c933311a62(i32)	[c48ea0385bf0461c9736705c26c5ecc1](https://us-east-2.console.aws.amazon.com/ecs/v2/clusters/gossamer-stg/services/gossamer-westend-svc/tasks/c48ea0385bf0461c9736705c26c5ecc1?region=us-east-2)	gossamer-westend
November 11, 2023 at 20:39 (UTC-4:00)	.pallet_session::historical::ProvingTrie<T>::generate_for::h1bf55405a66df38a(i32,i32)	[c48ea0385bf0461c9736705c26c5ecc1](https://us-east-2.console.aws.amazon.com/ecs/v2/clusters/gossamer-stg/services/gossamer-westend-svc/tasks/c48ea0385bf0461c9736705c26c5ecc1?region=us-east-2)	gossamer-westend
November 11, 2023 at 20:39 (UTC-4:00)	.<(TupleElement0,TupleElement1) as frame_support::traits::hooks::OnInitialize<BlockNumber>>::on_initialize::h8fe686fce2635624(i32) i64	[c48ea0385bf0461c9736705c26c5ecc1](https://us-east-2.console.aws.amazon.com/ecs/v2/clusters/gossamer-stg/services/gossamer-westend-svc/tasks/c48ea0385bf0461c9736705c26c5ecc1?region=us-east-2)	gossamer-westend
November 11, 2023 at 20:39 (UTC-4:00)	.frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::initialize_block::h413dbea923e2629a(i32)	[c48ea0385bf0461c9736705c26c5ecc1](https://us-east-2.console.aws.amazon.com/ecs/v2/clusters/gossamer-stg/services/gossamer-westend-svc/tasks/c48ea0385bf0461c9736705c26c5ecc1?region=us-east-2)	gossamer-westend
November 11, 2023 at 20:39 (UTC-4:00)	.Core_execute_block(i32,i32) i64 pkg=sync