bug: hummock wait for current epoch panicked at current epoch can't read, because the epoch in storage is not updated

[BugenZhao]

Occurred on CI parallel e2e tests:
https://buildkite.com/risingwavelabs/pull-request/builds/14943#018576c6-64b0-4156-8258-9845ebfa8bcc

Backtrace:

thread 'risingwave-batch-tasks' panicked at 'current epoch can't read, because the epoch in storage is not updated, epoch3637285672779776, sealed epoch3637285671731200', src/storage/src/hummock/state_store.rs:189:21
stack backtrace:
0: rust_begin_unwind
             at /rustc/bdb07a8ec8e77aa10fb84fae1d4ff71c21180bb4/library/std/src/panicking.rs:575:5
   1: core::panicking::panic_fmt
             at /rustc/bdb07a8ec8e77aa10fb84fae1d4ff71c21180bb4/library/core/src/panicking.rs:64:14
   2: risingwave_storage::hummock::state_store::<impl risingwave_storage::store::StateStore for risingwave_storage::hummock::HummockStorage>::try_wait_epoch::{{closure}}
             at ./src/storage/src/hummock/state_store.rs:189:21
   3: <S as risingwave_storage::store_impl::boxed_state_store::DynamicDispatchedStateStoreExt>::try_wait_epoch::{{closure}}
             at ./src/storage/src/store_impl.rs:828:39
   4: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/bdb07a8ec8e77aa10fb84fae1d4ff71c21180bb4/library/core/src/future/future.rs:124:9
   5: <async_stack_trace::StackTraced<F,_> as core::future::future::Future>::poll
             at ./src/utils/async_stack_trace/src/lib.rs:149:36
   6: <risingwave_storage::monitor::monitored_store::MonitoredStateStore<S> as risingwave_storage::store::StateStore>::try_wait_epoch::{{closure}}
             at ./src/storage/src/monitor/monitored_store.rs:177:17
   7: risingwave_storage::table::batch_table::storage_table::StorageTableIterInner<S>::new::{{closure}}
             at ./src/storage/src/table/batch_table/storage_table.rs:550:36
   8: risingwave_storage::table::batch_table::storage_table::StorageTable<S>::iter_with_encoded_key_range::{{closure}}::{{closure}}::{{closure}}
             at ./src/storage/src/table/batch_table/storage_table.rs:354:17
   9: <F as futures_core::future::TryFuture>::try_poll
             at ./.cargo/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/futures-core-0.3.25/src/future.rs:82:9
  10: <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll
             at ./.cargo/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/futures-util-0.3.25/src/future/try_future/into_future.rs:34:9
  11: <futures_util::stream::futures_ordered::OrderWrapper<T> as core::future::future::Future>::poll
             at ./.cargo/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/futures-util-0.3.25/src/stream/futures_ordered.rs:55:9
  12: <futures_util::stream::futures_unordered::FuturesUnordered<Fut> as futures_core::stream::Stream>::poll_next
             at ./.cargo/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/futures-util-0.3.25/src/stream/futures_unordered/mod.rs:515:17
  13: futures_util::stream::stream::StreamExt::poll_next_unpin
             at ./.cargo/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/futures-util-0.3.25/src/stream/stream/mod.rs:1626:9
  14: <futures_util::stream::futures_ordered::FuturesOrdered<Fut> as futures_core::stream::Stream>::poll_next
             at ./.cargo/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/futures-util-0.3.25/src/stream/futures_ordered.rs:194:26
  15: <S as futures_core::stream::TryStream>::try_poll_next
             at ./.cargo/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/futures-core-0.3.25/src/stream.rs:196:9
  16: <futures_util::stream::try_stream::try_collect::TryCollect<St,C> as core::future::future::Future>::poll
             at ./.cargo/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/futures-util-0.3.25/src/stream/try_stream/try_collect.rs:46:26
  17: <futures_util::future::try_join_all::TryJoinAll<F> as core::future::future::Future>::poll
             at ./.cargo/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/futures-util-0.3.25/src/future/try_join_all.rs:174:44
  18: risingwave_storage::table::batch_table::storage_table::StorageTable<S>::iter_with_encoded_key_range::{{closure}}
             at ./src/storage/src/table/batch_table/storage_table.rs:360:9
  19: risingwave_storage::table::batch_table::storage_table::StorageTable<S>::iter_with_pk_bounds::{{closure}}
             at ./src/storage/src/table/batch_table/storage_table.rs:494:9
  20: risingwave_storage::table::batch_table::storage_table::StorageTable<S>::batch_iter_with_pk_bounds::{{closure}}
             at ./src/storage/src/table/batch_table/storage_table.rs:507:13
  21: risingwave_batch::executor::row_seq_scan::RowSeqScanExecutor<S>::execute_range::{{closure}}
             at ./src/batch/src/executor/row_seq_scan.rs:409:14

[BugenZhao]

cc @zwang28 Could you please help to take a look? 🥺

[zwang28]

seal_epoch is handled by HummockEventHandler asynchronously.

Need a fix to ensure it's done before notifying barrier finishing.