bug: compute node panicked at minstant time goes backward

[BugenZhao]

Found this bug when benchmarking the nexmark materialized source on arm64 ec2 for 2 hours. Maybe a bug in minstant? cc @skyzh

thread 'tokio-runtime-worker' panicked at 'supplied instant is later than self', /home/ubuntu/.cargo/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/minstant-0.1.1/src/instant.rs:49:9
stack backtrace:
   0: rust_begin_unwind
             at ./rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/std/src/panicking.rs:584:5
   1: core::panicking::panic_fmt
             at ./rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/core/src/panicking.rs:142:14
   2: core::panicking::panic_display
             at ./rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/core/src/panicking.rs:72:5
   3: core::panicking::panic_str
             at ./rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/core/src/panicking.rs:56:5
   4: core::option::expect_failed
             at ./rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/core/src/option.rs:1874:5
   5: core::option::Option<T>::expect
             at ./home/ubuntu/.cargo/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/minstant-0.1.1/src/instant.rs:49:9
   6: minstant::instant::Instant::duration_since
             at ./home/ubuntu/.cargo/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/minstant-0.1.1/src/instant.rs:49:9
   7: <minstant::instant::Instant as core::ops::arith::Sub>::sub
             at ./home/ubuntu/.cargo/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/minstant-0.1.1/src/instant.rs:205:9
   8: minstant::instant::Instant::elapsed
             at ./home/ubuntu/.cargo/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/minstant-0.1.1/src/instant.rs:120:9
   9: risingwave_stream::executor::dispatch::DispatchExecutorInner::dispatch::{{closure}}
             at ./ebs/risingwave/src/stream/src/executor/dispatch.rs:85:29
  10: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at ./rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/core/src/future/mod.rs:91:19
  11: <async_stack_trace::StackTraced<F> as core::future::future::Future>::poll
  12: <futures_util::future::future::fuse::Fuse<Fut> as core::future::future::Future>::poll
             at ./home/ubuntu/.cargo/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/futures-util-0.3.21/src/future/future/fuse.rs:86:37
  13: <risingwave_stream::executor::dispatch::DispatchExecutor as risingwave_stream::executor::StreamConsumer>::execute::{{closure}}
             at ./ebs/risingwave/src/stream/src/executor/dispatch.rs:272:22

I've also opened an issue on the upstream: tikv/minstant#23

[BugenZhao]

BTW, have we tested how much minstant performs better than coarsetime?

[skyzh]

ministant should fall back to something similar to coarsetime on arm64, as arm64 doesn't support TSO like x86 did. It would be okay to switch to coarsetime.

[skyzh]

Also I guess this issue happens because the future is being run on another core than Instant::now is captured.