bug: panicked at 'supplied instant is later than self' on c6g.2xlarge EC2 (arm64)

[BugenZhao]

Thanks for your hard work on this excellent crate! Here is a bug report: when I was using it on an EC2 with arm64 architecture (c6g.2xlarge), I encountered 'supplied instant is later than self' frequently. Below is the backtrace.

Looks like the elapsed directly unwraps the checked_duration_since and panics on time going backward. However, the Instant from the standard library says,

Previous rust versions panicked when self was earlier than the current time. Currently this method returns a Duration of zero in that case. Future versions may reintroduce the panic.

What about also conforming to the behavior of std, so that there'll be no need to call now().saturating_duration_since(start_time)?

thread 'tokio-runtime-worker' panicked at 'supplied instant is later than self', /home/ubuntu/.cargo/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/minstant-0.1.1/src/instant.rs:49:9
stack backtrace:
   0: rust_begin_unwind
             at ./rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/std/src/panicking.rs:584:5
   1: core::panicking::panic_fmt
             at ./rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/core/src/panicking.rs:142:14
   2: core::panicking::panic_display
             at ./rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/core/src/panicking.rs:72:5
   3: core::panicking::panic_str
             at ./rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/core/src/panicking.rs:56:5
   4: core::option::expect_failed
             at ./rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/core/src/option.rs:1874:5
   5: core::option::Option<T>::expect
             at ./home/ubuntu/.cargo/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/minstant-0.1.1/src/instant.rs:49:9
   6: minstant::instant::Instant::duration_since
             at ./home/ubuntu/.cargo/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/minstant-0.1.1/src/instant.rs:49:9
   7: <minstant::instant::Instant as core::ops::arith::Sub>::sub
             at ./home/ubuntu/.cargo/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/minstant-0.1.1/src/instant.rs:205:9
   8: minstant::instant::Instant::elapsed
             at ./home/ubuntu/.cargo/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/minstant-0.1.1/src/instant.rs:120:9
   9: risingwave_stream::executor::dispatch::DispatchExecutorInner::dispatch::{{closure}}
             at ./ebs/risingwave/src/stream/src/executor/dispatch.rs:85:29
  10: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at ./rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/core/src/future/mod.rs:91:19
  11: <async_stack_trace::StackTraced<F> as core::future::future::Future>::poll
             at ./ebs/risingwave/src/utils/async_stack_trace/src/lib.rs:171:23
  12: <futures_util::future::future::fuse::Fuse<Fut> as core::future::future::Future>::poll
             at ./home/ubuntu/.cargo/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/futures-util-0.3.21/src/future/future/fuse.rs:86:37
  13: <risingwave_stream::executor::dispatch::DispatchExecutor as risingwave_stream::executor::StreamConsumer>::execute::{{closure}}
             at ./ebs/risingwave/src/stream/src/executor/dispatch.rs:272:22
  14: <futures_async_stream::try_stream::from_generator::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at ./home/ubuntu/.cargo/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/futures-async-stream-0.2.5/src/lib.rs:430:31
  15: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at ./home/ubuntu/.cargo/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/futures-core-0.3.21/src/stream.rs:120:9
  16: <&mut S as futures_core::stream::Stream>::poll_next
             at ./home/ubuntu/.cargo/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/futures-core-0.3.21/src/stream.rs:104:9
  17: <tokio_stream::stream_ext::next::Next<St> as core::future::future::Future>::poll
             at ./home/ubuntu/.cargo/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/tokio-stream-0.1.9/src/stream_ext/next.rs:42:9
  18: <tracing_futures::Instrumented<T> as core::future::future::Future>::poll
             at ./home/ubuntu/.cargo/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/tracing-futures-0.2.5/src/lib.rs:283:9
  19: <async_stack_trace::StackTraced<F> as core::future::future::Future>::poll
             at ./ebs/risingwave/src/utils/async_stack_trace/src/lib.rs:171:23
  20: <futures_util::future::future::fuse::Fuse<Fut> as core::future::future::Future>::poll
             at ./home/ubuntu/.cargo/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/futures-util-0.3.21/src/future/future/fuse.rs:86:37
  21: risingwave_stream::executor::actor::Actor<C>::run::{{closure}}
             at ./ebs/risingwave/src/stream/src/executor/actor.rs:165:16

Downstream issue: risingwavelabs/risingwave#4547

[zhongzc]

Thanks for your report!

On Linux with arm64, minstant doesn't use CPU timing instruction to produce clock time, instead, it will fall back to coarse time:

libc::clock_gettime(libc::CLOCK_MONOTONIC_COARSE, ...)

The value of clock time is really not expected to jump back, ref man clock_gettime 2:

  CLOCK_MONOTONIC
         Clock that cannot be set and represents monotonic time since—as described by POSIX—"some unspecified point in the past".  On  Linux,  that
         point corresponds to the number of seconds that the system has been running since it was booted.

         The  CLOCK_MONOTONIC  clock  is not affected by discontinuous jumps in the system time (e.g., if the system administrator manually changes
         the clock), but is affected by the incremental adjustments performed by adjtime(3) and NTP.  This clock does not count time that the  sys‐
         tem is suspended.

  CLOCK_MONOTONIC_COARSE (since Linux 2.6.32; Linux-specific)
         A  faster but less precise version of CLOCK_MONOTONIC.  Use when you need very fast, but not fine-grained timestamps.  Requires per-archi‐
         tecture support, and probably also architecture support for this flag in the vdso(7).

You can check if your system is encountering other clock issues.

However, I have no objection to following std. You can send a PR to do that and I will help release it, then the panic will not happen in the next version.

[BugenZhao]

However, I have no objection to following std. You can send a PR to do that and I will help release it, then the panic will not happen in the next version.

Cool! I've opened a pull request at #24.