Erlang Error - Monotonic time stepped backwards - rabbit ouage #8940
-
Describe the bugRunning rabbit on Ubuntu using docker with rabbit version of 3.11 could not locate the core dump, but the error is clear, searched for solution and saw something that might work.
any ideas on how I can debug the issue, or prevent it in the future, could not find anything in the documentation. Reproduction steps
Expected behaviorRabbit should not collapse/add an option with a flag to address this kind of issue. Additional contextNo response |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 2 replies
-
|
RabbitMQ does not adjust your operating system clock and cannot be assumed to be responsible for these very rare events. RabbitMQ 3.11 and 3.12 both require Erlang 25 and thus both use the same modern time API elements. NTP events can moved OS time backwards a tiny bit. Maybe such correction are logged in various OS logs. RabbitMQ allows you to specify any runtime flags you want to use, including those related to monotonic time and the time warp mode. A quick search reveals relevant settings in various operating system kernels, time warp handling in other data services, and many discussions of how Os clocks are not perfect. It is a very broad area to try to narrow down. RabbitMQ allows you to adopt any setting that Erlang provides. |
Beta Was this translation helpful? Give feedback.
-
|
The enforcement of monotonic time (at least by default) first shipped in 25.0 and was introduced in erlang/otp@debc443. I could not find a single issue or PR that would point to any incorrectness in the runtime. You can try using to get a leas strict behavior. Except for things like stream retention and TTL, RabbitMQ nodes are not as sensitive to time warp. Raft keeps track of sequences using a bunch of counters. Heartbeat tracking would not be affected by a single ms stepping backwards since the recommended range is usually in the 15-60s range. So both old and modern time warp defaults should work OK for RabbitMQ but one is obviously much stricter than the other, which can be an annoyance in some environments. |
Beta Was this translation helpful? Give feedback.
-
|
I'm also having the same issue on my RabbitMQ cluster service. I have NTP services enabled on all of my servers, and I suspect this error was triggered while time syncing, but I didn't find the relevant logs of system time modifications, so I can't fully confirm that. In this case, should I disable the NTP service or configure no_time_warp mode, which really works? |
Beta Was this translation helpful? Give feedback.
-
|
Do I recommend disabling NTP above or do I recommend something else? Also, how can we answer the "which one really works" question since you are guessing as to what has happened in your system? |
Beta Was this translation helpful? Give feedback.
The enforcement of monotonic time (at least by default) first shipped in 25.0 and was introduced in erlang/otp@debc443.
I could not find a single issue or PR that would point to any incorrectness in the runtime.
You can try using
to get a leas strict behavior. Except for things like stream retention and TTL, RabbitMQ nodes are not as sensitive to time warp.
Raft keeps track of sequences using a bunch of counters. Heartbeat tracking would not be affected by a single ms stepping backwards since the recommended range is usually in the 15-60s range.
So both old and modern time warp defaults should work OK for RabbitMQ but one is obviously much stricter than the other, which …