bug: hummock panic at pinning an older version

[zwang28]

Describe the bug

Namespace: longevity-20230118-170814
Pod: risingwave-compute-0, risingwave-compute-1, risingwave-compute-2
image: nightly-20230117

thread 'risingwave-main' panicked at 'pinning a older version 30232. Current is 31412', 

- risingwave_storage::hummock::local_version::pinned_version::PinnedVersion::new_pin_version::hc5588fd26f62bf21
      at /risingwave/src/storage/src/hummock/local_version/pinned_version.rs:102:9
- risingwave_storage::hummock::event_handler::hummock_event_handler::HummockEventHandler::handle_version_update::hc29f24e22ccdfb17
      at /risingwave/src/storage/src/hummock/event_handler/hummock_event_handler.rs:382:34
- risingwave_storage::hummock::event_handler::hummock_event_handler::HummockEventHandler::start_hummock_event_handler_worker::{{closure}}::ha4ac2dea3a0fa16c
      at /risingwave/src/storage/src/hummock/event_handler/hummock_event_handler.rs:450:29

compute-0.csv
compute-1.csv
compute-2.csv

To Reproduce

No response

Expected behavior

No response

Additional context

No response

[hzxa21]

I saw notification error right before the panic:

2023-01-18T17:55:47.646027Z ERROR risingwave_common_service::observer_manager: Receives meta's notification err Status { code: Unknown, message: \""error reading a body from connection: broken pipe\"", source: Some(hyper::Error(Body, Error { kind: Io(Kind(BrokenPipe)) })) }

I suspect there is something wrong with meta. @zwang28 Do you by any chance save the meta log?

[zwang28]

Do you by any chance save the meta log

No.

I suspect there is something wrong with meta

I have the same feeling.

Recently there was a fix in the way k8s managing meta node restart. In short, previously when k8s is to restart a meta node, it starts a new one first, and then kill the old one, which can lead to "read an old meta state version" like this issue.

Unfortunately the cloud log is expired too, we cannot tell whether there is any "k8s is to restart a meta node" when this issue happens.

[hzxa21]

Do you by any chance save the meta log

No.

I suspect there is something wrong with meta

I have the same feeling.

Recently there was a fix in the way k8s managing meta node restart. In short, previously when k8s is to restart a meta node, it starts a new one first, and then kill the old one, which can lead to "read an old meta state version" like this issue.

Unfortunately the cloud log is expired too, we cannot tell whether there is any "k8s is to restart a meta node" when this issue happens.

Sad. Let's wait for a while to see whether this issue still exists.

[hzxa21]

We haven't seen this issue for two weeks. Let's close the issue for now. Feel free to reopen if needed.