[flay test] test_rawkv::test_leader_transfer

[tonyxuqqi]

https://ci.pingcap.net/blue/organizations/jenkins/tikv_ghpr_test/detail/tikv_ghpr_test/43640/tests/

Stacktrace
thread 'cases::test_rawkv::test_leader_transfer' panicked at '', components/test_raftstore/src/util.rs:1422:5
stack backtrace:
0: rust_begin_unwind
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/panicking.rs:575:5
1: core::panicking::panic_fmt
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/core/src/panicking.rs:65:14
2: test_raftstore::util::must_raw_put
3: failpoints::cases::test_rawkv::TestSuite::must_raw_put
at /home/jenkins/tikv-src/tests/failpoints/cases/test_rawkv.rs:65:9
4: failpoints::cases::test_rawkv::test_leader_transfer
at /home/jenkins/tikv-src/tests/failpoints/cases/test_rawkv.rs:196:9
5: failpoints::cases::test_rawkv::test_leader_transfer::{{closure}}
at /home/jenkins/tikv-src/tests/failpoints/cases/test_rawkv.rs:153:27
6: core::ops::function::FnOnce::call_once
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/core/src/ops/function.rs:513:5
7: test_util::runner::run_test_with_hook::{{closure}}::{{closure}}
8: core::ops::function::FnOnce::call_once{{vtable.shim}}
9: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/alloc/src/boxed.rs:2000:9
note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace.

[pingyu]

'cases::test_rawkv::test_leader_transfer' panicked at ''

It's strange that must_raw_put met a region error but it was empty. As there is no tikv logs in CI, currently I don't know what is the error and where it raised.

Maybe we should try to package tikv logs in CI first.

[pingyu]

After add some debug logs in future_raw_put:

    async move {
        let v = match res {
            Err(e) => {
                error!("[debug] raw_put"; "e" => ?e);
                Err(e)
            }
            Ok(_) => f.await?,
        };
        info!("[debug] raw_put"; "v" => ?v);
        let mut resp = RawPutResponse::default();
        if let Some(err) = extract_region_error(&v) {
            error!("[debug] raw_put"; "err" => ?err);
            resp.set_region_error(err);
        } else if let Err(e) = v {
            resp.set_error(format!("{}", e));
        }
        Ok(resp)
    }

It shows that it's a error of MaxTimestampNotSynced:

test_region_merge-0 2024/04/13 17:39:57.625 kv.rs:1926: [INFO] [debug] raw_put, v: Err(Error(Txn(Error(MaxTimestampNotSynced { region_id: 1004, start_ts: TimeStamp(0) })))), thread_id: 863
test_region_merge-0 2024/04/13 17:39:57.625 kv.rs:1929: [ERRO] [debug] raw_put, err: , thread_id: 863