TiFlash failed to replicate data from TiKV in the master branch

[JaySon-Huang]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

3. What did you see instead (Required)

https://do.pingcap.net/jenkins/blue/organizations/jenkins/pingcap%2Ftiflash%2Fpull_integration_test/detail/pull_integration_test/492/pipeline/372

In the CI intergrate testing, tiflash failed to replica data from TiKV, with error logging like

[2024/06/14 11:44:39.043 +08:00] [ERROR] [kv.rs:774] ["dispatch raft msg from gRPC to raftstore fail"] [err=Grpc(Codec(WireError(InvalidEnumValue(15))))] [thread_id=0x5]
[2024/06/14 11:44:39.043 +08:00] [ERROR] [kv.rs:781] ["KvService::batch_raft send response fail"] [err=RemoteStopped] [thread_id=0x5]
[2024/06/14 11:44:44.046 +08:00] [INFO] [kv.rs:737] ["batch_raft RPC is called, new gRPC stream established"] [source_store_id=Some(1)] [thread_id=0x5]
[2024/06/14 11:44:44.047 +08:00] [ERROR] [kv.rs:774] ["dispatch raft msg from gRPC to raftstore fail"] [err=Grpc(Codec(WireError(InvalidEnumValue(15))))] [thread_id=0x5]
[2024/06/14 11:44:44.047 +08:00] [ERROR] [kv.rs:781] ["KvService::batch_raft send response fail"] [err=RemoteStopped] [thread_id=0x5]
[2024/06/14 11:44:49.054 +08:00] [INFO] [kv.rs:737] ["batch_raft RPC is called, new gRPC stream established"] [source_store_id=Some(1)] [thread_id=0x5]

It's because tikv introduce a new precheck process before snapshot generation tikv/tikv#17019 for
resolving issue tikv/tikv#15972. While tiflash-proxy does not updated and don't recognize the new type of raft message.

4. What is your TiFlash version? (Required)

master

[Yui-Song]

@CalvinNeo Do we have UT for replicating data from TiKV? From my point of view, it's a very basic feature.

[CalvinNeo]

@CalvinNeo Do we have UT for replicating data from TiKV? From my point of view, it's a very basic feature.

We have ut in both tiflash and tiflash proxy. Check https://github.com/pingcap/tidb-engine-ext/tree/raftstore-proxy/proxy_tests for example.
However, this is not a case that a UT should cover, because the mutation happened in TiKV.

[seiya-annie]

/found customer

[JaySon-Huang]

/remove-label found/customer

[ti-chi-bot]

@JaySon-Huang: The label(s) found/customer cannot be applied. These labels are supported: tide/merge-method-rebase, tide/merge-method-squash, tide/merge-method-merge, duplicate, first-time-contributor, good first issue, ok-to-test, needs-ok-to-test, help wanted, invalid, question, wontfix, make-local-great-again, needs-cherry-pick-release-5.4, needs-cherry-pick-release-6.1, needs-cherry-pick-release-6.5, needs-cherry-pick-release-7.1, needs-cherry-pick-release-7.5, needs-cherry-pick-release-8.1, affects-5.4, affects-6.1, affects-6.5, affects-7.1, affects-7.5, affects-8.1, may-affects-5.4, may-affects-6.1, may-affects-6.5, may-affects-7.1, may-affects-7.5, may-affects-8.1, needs-rebase.

In response to this:

/remove-label found/customer

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.