testkit: add a WithTiKV flag to support run unit test on real TiKV

[tiancaiamao]

What problem does this PR solve?

Issue Number: close #35646

Problem Summary:

Add a -with-tikv flag to the unit test.

What is changed and how it works?

After this change, I can run any test case with a real TiKV, for example:

cd executor
go test -run TestFix31530 -with-tikv 127.0.0.1:2379

I can verify out test cases with real tikv. Here is how I do that:

1. Create a docker image:

cat Dockerfile
FROM buildpack-deps:focal as base

COPY pd-server /bin/
COPY tikv-server /bin/
COPY start.sh /

ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8

FROM base AS branch-version-1

cat start.sh
#!/bin/bash
set -eu

/bin/pd-server --client-urls http://0.0.0.0:2379 --data-dir "/data/pd" &
/bin/tikv-server --status-addr 0.0.0.0:20180  --pd 0.0.0.0:2379 -A 127.0.0.1:20160 -C "/conf/tikv.toml" -s "/data/tikv"

2. build the image

docker build -t tikv-img .

3. Run the tikv in container

docker run --net=host --rm --name tmp1 tikv-img:bootstrap /start.sh

--net use the local port, --rm means remove the container after the test is done.

4. Run unit test with the tikv docker container

For example:

go test -run TestFix31530 -with-tikv 127.0.0.1:2379

5. Automate the test script

I modify the make ut script to automate the tests.
Create the tikv docker container before test and cleanup them.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• bb7133
• xhebox

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[hawkingrei]

In fact, @tisonkun has moved the real-tikv test case on here. If you would like to run the real-tikv test case. all are under the same folder.

[tiancaiamao]

The background is, I need to test the paging protocol with the real tikv.
So I need rewrite all the test case or move them all?
A more reasonable solution is just add a flag ... if you don't use it, it doesn't change anything.

[tisonkun]

This patch just add a flag, which won't affect CI settings. The reason I move all with real tikv tests to a dedicated test is for tagging tests in CI env.

[tisonkun]

@tiancaiamao would you like to change CI logic? Or adding these code for conveniently running tests with real tikv locally?

[tisonkun]

Also, with-tikv is a string flag while its help info said "whether tests run with real TiKV", which is confusing...

Perhaps

// This flag is only used for debugging locally with real tikv cluster.
var tikvAddr = flag.String("tikv-addr", "", "address of tikv cluster, if set, running test with real tikv cluster")

[tiancaiamao]

Also, with-tikv is a string flag while its help info said "whether tests run with real TiKV", which is confusing...

Perhaps

// This flag is only used for debugging locally with real tikv cluster.
var tikvAddr = flag.String("tikv-addr", "", "address of tikv cluster, if set, running test with real tikv cluster")

Updated.

[tiancaiamao]

@tiancaiamao would you like to change CI logic? Or adding these code for conveniently running tests with real tikv locally?

If you want to see the full changes, it's in my branch https://github.com/tiancaiamao/tidb/tree/test-with-tikv
But not all the test cases can run under TiKV, I check and document the result here:

https://pingcap.feishu.cn/docx/doxcnEzyRXsTdCJ2EE6EZwIP2oe

I'm not adding a new CI pipeline, I just make the code base more friendly for those who want to run some test cases with a real TiKV.

[hawkingrei]

/run-all-tests

[bb7133]

LGTM

[hawkingrei]

If the WithTikv is added in the testkit, We will run the with-tikv tests with the all tidb repo. so we have to build all the test cases to run them. It will take more time to build it. so I suggest the with-tikv cases in the specific folders. so It can reduce the objects to be compiled.

[hawkingrei]

@tiancaiamao Please gofmt the tools/check/ut.go. I find it is useless for ut.go when gofmt is run in the root folder.

[sre-bot]

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/d03ba526a464a86146ba548ed1ebf6714b8cb070

[tiancaiamao]

/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: d03ba52

[tiancaiamao]

/run-check_dev_2

[sre-bot]

TiDB MergeCI notify

✅ Well Done! New fixed [1] after this pr merged.

CI Name	Result	Duration	Compare with Parent commit
idc-jenkins-ci-tidb/integration-common-test	✅ all 11 tests passed	21 min	Fixed
idc-jenkins-ci/integration-cdc-test	🟢 all 35 tests passed	29 min	Existing passed
idc-jenkins-ci-tidb/common-test	🟢 all 12 tests passed	13 min	Existing passed
idc-jenkins-ci-tidb/tics-test	🟢 all 1 tests passed	10 min	Existing passed
idc-jenkins-ci-tidb/integration-ddl-test	🟢 all 6 tests passed	6 min 23 sec	Existing passed
idc-jenkins-ci-tidb/sqllogic-test-2	🟢 all 28 tests passed	5 min 38 sec	Existing passed
idc-jenkins-ci-tidb/sqllogic-test-1	🟢 all 26 tests passed	5 min 4 sec	Existing passed
idc-jenkins-ci-tidb/integration-compatibility-test	🟢 all 1 tests passed	3 min 17 sec	Existing passed
idc-jenkins-ci-tidb/mybatis-test	🟢 all 1 tests passed	2 min 59 sec	Existing passed
idc-jenkins-ci-tidb/plugin-test	🟢 build success, plugin test success	4min	Existing passed