test(KVStore): Support Proxy's apply pattern on normal write and compact log #5623

CalvinNeo · 2022-08-15T07:48:58Z

Signed-off-by: CalvinNeo [email protected]

What problem does this PR solve?

Issue Number: ref #5170

Problem Summary:

MockRaftStoreProxy will act like TiFlash Proxy:

Functions like normalWrite/compactLog to generate and "commit" raft entries.
Functions like doApply to apply raft entries.
Functions like replay to replay raft entries from last applied index.

What is changed and how it works?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

ti-chi-bot · 2022-08-15T07:48:59Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

breezewish
solotzg

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

CalvinNeo · 2022-08-15T07:49:04Z

/hold

Signed-off-by: CalvinNeo <[email protected]>

JaySon-Huang · 2022-08-24T10:06:23Z

dbms/src/Debug/MockRaftStoreProxy.cpp

+        }
+    }
+
+    if (cond.fail_before_kvstore) return;


maybe use FailPointHelper and fiu_do_on instead of passing a FailCond to mock the failure?

I am afraid that failpoints take its effects on the whole program, by giving a explicit fail condition for every write can provide more flexibility in such tests.

Signed-off-by: CalvinNeo <[email protected]>

ti-chi-bot · 2022-09-26T08:07:53Z

@CalvinNeo: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

…tests

CalvinNeo · 2022-09-26T08:46:14Z

/merge

ti-chi-bot · 2022-09-26T08:46:15Z

@CalvinNeo: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-09-26T08:46:17Z

This pull request has been accepted and is ready to merge.

Commit hash: dd3ef16

CalvinNeo · 2022-09-26T08:46:21Z

/run-all-tests

CalvinNeo · 2022-09-26T08:52:59Z

/run-unit-test

CalvinNeo · 2022-09-26T08:54:49Z

/merge

ti-chi-bot · 2022-09-26T08:54:50Z

@CalvinNeo: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-09-26T08:54:51Z

This pull request has been accepted and is ready to merge.

Commit hash: 5289e03

Signed-off-by: CalvinNeo <[email protected]>

CalvinNeo · 2022-09-26T08:56:15Z

/merge

ti-chi-bot · 2022-09-26T08:56:16Z

@CalvinNeo: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-09-26T08:56:18Z

This pull request has been accepted and is ready to merge.

Commit hash: 5f1fb29

CalvinNeo · 2022-09-26T09:01:23Z

/rebuild

CalvinNeo · 2022-09-26T09:01:35Z

/run-integration-test

sre-bot · 2022-09-26T09:09:07Z

Coverage for changed files

Filename                                         Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Debug/MockRaftStoreProxy.cpp                         129                 4    96.90%          38                 0   100.00%         382                18    95.29%          76                 8    89.47%
Debug/MockRaftStoreProxy.h                             7                 0   100.00%           7                 0   100.00%          21                 0   100.00%           0                 0         -
Storages/Transaction/KVStore.h                         5                 3    40.00%           5                 3    40.00%           5                 3    40.00%           0                 0         -
Storages/Transaction/Region.cpp                      353                12    96.60%          61                 1    98.36%         639                27    95.77%         136                35    74.26%
Storages/Transaction/Region.h                         20                 2    90.00%          10                 2    80.00%          28                 2    92.86%          14                 4    71.43%
Storages/Transaction/RegionMeta.cpp                   95                 8    91.58%          44                 1    97.73%         298                24    91.95%          48                12    75.00%
Storages/Transaction/RegionMeta.h                      1                 0   100.00%           1                 0   100.00%           7                 0   100.00%           0                 0         -
Storages/Transaction/tests/gtest_kvstore.cpp         314                 7    97.77%          32                 2    93.75%        1450                38    97.38%           6                 1    83.33%
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                924                36    96.10%         198                 9    95.45%        2830               112    96.04%         280                60    78.57%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18357      7512             59.08%    214843  77246        64.05%

full coverage report (for internal network access only)

sre-bot · 2022-09-26T09:15:42Z

Coverage for changed files

Filename                                         Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Debug/MockRaftStoreProxy.cpp                         129                 4    96.90%          38                 0   100.00%         382                18    95.29%          76                 8    89.47%
Debug/MockRaftStoreProxy.h                             7                 0   100.00%           7                 0   100.00%          21                 0   100.00%           0                 0         -
Storages/Transaction/KVStore.h                         5                 3    40.00%           5                 3    40.00%           5                 3    40.00%           0                 0         -
Storages/Transaction/Region.cpp                      353                12    96.60%          61                 1    98.36%         639                27    95.77%         136                35    74.26%
Storages/Transaction/Region.h                         20                 2    90.00%          10                 2    80.00%          28                 2    92.86%          14                 4    71.43%
Storages/Transaction/RegionMeta.cpp                   95                 8    91.58%          44                 1    97.73%         298                24    91.95%          48                12    75.00%
Storages/Transaction/RegionMeta.h                      1                 0   100.00%           1                 0   100.00%           7                 0   100.00%           0                 0         -
Storages/Transaction/tests/gtest_kvstore.cpp         314                 7    97.77%          32                 2    93.75%        1450                38    97.38%           6                 1    83.33%
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                924                36    96.10%         198                 9    95.45%        2830               112    96.04%         280                60    78.57%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18357      7514             59.07%    214843  77228        64.05%

full coverage report (for internal network access only)

sre-bot · 2022-09-26T09:18:19Z

Coverage for changed files

Filename                                         Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Debug/MockRaftStoreProxy.cpp                         129                 4    96.90%          38                 0   100.00%         382                18    95.29%          76                 8    89.47%
Debug/MockRaftStoreProxy.h                             7                 0   100.00%           7                 0   100.00%          21                 0   100.00%           0                 0         -
Storages/Transaction/KVStore.h                         5                 3    40.00%           5                 3    40.00%           5                 3    40.00%           0                 0         -
Storages/Transaction/Region.cpp                      353                12    96.60%          61                 1    98.36%         639                27    95.77%         136                35    74.26%
Storages/Transaction/Region.h                         20                 2    90.00%          10                 2    80.00%          28                 2    92.86%          14                 4    71.43%
Storages/Transaction/RegionMeta.cpp                   95                 8    91.58%          44                 1    97.73%         298                24    91.95%          48                12    75.00%
Storages/Transaction/RegionMeta.h                      1                 0   100.00%           1                 0   100.00%           7                 0   100.00%           0                 0         -
Storages/Transaction/tests/gtest_kvstore.cpp         314                 7    97.77%          32                 2    93.75%        1450                38    97.38%           6                 1    83.33%
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                924                36    96.10%         198                 9    95.45%        2830               112    96.04%         280                60    78.57%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18357      7514             59.07%    214843  77272        64.03%

full coverage report (for internal network access only)

CalvinNeo · 2022-09-26T09:42:43Z

/run-integration-test

ti-chi-bot added do-not-merge/needs-linked-issue release-note-none Denotes a PR that doesn't merit a release note. labels Aug 15, 2022

ti-chi-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Aug 22, 2022

p1

0da5af9

Signed-off-by: CalvinNeo <[email protected]>

CalvinNeo force-pushed the enrich-tests branch from a9baa78 to 0da5af9 Compare August 24, 2022 08:11

ti-chi-bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Aug 24, 2022

JaySon-Huang reviewed Aug 24, 2022

View reviewed changes

f

0c6e8a3

Signed-off-by: CalvinNeo <[email protected]>

CalvinNeo force-pushed the enrich-tests branch 8 times, most recently from 33e98c8 to 61fd7d4 Compare August 25, 2022 09:01

f

b72ed93

Signed-off-by: CalvinNeo <[email protected]>

CalvinNeo force-pushed the enrich-tests branch from 61fd7d4 to b72ed93 Compare August 25, 2022 09:07

CalvinNeo added 4 commits August 30, 2022 13:41

f

e373692

Signed-off-by: CalvinNeo <[email protected]>

update

87d5be8

Signed-off-by: CalvinNeo <[email protected]>

f

fd34595

Signed-off-by: CalvinNeo <[email protected]>

f

c57c4d2

Signed-off-by: CalvinNeo <[email protected]>

ti-chi-bot added 2 commits September 26, 2022 15:13

Merge branch 'master' into enrich-tests

50866da

Merge branch 'master' into enrich-tests

6dec87c

Merge branch 'enrich-tests' of github.com:calvinneo/tics into enrich-…

dd3ef16

…tests

ti-chi-bot removed the status/can-merge Indicates a PR has been approved by a committer. label Sep 26, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Sep 26, 2022

ti-chi-bot removed the status/can-merge Indicates a PR has been approved by a committer. label Sep 26, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Sep 26, 2022

fix

5f1fb29

Signed-off-by: CalvinNeo <[email protected]>

CalvinNeo force-pushed the enrich-tests branch from 5289e03 to 5f1fb29 Compare September 26, 2022 08:56

ti-chi-bot removed the status/can-merge Indicates a PR has been approved by a committer. label Sep 26, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Sep 26, 2022

ti-chi-bot merged commit 3bdb49e into pingcap:master Sep 26, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test(KVStore): Support Proxy's apply pattern on normal write and compact log #5623

test(KVStore): Support Proxy's apply pattern on normal write and compact log #5623

CalvinNeo commented Aug 15, 2022 •

edited

Loading

ti-chi-bot commented Aug 15, 2022 •

edited

Loading

CalvinNeo commented Aug 15, 2022

JaySon-Huang Aug 24, 2022

CalvinNeo Sep 21, 2022

ti-chi-bot commented Sep 26, 2022

CalvinNeo commented Sep 26, 2022

ti-chi-bot commented Sep 26, 2022

ti-chi-bot commented Sep 26, 2022

CalvinNeo commented Sep 26, 2022

CalvinNeo commented Sep 26, 2022

CalvinNeo commented Sep 26, 2022

ti-chi-bot commented Sep 26, 2022

ti-chi-bot commented Sep 26, 2022

CalvinNeo commented Sep 26, 2022

ti-chi-bot commented Sep 26, 2022

ti-chi-bot commented Sep 26, 2022

CalvinNeo commented Sep 26, 2022

CalvinNeo commented Sep 26, 2022

sre-bot commented Sep 26, 2022

sre-bot commented Sep 26, 2022

sre-bot commented Sep 26, 2022

CalvinNeo commented Sep 26, 2022

test(KVStore): Support Proxy's apply pattern on normal write and compact log #5623

test(KVStore): Support Proxy's apply pattern on normal write and compact log #5623

Conversation

CalvinNeo commented Aug 15, 2022 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Aug 15, 2022 • edited Loading

CalvinNeo commented Aug 15, 2022

JaySon-Huang Aug 24, 2022

Choose a reason for hiding this comment

CalvinNeo Sep 21, 2022

Choose a reason for hiding this comment

ti-chi-bot commented Sep 26, 2022

CalvinNeo commented Sep 26, 2022

ti-chi-bot commented Sep 26, 2022

ti-chi-bot commented Sep 26, 2022

CalvinNeo commented Sep 26, 2022

CalvinNeo commented Sep 26, 2022

CalvinNeo commented Sep 26, 2022

ti-chi-bot commented Sep 26, 2022

ti-chi-bot commented Sep 26, 2022

CalvinNeo commented Sep 26, 2022

ti-chi-bot commented Sep 26, 2022

ti-chi-bot commented Sep 26, 2022

CalvinNeo commented Sep 26, 2022

CalvinNeo commented Sep 26, 2022

sre-bot commented Sep 26, 2022

Coverage for changed files

Coverage summary

sre-bot commented Sep 26, 2022

Coverage for changed files

Coverage summary

sre-bot commented Sep 26, 2022

Coverage for changed files

Coverage summary

CalvinNeo commented Sep 26, 2022

CalvinNeo commented Aug 15, 2022 •

edited

Loading

ti-chi-bot commented Aug 15, 2022 •

edited

Loading