storage: Support transform SST to multiple DTFile #5906

breezewish · 2022-09-16T02:34:34Z

Signed-off-by: Wish [email protected]

What problem does this PR solve?

Issue Number: ref #5237

This is one of a series of PRs for #5237, where you can see the overall roadmap.

What is changed and how it works?

Support transform SST into multiple DTFiles, split by rows or size.

Note: this is currently not used in the production.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

Signed-off-by: Wish <[email protected]>

ti-chi-bot · 2022-09-16T02:34:36Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

flowbehappy
lidezhu

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

Signed-off-by: Wish <[email protected]>

breezewish · 2022-09-16T04:00:57Z

/run-all-tests

breezewish · 2022-09-16T04:01:29Z

dbms/src/Storages/Transaction/ApplySnapshot.cpp

                bounded_stream,
                storage,
                schema_snap,
                snapshot_apply_method,
                job_type,
-                tmt);
+                /* split_after_rows */ 0,


For reviewers: Currently we set to 0 to not use it.

Signed-off-by: Wish <[email protected]>

…sh into wenxuan/sst_to_multiple_dt

sre-bot · 2022-09-16T04:24:31Z

Coverage for changed files

Filename                                         Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
DeltaMerge/SSTFilesToDTFilesOutputStream.cpp         124                40    67.74%           8                 1    87.50%         186                48    74.19%          54                24    55.56%
DeltaMerge/SSTFilesToDTFilesOutputStream.h             8                 0   100.00%           8                 0   100.00%          20                 0   100.00%           0                 0         -
DeltaMerge/tests/gtest_sst_files_stream.cpp           18                 1    94.44%           7                 0   100.00%          96                 0   100.00%           4                 1    75.00%
Transaction/ApplySnapshot.cpp                        295               161    45.42%          12                 0   100.00%         414               186    55.07%         106                71    33.02%
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                445               202    54.61%          35                 1    97.14%         716               234    67.32%         164                96    41.46%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18800      8103             56.90%    217960  83148        61.85%

full coverage report (for internal network access only)

lidezhu

LGTM

JaySon-Huang

Rest LGTM

dbms/src/Storages/DeltaMerge/SSTFilesToDTFilesOutputStream.h

dbms/src/Storages/Transaction/PartitionStreams.h

Signed-off-by: Wish <[email protected]>

JaySon-Huang

LGTM

breezewish · 2022-09-16T08:11:01Z

/merge

ti-chi-bot · 2022-09-16T08:11:02Z

@breezewish: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-09-16T08:11:03Z

This pull request has been accepted and is ready to merge.

Commit hash: 0d7ccea

breezewish · 2022-09-16T08:56:56Z

/merge

ti-chi-bot · 2022-09-16T08:56:58Z

@breezewish: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

sre-bot · 2022-09-16T09:20:33Z

Coverage for changed files

Filename                                         Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
DeltaMerge/SSTFilesToDTFilesOutputStream.cpp         124                40    67.74%           8                 1    87.50%         186                48    74.19%          54                24    55.56%
DeltaMerge/SSTFilesToDTFilesOutputStream.h             8                 0   100.00%           8                 0   100.00%          20                 0   100.00%           0                 0         -
DeltaMerge/tests/gtest_sst_files_stream.cpp           18                 1    94.44%           7                 0   100.00%          96                 0   100.00%           4                 1    75.00%
Transaction/ApplySnapshot.cpp                        295               161    45.42%          12                 0   100.00%         414               186    55.07%         106                71    33.02%
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                445               202    54.61%          35                 1    97.14%         716               234    67.32%         164                96    41.46%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18812      8112             56.88%    218130  83259        61.83%

full coverage report (for internal network access only)

sre-bot · 2022-09-16T09:26:49Z

Coverage for changed files

Filename                                         Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
DeltaMerge/SSTFilesToDTFilesOutputStream.cpp         124                40    67.74%           8                 1    87.50%         186                48    74.19%          54                24    55.56%
DeltaMerge/SSTFilesToDTFilesOutputStream.h             8                 0   100.00%           8                 0   100.00%          20                 0   100.00%           0                 0         -
DeltaMerge/tests/gtest_sst_files_stream.cpp           18                 1    94.44%           7                 0   100.00%          96                 0   100.00%           4                 1    75.00%
Transaction/ApplySnapshot.cpp                        295               161    45.42%          12                 0   100.00%         414               186    55.07%         106                71    33.02%
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                445               202    54.61%          35                 1    97.14%         716               234    67.32%         164                96    41.46%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18832      8110             56.94%    218680  83316        61.90%

full coverage report (for internal network access only)

breezewish · 2022-09-16T10:09:01Z

/merge

ti-chi-bot · 2022-09-16T10:09:02Z

@breezewish: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

sre-bot · 2022-09-16T10:38:24Z

Coverage for changed files

Filename                                         Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
DeltaMerge/SSTFilesToDTFilesOutputStream.cpp         124                40    67.74%           8                 1    87.50%         186                48    74.19%          54                24    55.56%
DeltaMerge/SSTFilesToDTFilesOutputStream.h             8                 0   100.00%           8                 0   100.00%          20                 0   100.00%           0                 0         -
DeltaMerge/tests/gtest_sst_files_stream.cpp           18                 1    94.44%           7                 0   100.00%          96                 0   100.00%           4                 1    75.00%
Transaction/ApplySnapshot.cpp                        295               161    45.42%          12                 0   100.00%         414               186    55.07%         106                71    33.02%
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                445               202    54.61%          35                 1    97.14%         716               234    67.32%         164                96    41.46%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18833      8103             56.97%    218702  83235        61.94%

full coverage report (for internal network access only)

breezewish · 2022-09-16T10:52:10Z

/merge

ti-chi-bot · 2022-09-16T10:52:11Z

@breezewish: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-09-16T10:52:26Z

@breezewish: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

sre-bot · 2022-09-16T11:14:33Z

Coverage for changed files

Filename                                         Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
DeltaMerge/SSTFilesToDTFilesOutputStream.cpp         124                40    67.74%           8                 1    87.50%         186                48    74.19%          54                24    55.56%
DeltaMerge/SSTFilesToDTFilesOutputStream.h             8                 0   100.00%           8                 0   100.00%          20                 0   100.00%           0                 0         -
DeltaMerge/tests/gtest_sst_files_stream.cpp           18                 1    94.44%           7                 0   100.00%          96                 0   100.00%           4                 1    75.00%
Transaction/ApplySnapshot.cpp                        295               161    45.42%          12                 0   100.00%         414               186    55.07%         106                71    33.02%
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                445               202    54.61%          35                 1    97.14%         716               234    67.32%         164                96    41.46%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18834      8104             56.97%    218764  83203        61.97%

full coverage report (for internal network access only)

storage: Support transform SST to multiple DTFile

c435562

Signed-off-by: Wish <[email protected]>

ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Sep 16, 2022

Add unit tests

342a0b2

Signed-off-by: Wish <[email protected]>

ti-chi-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Sep 16, 2022

Merge branch 'master' into wenxuan/sst_to_multiple_dt

d3161a5

breezewish commented Sep 16, 2022

View reviewed changes

Reformat

66a0424

Signed-off-by: Wish <[email protected]>

breezewish requested review from lidezhu, JaySon-Huang and flowbehappy September 16, 2022 04:03

breezewish mentioned this pull request Sep 16, 2022

Big region snapshot adaption and applying snapshot improvement #5237

Closed

Merge branch 'wenxuan/sst_to_multiple_dt' of github.com:pingcap/tifla…

1373e3b

…sh into wenxuan/sst_to_multiple_dt

lidezhu approved these changes Sep 16, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Sep 16, 2022

flowbehappy approved these changes Sep 16, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Sep 16, 2022

JaySon-Huang reviewed Sep 16, 2022

View reviewed changes

dbms/src/Storages/DeltaMerge/SSTFilesToDTFilesOutputStream.h Show resolved Hide resolved

dbms/src/Storages/Transaction/PartitionStreams.h Outdated Show resolved Hide resolved

Address comments

0d7ccea

Signed-off-by: Wish <[email protected]>

JaySon-Huang approved these changes Sep 16, 2022

View reviewed changes

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Sep 16, 2022

Merge branch 'master' into wenxuan/sst_to_multiple_dt

c1a971a

Merge branch 'master' into wenxuan/sst_to_multiple_dt

073b415

Merge branch 'master' into wenxuan/sst_to_multiple_dt

49bd040

Merge branch 'master' into wenxuan/sst_to_multiple_dt

f7cb8a9

ti-chi-bot merged commit befe835 into master Sep 16, 2022

JaySon-Huang deleted the wenxuan/sst_to_multiple_dt branch September 16, 2022 11:50

JaySon-Huang mentioned this pull request Jul 10, 2023

[release-6.1] Add file ingest check #7773

Closed

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

storage: Support transform SST to multiple DTFile #5906

storage: Support transform SST to multiple DTFile #5906

breezewish commented Sep 16, 2022 •

edited

Loading

ti-chi-bot commented Sep 16, 2022 •

edited

Loading

breezewish commented Sep 16, 2022

breezewish Sep 16, 2022

sre-bot commented Sep 16, 2022

lidezhu left a comment

JaySon-Huang left a comment

JaySon-Huang left a comment

breezewish commented Sep 16, 2022

ti-chi-bot commented Sep 16, 2022

ti-chi-bot commented Sep 16, 2022

breezewish commented Sep 16, 2022

ti-chi-bot commented Sep 16, 2022

sre-bot commented Sep 16, 2022

sre-bot commented Sep 16, 2022

breezewish commented Sep 16, 2022

ti-chi-bot commented Sep 16, 2022

sre-bot commented Sep 16, 2022

breezewish commented Sep 16, 2022

ti-chi-bot commented Sep 16, 2022

ti-chi-bot commented Sep 16, 2022

sre-bot commented Sep 16, 2022

storage: Support transform SST to multiple DTFile #5906

storage: Support transform SST to multiple DTFile #5906

Conversation

breezewish commented Sep 16, 2022 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Sep 16, 2022 • edited Loading

breezewish commented Sep 16, 2022

breezewish Sep 16, 2022

Choose a reason for hiding this comment

sre-bot commented Sep 16, 2022

Coverage for changed files

Coverage summary

lidezhu left a comment

Choose a reason for hiding this comment

JaySon-Huang left a comment

Choose a reason for hiding this comment

JaySon-Huang left a comment

Choose a reason for hiding this comment

breezewish commented Sep 16, 2022

ti-chi-bot commented Sep 16, 2022

ti-chi-bot commented Sep 16, 2022

breezewish commented Sep 16, 2022

ti-chi-bot commented Sep 16, 2022

sre-bot commented Sep 16, 2022

Coverage for changed files

Coverage summary

sre-bot commented Sep 16, 2022

Coverage for changed files

Coverage summary

breezewish commented Sep 16, 2022

ti-chi-bot commented Sep 16, 2022

sre-bot commented Sep 16, 2022

Coverage for changed files

Coverage summary

breezewish commented Sep 16, 2022

ti-chi-bot commented Sep 16, 2022

ti-chi-bot commented Sep 16, 2022

sre-bot commented Sep 16, 2022

Coverage for changed files

Coverage summary

breezewish commented Sep 16, 2022 •

edited

Loading

ti-chi-bot commented Sep 16, 2022 •

edited

Loading