Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

statistics: implement batch size for auto analyze partition table in dynamic mode #38153

Merged
merged 3 commits into from
Sep 27, 2022
Merged

statistics: implement batch size for auto analyze partition table in dynamic mode #38153

merged 3 commits into from
Sep 27, 2022

Conversation

Yisaer
Copy link
Contributor

@Yisaer Yisaer commented Sep 26, 2022 •
edited
Loading

Signed-off-by: yisaer [email protected]

What problem does this PR solve?

Issue Number: ref #37977

Problem Summary:

What is changed and how it works?

implement batch size for auto analyze partition table in dynamic mode

Check List

Tests

  • Unit test
  • Manual test (add detailed scripts or steps below)

The result of auto analyze table with 4 partitions with different batch size is like following:

// batch size = 1
[2022/09/26 11:28:23.052 +08:00] [INFO] [update.go:1200] ["[stats] auto analyze triggered"] [table=t] [partitions="[\"p0\"]"]
[2022/09/26 11:28:23.113 +08:00] [INFO] [handle.go:1299] ["[stats] incrementally update modifyCount"] [tableID=78] [curModifyCnt=2500] [results.BaseModifyCnt=2500] [modifyCount=0]
[2022/09/26 11:28:23.113 +08:00] [INFO] [handle.go:1321] ["[stats] directly update count"] [tableID=78] [results.Count=7500] [count=7500]
[2022/09/26 11:28:23.125 +08:00] [INFO] [analyze.go:429] ["analyze table `test`.`t` has finished"] [partition=p0] ["job info"="auto analyze table all columns with 256 buckets, 500 topn, 1 samplerate"] ["start time"=2022/09/26 11:28:23.055 +08:00] ["end time"=2022/09/26 11:28:23.125 +08:00] [cost=70.046695ms]
[2022/09/26 11:28:23.187 +08:00] [INFO] [update.go:1200] ["[stats] auto analyze triggered"] [table=t] [partitions="[\"p1\"]"]
[2022/09/26 11:28:23.264 +08:00] [INFO] [handle.go:1299] ["[stats] incrementally update modifyCount"] [tableID=79] [curModifyCnt=2500] [results.BaseModifyCnt=2500] [modifyCount=0]
[2022/09/26 11:28:23.264 +08:00] [INFO] [handle.go:1321] ["[stats] directly update count"] [tableID=79] [results.Count=7500] [count=7500]
[2022/09/26 11:28:23.279 +08:00] [INFO] [analyze.go:429] ["analyze table `test`.`t` has finished"] [partition=p1] ["job info"="auto analyze table all columns with 256 buckets, 500 topn, 1 samplerate"] ["start time"=2022/09/26 11:28:23.187 +08:00] ["end time"=2022/09/26 11:28:23.278 +08:00] [cost=90.894112ms]
[2022/09/26 11:28:23.346 +08:00] [INFO] [update.go:1200] ["[stats] auto analyze triggered"] [table=t] [partitions="[\"p2\"]"]
[2022/09/26 11:28:23.396 +08:00] [INFO] [handle.go:1299] ["[stats] incrementally update modifyCount"] [tableID=80] [curModifyCnt=2500] [results.BaseModifyCnt=2500] [modifyCount=0]
[2022/09/26 11:28:23.396 +08:00] [INFO] [handle.go:1321] ["[stats] directly update count"] [tableID=80] [results.Count=7500] [count=7500]
[2022/09/26 11:28:23.409 +08:00] [INFO] [analyze.go:429] ["analyze table `test`.`t` has finished"] [partition=p2] ["job info"="auto analyze table all columns with 256 buckets, 500 topn, 1 samplerate"] ["start time"=2022/09/26 11:28:23.347 +08:00] ["end time"=2022/09/26 11:28:23.408 +08:00] [cost=60.617167ms]
[2022/09/26 11:28:23.462 +08:00] [INFO] [update.go:1200] ["[stats] auto analyze triggered"] [table=t] [partitions="[\"p3\"]"]
[2022/09/26 11:28:23.512 +08:00] [INFO] [handle.go:1299] ["[stats] incrementally update modifyCount"] [tableID=81] [curModifyCnt=2500] [results.BaseModifyCnt=2500] [modifyCount=0]
[2022/09/26 11:28:23.512 +08:00] [INFO] [handle.go:1321] ["[stats] directly update count"] [tableID=81] [results.Count=7500] [count=7500]
[2022/09/26 11:28:23.521 +08:00] [INFO] [analyze.go:429] ["analyze table `test`.`t` has finished"] [partition=p3] ["job info"="auto analyze table all columns with 256 buckets, 500 topn, 1 samplerate"] ["start time"=2022/09/26 11:28:23.463 +08:00] ["end time"=2022/09/26 11:28:23.521 +08:00] [cost=57.70313ms]

// batch size = 2
[2022/09/26 12:06:51.388 +08:00] [INFO] [update.go:1199] ["[stats] auto analyze triggered"] [table=t] [partitions="[\"p0\",\"p1\"]"]
[2022/09/26 12:06:51.625 +08:00] [INFO] [handle.go:1299] ["[stats] incrementally update modifyCount"] [tableID=78] [curModifyCnt=2500] [results.BaseModifyCnt=2500] [modifyCount=0]
[2022/09/26 12:06:51.625 +08:00] [INFO] [handle.go:1321] ["[stats] directly update count"] [tableID=78] [results.Count=20000] [count=20000]
[2022/09/26 12:06:51.637 +08:00] [INFO] [analyze.go:429] ["analyze table `test`.`t` has finished"] [partition=p0] ["job info"="auto analyze table all columns with 256 buckets, 500 topn, 1 samplerate"] ["start time"=2022/09/26 12:06:51.391 +08:00] ["end time"=2022/09/26 12:06:51.636 +08:00] [cost=244.760351ms]
[2022/09/26 12:06:51.637 +08:00] [INFO] [handle.go:1299] ["[stats] incrementally update modifyCount"] [tableID=79] [curModifyCnt=2500] [results.BaseModifyCnt=2500] [modifyCount=0]
[2022/09/26 12:06:51.637 +08:00] [INFO] [handle.go:1321] ["[stats] directly update count"] [tableID=79] [results.Count=20000] [count=20000]
[2022/09/26 12:06:51.649 +08:00] [INFO] [analyze.go:429] ["analyze table `test`.`t` has finished"] [partition=p1] ["job info"="auto analyze table all columns with 256 buckets, 500 topn, 1 samplerate"] ["start time"=2022/09/26 12:06:51.510 +08:00] ["end time"=2022/09/26 12:06:51.648 +08:00] [cost=138.490648ms]
[2022/09/26 12:06:51.710 +08:00] [INFO] [update.go:1199] ["[stats] auto analyze triggered"] [table=t] [partitions="[\"p2\",\"p3\"]"]
[2022/09/26 12:06:51.931 +08:00] [INFO] [handle.go:1299] ["[stats] incrementally update modifyCount"] [tableID=80] [curModifyCnt=2500] [results.BaseModifyCnt=2500] [modifyCount=0]
[2022/09/26 12:06:51.931 +08:00] [INFO] [handle.go:1321] ["[stats] directly update count"] [tableID=80] [results.Count=20000] [count=20000]
[2022/09/26 12:06:51.942 +08:00] [INFO] [analyze.go:429] ["analyze table `test`.`t` has finished"] [partition=p2] ["job info"="auto analyze table all columns with 256 buckets, 500 topn, 1 samplerate"] ["start time"=2022/09/26 12:06:51.711 +08:00] ["end time"=2022/09/26 12:06:51.941 +08:00] [cost=229.815655ms]
[2022/09/26 12:06:51.942 +08:00] [INFO] [handle.go:1299] ["[stats] incrementally update modifyCount"] [tableID=81] [curModifyCnt=2500] [results.BaseModifyCnt=2500] [modifyCount=0]
[2022/09/26 12:06:51.942 +08:00] [INFO] [handle.go:1321] ["[stats] directly update count"] [tableID=81] [results.Count=20000] [count=20000]
[2022/09/26 12:06:51.953 +08:00] [INFO] [analyze.go:429] ["analyze table `test`.`t` has finished"] [partition=p3] ["job info"="auto analyze table all columns with 256 buckets, 500 topn, 1 samplerate"] ["start time"=2022/09/26 12:06:51.823 +08:00] ["end time"=2022/09/26 12:06:51.953 +08:00] [cost=129.757718ms]


// batch size = 3
[2022/09/26 12:04:51.385 +08:00] [INFO] [update.go:1199] ["[stats] auto analyze triggered"] [table=t] [partitions="[\"p0\",\"p1\",\"p2\"]"]
[2022/09/26 12:04:51.723 +08:00] [INFO] [handle.go:1299] ["[stats] incrementally update modifyCount"] [tableID=78] [curModifyCnt=2500] [results.BaseModifyCnt=2500] [modifyCount=0]
[2022/09/26 12:04:51.723 +08:00] [INFO] [handle.go:1321] ["[stats] directly update count"] [tableID=78] [results.Count=17500] [count=17500]
[2022/09/26 12:04:51.734 +08:00] [INFO] [analyze.go:429] ["analyze table `test`.`t` has finished"] [partition=p0] ["job info"="auto analyze table all columns with 256 buckets, 500 topn, 1 samplerate"] ["start time"=2022/09/26 12:04:51.388 +08:00] ["end time"=2022/09/26 12:04:51.734 +08:00] [cost=345.96014ms]
[2022/09/26 12:04:51.734 +08:00] [INFO] [handle.go:1299] ["[stats] incrementally update modifyCount"] [tableID=79] [curModifyCnt=2500] [results.BaseModifyCnt=2500] [modifyCount=0]
[2022/09/26 12:04:51.734 +08:00] [INFO] [handle.go:1321] ["[stats] directly update count"] [tableID=79] [results.Count=17500] [count=17500]
[2022/09/26 12:04:51.748 +08:00] [INFO] [analyze.go:429] ["analyze table `test`.`t` has finished"] [partition=p1] ["job info"="auto analyze table all columns with 256 buckets, 500 topn, 1 samplerate"] ["start time"=2022/09/26 12:04:51.497 +08:00] ["end time"=2022/09/26 12:04:51.748 +08:00] [cost=251.003429ms]
[2022/09/26 12:04:51.749 +08:00] [INFO] [handle.go:1299] ["[stats] incrementally update modifyCount"] [tableID=80] [curModifyCnt=2500] [results.BaseModifyCnt=2500] [modifyCount=0]
[2022/09/26 12:04:51.749 +08:00] [INFO] [handle.go:1321] ["[stats] directly update count"] [tableID=80] [results.Count=17500] [count=17500]
[2022/09/26 12:04:51.761 +08:00] [INFO] [analyze.go:429] ["analyze table `test`.`t` has finished"] [partition=p2] ["job info"="auto analyze table all columns with 256 buckets, 500 topn, 1 samplerate"] ["start time"=2022/09/26 12:04:51.614 +08:00] ["end time"=2022/09/26 12:04:51.760 +08:00] [cost=146.255322ms]
[2022/09/26 12:04:51.828 +08:00] [INFO] [update.go:1199] ["[stats] auto analyze triggered"] [table=t] [partitions="[\"p3\"]"]
[2022/09/26 12:04:51.929 +08:00] [INFO] [handle.go:1299] ["[stats] incrementally update modifyCount"] [tableID=81] [curModifyCnt=2500] [results.BaseModifyCnt=2500] [modifyCount=0]
[2022/09/26 12:04:51.929 +08:00] [INFO] [handle.go:1321] ["[stats] directly update count"] [tableID=81] [results.Count=17500] [count=17500]
[2022/09/26 12:04:51.938 +08:00] [INFO] [analyze.go:429] ["analyze table `test`.`t` has finished"] [partition=p3] ["job info"="auto analyze table all columns with 256 buckets, 500 topn, 1 samplerate"] ["start time"=2022/09/26 12:04:51.829 +08:00] ["end time"=2022/09/26 12:04:51.938 +08:00] [cost=109.006329ms]

// batch size = 4
[2022/09/26 12:19:34.934 +08:00] [INFO] [update.go:1188] ["[stats] start to auto analyze"] [table=t] [partitions="[\"p0\",\"p1\",\"p2\",\"p3\"]"] ["analyze partition batch size"=4]
[2022/09/26 12:19:34.934 +08:00] [INFO] [update.go:1203] ["[stats] auto analyze triggered"] [table=t] [partitions="[\"p0\",\"p1\",\"p2\",\"p3\"]"]
[2022/09/26 12:19:35.735 +08:00] [INFO] [handle.go:1299] ["[stats] incrementally update modifyCount"] [tableID=78] [curModifyCnt=2500] [results.BaseModifyCnt=2500] [modifyCount=0]
[2022/09/26 12:19:35.736 +08:00] [INFO] [handle.go:1321] ["[stats] directly update count"] [tableID=78] [results.Count=27500] [count=27500]
[2022/09/26 12:19:35.748 +08:00] [INFO] [analyze.go:429] ["analyze table `test`.`t` has finished"] [partition=p0] ["job info"="auto analyze table all columns with 256 buckets, 500 topn, 1 samplerate"] ["start time"=2022/09/26 12:19:34.937 +08:00] ["end time"=2022/09/26 12:19:35.748 +08:00] [cost=811.235411ms]
[2022/09/26 12:19:35.748 +08:00] [INFO] [handle.go:1299] ["[stats] incrementally update modifyCount"] [tableID=79] [curModifyCnt=2500] [results.BaseModifyCnt=2500] [modifyCount=0]
[2022/09/26 12:19:35.748 +08:00] [INFO] [handle.go:1321] ["[stats] directly update count"] [tableID=79] [results.Count=27500] [count=27500]
[2022/09/26 12:19:35.760 +08:00] [INFO] [analyze.go:429] ["analyze table `test`.`t` has finished"] [partition=p1] ["job info"="auto analyze table all columns with 256 buckets, 500 topn, 1 samplerate"] ["start time"=2022/09/26 12:19:35.130 +08:00] ["end time"=2022/09/26 12:19:35.760 +08:00] [cost=629.353255ms]
[2022/09/26 12:19:35.761 +08:00] [INFO] [handle.go:1299] ["[stats] incrementally update modifyCount"] [tableID=80] [curModifyCnt=2500] [results.BaseModifyCnt=2500] [modifyCount=0]
[2022/09/26 12:19:35.761 +08:00] [INFO] [handle.go:1321] ["[stats] directly update count"] [tableID=80] [results.Count=27500] [count=27500]
[2022/09/26 12:19:35.771 +08:00] [INFO] [analyze.go:429] ["analyze table `test`.`t` has finished"] [partition=p2] ["job info"="auto analyze table all columns with 256 buckets, 500 topn, 1 samplerate"] ["start time"=2022/09/26 12:19:35.307 +08:00] ["end time"=2022/09/26 12:19:35.770 +08:00] [cost=463.237294ms]
[2022/09/26 12:19:35.771 +08:00] [INFO] [handle.go:1299] ["[stats] incrementally update modifyCount"] [tableID=81] [curModifyCnt=2500] [results.BaseModifyCnt=2500] [modifyCount=0]
[2022/09/26 12:19:35.771 +08:00] [INFO] [handle.go:1321] ["[stats] directly update count"] [tableID=81] [results.Count=27500] [count=27500]
[2022/09/26 12:19:35.782 +08:00] [INFO] [analyze.go:429] ["analyze table `test`.`t` has finished"] [partition=p3] ["job info"="auto analyze table all columns with 256 buckets, 500 topn, 1 samplerate"] ["start time"=2022/09/26 12:19:35.496 +08:00] ["end time"=2022/09/26 12:19:35.781 +08:00] [cost=285.375283ms]

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

@Yisaer
impl
86cfa3d 
Signed-off-by: yisaer <[email protected]>
@Yisaer Yisaer requested a review from a team as a code owner September 26, 2022 04:31
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Sep 26, 2022 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • qw4990
  • xuyifangreeneyes

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Sep 26, 2022
@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Sep 26, 2022
@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Sep 27, 2022
@Yisaer
address the comment
3e1cd9c 
Signed-off-by: yisaer <[email protected]>
@Yisaer
Copy link
Contributor Author

Yisaer commented Sep 27, 2022

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 3e1cd9c

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Sep 27, 2022
@ti-chi-bot ti-chi-bot merged commit 47739ff into pingcap:master Sep 27, 2022
@sre-bot
Copy link
Contributor

sre-bot commented Sep 27, 2022

TiDB MergeCI notify

CI Name Result Duration Compare with Parent commit
idc-jenkins-ci-tidb/integration-ddl-test 🟢 all 6 tests passed 31 min Existing passed
idc-jenkins-ci/integration-cdc-test 🟢 all 37 tests passed 29 min Existing passed
idc-jenkins-ci-tidb/integration-common-test 🟢 all 17 tests passed 16 min Existing passed
idc-jenkins-ci-tidb/common-test 🟢 all 11 tests passed 10 min Existing passed
idc-jenkins-ci-tidb/tics-test 🟢 all 1 tests passed 5 min 59 sec Existing passed
idc-jenkins-ci-tidb/sqllogic-test-2 🟢 all 28 tests passed 5 min 58 sec Existing passed
idc-jenkins-ci-tidb/sqllogic-test-1 🟢 all 26 tests passed 5 min 47 sec Existing passed
idc-jenkins-ci-tidb/mybatis-test 🟢 all 1 tests passed 3 min 56 sec Existing passed
idc-jenkins-ci-tidb/integration-compatibility-test 🟢 all 1 tests passed 3 min 41 sec Existing passed
idc-jenkins-ci-tidb/plugin-test 🟢 build success, plugin test success 4min Existing passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xuyifangreeneyes xuyifangreeneyes xuyifangreeneyes approved these changes

@qw4990 qw4990 qw4990 approved these changes

Assignees
No one assigned
Labels
release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@Yisaer @ti-chi-bot @sre-bot @qw4990 @xuyifangreeneyes