Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

schemaStorage: reduce memory usage and add unit tests #1115

Merged
merged 9 commits into from
Nov 26, 2020
Merged

schemaStorage: reduce memory usage and add unit tests #1115

merged 9 commits into from
Nov 26, 2020

Conversation

zier-one
Copy link
Contributor

@zier-one zier-one commented Nov 24, 2020 •
edited
Loading

What problem does this PR solve?

fix #1098

  • reduce memory usage in schema storage by avoiding table info replication
  • add many unit tests

500 table created:
before this PR:
image

after this PR:
image

Check List

Tests

  • Unit test
  • Integration test

Release note

@zier-one zier-one added this to the v4.0.9 milestone Nov 25, 2020
@zier-one zier-one changed the title refine table schema schemaStorage: reduce memory usage and add unit tests Nov 25, 2020
@zier-one zier-one added status/ptal Could you please take a look? needs-cherry-pick-release-4.0 Should cherry pick this PR to release-4.0 branch. labels Nov 25, 2020
@zier-one zier-one marked this pull request as ready for review November 25, 2020 08:30
@amyangfei
Copy link
Contributor

/run-integration-tests

@zhangjinpeng87
Copy link
Contributor

How about 10K tables? Maybe we should load the table schema lazily? Using the LRU strategy?

@zier-one
Copy link
Contributor Author

zier-one commented Nov 25, 2020 •
edited
Loading

How about 10K tables? Maybe we should load the table schema lazily? Using the LRU strategy?

The way to reduce memory usage in this PR is cheap and effective.
To support 10K tables, only reduce the memory usage is not enough. We need to consider memory, ETCD access frequency, and so on. We have a plan to solve the problem by optimizing the replication model. The schema storage will have further optimization at that time.

amyangfei
amyangfei reviewed Nov 26, 2020
View reviewed changes
cdc/entry/schema_storage.go Show resolved Hide resolved
cdc/entry/schema_storage.go Show resolved Hide resolved
zier-one
zier-one commented Nov 26, 2020
View reviewed changes
cdc/entry/schema_storage.go Outdated Show resolved Hide resolved
cdc/entry/schema_storage.go Outdated Show resolved Hide resolved
amyangfei
amyangfei reviewed Nov 26, 2020
View reviewed changes
Copy link
Contributor

@amyangfei amyangfei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-srebot ti-srebot added the status/LGT1 Indicates that a PR has LGTM 1. label Nov 26, 2020
@liuzix
Copy link
Contributor

liuzix commented Nov 26, 2020

LGTM

@ti-srebot ti-srebot removed the status/LGT1 Indicates that a PR has LGTM 1. label Nov 26, 2020
@ti-srebot ti-srebot added the status/LGT2 Indicates that a PR has LGTM 2. label Nov 26, 2020
@amyangfei
Copy link
Contributor

/merge

@ti-srebot ti-srebot added the status/can-merge Indicates a PR has been approved by a committer. label Nov 26, 2020
@ti-srebot
Copy link
Contributor

/run-all-tests

@codecov-io
Copy link

Codecov Report

Merging #1115 (a31f86f) into master (5e25cbb) will increase coverage by 0.4067%.
The diff coverage is 92.5000%.

@@               Coverage Diff                @@
##             master      #1115        +/-   ##
================================================
+ Coverage   39.3036%   39.7103%   +0.4066%     
================================================
  Files           112        112                
  Lines         11719      11740        +21     
================================================
+ Hits           4606       4662        +56     
+ Misses         6641       6605        -36     
- Partials        472        473         +1

@amyangfei amyangfei merged commit c852d05 into pingcap:master Nov 26, 2020
ti-srebot pushed a commit to ti-srebot/ticdc that referenced this pull request Nov 26, 2020
@ti-srebot
cherry pick pingcap#1115 to release-4.0
22f2635 
Signed-off-by: ti-srebot <[email protected]>
@ti-srebot ti-srebot mentioned this pull request Nov 26, 2020
Merged
@ti-srebot
Copy link
Contributor

cherry pick to release-4.0 in PR #1127

ti-srebot added a commit that referenced this pull request Nov 26, 2020
@ti-srebot
schemaStorage: reduce memory usage and add unit tests (#1115) (#1127)
2c00498 
Signed-off-by: ti-srebot <[email protected]>
@zier-one zier-one deleted the refine_schema_storage branch March 1, 2021 06:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@amyangfei amyangfei amyangfei left review comments

@ti-srebot ti-srebot ti-srebot approved these changes

Assignees
No one assigned
Labels
needs-cherry-pick-release-4.0 Should cherry pick this PR to release-4.0 branch. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2. status/ptal Could you please take a look?
Projects
None yet
Milestone
v4.0.9
Development

Successfully merging this pull request may close these issues.

TiCDC costs too much memory if table count is large or DDL is frequent
6 participants
@zier-one @amyangfei @zhangjinpeng87 @liuzix @ti-srebot @codecov-io