Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

*: don't use chunk rpc protocol for internal session #31723

Merged
merged 13 commits into from
Jan 18, 2022
Merged

*: don't use chunk rpc protocol for internal session #31723

merged 13 commits into from
Jan 18, 2022

Conversation

tiancaiamao
Copy link
Contributor

@tiancaiamao tiancaiamao commented Jan 17, 2022 •
edited
Loading

What problem does this PR solve?

Issue Number: close #30880

This is a workaround rather than a direct fix.

Problem Summary:

Chunk RPC protocol have the memory leak problem #30880

Before we figure out how it's happening and fix the bug, we can avoid using it for internal sessions.

What is changed and how it works?

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)

After avoiding using chunk RPC protocol for internal session, the memory usage is low.

./bin/dumpling  -h "172.16.5.59" -P "4897" -u "root" --output "s3://tmp/dumpling-outputc1144ef2-c669-4aea-8482-87b3ab6efbc2?access-key=XXXXXXX&secret-access-key=XXXXXXXX&endpoint= XXXXXXXX&force-path-style=true" --filetype "csv" --filesize "256M" --rows "200" --threads 4 --params "tidb_enable_chunk_rpc=0"
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

None

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Jan 17, 2022 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • hawkingrei
  • wshwsh12

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jan 17, 2022
@tiancaiamao tiancaiamao added type/bugfix This PR fixes a bug. affects-5.4 This bug affects the 5.4.x(LTS) versions. labels Jan 17, 2022
@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Jan 17, 2022
@sre-bot
Copy link
Contributor

sre-bot commented Jan 17, 2022 •
edited
Loading

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/a75b76009c3ed8fb7a7793059b5032d4267b8547

@tiancaiamao
Copy link
Contributor Author

/run-check_dev

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jan 17, 2022
@xhebox
Copy link
Contributor

xhebox commented Jan 17, 2022

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: dd04594

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jan 17, 2022
@tiancaiamao tiancaiamao added the needs-cherry-pick-release-5.4 Should cherry pick this PR to release-5.4 branch. label Jan 17, 2022
@tiancaiamao
Copy link
Contributor Author

/merge

@tiancaiamao
Copy link
Contributor Author

/run-unit-test

@tiancaiamao
Copy link
Contributor Author

/run-check_dev_2

@hawkingrei
Copy link
Member

/run-mysql-test

@tiancaiamao
Copy link
Contributor Author

/run-check_dev_2

@tiancaiamao
Copy link
Contributor Author

/run-unit-test

@hawkingrei
Copy link
Member

/run-check_dev

@hawkingrei
Copy link
Member

/run-unit-test

@hawkingrei
Copy link
Member

/run-check_dev_2

@hawkingrei
Copy link
Member

/run-unit-test

@tiancaiamao tiancaiamao mentioned this pull request Jan 17, 2022
Merged
12 tasks
@hawkingrei
Copy link
Member

/run-unit-test

@tiancaiamao
Copy link
Contributor Author

/merge

@ti-chi-bot ti-chi-bot merged commit acc3ae6 into pingcap:master Jan 18, 2022
@ti-srebot
Copy link
Contributor

cherry pick to release-5.4 failed

@tiancaiamao tiancaiamao deleted the use-chunk-rpc branch January 18, 2022 02:59
@guo-shaoge guo-shaoge added the needs-cherry-pick-release-5.3 Type: Need cherry pick to release-5.3 label Feb 15, 2022
@ti-srebot ti-srebot mentioned this pull request Feb 15, 2022
Merged
12 tasks
ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Feb 15, 2022
@tiancaiamao @ti-srebot
cherry pick pingcap#31723 to release-5.3
9232b0b 
Signed-off-by: ti-srebot <[email protected]>
@ti-srebot
Copy link
Contributor

cherry pick to release-5.3 in PR #32344

ti-chi-bot pushed a commit that referenced this pull request Jun 15, 2022
@ti-srebot
*: don't use chunk rpc protocol for internal session (#31723) (#32344)
f926183 
close #30880
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wshwsh12 wshwsh12 wshwsh12 approved these changes

@hawkingrei hawkingrei hawkingrei approved these changes

@winoros winoros Awaiting requested review from winoros

@fzhedu fzhedu Awaiting requested review from fzhedu

Assignees
No one assigned
Labels
affects-5.4 This bug affects the 5.4.x(LTS) versions. needs-cherry-pick-release-5.3 Type: Need cherry pick to release-5.3 needs-cherry-pick-release-5.4 Should cherry pick this PR to release-5.4 branch. release-note-none Denotes a PR that doesn't merit a release note. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2. type/bugfix This PR fixes a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

TiDB OOM during dumpling 2500 tables
8 participants
@tiancaiamao @ti-chi-bot @sre-bot @xhebox @hawkingrei @ti-srebot @wshwsh12 @guo-shaoge