Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Lightning: performance issue when importing from cloud storage #57413

Closed
xuanyu66 opened this issue Nov 15, 2024 · 0 comments · Fixed by #57454
Closed

Lightning: performance issue when importing from cloud storage #57413

xuanyu66 opened this issue Nov 15, 2024 · 0 comments · Fixed by #57454
Labels
affects-5.3 This bug affects 5.3.x versions. affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. affects-8.5 This bug affects the 8.5.x(LTS) versions. component/lightning This issue is related to Lightning of TiDB. severity/moderate type/bug The issue is confirmed as a bug.

Comments

@xuanyu66
Copy link
Contributor

xuanyu66 commented Nov 15, 2024 •
edited
Loading

Enhancement

When importing from cloud storage, high concurrency can lead to bottlenecks in mydump.PooledReader.Seek(0, io.SeekCurrent) due to the application of ioWorkers. Seek(0, io.SeekCurrent) does not cause disk IO operations, applying ioWorkers is unnecessary.
image
@xuanyu66 xuanyu66 added the type/enhancement The issue or PR belongs to an enhancement. label Nov 15, 2024
@xuanyu66 xuanyu66 mentioned this issue Nov 18, 2024
Merged
13 tasks
@lance6716 lance6716 added type/bug The issue is confirmed as a bug. affects-5.3 This bug affects 5.3.x versions. affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. affects-8.5 This bug affects the 8.5.x(LTS) versions. severity/moderate and removed type/enhancement The issue or PR belongs to an enhancement. labels Nov 19, 2024
@ti-chi-bot ti-chi-bot bot closed this as completed in 0e3efb6 Nov 19, 2024
@jebter jebter added the component/lightning This issue is related to Lightning of TiDB. label Dec 2, 2024
@ti-chi-bot ti-chi-bot mentioned this issue Dec 3, 2024
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Dec 4, 2024
@ti-chi-bot
Lightning: Skip apply ioworkers when seek(0, io.SeekCurrent) (#57454) (…
7723772 
…#57930)

close #57413
@ti-chi-bot ti-chi-bot mentioned this issue Feb 14, 2025
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Feb 17, 2025
@ti-chi-bot
Lightning: Skip apply ioworkers when seek(0, io.SeekCurrent) (#57454) (…
017dd17 
…#59544)

close #57413
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-5.3 This bug affects 5.3.x versions. affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. affects-8.5 This bug affects the 8.5.x(LTS) versions. component/lightning This issue is related to Lightning of TiDB. severity/moderate type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Lightning: Skip apply ioworkers when seek(0, io.SeekCurrent) xuanyu66/tidb
3 participants
@lance6716 @jebter @xuanyu66