Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Initial query stages read parquet files and repartition them needlessly #243

Closed
Tracked by #129
andygrove opened this issue Sep 19, 2022 · 3 comments
Closed
Tracked by #129

Initial query stages read parquet files and repartition them needlessly #243

andygrove opened this issue Sep 19, 2022 · 3 comments
Labels
bug Something isn't working performance

Comments

@andygrove
Copy link
Member

Describe the bug

shuffle-parquet

To Reproduce

Expected behavior
We should not shuffle the input files like this

Additional context
@andygrove andygrove added bug Something isn't working performance labels Sep 19, 2022
@andygrove
Copy link
Member Author

@yahoNanJing @thinkharderdev I found this during testing today with #242

@andygrove andygrove mentioned this issue Sep 19, 2022
Open
18 tasks
@andygrove
Copy link
Member Author

I may have been premature in creating this issue. This stage is an input to a join. I will think about this some more.

join

@yahoNanJing
Copy link
Contributor

Hi @andygrove, I don't think it's an issue. For a hash join, we need to do repartition each tables.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working performance
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@andygrove @yahoNanJing