-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add need_data_exchange in the ExecutionPlan to indicate whether a physical operator needs data exchange #4585
Comments
This might be a dumb question, but can this not be inferred from if the ExecutionPlan alters the partitioning of its input? |
I tried to think of a case where checking if input partitioning and output partitioning match was insufficient and I could not 🤔 |
My preference is to keep the ExecutionPlan Trait simple and leverage other methods or pattern matching to decide what kind of actions need to take to adjust the plan tree. |
Agree |
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Currently there's no explicit way to indicate whether an execution plan needs a data exchange, which is not so clear for the caller, like Ballista, to judge whether it's necessary to add an additional data exchange operator to deal with the shuffling.
Describe the solution you'd like
It's better to classify and clarify which execution plans needs data exchange explicitly.
Currently there are 3 kinds of execution plan which needs data exchange
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
The text was updated successfully, but these errors were encountered: