Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Don't materialize join columns that are dropped right after #8831

Open
revans2 opened this issue Jul 27, 2023 · 1 comment
Open

[FEA] Don't materialize join columns that are dropped right after #8831

revans2 opened this issue Jul 27, 2023 · 1 comment
Labels
feature request New feature or request performance A performance related task/issue reliability Features to improve reliability or bugs that severly impact the reliability of the plugin

Comments

@revans2
Copy link
Collaborator

revans2 commented Jul 27, 2023

Is your feature request related to a problem? Please describe.
When doing a join for most join types, the join keys for both side of the join are always materialized and then dropped in a project right afterwards. For us we get a gather map, and then gather the columns we want to output using that map. It would be much more memory efficient, and performant to just never materialize those columns in the first place. We can use them to produce the gather map, but not output them. This would require an API change to join, but it should not be that big of a deal. It would also require us to have a post processing rule that could detect this and rewrite the plan accordingly.

@revans2 revans2 added feature request New feature or request ? - Needs Triage Need team to review and classify performance A performance related task/issue labels Jul 27, 2023
@revans2
Copy link
Collaborator Author

revans2 commented Jul 27, 2023

I thought I filed something for this before, but I could not find it so I am doing it again.

@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Aug 8, 2023
@revans2 revans2 added the reliability Features to improve reliability or bugs that severly impact the reliability of the plugin label Sep 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request performance A performance related task/issue reliability Features to improve reliability or bugs that severly impact the reliability of the plugin
Projects
None yet
Development

No branches or pull requests

2 participants