[FEA] Don't materialize join columns that are dropped right after #8831
Labels
feature request
New feature or request
performance
A performance related task/issue
reliability
Features to improve reliability or bugs that severly impact the reliability of the plugin
Is your feature request related to a problem? Please describe.
When doing a join for most join types, the join keys for both side of the join are always materialized and then dropped in a project right afterwards. For us we get a gather map, and then gather the columns we want to output using that map. It would be much more memory efficient, and performant to just never materialize those columns in the first place. We can use them to produce the gather map, but not output them. This would require an API change to join, but it should not be that big of a deal. It would also require us to have a post processing rule that could detect this and rewrite the plan accordingly.
The text was updated successfully, but these errors were encountered: