Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase the coupling of Ibis and the DataFusion Backend #32

Closed
mesejo opened this issue May 8, 2024 · 0 comments
Closed

Increase the coupling of Ibis and the DataFusion Backend #32

mesejo opened this issue May 8, 2024 · 0 comments

Comments

@mesejo
Copy link
Collaborator

mesejo commented May 8, 2024

Ibis as a library made the design choice of being highly decoupled from each backend specific implementation, this makes sense because it simplifies maintenance. For LETSQL since we only cater to one backend we should increase the coupling with DataFusion.

This is impactful for the following reasons:

  1. Access all functionality, for example, the unnest function is available as part of the LogicalPlan and exposed to dataframes, but not through SQL (or hidden)
  2. Drop sqlglot as a dependency, this will reduce the size of the library, the number of dependencies and avoid relying on a library that is governed by a single individual/company
  3. Drop generic implementations such as to_parquet that may be missing out on DataFusion performance improvements (see benchmarks). The implementation of to_parquet uses a generic to_pyarrow_batches in conjuction with the pyarrow.parquet.ParquetWriter
  4. Less important, directly generating the LogicalPlan will improve (probably and marginally) the processing speed, currently we construct a string via sqlglot only to parse (deconstruct) it into the LogicalPlan
@mesejo mesejo changed the title Generate the LogicalPlan directly from Ibis Expr Increase the coupling of Ibis and the DataFusion Backend May 8, 2024
@mesejo mesejo closed this as completed Feb 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant