Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: serialize/deserialize logical and execution plan via substrait #317

Merged
merged 11 commits into from
Oct 24, 2022

Conversation

waynexia
Copy link
Member

@waynexia waynexia commented Oct 18, 2022

This patch implements the serialization and deserialization of our plans (DF's logical plan and execution plan for now). It's required by #291.

I choose https://substrait.io as the intermediate representation of plans, which is (or aim to be) a well-defined, cross-language specification for data compute operations. This kind of impl-unrelated representation can let us not only to communicate plans between Frontend and Datanode, but also gives the possibility to communicate with other systems in the future (but not the near future I suppose 🫣 it's not that widely used at present (I only find duckdb supports this among DBs). More on plan like apache/datafusion-ballista#30 (comment)).

And for the ser/de target, I'm not sure which one is better (logical or physical plan) so I implement both. Maybe for a non-hybrid computation (i.e. GT to GT) we can provide enough information to reassemble a physical plan (I guess)? But whatever, serializing a physical plan needs tremendous downcast and logical plan is more ergonomic...

Back to this patch, it only ships an implementation in the very early stage -- only bare table scan is supported. Other things like translating substrait's expression or schema, and the remaining plans are expected to be done later. I'll open a detailed list to track them.

@waynexia waynexia added the WIP label Oct 18, 2022
@waynexia
Copy link
Member Author

I decide to drop support of ExecutionPlan (at least for now).

@waynexia waynexia marked this pull request as ready for review October 21, 2022 09:50
Signed-off-by: Ruihang Xia <[email protected]>
@codecov
Copy link

codecov bot commented Oct 21, 2022

Codecov Report

Merging #317 (be08153) into develop (bc9a2df) will decrease coverage by 0.11%.
The diff coverage is 52.85%.

@@             Coverage Diff             @@
##           develop     #317      +/-   ##
===========================================
- Coverage    84.16%   84.05%   -0.12%     
===========================================
  Files          348      351       +3     
  Lines        31790    32152     +362     
===========================================
+ Hits         26757    27025     +268     
- Misses        5033     5127      +94     
Flag Coverage Δ
rust 84.05% <52.85%> (-0.12%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/common/grpc/src/physical/plan.rs 65.38% <0.00%> (ø)
src/common/substrait/src/error.rs 0.00% <0.00%> (ø)
src/common/substrait/src/df_logical.rs 62.14% <62.14%> (ø)
src/common/substrait/src/lib.rs 100.00% <100.00%> (ø)
src/meta-srv/src/service/heartbeat.rs 53.40% <0.00%> (-11.53%) ⬇️
src/meta-srv/src/error.rs 64.77% <0.00%> (-10.59%) ⬇️
src/meta-client/src/client/heartbeat.rs 79.51% <0.00%> (-8.37%) ⬇️
src/servers/src/error.rs 22.95% <0.00%> (-6.56%) ⬇️
src/meta-client/src/client.rs 77.90% <0.00%> (-2.23%) ⬇️
src/meta-client/src/client/router.rs 91.22% <0.00%> (-0.88%) ⬇️
... and 16 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Signed-off-by: Ruihang Xia <[email protected]>
@waynexia waynexia merged commit 8ab43b6 into develop Oct 24, 2022
@waynexia waynexia deleted the scan-phy-serde branch October 24, 2022 07:29
paomian pushed a commit to paomian/greptimedb that referenced this pull request Oct 19, 2023
…reptimeTeam#317)

* fix: change Utf8Array indice type

Signed-off-by: Ruihang Xia <[email protected]>

* refactor: remove unused sub-crate

Signed-off-by: Ruihang Xia <[email protected]>

* feat: impl for both Logical and Execution plan

Signed-off-by: Ruihang Xia <[email protected]>

* refactor: move test-util subcrate into table

Signed-off-by: Ruihang Xia <[email protected]>

* test: table scan logical plan round trip

Signed-off-by: Ruihang Xia <[email protected]>

* drop support of physical plan

Signed-off-by: Ruihang Xia <[email protected]>

* fix warnings

Signed-off-by: Ruihang Xia <[email protected]>

* rename trait fns to encode/decode

Signed-off-by: Ruihang Xia <[email protected]>

* address review comments

Signed-off-by: Ruihang Xia <[email protected]>

Signed-off-by: Ruihang Xia <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants