Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(substrait): consuming AggregateRel as last node #12875

Merged
merged 1 commit into from
Oct 16, 2024

Conversation

tokoko
Copy link
Contributor

@tokoko tokoko commented Oct 11, 2024

Which issue does this PR close?

Closes #12869.

Rationale for this change

consumer bakes in plan names into last Aggregate, but only handled metrics expressions before. This PR splits schema fields into grouping and metrics fields and renames each accordingly.

Are these changes tested?

yes

Are there any user-facing changes?

yes

Copy link
Contributor

@vbarua vbarua left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me from the Substrait side of things.

It also looks reasonable to me from the DataFusion side of thing, but I have less experience there 😅

],
"version": {
"minorNumber": 54,
"producer": "subframe"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a publicly available producer? It's useful to know how the plans where generated, in case we ever need to regenerate them to pull in updates from the producers.

The plan itself is perfectly valid and fine as-is.

If we ever want/need to generate an equivalent plan we can use https://github.com/substrait-io/substrait-java/tree/main/isthmus-cli with a query like:

./isthmus --create "CREATE TABLE data(a INT)" "SELECT a, COUNT(*) AS c FROM data GROUP BY a"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well, kinda.. it's a personal project. for me, it was obviously easiest to use, so I went with it, but it's far from complete, so if we want to make all files "re-generatable", we would have to use something like isthmus.

Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems reasonable to me 👍

Thank you @tokoko and @vbarua

@alamb alamb merged commit 87c2081 into apache:main Oct 16, 2024
24 checks passed
@alamb
Copy link
Contributor

alamb commented Oct 16, 2024

🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Substrait: Plans with AggregateRels as a last node fail to be consumed
3 participants