Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support more ScalarFunction in Ballista #2008

Merged
merged 9 commits into from
Mar 16, 2022
Merged

Conversation

Ted-Jiang
Copy link
Member

@Ted-Jiang Ted-Jiang commented Mar 14, 2022

Which issue does this PR close?

Closes #2007 .

Rationale for this change

Now support more ScalarFunction in ballista. like "random"
start 1 scheduler + 1 executor
run client with cargo run --features=ballista --bin datafusion-cli -- --host localhost --port 50050
Before:

select * from lineitem where l_quantity = random()*10;

Internal("failed to serialize logical plan: General(\"Unsupported scalar function Random\")")

Now:

select * from lineitem where l_quantity = trunc(random() * 9 + 1) limit 3;
+------------+-----------+-----------+--------------+------------+-----------------+------------+-------+--------------+--------------+------------+--------------+---------------+-------------------+------------+-------------------------+
| l_orderkey | l_partkey | l_suppkey | l_linenumber | l_quantity | l_extendedprice | l_discount | l_tax | l_returnflag | l_linestatus | l_shipdate | l_commitdate | l_receiptdate | l_shipinstruct    | l_shipmode | l_comment               |
+------------+-----------+-----------+--------------+------------+-----------------+------------+-------+--------------+--------------+------------+--------------+---------------+-------------------+------------+-------------------------+
| 7          | 145243    | 7758      | 2            | 9          | 11594.16        | 0.08       | 0.08  | N            | O            | 1996-02-01 | 1996-03-02   | 1996-02-19    | TAKE BACK RETURN  | SHIP       | es. instructions        |
| 35         | 120896    | 8433      | 3            | 7          | 13418.23        | 0.06       | 0.04  | N            | O            | 1996-01-19 | 1995-12-22   | 1996-01-29    | NONE              | MAIL       |  the carefully regular  |
| 163        | 192642    | 5162      | 4            | 5          | 8673.2          | 0.02       | 0     | N            | O            | 1997-11-17 | 1997-10-09   | 1997-12-05    | DELIVER IN PERSON | TRUCK      |  must belie             |
+------------+-----------+-----------+--------------+------------+-----------------+------------+-------+--------------+--------------+------------+--------------+---------------+-------------------+------------+-------------------------+
3 rows in set. Query took 1.897 seconds.

What changes are included in this PR?

Are there any user-facing changes?

@liukun4515
Copy link
Contributor

Thanks, @Ted-Jiang I will go through this pr later.

@github-actions github-actions bot added the datafusion Changes in the datafusion crate label Mar 15, 2022
@@ -979,8 +1008,6 @@ impl TryFrom<&protobuf::LogicalExprNode> for Expr {
ScalarFunction::Sin => Ok(sin((&args[0]).try_into()?)),
ScalarFunction::Cos => Ok(cos((&args[0]).try_into()?)),
ScalarFunction::Tan => Ok(tan((&args[0]).try_into()?)),
// ScalarFunction::Asin => Ok(asin(&args[0]).try_into()?)),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the commented-out Asin, Acos, Concat, Array, Nullif, and md5 are removed?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opps, forgot the deleted line, have added.

Copy link
Contributor

@liukun4515 liukun4515 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
It's would be better if we had the tests for the build-in function on the ballista side.
@Ted-Jiang Can you help to add some tests for this in the follow-up pull request?

@doki23
Copy link
Contributor

doki23 commented Mar 16, 2022

may need a follow-up pr when #1969 merged

Copy link
Member

@yjshen yjshen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Ted-Jiang, LGTM.

@yjshen yjshen merged commit dbc5815 into apache:master Mar 16, 2022
@yjshen yjshen added the api change Changes the API exposed to users of the crate label Mar 16, 2022
@yjshen yjshen mentioned this pull request Mar 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api change Changes the API exposed to users of the crate datafusion Changes in the datafusion crate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Ballista] Support more ScalarFunction in Ballista
4 participants