Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Decimal128 Support [databricks] #4139

Merged
merged 38 commits into from
Nov 19, 2021
Merged

Decimal128 Support [databricks] #4139

merged 38 commits into from
Nov 19, 2021

Conversation

revans2
Copy link
Collaborator

@revans2 revans2 commented Nov 17, 2021

This adds in support for decimal 128 and has been a collaboration with lots of different people.

revans2 and others added 30 commits October 21, 2021 15:04
@revans2 @kuhushukla
@sperlingxx @abellina @nartal1
Initial decimal 128 support (#3875)
e56e49b 
Signed-off-by: Robert (Bobby) Evans <[email protected]>
Signed-off-by: Alessandro Bellina <[email protected]>
Co-authored-by: Kuhu Shukla <[email protected]>
Co-authored-by: sperlingxx <[email protected]>
Co-authored-by: Alfred Xu <[email protected]>
Co-authored-by: Alessandro Bellina <[email protected]>
Co-authored-by: Niranjan Artal <[email protected]>
@revans2
Remove decimal enable config (#3893)
b747a82 
Signed-off-by: Robert (Bobby) Evans <[email protected]>
@revans2
Extend range of supported decimal divide operations (#3891)
e52b6fb 
Signed-off-by: Robert (Bobby) Evans <[email protected]>
@revans2
Decimal Average Support (#3898)
49eaf7a 
Signed-off-by: Robert (Bobby) Evans <[email protected]>
@revans2
Improve the range for mixed Decimal Multiply (#3900)
c69f426 
Signed-off-by: Robert (Bobby) Evans <[email protected]>
@revans2
Merge branch 'branch-21.12' into decimal128_upmerge
1bb791b
@revans2
More generic overflow checking and docs for decimal 128 (#3936)
099541c 
Signed-off-by: Robert (Bobby) Evans <[email protected]>
@revans2
Merge branch 'branch-21.12' into decimal128_upmerge
7a1af60
@revans2
Addressed review comments
9014610
@revans2
improved overflow test
ad12058
@revans2
Merge pull request #3989 from revans2/decimal128_upmerge
ad17764 
Upmerge Decimal128 to latest spark-rapids branch
@revans2
Fix columnar to columnar row conversion for decimal128
8ffbde9 
Signed-off-by: Robert (Bobby) Evans <[email protected]>
@revans2
Merge pull request #4020 from revans2/decimal128_columnar_movement
85bba72 
Fix CPU columnar to GPU columnar conversion for decimal128
@sperlingxx
Support Read/Write ORC on Decimal128 (#4037)
b88cac2 
Signed-off-by: sperlingxx [email protected]

Closes #3882
@sperlingxx
add missing supports in 320 shim
6dcba4a 
Signed-off-by: sperlingxx <[email protected]>
@revans2
Merge pull request #4050 from sperlingxx/fix_some_missing
334e2aa 
Decimal 128: Add missing supports for Literal and WindowExpr in 320
@revans2
Revert "Support Read/Write ORC on Decimal128 (#4037)"
b15512b 
This reverts commit b88cac2.
@revans2
Merge pull request #4055 from NVIDIA/revert-4037-dec128_orc
09f2c87 
Revert "Support Read/Write ORC on Decimal128"
@revans2
Merge branch 'branch-21.12' into decimal128_upmerge
6e4e031
@revans2
Cleanup docs
c25d1cb
@revans2
Merge branch 'decimal128' into decimal128_upmerge
d970325
@revans2
Fixed some tests
930fabf
@revans2
Fixed merge conflict
032d4b6
@revans2
Merge pull request #4056 from revans2/decimal128_upmerge
f5e6288 
Decimal128 upmerge
@sperlingxx
Decimal 128 Support: HiveGenericUDF, HiveSimpleUDF, ScalaUDF (#4068)
95d8939 
Signed-off-by: sperlingxx <[email protected]>
@kuhushukla
Add Decimal128 support for MapKeys, MapValues, MapEntries (#4074)
1978b74 
Signed-off-by: Kuhu Shukla <[email protected]>

Co-authored-by: Kuhu Shukla <[email protected]>
@revans2
Update docs for decimal overflow (#4077)
682a7dc 
Signed-off-by: Robert (Bobby) Evans <[email protected]>
@kuhushukla
Decimal128 support for Concat (#4086)
f702200 
Signed-off-by: Kuhu Shukla <[email protected]>

Co-authored-by: Kuhu Shukla <[email protected]>
@sperlingxx
Support Read/Write ORC on Decimal128 (#4095)
5b5aaa4 
Signed-off-by: sperlingxx <[email protected]>
@revans2
Merge branch 'branch-21.12' into decimal128
283de95
@revans2
Copy link
Collaborator Author

revans2 commented Nov 17, 2021

build

jlowe
jlowe previously approved these changes Nov 17, 2021
View reviewed changes
@revans2
Copy link
Collaborator Author

revans2 commented Nov 17, 2021

build

kuhushukla
kuhushukla previously approved these changes Nov 17, 2021
View reviewed changes
@revans2
Copy link
Collaborator Author

revans2 commented Nov 18, 2021

Tests are failing because of some issues with changes to SUM in 3.1.0. I am trying to figure out how to make this work with the older versions of Spark.

@revans2
Copy link
Collaborator Author

revans2 commented Nov 18, 2021

build

@revans2
Copy link
Collaborator Author

revans2 commented Nov 18, 2021

@jlowe @kuhushukla

I added in a basic fix for decimal. It is not great because there is a circular dependency between different files now, and I want to break up AggregateFunctions.scala, but that is too large of a change to make here.

jlowe
jlowe previously approved these changes Nov 18, 2021
View reviewed changes
@revans2
Copy link
Collaborator Author

revans2 commented Nov 18, 2021

Test failed with hash percentile approximate for decimal. This is an intermittent failure and is still being investigated #4060

@revans2
Copy link
Collaborator Author

revans2 commented Nov 18, 2021

build

@revans2
Copy link
Collaborator Author

revans2 commented Nov 18, 2021

It failed on 3.3.0 compile even after nothing changed. It might be a Spark change itself.

[2021-11-18T17:32:54.272Z] [ERROR] [Error] /home/jenkins/agent/workspace/jenkins-rapids_premerge-github-3312/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuOrcFileFormat.scala:132: not found: value OrcFileFormat

@revans2
Copy link
Collaborator Author

revans2 commented Nov 18, 2021

Yup. apache/spark@6f6a066 removed the Object OrcFileFormat

@revans2
Copy link
Collaborator Author

revans2 commented Nov 18, 2021

build

jlowe
jlowe previously approved these changes Nov 18, 2021
View reviewed changes
@revans2
Copy link
Collaborator Author

revans2 commented Nov 18, 2021

build

@revans2
Copy link
Collaborator Author

revans2 commented Nov 18, 2021

Same failure as before :(

@revans2
Copy link
Collaborator Author

revans2 commented Nov 18, 2021

build

@nartal1 nartal1 mentioned this pull request Nov 19, 2021
Merged
@revans2
Copy link
Collaborator Author

revans2 commented Nov 19, 2021

This time it was test_hash_groupby_approx_percentile_long_repeated_keys. This has a fix that is blocking other things so I am going to just merge it.

@revans2 revans2 merged commit de6d454 into branch-21.12 Nov 19, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jlowe jlowe jlowe approved these changes

@kuhushukla kuhushukla kuhushukla left review comments

Assignees

@revans2 revans2

@kuhushukla kuhushukla

Labels
feature request New feature or request
Projects
None yet
Milestone
Nov 15 - Nov 26
Development

Successfully merging this pull request may close these issues.
4 participants
@revans2 @jlowe @kuhushukla @sperlingxx