Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Fix merge conflicts from merging comet-parquet-exec into main #1320

Merged
merged 1 commit into from
Jan 22, 2025

Conversation

andygrove
Copy link
Member

@andygrove andygrove commented Jan 22, 2025

Which issue does this PR close?

N/A

Rationale for this change

Fix issues from #1318

What changes are included in this PR?

  • Remove some orphaned source files
  • Revert a documentation change

How are these changes tested?

@andygrove andygrove marked this pull request as ready for review January 22, 2025 02:48
Copy link
Contributor

@parthchandra parthchandra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @andygrove

Copy link
Contributor

@comphead comphead left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm thanks @andygrove

@andygrove andygrove merged commit f09f8af into apache:main Jan 22, 2025
76 checks passed
@andygrove andygrove deleted the fix-merge-comet-parquet-exec branch January 22, 2025 17:17
@kazuyukitanimura
Copy link
Contributor

kazuyukitanimura commented Jan 22, 2025

@andygrove @parthchandra @comphead
I still see more merge issues

Looks like common/src/main/scala/org/apache/spark/sql/comet/util/Utils.scala and common/src/main/scala/org/apache/spark/sql/comet/CometArrowUtils.scala are unmerged/duplicates

@kazuyukitanimura
Copy link
Contributor

native/core/src/parquet/parquet_support.rs and native/spark-expr/src/conversion_funcs/cast.rs are also unmerged/duplicates

I think we really should work on revert the this commit and previous commit and do a proper merge
@andygrove @parthchandra @comphead

@parthchandra
Copy link
Contributor

@kazuyukitanimura I don't see duplicates of any of the above files. Can you clarify what you mean by unmerged?

@kazuyukitanimura
Copy link
Contributor

kazuyukitanimura commented Jan 22, 2025

@parthchandra E.g. for the utils file fromArrowType is defined in both files

Another potential issue
spark/src/test/scala/org/apache/spark/sql/comet/CometPlanStabilitySuite.scala](https://github.com/apache/datafusion-comet/blob/main/spark/src/test/scala/org/apache/spark/sql/comet/CometPlanStabilitySuite.scala#L349
is testing only when the mode is SCAN_NATIVE_COMET?

@kazuyukitanimura
Copy link
Contributor

kazuyukitanimura commented Jan 22, 2025

https://github.com/apache/datafusion-comet/blob/main/spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala#L1267

looks like offheap mode check is re-introduced

@parthchandra
Copy link
Contributor

@parthchandra E.g. for the utils file fromArrowType is defined in both files

Oh, I get it now. We did duplicate code in the feature branch (CometArrowUtils and Utils both duplicate code from Spark. Let me clean this up in a followup)
The cast.rs and parquet_support.rs duplication is intentional. We used the former as the starting point for parquet type conversions. The parquet_support code will change as we address more data types.

Another potential issue spark/src/test/scala/org/apache/spark/sql/comet/CometPlanStabilitySuite.scala](https://github.com/apache/datafusion-comet/blob/main/spark/src/test/scala/org/apache/spark/sql/comet/CometPlanStabilitySuite.scala#L349 is testing only when the mode is SCAN_NATIVE_COMET?

Yes. The plans are different when the scans are different. We generated new plans for the different scan types but did not turn them on. The two new scan implementations do not yet pass all the unit tests so turning on plan stability for them is a little premature

@andygrove
Copy link
Member Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants