Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lexsort_to_indices should not fallback to non-lexical sort if the datatype is not supported #6226

Closed
viirya opened this issue Aug 11, 2024 · 1 comment · Fixed by #6225
Closed
Labels
arrow Changes to the arrow crate bug

Comments

@viirya
Copy link
Member

viirya commented Aug 11, 2024

Describe the bug

Comet encountered some errors on Spark test like following. It is because DataFusion SortExec calls lexsort_to_indices and the function fallbacks to sort_to_indices for single column case. However, sort_to_indices doesn't support all data types, e.g., struct. For unsupported data types, lexsort_to_indices should not fallback to sort_to_indices.

[info] - vectorized reader: struct *** FAILED *** (515 milliseconds)        
[info]   org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 644.0 failed 1 times, most recent failure: Lost task 0.0 in stage 644.0 (TID 862) (192.168.86.44 executor driver): org.apache.comet.CometNa
tiveException: Compute error: Sort not supported for data type Struct([Field { name: "_1", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "_2", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }])                                                                                                                                                                                    

To Reproduce

Expected behavior

Additional context

@alamb
Copy link
Contributor

alamb commented Aug 31, 2024

label_issue.py automatically added labels {'arrow'} from #6225

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arrow Changes to the arrow crate bug
Projects
None yet
2 participants