Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix build regression against Spark 3.2.x [databricks] #10179

Merged
merged 4 commits into from
Jan 11, 2024

Conversation

andygrove
Copy link
Contributor

@andygrove andygrove commented Jan 10, 2024

Closes #10174

The reason this code was failing against 3.2 and not against 3.3 is that 3.2 will populate timestampFormat with a default if none is supplied, whereas 3.3 does not.

Integration tests passed locally for me with this change against both 3.2.3 and 3.3.2

@andygrove andygrove self-assigned this Jan 10, 2024
@andygrove andygrove added the build Related to CI / CD or cleanly building label Jan 10, 2024
Signed-off-by: Andy Grove <[email protected]>
@andygrove
Copy link
Contributor Author

build

@andygrove andygrove marked this pull request as ready for review January 10, 2024 20:10
@andygrove andygrove changed the title Fix build regression against Spark 3.2.x Fix build regression against Spark 3.2.x [databricks] Jan 10, 2024
@andygrove
Copy link
Contributor Author

build

@andygrove
Copy link
Contributor Author

andygrove commented Jan 10, 2024

CI failed due to unrelated issue (on databricks):

2024-01-10T23:39:01.0960149Z [2024-01-10T23:34:21.452Z] [2024-01-10T23:33:07.595Z] ^[[31mFAILED^[[0m ../../src/main/python/dpp_test.py::^[[1mtest_dpp_bypass[true-5-parquet][DATAGEN_SEED=1704919540, INJECT_OOM, IGNORE_ORDER]^[[0m - py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.s...
2024-01-10T23:39:01.0961556Z [2024-01-10T23:34:21.452Z] [2024-01-10T23:33:07.595Z] ^[[31mFAILED^[[0m ../../src/main/python/dpp_test.py::^[[1mtest_dpp_bypass[true-5-orc][DATAGEN_SEED=1704919540, IGNORE_ORDER]^[[0m - py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.s...
2024-01-10T23:39:01.0963105Z [2024-01-10T23:34:21.452Z] [2024-01-10T23:33:07.595Z] ^[[31mFAILED^[[0m ../../src/main/python/dpp_test.py::^[[1mtest_dpp_via_aggregate_subquery[true-5-parquet][DATAGEN_SEED=1704919540, INJECT_OOM, IGNORE_ORDER]^[[0m - py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.s...
2024-01-10T23:39:01.0964581Z [2024-01-10T23:34:21.452Z] [2024-01-10T23:33:07.595Z] ^[[31mFAILED^[[0m ../../src/main/python/dpp_test.py::^[[1mtest_dpp_via_aggregate_subquery[true-5-orc][DATAGEN_SEED=1704919540, IGNORE_ORDER]^[[0m - py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.s...

assertion failed: Could not find DynamicPruningExpression in the **** plan

This seems to be related to #10147

@andygrove
Copy link
Contributor Author

I filed #10182 for the dpp issue

@andygrove
Copy link
Contributor Author

build

@andygrove andygrove merged commit 490b77c into NVIDIA:branch-24.02 Jan 11, 2024
@andygrove andygrove deleted the fix-json-timestamp branch January 11, 2024 14:40
@sameerz sameerz added bug Something isn't working and removed build Related to CI / CD or cleanly building labels Jan 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] json_test.py::test_from_json_struct_timestamp failed on: Part of the plan is not columnar
5 participants