refactor: parallelize parquet_exec test case single_file

[waynexia]

Signed-off-by: Ruihang Xia [email protected]

Which issue does this PR close?

Closes #.

Rationale for this change

I found this test is very slow (the last one datafusion::parquet_exec parquet::filter_pushdown::single_file):

        PASS [   0.061s]        datafusion-benchmarks::bin/tpch tests::run_q5
        PASS [   0.091s]        datafusion-benchmarks::bin/tpch tests::run_q21
        PASS [   0.082s]        datafusion-benchmarks::bin/tpch tests::run_q9
        PASS [   0.056s]          datafusion::user_defined_plan normal_query_without_schemas
        PASS [   0.047s]          datafusion::user_defined_plan topk_plan
        PASS [   0.081s]          datafusion::user_defined_plan normal_query
        PASS [   0.015s]          datafusion::user_defined_plan topk_query
        PASS [   0.120s]        datafusion-benchmarks::bin/tpch tests::run_q7
        PASS [   0.112s]    datafusion::user_defined_aggregates test_udf_returning_struct_sq
        PASS [   0.114s]    datafusion::user_defined_aggregates test_udf_returning_struct
        PASS [   0.131s]        datafusion-benchmarks::bin/tpch tests::run_q8
        PASS [   1.090s]             datafusion::tpcds_planning tpcds_physical_q88
        PASS [  23.772s]               datafusion::parquet_exec parquet::filter_pushdown::single_file
------------
     Summary [  27.234s] 2669 tests run: 2667 passed, 2 failed, 30 skipped

and always hold the whole test process for seconds.

What changes are included in this PR?

This test is composed of serval little cases. I change it to run them in parallel, making the unit test faster by ~10s.

        PASS [   0.082s]    datafusion::user_defined_aggregates test_udf_returning_struct
        PASS [   0.023s]          datafusion::user_defined_plan topk_plan
        PASS [   0.070s]          datafusion::user_defined_plan normal_query_without_schemas
        PASS [   0.013s]          datafusion::user_defined_plan topk_query
        PASS [   0.086s]          datafusion::user_defined_plan normal_query
        PASS [   0.101s]        datafusion-benchmarks::bin/tpch tests::run_q9
        PASS [   0.116s]        datafusion-benchmarks::bin/tpch tests::run_q7
        PASS [   0.119s]        datafusion-benchmarks::bin/tpch tests::run_q8
        PASS [   0.905s]             datafusion::tpcds_planning tpcds_physical_q88
------------
     Summary [  16.256s] 2669 tests run: 2667 passed, 2 failed, 30 skipped

Are these changes tested?

by it self

Are there any user-facing changes?

no

[alamb]

Makes sense to me -- thank you @waynexia . I think the return code from the task needs to be verified but otherwise this look good to me. Thank you

I like how it will use more threads if available and degrade gracefully if not.

On my machine, master takes 20s
≈com

cargo test -p datafusion --test parquet_exec
...
test result: ok. 52 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 20.93s

With this PR, it takes 5.6s 👨‍🍳 👌

cargo test -p datafusion --test parquet_exec
...
test result: ok. 52 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.68s

cc @tustvold as he previously had some concerns with parallelizing this test, but I think his original concerns were related to writing the test file multiple times, which this PR does not do.

[tustvold]

This looks good to me, other than the panic handling @alamb has already highlighted. I think we could potentially also look to reduce the page and row group size of the writer, so that we can potentially get the same coverage using a significantly smaller parquet file

[waynexia]

Thanks for your review!

I think we could potentially also look to reduce the page and row group size of the writer, so that we can potentially get the same coverage using a significantly smaller parquet file

I try to set row_limit to 4096 in 8799a64, and some test results change. But I'm not sure about if those changes are expected... Could you please have a look @tustvold?

[tustvold]

This wasn't quite what I meant, this just alters the internal batch size that the file is generated in. I'll add it to my list to get a PR in to adjust the generator to not generate such unnecessarily large files

[alamb]

Note I carefully chose the values of the predicates in this test to cover various cases based on how the values were distributed in the data generator. I am sure it will be possible to reduce the size of the data file, but it will require some carefulness to ensure the coverage is retained

[tustvold]

PR in #4743

[ursabot]

Benchmark runs are scheduled for baseline = 734c211 and contender = 38a24c0. 38a24c0 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java