test: add test for decimal and pruning for decimal column #2960
Conversation
|
|
||
| // parquet use the fixed length binary as the physical type to store decimal | ||
| // TODO: arrow-rs don't support convert the physical type of binary to decimal | ||
| // let exec = get_exec("byte_array_decimal.parquet", None, None).await?; |
There was a problem hiding this comment.
arrow-rs doesn't support read decimal value from parquet with binary physical type.
why not support this?
There was a problem hiding this comment.
Why would you have a variable length decimal type? What would this mean?
There was a problem hiding this comment.
@tustvold
In the parquet of Java version, we can write decimal with the binary or fixed-length binary, so we can get the decimal using the physical type of binary.
we can get the definition of decimal from the format spec
Decimal can be stored in the parquet as INT32, INT64, FIXED_LEN_BYTE_ARRAY or BYTE_ARRAY.
I can help to implement the BYTE_ARRAY in the arrow-rs.
There was a problem hiding this comment.
pr apache/arrow-rs#2160 for supporting read decimal from binary physical type in parquet.
liukun4515
force-pushed
the
add_decimal_parquet_prune_test
branch
from
ee1cc40 to
bb5dece
Compare
Codecov Report
@@ Coverage Diff @@
## master #2960 +/- ##
==========================================
+ Coverage 85.43% 85.64% +0.21%
==========================================
Files 275 279 +4
Lines 49839 51033 +1194
==========================================
+ Hits 42578 43707 +1129
- Misses 7261 7326 +65
Help us with your feedback. Take ten seconds to tell us how you rate us. |
| 23, | ||
| 2, | ||
| ), | ||
| ); |
There was a problem hiding this comment.
Can you add some large decimal numbers to test the decimal(23,2) cases?
There was a problem hiding this comment.
This test case is just used to test the pruning logic.
I will file follow-up pull request to fix the #2962 with parquet rowgroup filter/prune.
liukun4515
changed the title
|
Thanks @liukun4515 |
Co-authored-by: Andrew Lamb <[email protected]>
|
Benchmark runs are scheduled for baseline = f386f7a and contender = 0f19990. 0f19990 is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
Which issue does this PR close?
Closes #.
Rationale for this change
What changes are included in this PR?
Are there any user-facing changes?