-
Notifications
You must be signed in to change notification settings - Fork 877
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement parsing for ParquetWriterOptions #4693
Comments
Add doc link reference to apache/arrow-rs#4693
Add doc link reference to apache/arrow-rs#4693
Looks you want the last like this: let writer_properties = WriterProperties::from_str("
set parquet.writer_version = 2.0
set parquet.compression = zstd(5)").unwrap(); I want to try it. and should we really need the prefix word 'set' and |
Hi @fansehep -- thank you.
No. I am sorry for the confusion, the so for example, we would implement // call `parse` to implement FromStr
let encoding: Encoding = "PLAIN".parse().unwrap();
// call parse again, case insensitive
let encoding: Encoding = "plain".parse().unwrap(); |
Thanks for your help. 😃 |
Thank YOU! |
This one seems to be finished by: @alamb is there anything missing from what you were expecting? Can we close it? |
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
We are implementing configurable parquet writing in DataFusion
We want to be able to allow users to specify the parquet writing options (like compression) via a string like
Describe the solution you'd like
Implement
FromStr
for the following structures, with some tests.Each of these can be done via a separate PR
The basic code can probably be ported from DataFusion here and add some unit tests: https://github.com/apache/arrow-datafusion/blob/ed85abbb878ef3d60e43797376cb9a40955cd89a/datafusion/core/src/datasource/file_format/parquet.rs#L13
Bonus points for good error messages that give example values (like "Invalid encoding. Valid values: plain_dictionary, rle, etc
Describe alternatives you've considered
Additional context
@devinjdangelo implemented parsing for these in apache/datafusion#7244 however, I think these features could be more generally useful to others
The text was updated successfully, but these errors were encountered: