We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Describe the bug
Column order needs to be written in FileMetadata, see thrift spec:
https://github.com/apache/parquet-format/blob/066f9817332da32bdc6dc6dea833b6ee9c269934/src/main/thrift/parquet.thrift#L1133-L1149
To Reproduce
In arrow_writer/mod.rs:
arrow_writer/mod.rs
#[test] fn test_123() { let a = Int32Array::from(vec![1, 2, 3, 4, 5]); let b = IntervalDayTimeArray::from(vec![0; 5]); let batch = RecordBatch::try_from_iter(vec![ ("a", Arc::new(a) as ArrayRef), ("b", Arc::new(b) as ArrayRef), ]) .unwrap(); let mut buf = Vec::with_capacity(1024); let mut writer = ArrowWriter::try_new(&mut buf, batch.schema(), None).unwrap(); writer.write(&batch).unwrap(); writer.close().unwrap(); let bytes = Bytes::from(buf); let options = ReadOptionsBuilder::new().with_page_index().build(); let reader = SerializedFileReader::new_with_options(bytes, options).unwrap(); dbg!(reader.metadata().file_metadata().column_orders()); }
Currently outputting None
None
Expected behavior
Should write ColumnOrder for each column
Additional context
See here:
arrow-rs/parquet/src/file/writer.rs
Lines 326 to 336 in 6d4b8bb
Currently being set to None always when writing
The text was updated successfully, but these errors were encountered:
Planning to take a shot at this
Sorry, something went wrong.
label_issue.py automatically added labels {'parquet'} from #5147
label_issue.py
Successfully merging a pull request may close this issue.
Describe the bug
Column order needs to be written in FileMetadata, see thrift spec:
https://github.com/apache/parquet-format/blob/066f9817332da32bdc6dc6dea833b6ee9c269934/src/main/thrift/parquet.thrift#L1133-L1149
To Reproduce
In
arrow_writer/mod.rs
:Currently outputting
None
Expected behavior
Should write ColumnOrder for each column
Additional context
See here:
arrow-rs/parquet/src/file/writer.rs
Lines 326 to 336 in 6d4b8bb
Currently being set to
None
always when writingThe text was updated successfully, but these errors were encountered: