Support Reading Nested List Arrays

[andrei-ionescu]

Describe the bug

When trying to read parquet files with deeply nested fields we end up with the following error:

Parquet reader thread terminated due to error: ParquetError(ArrowError("reading List(GroupType {
...
}) into arrow not supported yet"))

To Reproduce

This is easily visible from the code found at array_reader.rs#L1516-L1522

Expected behavior

To have support for reading nested parquet files into arrow.

Additional context

This issue, in my particular case, has been hidden under #1383.

[alamb]

This may have been fixed as there has been significant work on reading of nested structures by @helgikrs and others

[Igosuki]

@alamb https://github.com/apache/arrow-rs/blob/master/parquet/src/arrow/array_reader.rs#L1622

[kesavkolla]

The nested data structure support is been pending for long time. I am also eagerly looking for this support. I have a fairly large JSON data which I used apache spark to write to parquet files. I wanted to read them via rust arrow have no success.

[alamb]

Hi @kesavkolla -- could you possibly share an example file we could use to test any fix?

[Igosuki]

I have a parquet file https://github.com/Igosuki/arrow2/blob/main/part-00000-b4749aa1-94e4-4ddb-bab2-954c4d3a290f.c000.snappy.parquet it uses List(List(Float64))

[chauhanVritul]

Here is a sample parquet file https://github.com/chauhanVritul/sampleparquet/blob/main/part-00000-8e5acb24-eb4e-491c-8c85-88799f25d1f0-c000.snappy.parquet

[Igosuki]

Yes, this use case is supported in datafusion. For instance, you can read nested avro and json schemas, so the issue should be raised with arrow-rs, I added a comment here #720 with our files.