Generic geometry for GeoArrow encoded columns.

[ninok]

Hi folks,

I was just reading through the 1.1 spec.which will add GeoArrow support.
It seems, that only columns that are restricted to a certain geometry type, like point or polygon, can use this encoding.
Why do we have this restriction? For WKB this restriction does not exist, since it encodes the type of the geometry.
Wouldn't it make sense to also just encode the geometry type for GeoArrow as well (i.e. one byte in the beginning).
From a performance point of view it should not make a difference. After reading the type an implementation can cast to the concrete type or just ignore the type,if the whole column has a non-generic geometry type.

[ninok]

Additionally I think we are mixing encoding and column type in the new spec.
Instead of e.g. "point" it should be "native"( or arrow or however we want to call it) and the column_type should be point.
How are we going to support other encodings in the future, e.g. non-interleaved and interleaved arrow?

[paleolimbot]

We went back and forth on that one in the initial discussion, and I think there was not appetite for ever having more than one encoding for a "point", for example, and so we decided to keep it all in the encoding field for now. If it ever comes up we need another way to encode a point, we can add another encoding (but I don't envision there being any motivation to do so in the near future).

[kylebarron]

Parquet doesn't have a union type, so it's hard to encode native geometries of unknown type in the way GeoArrow defines them.

[ninok]

Oh, I see. That's a pity and will block certain use cases, e.g. having rivers in a column, which are sometimes linestrings and sometimes polygons, depending on their size.
Is there any timeline when this support is planned(if it is planned)?

A workaround could be, to use the most generic type(multipolygon) for the encoding (i.e. a point would be multipolygon with 1 polygon which has 1 ring with 1 point), but that's quite hacky(i.e. invalid geometries) and would defeat the purpose of having an encoding that matches the native memory layout.

So I guess we just have to wait for union types.

[jiayuasu]

This is an issue we have been aware of. In particular, this has been happening to the Overture waterbody data.

It looks like that the Union type might never get merged (apache/parquet-format#44). We should work with the GeoArrow spec to figure out an alternative implementation without Union.

[kylebarron]

I've been working on support for generic geometry in GeoArrow for over a year: geoarrow/geoarrow#43 (and it's ready to merge pending final approvals). Because Arrow natively supports unions, it is much easier to define generic geometries in terms of an Arrow union. That means it's zero-copy to an Arrow union array, and also quite easy to cast to an array of simpler type.

I think it's fine to require generic geometries to be serialized to WKB. Especially because GeoParquet keeps metadata about the geometry type, so a GeoParquet to GeoArrow reader can know ahead of time which GeoArrow type to parse the WKB into.

[paleolimbot]

It is true that Arrow implementations provide some tools to deal with unions and geospatial applications might be able to leverage them if the Arrow type happens to be a union. For anybody that wants to use a Union to represent arbitrary GeoArrow encoded things, geoarrow/geoarrow#43 is almost certainly the way to go. That said, I am not sure under what conditions it actually makes sense to do this (and I'm not sure we've sorted this either!).

[paleolimbot]

We're just settling in to using GeoArrow to encode things in Parquet and I think we're still learning about what is or isn't faster. Eventually, we will probably do one of:

• Keep writing WKB for anything that can't work with a single-type geometry column. Maybe it's faster? Maybe it won't make a difference? I don't think we actually know right now and at least this version interoperates with other spatial software.
• Use an encoding that keeps all of the coordinates in a single array and encodes the geometry type at some level of nesting ("Spatial Parquet" ish)
• Use an encoding that separates coordinates of various geometry types ("Union" ish).