feat(arrow): add GeoArrow utils for ArrowLoader #2738
Conversation
Signed-off-by: Xun Li <[email protected]>
|
|
||
| // get geometry columns from arrow table | ||
| export function getGeometryColumnsFromArrowTable(arrowTable: Table): Map<string, {encoding: string}> { | ||
| const geometryColumns = new Map<string, any>(); |
There was a problem hiding this comment.
Better typing than any?
| function updateBoundsFromGeoArrowSamples( | ||
| flatCoords: Float64Array, | ||
| nDim: number, | ||
| bounds: [number, number, number, number], | ||
| sampleSize: number = 100 | ||
| ): void { | ||
| const numberOfFeatures = flatCoords.length / nDim; | ||
| const sampleStep = Math.max(Math.floor(numberOfFeatures / sampleSize), 1); | ||
|
|
||
| for (let i = 0; i < numberOfFeatures; i += sampleStep) { | ||
| const lng = flatCoords[i * nDim]; | ||
| const lat = flatCoords[i * nDim + 1]; | ||
| if (lng < bounds[0]) { | ||
| bounds[0] = lng; | ||
| } | ||
| if (lat < bounds[1]) { | ||
| bounds[1] = lat; | ||
| } | ||
| if (lng > bounds[2]) { | ||
| bounds[2] = lng; | ||
| } | ||
| if (lat > bounds[3]) { | ||
| bounds[3] = lat; | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
I haven't yet but plan to create a repo like geoarrow-js to consolidate some operations like this. I implemented this "totalBounds" operation directly on an Arrow JS Vector here: https://github.com/geoarrow/deck.gl-layers/blob/8449634e9403f8159b987264fa35b1085affbbc0/src/geoarrow/bbox.ts
| const ringData = polygonData.children[0]; | ||
| const pointData = ringData.children[0]; | ||
| const coordData = pointData.children[0]; | ||
| const nDim = pointData.stride; |
There was a problem hiding this comment.
I found that accessing .children directly didn't maintain typing information. In deck.gl-layers I have helpers to maintain strong typing for geometry types: https://github.com/geoarrow/deck.gl-layers/blob/8449634e9403f8159b987264fa35b1085affbbc0/src/utils.ts#L468-L471
There was a problem hiding this comment.
Is the typing for pointData.type.listSize? Should the strideForType() or pointData.stride take care of it? See https://github.com/apache/arrow/blob/f2995b0eb9e6792a2cca8ce9e8d9b933d4a9bbfd/js/src/type.ts#L701 Thanks!
There was a problem hiding this comment.
I use type.listSize myself. I haven't used stride
| const coordData = pointData.children[0]; | ||
| const nDim = pointData.stride; | ||
| const geomOffset = ringData.valueOffsets; | ||
| const flatCoordinateArray = coordData.values; |
There was a problem hiding this comment.
For arbitrary geoarrow input, you should assert that the low-level coords are interleaved, not separated (i.e. a struct of x and y columns) because geoarrow allows both in its spec
There was a problem hiding this comment.
Good suggestion! Will update. Thanks!
|
This was closed by #2744. Unfortunately, I suspect some of the comments provided on this PR were not addressed there. |
Add GeoArrow util functions:
The geometry types supported: (see https://github.com/geoarrow/geoarrow/blob/main/extension-types.md#extension-names):
These functions are converted from keplergl/kepler.gl#2385.
ToDos:
tableFromIPCin test casesgetBinaryGeometriesFromArrowgetGeometryColumnsFromArrowTablegetGeoArrowEncoding