Spec: Support geo type

[szehon-ho]

This is the spec change for #10260.

Also this is based closely on the decisions taken in the Parquet proposal for the same : apache/parquet-format#240

[szehon-ho]

@jiayuasu do you have any suggestion for default here?

[hsiang-c]

GeoMesa has 12 as the default: https://github.com/locationtech/geomesa/blob/geomesa-4.0.3/geomesa-z3/src/main/scala/org/locationtech/geomesa/curve/XZSFC.scala#L13

[jiayuasu]

12 sounds fine. CC @Kontinuation

[Kontinuation]

GeoMesa uses a high XZ2 resolution when working with key-value stores such as Accumulo and HBase, it is not appropriate to always use a resolution that high for partitioning data (for instance, GeoMesa on FileSystems).

XZ2 resolution 11~12 works for city-scale data, but will generate too many partitions for country-scale or world-scale data. I'd like to have a smaller default value such as 7 to be safe on various kinds of data.

[szehon-ho]

I could add this section later too, once its implemented (same for ORC below)

[RussellSpitzer]

[Appendix G](#appendix-g)

[rdblue]

How does this work? Does Iceberg need to interpret each WKB to produce this value? Will it be provided by Parquet?

[szehon-ho]

Yes, once we switch to Geometry logical type from Parquet we will get these stats from Parquet.

[flyrain]

Should we mention that it is the parquet type BoundingBox?

[szehon-ho]

yea will add a footnote here

[jiayuasu]

@szehon-ho BTW, the reason why we had a separate bbox statistics in havasu is to be compatible with existing Iceberg tables. Since this is to add the native geometry support, so lower_bound and upper_bound are good choices.

[Kontinuation]

I thought that the bounds were stored as WKB-encoded points (according to Appendix D and G), and WKB encodes dimensions of geometries in the header. It is more consistent to make bound values the same type/representation as the field data type.

More sophisticated coverings in Parquet statistics cannot be easily mapped to lower_bounds and upper_bounds, so do we simply use the bbox statistics and ignore the coverings for now?

[zhangfengcdt]

I think it is ok since these two bounds are optional and in case they are not presented, it still follow the spec.

[jiayuasu]

Does it support different dimensions like XY, XYZ, XYM, XYZM? If yes, how can we tell if the binary is for XYZ or XYM?

We should say For Geometry type, this is a WKB-encoded Point composed of the min value of each dimension in all Points in the Geometry. Then we don't have to worry about the Z and M value.

CC @szehon-ho

[szehon-ho]

We should say For Geometry type, this is a WKB-encoded Point composed of the min value of each dimension in all Points in the Geometry. Then we don't have to worry about the Z and M value.

@jiayuasu @Kontinuation @wgtmac Done, thanks.

[szehon-ho]

@flyrain

Should we mention that it is the parquet type BoundingBox?

Actually looking again after some time, not sure how to mention this here, as that is filetype specific. This is an optional field, and only set if type is parquet and bounding_box is set, but that's implementation detail .

[rdblue]

Except for geometry?

[szehon-ho]

@rdblue @flyrain hmm is there a specific reason? I think it could work technically as Geometry can be compared, unless I miss something

[flyrain]

Maybe that's fine if it is comparable, but practically people will always use xz2, right? I'm not sure, but wondering if there is some implications, e.g., too expensive, or super high cardinality, so that we don't recommend user to use the original GEO value as the partition spec.

[szehon-ho]

yea I think its possible to do (it's just the wkb value after all), you are right , not sure if any good use case. Yea we have to get the wkb in any case, i am not sure if its that expensive, but can check. But I guess the cardinality is the same consideration as any other type (uuid for example), and we let the user choose ?

[flyrain]

Are we going to support bucketing on GEO?

[szehon-ho]

I think its possible, again not sure the utility. Geo boils down to just WKB bytes

[flyrain]

I feel that the argument for identity can apply here as well. In that case, we can support it, but it's users' call to use it or not.

[rdblue]

We may want to change this to be like identity, using Any except [...].

I would not include geo as a source column for bucketing because there is not a clear definition of equality for geo. The hash would depend on the structure of the object and weird things happen when two objects are "equal" (for some definition) but have different hash values.

[jiayuasu]

1. When we say OGC:CRS84, the value you put in this field should be the following PROJJSON string (see GeoParquet spec)

{
    "$schema": "https://proj.org/schemas/v0.5/projjson.schema.json",
    "type": "GeographicCRS",
    "name": "WGS 84 longitude-latitude",
    "datum": {
        "type": "GeodeticReferenceFrame",
        "name": "World Geodetic System 1984",
        "ellipsoid": {
            "name": "WGS 84",
            "semi_major_axis": 6378137,
            "inverse_flattening": 298.257223563
        }
    },
    "coordinate_system": {
        "subtype": "ellipsoidal",
        "axis": [
        {
            "name": "Geodetic longitude",
            "abbreviation": "Lon",
            "direction": "east",
            "unit": "degree"
        },
        {
            "name": "Geodetic latitude",
            "abbreviation": "Lat",
            "direction": "north",
            "unit": "degree"
        }
        ]
    },
    "id": {
        "authority": "OGC",
        "code": "CRS84"
    }
}

2. Both crs and crs_kind field are optional. But when the CRS field presents, the crs_kind field must present. In this case, since we hard code this crs field in this phase, then we need to set crs_kind field (string) to PROJJSON.

[wgtmac]

Should we include this example in the parquet spec as well?

[wgtmac]

BTW, should we advise accepted forms or values for CRS and Edges?

[jiayuasu]

@wgtmac We should add this value to the Parquet spec for sure. CC @zhangfengcdt

@szehon-ho There is another situation mentioned in the GeoParquet spec: If the CRS field presents but its value is null, it means the data is in unknown CRS. This situation happens sometimes because the writer somehow cannot find or lose the CRS info. Do we want to support this? I think we can use the empty string to cover this case

[jiayuasu]

accepted forms or values for CRS and Edges

If we borrow the conclusion from Parquet Geometry proposal, then C,T,E fields are the follows:

C is a string. Based on what I understand from this PR, @szehon-ho made this field a required field, which is fine.
T is optional and a string. Currently, it only allows this value PROJJSON. When it is not provided, it defaults to PROJJSON too.
E is a string. The only allowed value is PLANAR in this phase. Based on what I understand from this PR, @szehon-ho made this field a required field, which is fine. @szehon-ho According to our meeting with Snowflake, I think maybe we can allow SPHERICAL too? We can add in the spec that: currently it is unsafe to perform partition transform / bounding box filtering when E = SPHERICAL because they are built based on PLANAR edges. It is the reader's responsibility to decide if they want to use partition transform / bounding box filtering.

[jiayuasu]

@szehon-ho

1. Does it make sense to include the following CRS84 example from the Parquet Geometry PR?

  /**
   * Coordinate Reference System, i.e. mapping of how coordinates refer to
   * precise locations on earth. Writers are not required to set this field.
   * Once crs is set, crs_encoding field below MUST be set together.
   * For example, "OGC:CRS84" can be set in the form of PROJJSON as below:
   * {
   *     "$schema": "https://proj.org/schemas/v0.5/projjson.schema.json",
   *     "type": "GeographicCRS",
   *     "name": "WGS 84 longitude-latitude",
   *     "datum": {
   *         "type": "GeodeticReferenceFrame",
   *         "name": "World Geodetic System 1984",
   *         "ellipsoid": {
   *             "name": "WGS 84",
   *             "semi_major_axis": 6378137,
   *             "inverse_flattening": 298.257223563
   *         }
   *     },
   *     "coordinate_system": {
   *         "subtype": "ellipsoidal",
   *         "axis": [
   *         {
   *             "name": "Geodetic longitude",
   *             "abbreviation": "Lon",
   *             "direction": "east",
   *             "unit": "degree"
   *         },
   *         {
   *             "name": "Geodetic latitude",
   *             "abbreviation": "Lat",
   *             "direction": "north",
   *             "unit": "degree"
   *         }
   *         ]
   *     },
   *     "id": {
   *         "authority": "OGC",
   *         "code": "CRS84"
   *     }
   * }
   */

2. It is ok to have them all fixed to default values for this phase.

[szehon-ho]

@jiayuasu i put it in the example (if you render the page). let me know if its not what you meant.

[desruisseaux]

Note that this example is non-standard. A CRS JSON encoding is in progress at OGC, but we don't know yet what it will look like. The only standards at this time are GML, WKT, or simply put a reference to the EPSG database or OGC registry (e.g. https://www.opengis.net/def/crs/OGC/0/CRS84 or "urn:ogc:def:crs:OGC::CRS84").

[desruisseaux]

Coordinate Reference System, i.e. mapping of how coordinates refer to precise locations on earth. Defaults to "OGC:CRS84".

Maybe remove the word "precise"? Not all CRS are precise. The proposed default, OGC:CRS84, has an uncertainty of about 2 meters (unrelated to floating point precision). Maybe it would be worth to change the sentence to: Defaults to "OGC:CRS84", which provides an accuracy of about 2 meters.

Another reason for removing the "precise" word is that the CRS alone is not sufficient for really precise locations. We also need the epoch, which is a topic that has not been covered at all in this PR.

[szehon-ho]

Removed precise

[wgtmac]

What syntax to use for an engine to create the geometry type? Does it require C/T/E to appear in the type?

[szehon-ho]

Related to above comment, I think these will all be optional (take a default value if not specified).

[szehon-ho]

Hi all fyi i have unfortunately encountered some problems while remote and probably cant update this, will come back to this after i get back home in two weeks.

[szehon-ho]

@jiayuasu @Kontinuation @wgtmac @flyrain @rdblue sorry for the delay, as I only got access now. Updated the pr.

[dmitrykoval]

Can we maybe explicitly mention here that both "planar" and "spherical" are supported as edge type enum values?

[szehon-ho]

@dmitrykoval i was debating this.

I guess we talked about it before, but the Java reference implementation, we cannot easily do pruning (file level, row level, or partition level) because the JTS library and the XZ2 only support non-spherical. We would need new metrics types, new Java libraries , and new partition transform proposals if we wanted to support it in Java reference implementation.

But if we want to support it, Im ok to list it here and have checks to just skip pruning for spherical geometry columns.
@flyrain @jiayuasu @Kontinuation does it make sense?

[dmitrykoval]

I see. I think if "planar" is the default edge type, then there shouldn't be many changes to the planar geometry code path, except for additional checks to skip some partitioning/pruning cases, right?

Regarding the reference implementation of the "spherical" type, do we need to fully support it from day one, or can we maybe mark it as optional in the initial version of the spec? For example, it would work if the engine supports it, but by default, we would fall back to the planar edge type?

[jiayuasu]

We could list spherical as an allowed edge type here. Maybe just mark it that it is not safe to perform partition transform or lower_bound/upper_bound filtering when the edge is spherical. We did the same in the Parquet Geometry PR.

[szehon-ho]

Yea , forgot to mention explicitly that in Iceberg, pruning is always an optional feature for reads, so no issue.

[desruisseaux]

Crs-encoding (coordinate reference system encoding) is the type of crs field.

• Please change "is the type" by "is the encoding". The CRS type is something completely different than the information put in this field.
• (minor detail) "crs" should be upper-case "CRS" as it is an acronym for "Coordinate Reference System".

Defaults to "PROJJSON".

I suggest to remove this specification. PROJJSON is non-standard and may never be (we don't know yet what will be OGC's decision). I don't think that we want to tie Iceberg forever to a format specific to one project.

[wgtmac]

For geometry type, this is a WKB-encoded point composed of the min value of each dimension among all component points of all geometry objects for the file.

As we are finishing the PoC on the Parquet side, the remaining issue is what value to write to min_value/max_value fields of statistics and page index. To give some context, Parquet requires min_value/max_value fields to be set for page index and statistics are used to generate page index. The C++ PoC is omitting min_value/max_value values and the Java PoC is pretending geometry values are plain binary values while collecting the stats. Should we do similar things here? Then the Iceberg code can directly consume min_value/max_value from statistics instead of issuing another call to get the specialized GeometryStatistics which is designed for advanced purpose.

@Kontinuation @zhangfengcdt @paleolimbot @jiayuasu

[jiayuasu]

@wgtmac do you mean that Iceberg uses the Parquet Geometry GeometryStatistics or Parquet Geometry uses the min_value/max_value idea from Iceberg?

[wgtmac]

I mean the latter. The ColumnOrder of the new geometry type is undefined as specified at https://github.com/apache/parquet-format/pull/240/files#diff-834c5a8d91719350b20995ad99d1cb6d8d68332b9ac35694f40e375bdb2d3e7cR1144. It means that the min_value/max_value fields are meaningless and should not be used. I'm not sure if it is a good idea to set min_value/max_value fields in the same way as lower_bounds/upper_bounds of Iceberg.

[Kontinuation]

I suggest defining the sort order of geometry columns as WKB-encoded points in the parquet format spec. This is the most simple yet useful way of defining the min and max bounds for geometry columns, and the sort order is better to be well-defined rather than left undefined.

[wgtmac]

I agree that it is better to explicitly define the column order than being undefined. If we go with this approach, the format PR and two PoC impls need to reflect this change, which might get more complicated.

[szehon-ho]

Is there anything for this specific line we need to change? As long as we get from Parquet some way we are ok here, but is the format of the lower/upper bound still ok?

[wgtmac]

No, I was thinking if Parquet could do better by doing similar things in the future.

[RussellSpitzer]

I think maybe we should just link out for the requirements here since it's a bit complicated.

The description as well could be

Simple feature geometry Appendix G, Parameterized by ....

I also don't think we should allow it to be unset ... can we just require that a subclass is always picked? We could recommend a set of defaults for engines to set on field creation but I'm not sure we need to be that opinionated here.

[rdblue]

Like above, this table covers arbitrary geometry and geography objects. I think it is fine to use WKT here and not have a separate way to store points, but we should be aware that the implication is that WKT is required for any JSON representation.

[flyrain]

LGTM

[dmitrykoval]

Just a minor clarification: since we are not specifying the default value for A, if the user provides a single argument geography specialization, it would be treated as geography(A), correct?

[szehon-ho]

That's a good point, what do you think to put 'spherical' as default? cc @jiayuasu @mkaravel @paleolimbot as well

[dmitrykoval]

I think making 'spherical' a default is a good idea.

[szehon-ho]

Done.

[dmitrykoval]

Thanks @szehon-ho!

[paleolimbot]

No objections from me in either direction!

[jiayuasu]

LGTM！

[dmitrykoval]

lgtm

[milastdbx]

What does Custom CRS mean ? Is it any non-default CRS value ?

Can values of CRS be of format format $authorithy:$identifier similar to that of default value ?

[rdblue]

Yes, $authority:$identifier is the intent.

[rdblue]

Nit: I think the idea is that this is an interpolated string, but we don't use $ elsewhere so people may embed the character. What about simply "type:identifer, where type is ..."

[wgtmac]

Is it accurate to use Geometry features for geography type? Or Geometry features is a standard term?

[szehon-ho]

Fixed to use geospatial, as discussed offline

[jiayuasu]

LGTM. Thanks for updating this!

[rdblue]

Thanks!

[redblackcoder]

There is some discrepency in the description here. CRS values are mentioned to be specified in two format:

1. srid:identifier
2. projjson: identifier

The default value of OGC:CRS84 doesn't follow the pattern. If OGC:CRS84 is the identifier itself, then should be mentioend as srid:OGC:CRS84?

[jiayuasu]

@redblackcoder OGC:CRS84 is not an identifier. It is corresponding to the following projjson string: https://github.com/opengeospatial/geoparquet/blob/main/format-specs/geoparquet.md#ogccrs84-details

It is also equivalent to srid:4326 but flip the axis order to lon/lat

[redblackcoder]

Isn't SRID database specific? From the open specification, it is like following:
SRID, AUTH_NAME, AUTH_SRID, SRTEXT

OGC:CRS84 corresponds to AUTH_NAME=OGC and AUTH_SRID=CRS84
Not sure what does SRID:4326 mean from Auth_Name and Auth_Srid perspective.

[redblackcoder]

Does SRID:4326 mean EPSG:4326? Is it common to assume EPSG as the default authority for the codes?

[desruisseaux]

Does SRID:4326 mean EPSG:4326? Is it common to assume EPSG as the default authority for the codes?

Not exactly. SRID usually means "the primary key in the context of those data". In a spatial database, this is the primary key of the spatial_ref_sys table. Theoretically, this is independent of EPSG codes. However, it is a common practice to use the same numerical values for convenience.

For mapping SRID to EPSG code in a spatial database, we need to look at the other columns. If and only if the value of AUTH_NAME is EPSG, then the value of AUTH_SRID is the EPSG code. But we could also have the IAU value in the authority column, in which case the authority code is for some astronomical body (Moon, Mars, Jupiter, etc.).

Note that even when SRID = 4326, AUTH_NAME = "EPSG" and AUTH_SRID = 4326, we do not really have "SRID:4326" = "EPSG:4326" because the authoritative definition of the latter has (latitude, longitude) axis order. Any other axis order is not EPSG:4326, but rather some derivative of EPSG:4326. Therefore, SRID:4326 should rather be understood as "related to EPSG:4326" instead of "same as EPSG:4326".

[redblackcoder]

Thanks for the explanation.
In absense of any spec around spatial_ref_sys table for iceberg, should the spec stick with AUTH_NAME:AUTH_SRID as the identifier when using srid type as the custom CRS? Currently, it is skipping the AUTH_NAME from the identifier and assuming AUTH_SRID to be unique across all.

[redblackcoder]

Do lower_bounds and upper_bounds mean the bounding box for the geospatial data points in the file? Can it be expanded to explain that bounds are based on a bounding box and these values are corners of the bounding box. Maybe this can be added to the Appendix.