Request for additional FeatureCollection documentation #2
Comments
|
I found another related issue. It looks like each I don't really have experience administering a server, but I thought that generally a Feature Service ("buildings_frankfurt" above) can only host one type of geometry. Is mixed geometry possible? If not, why is there a |
|
Sorry for the late response @thw0rted! Thought I was subscribed to this repo 😅
Mixed geometry is not possible in this case. Let me follow-up with that. In the JSAPI parser we don't actually parse this field at all. (We don't release that parser directly because it's fairly difficult to use and very specific to our internal workflows, basically we manually iterate over the pbf payload to build an offset table and use that to directly read out values lazily, overkill for most use cases but needed on the API side to avoid per-feature allocations. Also avoid parsing anything we don't explicitly use). For decoding the fields, this might help. This is what the payloads look side-by-side when parsed literally into JSON:
We can see that the message originating from PBF is more structured, and to get to the actual features, you needed to access queryResult.featureResult.features (whereas in f=pbf the featureResult is an anonymous object). "FeatureCollection" you can basically think of as the namespace for all the f=pbf formats that may be supported by feature services in the future. Currently only qureyResult.featureResult is supported. Opening up a single feature we get: At this point you are probably asking how we know that
We can see that the first field is objectId, and the second is population. |
|
Thanks for replying @mmgeorge . It sounds like some of the fields in the PBF definition aren't actually used. Would it make sense to trim them out of the definition? Or maybe just update the docs to note that the fields are deprecated / legacy / reserved / whatever? Did you see my question about applying the For comparison, this is
{
"objectIdFieldName" : "OBJECTID",
"uniqueIdField" :
{
"name" : "OBJECTID",
"isSystemMaintained" : true
},
"globalIdFieldName" : "",
"geometryProperties" :
{
"shapeAreaFieldName" : "Shape__Area",
"shapeLengthFieldName" : "Shape__Length",
"units" : "esriDecimalDegrees"
},
"geometryType" : "esriGeometryPolygon",
"spatialReference" : {
"wkid" : 4326,
"latestWkid" : 4326
},
"fields" : [
{
"name" : "name",
"type" : "esriFieldTypeString",
"alias" : "name",
"sqlType" : "sqlTypeOther",
"length" : 2048,
"domain" : null,
"defaultValue" : null
}
],
"features" : [
{
"attributes" : {
"name" : "Commerzbank DLZ 1"
},
"geometry" :
{
"rings" :
[
[
[8.65442100000007, 50.1062332000001],
[8.65432050000004, 50.106339],
[8.65446150000002, 50.1063941],
[8.65428550000007, 50.1065795000001],
[8.65414460000005, 50.1065244000001],
[8.65404280000007, 50.1066317],
[8.65526350000005, 50.1071017],
[8.65531910000004, 50.1071231000001],
[8.65571470000003, 50.1072820000001],
[8.65616050000006, 50.1068196000001],
[8.65598180000006, 50.1067498],
[8.65591740000002, 50.1068175],
[8.65524090000002, 50.1065534000001],
[8.65504730000004, 50.1064778000001],
[8.65501150000006, 50.1064635000001],
[8.65442100000007, 50.1062332000001]
],
[
[8.65536580000003, 50.1069989000001],
[8.65552820000005, 50.1068259],
[8.65557490000003, 50.1068439000001],
[8.65576940000005, 50.1069191],
[8.65560710000005, 50.1070920000001],
[8.65541540000004, 50.107018],
[8.65536580000003, 50.1069989000001]
],
[
[8.65493320000007, 50.1068226],
[8.65509250000002, 50.1066524],
[8.65533340000007, 50.1067451000001],
[8.65517400000005, 50.1069153],
[8.65493320000007, 50.1068226]
],
[
[8.65446870000005, 50.1066457000001],
[8.65464290000006, 50.1064629000001],
[8.65491170000007, 50.1065683],
[8.65473750000007, 50.1067511000001],
[8.65446870000005, 50.1066457000001]
]
]
}
}
]
}
FeatureCollectionPBuffer {
queryResult: QueryResult {
featureResult: FeatureResult {
fields: [ Field { name: 'name', fieldType: 4, alias: 'name' } ],
values: [],
features: [
Feature {
attributes: [ Value { stringValue: 'Commerzbank DLZ 1' } ],
geometry: Geometry {
lengths: [ 16, 7, 5, 5 ],
coords: [
632527880, -1677, -100500, -105800, 141000,
-55100, -176000, -185400, -140900, 55100,
-101800, -107300, 1220700, -470000, 55600,
-21400, 395600, -158900, 445800, 462400,
-178700, 69800, -64400, -67700, -676500,
264100, -193600, 75600, -35800, 14300,
-590500, 230300, 633472680, -1677, 162400,
173000, 46700, -18000, 194500, -75200,
-162300, -172900, -191700, 74000, -49600,
19100, 633040080, -1677, 159300, 170200,
240900, -92700, -159400, -170200, -240800,
92700, 632575580, -1677, 174200, 182800,
268800, -105400, -174200, -182800, -268800,
105400
]
}
}
],
objectIdFieldName: 'OBJECTID',
uniqueIdField: UniqueIdField { name: 'OBJECTID', isSystemMaintained: true },
geometryProperties: GeometryProperties {
shapeAreaFieldName: 'Shape__Area',
shapeLengthFieldName: 'Shape__Length',
units: 'esriDecimalDegrees'
},
geometryType: 3,
spatialReference: SpatialReference { wkid: 4326, lastestWkid: 4326 },
transform: Transform {
scale: Scale { xScale: 1e-9, yScale: 1e-9 },
translate: Translate { xTranslate: -400, yTranslate: -400 }
}
}
}
}I suspect that the first pair in each ring is not being parsed correctly ( |
Well some of them are used outside of the context of the JSAPI specifically, and some are used for parity with f=json. GeometryType (in the Feature message) though should probably be omitted. Will look into removing that
Ah ok yes the geometry decoding is a little complicated, especially if you are not already using quantization/generalization. Basically for PBF, we need to encode the coordinates array as integers (well we could encode them as floats, but varint variable-length integer encoding is one of the primary reasons for using PBF). Because of this, f=pbf always returns quantized coordinates, even if quantization parameters are not specified. In the event of f=pbf and not quantization parameters, the quantization parameters default to the upper left of the map. To unpack these, you need to un-delta encode and then multiply by the included transformation. The f=pbf mode was originally designed specifically for "tile" queries against feature services. Here's quick walkthrough of how we can unpack one of these queries: Delta-encodingBasically on the client we divide up the world into 512x512 pixels tiles, and query such that features in those tiles are relative to the top-left corner of each tile. For example (apologies for the bad paint job 😅): The numbers here are fake, but let's suppose this polygon returns In this case you can see the first vertex position is much larger in magnitude than the subsequent vertices. This is because it is relative to the tile's upper left corner, i.e., it refers to the blue line in the image above. Every other coordinate is a relative, so (20, 0) means "move 20 units to the right". Note that in this example, because coordinates are relative to the tile's upper-left corner, positive y is downward much like it is in screen-space. So (0, 20) means "move down 20 units". Why encode relative vertices this way? Mainly to reduce the size of the payload -- smaller numbers use less space for PBF varints. To unpack the delta-encoded vertices we can do: for (let i = 1; i < lengths[0].length; i++) { // Start at 1
coords[2 * i] += coords[2 * (i -1)];
coords[2 * i + 1] += coords[2 * (i -1) + 1];
}This will give us: Converting to world coordinatesWe now have unpacked, non-delta encoded vertices, but they are still relative to the tile's upper-left origin. If you now need to convert back to world coordinates, you need to multiply each vertex by the included transformation in the featureSet:
To do this, you can just do: const xWorld = x * scale.x + translate.x;
const yWorld = translate.y - y * scale.y;Note that in the above query we request the payload in spatialReference 3857 (web-mercator), so xWorld and yWorld will be in mercator map units. |
|
Actually I just realized that the PBF file posted in this repository is out of date, coords should be sint64 not sint32. We made this change to support non-tiled queries which will otherwise overflow. That might be what you are running into @thw0rted ... we will get that fixed. |
|
Thanks, the delta-encoding was the confusing part, but I totally understand why they do it now -- protobuf encodes smaller values in fewer bytes. It does raise the question of why the server I linked to uses a huge initial offset for each ring, with a (relatively) tiny Just to close the loop here, in my posted example, the first coordinate of the ring does not change and would be I found a blurb in the REST docs that says
(emphasis mine) The layer definition gives the layer SR ( I tried adding quantizationParameters. That should decode to {
"extent": {
"spatialReference": {
"latestWkid": 4326,
"wkid": 4326
},
"xmin": 50.1,
"ymin": 8.6,
"xmax": 50.2,
"ymax": 8.7
},
"mode": "view",
"originPosition": "upperLeft"
}but the PBF is exactly the same as above, i.e. with transformed world coordinates very close to |
|
@thw0rted I think your first coordinate is getting truncated because the published pbf proto file is typing the coords as sint32 instead of sint64. This is what I'm seeing using a parser built using the correct schema: Query result
Transform
Calculating X:
|
|
Beautiful, thanks! Protobuf is still a bit of a mystery though -- why on earth is it better to encode Anyway this should get me going. If I have a chance, I'll come back and post some code here with whatever I figure out. |
|
@thw0rted TBH I'm not entirely sure what secret sauce the server is using to compute this, next time I run into the server folks I'll have to ask 😆. The original release of the PBF stuff was only for our tile queries (I work on FeatureLayer rendering & helped out with the initial work on this), and in that case the transform is a well defined thing based on what's passed in the actual request. Probably there's some implementation reason the transform is returned like this (for this specific case)-- e.g., maybe the transform is computed based on more than just the single feature that winds up in the final selection because it hits some internal server cache, it actually is the best for reason X, or it just has to be done this way, etc. That's just speculation on my part though. I wouldn't expect this to really have any noticeable impact on decode performance though. |
|
As promised, here's what I'm using now: import type { Position } from "@esri/arcgis-rest-types";
import { esriPBuffer } from "./FeatureCollection";
// Extract one line-string with `count` 2D (hasZ=false) or 3D (hasZ=true)
// coordinates, from a delta-encoded number array, starting at the given
// `offset`. Then, apply the provided `transform` to each coordinate.
export function transformLine(coords: number[], offset: number, count: number, hasZ: boolean, transform: esriPBuffer.FeatureCollectionPBuffer.Transform): Position[] {
const ret: Position[] = [];
const size = hasZ ? 3 : 2;
const decoded = coords.slice(offset, (offset + size) * count);
deltaDecode(decoded, hasZ);
for (let i = 0; i <= decoded.length + size; i += size) {
const pos = transformTuple(decoded.slice(i, i + size), transform);
// Shouldn't return undefined, hopefully
if (pos) { ret.push(pos); }
}
return ret;
}
// Apply the provided transform to a single point
function transformTuple([x,y,z]: number[], transform: esriPBuffer.FeatureCollectionPBuffer.Transform): Position | undefined {
if (undefined === x || undefined === y) { return; } // Shouldn't happen
if (transform.scale) {
x *= transform.scale.xScale;
y *= transform.scale.yScale;
if (undefined !== z) { z *= transform.scale.zScale; }
}
if (transform.translate) {
x += transform.translate.xTranslate;
y += transform.translate.yTranslate;
if (undefined !== z) { z += transform.translate.zTranslate; }
}
const ret = [x,y] as Position;
if (undefined !== z) { ret.push(z); }
return ret;
}
// Unpack a delta-encoded 2D (hasZ=false) or 3D (hasZ=true) linear-ring in-place.
function deltaDecode(coords: number[], hasZ: boolean): void {
const size = hasZ ? 3 : 2;
for (let i = size; i + size <= coords.length; i += size) {
// Assert because the input array is not sparse
// eslint-disable-next-line @typescript-eslint/no-non-null-assertion
coords[i] += coords[i - size]!;
// eslint-disable-next-line @typescript-eslint/no-non-null-assertion
coords[i + 1] += coords[i - size + 1]!;
// eslint-disable-next-line @typescript-eslint/no-non-null-assertion
if (3 === size) { coords[i + 2] += coords[i - size + 2]!; }
}
}The result of calling ETA: Almost forgot, I wrote ETA again: I also had to add // Default or "upper" origin means Y is flipped
if (transform?.scale && transform.quantizeOriginPostion !== esriPBuffer.FeatureCollectionPBuffer.QuantizeOriginPostion.lowerLeft) {
transform.scale.yScale *= -1;
}just after initially decoding the payload. It's effectively the same as the formula a few posts back, where in "upperLeft" origin mode (the default), the world Y coordinate is given as |
|
@thw0rted are you planning on publish an arcgis-pbf-arsing library by any chance? I'd be happy to collaborate on something if you were going to open source it. |
|
Ookie dokie well I've got a geometry parsing set up, needs a bit of tidying but this is the guts of it. There are a few bits hard-coded in there but hopefully it provides enough insight as to what needs to happen. Cheers |
|
Yeah, I could publish some of this, but I figured what I had pasted was enough to be useful without being "big" enough to warrant a whole project. Your usage looks good, though it sounds like you're doing a side-effect import of Long and leaving the coords as I don't really get why they're using Longs for this anyway. Native JS |
|
Hmm @thw0rted not sure exactly what you mean. 32 bit integer coords is not enough precision to represent the world, you'll get a wobble when zooming in, unless you specify those coords relative to some local origin. However, JS can represent 53 bit integers safely, so it isn't necessary to generate longs even though you have > 32 bits of precision required. Note that pbf allows for 32 or 64 bit ints, not values in between. When using protobufjs, you can use --force-number to avoid creating longs, which in general I would recommend. |
|
Ah! I need to spend more time with protobuf, it didn't even occur to me that the choice was between 32 vs 64, rather than 53 vs 64. (When I said "15 digits", I meant that a 53-bit float is accurate to around 15 decimal places.) That's a good tip about |
|
FYI I just got around to trying I'm using For the time being, rather than wrestling with Webpack to make it exclude |
|
The other thing I discovered on this front was that if you generate your own parser using the pbf library from mapbox then it automatically handles the long issue by simply not supporting them :) FWIW I've published my work here https://github.com/rowanwins/arcgis-pbf-parser |
|
Thanks for the heads-up, I wasn't aware of Mapbox's competing library and will definitely have to look into it. My only hangup is that the Mapbox version doesn't seem to generate any kind of type information (JSDoc or When you say long is "not supported", you just mean that higher-precision values will be decoded and stored in a |
Your interpretation is correct :) |
|
Potentially would also be good to flesh out more doc concerning transforming the geometry (#11). This repo does assume knowledge of the rest API, so it would probably be good to link to it. |
Thanks for posting this project! I can see that it was brought over from an internal tracker (there's still at least one link to
devtopia.esri.comin the README) so I thought some of the definition might be in-flux or subject to some cleanup. My question is about theFeatureCollectiontype.Calling the
/queryendpoint (without countOnly/idsOnly) returns aQueryResultwith aFeatureResult. TheFeatureResultcan have an array ofFeatures, each of which pairs an array ofValuess (attributes) with aGeometry. ButFeatureResultcan also have oneFeatureCollection, which has an array of attributes and one of coordinates. Despite the name of the repository, the docs don't actually describeFeatureCollection. Is it actually used? If so, when? How would one break up the array of attributes or coordinates into individual Features?The text was updated successfully, but these errors were encountered: