Support for PBF response from Feature Service?

[thw0rted]

In the docs for arcgis-rest-feature-layer it says that the f parameter can take a value of "pbf" to return protobuf, but that this requires the use of rawResponse: true, and that you have to parse the resulting binary yourself. This community discussion thread from last year has people asking for the corresponding protocol description file (.proto) but it doesn't sound like it's been released.

I haven't worked with protobuf extensively but my understanding was that you start with a schema in .proto format, pass it to a code generator like this one, then it gives you a helper class that can deserialize messages at runtime. Without knowing the format, I don't think the guidance to "parse it yourself" is actually possible.

Can Esri either add native support for the pbf transport to arcgis-rest-feature-layer, or at least release an official schema file so we can parse the response ourselves?

[jgravois]

back when i chimed in on #617 (comment) i was under the misguided impression that since individual Esri vector tiles adhere to the Mapbox spec you could also just lean on one of their hipster libraries to decode our .pbf query output.

see mapbox/vt2geojson#18 (comment)

i screwed around a little bit this evening and all i can say is, i was wrong.

[thw0rted]

I don't mean to sound too critical, but I don't understand what's hard here. Somebody, somewhere at Esri, wrote a .proto file describing these messages, because there's no way they're hand-jamming the bits out of the server. (Hopefully, they also have a Confluence page or something with a human-readable schema for what a message looks like.) There's no reason it should be encumbered by some incompatible license or full of proprietary Esri secrets they can't let out, right? Why not just throw it into the /support directory?

[jbartley]

@thw0rted plan is to release this on github like you describe. Will ping you once we have it out.

[ddbradshaw]

Adding a "me too" here as well. Very interested in decoding PBF queries.

[jbartley]

@thw0rted we finally got it out. Sorry it took so long.

https://github.com/Esri/arcgis-pbf

[thw0rted]

Thanks @jbartley , I will take a look when I have some spare cycles.

It looks like there's no example of using the included JS in the README but it's based on protobufjs so those docs should be enough. As far as I can tell, there should also be native support for the Fetch streaming API (ReadableStream) which is particularly helpful. It sounds like the hardest part will be implementing the delta math for myself -- maybe there are helpers for that elsewhere in the library?

[thw0rted]

Does anyone have a link to a public-facing / sample feature server that supports PBF? All the ones I can find only say "JSON" in the supported query format field. I tried changing f=json to f=pbf anyway, and the response is just

{"error":{"code":400,"message":"Invalid or missing input parameters.","details":[]}}

I found an Esri-hosted sample server with a bunch of different data types, so I had hoped that this "Military" service would respond to https://sampleserver6.arcgisonline.com/arcgis/rest/services/Military/FeatureServer/2/query?f=pbf&where=1%3D1&outFields=*&returnIdsOnly=true but that just gives the above error. Is there something the server admin has to do to enable PBF transport?

ETA: the linked Community thread pointed to a service on this public server. I picked one at random and it seems to respond correctly with PBF. I noticed that the first server ("sampleserver6") is running 10.71 while this is running 10.81. Maybe there was a bug in 10.71 that broke PBF support?

[jbartley]

@thw0rted any arcgis online hosted feature service will support PBF for queries. Here are some examples.

Also supported on Enterprise feature services and map services at 10.81 and later.

When checking the layer response look at supportedQueryFormats. If you see 'PBF' in the supportedQueryFormats it is supported.

[thw0rted]

Thanks, I think the disconnect was that the FeatureServer description document just says "JSON", and I was failing to look at a single layer under the service -- so, /Military/FeatureServer/ is JSON but /Military/FeatureServer/0 is JSON, geoJSON. I was also confused because the linked Community thread said PBF support was added in "10.7.1" which I took to be a typo for "10.71". I'll look for 10.81 in future.

Also, last night, I was able to use the JS library you linked above to parse out a response:

import { esriPBuffer } from "./FeatureCollection.js";

// ...

const resp = await queryFeatures(...) as Promise<Response>;
let ids: string[];
try {
    // TODO: streaming
    const ab = await resp.blob().then(b => b.arrayBuffer());
    const reader = Reader.create(new Uint8Array(ab));
    // Return types for `decode` are wrong
    ids = (esriPBuffer.FeatureCollectionPBuffer.decode(reader) as any)
        .queryResult.idsResult.objectIds;
} catch (err) {
    throw new Error("Failed to read response stream");
}

(I had to hand-jam the correct return type until #821 is fixed.) Today I'm going to work on decoding the features themselves. If anybody has suggestions how to do this stream-wise (using Response#body.getReader()) that would be really helpful. I'm probably going to ask about that over at the protobufjs tracker.

[ddbradshaw]

@thw0rted - Watching this thread for your final results if you're willing to share.

[thw0rted]

Happy to share, what I posted above already gives you the basics. I'm currently hung up on yak shaving (my tests go through Webpack, which seems to be mangling one of my sample PBFs before it gets to the parser...) but once that's done, I will update here.

[thw0rted]

I think I've nearly got this. I'm building a collection of Features using the provided decoder but I'm hung up on the transform. The sample service I've been using (https://services.arcgis.com/V6ZHFr6zdgNZuVG0/ArcGIS/rest/services/buildings_frankfurt/FeatureServer/0) provides features that look like

{
    "fields": [
        {
            "name": "OBJECTID",
            "fieldType": 6,
            "alias": "OBJECTID"
        },
        // ...
    ],
    "values": [],
    "features": [
        {
            "attributes": ...,
            "geometry": {
                "lengths": [
                    7,
                    5
                ],
                "coords": [
                    637561280,
                    -1677,
                    -225500,
                    -195200,
                    322200,
                    -153200,
                    188800,
                    163400,
                    -125300,
                    59500,
                    36800,
                    31900,
                    -197000,
                    93600,
                    637625080,
                    -1677,
                    -108900,
                    57000,
                    52300,
                    41100,
                    108900,
                    -57000,
                    -52300,
                    -41100
                ]
            }
        },
   ],
    "objectIdFieldName": "OBJECTID",
    "uniqueIdField": {
        "name": "OBJECTID",
        "isSystemMaintained": true
    },
    "geometryProperties": {
        "shapeAreaFieldName": "Shape__Area",
        "shapeLengthFieldName": "Shape__Length",
        "units": "esriDecimalDegrees"
    },
    "geometryType": 3,
    "spatialReference": {
        "wkid": 4326,
        "lastestWkid": 4326
    },
    "transform": {
        "scale": {
            "xScale": 1e-9,
            "yScale": 1e-9
        },
        "translate": {
            "xTranslate": -400,
            "yTranslate": -400
        }
    }
}

That feature is supposed to draw a polygon somewhere in Frankfurt, so very roughly around (50.1, 8.6) degrees. I assumed the transform was supposed to be applied by adding translate then multiplying by scale, but that just leaves you with tiny numbers. Do I have to use the layer's extent min/max to translate to decimal degrees, or am I missing some other offset here?

ETA: adding the extent's minx / miny gives me reasonable-sounding values, except for the first coordinate in each ring. For some reason, each ring starts with a huge number like 637561280 above, which throws the whole thing out of whack. Could this be a problem with the generated decoder?

[rowanwins]

Have added some geometry parsing tips over here

[patrickarlt]

I think most of the issues related to this have been at least partially resolved by https://github.com/Esri/arcgis-pbf and https://github.com/rowanwins/arcgis-pbf-parser