WEB3D_quantized_attributes Specification

[lasalvavida]

There are a few things that came up while writing support for the proposed WEB3D_quantized_attributes extension into Cesium, that I'd like to discuss here.

First off, the implementation can be found on this branch of my fork of Cesium, and any comments and suggestions are welcome in my pull request.

So, firstly, the specification calls for the addition of a decodeMatrix, and optional decodeMin and decodeMax properties.

I'm not certain this is the best approach, largely because in practice, the formula with the decode matrix ends up looking like this:

I can't think of a case where any individual component would combine with any of the others, so I think that this could be more concisely stated. Perhaps by just making the accessor max and min attributes mandatory if the extension is to be used, and then including precision information in the extension attributes.

@mlimper Your input on this would be greatly appreciated

[MarcoHutter]

Just some remarks, summarized / derived from the README:

• The min/max accessor attributes (basically the bounding box of the model, when referring to the positions) are not necessarily equal to the decoding parameters.
• The SRC format, where this quantization was originally proposed, indeed used only vector properties (decodeOffset and decodeScale, basically just a slightly different representation of the min/max)
• The justification for choosing a matrix was that it may easily be integrated into (i.e. multiplied with) the MODELVIEW matrix, so that the shaders remain unaffected by the quantization

However, I also think that the matrix could simply be constructed at runtime, just like you described, but I may have overlooked something here.

[mlimper]

@lasalvavida Thanks a lot for the nice explanation, and thanks @MarcoHutter for your answers.

I guess marco summarized well what we also had in mind when writing the extension:

min/max are not necessary equal to the decoding parameters, although they might be the same in the most simple cases. The difference especially matters when you have a larger model and you want to subdivide it into multiple smaller meshes: if you use each meshes min/max directly to compute decodeOffset/decodeScale, visible cracks can occur at the boundaries between neighboring meshes. These artifacts originate from boundary vertices (which were split and hence belong to more than one mesh) being quantized to slightly different positions. The common solution is to use the same "quantization grid" for all the meshes, which will lead to a consistent, closed surface.

We found using a matrix would be actually more convenient to use for the decoder side, imposing a bit more complexity on the encoder side.

Both of the ideas, using a common grid for neighboring meshes to hide cracks along the seams, as well as using a decode matrix, were ideas we also discovered in this paper "Mesh Geometry Compression for Mobile Graphics" by Lee et al.:
http://cg.postech.ac.kr/research/mesh_comp_mobile/mesh_comp_mobile_conference.pdf

Of course, this is all open for discussion - please share your thoughts.

[lasalvavida]

@MarcoHutter @mlimper, thank you for your input. I'll try applying the decode matrix directly to the model view matrix, that will almost certainly be faster than the current approach (unpacking the bufferview using the decode matrix).

@mlimper Could you give a numerical example of when min and max would be different from decodeMin and decodeMax?

A few other thoughts. What should the accessor componentType be when we're using the extension? Should it be what the data type is in its encoded form(uint16), or what it should be treated as when it is decoded(float32)?

There's also some issue of precision here when the bounds of a model are fairly small. For example, quantizing a model with extrema (-1,-1,-1) and (1,1,1) results in the following decode matrix:

I could certainly include more decimal places, but if we divide a fairly small number by 2^16, there's going to be some rounding problems.

Maybe it would be better to separate the precision information out of the matrix as another property

so that the rounding is done by the end user, not by the model converter.

[mlimper]

I'll have a look after Eurographics - right now, I'm unfortunately a bit under water :-/
So, sorry for that in advance!

I just had a brief look, in our case it looks like the componentType will usually be UNSIGNED_SHORT (5123 - see the teapot example I sent you).

@mlimper Could you give a numerical example of when min and max would be different from decodeMin and decodeMax?

I don't have one at hand right now - but you can see an explanation and a graphical illustration in the mentioned paper of Lee and colleagues (http://cg.postech.ac.kr/research/mesh_comp_mobile/mesh_comp_mobile_conference.pdf), Figures 1 and 2 give the basic idea.
Section IV B provides details on the encoding process. In particular, step 1) might be interesting:

"Calculate the bounding cube for each mesh partition and find the largest x, y, and z bounding cube sizes among all partitions"

So, what the authors are doing to prevent the cracks is actually to quantize everything with respect to the largest mesh ("partition") within the scene. I think what they are doing is conceptually the same as using some "modulo scheme" to repeat this "master size", guaranteeing that every mesh will have quantized coordinates that are on the grid and can be represented within the quantized unit range (with respect to the "master size"). This was solved in a very similar fashion by Sander and Mitchell in their paper on "Progressive Buffers":
http://www.cs.ust.hk/~psander/docs/progbuffer.pdf

Quoting from that one:
"While the precision for the normals and texture coordinates is sufficient, 16-bit precision for the position in a large scene is not high enough. In order to address this, we initially considered storing the position as a delta from the chart’s center position. However, this would result in dropped pixels
at chart boundaries because of precision errors when decompressing vertices from adjacent charts whose positions must match exactly. In order to avoid this problem, we store the position modulo r (p % r), where r is the largest cluster bounding sphere. Given the stored position and the cluster’s
bounding sphere center position (which is made available through the constant store), the vertex shader can reconstruct the original position. Since the stored values (p % r) are identical for matching vertices on adjacent clusters, they are reconstructed to the exact same value."

[lasalvavida]

Didn't mean to close this.

[lasalvavida]

@mlimper I have a question emerging from my conversation with @pjcozzi in #3891.

I think it probably makes sense to make decodeMin and decodeMax required for POSITION attributes as in #593.

The only case I can think of where you really need the min and max is for computing bounding spheres. I can put together a pull request for the spec if that is agreeable to you.

[pjcozzi]

@mlimper I agree with @lasalvavida here because it will simplify the client implementation, which is inline with the spirit of glTF.

[mlimper]

The only case I can think of where you really need the min and max is for computing bounding spheres. I can put together a pull request for the spec if that is agreeable to you.

That would be great, thanks in advance!
Sounds good to me.

[lasalvavida]

@mlimper Another question for you; @pjcozzi and I were talking about an issue that I had run into while writing the cesium implementation.

If one attribute is quantized and the other is not, or two attributes are quantized with different accessors, using the modified shaders approach necessitates separate programs.

However, as the spec stands the burden is placed on the client to do something like what I did in cases where this conflict occurs (clone the program, technique and material and re-assign the mesh to it). This requires a lot of traversing the gltf tree in the reverse order of what it was designed for.

I propose modifying the extension to explicitly require that accessors have separate programs under these conditions to streamline the implementation. I would then move my pre-processing into gltf-pipeline as part of the conversion.

Thoughts?

[mlimper]

That sounds like an interesting point!

Not 100% sure that I get this correctly, though:
What exactly is the "modified shaders approach"? Does it mean you explicitly write the de-quantization code into the shader, instead of baking it into a matrix?

[lasalvavida]

The spec lists two approaches:

One way to do this would be to pass the matrix to a vertex shader and perform the decoding before multiplying with a model-view matrix. However, it is even possible to multiply the decode matrix with the model-view matrix in advance. This way, no special adaptions are necessary inside the shader in order to render compressed data.

The former is the approach I am referring to. Not all attributes (color for example) have a matrix that can be baked into, so for now I am using the approach of modifying the shader for better coverage.

[mlimper]

Ah, so you mean you will get a conflict when you have, say, one geometry that uses quantized positions, and one that uses floating-point positions. In that case, you would use two different shaders - correct?

Just out of curiosity: Could one also use the same shader and an identity matrix for the decode matrix in the floating-point case?

I think I see the point now: Basically the quantized_attributes extension might require you to modify a given glTF shader, correct? You proposed change to the spec (and the integration into gltf-pipeline) seems to prevent this issue by not allowing such cases. That seems reasonable to me.

[lasalvavida]

This is reflected in the dialogue in my cesium pull request, but I think it is important to state it here as well. It is not necessary to place restrictions on what programs quantized accessors can belong to. I will be sure to include some notes on the implementation to help anyone else who attempts to write their own in the future.

[pjcozzi]

@mlimper is it worth adding an implementation details or limitations to the WEB3D_quantized_attributes extension spec from this discussion? Or should we just close this?

[mlimper]

It seems worth adding some words to the extension spec, clarifying what kind of approach is recommended when dealing with custom shaders + quantized attributes at the same time. However, that would only apply to glTF 1.x, I guess, and the extension spec still says "Written against the glTF draft 1.0 spec."... maybe one could update it to match 2.0 (which would mean we could probably ignore the custom shader issue)?

[pjcozzi]

Sounds like you are suggesting that we deprecate the current extension spec. I don't know that we ever do that. However, it may be OK here since perhaps the extension is considered draft since it is written against the draft spec and if there are not many implementations and we are aware of all of them. Then the extension spec could be written against glTF 2.0 with an implementation section for using shaders with WEBGL_technique_webgl.

I'll circulate with the group on the next call.

[lexaknyazev]

@pjcozzi Any updates on compatibility with glTF 2.0 and KHR_technique_webgl?

[hypnosnhendricks]

Has this made any traction at all for GLTF 2.0? It seems like GLTF 2.0 was so close to being able to spec this directly into their accessor data. There is already the spec for a min/max value and a booean for normalized. it seems so natual to use that same min/max combined with the normalized boolean to be able to know how to reconstruct the values.

This extension is the closest I have found to match my needs of quantizing the morph target data in our GLB's, but it doesn't seem like this exists for GLTF 2.0

[donmccurdy]

@hypnosnhendricks perhaps KHR_mesh_quantization is what you're looking for?

[emackey]

I think this can be closed given that KHR_mesh_quantization is ratified for glTF 2.0 now. Let me know if you think otherwise, and we can reopen.

[hypnosnhendricks]

@hypnosnhendricks perhaps KHR_mesh_quantization is what you're looking for?

I have been looking at the various compression schemes available in GLTF 2.0.

Both Draco and EXT_meshop_compression operate as zip style compression schemes. Since they are zip style schemes, this doesn't give us what we want, which is a lossy quantization. Draco doesn't support morph targets, which is where the majority of our GLB size comes from. Also, we already serve are GLBs compressed using zstd. ie, we send "model.glb.zst" instead of "model.glb" to the client upon request.

That leaves KHR_mesh_quantizaiton as the only viable one. However, it changes the GLB structure by inserting nodes and altering the skinning bind matricies. I'm not sure if this will be acceptable to our clients, specifically those using Unity, Unreal & assimp. I'll find out more after the holiday break. This extension also appears to be designed to be used in place on the GPU, hence the altering of the nodes/skinning matricies. At this time, none of our clients can actually use these compressed vertex data, so they would need an easy way to convert back to an approximate floating point value, which doesn't appear to be easily doable with this extension and skinned meshes.

I found this WEB3D quantization approach & it seems like it is the most elegant solution. The quantization operators are specified in the accessor & it is easily reversible on the client side. The downsides are that this is a GLTF 1.0 extension & there doesn't appear to be any widespread support for it. One of our main goals is to make sure our GLBs are viable on the most clients possible, including Babylon, ThreeJs, Unity, Unreal, Etc. We have been able to keep to this goal so far & I really would like to keep it with a widely supported quantization approach for morph target data.

About 80% of our GLB data is compromised of morph target data. The vast majority of our morph target data exists within a range of +/- 3.0. Our goal for quantization would be to quantize to 8 or 16 bit based based on quality settings & then throw out all the zero values when we convert to sparse representation.

A quick test with gltfpack shows a reduction in our GLB size by about 30% with quantization alone.
Doing zstd compression without quantizaiton reduces our GLB size by 60%.
Adding both together reduces the size of our GLBs anywhere from 80 to 90%, so doing both quantization along with zstd compression is a pretty big win.

[donmccurdy]

Both Draco and EXT_meshopt_compression operate as zip style compression schemes. Since they are zip style schemes, this doesn't give us what we want, which is a lossy quantization.

Hm, I don't think that's really true of either — Draco is always lossy, and while Meshopt isn't, it's always paired with a lossy pre-processing step using KHR_mesh_quantization. (/cc @zeux)

Loading with dequantization parameters often requires either custom engine or material changes to apply the params, or costly decoding on the CPU. By contrast, KHR_mesh_quantization can be supported with virtually no changes at all to an existing glTF implementation, and no decoding overhead. In terms of gaining widespread support, that's really ideal, and introducing a new and different quantization mechanism could make things much more complicated.

Could I suggest putting the quantization parameters elsewhere in the file, such as .extras? If your application needs to decode back to the original values, it could use those.

At this time, none of our clients can actually use these compressed vertex data

Do you mean particular engines require float32 attributes and can't load smaller vertex attribute types? For Unity, GLTFast already supports KHR_mesh_quantization. Babylon.js and three.js support this out of the box as well.

[hypnosnhendricks]

@donmccurdy For our Clients, I'm talking Unity, Unreal and a few others that use assimp internally. I'm going to speak to our Unity team to see if the modifications to the object model ( inserting nodes, modifying the skinning matrices ) that KHR_mesh_quantization does will be viable for our various clients. If it isn't viable, we will have to go a custom quantization route, similar to what the WEB3D one does here, and have our client runtimes deal with decoding to the native formats.

Having the offset/scale in the accessor's extras for normalized accessors seems like the most straightforward approach. Its as shame I can't reuse the accessor bounds for this, as that could also store the same decoding information necessary to transform the normalized values to their correct floating point representation.

glTFast is interesting as I haven't stumbled upon that before, but looking at it's readme, it doesn't support animations or morph targets yet, so not viable at this time for our needs. But I'll be keeping track of it.

[zeux]

Hm, I don't think that's really true of either — Draco is always lossy, and while Meshopt isn't, it's always paired with a lossy pre-processing step using KHR_mesh_quantization. (/cc @zeux)

Yes - this is accurate. Draco uses quantization that's "invisible" to the consumer - as long as the Draco decoder is used, the decoded mesh will likely use float32 attributes. KHR_mesh_quantization + EXT_meshopt_compression provide a solution where the resulting quantization is exposed to the consumer; the benefit is that the resulting model can be uploaded directly to GPU memory with optimal memory consumption.

doing both quantization along with zstd compression is a pretty big win.

This is approximately what you get by using KHR_mesh_quantization + EXT_meshopt_compression (or rather for optimal size you probably need to use the above + zstd; zstd alone can't compress geometry data optimally).

our client runtimes deal with decoding to the native formats

FWIW "native" seems a bit of a misnomer here as Unity and Unreal must already support quantized attribute storage, even if glTF import pipeline isn't compatible with it. But yes - you do get extra nodes. The attraction of KHR_mq approach however is that the engine support tends to be simple and compatible with GPU-side storage, whereas special per-attribute dequantization matrices would require extra complexity unless the mesh is decoded at load time.