EXT_node_lod extension proposal

[takahirox]

Working repository: https://github.com/takahirox/EXT_node_lod

I would like to propose EXT_node_lod extension that allows LODs in glTF.

I know already MSFT_lod extension exists for LOD. The new extension is less flexible but simpler than MSFT_lod so it should be easier for the handlers (implementations) to handle.

JSON structure and examples

EXT_node_lod can be defined at the node level.

"nodes": [{
    "name": "High_LOD",
    "mesh": 0,
    "extensions": {
        "EXT_node_lod": {
            "lod": [
                {"node": 1, "coverage": 0.5},
                {"node": 2, "coverage": 0.2},
            ],
        },
    },
},
{
    "name": "Medium_LOD",
    "mesh": 1,
},
{
    "name": "Low_LOD",
    "mesh": 2,
}]

Difficulty of handling MSFT_lod

While I was writing a handler for MSFT_lod I realized that MSFT_lod has some difficulties to handle.

• Both node and material can define LOD. The behavior is unstated in the specification if material containing LOD is referred by node containing LOD.
• A node can specify other nodes so LOD node can form nest or even infinity loop. And the behavior for them is unstated in the specification.
• Threshold for switching LOD is defined in extra as optional. Unclear how to switch LOD for applications/viewers/engines if it is undefined because they don't know the qualities of LODs.

Simplification ideas

I would like to suggest to simplify the LOD specification. It will lose some flexibilities but be simpler to write an implementation. The differences from MSFT_lod specification are

• Allow to define LOD only at node level and drop material LOD. This change can avoid the case where both node and material have LOD. Material LOD capability is still covered by node LOD (refer to the same geometry attributes and refer to different materials).
• Add/Clarify some LOD node restrictions for simplicity.
  • Nested LOD nodes are disallowed
  • Child nodes are part of that LOD
  • LOD nodes are regarded as part of the scene node graph
• Require threshold values. This can avoid unknown threasholds for applications/viewers/engines.

Additional notes

• With the simplification suggestions, the extension will be easier to be handled so I think it would be good to be a multi-vendor extension (EXT_ prefix) rather than a vendor extension.

Discussions

I'm still thinking how threshold values should be, screen coverage, distance from camera, or others. In the current PR, I chose screen coverage for now just to follow MSFT_lod spec. Probably compatibility is important so we may need to choose the generic one that can be handled easily and consistently across applications/viewers/engines.

takahirox/EXT_node_lod#1

Difficulties of handling screen coverage threshold.

• Screen coverage requires all geometries sizes in the LOD and its descendant nodes.
• Screen coverage is affected by camera info and world matrix. Needs recomputing when they are updated.
• Some engines don't (natively) support eg. Three.js

Related threads

• Adding LODs to glTF  #1045
• Level of Detail (LoD) or Quality of glTF asset #1743
• MSFT_lod: Clarify the behavior of the case where node level and material level LODs are specified #1952
• MSFT_lod: Clarify whether recursive LOD is allowed or not #1953

[vpenades]

Question: at which point a node would stop being visible?

Based on the current implementation, I guess the last LOD, the one with smallest coverage, would point to an empty node?

Otherwide, I would add a property called "minCoverage" that would define at which point the node should not be rendered at all.

minCoverage should be smaller than the smallest coverage of any LOD, or 0 by default

Also, How coverage is calculated? I guess it would be based on spheres? in which case maybe it could be useful having a Radius as a hint?

[takahirox]

I guess the last LOD, the one with smallest coverage, would point to an empty node?

Yes because I think the implementation would be simple regardless of whether including invisible LOD like the following pseudo code.

const lod = new LOD();
lod.addLevel(loadNode(nodeIndex));
for (let i = 0; i < extension.lod.length; i++) {
  lod.addLevel(loadNode(extension.lod[i].node), extension.lod[i].coverage);
}

Also, How coverage is calculated? I guess it would be based on spheres? in which case maybe it could be useful having a Radius as a hint?

Perhaps sphere or box.

This LOD extension switches node so to calculate the node sphere/box the implementation needs to traverse descendant nodes. It may be troublesome so adding a hint (as required?) may be good. Or distance based coverage may be good because no calculation is needed.

[vpenades]

Yes because I think the implementation would be simple regardless of whether including invisible LOD like the following pseudo code.

const lod = new LOD();
lod.addLevel(loadNode(nodeIndex));
for (let i = 0; i < extension.lod.length; i++) {
  lod.addLevel(loadNode(extension.lod[i].node), extension.lod[i].coverage);
}

My concern is that currently, glTF validator gives warnings on empty nodes, and many tools not aware of the extension might "optimize" the models by removing empty nodes. My feeling is that from one reason or another, glTF discourages empty nodes and preffers explicit and concise definitions.

Also, How coverage is calculated? I guess it would be based on spheres? in which case maybe it could be useful having a Radius as a hint?

Perhaps sphere or box.

This LOD extension switches node so to calculate the node sphere/box the implementation needs to traverse descendant nodes. It may be troublesome so adding a hint (as required?) may be good. Or distance based coverage may be good because no calculation is needed.

The whole logic of the extension sits on top of the logic to switch between LODs, and this logic needs to be precisely defined to avoid multiple engines behaving diferently because they did different interpretations.

So if you use "coverage"... what it means, coverage of what? the whole geometry? a bounding sphere? if two engines calculate the sphere in a different way, the end result would be that the coverage would be different in both engines, and as a consequence, the two engines would trigger the LOD switch at different times. Tha'ts why I think that if Coverage is used, it should be defined in the extension to guarantee consistency across engines.

Distance would be an alternative option. But distance based LODs are sensitive to the screen size, so when rendering very large images would trigger lower LODs to soon, resulting in a loss of quality. If anything I would use Distance divided by screen height, to ensure similar behavior for different image sizes. Again, the extension need to define this to ensure consistest behavior across multiple engines

[takahirox]

My concern is that currently, glTF validator gives warnings on empty nodes, and many tools not aware of the extension might "optimize" the models by removing empty nodes

Some potential options in my mind are

1. The extension doesn't care about it. Empty node should be valid in the spec (, shouldn't it?). Encourage tools for allowing empty nodes. @donmccurdy Any thought about this from tooling end (gltf-transform) ?
2. The extension doesn't allow empty node. Users may use a node with a mesh that has like (0,0,0), (0,0,0), (0,0,0) positions for invisible LOD.
3. The extension allows invisible LOD in the extension definition like "lod": [{"node": null, "coverage": 0.5}] or minCoverage as you suggested.

So if you use "coverage"... what it means, coverage of what?

The goal is to clearly define coverage perhaps with fomula. I first want to discuss distance vs screen coverage. And especially if we choose screen coverage, I want to discuss how it should be consistently calculated next.

If screen coverage is preferable, I'm thinking if it's good to add sphere, box, or plane size for calculating screen coverage as required parameter.

[takahirox]

I need a double check because I'm not an expert of 3D CG algebra but perhaps screen coverage ratio can be calculated with this formula? Assuming sphere is in the viewport.

pos1 = projection_matrix * view_matrix * model_matrix * vec4(radius, 0.0, 0.0, 1.0);
pos2 = projection_matrix * view_matrix * model_matrix * vec4(0.0, 0.0, 0.0, 1.0);
radius = length(pos1.xyz - pos2.xyz);
ratio = radius * radius * pi / 4.0;

It can be more complex if we allow sphere to have un-unified scale.xyz and to rotate tho.

[takahirox]

Thinking if different radiuses per LOD level should be allowed and where bounding sphere radiuses for LODs should be defined.

Radiuses for lower LODs can be defined in elements of EXT_node_lod.lod array like

"nodes": [{
    "extensions": {
        "EXT_node_lod": {
            "lod": [
                {"node": 1, "coverage": 0.5, radius: 10.0},
                {"node": 2, "coverage": 0.2, radius: 10.0},
            ],
        },
    }
},
...
]

But where for the highest LOD. EXT_node_lod.radius may be another option (if we don't allow different radius per LOD level).

"nodes": [{
    "extensions": {
        "EXT_node_lod": {
            "lod": [
                {"node": 1, "coverage": 0.5},
                {"node": 2, "coverage": 0.2},
            ],
            "radius": 10.0,
        },
    }
},
...
]

[takahirox]

If glTF vs glXF will end up with self-contained vs scene (composition), defining the extension not in glTF but in glXF might be more straightforward and intuitive (one Level in one glTF file).