Add BINORMAL and TANGENT attribute semantics

[pjcozzi]

From CesiumGS/gltf-pipeline#203:

The glTF spec does not define NORMAL and TANGENT attributes. See

https://github.com/KhronosGroup/glTF/tree/master/specification/1.0#parametersemantic

Perhaps we should add them for 1.1 since it is trivial?

Otherwise, the PBR extension could define them, but they would need to be prefixed with _ here so the glTF asset is still valid before the extension is added.

@lexaknyazev @mlimper

[lexaknyazev]

Did you mean BINORMAL?

[pjcozzi]

Yes, I updated the title.

[javagl]

As this is supposed to affect the spec, a nitpick from me, quoting from http://mathworld.wolfram.com/BinormalVector.html :

In the field of computer graphics, two orthogonal vectors tangent to a surface are frequently referred to as tangent and binormal vectors. However, for a surface, the two vectors are more properly called tangent and bitangent vectors.

Some people have strong opinions about this, because the surface only has one normal, but it has infinitely many tangents, and one can only compute a bitangent for any given tangent.

(I know, "binormal" is so common that one might have the choice between "common terminology" and "right terminology" here. But some tutorials, e.g. http://www.opengl-tutorial.org/intermediate-tutorials/tutorial-13-normal-mapping/ are already trying to use the "right" one...)

[pjcozzi]

@mlimper what is the common PBR term? I suggest we use that.

[lexaknyazev]

If we can universally figure out following questions, it should be possible to include that in the core:

• Normals could deliberately have values that are different from actual surface normals (smooth/flat shading). Is similar behavior acceptable with these new vectors? Or are they strictly defined by existing positions/normals?
• Are Cesium equations sufficient for most common usecases?
• Are there any additional limitations on these semantics, such as:
  • if one is defined, other also must be defined, or
  • they must form only right (or left) hand system, etc?

[pjcozzi]

Normals could deliberately have values that are different from actual surface normals...Is similar behavior acceptable with these new vectors?

Yes, this is similar in the GL world, where, for example, vertex normals are not used by the pipeline for backface culling since they could be anything.

Are Cesium equations sufficient for most common usecases?

No, Cesium implements a general-purpose algorithm that is pretty good for most cases, but users can often do better if they know more about their mesh, e.g., because they have an equation for it, e.g., a sphere.

Are there any additional limitations on these semantics, such as:

I dunno. I'm tempted to say no so they can fit a variety of use cases, but I'm also tempted to say yes so users can get the best performance and then just define other semantics for custom cases. For example:

Guarantee that they form a right-handed system and are orthogonal to NORMAL. If only one is defined, the other can be computed from a cross product. Perhaps also require that NORMAL and these are normalized. Again, this will not be suitable to all use cases, but others can be handled with custom semantics.

Perhaps see what other formats and engines do for tangent space effects if someone has the bandwidth. I do not.

[lexaknyazev]

In a nutshell, should spec define these attributes meaning (to some extent), or just allow new tokens and let material/shaders define actual usage?

E.g., with custom (GLSL) technique, actual data type and usage of POSITION or NORMAL attributes could be anything. However, material extensions have some requirements. We could keep things that way.

CC @mlimper

[pjcozzi]

should spec define these attributes meaning (to some extent), or just allow new tokens and let material/shaders define actual usage?

Ah, interesting idea...perhaps it is OK to have them very loosely defined by the attribute semantic and precisely defined by the material. The downside would be different material extensions may not be swappable with the same attributes. I suspect this wouldn't be a problem in practice since (1) most extensions would probably have the same definition, and (2) swapping is unlikely to be a common use case.

[xelatihy]

My 2 cents.

Normal/displacement/bump maps are only uniquely defined if tangents are defined. So defining tangents is quite important for portability of assets in my opinion.

[mlimper]

However, material extensions have some requirements. We could keep things that way.

I suspect this wouldn't be a problem in practice since (1) most extensions would probably have the same definition, and (2) swapping is unlikely to be a common use case.

Agree with both!

I know, "binormal" is so common that one might have the choice between "common terminology" and "right terminology" here. But some tutorials, e.g. http://www.opengl-tutorial.org/intermediate-tutorials/tutorial-13-normal-mapping/ are already trying to use the "right" one...

Interesting point. I think as part of glTFs mission, it will be fine to help spreading the "right" term here, without much risk - I suppose people are smart enough to figure out that their "binormal" would be our "bitangent", and we could also briefly note this somewhere in the spec.

Perhaps we should add them for 1.1 since it is trivial?

Sounds like a good way to go!

[pjcozzi]

@erich666 do you know the answer to "binormal" vs. "bitangent?" I thought I read something on your blog at one point.

I know, "binormal" is so common that one might have the choice between "common terminology" and "right terminology" here. But some tutorials, e.g. http://www.opengl-tutorial.org/intermediate-tutorials/tutorial-13-normal-mapping/ are already trying to use the "right" one...

Interesting point. I think as part of glTFs mission, it will be fine to help spreading the "right" term here, without much risk - I suppose people are smart enough to figure out that their "binormal" would be our "bitangent", and we could also briefly note this somewhere in the spec.

[erich666]

Please do use the right term, "bitangent." Write Eric Lengyel if you want convincing. From Real-Time Rendering, 3rd edition, a footnote:

The bitangent vector is also, more commonly but less correctly, referred to as the binormal vector.

See http://mathworld.wolfram.com/BitangentVector.html as a reference, written by Eric Lengyel, who's a graphics programmer for games and an expert at math for graphics.

I also sent a long email thread to Patrick, letting Eric Lengyel and Spike Hughes hash it out in 2010. No great conclusion, but no one said "binormal" was good, and "bitangent" was basically the winner.

[pjcozzi]

Thanks @erich666!

OK, no question at all then, go with bitangent.

[pjcozzi]

should spec define these attributes meaning (to some extent), or just allow new tokens and let material/shaders define actual usage?

Ah, interesting idea...perhaps it is OK to have them very loosely defined by the attribute semantic and precisely defined by the material. The downside would be different material extensions may not be swappable with the same attributes. I suspect this wouldn't be a problem in practice since (1) most extensions would probably have the same definition, and (2) swapping is unlikely to be a common use case.

To follow up here, I was doing some reading and a normal map is often distorted so it fits the surface so the bitangent and tangent may not be perpendicular so this is even more reason to not over constraint these in the spec.

[pjcozzi]

In GPU Pro 5, there is a chapter, "Quaternions Revisited", that explains that the tangent space basis can be stored as a quaternion with a small loss in quality. I don't know enough about this to say if this would be preferred since it saves so much memory, if this should be an option to use instead of explicit vectors, or if we should just pass on it for now.

@erich666 have you looked at this closely?

[erich666]

"Quaternions Revisited"

Closely? No. I know that there's a small loss in quality by doing so, according to the article. I've asked the author to comment.

[PeterSikachev]

Hi,
I was one of the authors of the article, and the one who implemented the tech. YMMV, but we were never able to see any imprecision due to quaternion usage. You can also check out the accompanying demo for GPU Pro 5 (should be available from CRC Press website) - one of my co-authors has made a standalone demo with all the source code available.
Any further questions, let me know.
Peter

[pjcozzi]

@erich666 @PeterSikachev thanks for the prompt response. Sounds like this has great potential to reduce file size.

@mlimper @lexaknyazev what do you think? Should this be optional as a replacement to storing the vectors? Or could we just use a quaternion (assuming visual quality is acceptable)? It would likely require a bit more work for an exporter/converter, but the runtime would be simple and efficient by default.

[erich666]

I'd include @cedricpinson and ask for "what do people use?" I just recalled:

Malyshau, Dzmitry, “A Quaternion-Based Rendering Pipeline,” in Wolfgang Engel, ed., GPU Pro3, CRC Press, pp. 265–273, 2012.

He notes one additional bit you need to store: handedness. He explains how quaternions are different than matrices in this way. Malyshau also talks about how quaternions can also save on texture access and storage.

It's also worth pointing out that, for good or ill, the normal workflow is TBN (tangent, bitangent, normal) in most modelers, AFAIK.

[PeterSikachev]

You can convert TBN into quaternions. Nobody makes artists set quats directly, obviously. The only thing which quaternions do not allow is a skewed basis, but I would argue that for vertex basis you don't really need it.

[erich666]

You can convert TBN into quaternions.

Good point, and that's simple enough. If quaternions are the default, we need to document how to go in both directions (or give a good reference) to make it easy for users to adopt.

[xelatihy]

Not sure if adding quaternions implies removing NORMAL, since quaternions do imply an orthornomal basis. I would suggest no.

But in that case, using 4D tangents (x,y,z,handedness) feels like a better compromised. No need to store bitangent and trivial to encode and compute with. To get the bitangent, just take the cross product. In this manner, BITANGENT may be added in case one wants a non-orthonormal basis, but they are not required.

It seems storing tangents takes the same space as quaternions if normals are stored separately, and they may be encoded more efficiently using the same normal encoding tricks.

Just my 2cents.

[PeterSikachev]

Of course it does. In fact, xyz components of the quaternion are Normal_xyz * sin(theta/2).

[xelatihy]

I guess my post was not clear. Even if using quaternions, I would suggest to not remove NORMAL to ease adoption. But converting on input is always an option, I guess.

A curiosity. Which game engines use Quaternions directly instead of normals?.
And which editors do?

It is very common in the area of graphics where I work, but I am curious how widespread this use is.

[darksylinc]

In Ogre we use QTangents as described by CryTek.
The idea behind QTangents is:

• Use quaternions. Four 16-bit SNORM values.
• Exploit the property that Q = -Q to store whether the bitangent should be reflected in the sign.
• SNORM does not have negative 0; thus the trick above breaks if the value happens to be exactly zero, so a small tweak is needed (force the component storing the sign to never be zero; and renormalize the quaternion to keep it unit length)
• QTangents can't be decoded in the pixel shader as interpolation won't work. It must be done in the VS. There was a game I cannot recall now that managed to process meshes in a way that Quaternion interpolation was possible (decoding them in the PS instead of the VS). However it assumed you could flip the sign of the quaternion offline at will to satisfy the algorithm; which can't be done with QTangents.

We have already code that converts 3 float3 into four 16-bit SNORM values.
Encoding:
https://bitbucket.org/sinbad/ogre/src/18ebdbed2edc61d30927869c7fb0cf3ae5697f0a/OgreMain/src/OgreSubMesh2.cpp?at=v2-1&fileviewer=file-view-default#OgreSubMesh2.cpp-988

Decoding:

vec3 normal = xAxis( qtangent );
vec3 tangent = yAxis( qtangent );
float biNormalReflection = sign( qtangent.w );
vec3 bitangent = cross( normal, tangent ) * biNormalReflection;

The definitions of xAxis and yAxis can be found here: https://bitbucket.org/sinbad/ogre/src/18ebdbed2edc61d30927869c7fb0cf3ae5697f0a/Samples/Media/Hlms/Common/GLSL/QuaternionCode_piece_all.glsl?at=v2-1&fileviewer=file-view-default

Technically, RGBA10_2 could be used, reconstructing one of the components and storing sign in the alpha component, but one needs to be very careful, to not break edge cases when value is 0 or reflection is used.

Just Cause 2 uses an entirely different approach, and uses 4 bytes in total to store all of the data. See http://www.humus.name/Articles/Persson_CreatingVastGameWorlds.pdf

[Marc-B-Reynolds]

xAxis & yAxis have done almost all the work of zAxis...using cross instead is just adding issues.

[darksylinc]

If you're decoding on the CPU yes.

If you're decoding on the GPU, you have to decode them on the Vertex Shader; thus the cross product will very likely be done in the pixel shader so you end up sending 7 interpolants instead of 9 (and actually you may be able to send 6 if you manage to encode the sign in another interpolant).

[PeterSikachev]

You don't have to decode a quaterion to TBN. You can transform a normal from a bump map directly by quaternion in the pixel shader.

[darksylinc]

Quaternions don't interpolate well because as they may double cover.
Ensuring they don't double cover is far from a trivial task, hence there was a paper I said I couldn't remember its name.

[PeterSikachev]

Quaternions DO interpolate well (and better than TBN), they just need a proper alignment which is easily doable for any correctly UV-mapped object. Please, refer to our GPU Pro 5 article for further details.

[jeffdr]

Something I feel may be missing from the conversation here is a bit of consideration for where normal map content comes from. At the risk of saying stuff everyone already knows, here's a quick summary.

Normal maps are almost always "baked". That is, they are automatically generated by casting rays from the low poly mesh onto a high poly mesh to capture the detail missing in the low poly approximation. This process is very sensitive to the way in which tangent basis vectors are set up and interpolated across the triangles. If the baking tool and the renderer do not match in every regard, you will get rather bad normal mapping artifacts.

For game studios this is fairly easily solved. Either the renderer is crafted to match the tool the artists are using in-house, or custom baking tools are altered to produce renderer-friendly meshes and normal maps.

Projects like glTF do not have the benefit of such integration however. Since I presume the purpose of the glTF format is to be of general use, and broadly compatible with content and workflows "in the wild", then the variety of conventions in use by artists today should be a primary, not a secondary, concern.

A rough ordering of the top tools, in descending order of popularity, looks something like this:

xNormal
Substance
3DS Max
Maya
Knald
Blender
Marmoset

For the most part, none of these tools produce identical outputs, even though they all use TBN vectors. They are all slightly different, both in terms of tangent generation and tangent interpolation. This sucks, but it's where we are. I know this because I've recently written my own baker (Marmoset Toolbag) that supports all of the above conventions.

With this in mind I hope an idea like interpolating quaternions over the triangle would be immediately rejected, even though it's probably perfectly elegant engineering-wise, because that would set us directly at odds with essentially every baking tool on the market. At least, if we want to do more with glTF than send efficiently packaged Utah Teapots to each other.

So what would I recommend? Well, the closest thing to a standard in the space has been set by Morten Mikkelson (now at Unity I believe). He wrote a paper, and a C implementation of the algorithm, for generating stable tangent results on any mesh, which is I believe always orthogonal (bitangent can be generated in the shader) and requires no per-pixel renormalization or orthogonalization. It's in use in both Unity and Unreal, as well as several bakers (xNormal, Substance, Knald, Marmoset). So that would be a pretty strong vote for N3 + T4 vertex storage, and simple interpolation of the same, I think.

If you really, really wanted the extra space, this could be switched to quaternion (but you shouldn't interpolate via quaternion, you'd have to use N+T for that). If you do this though, you should provide a tool for converting, since essentially all of our user's input data will come in TBN form. Personally, I wouldn't want to burden the users or the import process with any of this, and I'd just stick with tangent vectors, which everyone is using already.

Hope this helps!

[javagl]

At the risk of saying stuff everyone already knows, here's a quick summary.

No worries. After the first ~20 comments here, I was about to ask: What is this all about?. Although I still don't know the answer (really), your comment at least leaves me with the high-level answer: Stuff about normals that is so complicated that even the major game studios don't have a silver bullet for it. I hope that here, a solution may be found that is sustainable and for which an undoubted, unambiguous (CC0) reference implementation will be provided.

[tparisi]

My $.02 and that's all it's worth, not my long suit here: TBN is SO much easier to understand for implementors. OTOH as long as tools and viewers can easily convert and the math is documented, that would be consistent with how to do rotations... for whatever that's worth.

[pjcozzi]

The tweet for this thread had some nice comments, adding them here so, at the least, we have a single archive of this discussion:

@kenpex

• quats are good also when U don't want to pass too much data from VS->PS
• another popular alternative is to store norm+tang+sign of bitangent
• the main deal is to make sure U use the same TF when U encode nmaps
• that's the "hardest" part, if U do that then precision is no prob

@Humus

• A couple of ways to store Tangent Space in 4 bytes in JC2: http://www.humus.name/Articles/Persson_CreatingVastGameWorlds.pdf

@ApecTobias

• Currently reconstructing TBN vectors from quat in vertex shader like Crytek. Interpolation of quats does not allow arbitrary UVs :/

@matiasgoldberg

• There is a game that managed to interpolate UVs in the PS, but I cannot recall the paper nor the game now
• However it was not compatible with Crytek's QTangents (i.e. no reflection, or refl. is stored separately)

@knarkowicz

• Most common - TB+sign. It's also default in tools like xNormal (imp for baking).

[pjcozzi]

I'll ask that we try to converge on this topic by this Wednesday, Feb 8, so we can get the spec and ecosystem in order.

---

As for the best approach, I originally suggested quaternions, but @jeffdr's comments on the toolchain make me think it might create ecosystem issues.

@mlimper's suggestion seems reasonable in that is is pretty efficient, easy to implement, and covers all cases:

• NORMAL + TANGENT4 (optimized, assumes orthonormal case)
• NORMAL + TANGENT + BITANGENT (fits standard pipelines, non-orthonormal case possible)

I'll also add that glTF does not define the datatype for vertex attribute semantics (however, it does for uniform semantics) so a user could short these as ushort for more saving as suggested by @jeffdr (the spec would need to define the sign for T4 since -1 isn't representable).

Any other thoughts on this or another approach?

[darksylinc]

What is glTF trying to solve?

If it's being a interchange format (aka "Binary Collada"); then NORMAL + TANGENT4 or NORMAL + TANGENT + BITANGENT is great. Last covers all cases, it's lossless; former one covers most cases, is not super fat.

Although if this is the aim, personally I'll probably stick to OpenGEX.

If the goal is being a format with the hopes of actually being used in the wild (WebGL apps, Indie/AAA games, simulations, mobile apps, etc) including VR applications, then something I learnt from VR is that it pushes EVERYTHING.

From runtime performance to loading performance. That means I need a small lossy quaternion representation (or the one used by Just Cause) and assume my tris are orthonormal. Consume as little as bandwidth as possible, and as less interpolators as possible.

That means I'll need these compact representations being discussed here. I could sure do the conversion during loading, right? Well then I hurt loading performance, which is also important for VR.
Also compact representations consume less disk space thus taking less time to download (i.e. WebGL apps)

IMHO if glTF wants to have the slightest chance of being used and not die as a format that nobody picks up it has to support compact/alternative representations.

[erich666]

Good summary @pjcozzi, and @darksylinc. Two quick comments:

In Ogre we use QTangents as described by CryTek.

Thanks! The presentation is here, PDF here< which I think is worth adding to the summary.

In that presentation they seem to agree with @PeterSikachev and his comment:

I agree, quats do not support non-orthogonal frames, but, frankly, it is usually a problem with DCC tools which should be fixed in the exporter.

Slide 10 of the Crytek presentation notes:

About Tangent Frames
Please make them orthogonal!
If they are not, you are introducing skewing

For me it seems to come down to whether non-orthogonal tangent frames are something that are the norm and are not corrected, and are hard to correct. I know very little here about content creation: how are these corrected? Peter says it can be fixed in the exporter, but it seems like if you skew a texture on a surface, that's something the artist needs to fix up front, an exporter can't fix the tangent frame AFAIK. But I'm uninformed, so I'd love to learn.

If it's easy to teach people to fix things up, "here's the code", then quaternions (in some form - there are two competing methods identified so far) give good compression (slide 17 of the QTangents says they're a win). If TBN is how most people do it (which from what I can tell is true) and there's no obvious way to clean up and move to quaternions, then TBN needs to be supported (probably storing just TB and reflection, and not the normal). If glTF is meant to be a flexible interchange format (I don't think that's a goal, but hey I'd love to have something that is...), then TBN is needed. However, two representations starts to wander into TIFF and VRML2 (and Collada, AFAIK) territory, of the file format meaning everything to everyone but no one being able to 100% interpret the other's data.

[lexaknyazev]

If glTF is meant to be a flexible interchange format (I don't think that's a goal, but hey I'd love to have something that is...), then TBN is needed.

glTF aims to be a "JPEG-for-3D", hence, runtime consumption format.

If we stick to orthonormal basis, this issue becomes about implementation (quats vs different flavors of optimized TBN). Otherwise, it looks like there's no alternative to full TBN support.

For me it seems to come down to whether non-orthogonal tangent frames are something that are the norm and are not corrected, and are hard to correct.

Also from @PeterSikachev:

Skewed basis isn't needed much, unless you have some exotic case like non-uniform scaling.

[bghgary]

Here is an implementation of NORMAL + TANGENT4 in BabylonJS: BabylonJS/Babylon.js#1911 (thanks to @dewadswo)

I've also updated @sbtron's test scene to use models that include TANGENT4 in the glTF. Here is a direct link: https://sbtron.github.io/BabylonJS-glTFLoader/?model=PillowPlane&environment=darkPark&pointLight=true&showTangentSpace=true

[mlimper]

Hey, great progress - good to see this!

I worked on a small C++ tangent generator, which is public domain:
https://github.com/mlimper/tgen/blob/master/README.md

I am using this code to generate tangents for export, and also for baking normal maps. It delivers useable results, at least for my use cases - two simple examples are shown on the page. Sketchfab is also able to render the assets with those maps (with its own tangents) - at least it does not seem totally broken to me:
https://skfb.ly/6oAG6

After the great explanations of @jeffdr above, I checked and tried the available implementation of Morten Mikkelson. However, I found the way it works a bit too "intrusive", as it completely rearranges your index / vertex data. My code is a bit more modular, I hope.

Looking forward to feedback , comments and contributions!

[bghgary]

@McNopper @mlimper @AurL Just merged the tangent support into the core spec. Can you check your implementations/samples to ensure they match the spec? I believe there is currently some discrepancies on the attribute name (TANGENT4 vs. TANGENT). The name TANGENT is what we have committed to the repo.

[McNopper]

I am already using VEC4 for TANGENT with the correct interpretation.
What I need to fix is, that the naming is TANGENT and nor TANGENT4. Also, I need to kick out binormal/bitangent - if still present.

[McNopper]

Okay, fixed it:
KhronosGroup/glTF-Sample-Models#46

[mlimper]

Thanks for the update! Just checked: TGen is not affected (doesn't generate glTF content), and our MOPS CLI tool already uses the TANGENT convention, as it was written against your reference glTF loader for BabylonJS ;-)

[AurL]

@bghgary Thanks. I will update this

[pjcozzi]

Updated in #826