-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
glTF 2.0: texture packing for Metallic-Roughness? #857
Comments
Somehow related to KhronosGroup/glTF#857
That actually sounds like a reasonable proposal - especially as it's already being used elsewhere like that. The ordering should not be an issue IMHO, if we can make it fit existing conventions, that sounds like a plus. Only thing we might want to make sure is that people can use the extension safely without having an occlusion map... which then probably means that somewhere inside the glTF material description we will want to specify whether the texture content has an occlusion channel or not. @sbtron and colleagues, what do you think? |
I fully support merging them and defining static channel mapping at least for MR. How would SpecGloss model fit with such change? |
Without swizzling support, this would require shader regeneration to cover arbitrary cases. |
One fixed mapping sounds OK. Why have multiple ones? Separate textures seems like an unnecessary step, given that we're aiming for a delivery format - it's not supposed to be authored as-is. |
The idea with the additional maps in the core spec was to be able to share them across multiple material models so other material models defined through extensions could also use them. The spec gloss material defined in extension can also use the common occlusion map as its currently proposed - https://github.com/sbtron/glTF/tree/KHRpbrSpecGloss/extensions/Khronos/KHR_materials_pbrSpecularGlossiness#additional-maps. Packing it with metal roughness while useful for metal roughness would make it unavailable to spec gloss or other future material extensions. |
@sbtron But an exported spec/gloss model wouldn't want a loose occlusion map would it? It would be more useful to bundle the occlusion and gloss maps together for a spec/gloss model, I would think. The ability to deliver a tightly packed model is more in line with the goals of glTF than the ability to re-use loose textures across extensions, I would think. |
@emackey the current specgloss texture doesn't have additional channels available as it is already packing spec (RGB) and gloss (A) together - https://github.com/sbtron/glTF/tree/KHRpbrSpecGloss/extensions/Khronos/KHR_materials_pbrSpecularGlossiness#specularglossinesstexture |
Ah OK, understood. Even so, the spec/gloss extension could simply define its own occlusion map and not rely on the core one, right? A single model won't use both approaches in the same file, so each approach should be optimized for itself, as opposed to re-usability between approaches. |
The intention is that there can be both approaches in the same file. See https://github.com/sbtron/glTF/tree/KHRpbrSpecGloss/extensions/Khronos/KHR_materials_pbrSpecularGlossiness#conformance-and-best-practices The example models have both at the same time. https://github.com/KhronosGroup/glTF-Sample-Models/tree/master/2.0/BoomBox/glTF-pbrSpecularGlossiness |
My mistake again. Is there no way to implement this without causing pain to the spec/gloss workflow? Since the proposal is Occlusion (R), Roughness (G), Metallic (B), could we say that rough/metal (G/B) are optional when extensions are present, and a grayscale version of the image can be supplied offering only occlusion? I do think there's some value in conforming to an existing RGB mapping instead of inventing a new one, albeit limited value, not worth messing up the spec/gloss extension. That said, the original post above contains a link to Substance Painter export settings, and I just confirmed that Substance Painter 2.4 can have user-defined channel mappings on export. So even if we stick to a channel mapping that's unique to glTF, it's not unreasonable to expect existing authoring tools to produce it. |
This also frees one texture unit. Possible layout with both models (note, that we still have unpopulated A channel in textures 2, 5, and 6):
One more possible caveat: we can't use JPEG for BaseColor / Diffuse textures, when they contain alpha channel. Is it OK with established workflows? |
Another thing to consider is whether the occlusion channel is in sRGB or linear space. Unity imports models using a grayscale texture in sRGB space while Unreal uses the aforementioned packed structure but in linear space. Supposedly using linear can cause banding, but I still have yet to find any real evidence that supports this. |
Although the compression for JPG heavily depends on the image contents, consider, for example, the
When the spec dictates (for example, for Spec/Gloss case) that the images must have an alpha channel, and thus, that they must be stored as PNG, then the arguments for combining these channels into a single (namely, that "glTF is a transmission format and has to be compact") are rendered absurd by the fact that two images could be created which (essentially) contain the same information, but only have one tenth of the size of the proposed format. Disclaimer: I don't have a clue about other image formats, KTX and whatnot. All I can say is that "common" textures as (highly-compressed) PNGs are still up to 10 times larger than a JPG with a high quality setting that is hardly distinguishable from the original PNG. So I'd at least advocate for not enforcing the use of PNG in the spec. |
Lossy JPG, even at high quality settings, may not be the best format for metal/rough/occlusion. In JPG, high-contrast lines in a single color channel tend to bleed into the other two channels, because for normal images that's not too noticable to the eye. JPG lossiness is based on what human vision can't see in an image. When the 3 channels represent un-related linear values, such as occlusion, metal, and roughness, a high-contrast boundary in one channel may not be present in the other channels, and JPG could easly allow such details to bleed into the wrong channels. PNG is non-lossy compression and shouldn't have this particular problem (although browsers do a terrible job auto-converting color spaces in PNG files in uncontrollable ways, but I digress). |
That's valid reason against JPGs for MR textures. What about
An updated texture/image definitions PR explicitly demands disabling any browser conversions. Please see #860. |
JPG should be fine for basecolor RGB. It does not support Alpha, though. Likewise, for a specular map (although I've demonstrated some spec/gloss ignorance in this thread already), I'd wager that an RGB specular map is essentially an image that JPG would preserve nicely. But again, the gloss map needs the alpha channel, which JPG does not offer. So that's a non-starter unless the gloss map is moved elsewhere. I'm not sure what JPG would do to a normal map. At least it doesn't need the alpha channel, and the RGB channels are more directly related than a roughness channel is to a metallic channel. Even so I'm not sure what damage JPG's compression would do to vectors. |
@emackey Sure, sharp edges may cause artifacts. And it's more likely that the "bleeding" will be more visible when it affects e.g. an emission channel than e.g. an occlusion channel. So in general, it's difficult to make a statement about how "noticable" they are for certain channels, certain textures and certain compression factors. I'm just mentioning this, and leave it to others to either draw conclusions from that or not, but the point is: The potential size difference between PNG and JPG is huge. So huge that one Metallic/Roughness/Occlusion PNG may easily still be three times larger than three individual JPGs, each storing one channel... |
Agreed, but keep in mind, this size difference is only in terms of file size or network bandwidth, not GPU memory. Both formats result in un-compressed arrays of pixels in memory. So using three JPGs in place of a single PNG may save network traffic, but costs both texture memory and texture units. Here's my impression of the various goals and priorities that have been discussed on this thread so far. Let me know if I got this all straight.
|
I'm not an expert on loading images on the web. Is there an efficient way to load two images (say an RGB jpg and a grayscale jpg) into a single 4-channel texture (where the grayscale jpg is loaded into the alpha channel)? I had always assumed that separating the images for each type of data would be an issue for texture bandwidth, but if it's efficient to load multiple images into one texture, then perhaps it's not an issue. If it is efficient, then maybe we should separate each type of data as different unpacked images so that jpg can be used when appropriate? |
All WebGL texture functions operate on one texture source at a time. Thoughts? |
I propose that we move the discussion about png/jpg into a separate issue where I will try to quote / summarize what happened here. The original issue is regarding packing of the textures which we can continue to resolve here. Any objections? |
I think the core issue here is selecting which channels go where, and unfortunately JPG has some influence on that by failing to provide a proper alpha channel, encouraging us to sway towards RGB textures instead of RGBA in the spec. There's another reason (outside of JPG) to sway towards RGB as well: Many image manipulation software packages deliberately ignore or discard RGB data for pixels where
Obviously this is not a problem for the baseColor or diffuse map (as the alpha, if supplied, is intended to be used the traditional way where zero values mean discard). I suspect this may not be a problem for Spec/Gloss (Does the spec color matter at all when gloss == 0? I would expect it to not matter). But I would advise against dumping occlusion into some random alpha channel of an unrelated map.
Now this is an interesting option: Any given glTF runtime implementation has the option to do this kind of pre-pass on any glTF assets, to make them conform to the local engine's needs. If an engine wants the occlusion channel jammed into the alpha channel of the normal map, sure, do it in a prepass step, the glTF spec doesn't mind at all. glTF is a "transmission format" spec after all, we should worry less about what layout the engine needs internally and more about how to best pack these assets for transmission, with the knowledge that engines can unpack and rearrange assets at will. I could imagine a prepass step to convert any metal/rough-only assets to spec/gloss, and then offering support of both types of glTF while internally only needing a spec/gloss PBR shader with a condensed set of maps. So in summaryAfter all this discussion, I'm still leaning back to the same idea we had a few days ago for the transmission format, now with the knowledge that individual engines have the option to remap all this during a pre-pass step if they don't like the delivered channel mappings.
All of these alpha channels are optional (for models that don't use translucency, which no web developer expects JPG to provide), with the exception of (4) the glossiness one. But that's in an extension, so if someone thought it was critical to avoid, they could avoid the extension, and produce a core glTF 2.0 file with no alpha channels at all. Their engine could remap the channels at load time and maybe save a texture unit or two without their knowledge. Thoughts? |
Whoops, correction on (2) above: Occlusion is (R), Roughness is (G), and Metallic is (B), per the original post in this issue. |
I don't know what the effect is, but that concern makes sense. Would it be kicking the can down the road to change the metallicRoughness texture to use the Original post notwithstanding, I don't have a strong preference here. Wanted to suggest UE4's ordering as nice-to-have in the absence of other considerations, of which there are apparently plenty. 🙂 |
Hmm, I don't think so. With @donmccurdy's most recent proposal, the difference is really whether or not occlusion shares an index with metalRough, regardless of the presence of other workflows. I like the strategy of saying occlusion is always Does that resolve any of the concerns here? |
The combination of a grayscale with a 2-channel texture is also a nice-to-have in my humble opinion 😄 |
If you look carefully at the proposal, there is a difference between "metallic-roughness only" and "both workflows" in that occlusion is stored in a separate texture for the latter. Sampling one texture vs sampling two textures in the shader will have different performance characteristics. |
Sure, but that's only because the Does the current spec forbid reuse of index values? I could imagine a test model where the normal map index was copy-pasted to the base color index, resulting in an oddly-purple-colored model, but is there anything technically preventing re-use of index values like this? I'm saying I don't think this is a special-case of index value re-use. It's just a case where it makes sense to do so. |
Hm, let's assume 2 textures with If we stick with the current spec, we will have imposed this lower-performance scenario on all Metal-Rough assets. Alternatives to this appear to be:
I can't think of a 4th combination, unless we bundle occlusion with a less-common data type. |
Thanks @donmccurdy. I can't think of another combination either. We have 4 choices where all of them have downsides. None of the alternatives jump out to me as the solution that just works. Personally, I prefer consistent behavior over more complicated solutions, considering performance trade-offs. And given that we already have multiple implementations out there, I would want a solid solution to switch to. Thoughts? |
I think the metal/rough-only workflow could benefit from a slight change to the 2.0 spec as it stands today:
|
My own preference would be to do the If that updating cost is already too high, then one more option along the same lines:
This is backwards-compatible with existing authoring tools, but at least leaves open the possibility of packing metallic + roughness + occlusion later. |
Regarding the JPG/PNG discussion, there is a tool called pngquant that quantized PNGs into a palette 8-bit image which reduces the size a lot. Note that I had to modify the code a bit to make it not automatically premultiply alpha. Quality settings for JPG and quantized PNG are both 90. Results:
See the difference for yourself. BoomBox - https://sbtron.github.io/BabylonJS-glTFLoader/?imageFormat=jpg-with-quantized-png&model=BoomBox FarmLandDiorama - https://sbtron.github.io/BabylonJS-glTFLoader/?imageFormat=jpg-with-quantized-png&model=FarmLandDiorama I've also included a per-channel pixel diff between PNG and PNG-quantized in the corresponding quantized folders of each model. FarmLandDiorama - https://github.com/sbtron/BabylonJS-glTFLoader/tree/master/models/2.0/jpg-with-quantized-png/FarmLandDiorama/glTF-pbrSpecularGlossiness/diff Here are some stats for BoomBox and FarmLandDiorama models:
|
Slightly related - perceptual JPEG encoder: |
Here is what I'm testing out with the packing: For assets without the KHR_materials_pbrSpecularGlossiness extension:
For assets with the KHR_materials_pbrSpecularGlossiness extension:
Here are the implementation changes required to do this in BabylonJS: Here are the changes to the models: |
@bghgary That looks good to me Looks like the sample models stick with |
Beat me to the punch. Was about to reply with this. Implementing this change in the exporter was slightly more complicated, but it wasn't terrible.
+1 if you agree and we can close on this issue. |
No, fixed the second one. |
This looks great, thanks @bghgary! We've got an implementation and demo for metal-rough in three.js now. Still tracking down a visual artifact. Spec-gloss in progress, thanks to @takahirox. |
Thanks @donmccurdy for pushing glTF to be compatible with my change to ThreeJS here from January: mrdoob/three.js#10691 This is awesome stuff and nice to see everything converging on best practice. Hopefully this standard can spread to Unity in the future. |
Thanks @donmccurdy @bghgary, we will apply the update on our side. |
@bghgary I'm back here to ask about something that is unclear for me (and sorry in advance if I missed something) How should we know if we should use the Looking at the samples Avocado vs BoomBox, I understand that we should take the R channel (occlusion) into account only if |
Well, sorry for the dumb question, I have the answer. |
Updated in #826 |
From this paragraph, I see that the Metallic-Roughness workflow will make use of texture packing by assigning:
Were there plans for using the
B
channel later? If Occlusion is the obvious next choice, is it worth altering the order to match UE4's Occlusion/Roughness/Metallic R/G/B ordering?I'm not very familiar with texture creation workflows, so maybe this is a non-issue and easy for exporters/converters to deal with. But thought I'd ask, as THREE.MeshStandardMaterial is already configured for that OcclusionRoughnessMetallic ordering. Thanks!
The text was updated successfully, but these errors were encountered: