Broken AOMap in USDZ with Transform2d

[kolodi]

After introducing support for texture repeat and offset in #21852, the ambient occlusion map is repeated many times, despite the fact that we are setting the scale separately for each texture:
float2 inputs:scale = ${ buildVector2( texture.repeat ) }

To Reproduce

Steps to reproduce the behavior:

1. Have a model with tiled diffuse map and (obviously not tiled) AO map.
2. Export to USDZ
3. unzip USDZ archive and open USDA file
4. notice different scaling for each textures, diffuse texture may have scale major/minor from (1,1) and AO map should have a scale of (1,1)
5. Transfer USDZ file to iPhone/iPad
6. Open it in QuickLook and see AO map being tiled multiple times (probably as many as diffuse)

Expected behavior

AO map should not be tiled/repeated.

Screenshots

from #21871

Platform:

• Device: [Desktop, Mobile]
• OS: [Windows, MacOS, Linux, iOS]
• Browser: [Chrome, Firefox, Safari, Edge]
• Three.js version: [dev, r129]

[donmccurdy]

If you hand edit the file to have the transform you expect, is it working in QuickLook? Based on the notes in https://github.com/google/usd_from_gltf#emulated-functionality-for-ar-quick-look, I'd thought it didn't support this:

• Texture UV transforms [are not supported]. These are emulated by baking transforms into vertex UVs.
• Multiple UV sets [are not supported].

^Since three.js requires multiple UVs for AO, and QuickLook apparently doesn't support them, that's a bit of an obstacle. 😕

[kolodi]

Exported USDZ file has the correct transform and AO map has the scale of (1,1). I have tried to convert the same 3D model with:

1. USDZExporter
2. USDPython v 0.64 (by Apple)
3. USDTools by Pixar

The resulting USDA may differ in structure but Transform2d for textures is the same.
You actually can open the USDZ file in different viewers (like xCode) and it is rendered just fine. And as we can see, Transform2d actually works in QuickLook, but, apparently, there are additional limitations for USD, specifically, AO map transform is not working correctly and (probably) repeated as many times as diffuse map, BUT, if your diffuse map is exactly (1,1) then AO map works well.

I want to make an important note about UVs. If you have AO map on UV2 which is completely different from UV1 (intentionally made by 3D artist ), then of course, you can't put it on UV1 and hope that it will work well.

I have experimented a bit trying to add the UV2 to USDZ meshes during the export. Then I tried to use st2 PrimVar based on UV2 for transformations. The result was not quite right. I mean it was somehow working, but it seems like tiling was extremely high, like 100x instead of 6-8 set in transform scale. Overall I think it is a good idea dig more into it.

Workaround

As for now, you simply must sacrifice AO map in favor of tiling or vice-versa. One possible solutions if you want to have both is:

1. Use only UV1
2. Scale UV1 of the geometry (down) in such a way that your tiling maps (diffuse, normal...) is repeated many times
3. Scale (down) AO map so it covers exactly the whole UV.

This solutions has 2 major drawbacks:

1. All existing 3d models need to be modified using this particular workaround, which adds many troubles now and later.
2. AO map becomes tiny if you want more repeats (tiles) to the point that it is so bad you don't want to have it at all.
   Basically, you either go for no scale at all and single UV for small objects or a regular tiling but sacrifice AO for bigger ones.

Possible solution

Can we still do something so we can have both AO and tiling in QuickLook?
I think yes, and I am currently working on a possible solution. The idea is:

1. Duplicate meshes for which we have AO map scaled differently than diffuse.
2. Use UV2 in slot for UV1
3. Create specific materials for those meshes that include only AO map

So far I have successfully done 1 and 2. As for n3 there are a couple of challenges. In particular, we want to render parts of the geometry completely transparent and do not interact with lighting where AO map is near white .

If you want to help me with this, do not hesitate ;-)

[elalish]

This all sounds great, but I wonder if we could have a simpler solution short-term? Like detect when the AO map is not going to work right and just disable it? The render above is a big problem, but without AO at all, it would be decent, if not ideal.

[kolodi]

This all sounds great, but I wonder if we could have a simpler solution short-term? Like detect when the AO map is not going to work right and just disable it? The render above is a big problem, but without AO at all, it would be decent, if not ideal.

Looking in GLTF loader, it assumes that AO map should be on the second UV channel and if a geometry where there is a material with AO map does not have one, there is a workaround to create the second UV channel from the first one. The information about whenever the workaround was applied is unknown during the USDZ export. AO map won't work correctly in USDZ if it was initially on secondary UV channel by design. Maybe @Mugen87 knows how to extend the workaround in the way we can preserve the information about initial UV set used by AO map so we can use it to decide to exclude AO map from USDZ export. Please also correct me if any of the assumption I've made is wrong.

There is another deciding factor to exclude AO map from export - when diffuse map is repeated more than once. I will make PR with this workaround soon.

[mrdoob]

This should be now fixed: #25996