Feature Request: OnBeforeRender on Material

[DavidPeicho]

Is your feature request related to a problem? Please describe.

Let's say I have a custom material that is shared by two meshes. This material uses the inverse transform of the mesh. In this case, I have no way to send the inverse transform because it's dependent from the actual bound mesh.

When creating an app with Three.js, it's not really an issue because we control the flow of data. However, when wrapping Three.js in another library, it makes it somewhat difficult to share a ShaderMaterial whose parameters aren't independent from the camera / mesh drawn.

Describe the solution you'd like

On the top of my mind:

• Create a onBeforeRender callback on the material, that would be in charge of updating per-object uniforms. This is similar
  to the onBeforeCompile callback, but called each time something is rendered with this material.

Describe alternatives you've considered

• Use onBeforeRender on each meshes. Annoying because all meshes using this material must be updated and a user would need to attach the callback himself
• Clone the material for each mesh

Additional context

This is highly related to the issue #16922. I think the problem is similar.

[mrdoob]

• Use onBeforeRender on each meshes. Annoying because all meshes using this material must be updated and a user would need to attach the callback himself

I was going to suggest that...

This whole onBefore*, onAfter* part of the API is ugly and fragile... Have you tested NodeMaterial?

[DavidPeicho]

I haven't because I will most likely just add my entire GLSL in one node 😄 Also Right now I like how my shaders are clearly separated and I just use includes. Maybe I should take a closer look at NodeMaterial, any feature there that could help me achieve that on a material level?

I wish I could just do it on the meshes but that's not that easy, because I just don't know in advanced where my material will be attached.

It's fragile I agree, but right now what's happening is that there is a custom path basically for the default materials. Ideally I think it would be nice to be able to do exactly the same thing with any custom material, including having a way to modify transforms / camera.

[mrdoob]

@sunag how do you think this could be done with NodeMaterial?

[sunag]

@sunag how do you think this could be done with NodeMaterial?

Yes. This already is possible in the new NodeMaterial system.

three.js/examples/jsm/renderers/nodes/accessors/ModelNode.js

Line 17 in ea11dda

this.updateType = NodeUpdateType.Object;

You can update a shared uniform for each render object.

three.js/examples/jsm/renderers/nodes/accessors/ModelNode.js

Lines 39 to 55 in ea11dda

	update( frame ) {
	const object = frame.object;
	const inputNode = this._inputNode;
	const scope = this.scope;
	if ( scope === ModelNode.VIEW ) {
	inputNode.value = object.modelViewMatrix;
	} else if ( scope === ModelNode.NORMAL ) {
	inputNode.value = object.normalMatrix;
	}
	}

Is possible update per frame too with objects like camera

three.js/examples/jsm/renderers/nodes/accessors/CameraNode.js

Line 16 in ea11dda

this.updateType = NodeUpdateType.Frame;

[sunag]

I can implement a solution for WebGL too after this PR ( #21117 )... I am currently implementing a light system for selective lights or global using NodeMaterial.

[DavidPeicho]

I had a look a bit at the Node system today. I still need to connect myself the mesh to the update though no? I don't see how that would work automatically.

Where is the frame.object attribute field?

EDIT: Ok my bad, now I understand the code to hook that up is only for WebGPU right now

[sunag]

https://raw.githack.com/sunag/three.js/nodematerial-shared-uniforms/examples/webgpu_instance_uniform.html

Source: https://github.com/sunag/three.js/blob/8f9cbfa9c15d45a5ce8d3edba8cebc246c039592/examples/webgpu_instance_uniform.html

I made a example, the color is updated per mesh.color in render object moment.
But it seems there is a problem... WebGPURenderer still does not support material instance? @Mugen87

[Mugen87]

WebGPURenderer still does not support material instance?

What do you mean with "material instance"?

[sunag]

What do you mean with "material instance"?

A single material for multiples meshes.

[sunag]

If replace materialTemp to material in addMesh() only a mesh is rendered on screen at a time.
https://github.com/sunag/three.js/blob/8f9cbfa9c15d45a5ce8d3edba8cebc246c039592/examples/webgpu_instance_uniform.html#L104-L109

[Mugen87]

Okay, after investigating this issue a refactoring of how bindings are managed is required. Right now, the binding and the actual data are handled in a single object. In the below example, bufferGPU represents the UBO and array the actual data.

three.js/examples/jsm/renderers/webgpu/WebGPUUniformsGroup.js

Lines 22 to 23 in cb99e7e

	this.array = null; // set by the renderer
	this.bufferGPU = null; // set by the renderer

If a material is shared among multiple 3D objects, there is only one instance of bindings objects and thus only one data storage. This is not a problem for textures (because textures are defined on material level) however model related data like the MVP matrix can't be processed this way. They have to be managed per 3D object and the UBO needs to be updated per object with the correct data.

I need some time to think about a possible solution for this...

[DavidPeicho]

@sunag let me know if you need any help for the WebGL version (or even WebGPU).

[sunag]

@Mugen87 I found the problem, is related with the implementation of NodeMaterial... I will create um PR to fix this.

[Mugen87]

I'm curious about your PR 🤓. This can also be fixed when multiple WebGPUUniformsGroup objects share a single bufferGPU. The render and material system just have to inject an existing UBO that is shared across multiple 3D objects. It is not required that such a UBO needs to exist per 3D object. If multiple 3D objects share a program, they can also share all WebGPU resources.

I'm waiting now with my PR until I've seen your change.

[sunag]

They have to be managed per 3D object and the UBO needs to be updated per object with the correct data.

I mean in case of just update the uniforms with the new mesh and draw again with the same material and uniforms but objects differents... This approach work with WebGL but with WebGPU I never tested.

[Mugen87]

Ah okay, I only focus on WebGPU in this topic^^.

[sunag]

I'm curious about your PR

I did not do anything particular... I just made WebGPUNodes work via .get( objects ) instead of .get( material ). A NodeBuilder for each object.

[Mugen87]

If this works, it's a good start! Fell free to file the PR 👍

I'd like to find out if it is possible to further optimize the renderer and material system by only having different bindings if required. However, this will take a while and some experimenting.

I guess we need similar logic anyway if objects should be able to share render pipelines.

[DavidPeicho]

Any news on the WebGL side? I am available few hours a week to help on that 😄

[Mugen87]

@DavidPeicho In context of WebGL, we have to solve #21117 first.

[DavidPeicho]

Any news here now that #21117 is solved?
If you need any help to go forward with that, let me know what you want me to work on and I can give some of my time for that!

[WestLangley]

@DavidPeicho

This material uses the inverse transform of the mesh

If the transform is unscaled, this may help you avoid passing the inverse transform to the shader:
https://en.wikibooks.org/wiki/GLSL_Programming/Applying_Matrix_Transformations

[DavidPeicho]

@WestLangley thanks for the link but it was just one example out of several. It can happen that some data comes from the mesh and isn't easily accessible.

[Mugen87]

The idea is to apply the pattern from the following WebGPU example to WebGLRenderer.

https://threejs.org/examples/webgpu_instance_uniform

However, this requires a deeper integration of the node material into WebGLRenderer so it's possible to process node updates per frame. #21117 was a first step but WebGLRenderer needs to call a similar method like below (which is currently used by WebGPURenderer) so it can utilize more features of the new node system:

three.js/examples/jsm/renderers/webgpu/nodes/WebGPUNodes.js

Lines 44 to 61 in 8837fac

	update( object, camera ) {
	const material = object.material;
	const nodeBuilder = this.get( object );
	const nodeFrame = this.nodeFrame;
	nodeFrame.material = material;
	nodeFrame.camera = camera;
	nodeFrame.object = object;
	for ( const node of nodeBuilder.updateNodes ) {
	nodeFrame.updateNode( node );
	}
	}

[Mugen87]

WebGPURenderer uses a node system by default. However, we can not include the entire node system in the main build of three.js so we have to find another way for WebGLRenderer.

Right now, it's required that the app includes a certain file (WebGLNodes). However, I'm thinking about a different approach based on dependency injection. Meaning if users want to use the node path of WebGLRenderer, they would have to call a new method that injects node classes for further usage.

[DavidPeicho]

Thanks for the info.

If I sum up, right now we simply need to inject the same method that would allow to do the update? Because the build looks already injected by WebGLNodes.

Otherwise the other type of injection sounds nice. Something like renderer.setNodeBuilder() would be great.

I recently tried to start new shaders using the node system, but I have to admit it's a bit difficult and time consuming to use in WebGL in its current state.

[sunag]

@DavidPeicho Look like we have a WebGL version for that :) #22504

[DavidPeicho]

That's amazing news, thanks! I guess now I will have to convert my custom materials to the node system

[DavidPeicho]

Material.onBeforeRender is actually implemented now. Howerver, it's missing some doc. Will try to make a PR.

[Mugen87]

Unfortunately, you can't use Material.onBeforeRender() to update per-object uniforms. Although both spheres use Material.onBeforeRender() as intended, they share the same opacity value depending on the render order.

https://jsfiddle.net/03xm49dp/

Material.onBeforeRender() was only introduced to make the new node material system work with WebGLRenderer. At some point, we should maybe stop this support and refer devs to WebGPURenderer with its WebGL backend instead. And then remove Material.onBeforeRender() since I don't think it adds any additional value (and right now produces only confusing).

[LeviPesin]

At some point, we should maybe stop this support 

I agree with stopping the support for WebGLRenderer+Nodes (regulated by the renderers/webgl-legacy/WebGLNodeBuilder.js). @sunag What do you think?
IIRC it is in the legacy directory for quite a some time now, so isn't it time to officially announce its deprecation and then remove it after 10 releases?

[Mugen87]

Closing. With the removal of the node material integration from WebGLRenderer, the related material callbacks onBuild() and onBeforeRender() have been removed as well (see #28702).

Please use WebGPURenderer if you want to change uniform values per object.