Add layers and layers_cull_mode properties to BaseMaterial3D

[thompsop1sou]

This PR adds two new properties to BaseMaterial3D which allow it to render only on selected layers:

• layers - Determines which layers this material will be rendered on.
• layers_cull_mode - Determines how to cull the material on the layers where it is not rendered. Has three options:
  • LAYERS_CULL_DISABLED - No layer-based culling will occur. (Exactly the same as current BaseMaterial3D.)
  • LAYERS_CULL_VERTEX - Layer-based culling will occur via code generated in the vertex() function.
  • LAYERS_CULL_FRAGMENT - Layer-based culling will occur via code generated in the fragment() function.

Motivation

The purpose of this is to allow users to apply StandardMaterial3D or ORMMaterial3D to an object only on certain render layers. Godot shaders can already accomplish this, and other effects, through use of CAMERA_VISIBLE_LAYERS (see #67387). This PR exposes some of that same functionality to users of StandardMaterial3D and ORMMaterial3D. Combined with the ability to add more materials via next_pass, this will allow users a greater level of control over how their objects render on different layers.

Implementation

If either LAYERS_CULL_VERTEX or LAYERS_CULL_FRAGMENT are selected, a uniform will be added to the generated Godot shader code. It will be controlled by the new layers property.

uniform uint layers = 1u;

If the mode LAYERS_CULL_VERTEX is selected, the following will be added inside the vertex() function in the generated Godot shader code:

if ((CAMERA_VISIBLE_LAYERS & layers) == 0u) { // Begin vertex cull mode
	VERTEX = vec3(1.0 / 0.0);
} else {
	// other generated vertex shader code...
} // End vertex cull mode

Note: Assuming this PR is considered, I could definitely use some input on the vertex cull mode. I know this works on my own machine. However, I'm not familiar enough with shaders and rendering to know whether this will work on all machines. (Or whether there is a more performant way to do it.)

If the mode LAYERS_CULL_FRAGMENT is selected, the following will be added at the beginning of the fragment() function in the generated Godot shader code:

if ((CAMERA_VISIBLE_LAYERS & layers) == 0u) {
	discard;
}

---

Bugsquad edit: Implements and closes godotengine/godot-proposals#11210

[tetrapod00]

You may find it helpful to run pre-commit locally: https://docs.godotengine.org/en/latest/contributing/development/code_style_guidelines.html#pre-commit-hook

[clayjohn]

My experience with this trick to discard a triangle as been mixed at best. On at least a few devices I tested on this resulted in corrupted triangles instead of discarding the triangle

[thompsop1sou]

Do you know if there is a best way to do this? I've seen several suggestions online, but i don't have the experience to know which might work the best:

• Set all the vertices to NaN/Inf
• Set all vertices outside of the view frustum
• Set all the vertices to the same value so that the primitive collapses to a point

I did make a note in the class documentation about the vertex method potentially not working on all systems. Don't know if that would be enough to allow this through:

[clayjohn]

I've had the best luck setting the vertex outside the view frustum. To do that here you would need to use the POSITION built in instead of VERTEX because VERTEX is the view space position.

Overall though I'm not certain the shader is the best place for this check. With this approach the draw call still happens even if it's filtered out.

You could pass the later into the rendering server and then filter the material when the instance buffer is being set up. That way you can skip the draw call entirely (and all the associated setup cost)

[thompsop1sou]

Okay, that makes sense about stopping it before the draw call. I think both lyuma and tetrapod (in the proposal) mentioned the same thing. The main reason i went about it this way is i wasn't sure how to do it the other way 😅 (I know C++ and Godot shader code, but not OpenGL or Vulcan.) I might look into it, though, and see if i can figure it out. Would you have any pointers on where i could start, such as where visual instances are culled?

If i'm able to figure it out, would it make more sense for this property to go on Material instead of BaseMaterial3D? I suppose there's probably a significant difference between 3D and 2D layers/rendering, which might make things more complicated if working with just Material. But maybe i could copy how next_pass/render_priority work and only expose the layers property for 3D materials.

[Calinou]

I used VERTEX = vec3(-INF); to hide particles on contact back when I implemented particle collision (the current implementation sets ACTIVE = false instead, a particles-only built-in).

[lyuma]

Set all the vertices to NaN/Inf

I believe your code for this was not correct. You were doing `vec3(1.0/0.0)` but the problem is many compilers/optimizers (these are inside the driver itself) will optimize out certain statements containing values known at compile time to be NaN. I usually use sqrt(value not known at compile time but always negative). For example, `VERTEX = vec3(sqrt(-VIEW_MATRIX[3].w));` If done right without triggering compiler optimizers, this approach has worked on all PC hardware I have tested as well as all android based VR devices (tested on Quest 1 and newer). I have not tested on other mobile chipsets so I will defer to clayjohn's expertise.

Set all vertices outside of the view frustum

Can also do this, but it doesn't help if everything is degenerate anyway.

Set all the vertices to the same value so that the primitive collapses to a point

This is the correct/best answer if the material setting affects the whole mesh. No need to depend on undocumented behavior if you can achieve the same effect by making all triangles degenerate. Personally, though, I would like to question the overall approach of doing this. Why are we submitting the draw call and then culling it in the vertex shader? It seems to me that this culling property makes more sense to be applied in RenderingServer similar to how layer_cull_mask currently works on visual instances. The proposal indicates a desire to make the debug materials on a demo project easier to test, but I think there ought to be a better way to do this that doesn't involve swapping around layers on materials. It just doesn't feel like the right place to do this even if it is possible, given that it includes a performance cost that users might not expect. (You can write this code in a shader material if needed)

[thompsop1sou]

@clayjohn @lyuma I looked into culling before the draw call, but it's well beyond me. If you all definitely want to approach it from that direction, i could make a new issue (or amend the current issue) asking for it to be implemented on all 3D materials, and then see if someone else comes along who wants to tackle it.

If you think the current solution could work with minor tweaks, I'd be happy to make any necessary changes, such as changing how it is culled in the vertex function. If so, let me know what the best solution for this is... seems like maybe it is setting the vertices outside of the view frustum, per clayjohn's suggestion above.

[thompsop1sou]

The proposal indicates a desire to make the debug materials on a demo project easier to test, but I think there ought to be a better way to do this that doesn't involve swapping around layers on materials. It just doesn't feel like the right place to do this even if it is possible, given that it includes a performance cost that users might not expect. (You can write this code in a shader material if needed)

@lyuma Regarding this, my original goal for this PR was to provide a way for users who don't know shaders to more easily use an example project i'm working on, which you can find here. (I know this project isn't an optimal solution. I've just put it up as a possible workaround until the compositor is complete.)

I also thought being able to render different materials on different layers might be helpful for other visual effects, such as thermal vision. Of course that's already possible in shaders, but I thought it could be helpful to make that more accessible.