Add a post-light function in the spatial shader, so that proper cel shading can be done

[QbieShay]

Describe the project you are working on:
A game with the type of graphics similar to breath of the wild

Describe the problem or limitation you are having in your project:
TLDR: There is currently no way in Godot to obtain proper cel shading.

Godot's built in toon shader produces a number of bands (one per light that affects the object) and that's due to the fact that each light contribution has a smoothstep applied (depending on the roughness, which i found rather weird) and not the sum of all lights.

After a while, I managed to use Godot's spatial material to produce a toon effect

As you can see in the image there are multiple bands, which is generally not desired. Or rather, each new light creates an additional band, while generally artists would like to control the number of bands and the tint, if any.

An example on how a correct toon material would look like:

This shader (courtesy of DaveDaDev) looks right, but it has the limitation that it can be affected by only one light.
If the code at the end is changed from
DIFFUSE_LIGHT = diffuse.rgb; to DIFFUSE_LIGHT += diffuse.rgb;, it suffers from the same banding problem:

There has been also an attempt to read the light information by rendering the model completely white with normal lighting, and then using a second pass on the material to read the white tint and discretise it.
This approach worked:

But with a huge limitation: the new shaded character is drawn always in front of transparent object because it does a screen read.

In the picture you can see the plane being drawn behind the character, even if it's supposed to be in front.

If the plane is set to depth draw always, the second pass won't draw:

Describe the feature / enhancement and how it helps to overcome the problem or limitation:
What I would like to see is an additional function in the shader which is executed after all the lights accumulate and receives the final diffuse light and specular light.
So, it's one function with 2 built-ins.

Writing a cel shader, with this function, would look like:

light_post(){
    DIFFUSE_LIGHT = step(DIFFUSE_LIGHT, vec3(0.5));
}

Or also, in case a tint is used:

uniform sampler2D ramp;

light_post(){
    // sample from the ramp depending on the light intensity
    vec2 ramp_uv =  vec2(get_hsv_v(DIFFUSE_LIGHT), 0.5).rgb;
    DIFFUSE_LIGHT = texture(ramp, ramp_uv);
}

This function would be enough to do all the desired effect for cel shading such as step and using color ramps to shift the tint of parts in light/shade.

Describe how your proposal will work, with code, pseudocode, mockups, and/or diagrams:
This proposal would mean to add code into Godot's default shader and an additional step in the string substitution in the shader.cpp file. It should be possible to use a light_post() entry function in the shader code, and inside of this function DIFFUSE_LIGHT and SPECULAR_LIGHT should be available with all the accumulated effects from adding all the lights.

If this enhancement will not be used often, can it be worked around with a few lines of script?:
There is no way, with the current architecture, to obtain cel shading with a single band with multiple lights.

Is there a reason why this should be core and not an add-on in the asset library?:
It can't be done as an addon or library, since this functionality is not achieavable without touching the core (not enough information is exposed to the users to obtain this effect)

[jitspoe]

I'd also be interested in this feature. My use case is a bit different:

I'm creating a retro FPS game, and I want to palette the textures POST-lighting, but PRE-post effects, so I can have that nice paletted look using more modern lighting with normal maps and whatnot, but also I want fog and stuff to use full color depth, so it looks smooth. I need something between the lighting step and post processing.

[morganholly]

this would also help me for a shader i made recently, it's doing a bit of math per light but i could put most of it in a post light function, which would allow me to make that more customizable while also being more performant if lit by several lights

[QbieShay]

Hey all!

I'm both happy and sad that this issue is still relevant after so long.

To give more context on this, lights are implemented with a multi pass approach in the compatibility renderer, which makes it impossible to use a post light function. Changing this is first, a lot of work, second, a very bad idea for mobile.
Here we're paying the cost of trying to support so many platforms at once.

You can continue following the discussion here #8050 , but unfortunately this can't come anytime soon.

[jitspoe]

Ah, yeah, I could see that being an issue. How is fog handled in multipass then?

I tried running my game in the compatibility render and 90% of the lights didn't work in any case, so if this is something that only worked in 1 render, that would be acceptable for me, since it doesn't seem my scenes have any hope of working anyway. 😅

[QbieShay]

Well, the fact that doesn't work now doesn't mean it won't work forever. In fact getting the compatibility renderer up to speed is something that's on our mind for sure.

One thing we do really care about (i use plural we as the rendering team) is to maintain compatibility between all renderers when writing shaders, because we don't want to end up with many renderers each with their own special features incompatible with each other.

For this reason this feature is particularly difficult to add.

[HydrogenC]

Yes this is crucial for applying a ramp texture after accumulating all the lights. Will godotengine/godot#94427 be a possible workaround, since we can modify how the lighting is processed by replacing the base shader? If this PR allow modifying parts of the base shader in https://github.com/godotengine/godot/blob/master/servers/rendering/renderer_rd/shaders/forward_clustered/scene_forward_clustered.glsl, then this problem could be solved. Anyway, it's likely that eventually this could be solved when a deferred renderer is implemented in Godot.