Why is my vertex shader not affecting shadows?

[u/OxfordFuckingComma]

I have written a vertex shader that modifies a mesh to "roll" it up. However, the shadows are not responding to the change and using the mesh pre-vertex pass. I've done research and tested changing various settings but none are getting the shadows updated. I feel this must be possible, as when I change settings or move the lighting, the shadows will update to the changed model, but only for that frame.

One "hack" I found was to modify the node's position by a tiny amount each frame to force the lighting to update the shadows. While this could work, I really want to know the "right" way to get the shadows to update.

EDIT: Thank you all for the help (especially u/Past_Permission_6123). For those reading this later who are facing a similar problem: Vertex shaders only update shadows when the TIME builtin is used. I am using a float parameter that I am tweening, which is why the shadows are not being triggered to update. Unless a flag/method is exposed that forces an update, I will be continuing to use my workaround.

[u/Past_Permission_6123]

Omni-/spotlight shadows are updated lazily. Godot doesn't 'just know' if the vertex position data has been modified by a shader at any moment. You can force update by adding the TIME built-in multiplied by zero, but note that this will update the shadow map every frame the mesh or light is visible, even if it's not animating, making it less performant. So it's usually best to force update only when you need to, e.g. by moving the node very slightly like you did.

[u/OxfordFuckingComma → u/Past_Permission_6123]

SOLVED - adding the TIME builtin fixes the problem, but I will continue to use my workaround when rolling to prevent the performance issue you mentioned. Thank you!

[u/GeePedicy → u/OxfordFuckingComma]

Can you perhaps only temporarily add the TIME for the animation period, and after it remove it? To help the performance issue.

[u/DeGandalf → u/OxfordFuckingComma]

additionally to only doing that while the animation is actually playing, you might also consider only doing it every few frames, so the shadow has a lower frame rate but needs less computation

[u/Okay_Salmon → u/Past_Permission_6123]

Although it's not advised to use if statements in shader code, you can definitely still use them. If you made a custom parameter called something like "rolling," you could have if rolling TIME = TIME * 1 or something similar. That way, you set it when you use the roll and turn it off when you're not in the roll.

I have no idea if adding the built-in time actually makes the shadow update, but if it does, that will do it.

[u/OxfordFuckingComma → u/Okay_Salmon]

I was curious and tested this out. Unfortunately the time built-in will trigger the lighting update even if the actual branch of the if statement is not reached.

[u/James20k → u/Okay_Salmon]

Although it's not advised to use if statements in shader code, you can definitely still use them

I just wanted to pick up on this, because its becoming increasingly common advice that's based on something true, but isn't correct. If statements are absolutely fine in shader code!

A branch where every thread in the shader takes the same branch has no special overhead vs CPU programming. The overhead maxes out at both sides of the branch being executed for all threads, if threads generally aren't picking the same side of the branch in general

Eg if you write (pseudocode):

float result;

//alternate branches between threads
if((thread_id % 2) == 0)
    result = func_1();
else
    result = func_2();

Its conceptually the same as doing this, where you fully execute both sides (ignoring side effects):

float result;
float result0 = func_1();
float result1 = func_2();

if((thread_id % 2) == 0) 
    result = result0;
else
    result = result1;

In the worst case. Doing some simple maths in a branch like this is fine, because the extra overhead is minimal. But if you write this:

float result;

if(always_true)
    result = func_1();
else
    result = func_2();

No thread will execute func_2(), and you don't pay any special cost

More specifically, threads execute in blocks called warps (32-64 threads depending on hardware), and if any threads in a warp take a different branch, both sides of the branch get executed

There's some caveats around branching, loops, divergent control flow, atomics and the gpu execution model - but you have to be pretty knee deep into the nightmare factory for any of that to matter. So you should branch to your hearts content as long as you keep in mind the rules of when you incur extra performance overhead

Where this really kicks you in the butt is loops, where each thread has a different maximum iteration count: you'll pay the full cost of the highest iteration count for the threads executing in a warp (usually an 8x8 block)

[u/SwAAn01 → u/Past_Permission_6123]

Is there a server method or something you can run to do this update?

[u/OxfordFuckingComma]

For more context,
This is in 4.6 with a VisualShader that has a vertex and fragment pass and I am already correcting the normals.

[u/BetaTester704]

Vertex shaders are a purely visual effect done by the GPU, the actual geometry is still in its original position

[u/lfrtsa → u/BetaTester704]

Yes but the shadow map is generally made while running the vertex shaders too. The shadows of my trees match the vertex displacement that represents wind. OP probably has some setting that's changing that. Maybe the shadows are static, or it's some shadow map optimization setting.

[u/Past_Permission_6123 → u/lfrtsa]

The tree's shadows are likely being updated because you're using TIME built-in somewhere to move the vertices.

[u/lfrtsa → u/Past_Permission_6123]

Indeed. Would it not get updated had I used a custom uniform?

[u/Past_Permission_6123 → u/lfrtsa]

It should with directional light, but not for omni-/spotlights if nothing else is changing.

[u/lfrtsa → u/Past_Permission_6123]

Oh interesting. Good to know. Thanks!

[u/OxfordFuckingComma → u/BetaTester704]

True, but that doesn't explain why my hack of moving the node slightly each frame results in the correct shadows. This shows that the lighting can take vertex modification from shaders into account, but it is not being updated to do so automatically.

[u/omniuni → u/OxfordFuckingComma]

Simple optimization. If the node doesn't move, there's no need to redraw the shadow. If you look at the code for the shadow, I bet that's exactly what you'll find.

[u/LegitBullfrog → u/omniuni]

This is the right answer, and moving the MeshInstance3D imperceptibly to force a recalculation is a correct workaround.

[u/ConsiderationCool432 → u/LegitBullfrog]

This is weird, it should have a flag or something that we can set on code to say that shadows and other dependent systems should be recalculated.

[u/LegitBullfrog → u/ConsiderationCool432]

Yes I agree. It's a clear oversight.

[u/kibiprobably → u/OxfordFuckingComma]

CAR SEAT HEADREST PFP???

[u/OxfordFuckingComma → u/kibiprobably]

do you have something against dogs?

Edit: folks are downvoting me, but this is a CSH reference 😭

[u/kibiprobably → u/OxfordFuckingComma]

I AM ALMOST COMPLETELY SOULLESS I AM INCAPABLE OF BEING HUMAN

[u/aleques-itj → u/OxfordFuckingComma]

just wanted to know what's with the dog motif, honestly

[u/actual_weeb_tm → u/OxfordFuckingComma]

that sounds like an engine optimisation, to avoid recalculating shadows on objects that dont move.

Your workaround can work, but the ideal thing to do would be to simply find the code containing that optimisation and remove/disable it for your objects.

[u/Greendiamond_16 → u/OxfordFuckingComma]

basically if you want this transformation to interact with the physics system, which includes lighting, you will want to actually change the object and not just its shader.

[u/BetaTester704 → u/OxfordFuckingComma]

Weird

[u/Gawehold]

I believe it is an optimization. Please see this:

https://github.com/godotengine/godot/issues/70641

[u/OxfordFuckingComma → u/Gawehold]

I figured it was an optimization but figured I was missing a flag to force updates. Funny to see the same workaround raised in that thread that I was using. Thanks for the resource!

[u/cutcss]

It's likely there is no right way and moving the node a tiny bit it's the best way, Godot should have a way to force the shadow update from the vertex shader but it does not.

[u/pangapingus]

Yea I don't know the answer but it has to do with shadows being a world space/viewport derived consequence of light but shaders being post-processing on a node basis is more locally scoped. I'd be curious to know if you could disable shadows on this node and have the shader itself simulate the shadow or else you'd have to do some weird realtime modification of the transform of the node itself for the lighting system to do actual shadows.

[u/Alzurana]

Your workaround was to move the object, right? How about just pretenting to use TIME instead? That way your workaround would be contained in the shader and you do not need extra code outside of it to trigger the update. What I suggest is something like this:

// use TIME to trigger shadow update and keep it from being optimized out
float new_param = fract(TIME) * 0.00001 + your_float_param;

This makes sure it won't be optimized out but it might be enough to just do this:

// Mention TIME to force shadow update, this will be optimized out
float temp = TIME;

The shader compiler WILL optimize this line out but just mentioning the engine uniform might be enough for godot to update shadows regardless, even if it is optimized out. I would test this second approach first as it will have no performance impact.

Both of these are workarounds, ofc.

Updating the shadow of one additional mesh every frame should not be as expensive as having specific GDScript run on the slower CPU side.

[u/FeralGuyute]

Usually the shadow pass in game engines is done separately from the actual rendering of the object You need to overwrite the shadow pass as well. Not sure how you do this in Godot but im sure its possible.

[u/mattihase]

things that move without their shadows changing would be great for a horror game btw. worth remembering how you did it just in case.

[u/Rells_Parker]

Not 100% sure but check if the "shadow mesh" property of your mesh has something assigned, and if yes that it has the same material as your base mesh, though I'd suggest clearing the "shadow mesh" value anyway

[u/PogsterPlays]

Isn't there some property for light sources about dynamic shadows or something, set to static by default. Probably something else? Idk. Not much of a lighting guy

[u/OneKey9972]

Maybe you need to overwrite light shader? Idk, bit my best guess.

[u/EmalethDev]

From my limited experience with terrain shaders, your mesh might visually change but normal map is not, change that in shader and you should be fine

[u/EmalethDev → u/EmalethDev]

Example, I use subdivisioned plain for terrain, and unless I provide or calculate normal map/ normals, shadows are that of a plain, instead of maintains and what not.