koobas.hobune.stream
Shadows have a mind of their own
I'm trying to make a shader that receives shadows and selectively outputs transparency based on the texture.
Even though my shader works, when I output transparent fragments Unity is still applying a translucent shadow in 3d space, and is using the geometry to occlude any background shadows based on the current viewing angle. This can be seen in the attached screenshot. Note the following:
Even though the female figure is transparent (the floor and cube can be seen behind it), there is a shadow being applied to the model.
The male character's shadow and the cube's shadow are occluded by her hip and hand, respectively.
Here's the subshader header. The contents of the shader don't matter because even if I change the shader to simply output fixed4(0,0,0,0) I still get this behavior. If I change the queue to Transparent then the behavior stops but causes SHADOW_ATTENUATION to always return 1, making my shader useless.
SubShader {
Tags {"Queue" = "Geometry" "RenderType" = "Opaque"}
Pass {
ZWrite Off
Blend SrcAlpha OneMinusSrcAlpha
Now, I can live with the shadow occlusion (even though I would like to understand why it happens) but the extra automatic pass to apply shadows to my model is something I would like to remove, as it feels utterly pointless. Can I indicate somehow to Unity that I am handling shadows myself and can you please leave my poor mesh alone?
Thanks in advance for any advice.
Another important observation is that the ghost shadow is only rendered against $$anonymous$$eshRenderer's marked as "receive shadow". In this screenshot the sand-colored plane is marked as 'receive shadows' and the water-colored plane is not. 
So to clarify my last comment, a 2d surface marked with "receive shadows" causes a 3d shadow overlay to be rendered on top of it, regardless of the output of the shader.
This answer is partial, but I hope it puts you on the right path.
The basic idea of how realtime shadows work is that each shadow casting light renders a depth buffer from the "view" of the light. Because it's a depth buffer, only opaque objects can go into it (i.e. only opaque objects can cast shadows). During normal rendering, the shader takes the current pixel position, and computes the depth of that position from the point of view of the light, and then checks the depth of that position (in light space) with the corresponding shadow depth buffer pixel. If the current pixel depth is greater than the shadow depth buffer pixel, then the pixel is shadowed. If it's less, then it's not shadowed.
While only opaque objects can cast shadows, it should be possible for transparent objects to read shadows. That means that putting your object into the transparent pass (which is where it should go because it is outputting transparency) should be able to get shadow attenuation values other than one. The fact that you always get 1 is the central problem, and what you should try to figure out. If you are using deferred rendering, you should try forward rendering. That might force Unity to do the right thing for transparent shadows. If that doesn't work, do some research on transparent shadow receivers in Unity. You should be able to find a solution (I hope).
Thank you for the detailed explanation! So if I understand correctly, the shadow depth buffer is translated to camera space, and when a 'receive shadow' surface is rendered, any shadow pixel with a lower depth value is rendered as a shadow on the target surface. This would explain both the ghost shadow (applied to a nearer object) and the occlusion of background shadows (no shadow visible from the current angle).
I guess then the question is whether it is possible to compute and sample realtime shadows in light space, not in camera space. This would allow me to avoid occlusion based on camera space translation. Obviously at a performance cost but since my art is $$anonymous$$imalistic and tightly controlled this won't be a problem.
(Using the Transparent queue means the models won't cast shadows. The art style I am going for requires self shadowing, so I do not want to bypass the shadow caster. SHADOW_ATTENUATION returns 1 for the self shadow, which is non-existent - I hadn't realized that at the time of my original post.)
You can do the projection from light to camera or vice versa. I forget which approach is the most common. In any event, because you're doing such a specialized process and want self shadowing, I would recommend you render a custom shadow map per character that only handles self shadowing. Then you lookup into this shadow map as well as the global shadow map to get the effect you're looking for.
That's a good point and I will consider it. An orthographic camera rendering the characters with a depth map could simulate a directional light. I'm shaky on the projection math though...
Some progress being made here: http://forum.unity3d.com/threads/custom-depth-buffer-rendering.323279/
