Question by jilleJr · Feb 25, 2016 at 01:38 PM · shader shader writing stencil ztest

Stencils and zTest be buggin

What is going on?!?!? I like shaders 'cause they're cool and all, but gosh they're annoying to debug.

Now. I have a simple stencil shader setup, where I have the player writing to the stencil when it's behind something, like so:

 (Image this being inside the shader, I cut out the boring parts)
 Properties
 {
     _MainTex ("Texture", 2D) = "white" {}
 }
 SubShader
 {
     // Just write to the stencil
     Pass {
         Cull Off
         ZWrite Off
         ZTest Greater
         ColorMask 0
         Stencil {
             Ref 1
             Comp always
             Pass replace
         }
     }
 
     // Draw normally ...
     2nd pass {

Then I have an image effect that writes with a color if stencil=1, real simple. Reason I am doing it this way is that via an image effect shader I can easily do the silhouette as postprocessing, without having to wonder about draw order and such. Also, this way I can have a transparent silhouette, and it won't stack so there will only be one "layer" of transparent color on top. Just the way I like it.

Good system, I'ma noob at shaders so I'm glad it works. But then stuff starts acting weird.

How I understand it, when the ZTest says the pixel is behind something, the stencil can do it's magic, but for some reason it doesn't always work(?). Inspect the following gif:

It may be unclear at first, but the top half is the scene view, and the bottom half is the game view. Sideeffect of doing the silhouettes the stencil way is that I can only see the effect in the game view. Butt. In the scene view, I'm dragging a plane in and out of the big tree in the first shot and the ground in the second, and the plane has a shader with ONLY the "write-to-stencil" pass, no second pass (Also, could someone pls tell me why the plane is visible in the first shot, it's transparent black, for some reason?? There's literally zero code that actually makes it draw, wtf). Now in the tree shot, the stencil part only works when the plane is like >50% inside, and in the ground shot, the stencil part is kinda clipped.

Now I should note that both the tree and the ground has an unlit texture shader, tho the tree has a slight modification where I added Cull Off (yes it's necessary, workaround is not what I'm looking for) and the ground is actually a terrain, believe those renderes differently, not sure tho.

If I change the tree from Cull Off shader to an unlit texture shader, i.e. same as before but w/o the Cull Off part (so it defaults to Cull Back), it works perfectly, without activating half into the tree for no apparent reason.

My question here is do anyone here know what is going on? And if it's fix'able, how? Is there maybe something I missed 'cause as I've said I'm pretty new to this shader thing so errors are bound to happen. Please tell me if you need more info/code and I'll supply it.

Here's a small showcase of one time I got it to work long enough for the gif, so that's what I am aiming for: silhouette