koobas.hobune.stream
Help me understand stencil shader logic
I am trying to write a stencil shader in a way that makes sense for me. The one I am working on right now displays a different color depending on how many objects are overlapping. The photo below shows the behavior I want (I wrote it with code that didn't make sense to me via trial and error).
What I think should be written from what I understand of stencil buffers is for each pass:
ref [int]
Comp Equal
Pass IncrSat
with int starting at 0 for the first subshader and then incrementing up for each subshader. I thought that way the next subshader would only render if the pixel was on another object at the same location and incremented the buffer value up
However, when I write that then each pixel renders the last subshader, even if they are not overlapping. Clearly I am not understanding stencils. What am I misunderstanding? Are there any good comprehensive and clearly written texts out there that explain everything about stencil shaders? I have looked at many online tutorials and I still do not understand the logic.
// Upgrade NOTE: replaced 'mul(UNITY_MATRIX_MVP,*)' with 'UnityObjectToClipPos(*)'
Shader "Unlit/Stencil/Simple"
{
Properties
{
_MainTex ("Texture", 2D) = "white" {}
_Color ("Main Color", Color) = (1,1,1,1)
_SecondaryColor ("Secondary Color", Color) = (0,0,0,1)
_TertiaryColor ("Tertiary Color", Color) = (0,0,0,1)
_QuaternaryColor ("Quaternary Color", Color) = (0,0,0,1)
_MasterAlpha ("Master Alpha", Range(0,1)) = 1
}
SubShader
{
Tags {"Queue"="Transparent" "RenderType"="Transparent" }
LOD 100
Blend SrcAlpha OneMinusSrcAlpha
Pass
{
Stencil {
Ref 0
Comp Equal
Pass IncrSat
Fail IncrSat
}
CGPROGRAM
#pragma vertex vert
#pragma fragment frag
#include "UnityCG.cginc"
fixed4 _Color;
fixed _MasterAlpha;
//float4 _MainTex_ST;
struct appdata
{
float4 vertex : POSITION;
//float2 uv : TEXCOORD0;
};
struct v2f
{
float4 vertex : SV_POSITION;
//float2 uv : TEXCOORD0;
};
v2f vert (appdata v)
{
v2f o;
o.vertex = UnityObjectToClipPos(v.vertex);
//o.uv = TRANSFORM_TEX(v.uv, _MainTex);
return o;
}
fixed4 frag (v2f i) : SV_Target
{
_Color.a *= _MasterAlpha;
return _Color;
}
ENDCG
}
Pass
{
Stencil {
Ref 1
Comp Less
//Pass DecrSat
}
CGPROGRAM
#pragma vertex vert
#pragma fragment frag
#include "UnityCG.cginc"
#define PI 3.1415926538
fixed4 _SecondaryColor;
fixed _MasterAlpha;
struct appdata
{
float4 vertex : POSITION;
};
struct v2f
{
float4 vertex : SV_POSITION;
};
v2f vert (appdata v)
{
v2f o;
o.vertex = UnityObjectToClipPos(v.vertex);
return o;
}
fixed4 frag (v2f i) : SV_Target
{
_SecondaryColor.a *= _MasterAlpha;
return _SecondaryColor;
}
ENDCG
}
Pass
{
Stencil {
Ref 2
Comp Less //
}
CGPROGRAM
#pragma vertex vert
#pragma fragment frag
#include "UnityCG.cginc"
#define PI 3.1415926538
fixed4 _TertiaryColor;
fixed _MasterAlpha;
struct appdata
{
float4 vertex : POSITION;
};
struct v2f
{
float4 vertex : SV_POSITION;
};
v2f vert (appdata v)
{
v2f o;
o.vertex = UnityObjectToClipPos(v.vertex);
return o;
}
fixed4 frag (v2f i) : SV_Target
{
_TertiaryColor.a *= _MasterAlpha * _MasterAlpha;
return _TertiaryColor;
}
ENDCG
}
Pass
{
Stencil {
Ref 3
Comp Less //
}
CGPROGRAM
#pragma vertex vert
#pragma fragment frag
#include "UnityCG.cginc"
#define PI 3.1415926538
fixed4 _TertiaryColor;
fixed4 _QuaternaryColor;
fixed _MasterAlpha;
struct appdata
{
float4 vertex : POSITION;
};
struct v2f
{
float4 vertex : SV_POSITION;
};
v2f vert (appdata v)
{
v2f o;
o.vertex = UnityObjectToClipPos(v.vertex);
return o;
}
fixed4 frag (v2f i) : SV_Target
{
_QuaternaryColor.a *= _MasterAlpha * _MasterAlpha * _MasterAlpha;
return _QuaternaryColor;
//return lerp(_TertiaryColor, _QuaternaryColor, _MasterAlpha);
}
ENDCG
}
}
}
[1]: /storage/temp/157501-wowow.gif
Well, there are two major parts to stencil operations. First is the comparison method. This is just like the depth test or alpha test. The test specifies when to actually perform the stencil operation and also control if the pixel is actually rendered. So if the test fails the stencil operation is not carried out and the pixel is not rendered at all.
The stencil operator itself just modifies the value in the stencil buffer but only when the depth buffer test passes and stencil comparison passes (PASS), when the stencil comparison fails (FAIL) or when the depth buffer test fails (ZFAIL).
I've written some pseudo code how the stencil test, stencil operations and the actual rendering are connected:
if (depthTestPass)
{
if (stencilTest()){
PASS_Operation();
} else {
FAIL_Operation();
return;
}
RenderPixel();
}
else if (stencilTest())
{
if (stencilTest()) {
ZFAIL_Operation();
}
}
I just write this on top of my head ^^. There might be a mistake but that's essentially how it works. So when a stencil test is active it's an additional test for the actual rendering (writing to the framebuffer) as well as a condition for the 3 different stencil conditions.
Note that subshaders are essentially just seperate shaders which are ALL executed one after another. When a certain subshader is executed of course can be controlled by the rendering queue. But if two subshaders belong to the same rendering queue they are executed in order. Note that if your shader has multiple subshaders only one of them is actually used. If a subshader has multiple passes it means the object is rendered that many times, once for every pass.
We don't have enough information to actually understand how you to use the stencil buffer. It's not clear how many different shaders you use on how many objects. Also note if the objects you're rendering are sprites, transparent pixels will also affect the stencil buffer. Only the mesh is what matters. The shading or blending operations which might take place in the fragment shader are irrelevant for stencil operations. I'm not sure if cutout shaders would work correctly (so if the shader uses "discard" in the fragment shader)
So without seeing your actual shaders and knowing what kind of objects are involved it's hard to tell anything about your issue or what your wanted result is.
Just for reference Unity's stencil documentation
There is only one shader used on the objects. I edited the post to include the code and an easier image to work off of. The image shows the behavior I want, and I have set darker colors to areas with more overlaps. This code works but the logic doesn't make sense to me; it doesn't seem to me that it should be working. I don't know why the _SecondaryColor would render if the stencil buffer value is LESS than 1, I thought it should render if it is 1 (and so forth for each pass). That's why I thought it should be ref [int] Comp Equal Pass IncrSat but if I do that everything is black (_QuaternaryColor).
But if two subshaders belong to the same rendering queue they are executed in order. Note that if your shader has multiple subshaders only one of them is actually used. If a subshader has multiple passes it means the object is rendered that many times, once for every pass.
Do you mean the subshaders are executed in the order you write them? Are passes executed in order you write them? Should I be using subshaders or passes for the different colors? $$anonymous$$aybe it is these two things that I don't understand and is messing me up.
Of course all your passes will be rendered in order. They are named pass 0 up to pass 3. All four passes will be executed for every object you're rendering with this shader. So you quadrupled the times a single object is actually rendered. That's the point of having multiple passes if you have effects that requires multiple passes ^^. Subshaders are a totally different case. They just allow you to essentially put different implementations within the same shader file. Which subshader is actually used is deter$$anonymous$$ed either by the hardware support on the executing device or if you did specify certain tags.
When you use the stencil buffer it's actually quite common to have a preprocessing pass that actually does perform the stencil operations and following only passes which do not change the stencil value but just test against it. Like it is explained over here.
Keep in $$anonymous$$d that each pass means the object is rendered again. The stencil test might filter out all fragments so nothing would actually be written to the framebuffer, but the overhead is still there, The object is processed once for each pass.
Ok, I think I understand more about why my code doesn't work and that example does. One thing I still can't figure out is why it needs the Fail IncrSat line. Since it is the first pass, shouldn't it always pass a Comp Equal to 0? Or are all the first passes for all the objects first rendered and then the second passes for all the objects are rendered, and so on (so the stencil buffer value might be higher because the first pass of a different object increased it)?
Do you know of any good learning resources for shaders? $$anonymous$$aybe a textbook? I looked at all of the things listed here as well as online tutorials but they are either not thorough enough or I have a hard time understanding them.
