Cutout toon shader

Question

Hi, I made a cutout toon/outline shader. It works fine, but there are some artifacts that cant get rid of. Am I doing something wrong or is this a natural side effect of this kind of tooning?

Link to fullsized screenshot

alt text

 SubShader {
     Tags {  "ForceNoShadowCasting"="True" "Queue"="AlphaTest" "RenderType"="TransparentCutout" }
     Pass {
         Cull Front
         ZWrite On
         Tags { "LightMode" = "Always" }
         CGPROGRAM
         #pragma exclude_renderers gles flash d3d11 xbox360
         #pragma vertex vert
         #pragma fragment frag
         #pragma fragmentoption ARB_precision_hint_fastest
 
         
         struct tv2f {
             float4 pos : POSITION;
             float2 uv : TEXCOORD0;
             float4 color : COLOR;
             float4 vpos : TEXCOORD1;
             float2 texcoord : TEXCOORD0;
         };
 
         uniform float _Outline;
         uniform float4 _OutlineColor;
                 uniform sampler2D _MainTex;
             uniform float _Cutout;
 
         tv2f vert(a2v v) {
             tv2f o;
             o.pos = mul(UNITY_MATRIX_MVP, v.vertex);
             o.vpos = v.vertex;
             float3 norm = mul ((float3x3)UNITY_MATRIX_MV, v.normal);
             float2 offset = TransformViewToProjection(norm.xy);
 
             o.texcoord = TRANSFORM_TEX(v.texcoord, _MainTex);
             o.pos.xy += offset * o.pos.z * _Outline;
 
             o.color = _OutlineColor;
             return o;
         }
         
         float4 frag(tv2f i) : COLOR {
             
                 clip(tex2D(_MainTex, i.texcoord).a - _Cutout);
             return i.color;
         }
         ENDCG
     }
     
     Pass {
         Cull Back
         ZWrite On
         Tags { "LightMode" = "Always" }
         
         CGPROGRAM
         #pragma exclude_renderers gles flash d3d11 xbox360
         #pragma vertex vert
         #pragma fragment frag
         #pragma fragmentoption ARB_precision_hint_fastest
 
         uniform float _Outline;
         uniform float4 _OutlineColor;
                 uniform float4 _Color;
                 uniform sampler2D _MainTex;
         uniform samplerCUBE _ToonShade;
             uniform float _Cutout;
         
         struct tv2f {
             float4 pos : POSITION;
             float2 texcoord : TEXCOORD0;
             float3 cubenormal : TEXCOORD1;
             float4 vpos : TEXCOORD2; 
         };
         
         tv2f vert(a2v v) {
             tv2f o;
             o.pos = mul(UNITY_MATRIX_MVP, v.vertex);
             o.vpos = v.vertex;
             o.texcoord = TRANSFORM_TEX(v.texcoord, _MainTex);
             o.cubenormal = mul (UNITY_MATRIX_MV, float4(v.normal,0));
             return o;
         }
         
         float4 frag(tv2f i) : COLOR {
             
                 clip(tex2D(_MainTex, i.texcoord).a - _Cutout);
             float4 col = _Color * tex2D(_MainTex, i.texcoord);
             float4 cube = texCUBE(_ToonShade, i.cubenormal);
             return float4(2.0f * cube.rgb * col.rgb, col.a);
         }
         ENDCG
     }
 }

Answer 1

This problem is common: It occurs in all outline shaders that extrude along the normal, as Kajos said in a comment. It happens on parts of the model where vertices are doubled to produce crease (hard) edges.

I made a small example to illustrate precisely what's going on. Compare this sphere to the cube:

alt text

Both are shaded with the standard outline toon shader you can get from Assets->Import Package. You'll notice that the sphere is fine, but the cube clearly suffers the same artifact your buildings also experience. That's because the sphere is entirely smooth - it has no hard edges, so no two vertices share a position in order to have different normals in the same location. The box doesn't have this luxury. It has 3 vertices in each corner, because each corner connects to three faces, and each face must have its own normal, or else, the box won't have hard edges. But when outline shaded in this simple way, the vertex extrusion along the normal vector splits the model apart - each face that the model is actually composed of gets pushes away from the others, and then those gaps appear.

If you wish to fix it, you have to either make sure your model doesn't have any hard edges (impractical) or you have to come up with a different way to create outlines than the one the standard Toon shader uses.

One way is to do screen space silhouette detection as an image effect. Another is to keep extruding vertices, but choose some other direction than the normal vector, e.g. vector to the object's center, for example. Doing so would fix the problem for the box while also working for the sphere, but requires the mesh to be a convex hull, which complex geometry almost never is. Your buildings most certainly aren't.

Cutout toon shader

1 Reply

Your answer

Follow this Question

Related Questions