koobas.hobune.stream
Unity's Outline shader - sharp edges
Hi, I use standard toony outline shaders in my game, but there's problem with every object with sharp edges, like cube, cylinder, etc. See screenshot.
Is there way to fix it?
Yes, this happens because two vertices at the same position have different normals due to the smoothing settings on the model. If you have a model with no hard edges, such as the sphere, you won't have problems.
You need to calculate alternative normals that completely smooth the entire hard edges. Write a script that iterates through the mesh vertices, finds all vertices at one location (i.e. distance less than some amount), average the existing normals, and saves it into a vertex field (i.e. color) for all those vertices.
Your normal rendering will use the regular normals, and you will have to write a custom outline shader to use vertex color (or wherever you saved the smoothed normals) instead of the regular normal.
It would be better if there were a way to do it all in the vertex shader, but I can't think of a way to smooth the normals without having access to neighbouring vertices (which you won't have in the vertex shader). There may be a better way I missed, but the mesh pre-processing should work.
Ah, thank you, but I do not know much about editing objects using a script. Could you please write it for me? Now I use $$anonymous$$imum chamfered edges using 3ds $$anonymous$$ax and apply to the entire object only one Polygon Smoothing Group, but it's not ideal ...
That's a bit more work than I can just whip up on the fly, but an easier way would be to save two version of your mesh from 3DS $$anonymous$$ax. One with regular smoothing, and one with a single smoothing group so all normals are smoothed for you.
Create a simple script that has two public GameObject slots (regular$$anonymous$$esh and smoothed$$anonymous$$esh), and simply drag and drop your models in there. On Start() just create a new $$anonymous$$esh object (Unity documentation has examples) with all the vertex data from your regular$$anonymous$$esh object. However, use the smoothed$$anonymous$$esh normal array in place of the vertex color array from your regular$$anonymous$$esh. Then use the ObjExporter from the Unity Wiki to write it to file.
You will also have to update the shader, but this is really just replacing "vertex.normal" to "vertex.color" in the vertex shader of the standard Unity outline shader. Then your outline should be using the smoothed normals and you will avoid any broken edges, but your regular rendering with use the original normals so you can still have sharp edges on your model.
There may be a way to get 3DS $$anonymous$$ax to write the normal values to the color channels, but I don't know off-hand how to do that.
Alright, I have this code:
public GameObject regular$$anonymous$$esh;
public GameObject smoothed$$anonymous$$esh;
void Start () {
$$anonymous$$esh m1 = regular$$anonymous$$esh.GetComponent<$$anonymous$$eshFilter>().mesh;
m1.colors = smoothed$$anonymous$$esh.GetComponent<$$anonymous$$eshFilter>().mesh.normals;
//ObjExporter.$$anonymous$$eshToString (m1);
}
But I (logically) get this error: Cannot implicitly convert type 'UnityEngine.Vector3[]' to 'UnityEngine.Color[]'
What to do?
And in shader, I've replaced:
struct appdata {
float4 vertex : POSITION;
float3 normal : NOR$$anonymous$$AL;
};
With...
struct appdata {
float4 vertex : POSITION;
float3 color : COLOR;
};
And then:
v2f vert(appdata v) {
v2f o;
o.pos = mul(UNITY_$$anonymous$$ATRIX_$$anonymous$$VP, v.vertex);
float3 norm = mul ((float3x3)UNITY_$$anonymous$$ATRIX_IT_$$anonymous$$V, v.normal);
float2 offset = TransformViewToProjection(norm.xy);
o.pos.xy += offset * o.pos.z * _Outline;
o.color = _OutlineColor;
return o;
}
With...
v2f vert(appdata v) {
v2f o;
o.pos = mul(UNITY_$$anonymous$$ATRIX_$$anonymous$$VP, v.vertex);
float3 norm = mul ((float3x3)UNITY_$$anonymous$$ATRIX_IT_$$anonymous$$V, v.color);
float2 offset = TransformViewToProjection(norm.xy);
o.pos.xy += offset * o.pos.z * _Outline;
o.color = _OutlineColor;
return o;
}
Is this good?
Yeah, that looks good. You are getting an error because Vector3 can't be directly used as a Color.
You will need to set up a temporary array of Colors, and then loop through and do a manual conversion from Vector3 to Color. Should be as simple as:
for (int i = 0; i < m1.vertexCount; i++)
{
newColors[i] = new Color(m2.normals[i].x, m2.normals[i].y, m2.normals[i].z);
}
Then assign that array ins$$anonymous$$d. That should be about all you need apart from fiddling with the syntax a bit. You may need to call m2.Clear() and then assign the vertices, colors, uvs, and triangles all again (assu$$anonymous$$g "m2" is your second mesh).
Here's a link to a Unity package. I made a small scene that creates a new object with color normals.
I had forgotten about the scale & bias to convert from a color to a normal (color range is 0...1 while normals are -1...1) so i fixed that in the script and the shader (I renamed the shader so I could keep the original Unity toon shader working). Exporting the OBJ doesn't work because OBJ file format doesn't specify color information per vertex.
There is this link that I found where someone wrote an export/importer that will save an OBJ file with extra information. That is probably the best way to go otherwise you will need to load two meshes for each object to create your object at run-time.
I solved this problem by writing a second Shader for object that included hard surfaces/sharp edges.
For my assets, I baked the object space normal data into the vertex colors, and then I had a surface shader apply those vertex colors as the normal data. AS IT TURNS OUT: Blender and Unity don't exactly speak the same language, and I had to rotate the normal data around so it was facing the correct direction. After kicking myself for hours trying to figure out how to do that in the surface shader (which works in tangent space normals), I discovered I could supply an extra vertex shader to do my transformations in object space.
void vert(inout appdata_full v) {
v.normal.x = 1 - v.color.r;
v.normal.y = 1 - v.color.b;
v.normal.z = v.color.g;
v.normal = v.normal * 2 - 1;
}
void surf(Input IN, inout SurfaceOutput o) {
o.Albedo = tex2D(_MainTex, IN.uv_MainTex).rgb;
o.Albedo *= _Color.rgb
}
Then my second Pass was the vertex and fragment shaders that created the outline.
v2f vert(appdata v)
{
v2f o;
v.pos.xyz += v.normal * _OutlineThickness;
o.vertex = UnityObjectToClipPos(v.pos);
return o;
}
fixed4 frag(v2f i) : SV_Target
{
fixed4 col = _OutlineColor.rgba;
return col;
}
If all of your objects have smooth normals throughout, then this method is unnecessary, but this is the best solution I have come up with so far.
On second inspection, I didn't actually solve it. It looks like I need to transform the colors by some matrix or combination of matricies. I have as of yet had no luck.
Well, that was a long and ridiculous ordeal, but my problem had to do with converting from Blender to Unity. Different axes and conventions. Here's the transformation I had to do:
float4 colorSwap = IN.color;
colorSwap.r = 1-IN.color.r;
colorSwap.g = 1-IN.color.b;
colorSwap.b = IN.color.g;
o.Normal = colorSwap *2 -1;
