- Home /
How can I transform a world view dir into tangent space in a Cg only shader
Hello, I've been trying to figure this one out for a while now and I can't seem to make it work. The only place it fails is when I rotate the object in any way.
My plan was to write a custom Cg shader that would transform the viewDir into tangent space so I could do my calculation there. This method is supposed to be much faster than transforming the tangent space normal into world space, and less complicated, so I thought :).
Here was my line of thinking. I create a TBN matrix with TANGENT_SPACE_ROTATION and multiply it with unity_ObjectToWorld so that the TBN will be in world space. Then I multiply the world space view dir UNITY_MATRIX_V[2].xyz by the transpose of that matrix to bring the view direction into tangent space.
Update: Updated to working code.
Here's the code:
Shader "CgRimShader"
{
Properties
{
_Color("Color", COLOR) = (0,0,0,1)
_RimColor("Rim Color", COLOR) = (1,1,1,1)
_RimPower("Rim Power", Float) = 2
_BumpMap ("Normalmap", 2D) = "bump" {}
}
SubShader
{
Tags {"RenderType"="Opaque" }
LOD 100
Pass
{
CGPROGRAM
#pragma vertex vert
#pragma fragment frag
#include "UnityCG.cginc"
struct appdata
{
float4 vertex : POSITION;
float2 uv : TEXCOORD0;
float3 normal : NORMAL;
float4 tangent : TANGENT;
};
struct v2f
{
float4 vertex : SV_POSITION;
float2 uv : TEXCOORD0;
float3 viewDir : TEXCOORD1;
};
sampler2D _BumpMap;
float4 _BumpMap_ST;
float4 _Color;
float4 _RimColor;
float _RimPower;
v2f vert (appdata v)
{
v2f o;
o.vertex = UnityObjectToClipPos(v.vertex);
o.uv = TRANSFORM_TEX(v.uv, _BumpMap);
o.viewDir = COMPUTE_VIEW_NORMAL;
return o;
}
fixed4 frag (v2f i) : SV_Target
{
float3 Normal = UnpackNormal(tex2D(_BumpMap, i.uv));
float3 ViewDir = normalize(i.viewDir);
float NdotV = max(0, dot(Normal, ViewDir));
fixed4 col = _Color + pow(1-NdotV, _RimPower) * _RimColor;
return col;
}
ENDCG
}
}
}
Well the good news is that whatever I'm doing wrong, I'm doing it consistently. I transformed the object space normal into world space to see what that would look like and it's the same issue. I also brought the view space normal into object space and same thing. I can rotate the camera any way I want and the object looks right, but as soon as I rotate the object, it breaks.
The articles I've read, and what I already understand about matrices says this should be working, so this is driving me nuts. I'm just going in circles at this point.
One other thing about the version above. UNITY_$$anonymous$$ATRIX_V[2] is not the view direction in world space. It's supposed to be the transpose of the camera matrix. So I did a transpose on it to get the world view direction and that didn't work either. Then I found a post by someone saying that the world space view direction is in UNITY_$$anonymous$$ATRIX_T_$$anonymous$$V[2], but that didn't work either.
Then I was like fine, I'll see if I can translate a forward vector into tangent space using a trial and error method. Then finally it worked! I sort of get why it works for both the camera and model rotation, but not completely.
Answer by weatxyz · Nov 06, 2016 at 06:55 PM
OK, so I took a different approach with this and finally got it to work. Although I hate the fact that I stumbled onto this solution and still don't know why it works. Just replace the vert function above with this one and it works great.
Update: I kept optimizing this and optimizing this and discovered that the solution was staring me right in the face. there's one simple macro that does all that is needed called COMPUTE_VIEW_NORMAL! I updated the code below to reflect this change.
v2f vert (appdata v)
{
v2f o;
o.vertex = UnityObjectToClipPos(v.vertex);
o.uv = TRANSFORM_TEX(v.uv, _MainTex);
o.viewDir = COMPUTE_VIEW_NORMAL;
return o;
}
The definition of that macro is this: mul((float3x3)UNITY_MATRIX_IT_MV, v.normal)
This version also works and is the version I optimized my shader to before I found the macro: mul(v.normal, (float3x3)UNITY_MATRIX_T_MV)
Answer by ifurkend · Aug 22, 2020 at 03:05 PM
Kudos to Harry Alisavakis for providing the complete code to obtain the viewDir in tangent space:
https://halisavakis.com/my-take-on-shaders-parallax-effect-part-ii/
Your answer
Follow this Question
Related Questions
How to force the compilation of a shader in Unity? 5 Answers
How to determine shader center? 0 Answers
How to access another sprite's texture from a 2D shader 0 Answers
Meta pass and Precomputed Realtime GI 0 Answers
Projected Texture Shader Issues 1 Answer