How to render perspective corrected surface direction?

Hi there I'm having to solve quite an interesting problem. For a post processing effect I need to render the perceived direction of the surfaces of objects. This can be driven by artist from a flow map so in some ways it's very similar to normal mapping except for one key difference. This needs to work with the camera's perspective transformation which is where I'm running into issues. To visualize it I’ve drawn here with red arrows what I’m trying to achieve:

So given this needs to work with perspective my current approach is to transfer tangent, bitangent and vertex position into world space in the vertex shader:

 float3 normalDir = normalize(mul(unity_ObjectToWorld, float4(v.normal, 0).xyz));
 o.tangentDir = normalize(mul(unity_ObjectToWorld, float4(v.tangent.xyz, 0)).xyz);
 o.bitangentDir = normalize(cross(normalDir, o.tangentDir) * v.tangent.w);
 o.worldPosition = mul(unity_ObjectToWorld, v.vertex);

Then in the vertex shader I take the world position and world position offset by tangent and bitangent values obtained from the flow map:

 float4 dirTexel = tex2D(_DirectionTex, i.uv);
 float2 dirRG = (dirTexel.rgb.rg * 2 - 0.5);
 float3 dir = i.tangentDir * dirRG.r + i.bitangentDir * dirRG.g;

bring them into the camera frustum:

 float3 p0 = mul(UNITY_MATRIX_VP, float4(i.worldPosition.xyz, 1));
 float3 p1 = mul(UNITY_MATRIX_VP, float4(i.worldPosition.xyz + dir, 1));

Calculate the difference and render out the angle into a color channel I then read at a later stage in the post processing:

 float2 dir2D = p1.xy - p0.xy;
 
 dir.x = cos(3.14159265359 * -0.25);
 dir.y = sin(3.14159265359 * -0.25);
 dir.xy = float2(-1, );
 float angleRadians = atan2(dir2D.y, dir2D.x);
 angleRadians += 6.28318530718;
 angleRadians /= 6.28318530718;
 col.b = fmod(angleRadians, 1);

This is giving me a result I can’t really make heads or tails off. Here I have rendered out the transformed tangent directions:

Interestingly if I only multiply with UNITY_MATRIX_V and set the camera to ortographic I get the result I want so you'd think it was a problem with perspective projection but UNITY_MATRIX_VP still breaks so something goes wrong from the conversion from view space to projection space. Unity Answers only allows for 2 images to be attached to a question so I put some more screenshots here. Any help wuld be greatly appreciated!