How do I implement ScreenPointToRay in a compute shader?

I've been scratching my head for a while trying to figure this one out. The nearest I can figure is that you take the XY of the screen position, feed it to the projection matrix and the worldToCameraMatrix and then magically you get the right direction of the ray, then you can derive the ray's origin from the camera position. Seems simple right? I cannot get that to work.

Nearest I have is something along the lines of:

 Matrix4x4 V = Camera.main.worldToCameraMatrix;
 Matrix4x4 P = Camera.main.projectionMatrix;
 Matrix4x4 InvVP = (P*V).inverse;

which then gets fed into the compute shader to a function like this:

 float3 ComputeWorldPosition(float3 screenPos)
  {
      // Transform pixel space to clipping space
      screenPos.x = 2.0f*screenPos.x - 1.0f;
      screenPos.y =  1.0f - 2.0f * screenPos.y;
      
      // Do inverse VP multiplication
      float4 worldPos = mul(float4(screenPos, 1), InvVP;
      
      // Calculate final world position
      return worldPos.xyz * worldPos.w;
  }

Screenpos x and y are always between 0 and 1 and z is always 0, this results in nonsensical output, I know I'm doing something wrong but I cannot figure out what.