Code behind 'Camera.ViewportPointToRay'?

Question

I have a situation where I'm dealing with just the matrices behind camera transformations, but not actual cameras. I already have working equivalents for the ViewportToWorld and WorldToViewport methods, as well as a camera projection matrix, a worldToLocal matrix, and a localToWorld matrix.

I'd like to have the functionality of the method 'ViewportPointToRay' in this class also, does anyone know how it might work behind the scenes?

Alternatively, if anyone knows how to find the world coordinates for where the corners of the viewport intersect with a plane at y = 0 using only matrices and not the 'ViewportPointToRay' method, that would also work!

For reference here is a gizmo I'm drawing just to show the frustum where y = 0. I basically want to not have to reference a camera in this method.

 public void OnDrawGizmos()
 {
     Plane plane = new Plane(Vector3.up, Vector3.zero);
 
     Vector3 bottomLeftViewport = new Vector3(0f, 0f, 0f);
     Vector3 topLeftViewport = new Vector3(0f, 1f, 0f);
     Vector3 topRightViewport = new Vector3(1f, 1f, 0f);
     Vector3 bottomRightViewport = new Vector3(1f, 0f, 0f);
 
     Ray ray;
     float dist;
 
     Vector3[] maxZoomFrustum = new Vector3[4];
 
     ray = m_camera.ViewportPointToRay(bottomLeftViewport);
     plane.Raycast(ray, out dist);
     maxZoomFrustum[0] = ray.GetPoint(dist);
 
     ray = m_camera.ViewportPointToRay(topLeftViewport);
     plane.Raycast(ray, out dist);
     maxZoomFrustum[1] = ray.GetPoint(dist);
 
     ray = m_camera.ViewportPointToRay(topRightViewport);
     plane.Raycast(ray, out dist);
     maxZoomFrustum[2] = ray.GetPoint(dist);
 
     ray = m_camera.ViewportPointToRay(bottomRightViewport);
     plane.Raycast(ray, out dist);
     maxZoomFrustum[3] = ray.GetPoint(dist);
 
     Handles.color = Color.green.SetAlpha(0.3f);
     Handles.DrawAAConvexPolygon(maxZoomFrustum);
 }

Answer 1

Best Answer

Answer by Bunny83 · Jan 31, 2020 at 01:44 PM

I've just tested this one and it works quite well. The numerical error compared to Unity's own ViewportPointToRay method is at the 5th decimal place. Drawing the rays 100 world units into the scene you can barely see the error. Unity might use some different internal order. However the results are pretty spot on.

 public static Ray ViewportPointToRay(Vector2 aP, Matrix4x4 aProj, Matrix4x4 aCam)
 {
     var m = aProj * aCam;
     var mInv = m.inverse;
     // near clipping plane point
     Vector4 p = new Vector4(aP.x*2-1, aP.y*2-1, -1, 1f);
     var p0 = mInv * p;
     p0 /= p0.w;
     // far clipping plane point
     p.z = 1;
     var p1 = mInv * p;
     p1 /= p1.w;
     return new Ray(p0, (p1-p0).normalized);
 }

Note that aProj need to be the camera's projection matrix and aCam need to be the camera's worldToCameraMatrix. Also note that worldToCameraMatrix is just the camera's worldToLocal matrix but with the z axis inverted like this:

 var w2c = Matrix4x4.Scale(new Vector3(1, 1, -1)) * cam.transform.worldToLocalMatrix;

I've tested this method with a perspective and an orthographic camera and it works like expected.

Add comment · Show 1 · Share

Bunny83 · Jan 31, 2020 at 01:52 PM 1

Share

Please note that this method is only a proof of concept and if you want to use it as it is it shouldn't be used more than a couple of times pre frame. If you need to do this conversion several times you really should cache the "mInv" matrix since combining and inverting a matrix isn't that cheap. Though that's one of the main point why we use matrices in the first place. You set them up once and use them on a large amount of data / points.

edit

Here's a more optimised method which calculates a single "unproject matrix":

 private static Matrix4x4 m_View2NDC = Matrix4x4.Translate(-Vector3.one) * Matrix4x4.Scale(Vector3.one * 2);
 public static Matrix4x4 CalculateUnprojectMatrix(Matrix4x4 aProj, Matrix4x4 aWorld2Cam)
 {
     var m = aProj * aWorld2Cam;
     return m.inverse * m_View2NDC;
 }
 public static Ray ViewportPointToRay(Vector3 aP, Matrix4x4 aUnprojectMatrix)
 {
     aP.z = 0;
     var p0 = aUnprojectMatrix.MultiplyPoint(aP);
     aP.z = 1;
     var p1 = aUnprojectMatrix.MultiplyPoint(aP);
     return new Ray(p0, (p1-p0).normalized);
 }

I just cached the view to NDC conversion since it's the same all the time. So when constructing the unproject matrix we only need to perform 2 matrix multiplication and one inverse. Once we have the unproject matrix for the current camera view we can use it as often as we want to translate a viewport point (coordinate range 0 - 1 for all axis) to worldspace point.

All the ViewportPointToRay method does now is calculating the world point for the given viewport point on the near and far clipping plane and construct the ray for those points.

Answer 2

Answer by Namey5 · Jan 31, 2020 at 06:18 AM

ViewportToRay would work very similar to ViewportToWorld, but with the added step of finding the direction of said point from the camera;

 Ray ViewportPointToRay (Vector3 pos, Camera cam)
 {
     //Remap to NDC-space [-1,1]
     pos = pos * 2.0f - Vector3.one;
     pos.z = 1f;

     //Find the world-space position of the point at the camera's far plane
     Vector3 worldPos = cam.cameraToWorldMatrix.MultiplyPoint (cam.projectionMatrix.inverse.MultiplyPoint (pos));

     //The ray's origin is just the camera's position. Alternatively, you could use the same 
     //matrix logic above and find the same point at the camera's near plane and use that instead
     Vector3 origin = cam.transform.position;
 
     return new Ray (origin, worldPos - origin);
 }

The Unity documentation for the internal function makes a very interesting point; the z-position is ignored. In this case, because distance is irrelevant in relation to a ray, we just assume the viewport point lies on the camera's far plane.

Add comment · Show 4 · Share

EmmetOT · Jan 31, 2020 at 09:52 AM 0

Share

Great! This is exactly what I'm looking for. I just need to figure out how to get the 'cameraToWorld$$anonymous$$atrix' and I'm golden.

Bunny83 EmmetOT · Jan 31, 2020 at 12:10 PM 0

Share

CameraToWorld is essentially just the localToWorld matrix of the camera transform. However depending on the platform and API (OpenGL, DirectX, ...) certain things might be inverted. Likewise Unity has the method GL.GetGPUProjection$$anonymous$$atrix to get the right projection matrix that is used / required on the actual GPU / shader. There also might be tiny differences (z inverted so right handed / left handed; or maybe y inverted; sometimes the depth buffer value might be inverted; ...).

Note that there a few issues with this approach. . First of all this probably won't work with perspective projection since the projection is achieved through the w component of the homogeneous coordinates and the additional normalization(homogeneous divide). So just using $$anonymous$$ultiplyPoint with a Vector3 position won't work. It probably should work for orthographic cameras.

Second issue is that Unity's ViewportToRay method places the ray origin at the near clipping plane and not just the camera origin. It's just a tiny difference but might be important. For example using just using cam.transform.position as origin won't work for orthographic cameras.

Properly "unprojecting" is quite tricky, especially just using the matrices like you can see in this question. Usually the projection matrix is defined for a right handed system with z being inverted. In that case the projection matrix looks like the one mentioned in this question. Note that the projected NDC z value usually goes from -1 to 1. That's actually true for all 3 components (x,y,z).

Bunny83 Bunny83 · Jan 31, 2020 at 02:02 PM 0

Share

Small correction to my comment. I just had another look and $$anonymous$$ultiplyPoint actually performs a homogeneous divide and assumes an inco$$anonymous$$g w value of 1. So it does the projection correctly. Though it sill would not work with an orthographic camera.

Show more comments

Code behind 'Camera.ViewportPointToRay'?

2 Replies

Your answer

Follow this Question

Related Questions