Character controller for fixed perspective camera (moving relative to arbitrary point on screen)

How do I make a character always move relative to the camera, even near the edges of the screen?

Here is the effect I am trying to avoid:

https://giant.gfycat.com/TameGleefulAsianlion.webm

===========

The Setup

===========

I have a fixed perspective camera in the world, and a player controlled character that needs to move freely around the space.

Figure 1 - Character and camera set up (the camera won't always be in this position, it can move)

Moving vertically should always move them in the direction of the screen vertical, and similarly for moving horizontaly.

Sounds easy enough, right?

===============

The Problems

===============

Distortion near the edges

Unlike orthogonal cameras, perspective cameras distort near the edges of the screen.

The typical solution for a 3rd person camera that follows the player would be to:

1. Project the forward vector of the camera onto a plane with the up vector of the player

2. Calculate the horizontal axis via a cross product

3. Multiply the normalized input axes by those flattened axes.

That does not work in this case.

With the setup above, if positioned near the left side of the screen the character moves slightly in the z-direction when moving horizontally. This axis evens out towards the center:

Figure 2 - Crappy diagram of how the horizontal axis warps near the screen edges

Projected axis misalignment

Even if this effect is accounted for, the final movement axes not be perfectly 90 degrees from one another near the edges. Horizontal movement is mostly stable but vertical movement swings significantly to compensate for the projection.

Figure 3 - A top down view of how the forward and right axes can end up near the edge of the screen

This can mean joystick input just doesn't feel right when moving diagonally. In the diagram above, moving towards the top right gives you finer control than moving towards the bottom right.

======================

Attempted solution

======================

• Find the screenspace position of the character

• Add 1 in the x and y directions to this position in screen space

• Reproject those points to a plane to find the forward and right vectors

  ---

Figure 4 - Calculating the forward and right vectors in screen space

       // Find the character's position on the screen
     Vector3 screenPos = Camera.main.WorldToScreenPoint(transform.position);
     // Remove z-axis
     screenPos.Scale(new Vector3(1, 1, 0));
 
     // Find a point on the screen 1 unit above the character
     Vector3 screenUp = ScreenPos + Vector3.up;
     // Find a point on the screen 1 unit to the right of the character
     Vector3 screenRight = ScreenPos - Vector3.left;
 
     // Cast a rays into the scene at these points
     Ray rayUp = Camera.main.ScreenPointToRay(screenUp);
     Ray rayRight = Camera.main.ScreenPointToRay(screenRight);
 
     // Calculate vectors from the ray origins to the transform origins, then normalize
     float upDelta = rayUp.origin.y - transform.position.y;
     Vector3 upDirectionNormalised = rayUp.direction / rayUp.direction.y;
     Vector3 upIntersectionPos = rayUp.origin - upDirectionNormalised * upDelta;
 
     float rightDelta = rayRight.origin.y - transform.position.y;
     Vector3 rightDirectionNormal = rayRight.direction / rayRight.direction.y;
     Vector3 rightIntersectionPos = rayRight.origin - rightDirectionNormal * rightDelta;
 
     // Find the forward and right vectors from these intersection points
     Vector3 newForwardAxis = upIntersectionPos - transform.position;
     Vector3 newRightAxis = rightIntersectionPos - transform.position;
 
     // Flatten the y axes
     newForwardAxis.Scale(new Vector3(1, 0, 1));
     newRightAxis.Scale(new Vector3(1, 0, 1));

This works for the most part, but has a few big problems:

• HUGE jitter on these axes. For some reason they won't sit still. I have tried lerping the axis over time, which smooths this out - but also means if the camera moves faster than the lerp speed, the axes can't keep up!

• When the camera sits very low (facing the player), the center and forward points converge on the same world point.

  ---

Anybody faced this problem before? Any pointers would be a HUGE help! The camera must be a perspective camera, unfortunately.

Cheers