Raycast - Clamp Math Issue

Hello,

I have a ball, in which the camera can rotate around. I have a cube that moves on the x axis of the ball depending where the centre of the screen is. Using a raycast from the camera, to an invisible plane where the ball is, I am able to move the cube along the X axis of the raycast, using a Clamp so the cube never leaves the surface of the ball.

The cube is used to show the user where force is going to be applied to the ball (like an angle shot in a game of pool/billiards). So its a visual aid.

The issue I'm having is that if I rotate my camera around the ball, the cube position is off (but the Raycast is still fine), so it must be a transform + maths issue I'm having. By 'off', I mean that if the ball is in the centre of the screen, the cube does not place itself central.

 var start = Camera.main.transform;
 var direction = start.TransformDirection(Vector3.forward);
 var centreHit : RaycastHit;
 
 if (Physics.Raycast (start.position, direction, centreHit, Mathf.Infinity, playerRay.value)) {
     
     Debug.DrawLine (start.position, centreHit.point, Color.yellow, 0.5);
 
     axisLock.x = Mathf.Clamp(centreHit.point.x - ball.position.x, -0.3, 0.3) + ball.position.x;
 
     forceHitVisual.position.x = axisLock.x;
 }

Further explanation

Below is a top down diagram showing my issue. Something I didn't say before is that my ball always rotates itself to face the camera. The camera is off shot, but all the raycasts you see are coming from it (The white lines are not raycasts, but the camera frame).

--- UPDATE ---

After some debugging - I came across that when I move the camera around on 0 degree rotatation (photo 1), the Z axis of the clamp ray is always the same (8.5), and the X axis changes depending where the centre is. But if I rotate the camera around the ball 90 degrees (same as in 3 and 4), both my X and Z axis start changing. How would I get it so that only my X axis affects the clamp ray, and Z stays as my depth?