Raycasting with Oculus and Leap Motion

So essentially I'm creating an on-rails first person shooter using the Oculus and Leap Motion. Adding the Oculus causes the raycast that controls the hit detection to veer off target. I've found a couple of threads that talk about this problem but none that sufficiently solve my issue.

Initially the raycast was done

     InteractionBox iBox = frame.InteractionBox;
     HandList hands = frame.Hands;
     Hand firstHand = hands [0];
     Vector stabilizedPosition = firstHand.StabilizedPalmPosition;
     
     Vector handCenter = stabilizedPosition;
     Vector normalizedPosition = iBox.NormalizePoint(handCenter);
     
     float x = normalizedPosition.x;
     float y = normalizedPosition.y;
 
     Camera.main.ScreenPointToRay(new Vector2(x,y));
     if (Physics.Raycast (ray, out hit, Range))
     {   
          //hit stuff
     }

But using ScreenPointToRay caused the ray to be distorted by the Oculus.

The current set up is that the weapons and the shooting script attached to them are all attached to the right camera. The ray is cast using:

 float x = normalizedPosition.x;
 float y = normalizedPosition.y;
 float z = normalizedPosition.z;
 
 Vector3 cameraPos = transform.position;
 Vector3 cameraDir = transform.forward;
 Vector3 cursorPos = new Vector3(x,y,z);
 Vector3 direction = cameraDir + cursorPos;
 
 //denormalize
 float directionx = direction.x * screenWidth;
 float directiony = direction.y * screenHeight;
 float directionz = direction.z * screenWidth;
 Vector3 rayDirection = new Vector3(directionx,directiony,directionz);
 
 if (Physics.Raycast (cameraPos, rayDirection, out hit, Range)){
 }

However, the raycast doesn't seem to be following the camera's movements exactly as it should and there also still seems to be some sort of distortion in that straight up and down movement seems to cause a curve. Even the Debug.DrawRay doesn't seem to be matching up with where the hit markers appear.

So what I want to know is there any way to negate the distortion effect for the first approach? Failing that is there a way to improve my second raycasting approach?