Panning camera based on a fixed point.

I have a 3D perspective camera looking down on my scene at a fixed 40-degree angle that can be panned around the scene by tapping and dragging. Pinching is also used to move the camera in and out (for zooming). The camera should never rotate from its original rotation, only translate.

I want a system where as I pan the camera around, the touch position perfectly tracks the object that was touched when the pan started, regardless of the distance that the camera is from the object.

My proposed solution was to raycast to the touch position to determine how far away it is and modify the pan amount by the distance from the camera to the raycast hit position. I quickly found that the required relation needed was not linear and seemed to be exponential (i.e as the camera moves further away from the selected object, it should pan faster to keep up). I have been playing around with values trying to get it right but it never seems to work correctly.

Is there a simple way of solving this that I am missing? I'm thinking maybe I'm missing a trig relationship to do with the camera being rotated 40 degrees on the X axis. Any help would be greatly appreciated.