Mouseclick GameObject through the view of a RenderTexture

My objective is to get it so that the user can click on a UI Panel (RenderTexture) which is showing a series of GameObjects, and the player can interact with those GameObjects through the RenderTexture. Sort of like an "Interactive Security Camera".

I currently have it working so long as the X and Z rotation of the 'RenderTexture' Camera (or the parent object) are 0. The Y rotation works fine regardless (basically so long as the Camera is perpendicular to the gameobjects it works fine)

I'm using a series of Raycasts to do all the checks etc.

However as soon as I rotate up/down or I roll the camera, it no longer works and I'm unsure why. Any help would be appreciated. Below is the code for the raycasts I currently have;

 public Camera consoleCamera;
 public LayerMask mask = -1;

 public void OnBlipPressed()
 {
     consoleCamera = GameObject.FindGameObjectWithTag("ConsoleCamera").GetComponent<Camera>();

     RaycastHit hit;

     Ray ray = consoleCamera.ScreenPointToRay(Input.mousePosition);

     if (Physics.Raycast(ray, out hit, mask))
     {
         if (hit.collider.gameObject.name == "SRSDisplayTarget")
         {
             var loc = hit.textureCoord;

             Camera srsCamera = transform.parent.GetComponent<HelmBehaviour_SRSController>().shipController.transform.GetChild(0).GetChild(0).gameObject.GetComponent<Camera>();
             Ray srsRay = srsCamera.ScreenPointToRay(new Vector2(loc.x * srsCamera.pixelWidth, loc.y * srsCamera.pixelHeight));
             srsRay.origin = new Vector2(srsRay.origin.x * -1, srsRay.origin.y);

             //Previous method (where I inverted the Y coords/the ray origin) needed me to reverse the direction, however I just need to invert the X coord (for some reason)
             //srsRay.direction *= -1;

             Debug.DrawRay(srsRay.origin, srsRay.direction, Color.red, 30.0f);

             //Debugging to show me where the ray is being drawn at rather than hunting for it
             GameObject test = Instantiate(new GameObject("Test"));
             test.transform.position = srsRay.origin;

             RaycastHit srsHit;
             if (Physics.Raycast(srsRay, out srsHit))
             {
                 Debug.Log(srsHit.collider.gameObject);
             }
         }
     }

 }