Steam VR 1 out of 4 angles not registering correctly

We are making a VR block puzzler game in Unity 5.4 (VR Mode enabled/single pass stereo rendering on) using SteamVR + VRTK and HTC Vive HMD & controllers. In this game we rotate gameobjects based on the camera position and respective object position so we get a logical rotation form perspective of the player. Everything is working great except one view angle, the script is not receiving the right info about the positions or maybe calculations are not right. It is very strange because all view angles are working perfectly except that one (the angle from the left of the starting position if you’re looking at the center of your roomscale play area), from that specific angle, it works very sporadically, almost as if the coordinates are not registering. Note: our swiping mechanic works fine from that angle, it just seems to be the rotate left, right, forward and back actions that don’t register correctly.

Screenshot example: https://dl.dropboxusercontent.com/u/7673/RLTYCHK/Bad%20angle.PNG

Here's the code we're using:

 if(controller && controller.touchpadTouched)
 {
     Vector3 difference = transform.position - cam.transform.position;
     float z = difference.z;
     float x = difference.x;
     if(!isTouching)
     {
         initialX = controller.GetTouchpadAxis().x;
 
 
         isTouching = true;
 
     }
     else
     {
         xOff = controller.GetTouchpadAxis().x - initialX;
         if(xOff > 0.3f && !swiped) //swipe right
         {
             Debug.Log("Swipe Right");
             if (Mathf.Abs (z) > Mathf.Abs (x)) {
                 if (z >= 0) {
                     transform.RotateAround (transform.position, Vector3.forward, -90);
                 } 
                 else {
                     //PROBLEMATIC ANGLE
                     transform.RotateAround (transform.position, Vector3.forward, 90);
                 }
             }
             else if (Mathf.Abs (z) < Mathf.Abs (x) ) {
                 if (x >= 0) {
                     transform.RotateAround (transform.position, Vector3.right, -90);
                 } 
                 else {
                     transform.RotateAround (transform.position, Vector3.right, 90);
                 }
             }
             SoundManager.main.PlaySound ("RotateBottom", GetComponent<AudioSource> ());
             swiped = true;
         }
         else if(xOff < -0.3f && !swiped)
         {
             Debug.Log("Swipe Left");
             if (Mathf.Abs (z) > Mathf.Abs (x)) {
                 if (z >= 0) {
                     transform.RotateAround (transform.position, Vector3.forward, 90);
                 } 
                 else {
                     transform.RotateAround (transform.position, Vector3.forward, -90);
                 }
             }
             else if (Mathf.Abs (z) < Mathf.Abs (x) ) {
                 if (x >= 0) {
                     transform.RotateAround (transform.position, Vector3.right, 90);
                 } 
                 else {
                     transform.RotateAround (transform.position, Vector3.right, -90);
                 }
             }
             SoundManager.main.PlaySound ("RotateUp", GetComponent<AudioSource> ());
             swiped = true;
         }
     }
 }
 else
 {
     
     xOff = 0;
     isTouching = false;
     swiped = false;
 }