Rigidbody going through wall during pinch-to-zoom

So I have a rigidbody on an object and the child of the object is the camera. When I pinch to zoom, I need to translate/MovePosition the camera forward/backwards depending on the pinch. The camera is in a room that can be a various sizes and it needs to stay in the room, but somehow when I pinch, I still leave the room. What's wrong?

There are box colliders on the walls. I used the code from the Unity tutorial on pinchZoom and adjusted it to my needs.

 public class PinchZoom : MonoBehaviour
 {
     public float perspectiveZoomSpeed = 0.5f;        // The rate of change of the field of view in perspective mode.
     public float orthoZoomSpeed = 0.5f;        // The rate of change of the orthographic size in orthographic mode.
     public GameObject FPSCam;
     private Camera cam;
     private Rigidbody rb;
 
     void Start(){
         cam = FPSCam.GetComponentInChildren<Camera> ();
         rb =FPSCam.GetComponent<Rigidbody>();
 
     
     }
     void FixedUpdate()
     {
         //camera.transform.GetComponent<Rigidbody> ().velocity = Vector3.zero;
 
         // If there are two touches on the device...
         if (Input.touchCount == 2)
         {
             // Store both touches.
             Touch touchZero = Input.GetTouch(0);
             Touch touchOne = Input.GetTouch(1);
 
             // Find the position in the previous frame of each touch.
             Vector2 touchZeroPrevPos = touchZero.position - touchZero.deltaPosition;
             Vector2 touchOnePrevPos = touchOne.position - touchOne.deltaPosition;
 
             // Find the magnitude of the vector (the distance) between the touches in each frame.
             float prevTouchDeltaMag = (touchZeroPrevPos - touchOnePrevPos).magnitude;
             float touchDeltaMag = (touchZero.position - touchOne.position).magnitude;
 
             // Find the difference in the distances between each frame.
             float deltaMagnitudeDiff = prevTouchDeltaMag - touchDeltaMag;
 
             rb.velocity = Vector3.zero;
 
             rb.MovePosition(FPSCam.transform.position + cam.transform.forward * .4f * deltaMagnitudeDiff);
 
         }
     }
 }