Mouse control rotate around player

I have a third person camera that smoothly looks at and follows the player. While the player is not in battle, I want the player to be able to hold RMB and rotate around himself so that you can look at the player from whatever angle you want. Here is the relevant block of code I have so far:

             //free look when we can and RMB down
             if (canFreeLook)
             {
                 cam.transform.rotation = Quaternion.LookRotation(cameraTarget.position - cam.transform.position);
                 if (Input.GetMouseButton(1))
                 {
                     float x = Input.GetAxis("Mouse X");
                     float y = Input.GetAxis("Mouse Y");
                     if((x <= 0 && currentFreeLookOffset.x >= -maxDist) || (x >= 0 && currentFreeLookOffset.x <= maxDist)) 
                         currentFreeLookOffset += Vector3.right * x;
                     if((y <= 0 && currentFreeLookOffset.y >= cameraTarget.position.y) || (y >=0 && currentFreeLookOffset.y <= maxHeight))
                         currentFreeLookOffset += Vector3.up * y;
                 }
             }

This moves the camera left right up and down and orients the camera to always face the player. But it's not what I want. I want the camera to stay the same distance away from the player and just rotate around him.