Maintaining Camera Rotation between modes

I am creating a custom camera to view a 3D object. The camera is normally a fly cam but on drag of an object, I want the camera to orbit the object. I have achieved this already but the camera jumps between rotations when swapping between the modes reseting to the previous rotation value. How would I smooth this so the camera maintains the same rotation after orbit and the user can continue on the fly cam mode from this rotation. Here is my code:

 if (!rotateToggle)
 {
     if (Input.GetMouseButton(0)){
         rotationX += Input.GetAxis("Mouse X") * cameraSensitivity * Time.deltaTime;
         rotationY += Input.GetAxis("Mouse Y") * cameraSensitivity * Time.deltaTime;
     }
     rotationY = Mathf.Clamp(rotationY, -90, 90);

     transform.localRotation = Quaternion.AngleAxis(rotationX, Vector3.up);
     transform.localRotation *= Quaternion.AngleAxis(rotationY, Vector3.left);
     if (Input.GetKey(KeyCode.LeftShift) || Input.GetKey(KeyCode.RightShift))
     {
         cameraSpeed = 150f;
     }
     else
     {
         cameraSpeed = 50f;
     }


     transform.position += transform.forward * cameraSpeed * Input.GetAxis("Vertical") * Time.deltaTime;
     transform.position += transform.right * cameraSpeed * Input.GetAxis("Horizontal") * Time.deltaTime;
 }else{

     rotationX = Input.GetAxis("Mouse X") * RotateAmount;
     rotationY = Input.GetAxis("Mouse Y") * RotateAmount;

     Vector3 angles = transform.eulerAngles;
     angles.z = 0;
     transform.eulerAngles = angles;
     transform.RotateAround(rotationTarget.position, Vector3.up, rotationX);
     transform.RotateAround(rotationTarget.position, Vector3.left, -rotationY);
     transform.LookAt(rotationTarget);

 }

rotateToggle is true when the user is dragging the object and rotationTarget is the orbit target's transform.