3rd person camera with toggle-able free look, problem with transform.Rotate

I have a camera that is locked onto the character. The player can then press alt to freely move the camera independently from the character. The problem I am having is with transform.Rotate, as it updates every frame, something weird happens and the camera starts rotating around its z-axis turning upside down.

So I want to use:

 mouseX = Input.GetAxis ("Mouse X");
 mouseY = Input.GetAxis ("Mouse Y");
 
 character.Rotate (0, mouseX, 0);
 transform.Rotate (-mouseY, 0, 0);

while also using this:

 mouseX += Input.GetAxis ("Mouse X");
 mouseY -= Input.GetAxis ("Mouse Y");
 
 centerPoint.localRotation = Quaternion.Euler (mouseY, mouseX, 0);

However, if both are used, then you can't change smoothly between locked and free-look. Here is the whole code:

 using System.Collections;
 using System.Collections.Generic;
 using UnityEngine;
 
 //Script is attached to the centerpoint go that the main camera uses as a pivot point.
 
 public class Help : MonoBehaviour {
 
     public float cameraReturnSpeed = 300.0f;
     public Transform character;
     public Transform NeutralPoint;
 
     private bool returnCamera = false;
     private float mouseX, mouseY;
 
     //TODO: When exiting free look, rotate y back to previous position, not back to neutral.
     //TODO: Don't turn the camera upside down in free look.
 
     // Use this for initialization
     void Start () {
         
     }
     
     // Update is called once per frame
     void Update () {
 
         //Rotate character around y and rotate camera view around x with the mouse.
         mouseX = Input.GetAxis ("Mouse X");
         mouseY = Input.GetAxis ("Mouse Y");
 
         //Allow free look when left alt key is pressed.
         if (Input.GetKey (KeyCode.LeftAlt)) {
             returnCamera = false;
             //Rotate adds to the rotation every frame, this causes the problem. 
             transform.Rotate (-mouseY, mouseX, 0);
         }
         else
         {
             character.Rotate (0, mouseX, 0);
             transform.Rotate (-mouseY, 0, 0);
         }
         if (Input.GetKeyUp (KeyCode.LeftAlt)) {
             StartCoroutine(ReturnCamera ());
         }
 
         if (returnCamera == true) {
             if (Mathf.Approximately(transform.rotation.x, NeutralPoint.rotation.x)) {
                 returnCamera = false;
             }
             else {
                 transform.localRotation = Quaternion.RotateTowards (transform.localRotation, Quaternion.Euler (transform.localRotation.y, 0, 0), Time.deltaTime * cameraReturnSpeed);
             }
         }
         
     }
 
     IEnumerator ReturnCamera (){
         returnCamera = true;
         yield return new WaitForSeconds (0.6f);
         transform.localRotation = Quaternion.Euler (transform.localRotation.y, 0, 0);
         returnCamera = false;
     }
 }

I've been struggling with this for some time now, hopefully you guys can help me out.