Camera moving independent of players rotation

My first person camera is designed so that when the player holds the middle mouse button down he/she can move the characters head on its shoulders. Only problem is when they let go it snaps back to be aligned with the character fine but when the button is pressed again it snaps back to its previous location.

I have tried everything from storing the bodies rotation in a quaternion and then making the camera transform match to having the characters rotation be reassigned to a vector3.zero and I cannot for anything get it to work correctly.

This is my switch trigger:

 void RotateHead()
     {
 
         if (Input.GetMouseButtonDown(2))
         {
 
             MBlook.headToggle = true;
         }
         else if(Input.GetMouseButtonUp(2))
         {
             if (transform.rotation != savedRotation)
             {
                 transform.rotation = Quaternion.Lerp(savedRotation, transform.rotation, Time.time*smooth);
 
             }
             MBlook.headToggle = false;
         }
     }

What I'm wanting to do is so that the camera will not snap to a different direction or rotation when the button is pressed but shift fluidly into the camera movement in the same spot the player is looking at when they press the button. I also tried taking the camera and storing its rotation directly prior to switching the head toggle to no avail. Any help would be greatly appreciated.