Orbit Cam with different Views

Hey guys

I am trying to make an orbit cam with different views. I have 3 Views (Top, Outside, Inside) for a building and I want to use just one camera. The problem is, that when I swtich between these views (in smooth follow), the cam flies to the position, which works fine, but than the cam goes into Orbit mode (because the inside view has a "walk" mode and no orbit mode). In Orbit mode, it grabs the wrong position. I think thats because the ClampAngle value/function grabs the yMinLimit as position height and not the value where the camera flies too. So after the flying mode, the cam suddenly jumps to the yMinLimit position instead of keeping the position. Sorry if this is not really understandable :D

I hope some can help me???

 //Smooth Follow
 if(outside){
 Vector3 wantedPosition = target.TransformPoint(0, switchViewheight, -switchViewdistance);
 transform.position = Vector3.Lerp (transform.position, wantedPosition, Time.deltaTime * damping);
 
 Quaternion wantedRotation = Quaternion.LookRotation(target.position - transform.position, target.up);
 transform.rotation = Quaternion.Slerp (transform.rotation, wantedRotation, Time.deltaTime * rotationDamping);
 }
 
 //Orbit Mode right after the smooth follow is done
 if(outside){
 if (Input.GetMouseButton(0)){
      velocityX += xSpeed * Input.GetAxis("Mouse X") * outside_distance * 0.02f;
      velocityY += ySpeed * Input.GetAxis("Mouse Y") * 0.02f;
 }
      rotationYAxis += velocityX;
      rotationXAxis -= velocityY;
 
     rotationXAxis = ClampAngle(rotationXAxis, yMinLimit, yMaxLimit); 
 
     Quaternion rotation = Quaternion.Euler(rotationXAxis, rotationYAxis, 0);
 
     Vector3 position = rotation * (new Vector3(0.0f, 0.0f, -outside_distance)) + target.position;
 
     transform.rotation = rotation;
     transform.position =  position;
 
     velocityX = Mathf.Lerp(velocityX, 0, Time.deltaTime * smoothTime);
     velocityY = Mathf.Lerp(velocityY, 0, Time.deltaTime * smoothTime);
 }
 
  public static float ClampAngle(float angle, float min, float max){
         if (angle < -360F)
             angle += 360F;
         if (angle > 360F)
             angle -= 360F;
         return Mathf.Clamp(angle, min, max);
     }