How to maintain player chosen camera height?

So I have a camera that the player controls. The camera can be raised or lowered by use of the middle mouse wheel. There is a limit to how close the camera may get to the ground or far away from the ground by use of a Raycast. If I roll over a bump, it will raise my camera accordingly but after the bump it won't lower it to the previous height that the player chose to have. Here is the code:

 private void moveCamera()
     {
         //Define the y Vector position
         var y = transform.position.y;
         var newy = y;
         
         //Find Mouse X and Y Position on Screen
         float xMouse = Input.mousePosition.x;
         float yMouse = Input.mousePosition.y;
         
         //Move Player Forward                                            
         if(Input.GetKey(KeyCode.W) || yMouse <= Screen.height && yMouse > Screen.height - moveBorderWidth)
             transform.Translate(Vector3.forward * moveSpeed * Time.deltaTime);
         
         //Move Player Backward
         if(Input.GetKey(KeyCode.S) || yMouse >= 0 && yMouse < moveBorderWidth)
             transform.Translate(-Vector3.forward * moveSpeed * Time.deltaTime);
         
         //Move Player Left
         if(Input.GetKey(KeyCode.A) || xMouse >= 0 && xMouse < moveBorderWidth)
             transform.Translate(Vector3.left * moveSpeed * Time.deltaTime);
         
         //Move Player Right
         if(Input.GetKey(KeyCode.D) || xMouse <= Screen.width && xMouse > Screen.width - moveBorderWidth)
             transform.Translate(Vector3.right * moveSpeed * Time.deltaTime);
         
         //Cast a ray downward, return distance        
         if(Physics.Raycast(transform.position,-Vector3.up,out hit))
         {
             groundPos = hit.point.y;
             print (groundPos);
             distanceToGround = hit.distance;
         }
                                         
         //Use the middle mouse scroll to adjust height of camera 
         y -= moveSpeed * Input.GetAxis("Mouse ScrollWheel");
     
         //Keep height between 10 above ground and 40 above ground
         float MinCameraHeight = groundPos + 10;
         float MaxCameraHeight = groundPos + 40;
         y = Mathf.Clamp(y, MinCameraHeight, MaxCameraHeight);
                                                 
         //Keep the current y Vector position and update
         var pos = transform.position;
         pos.y = y;
         transform.position = pos;
                     
     }

So how would I go about forcing the camera to go back down after the bump(or back up after a valley) according to the desired height the player chooses?