Camera bounds with zoomable camera

I'm making a top-down 2D shooter and I have a camera control script that allows you to zoom in and out with the scroll wheel. But I need to add bounds to the camera. This is what I have at the moment.

 void CameraZoom()
     {
         zoom -= Input.GetAxis("Mouse ScrollWheel") * zoomSensitivity;
         zoom = Mathf.Clamp(zoom, zoomMin, zoomMax);
         cam.orthographicSize = Mathf.Lerp(cam.orthographicSize, zoom, Time.deltaTime * zoomSpeed);
         CalculateCameraBounds();
     }
 
     void CalculateCameraBounds()
     {
         vertBounds = cam.orthographicSize;
         horzBounds = vertBounds * cam.aspect;
 
         minX = horzBounds - mapX / 2.0f;
         maxX = mapX / 2.0f - horzBounds;
         minY = vertBounds - mapY / 2.0f;
         maxY = mapY / 2.0f - vertBounds;
     }
 
     void CameraFollow()
     {
         Vector3 targetPosition = new Vector3(target.position.x, target.position.y, transform.position.z);
 
         Vector3 boundPosition = new Vector3(
             Mathf.Clamp(targetPosition.x, maxX, minX),
             Mathf.Clamp(targetPosition.y, minY, maxY),
             transform.position.z);
 
         transform.position = Vector3.Lerp(transform.position, boundPosition, smoothing);
     }

This works for bounding the camera's Y position,. but doesn't for the X. It just makes the minX and maxX values larger, unlike the the minY and maxY values getting smaller like I want. I'm pretty sure I know the issue lies here:

 vertBounds = cam.orthographicSize;
 horzBounds = vertBounds * cam.aspect;

As I zoom the camera out, the orthographic size gets larger, which is then being multiplied by the aspect , thus making the horizontal bounds larger rather than smaller. I'm aware of this and have spent hours trying to figure out a way to change it up so that doesn't happen, but nothing comes to mind. Would appreciate any ideas or solutions.