Camera movement and zoom bounds

I have a perspective camera that can move left/right and can zoom in/out.

I need to keep it in some bounds so it doesn't view outside the map. I managed to make it work for a normal non-zoomed state, but the problem happens when I zoom.( I change the camera's Z-axis and it's FOV ) it either sees outside the map or it doesn't see all of it.

I have tried using the frustum size using this code :

 float distance = Vector3.Distance(transform.position, (leftLimit.position + rightLimit.position) * 0.5f);
 frustumHeight = FrustumHeightAtDistance(distance);
 frustumWidth = frustumHeight * camera.aspect;
 
 newPos.x = Mathf.Clamp(newPos.x, leftLimit.position.x + frustumWidth * 0.5f, rightLimit.position.x - frustumWidth * 0.5f);
 transform.position = newPos;

But this doesn't work either. Any thoughts ?