(2D) Camera negative boundaries

I have a camera that uses raycast to find a maximum value that I use in clamping, but my problem lies with the minimum value.

I need the minimum value to be directly in front of the player (that works fine), but if the value in front of the player negative, the camera goes out of control.

Any ideas on how to prevent this without using many if statements?