Different Camera changes work against each other?

So in my Camera script, I have two separate, for lack of a better word, functions that allows you to zoom as well as move your camera around. (Both of them are in the update function, so just remember that it isn't actually a function.) You can drag the camera around by holding Middle Mouse button and dragging; and you can zoom in an out using your scroll wheel. However, this clashes together, because of my script. My script is as follows (pay attention to lines 18-24 the most. I believe it has something to do with the second if statement):

 void Update () {
 
         if (Input.GetMouseButtonDown(2)) {
             dragOrigin = Input.mousePosition;
         }
 
         if(!Input.GetMouseButton (2)) {
             return;
         }
 
         Vector3 pos = Camera.main.ScreenToViewportPoint(Input.mousePosition - dragOrigin);
         Vector3 move = new Vector3(pos.x * dragSpeed, 0, pos.y * dragSpeed);
         
         transform.Translate(move, Space.World);
 
         if(Input.GetAxis ("Mouse ScrollWheel") < 0) {
             if(Camera.main.fieldOfView <= 50) {
                 Camera.main.fieldOfView += 2f;
             }
             if(Camera.main.orthographicSize <= 20) {
                 Camera.main.orthographicSize += 0.5f;
             }
         }
 
         if(Input.GetAxis("Mouse ScrollWheel") > 0) {
             if(Camera.main.fieldOfView > 20) {
                 Camera.main.fieldOfView -= 2f;
             }
             if(Camera.main.orthographicSize >= 1) {
                 Camera.main.orthographicSize -= 0.5f;
             }
         }
 
     }