Transitioning from different number of touches causes behavior overlap

Hey all! So I have a functionally basic model of moving my dudes around my environment. However 2 of my control input methods are not playing nicely!

My basic control setup is that pinching and pulling the screen with 2 touches will zoom the camera and touching a valid location on the screen with 1 will move the character. This all works correctly. The issue is that often when putting down or lifting up 2 fingers their strikes are often milliseconds apart causing the movement behavior to fire at an undesired location. I'm pretty sure the problem lies in how I am gating the behaviors (currently using Input.touchCount singularly) but I can't really think of another way that doesn't feel extremely complicated. Here are both pieces of code.

 void Update () {
         if (Input.touchCount == 1 && Input.GetTouch(1).phase != TouchPhase.Ended) {
             Touch touch = Input.GetTouch(0);
             if (touch.phase == TouchPhase.Began) {
                 touchMark = Time.time;
                 if (interactionRay(touch.position)) {
                     switch (hit.collider.gameObject.layer) {
                     case (int)Indexes.Layers.playerActor:
                         selectedActor = hit.collider.gameObject.transform;
                         StartCoroutine(myCameraMan.panToTarget(selectedActor, 1f, true));
                         break;
                     default:
                         if (touchDuration > touchDurationGate) actorMotors[selectedActor.gameObject.name].moveToPoint(hit.point);
                         break;
                     }
                 }
             }    
             if (touchMark == 0) touchMark = Time.time;
             touchDuration = Time.time - touchMark;
             if (interactionRay(touch.position) && touchDuration > touchDurationGate && (touch.phase == TouchPhase.Moved || touch.phase == TouchPhase.Stationary)) {
                 actorMotors[selectedActor.gameObject.name].moveToPoint(hit.point);
             }
             if (touch.phase == TouchPhase.Ended) {
                 touchDuration = 0f;
                 touchMark = 0f;
             }
             touchesDebug.text = touch.phase.ToString();
         } else {
             touchesDebug.text = "none";
         }
         if (selectedActor != null) {
             targetDebug.text = selectedActor.gameObject.name;
         }
         //Debug.Log(touchDuration.ToString() + "mark" + touchMark.ToString());
     }

And here is the camera code for 2 touches (living in another script).

 //Deal with pinch/pull zoom @2 touches
         if (Input.touchCount == 2 && !cameraMoving) {
             Touch fingerOne = Input.GetTouch(0);
             Touch fingerTwo = Input.GetTouch(1);
             float touchDistance = Vector2.Distance(fingerOne.position, fingerTwo.position);
             if (fingerOne.phase == TouchPhase.Moved || fingerTwo.phase == TouchPhase.Moved) {
                 float deltaTouchDistance = Vector2.Distance(fingerOne.position - fingerOne.deltaPosition, fingerTwo.position - fingerTwo.deltaPosition);
                 float newFoV = Mathf.Clamp(mainCam.orthographicSize + Mathf.Clamp(deltaTouchDistance - touchDistance, zoomClamp  * -1f, zoomClamp), camSizeClamp[0], camSizeClamp[1]);
                 mainCam.orthographicSize = newFoV;
             }
             if (fingerOne.phase == TouchPhase.Ended || fingerTwo.phase == TouchPhase.Ended) {
                 if (mainCam.orthographicSize < camSizeClamp[0] + bounceOffset) StartCoroutine(zoomCamera(camSizeClamp[0] + bounceOffset, 2));
                 if (mainCam.orthographicSize > camSizeClamp[1] - bounceOffset) StartCoroutine(zoomCamera(camSizeClamp[1] - bounceOffset, 2));
             }
         }

You can see in the code pasted first moving the character I setup a timer to try and gate the touch. The lowest effective thresh hold that causes the behaviors to work as I want is around .25f which makes moving characters feel really unresponsive and bad...

Any help is of course appreciated!