Multi touch input finger problem

Alright so i want half of the screen to serve as input to look around, and one half of the screen to have buttons and such to move around. This is the script i have written. The problem that is happening, is that when i put my finger on the left side of the screen, and then i put one on the right, it THINKS that i moved my finger all the way from the left side to the right side in one frame(i think). With just one finger down the look works fine. When i have my LEFT finger down the direction becomes an absurdly large number. I hope i am explaining this okay, because i want unity to see that its on the left side of the screen and just stop, how do i do that?

 private Vector2 touchStart;
 private Vector2 direction;
 
 void Update(){
             for(int i=0; i < Input.touchCount; i++){
                 Touch touch = Input.GetTouch(i);
                 if(touch.position.x > Screen.width / 2 && touch.phase == TouchPhase.Moved){
                     direction = touch.position - touchStart;
                     direction = new Vector2(Mathf.Clamp(direction.x, -100, 100), Mathf.Clamp(direction.y, -100, 100));
 
                 }else{
                     direction = new Vector2(0, 0);
                 }
                 touchStart = touch.position;
             }
 }