Touch Input only working on Unity Remote

Well First post ever, here we go, I apologize for any bad english, not a native speaker. When I test my app on Unity Remote it works just fine, but when I install it on my phone, the touch, only the touch, is very laggy, like it has some sort of delay. The animations, the rendering, all other gameobjects seem to work just fine.

I optimized as much as I could, meshes, shadows off, shared materials, script optimization. I even switched the quality settings form "Fantastic" to "Fastest". And altought I did get an improvement the touch is still a bit laggy.

My FPS is around 500 - 700fps most of the time, altough it goes down to around 200fps in some parts of the courutinnes. (Around once in every minute). Insane drop I know, but as far as I know 200fps is still a very good number. Altough I am developing on a PC with a very good processor and I am testing on a very poor android device so... well you know.

Here is the script I use for my touch input, it controls a ball by swipping up,down,left and right:

private void Awake() { // Set up the reference. ball = GetComponent(); }

     private void Update()
     {
         if (Input.touchCount > 0)     
         {
         h = 0;
          v = 0; // stops character
         Touch touch = Input.touches[0];
         switch (touch.phase) 
             {
         case TouchPhase.Began:
             startPos = touch.position;
             break;    
         case TouchPhase.Ended:
                 float swipeDistVertical = Mathf.Abs(touch.position.y - startPos.y); // distance touch moved in Y axis
                 float swipeDistHorizontal = Mathf.Abs(touch.position.x - startPos.x); // distance touch moved in X axis
                 if (swipeDistVertical > swipeDistHorizontal) // compare Y movement and X movement to see where player was attempting to swipe
                 {
                     // player attempted to swipe vertically
                     v = Mathf.Sign(touch.position.y - startPos.y); // if -1 moved down, if 1 moved up;                     
                 }
                 else
                 {
                     // player attempted to swipe horizontally
                     h = Mathf.Sign(touch.position.x - startPos.x); // if 1 moved right, if -1 moved left
                 }
             break;
             }
         }
     move = (v*Vector3.forward + h*Vector3.right).normalized;
     }


     private void FixedUpdate()
     {
         // Call the Move function of the ball controller
         ball.Move(move);
     }