Why is my first touch interpreted as a mouse click?

I made this video that demonstrates the problem (60s total, I get to the point in 10s) https://youtu.be/tRrZ1qJzmFw

A description for those who prefer to read:

I have several rectangles on the screen and I wish to move them individually using multitouch. My code looks like this:

  void Update()
     {
     // ** Touches ** \\
         tapCount = Input.touchCount;
         if (Input.touchCount > 0){ //Look through all the touches
           for ( int i = 0 ; i < tapCount ; i++ ) 
             {
                 Touch touch = Input.GetTouch(i);
                 switch (touch.phase){
                     case TouchPhase.Began:
                         if (getTouchedMagnet(touch.position) != null){
                             //getTouchedMagnet compared the touch position with a list of magnets and checks if the touch point is inside any of the magnets. It also logs which touch point is touching the magnet in an int.
                             getTouchedMagnet(touch.position).startTouch(i, touch.position);
                         }
                         break;
                     case TouchPhase.Moved:
                         if (getMagnetByInt(i) != null){
                             //update the position as the touch is moved. getMagnetByInt gets the magnet touched by this touch point
                             getMagnetByInt(i).updatePosition(touch.position);
                         }
                         break;
                     case TouchPhase.Ended:
                         if (getMagnetByInt(i) != null){
                             //set the magnet back to default
                             endInput(i);
                         }
                         break;
                     default:
                 //         Debug.Log("checking touch "+i+" but nothings happening");
                         break;
                         
                 }
             }
         }
         else{ //if no touches are active 
             foreach(Magnet magnet in magnets){
                 if (magnet.touchedBy !=99){ // end all touches(except mouse input) in case touch ends aren't registered
                     magnet.endTouch();
                 }
             }
         }
     // ** Mouse ** \\
     if (Input.GetMouseButtonDown(0) && getTouchedMagnet(Input.mousePosition) != null){
         //get magnet by position and set it to 'mouse' int value to find later
             getTouchedMagnet(Input.mousePosition).startTouch(99, Input.mousePosition);
     } 
     else if (getMagnetByInt(99) != null && Input.GetMouseButton(0)){
         //whilst mouse is active, upate position of magnet touched by mouse
         getMagnetByInt(99).updatePosition(Input.mousePosition);
     }
     else if (getMagnetByInt(99) != null && Input.GetMouseButtonUp(0)){
         //set magnet back to default
         endInput(99);
     }        }

The Problem When using multitouch, the position of first item I touch is set to an average of all the total touch points instead of the actual position of the first touch point. Subsequent items and touches work as I intended - the position is set under the actual touch point.

Again, the video does a much better job of explaining this https://youtu.be/tRrZ1qJzmFw - you'll see the issue within 10s

Things that might be important:

• These objects are positioned using the Canvas UI (for carefully considered reasons)

• I have a touch screen laptop and I see this in the Unity Editor in Windows 10 90% of the time

• I see this on Android device 100% of the time, but have not tested iOS

EDIT I solved this on device by deleting my "mouse" code. I think that Unity is treating my first touch as a mouse input. I understand that its trying to be helpful. Instead of Input.GetMouseButtonDown() Is there a way to say Input.GetMouseButtonDownButCheckItsDefinitelyNotATouch() ?