Non consistant 2D Touch result

Hi,

I'm working on a game involving display of 10 to 100 game objects at the same time. Player has to touch one of the game object. And here's where my trouble start. Game is in 2D, using an orthographic camera. All my game objects have RigidBody2D and Box Collider 2D.

Most of the time, it works correctly, but sometimes game seems to fail to catch user's touch.

Here is the code I use in my "main" script.

 if (Input.touchCount > 0)
         {
             Touch userTouch = Input.GetTouch (0);
             Vector3 wp = Camera.main.ScreenToWorldPoint(userTouch.position);
             Vector2 touchPos = new Vector2(wp.x, wp.y);
             Collider2D hit = Physics2D.OverlapPoint (touchPos);
             MyOtherScript myGameObject = hit.attachedRigidbody.GetComponentInParent<MyOtherScript> ();
 
             if (!myGameObject.isClicked && userTouch.phase == TouchPhase.Began)
             {
                 myGameObject.SomeFunction ();
             } 
 
         }

At first, I suspected my phone to not catch correctly touch, but I've tested on different device, both iOS and Android, and I got the same result : touch works 95% of the time, but fail the rest. And it brings to pretty frustrating gameplay...

Some more information on gameplay : once a game object is touched, game test if it is the right one (SomeFunction() ), remove all game object from the screen and display some fresh game object.

Does anyone would have an idea to help me ?

Thanks in advance !