Trying to get just one object of duplicates to move

I'm trying to make a game where you can click on one object to select it for rotation, and am using Raycasting, but somehow it's selecting both objects connected to the same script even though they have different colliders?

 if (Input.GetMouseButtonDown(0))
         {
             Ray ray = Camera.main.ScreenPointToRay(Input.mousePosition);
             // Casts the ray and get the first game object hit
             if (Physics2D.GetRayIntersection(ray, Mathf.Infinity) == col)
             {
                 selected = true;
             }
             else
             {
                 selected = false;
             }

col is the collider I initialize in the Start() function with GetComponent. I was trying to use regular Physics.Raycast before but for some reason it wasn't working. Now it works: I can select and deselect, but only both at a time, no matter which one I click on.