Raycast calling both if and else statments

When i click on a gameobject in the first position of the list i should get points and remove the object from the list. If i click on a gameobject that is not in the first position in the list then it should be game over. But both the if and else are getting called at the same time. Why is this happening??? This is what i am doing:

Ray ray = Camera.main.ScreenPointToRay (Input.mousePosition);

RaycastHit hit;

             if (Physics.Raycast (ray, out hit)) 
             {
                     if (Input.GetMouseButtonDown (0)) 
                     {
                             
                         if(hit.collider.gameObject == list.playerButtons[0])
                             {
                                 list.playerButtons.Remove(list.playerButtons[0]);
                                 //Debug.Log (hit.collider.);
                                 gameController.AddScore ();
                                 Debug.Log("successfully");
                             }
                     

                         else
                             {
                                 gameOver = true;
                                 if(gameOver == true)
                                 {
                                     Debug.Log("failed");
                                     //Application.LoadLevel("Game Over");
                                 }
                             }
                     }

             }