Raycast To Destroy Object Requires Double Click?

So essentially I'm just attempting to very simply have it so I can click on an object in the scene and then destroy it, I realize there's the whole OnMouseDown thing but for how I am going to be building this project latter on I need the added functionality that the raycast method provides. Now for some reason completely inexplicable to me, when I run this script and click on gameobjects (with the correct tag and colliders) it won't destroy it. I have to click each object individually twice in order for it to be destroyed, the ammo counter goes up on each click, I've even had debug statements and tried moving around the Destroy code. Each object has to be clicked twice. And each gameobject seems to remember how many times it's been clicked, so if I click on one object then go click somewhere else in the scene and back to that object it will be Destroyed, it is really boggling my mind. I'm sure it's probably something really simple, but I cannot figure it out. Any help would be great, thanks.

 using UnityEngine;
 using System.Collections;
 
 public class Player : MonoBehaviour {
 
     public int ammo;
     RaycastHit hit;
     bool sHit;
 
     // Use this for initialization
     void Start () {
     
     }
     
     // Update is called once per frame
     void Update () {
         if (Input.GetButtonDown ("Fire1")) {
             Ray ray = Camera.main.ScreenPointToRay (Input.mousePosition);
 
             if (Physics.Raycast (ray.origin, ray.direction, out hit)) {
                 if (hit.collider.tag == "Shuriken") {
                     Destroy (hit.collider.gameObject);
                     ammo++;
                 }
             }
         }
     }
 }