Mouse click sometimes destroys an instantiated object, but it sometimes doesn't. Why?

Hi everyone!

I have a simple 2D game that I've been working on for a day where objects are instantiated and fly across the screen. The goal of the game is to click on that object, which will destroy that object, and you get a certain amount of points. My problem is that I can sometimes click on an instantiated object once and it destroys, which is what I want, but some objects I have to click multiple times and then it finally destroys. Sometimes I click and click and click and it just won't destroy. I'm using OnMouseDown(). Here is my MouseClick code:

 using UnityEngine;
 using System.Collections;
 
 public class MouseClick : MonoBehaviour
 {
     public int scoreValue;
     private ScoreManager scoreManager;
 
     // Use this for initialization
     void Start ()
     {
         scoreManager = GameObject.FindGameObjectWithTag("GameController").GetComponent<ScoreManager>();
     }
     
     // Update is called once per frame
     void Update ()
     {
        
     }
 
     void OnMouseDown()
     {
         if (Input.GetMouseButtonDown(0))
         {
             if (gameObject.tag == "Cat") //+10 points
             {
                 scoreManager.totalScore += scoreValue;
                 Destroy(gameObject);
             }
 
             if (gameObject.tag == "Pizza") //+20 points
             {
                 scoreManager.totalScore += scoreValue;
                 Destroy(gameObject);
             }
 
             if (gameObject.tag == "Coffee") //+30 points
             {
                 scoreManager.totalScore += scoreValue;
             }
 
             if (gameObject.tag == "Totoro") //+40 points
             {
                 scoreManager.totalScore += scoreValue;
             }
             
         }
     }
 }

I'm really curious as of why some objects get destroyed in one click, and some objects take multiple clicks to be destroyed. If you would like, I can post pictures/videos.

Thank you for your help. Have a great day!