Click To Delete Specific Object

I'm making a game where you go around collecting notes off of trees (much like Slender, but simpler, and with only trees) to give my friends a good scare and to help learn Unity. Right now, I'm having trouble getting the notes to disappear once clicked at a close range. I have this code attached to my object, Note1, and my first person camera (because I'm not sure which one the code for object deletion should go on). Right now, the code deletes anything that I click on (the tree, terrain, etc). Help please?

function Update ()
{
   if ( Input.GetMouseButtonDown(0) )
   {
      var hit : RaycastHit;
      var ray : Ray = Camera.main.ScreenPointToRay (Input.mousePosition);
      if (Physics.Raycast (ray, hit, 5.0))
      {
         Destroy(hit.collider.gameObject);
      }
   }
}