TouchPhase.Ended does not work if ray is not hitting object.

I am using a raycast to detect if a player is touching an object. When the player touches the object, it fires. Firing works great, however it seems as if the player releases the finger anywhere other than right on top of the object where the ray hits, it does not recognize that the player let go, and continues to fire. What would be a better way to approach this?

 var myTouches:Touch[] = Input.touches;
 for(var i:int=0;i<Input.touchCount;i++)
 {
  var hit : RaycastHit;
  var ray = Camera.main.ScreenPointToRay (myTouches[i].position);          
  
     if (Physics.Raycast (ray, hit, 1000))
     {  
      
         if(hit.collider.gameObject.tag == "handle") 
         { 
   var tower = hit.collider.transform.parent.gameObject.GetComponent(TowerController);
   
             if (myTouches[i].phase == TouchPhase.Began) 
             {           
                 tower.firing=true;
                 tower.handlefree=false;  
             }         
   
             if (myTouches[i].phase == TouchPhase.Ended)
            {   

tower.firing=false; tower.handlefree=true; } }