Detect touch lifted from object regardless of lift location

I need help figuring out how to detect when a touch is lifted from an object on screen. I'm raycasting from the touch position to detect a click/tap on my clickable objects. This works perfectly when the touch begins, i.e. TouchPhase.Began. This isn't a problem because most of my objects only need to detect the first tap, e.g. a TNT object gets tapped once then explodes.

Some of my objects need to perform some action until you lift your finger off of the object. This works fine, but only when the finger is lifted when directly over the object, i.e. if you press down on the object, then slide off of it and lift your finger, it won't detect the finger lift. I understand why it is happening due to my code, but I'm having trouble visualizing an efficient solution.

The problem should be evident in my code below, (I only test to see if the touch is lifted when a raycast returns the objects collider). I would love any feedback or suggestions you all have to offer. The code below is in a method called in Update()

 if(Input.touchCount > 0)
         {
             foreach(Touch t in Input.touches)
             {
                 hits = Physics2D.RaycastAll(Camera.main.ScreenToWorldPoint(t.position), Vector3.forward, Mathf.Infinity);
 
                 if (t.phase == TouchPhase.Began)
                 {
                     foreach(RaycastHit2D h in hits)
                     {
                         if(h.collider != null)
                         {
                             if(h.collider.gameObject.GetComponentInChildren<ClickableBlock>())
                                 h.collider.gameObject.GetComponentInChildren<ClickableBlock>().RayClick();
                         }
                     }
                 }
 
                 if (t.phase == TouchPhase.Ended)
                 {
                     foreach(RaycastHit2D h in hits)
                     {
                         if(h.collider != null)
                         {
                             if(h.collider.gameObject.GetComponentInChildren<ClickableBlock>())
                                 h.collider.gameObject.GetComponentInChildren<ClickableBlock>().RayOff();
                         }
                     }
                 }
             }
         }