Punching holes through a texture and making them regenerate back again in an efficient way

I'm working on a prototype where the user can use the mouse to punch holes through a texture, making this texture see-through in that area for a certain amount of time. After that time, the hole will "regenerate" back again, effectively just increasing the alpha value back to 1 until is opaque. I already have all this working, but I'm afraid my current implementation is not exactly good in terms of performance, and is definitely not suitable for big textures. This is what I have working right now, using small textures (the hair and the head are separate textures):

What I'm doing is using a depth mask shader and drawing the holes (black circles) in the second texture with every mouse click. This works perfectly well, but then I need to regenerate the holes, so what i do is (and I totally realize this is a really hacky approach) use the red channel in the color of each pixel to determine if that pixel should lerp back to opaque (go back to an alpha value of 1). As I said, this kind of works for small textures, but as soon as I try it on a big one, it is just unable to handle that amount of pixels at once. I tried to summarize the code of my implementation here:

 public class TextureHoles : MonoBehaviour
 {
     public Texture2D sourceTexture;
     public Camera sceneCamera;
     public LayerMask targetLayerMask;
 
     public Color drawColor;
     public float circleRadiusSize;
 
     private Texture2D currentTexture;
 
     private Renderer rend;
     private MeshCollider meshCollider;
 
     private Color[] colors;
     private Color[] targetColors;
 
     private List<Hole> holes = new List<Hole>();
     private float redChannelValue = 0f;
     private const float HOLES_MAX_TIME = 3f;
 
     [System.Serializable]
     public class Hole
     {
         public Hole(Color _color)
         {
             color = _color;
             timer = 0f;
         }
 
         public void Run()
         {
             timer += Time.deltaTime;
         }
 
         public Color32 color;
         public float timer;
     }
 
     private void Start()
     {
         // create copy of source tex
         currentTexture = new Texture2D(sourceTexture.width, sourceTexture.height, TextureFormat.RGBA32, false);
         currentTexture.SetPixels(sourceTexture.GetPixels());
         currentTexture.Apply();
 
         rend.material.SetTexture("_Mask", currentTexture);
 
         targetColors = currentTexture.GetPixels();
     }
 
     private void Update()
     {
         if(Input.GetMouseButtonDown(0))
         {
             CutHair();
         }
 
         for(int i = 0; i < holes.Count; i++)
         {
             RecoverHair(holes[i]);
         }
     }
 
     public void CutHair()
     {
         RaycastHit hit;
 
         if(Physics.Raycast(Cursor.Instance.GetRayInitialPos(), Vector3.forward, out hit, Mathf.Infinity, targetLayerMask))
         {
             Vector2 uv;
 
             uv.x = (hit.point.x - hit.collider.bounds.min.x) / hit.collider.bounds.size.x;
             uv.y = (hit.point.y - hit.collider.bounds.min.y) / hit.collider.bounds.size.y;
 
             Vector2 coord = new Vector2((int)(uv.x * currentTexture.width), (int)(uv.y * currentTexture.height));
 
             int circleRadius = (int)(Cursor.Instance.GetCircleSize() * circleRadiusSize);
 
             // we use the red channel value to identify which color we should lerp back to opaque
             redChannelValue += .03f;
             Color holeColor = new Color(redChannelValue, 0f, 0f, 0f);
 
             currentTexture.DrawCircle(holeColor, (int)coord.x, (int)coord.y, circleRadius);
             currentTexture.Apply(true);
 
             holes.Add(new Hole(holeColor));
         }
     }
 
     private void RecoverHair(Hole hole)
     {   
         if(hole.timer >= HOLES_MAX_TIME) return;
 
         hole.Run();
 
         colors = currentTexture.GetPixels();
 
         // go through all colors in the texture
         for(int i = 0; i < colors.Length; i++)
         {
             if(colors[i].r == 1) continue;  // dont touch colors with full red channel value
 
             if(((Color32)colors[i]).r == hole.color.r)
             {
                 // lerp alpha value of colors with matching red channel value
                 float newAlpha = Mathf.Lerp(hole.color.a, targetColors[i].a, hole.timer / HOLES_MAX_TIME);
                 colors[i] = colors[i].WithA(newAlpha);
 
                 int y = i / currentTexture.width;
                 int x = i - (y * currentTexture.width);
                 currentTexture.SetPixel(x, y, colors[i]);
             }
         }
 
         currentTexture.Apply();
     }
 }
 
 public static class TextureExtensions
 {
     public static Texture2D DrawCircle(this Texture2D tex, Color32 color, int x, int y, int radius = 3)
     {
         float rSquared = radius * radius;
 
         for (int u = x - radius; u < x + radius + 1; u++)
             for (int v = y - radius; v < y + radius + 1; v++)
                 if ((x - u) * (x - u) + (y - v) * (y - v) < rSquared)
                     tex.SetPixel(u, v, color);
 
         return tex;
     }
 }

I know I'm using GetPixels and SetPixels a lot, and that is probably what is draining all the resources from the CPU and giving me such bad performance in bigger textures, but I'm at a bit of a lost right now in terms of how should I approach optimizing this algorithm. And this is my main question really, what method can I use to regenerate (lerp alpha value) every individual hole at its own pace in a performant way? Every suggestion or hint on a potential approach is appreciated, thank you.