How to reduce iterations for each pixel?

I am building a game in Unity and I have a RawImage with the dimensions 1024 x 1024 and it doesn't have a texture initially. I build one at runtime. At the moment, I have a button that when pressed generates a new texture and sets it to the RawImage. Basically, I generate a random array of Vector2 that contains 4 random positions within the dimensions of the RawImage. I'm facing long processing times because I need to set each pixel of the RawImage to my desired background color and all of this for just 4 positions. In other words, I do 1024 x 1024 = 1048576 iterations when I would just need 4. Here is a graphical representation as an example:

*In this example I used a resolution of 40x40 in order to make the points more visible.

Here is the code that I use at the moment:

 public RawImage myImage;
 
     public void GenerateButtonPressed()
     {
         Texture2D topTexture = new Texture2D(40, 40);
         Vector2[] randomValues = new Vector2[4];
         Color pointColor = Color.black;
         Color backgroundColor = Color.cyan;
 
         for(int i=0; i < randomValues.Length; i++)
         {
             randomValues[i] = new Vector2(Random.Range(0, 40), Random.Range(0, 40));
         }
 
         int randomCounter = 0;
         for(int i = 0; i < topTexture.width; i++)
         {
             for(int j = 0; j < topTexture.width; j++)
             {
                 if(i == randomValues[randomCounter].x && j == randomValues[randomCounter].y)
                 {
                     topTexture.SetPixel(i, j, pointColor);
                     randomCounter++;
                 }
                 else
                 {
                     topTexture.SetPixel(i, j, backgroundColor);
                 }
                 
             }
         }
         topTexture.Apply();
         myImage.texture = topTexture;
     }

As you can see, I do 2 nested for loops with a pretty big index size just to set 4 points. Is there a way to reset the background without doing all those iterations and to set just the 4 points when the button is presed? Any amount of help would be appreciated because this improvement for sure will make my game more enjoyable. Thanks in advance!