SetPixels32 On A Sprite In Texture2D Atlas

So basically I'm trying to take a mouse click on an individual sprite and change the color of those pixels that were clicked within a predetermined area on the Texture2D atlas. The texture is set to read/write, it is an ARGB32 texture, and is not tightly packed.

I have been able to SetPixels32 on the entire textureRect of the sprite in the atlas and I have also been able to SetPixels32 on the center point of the sprite in the atlas however I cannot seem to reliably translate a RaycastHit2D.point to the corresponding point in the textureRect of the sprite within the atlas.

I've translated the world space hit to local space but that hasn't worked. I've tried to multiply the world space hit by the pixelsPerUnit of the sprite and that hasn't worked either.

Can anyone explain to me what the calculation would be to achieve this translation?