Raycast on sprite in 3D space, how to find texture coordinates

Hi there, I ran into a problem that just makes my head melt. I got a sprite with a BoxCollider, they have the same width and height. They can be positioned anywhere in 3D space, but always face the player. My plan is to Raycast towards the sprite from the player, and paint a pixel at the point of impact. Now normally I could use RaycastHit.textureCoords for something like this, but I didn't got it to work with a sprite and a BoxCollider.

The approach I want to try is to just get the point of impact on the BoxCollider and then calculate the coordinates of the pixel that is below this point of impact.

A quick visualization of this, I hope this is helpful. First a raycast hits (the upper edge in this case, for visualization purposes) and then the pixel on the sprite is calculated.

But I can't seem to figure out how to do that. I tested a few (well, more than a few) things, and I think I need to watch out for the pivot point, width, height and the pixel to unit conversion of the sprite. The object can also be oriented just about anywhere in space, the only sure thing is that it looks at the player.

Here is just a quick sample of my code, I included two of my failed attempts. Both of them sometimes result in colored pixels, but not always.

 //C#
 SpriteRenderer renderer = hit.collider.gameObject.GetComponent<SpriteRenderer>();
 Texture2D tex = renderer.sprite.texture;
 
 //First attempt: in local space
 //Resulting pixels are just off by a pretty large amount
 Vector3 localPos = hit.transform.InverseTransformPoint (hit.point);
 Vector2 texturePos = new Vector2(localPos.x, localPos.y);
 texturePos.x = texturePos.x * tex.width;
 texturePos.y = texturePos.y * tex.height;
 
 //Second attempt: in world space
 //Pixels are off and rotation seems to be a problem
 Vector2 texturePos = new Vector2();
 texturePos.x = ((hit.point.x - hit.collider.bounds.min.x) / hit.collider.bounds.size.x)*tex.width;
 texturePos.y = ((hit.point.y - hit.collider.bounds.min.y) / hit.collider.bounds.size.y)*tex.height;

texturePos is supposed to hold the coordinates at the end. I would be really grateful if someone could help me with this, because this is a pretty crucial part of what I'm building.

(PS: If you can think of a better solution than this projection thing, then I would be happy too. I just need the right pixel coordinates.)