Masking part of an object via raycasts

If you are attempting to mask part of an object, there are a few ways.

For what you describe, the most robust way would be to write custom code in the fragment program of a pixel shader, which sets certain pixels to be entirely transparent. You can pass the rays into the shader as vectors, and do the math entirely on the GPU. Unity provides the world to object matrices for doing the math. You'd probably want to alpha test, so that depth sorting works as expected. Of course, doing this to a lot of objects will dramatically increase your fill rate, since they are treated as transparent - could be a concern on mobile.

If you are in a space where you can guarantee render order (such as UI), you could render multiple cameras, and create 'mask' objects, which write to the depth buffer, but have ColorMask 0, so they write nothing to the color or alpha buffers. This would enable simple screen space masking.