Why are raycasts so unreliable?

Yes, raycasts are tricky -- the coding is more difficult, and lots of special rules. But they are reliable, once you figure them out.

You can't see them, ever. And if you use Debug.DrawRay, you can easily show the wrong line, making it even worse.

Returning true/false and also the RayCastHit by reference is a standard trick, but not many people are used to it. The 9 overloads are odd, and it's easy to enter distance where the layer mask should have been. Layermasks are in binary, but layers aren't. Sometimes a raycast is "good" if it hits the right thing, or if it misses everything... .

But treat them as something tricky, and go slow, set up simple tests and print a lot. You can find many, many examples here, of how layer masks work, etc.. . Really, there are dozens of "firing a bullet using a raycast" threads.

I feel your pain with RayCasts haha. My game relies entirely on using a ton of them from the player movement commands, selecting objects, firing laser beams, etc.

It is a LOT of work to set up a complicated game relying on so many RayCasts instead of simple collisions and more traditional player movement controls.

For just ignoring one layer and being able to hit others is very simple though once you know how to do it. Once you have a ton of different layer interactions that's when it becomes so tedious to make everything work perfectly.

Here's a little example of how to do what you want if I'm understanding your question correctly.

This example might be a little excessive (meaning some parts may be unnecessary I'd have to test to know for sure) but it works, some of the stuff I'm doing with Layer Masks seems strange but this is what I learned about it so far and it works at least.

 //start position of Rayshot
 public Transform rayStartPosition;
 
 //ray shot length
 float maxRayLength = 50;
 
 //variables to store the rayHit info
 GameObject rayHitObject;
 String rayStringName;
 Vector rayVectorHitPosition;
 
 //declare LayerMask variables for player
 LayerMask playerLayerMask;
 int playerLayerMaskInt;
 
 void Start()
 {
 //this is what seems weird but it works
 //The "8" is the "User Layer" when you
 //click on the "Add Layer" option in Inspector
 playerLayerMaskInt = ~(1<<8); 
 playerLayerMask = playerLayerMaskInt;
 }
 
 //casting the ray example
 RaycastHit rayHit;
 
 if (Physics.Raycast(rayStartPosition.position, rayStartPosition.forward, out rayHit, maxRayLength, playerLayerMask))
 {
 //get the gameObject
 rayHitObject = rayHit.collider.gameObject;
 
 //get the name of object
 rayStringName = rayHitObject.name;
 
 //get the position of the rayhit
 rayVectorHitPosition = rayHit.collider.gameObject.transform.position;
 }
 
 

You can add more layers to ignore by declaring them like this example and simply adding a & between each LayerMask variable so it ignores every layer you want. Example:

 if (Physics.Raycast(rayStartPosition.position, rayStartPosition.forward, out rayHit, maxRayLength, playerLayerMask & ignoreRayLayerMask))

So if you do it like that it's actually pretty simple for what you are trying to do, but it is strange to learn it, and setting up an entire game around raycasts is very tedious for something like a Diablo 2 style of gameplay where everything is controlled by mouse clicks, or mobile screen taps.