RaycastAll failing inconsistently

Hey everyone, I've been using raycasts just like this with no problem at all, but now they are failing. We are making an RTS and the camera and unit movement works great, but at a certain point in each battle, the raycasts just start failing on random frames. Once it has happened, it does not stop until we stop the game. Sometimes it happens at low framerate times, but other times it just happens near the beginning of the game. Anyway, here is one place I am doing a raycast, it would be nice if someone could tell me I'm doing something wrong, or that they've experienced it before.

RaycastHit[] hitInfo = Physics.RaycastAll(new Vector3(x, maxHeight, z), -Vector3.up, float.MaxValue, TerrainMask);
float highest = WaterLevel;
foreach (RaycastHit hit in hitInfo)
    if (hit.point.y > highest)
        highest = hit.point.y;

Basically, when the raycast fails, it returns WaterLevel. Before I didn't notice this problem, because it was setting the y value to NaN, so the Transform SetPosition failed, disallowing the new position. But now, since I added WaterLevel, it is more visible, and the units and camera jump up and down from the terrain to the water at random.

There is another place I am doing a raycast, and that raycast fails during the same frames as the other. Looks like this:

Ray ray = Camera.main.ScreenPointToRay(Input.mousePosition);
RaycastHit[] hitInfos = Physics.RaycastAll(ray, 3000, SelectionMask | TerrainMask);

However, I realize I am making another raycast that is never failing. This raycast is the only one that goes vertical, and is unlikely to hit the terrain. So it seems that any raycast that hits the terrain, causes the whole raycast to fail, but only every once and a while.

For my workaround, I am going to detect failure and return NaN again, since that seemed to give better results, but a real fix would be great!

Thanks everyone.