Voxel Raycasting Help

Hi all.

I have a voxel world, made up of cubes (you know, like Minecraft - except my game isn't like Minecraft. But it's voxel). And before, I was using mesh colliders for each chunk of 16x64x16 blocks. As you can imagine, mesh colliders are not ideal and were causing some hiccuping upon chunk loading. So I dumped the mesh colliders and decided to use box colliders that checked blocks around the player, and, if they were solid blocks that can be collided with, they would be enabled there.

This performs well. But it comes with a problem. Before, to place and delete blocks, I would cast a ray at the ground, get the hit point, use the normal of it to figure out where to place the block or to figure out which block to delete.

Well, now I don't have mesh colliders anymore and so there aren't colliders to hit with rays. I posted another question earlier about collider performance, but I don't think it's feasible to go with the collider approach. I think I need to use a custom raycast.

I wrote one that's really simple. Here it is (with a few details omitted for simplicity):

 public static bool Raycast(Ray ray, int distance, bool adding, out VoxelCoords coords)
 {
     float dist = 0.0f;
     Vector3 point;
     
     for (int i = 0; i < distance; i++)
     {
         dist += 1.0f;
         point = ray.GetPoint(dist);
         VoxelCoords tempCoords = new VoxelCoords(point);
 
         if (ReadBlock(tempCoords).type != Blocks.air)
         {
             coords = adding ? new VoxelCoords(ray.GetPoint(dist - 1.0f)) : tempCoords;
             return true;
         }
     }
     
     coords = new VoxelCoords(false);
     return false;
 }

You give it a ray, a distance, whether you're adding a block or removing a block, and some Voxel coordinates (which are just Vector3s but rounded to a block's center, for reading voxel data).

The ray I pass in is a ray obtained from using ScreenPointToRay from Input.mousePosition.

I figured I would just advance along the ray 1 unit at a time and use GetPoint to figure out the point there. Then I would round it to voxel coordinates, figure out if air is there. If there isn't air, then we hit something. Then if adding is true, I return 1 unit back on the ray, otherwise I return the coordinates hit.

So. This fails. It generally works, but it's not accurate. The blocks modified aren't quite where I click, they may be off and affect another block to the side of the one I'm trying to modify.

One issue I can see with it is that when I use the ray.GetPoint(dist - 1.0f), it's not acting like a normal that's perpendicular to the block face. I can probably use the ray to figure out the direction properly to fix that.

The part I don't understand is why, even when I'm deleting the blocks (adding = false), it's still not accurate. It just returns the coordinates of the block that was hit. But it seems to be hitting the wrong one at times.

So I think it's not as simple as I'm trying to make it here. My question, then, is I'm wondering if you guys have any advice or pointers to help me get this working better.

Thanks!