What am I doing wrong in my distance calculation?

Hi everyone,
For a little try-out project, I want to make a simple building simulator.
Now, I have to building in place, but for some reason, the size of the wall is not being calculated properly.
The length is being calculated by having a begin position and an end position which is a raycasted world mouse position. In some occasions it calculates like it should, but other times, it just doesn't work at all and has a weird size. In both cases, it is not as accurate as I would like to the mouse position. (See begin of gif)

I have some example footage here:
I suspect that it has something to do with the distance calculation. I have tried Vector3.distance, but this gave the same problem for some reason.
The distance is being calculated in the Input.getKey(KeyCode.Mouse0) under the name of zScale.

 void BuildWall(Vector3 hit)
     {
         if (isPlacing == false)
         {
             isPlacing = true;
             beginBuildPosition = new Vector3(hit.x, 0, hit.z);
             buildBlock = GameObject.CreatePrimitive(PrimitiveType.Cube);
             buildBlock.GetComponent<Collider>().enabled = false;
             rotationBox = GameObject.CreatePrimitive(PrimitiveType.Sphere);
             rotationBox.GetComponent<Collider>().enabled = false;
             rotationBox.transform.localScale = new Vector3(1f, 1f, 1f);
             rotationBox.name = "RotationBox";
             rotationBox.transform.position = beginBuildPosition + new Vector3(0, 1, 0);
             rotationBox.GetComponent<Renderer>().material = rotatorMaterial;
             buildBlock.transform.parent = rotationBox.transform;
             buildBlock.transform.localPosition = new Vector3(0, 1, 0);
             buildBlock.transform.localEulerAngles = new Vector3(0, 90, 0);
             buildBlock.transform.localScale = new Vector3(0.5f, 2, 0.25f);
             buildBlock.name = "Wall";
             buildBlock.GetComponent<Renderer>().material = placementMaterial;
         }
 
         if (Input.GetKeyUp(KeyCode.Mouse0))
         {
             buildBlock.GetComponent<Renderer>().material = wallMaterial;
             buildBlock.GetComponent<Collider>().enabled = true;
             buildBlock.transform.parent = null;
             Destroy(rotationBox);
             isPlacing = false;
         }
 
         if (Input.GetKey(KeyCode.Mouse0))
         {
             float zScale = (beginBuildPosition - hit).magnitude;
             buildBlock.transform.localPosition = new Vector3(0, 1, zScale / 2);
             buildBlock.transform.localScale = new Vector3(zScale, 2, 0.25f);
             rotationBox.transform.LookAt(new Vector3(hit.x, rotationBox.transform.position.y, hit.z));
         }
         if (isPlacing && !Input.GetKey(KeyCode.Mouse0))
         {
             rotationBox.transform.position = hit;
         }
     }

Does anyone know what I am doing wrong?
Thanks upfront,
Jenoah