RaycastHit.normal - what does it really return?

I'm testing for proximity of other objects by raycasting a sphere, considerably larger than the object itself (also a sphere). I then examine the array of RaycastHit structs, using the RaycastHit.normal vector to push away from the other object in proportion to the distance. So far, so good.

However, the normal returned isn't perpendicular to the surface of the object. In the case of a wall created from a cube, I would have expected the normal to point 90 from the surface, but it doesn't. In the following screen caps, the blue line is the vector returned by RaycastHit.normal:

The white line is the original direction of the sphere; the red line is the ray from the centre of the sphere to the collision point, for which the normal supposedly is computed. The documentation says the normal is "the normal of the surface of the ray hit". As you can see, the angle is not 90 against the surface.

Here's the same scene from a slightly different perspective:

As you can see, the normal points downwards, which drives the sphere, which will follow the yellow line, down into the ground. Not so good.

The plane beneath the other objects gives different results:

In this case, the normal is always along the hit ray.

When moving the sphere around, I can also sometimes see how the hit point snaps to one of the vertices of the wall and stays there; sometimes the hit point is clearly not the closest point. In the case of the plane, the hit point always seems to be along a triangle vertex.

Am I misunderstanding how normals and/or colliders work here? (I should add that I'm not using physics to move the sphere, but translation. As the next step, I'll implement the same type of steering using rigidbodies and physics; I want to compare the pros and cons of the two different approaches.)

I'm using Unity Pro 3.3. Here's the code attached to the sphere:

var target : Transform; var speed : float = 1.0; var comfortDist : float = 10.0;

 var pos : Vector3 = transform.position;
 var dist : float = Time.deltaTime * speed;
 var ray : Ray = new Ray(pos, target.position - pos);
 var dir : Vector3 = transform.TransformDirection(Vector3.forward);
 Debug.DrawRay(pos, dir * 10);

 var hits : RaycastHit[] = Physics.SphereCastAll(ray, comfortDist, dist);
 for (var h in hits) {
     if (h.transform != target && 
         h.collider != this.collider &&
         h.transform != skip) { 
         var realDist : float = Vector3.Distance(pos, h.point) - r;

         Debug.DrawLine(pos, h.point, Color.red);
         Debug.DrawRay(h.point, h.normal, Color.blue);

         var repulsion = (repulsionExtent - realDist) / repulsionExtent;

         var currentForward : Vector3 = transform.TransformDirection(Vector3.forward);
         var newdir : Vector3 = Vector3.Slerp(currentForward, h.normal, repulsion);
         transform.rotation = Quaternion.FromToRotation(currentForward, newdir) * transform.rotation;
     };
 };

 var resultdir : Vector3 = transform.TransformDirection(Vector3.forward);
 Debug.DrawRay(pos, resultdir * 10, Color.yellow);

 // Move forward
 if (move) transform.Translate(Vector3.forward * dist);

}

P.S.: If you put in more than one seeker sphere, you'll notice that they behave correctly in respect to each other, which seems to have to do with the fact that the sphere collision normals behave as expected. Or rather, as I would expect them to... ;-)