How can I get accurate collision contacts?

I'm trying to make a character controller based on a 3D box collider, but I'm not getting accurate collision data. Basically my strategy at the moment is to detect collision points and raycast down to check if the character is on the ground. This sort of works, but the puzzling nature of collision contacts is causing it to break in certain situations, particularly as seen here:

It "works" in that it detects (roughly) where the two objects are colliding, but the raycast straight down is clearly showing that it's not where it needs to be.

What can I do to get this to work properly?

Here is the relevant code:

     void OnCollisionStay(Collision collision)
     {
         contactPoints = collision.contacts;
     }

And in Update():

         foreach (ContactPoint point in contactPoints)
         {
             Debug.DrawRay(point.point + Vector3.up * .1f, Vector3.down, Color.red);
             Debug.Log(contactPoints.Length.ToString());
             if (Physics.Raycast(point.point + Vector3.up * .1f, Vector3.down, GroundCheckDistance, 1 << 10)) // 10 is the floor layer ID
             {
                 grounded = true;
                 break;
             }
             else
             {
                 grounded = false;
             }
         }