Two diffrent directions on normal when colliding with one flat surface?

Hi. I am making a rigidbody controller script were I project user given input for movement onto the surfaces the controllers CapsuleCollider is colliding with. Rotation on the x and z axis is frozen on the capsule.

A problem I ran into is that on some occasions when i slide along a wall there is a stutter and I've traced the problem to the normals on the collision point actually alternating quickly when my capsule slides along the surface.

What could be the cause of this and how do I avoid it? The only thing i can think of is the direction of the movement on my capsule playing into it but why would it? Should this even be possible?

An image displaying normals in magenta and horizontal direction orthogonal to the normal in green. The origin of the "pairs" are close but not equal.

Code that draws the normals:

 using UnityEngine;
 using System.Collections;
 
 public class CollisionNormalTest : MonoBehaviour {
     private ContactPoint c;
     FixedUpdate(){
        Debug.DrawRay(c.point,c.normal*5.0f,Color.magenta,2.5f);
     }
     void OnCollisionStay (Collision col)
     {
         foreach (ContactPoint c in col.contacts){
             this.c = c;
             }
     }
 }