CapsuleCast is detecting the wrong hit normal when colliding with two surfaces

Hey,

So I am using CapsuleCastAll for ground detection in my game. Because I'm using a rigidbody, I have to handle slopes myself, so I set my forward direction based on the cross vector (Vector3.cross) of transform.right and the hit.normal.

This works however I ran into an issue.

Here is how to forward vector looks on a sloped surface represented by the blue line, when on a flat surface, it is at a 90 degree angle:

My issue is, when standing on the corner of a sloped surface moving down (Moving up a sloped surface works fine for some reason) the player freezes for a few seconds and the forward vector is still reacting as if I'm on the slope:

I imagine that this is because the back of the capsule cast if hitting the sloped surface and still calculating the forward vector based on that normal.

Here is the code for my CapsuleCast:

 bool isCastGrounded()
     {
         float dist = col.height / 2 - col.radius;
 
         Vector3 p1 = transform.position + col.center + Vector3.up * dist;
         Vector3 p2 = transform.position + col.center - Vector3.up * dist;
 
         float radius = col.radius;
 
         RaycastHit[] hits = Physics.CapsuleCastAll(p1, p2, radius, -Vector3.up, 0.25f, ground);
 
         foreach(RaycastHit hit in hits)
         {
             hitInfo = hit;
             
             return true;
         }
         return false;
     }

And here is my CalculateForwardVector code:

 void CalculateForwardVector(RaycastHit hitInfo)
     {
         if (!isCastGrounded())
         {
             forwardDir = transform.forward;
             return;
         }
 
         forwardDir = Vector3.Cross(transform.right, hitInfo.normal);
     }

So I guess my question is how would I go about determining if my hit normal is on a corner and fixing it appropriately (or if anyone has a better solution i'm all ears)

Thanks!