Player getting stuck on collider/walls

I have a capsule for a character and have attached a character controller via C# script. Attached to the character is a rigidbody and a capsule collider. My terrain which is just flat planes has a box collider attached at a height of 10. I am using a player script that moves back and forward one step at a time and stops and when you hit right or left they turn 90 degrees and stop. I can move around on the field with no issues move next to the colliders/walls themselves and even stop in front of them and still turn and keep moving forward. However when I am facing a wall and I hit forward into the collider my player/capsule attempts to move as close as the colliders allow (almost touching each other) and gets stuck, I am noticing on my player/capsule 3d object the X rotation starts going crazy (increasing in the positive by 0.00001 amounts, when it hits this point and stops allowing me to move whatsoever in any direction I try. I have tried adding a frictionless element to both the collider/wall and player/capsule, attempted to check is kinematic (which then makes the player go through the colliders like they weren't even there) and have attempted everything I have found in the unity community and google searches. I am totally at a loss now and am unsure what I am doing wrong. I have included my controller script for reference if that helps any. I apologize for the long write-up here but I am trying to be as specific as possible. Thanks in advance.

Player Controller Script:

 public class PlayerController : MonoBehaviour
 {
 
     Vector3 pos;                                // For movement
     float speed = 5.0f;                         // Speed of movement
 
     void Start()
     {
         pos = transform.position;          // Take the initial position
     }
 
     void FixedUpdate()
     {
         if (Input.GetKeyDown(KeyCode.A) && transform.position == pos)
         {        // Left
             transform.Rotate(0, -90, 0);
             
         }
         if (Input.GetKeyDown(KeyCode.D) && transform.position == pos)
         {        // Right
             transform.Rotate(0,90,0);
             
         }
         if (Input.GetKeyDown(KeyCode.W) && transform.position == pos)
         {        // Up
             pos += transform.forward;
         }
         if (Input.GetKeyDown(KeyCode.S) && transform.position == pos)
         {        // Down
             pos += transform.forward * - 1;
             
         }
         transform.position = Vector3.MoveTowards(transform.position, pos, Time.deltaTime * speed);    // Move there
     }
 
 }