How to prevent steep wall climbing?

I had the same exact problem in my game. Another user shared this code designed for allowing characters to slip down hills. I modified it and it solved my problem perfectly. Note that speed, slopeLimit, and moveDir are in the default character controller script.

 private Vector3 hitNormal;
 bool m_IsGrounded;
 private float slopeLimit;

 private void Start()
 {
      slopeLimit = m_CharacterController.slopeLimit;
 }
 
 private void Update()
 {
     if (!m_IsGrounded)
     {
         m_MoveDir.x += (1f - hitNormal.y) * hitNormal.x * (speed * 1.2f);
         m_MoveDir.z += (1f - hitNormal.y) * hitNormal.z * (speed * 1.2f);
     }
 
     if (!(Vector3.Angle(Vector3.up, hitNormal) <= slopeLimit))
     {
         m_IsGrounded = false;
     }
     else
     {
         m_IsGrounded = m_CharacterController.isGrounded;
     }
 }
 
 private void OnControllerColliderHit(ControllerColliderHit hit)
 {
        hitNormal = hit.normal;
 }

You'll have to implement some kind of slip code on slopes to allow the controller to move downward away from the wall. Otherwise, the basic controller will think it is grounded and use your normal movement code.

A good place to start is in detecting the normal of the point you hit at and using vector math to determine the appropriate "down" direction and mark that the player is no longer 'grounded' or whatever your controller requires before you can jump if the values of the normals are too high. The normal is perpendicular to the face of the collider, so if the Y-value of the normal is too low, your character is on a slope.