Charactercontroller jittering while jumping into wall

I've got an entity with a CharacterController component. Everything is set to default aside from the capsule positioning. I've also added a separate ground check (using Physics.CheckSphere) and I've gotten my character to move around and jump. This is the code executing in character's movement FixedUpdate script

  void FixedUpdate()
     {        
         if (velocity.x != 0 || velocity.z != 0)
         {            
             transform.Rotate(Vector3.up * rotationAngle);
                         
             Vector3 forwardMovement = transform.forward * velocity.z;
             Vector3 rightMovement = transform.right * velocity.x;            
 
             Vector3 movement = forwardMovement + rightMovement;
 
             controller.Move(movement * Time.deltaTime);            
         }
 
         if (groundCheck.IsGrounded && velocity.y < 0f)
         {
             velocity.y = -0.2f;
         }
 
         velocity.y += Physics.gravity.y * Time.deltaTime;
         controller.Move(Vector3.up * velocity.y * Time.deltaTime);
     }

And this is the code executed on keypresses (coded as actions run within a Update)

 private void OnMovementAxes(Vector2 axes)
     {        
         if (axes == Vector2.zero)
         {
             velocity.x = 0f;
             velocity.z = 0f;
             return;
         }
 
         Vector2 newVelocity = axes.normalized * movementSpeed;
         velocity.x = newVelocity.x;
         velocity.z = newVelocity.y;
     }
 
     private void OnJumpPressed()
     {        
         if (controller.isGrounded)
         {
             velocity.y = Mathf.Sqrt(jumpHeight * -2f * Physics.gravity.y);
         }        
     }    
 
     private void OnCameraPosition(Vector3 cameraPosition, Vector3 cameraDirection)
     {        
         Debug.DrawRay(cameraPosition, cameraDirection * 10, Color.red);
         Vector3 movementDirection = Vector3.Scale(cameraDirection, Vector3.right + Vector3.forward);
         Debug.DrawRay(cameraPosition, movementDirection * 10, Color.blue);
 
         rotationAngle = Vector3.Angle(transform.forward, movementDirection);
 
         float crossProduct = Vector3.Cross(transform.forward, movementDirection).y;
 
         if (crossProduct < 0)
         {
             rotationAngle *= -1;
         }
     }

When my character reaches a wall that they should be able to jump onto (and they can if they jump from a bit further back), and tries jumping while holding forward movement, the character does lift but not even half the jump height it usually does, and it just jitters.

First I assumed that the GroundCheck sphere is too large so that the front of it is touching the wall and registering as "grounded" when colliding with the wall so I made it small enough that it's well enough behind the main capsule collider. Didn't help

Then I played with Skin WIdth property. Tried setting it from 0.08 to 0.1, 0.5 and even 1 (which was interesting to see) but that didn't help either.

Then I tried changing Slope Limit property from 45 to 90 and that seems to have fixed the issue but I'm not sure why that is. I don't quite know what "Slope Limit" actually does. Does it specify how steep a slope the character can "climb" up? Also that solution seems like a bit of a hack.

Is there a better solution to prevent this jitter when trying to jump while running into a wall?