IsGrounded flickers when hitting a wall

I have a CharacterController with a StepOffset of 0.3, and a 3x1x3 box. On my FixedUpdate, I call my move script:

 Vector3 velocity;
 Vector3 direction;
 direction = new Vector3(Input.GetAxis("Horizontal"), 0, Input.GetAxis("Vertical"));
 switch (_isSprinting) {
 case true:
   velocity = direction * _sprintSpeed;
 
   break;
 case false:
   velocity = direction * _speed;
 
   break;
 }
 if (_controller.isGrounded) {
   _vSpeed = 0;
 }
 
 if (_controller.isGrounded) {
 
   if (Input.GetKeyDown(KeyCode.Space)) {
     _vSpeed = _jumpVelocity;
   }
 }
 _vSpeed -= _gravity * Time.deltaTime;
 velocity.y = _vSpeed;
 velocity = transform.TransformDirection(velocity);
 _controller.Move(velocity * Time.deltaTime);

The issue I'm having is when I walk towards the 1u-high block, instead of just stopping, my character "crawls" closer to the block. If I debug isGrounded I can see it flickers on and off as the character crawls forward. If I set stepOffset = 0 it works flawlessly, however, my character can no longer use stairs. If I set it to 0.5, it kind of works, but when I release the "forward" key, it yanks a bit ( isGrounded flickers once). Here's an example: https://streamable.com/ygy025

This happens both with ProBuilder's box as well as Unity's default box primitive.

These are the controller values: