Jitter on Slerp - Raycast

Hello, I have an AI that is checking for walls in front of it using a CapsuleCast and then inside the CapsuleCast check, it's checking for walls on either side of it using a set distance of a Raycast so to determine the best way to rotate and lastly, rotate a set way if no side walls to detect. Now this works, more or less, still have some bugs to fix and changes to make so that it'll rotate in different environments, but for the life of me the rotation with Slerp/RotateTowards is jittery when it reaches the peak of the rotation until the rotation has ended. I've went through a lot of other posts but all of the suggestions work the same and have the same issue. I've spent days on and off this part of the AI but I don't understand why it would jitter. Any thoughts? Thanks.

Here's the code that's doing the checking and rotating.

  //check for anything in front
         if (Physics.CapsuleCast(point1, point2, controller.radius, transform.forward, out hitInfo, capsuleWallDetectDistance, ignoreEnemy))
         {
             //check for walls on the right then turn to the left
             if (Physics.Raycast(transform.position, transform.right, detectWallDistance, ignoreEnemy))
             {
                 direction = -transform.right;
             }
             //check for walls on the left then turn to the right
             else if (Physics.Raycast(transform.position, -transform.right, detectWallDistance, ignoreEnemy))
             {
                 direction = transform.right;
             }
             //otherwise if no walls are on either side just turn to the right.
             else
             {
                 direction = transform.right;
             }
     transform.rotation = Quaternion.Slerp(transform.rotation, Quaternion.LookRotation(direction), 2 * Time.deltaTime);
     }