RotateTowards won't rotate in X axis

I got a simple function for rotating an object at a certain speed, looking at a target object:

     private void RotateTowards(Transform target, bool verticalRotate = true)
     {
         //Target direction to rotate towards
         Vector3 direction = (target.position - transform.position); //.normalized
 
         //Quaternion to store the converted target direction
         Quaternion lookRotation;
 
         //If allowing vertical rotation, use the entire direction
         if (verticalRotate)
         {
             lookRotation = Quaternion.LookRotation(direction);
         }
         //If not allowing vertical rotation, set the Y axis as the same as the current transform, so that it's always looking straight forward
         else
         {
             lookRotation = Quaternion.LookRotation(new Vector3(direction.x, transform.position.y, direction.z));
         }
 
         //Rotate towards the generated lookRotation
         transform.rotation = Quaternion.RotateTowards(transform.rotation, lookRotation, Time.deltaTime * agent.angularSpeed);
     }

It's called every frame, gets the direction to rotate towards, converts it to a target quaternion rotation, and then uses RotateTowards to smoothly rotate in that direction.

However, while this will rotate the object fine horizontally, it won't look up or down. The generated lookRotation is correct, so something's not working with the RotateTowards function.

The object in question has a NavMeshAgent attached - could that somehow be interfering with the rotation? I do disable navmesh rotation when I call the function, as such:

         //Do not use navigation-based rotation
         agent.updateRotation = false;
 
         //Rotate to face the player
         RotateTowards(player.transform, aimVertically);

So presumably that wouldn't have impact, but that's all I can think of...

Thankful for any help figuring this out!