Why does my vector pan my character slightly to the side?

I'm modifying the default FirstPersonController script to have a jetpack, and I've got it mostly down. However, there's an annoying bug where if I fly at an angle to the x or z axes it'll make me strafe slightly to the side.

I think it's because I clamp the x and z distances. Please check my code and tell me what's wrong.

I did notice that when I set the FlyModifier to a very small number like 0.1, I did stop the strafing, but it took forever to start building momentum.

     // always move along the camera forward as it is the direction that it being aimed at
     Vector3 desiredMove = transform.forward*m_Input.y + transform.right*m_Input.x;

     // get a normal for the surface that is being touched to move along it
     RaycastHit hitInfo;
     Physics.SphereCast(transform.position, m_CharacterController.radius, Vector3.down, out hitInfo,
                        m_CharacterController.height/2f);
     desiredMove = Vector3.ProjectOnPlane(desiredMove, hitInfo.normal).normalized;

     //If just a regular jump, do regular movement, otherwise add momentum
     if (!m_Flying) {          
         
         //On the ground
         if (m_CharacterController.isGrounded) {
             m_MoveDir.x = desiredMove.x*speed;
             m_MoveDir.z = desiredMove.z*speed;    
         } else { //Falling through the air
             m_MoveDir.x += desiredMove.x*speed*m_flyModifier*Time.deltaTime;
             m_MoveDir.z += desiredMove.z*speed*m_flyModifier*Time.deltaTime;
     
             m_MoveDir.x = Vector2.ClampMagnitude(new Vector2(m_MoveDir.x, m_MoveDir.z), m_RunSpeed).x;
             m_MoveDir.z = Vector2.ClampMagnitude(new Vector2(m_MoveDir.x, m_MoveDir.z), m_RunSpeed).y;             
         }
     //Jetpacking                              
     } else {
         
         m_MoveDir.x += desiredMove.x*speed*m_flyModifier*Time.deltaTime;
         m_MoveDir.z += desiredMove.z*speed*m_flyModifier*Time.deltaTime;
         
         m_MoveDir.x = Vector2.ClampMagnitude(new Vector2(m_MoveDir.x, m_MoveDir.z), m_RunSpeed).x;
         m_MoveDir.z = Vector2.ClampMagnitude(new Vector2(m_MoveDir.x, m_MoveDir.z), m_RunSpeed).y;

     }