Character Moving in Elliptical Pattern

When I try moving using a rigidbody character going the character moves to the right before coming back to the center when it stops and the the inverse going backward. This pattern continues when facing in different directions and moving in different directions.

This doesnt happen if I disable the rotation of the camera affecting the player but I cant figure out why this is happening. The camera follows the player object and the player uses the camera's rotation to know which direction to move in. Usually I would keep the camera as a child object but I want to have airwalking disabled for this game.

Below is the update and fixedupdate from my player controller. Any help in fixing this would be much appreciated.

 void Update() {
         Vector2 playerInput;
         playerInput.x = Input.GetAxis("Horizontal");
         playerInput.y = Input.GetAxis("Vertical");
         playerInput = Vector2.ClampMagnitude(playerInput, 1f);
 
         Vector3 forward = playerInputSpace.forward;
         forward.y = 0f;
         forward.Normalize();
         Vector3 right = playerInputSpace.right;
         right.y = 0f;
         right.Normalize();
 
         desiredVelocity = (forward * playerInput.y + right * playerInput.x) * maxSpeed;
 
         desiredJump |= Input.GetButtonDown("Jump");
     }
 
     public void FixedUpdate() {
         velocity = body.velocity;
         float maxSpeedChange = maxAcceleration * Time.deltaTime;
         velocity.x = Mathf.MoveTowards(velocity.x, desiredVelocity.x, maxSpeedChange);
         velocity.z = Mathf.MoveTowards(velocity.z, desiredVelocity.z, maxSpeedChange);
 
         if(desiredJump) {
             desiredJump = false;
             Jump();
         }
 
         body.velocity = velocity;
 
         onGround = false;
     }

Update: I tried attaching the camera to the player and seeing if that affected the problem at all but it seems to be happening as before. I also noticed that the problem seems to not occur when my y rotation is any multiple of 45. Still unclear if that means that it is happening without my camera being involved after all but I'm not seeing it because it is always moving on an non-offending axis or it's just because those are vectors where only one axis is active or the two axes are matching which offset. Still more investigation to do. Any help would be great.