TransformDirection causing move Jitter

So here is the deal, I am moving the player with different control schemes while on the ground and in the air. On the ground the player's velocity (using a Rigibody BTW) is simply being set to the desired velocity, while in the air I slowly increase/decrease the values of the vector i am moving across so the player has less control in the air.

I am then making the movement relative to the player's forward direction

What I get is that it works perfectly while the player is facing forward with the world. The moment that changes, his ground movement is still working as intended but the air movement makes him jitter and not move in the desired direction at all.

When I take the transformdirection line out, it works but the character's movement is relative to the world.

I imagine it is because of how I am changing the vector while in the air. It is the only thing that I can think of at least.

     //Movement
     public float speed = 10f;
     public float airSpeed = 2.5f;
     Vector3 moveDir = new Vector3();
     // Jumping
     public bool isGrounded = true;
     Ray groundRay = new Ray();
     RaycastHit groundHit;
     public float rayDist;
     public float jumpForce;
     // Components
     Rigidbody RB;
 
     void Start ()
     {
         // Find/Get components
         if(this.GetComponent<Rigidbody>())
             RB = this.GetComponent<Rigidbody>();
         else
             RB = this.gameObject.AddComponent<Rigidbody>();
     }
     
     // Update is called once per frame
     void Update ()
     {
     //---------------Movement
         if(isGrounded)
         {
             moveDir.x = Input.GetAxis("Horizontal") * speed;
             moveDir.z = Input.GetAxis("Vertical") * speed;                
         }
         else
         {
             moveDir.x += (Input.GetAxis("Horizontal") * airSpeed) * Time.deltaTime;
             moveDir.z += (Input.GetAxis("Vertical") * airSpeed) * Time.deltaTime;
         }
         // Gravity
         moveDir.y = RB.velocity.y;
         moveDir = transform.TransformDirection(moveDir);
     //-----------Jumping 
         // Ground Check
         groundRay.origin = this.transform.position;
         groundRay.direction = Vector3.down;
         isGrounded = Physics.Raycast(groundRay, out groundHit, rayDist);
 
         if(Input.GetKeyDown("space"))
             moveDir.y = jumpForce;
     }
 
     void FixedUpdate()
     {
         RB.velocity = moveDir;
     }

So what gives? I have no idea why adding to the vector3's XY values with deltaTime should make this behaviour.

Thanks in advance for help.