Character floats upward when looking up and down

I have a first person character controller that works fine so far, using an Xbox controller instead of a keyboard/mouse. The biggest problem is when I point the camera downward and move the character backwards, it begins to float steadily upwards as it moves backward. The same when looking up and moving forwards, except it floats upward steadily as it moves forwards. This also happens when jumping. I have a vague idea of where in my code the problem is.

 void Movement()
 {
     CharacterController controller = GetComponent<CharacterController>();       

     //applies gravity to character controller
     if (controller.isGrounded)
     {
         moveDirection = new Vector3(Input.GetAxis("Horizontal"), 0, Input.GetAxis("Vertical"));
         moveDirection = transform.TransformDirection(moveDirection);
         moveDirection *= speed;

         if (Input.GetButton("Jump"))
             moveDirection.y = jumpSpeed;
     }

     else if(!controller.isGrounded)
     {
         moveDirection = new Vector3(Input.GetAxis("Horizontal"), moveDirection.y, Input.GetAxis("Vertical"));
         moveDirection = transform.TransformDirection(moveDirection);
         moveDirection.x *= speed / 2;
         moveDirection.z *= speed / 2;
     }

     moveDirection.y -= gravity * Time.deltaTime;
     controller.Move(moveDirection * Time.deltaTime);
     //moves camera with mouse
     rotationX += Input.GetAxis("Mouse X") * lookSpeed;
     rotationY -= Input.GetAxis("Mouse Y") * lookSpeed;
     transform.rotation = Quaternion.Euler(rotationY, rotationX, 0);
 }

Where it says moveDirection.y -= gravity * Time.deltaTime; seems to have some effect, because when I delete Time.deltaTime it eliminates this problem. Thing is, if i do that my character is no longer able to jump. Has anyone else encountered this problem?

Edit: Don't know if this is worth mentioning but my character is simply a sphere as opposed to an animated object like Ethan in standard assets.