Why does my characterController code not apply gravity correctly?

I'm using a characterController and while movement on the ground is just dandy, jumping and gravity are somewhat loopy.

1) Hitting jump while stationary behaves as expected - Starting at 1.08 in y, the character rises to roughly 2.5, then falls back to 1.08. However, jumping like this and then holding a direction causes the character to glide slowwwly back to ground level. Releasing a direction key causes gravity to kick back in and the character drops to ground. Not expected behaviour.

2) Hitting jump while moving results in my character hopping up a tiny increment, reaching 1.20, then quickly falling to 1.08 (ground level).

I've stared at my code until I can no longer understand any of it. I'm hoping for a solution... can anybody see where I'm going wrong?

 void FixedUpdate () {
 
         CharacterController controller = GetComponent<CharacterController> ();
 
         camForward = Vector3.Scale (camFieldSouth.forward, new Vector3 (1, 0, 1)).normalized;
 
         float inputX = Input.GetAxis ("Horizontal");
         float inputY = Input.GetAxis ("Vertical");
 
 
                 
         if(controller.isGrounded){
 
             moveDirection = new Vector3(inputX, -antiBumpFactor, inputY);
             moveDirection = moveDirection.z * camForward + moveDirection.x * camFieldSouth.right;
             if(moveDirection.magnitude > 1)moveDirection.Normalize();
             // moveDirection = transform.TransformDirection(moveDirection);
             moveDirection *= speed;
 
             if(Input.GetButton("Fire1")){
                 moveDirection.y = jumpSpeed;
                 jumped = true;
             }
         }else {
             // Check movement but don't touch the y component
             Vector3 camMoveRef = new Vector3(inputX, -antiBumpFactor, inputY);
             camMoveRef = camMoveRef.z * camForward + camMoveRef.x * camFieldSouth.right;
             // camMoveRef.y = 0.0f;
             if(camMoveRef.magnitude > 1)moveDirection.Normalize();
             moveDirection.x = camMoveRef.x * speed; // If this is just inputX * speed, it works fine - but not cam relative.
             moveDirection.z = camMoveRef.z * speed; // If this is just inputY * speed, it works fine - but not cam relative.
             Debug.Log (moveDirection.y);
             // moveDirection = transform.TransformDirection(moveDirection);
             jumped = false;
         }
         moveDirection.y -= gravity * Time.deltaTime;
         controller.Move (moveDirection * Time.deltaTime);
 
         // NPC movement code
         // Trimming the position List to remain with set size
         if (positionRecord.Count > positionRecordSize) {
             positionRecord.RemoveAt(0);
         }

Further information on request. Thanks for your time, folks.

--Rev