Problem with moving player relative to camera

Hi, I'm trying to achieve movement so that player can move up, down, right and left, but the character only rotates to match the sideways movement. (Similar to old beat-em up games like Battletoads and Double Dragon). The problem I have is when pressing the key A (moving left) the character moves to right instead of left. Every other direction works fine.

I have it setup like this: the character has a character controller inside it. Camera is a child of the player so it follows around. I'm processing input with a modified code based on this : http://unity3d.com/support/documentation/ScriptReference/CharacterController.Move.html and here's my code:

     void Update() 
 {
     CharacterController controller = GetComponent<CharacterController>(); 
     if (controller.isGrounded)     
         {
             
         moveDirection = new Vector3(Input.GetAxis("Vertical"), 0, -Input.GetAxis("Horizontal"));
         
         moveDirection = transform.TransformDirection(moveDirection);
         moveDirection *= speed;
             
             if(Input.GetKey("a")){
                 transform.rotation = Quaternion.Euler(0, 270, 0);
                 
             }
             if(Input.GetKey("d")){
                 transform.rotation = Quaternion.Euler(0, 90, 0);    
             }
         }
         
     controller.Move(moveDirection * Time.deltaTime);
         Debug.Log(moveDirection);
     }

Rotating the player obviously causes the problem here, as it flips the local axis around, but if I have understood right, using transform.TransformDirection should move the player in world space(?)... I have been stuck with this problem for a while now so any help is greatly appreciated!