Player rotation issues (Player returns to default forward rotation after releasing keys)

So basically what is happening is that my player rotates and faces the direction that he is currently being steered toward (which is wonderful). But after I release, let's say the A key, meaning he is currently facing and moving to the left, he goes back to facing forward after the key is released.

Here is my code:

 public class PlayerMovement : MonoBehaviour {
 public bool isGrounded;
 private float distToGround;
 private Vector3 v;
 private Vector3 h;
 public float speed;
 
 Rigidbody rb;
 Collider col;
 
 void Start() {
     rb = GetComponent<Rigidbody>();
     col = GetComponent<Collider>();
     distToGround = col.bounds.extents.y;
 }
 
 void Update() {
     if (Physics.Raycast (transform.position, -Vector3.up, distToGround + 0.1f)) isGrounded = true; else isGrounded = false;
     if (isGrounded) {
         v = new Vector3(0, 0, Input.GetAxis("Vertical")) * speed;
         h = new Vector3(Input.GetAxis("Horizontal"), 0, 0) * speed;
         rb.MovePosition(rb.position + v * Time.deltaTime);
         rb.MovePosition(rb.position + h * Time.deltaTime);
 
 //This deals with all the rotation
         Vector3 direction = new Vector3( Input.GetAxis ("Horizontal"), 0, Input.GetAxis ("Vertical") );
         Quaternion rotation = Quaternion.LookRotation(direction, Vector3.up);
         transform.rotation = rotation;
     }
   }
 }

I obviously know that the reason this is happening is because the rotation is being set to the axis currently being pressed, so when I release any key, the rotation goes back to the original position. I just can't quite figure out how to fix it...