Controlling velocity of rigidbody in air

Okay im a little stumped here. Basically i have a rigidbody that has a certain constant drag when in contact with floor. When it loses contact with the floor and is now in the "air", i set the drag to ".1", essentially 0. I do this because the ground drag is around 2.5 or so, which means that if the rigidbody is in the air, it would float down very slowly and unrealistically. By setting drag to .1 or 0, i get a proper falling object.

My problem is that since i control my rigidbody using addforce(), when the object enters the air while moving a certain direction, if i keep holding the movement key corresponding to that direction, the object will continue to speed up more and more. The speed up is unsatisfactory even if i lower addforce() when the rigidbody is in the air, which i did.

The reason i dont make addforce() very low or turn it off altogether when the rigidbody is in the air, is because i want to give the player a chance to adjust the position of rigidbody while it's in the air.

So what i want to do is essentially set a max X and Z velocity that the rigidbody can reach while in the air without clamping the velocity which i hear is bad practice when it comes to the physics engine. I was thinking of increasing drag to a point where the velocity won't increase past a certain point, but then the Y direction of the rigidbody is affected by drag as well, which is something i want to avoid.

I can think of one particular solution, where i manually check the velocity in the X and Z directions every frame while the object is in the air, and if it's past a certain velocity, then i can turn off the movement controls in that particular direction until the velocity goes below the threshold. This way i never manually mess around with the velocities.

I was wondering if there's a more elegant solution to my problem that some genius here can think of :)? thanks.