Using Navmash navigation for direct character control

I'm working on a 3D isometric game using traditional point and click control for the character. It's currently set up using Unity's built in Navmesh system to control the character navigation. However, I want to make the system extendable to also support direct character control, like with an XBOX/Playstation controller. The Navmesh system doesn't seem to support this type of control, it only seems to support the basics of navigating to a fixed point. Of course I can build a totally separate navigation system for direct control, but that seems pretty inefficient.

Using the Navmesh, the only thing I've come up is to detect the direction that the controller is pressed and just set a navigation point slightly in that direction so it's basically like it's using point and click, but I'm "clicking" right next to the character constantly in whichever direction the controller is being pressed. That is obviously a horrible system, and I would never want to actually do that, but I haven't come up with a way to use the existing Navmesh for this type of control.

Am I missing something in the Navmesh functionality, or is it just not supposed to be used in this way?