How to "add force" to a character controller

So I'm trying to make my character wall jump and i have the vectors and all the tricky stuff set up, but I need to somehow all of a sudden give the player velocity in order to jump off of the wall. Obviously I can't use AddForce because Im using the Character Controller, but how would I add momentum at least and then allow it to be counter-acted or something. For my movement script, I set the velocity to a certain thing and when I take my finger off the key it stops. But with the wall, I don't want it to keep on moving at that velocity. If i give my character a velocity of -1 m/s and the wall jump gave him a velocity of 5 m/s I want his velocity to go from 5 to 4 to 3 to 2 and then, if i take my finger off, it should stop at 2 m/s. This has stumped me for so long, thanks for any help :)