Apply physics material to a third person character?

What I'm trying to do is have my third person player react to a physics material, giving the illusion that he is walking on ice. I am not using any pre-made controller, it is custom made.

When the physics material is applied to the player, he just slides in one random direction and that's it. No matter where the player is facing and regardless of which direction he chooses to walk, he will always keep sliding slightly in that direction unless hit by an enemy with a rigidbody.

I suppose what I want is to have the player continue sliding a little bit after he stops moving in the direction he was walking. I'm new to physics materials and am not really sure how to use them or if what I am trying to achieve is even possible based on how the player is controlled.

Is there something I'm missing, or would I have to change something in the player movement to have him behave as I would want?