Make 3D character controller not stuck on walls

I am creating a rigidbody based character controller, and therefor not using the unity CharacterController. The common problem i have is, that when jumping into a wall, i get stuck on the wall. The typical solution i have read about, is adding af physics material to the player, with Dynamic Frction and Static Friction set to 0. This works, but in my situation i have a sliding mechanic which uses rigidbody.addforce. So when i slide with this physics material, i can slide forever. I tried setting the frictions according to if you are on the ground or in the air, but then i can't jump if i am grounded right next to a wall. Is there any other way to fix this probem?