Collision sticking?

I'm currently building a game where the player can "flick" between walls - sticking to them on collision. So when the game starts the player is stuck to the wall on the left, they flick right and their character jumps to a wall on the right, then they flick left and it jumps to the wall in the middle. This is illustrated in the below diagram.

When the player jumps I apply some force to its rigid body, the problem is that for 1-2 frames after the jump it still thinks its colliding with the wall its jumping from which causes it to "re-stick" to the wall (it sticks because touching the wall reduces the objects velocity to 0 and gravity is disabled in the game).

One solution would be to say "after a jump then temporarily ignore the wall that you're jumping from". The problem with that is if you imagine you're attached to the middle wall, to the right, and you attempt to jump left then you want to keep sticking to the wall - you want it to reset your velocity to 0. That could be solved by recasting or using the contact points to ensure that the player never tries to jump "into an object" (like the above case) but that becomes pretty ineffective where colliders are odd shapes or have holes in them.

Any help would be much appreciated! This is really causing me some problems!

Oh - as a quick side note - it's not caused by the colliders physics materials - they have 0 friciton.