Why does my character lose speed while jumping against the wall?

I'm making a 2D platformer with Unity's built in physics. My character has a Capsule Collider2D with zero firiction, wall also has zero firiction. When I try to jump against the wall, I mean jumping while thouching the wall while holding right (wall is at my right) it doesn't reach the intended height and fails the jump. If I take a step back and jump, it jumps perfectly fine. Is this problem related to Unity's physics, or should I be looking at my character controller script?