Character controller bouncing off colliders

I have a sidecrolling platforming game where i'm using a character controller on the player character. Problem is, if my character collides with a collider (fx box) that is so low that it hits the curved part of my character controller collider, my character bounces backwards when i'm jumping. In other words, if i run into a low collider and jump, my charcter will bounce backwards violently. I tested this by running into a collider that is taller than my character, and it doesn't happen.

Does anyone know a way around this problem?