Rigidbody2D - character jitter on collision

Hi!

I'm working on a bomberman clone, similar to Atomic Bomberman. I've made a background consisting of a 15x11 checkerboard with invisible circle colliders between the tiles (red dots). These are used to keep the players from running around in between tiles (blue arrows). Also, I have a character with a circle collider and rigidbody2d.

In Atomic Bomberman, the collision detection works without much jitter. If you run into one of the colliders between 4 tiles, it'll push you onto the right lane right away. You almost don't even notice it.

This works ok in my clone, unless you hit the collider right in the middle. If you keep pressing the button, the character will jitter alot before eventually being pushed onto the right lane.

Any suggestions on how I can solve this? The size of the colliders is minimal, btw.