Colliding to mesh seams when shouldn't

So I have many block like objects right next to each other. Their faces (and colliders) are perfectly aligned at the same position, yet this happens:

There are no sideways seams at all in the colliders (the mesh in between two blocks is removed), so this should be treated as a couple flat surfaces next to each other.

Why does this still happen, and how can I avoid it? I've tried setting the physics solver iteration count higher, but it didn't help. Running out of ideas here.