Composite Collider on Tilemap not really stopping player

So I'm working on a simple topdown shooter with some tiles for walls and floor. The walls have a tilemap collider using the composite collider. The problem is, sometimes when the player is walking, there are some specific areas that the player can walk through, specially if the player is moving fast. Is this a bug or it might be some problem with my configs or my code? (I'm using RigidBody2D.velocity to move the player btw).