Character controllers against each other

I have several characters. They need a character controller as a collider to not fall through the ground, stop at walls and jump physically. But I want that the characters avoid their character controllers when they come together as characters. They should be able to walk through each other like they had no collider.

How can i do this?