Character Controller's performance for massive groups

As generally advised, I have built enemies in my game using the character controller.

I have then tried spawning lots of them (probably hundreds) and the game got slow, mostly because of the controller script itself, and Physics.Simulate (which I assume is the physics engine process).

So I kind of expected that, but I am wondering: How should one go about having very large group of enemies, performance-wise?

In my case, my script was eating 66% of the time (which may be because of poor prototyping code so don't mind that) and physics 33%. Which means only the fact of the character controller existing already takes a third of the power (in my case, at least).

Are there ways to optimize the character controller? Or other alternatives for large groups? I should precise, I still want my enemies to have their AI, I'm just looking for ways to optimize them.

PS: The floor in my game isn't flat, so I need some sort of collision detection for them to keep moving on the floor