Adding and Removing lots of components, performance?

I'm making an action system for my game and am wondering if there is a significant performance hit by adding and removing lots of components from a gameobject.

The actions in question would be custom scripts like, "move to", "attack", "patrol", "use ability", etc. Some actions can work together, others not.

The alternative I am considering is a "lite component" system where actions are custom classes that operate inside existing unity components. The purpose being to reduce overhead and gain performance benefits.

These actions will operate on perhaps two dozen "units" at a time, perhaps more. This also needs to work well on an iPad.

So would it be worth it to make a specialized action system or should I just use Unity components?

Thanks!