Simple GameObjects. Unexpected movement.

Hopefully someone can point out what could be a potential cause of the problem.

Here is how my scene works. I spawn enemies along the borders of a small plane and they start moving towards the center of the plane (Player is in center) in regular intervals (every 1.5 seconds they move forward 1 unit using MoveTowards). Enemies get predefined positions of where they need to be before starting to move, that position is incremented by 1 unit every time an enemy is ready to move.

Movement is handled inside enemies themselves, but spawning is handled by the World script (it recycles enemy GameObjects, ie does SetActive(true/false)).

Here is my problem: If there are 1 - 2 enemies on the field, everything works as expected, but when I allow more enemies to exist on the field (without killing them to control the amount at 1 - 2), occasionally every 2 - 3 spawns once I spawn the enemy it starts sliding in a "random" direction, but then moves as expected (1 unit towards the direction it was meant to move).

I spend a few days on this problem and can't figure it out, I'll be happy to add anything else you might want to know.

Thank you for your attention in advance.