3D Colliders colliding with nothing

I am creating a tower defense game, and I'm adding collision between towers. I want to be able to test collision between the two towers, but have them pass freely through each other. To do this I am using colliders and use Physics.ignoreCollision between all the towers. The problem is that when the colliders are enabled, changing position with transform.position is very choppy as if it's colliding with the terrain (but the terrain is flat and collision between towers and the ground is ignored).

 -I'm using a raycast from the mouse position to set the tower position.
 -Only 1 tower moves at a time.
 -I am testing with box colliders and capsule colliders

How can I detect collision between two towers while having them ignore ALL collisions?

Thanks in advance