Moving many static colliders - Bad performance

I'm using Unity 5.6.1f1 and I have a game object "root" with a lot (several thousands) of children containing static (no rigidbody) capsule colliders (not a trigger). All objects containing a collider are in the layer "Grabbable" that only collides with another layer "Tool" in which there are only two sphere colliders. So all of those capsule colliders don't collide with each other.

The performance is great (about 80-90 fps) as long as I don't or only slowly move/rotate that root game object. The quicker I move/rotate the root object the lower the framerate (down to about 10fps). Thereby I update the transform.position/rotation every frame. With quick I only mean that the delta between two frames is big. What I see in the profiler is that I spend most of the time in Physics.Processing -> PhysX.PxsSap.sapUpdateWork

So it makes sense to me that PhysX has to adjust it's structs whenever you move colliders, but why does this depend on how far I move the colliders and how can I cope with that issue?

I've also tried to add a Rigidbody to the root object but this didn't help anything as expected according to this https://blogs.unity3d.com/2014/07/08/high-performance-physics-in-unity-5/