Physic spikes after few seconds of gameplay on Android

I have a ball and a parent object that contains world geometry. Parent object contains couple objects that may intersect each other and each of them has a collider. Some of those objects have animations.

Each frame parent object is slightly rotated around the ball and after few second of game play, physics start lagging and take as much as 90ms each frame instead of regular 5ms intervals.

How can I get rid of these lags, is something wrong with desing of my scene ?