Why Physics perfomance cost get huge when static geometry is added ?

Hi everybody! I have a problem I can't solve using the Profiler.

Even if I did all the possible optimizations in Unity (texture atlas,batching,culling,lightmapping...), I think I have missed something because Physics.Simulate still takes more than 30.0ms to be calculated, even if I only have 15 rigidbodies active each frame (6 cars, 8NPCs, and the player).

When I delete all the static geometry, Physics.Simulate takes only 3.0ms to be calculated (my static geometry is a complex city, but there are no mesh colliders on the roads).

I don't understand how geometry can have so much influence on Physics.Simulate ? Am I missing something else ? What should I do ? (I am targeting mobile platforms, and because of this, my game only works on iPhone 5S and higher...)

Thank you for your help.

PS: if you want to see what the static geometry is like, here a preview of the game : https://www.youtube.com/watch?v=vOH0LxtQD-g&feature=youtu.be&t=29s