What are the causes of Device.Present performance problems?

I've been under the impression in the past that Device.Present is an indicator that your graphics are too complicated, which causes the GPU to take too long, which causes a need to wait for the next VSync (especially on iOS devices where VSync is forced).

However, I'm doing some Physics testing right now in a graphically simple scene - 4 draw calls. When the number of static colliders in the scene is very small (around 20-40 static box colliders), the game runs very well on the device, around 60FPS.

When I get into large amounts of Physics (2000 static box colliders), the game STILL runs at 60FPS, all is well, EXCEPT I occasionally see Device.Present taking up 977ms (so, about 1 second of time). This is very noticeable in the game, which appears to freeze for that 1 second.

This doesn't make sense to me. First, on the frames with Device.Present problems, the frame is the same # of milliseconds as any other frame, so I don't see how I'm suddenly missing the VSync. Also, if Device.Present is related to GPU, why would colliders affect it (my colliders have no graphical component)? Physics.Simulate consistently finishes in ~1.6ms from 40-2000 static colliders in the scene. Why do these static colliders suddenly trigger Device.Present problems?

I could see this being a bug, but I think it more likely that I just don't understand the implications of Device.Present as well as I thought. Can anyone give any insight into why I would see such results?