Lots of !IsFinite() Errors and FPS crawling to a halt - How to debug?

Hello,

When I'm running my game, I'm getting tons of !IsFinite() errors, and this causes my game to really crawl (like under 0.5 FPS crawling).

Messages are very uninformative, and I have no idea how to debug. For example:

 !IsFinite(outDistanceForSort)
 UnityEditor.DockArea:OnGUI()

 !IsFinite(outDistanceAlongView)
 UnityEditor.DockArea:OnGUI()

Now these keep coming every second.

I know exactly which line of code I added to make this problem happen, but that line of code is essential, and I have no idea how to debug what is causing this. Any ideas on what could cause this or how to approach it for debugging, as the message doesn't point to any specific object?

The line I added to cause this is setting the Time.timeScale to 0:

 Time.timeScale = 0;

I do this because I want the game to start paused and only begin after the player clicks. This command happens in one of the Start() methods, but I also tried moving it to the first Update() call instead and it didn't help.

I should note: I have a pause button in game, and if I don't start the game paused, but the player manually pauses the game from that button, then it usually works fine. But pause it many times, and you might get this error again, though rarely. On the other hand when I pause at the beginning of the game, I ALWAYS get this error. Both pause mechanisms work the same.

Thanks!

Edit: Just making something clear - the errors and FPS issues continue even after I unpause the game and return the timescale to normal.

Edit 2: Ok, I managed to locate the problem object by removing them one by one until I stopped getting the errors. The problem object turned up being a Cloth object (with an interactive cloth component and cloth renderer). The cloth object doesn't have any script on it. Any ideas as to why setting timescale to 0 will cause these errors with the cloth object?