koobas.hobune.stream
How to track all Update() calls to find infinite loop
I'm trying to track down a suspected infinite loop within our game. We have a massive project containing many loops - some written by us, some written by a previous company, and some contained within third-party plugins. At random times when running the game, the player will stop rendering, the Windows blue spinning pointer will appear, and clicking the window will cause a Windows "Not Responding" dialog to pop-up.
We've tried attaching the process to both the MonoDevelop debugger as well as Visual Studio, but both give no information when crashing, which is leading us to assume an infinite loop is the problem. We tried pausing the debugger after it froze, but that actually sends Mono into a frozen "Not Responding" state and never comes out.
But one idea we had was to print (to a socket) an "I'm here..." statement from the Update() call of each GameObject, so that we could hopefully narrow down the last Update to be called before entering the frozen state. The only problem is that we have 1380 instances of GameObjects stemming from 214 MonoBehaviour classes, some of which are empty base classes used for multiple derived classes which then contain the Update() implementation. So I was wondering if there were any ideas out there for triggering a print statement any time a MonoBehaviour.Update() call was made while knowing the caller? Or maybe wrapping MonoBehaviour into a "CustomMonoBehaviour" which could extend the Update() function somehow? Any suggestions would be greatly appreciated!
(Building for Windows 7 using Unity 4.5.0f6)
Edit: Added details on Mono crashing when trying to pause after the player freezes up.
If you don't get an answer here, ask this question on the Unity Forums.
have you tried checking the event viewer for any logs that the system may have written... could contain the stack trace.
Thanks for the suggestion, @Landern. Would you happen to know where in the event viewer I would look? And I'm assu$$anonymous$$g you mean the Windows event viewer?
@plwalsh88 after the crash open event viewer, click and expand windows logs, click on application(might take a second to render), look for the red exclamation icon, level error at the top(should be sorted by date, newest at the top of the table). Try and find anything related to your project.
@Landern I gave a look through that area and saw all the red exclamation points related to my crashes; however, there didn't seem to be any useful information attached to the events. Thanks for the suggestion, though.
A better solution for your problem would be to run the game from MonoDevelop/Visual Studio in debug mode so you can set breakpoints. They are much more dynamic and fit better for finding little bugs and tweaking things to it's optimum when compared to unity's Debug.Log();
Cheers
Where would you set breakpoints, though, when there's no indication of where the problem may lie? The freeze occurs at completely random times -- sometimes within five $$anonymous$$utes, sometimes after hours, sometimes while using the controller, sometimes while letting it sit untouched. So wouldn't setting breakpoints just have you sitting there for hours clicking the Play button every time it paused for a break? To clarify, ins$$anonymous$$d of Debug.Log(), the plan is to send a string over a socket and listen for it through another program, so as not to slow it down with print statements.
Ok, the last sentence of yours should make breakpoints/Debug.Log() kinda equal.. I suggested those beucase many Debug.Logs give me an terrible performance. To come to your question, in general I'd have done the same with it breakpoints as you wanna do with the logmessages. But then again, as you said, where to put em.. Seems like I only got half of your question last time, sorry mate. Do you have any multithreading running in your game? This randomness pretty much sounds like threading problems.. if you have some, I would suggest taking a look there first. Threading can make things act in a way you'd never expect when used unsafely.
No problem, thanks for the info. We actually do have some multithreading going on, good point. We have two extra server threads being spawned from the C# code to handle messages. I will try to disable those threads and see if we still get the same freeze-ups. Thanks for the suggestion.
From you description, it cannot be determined whether the game halts or unity crashes? (is that the player or editor?)
Adding some print code to all Update methods is possible using a framework such as Mono.Cecil (Unity also uses this framework for various tasks).
Cecil lets you read assemblies and mess with them at the IL level, so a possible solution would be to read all assemblies that are executed in your project, and for all MonoBehaviour classes that implement an Update method, inject the IL code needed to call some custom reporting method that would log some information.
Personally, i would do this only as a last resort, since it involves coding, learning how to use Cecil, testing, etc.
I would attempt to use a native debugger (WinDbg) if on Windows. It can attach to a process and will (perhaps) allow you to get a clear stack trace when Unity crashes.
This will be a native stack trace though, but it may show what object/component the native code was calling when it crashes.
So, to sum it all up:
Try to use a native debugger on either the editor or a standalone windows player.
If that failed to produce any helpful information, attempt to implement a solution using Mono.Cecil that will inject IL code to your assemblies.
I would be happy to elaborate on both of these options, if needed.
Thanks for the response! Regarding your first question, we're also having trouble figuring that out. It seems to halt rather than crash. It freezes the window on the last rendered frame, but the window never closes/crashes on its own. You have to click on the window to get the Windows popup telling you the program has stopped responding and asking if you would like to kill it. This is when running in the player. The Player Log output reports no indication of an exception being raised and no stack trace. I added my own infinite loop to one of the C# scripts and built a new player, and when I trigger that loop, I see the exact same behavior as described above. So that test, combined with the fact that we never see an exception, has led us to think it may be an infinite loop we are somehow, somewhere getting caught in.
I will give WinDbg a shot, if I can get it downloaded and whatnot. It's been hard to get a stack trace from any debugger since it never actually crashes. But maybe WinDbg will allow me to pause execution when it gets into its frozen state and see what line is being executed. Both $$anonymous$$onoDevelop and VisualStudio crash when I try to pause after it freezes; interestingly, though, $$anonymous$$ono doesn't crash when pausing during my intentionally injected infinite loop.
If you're on pro - use the profiler. It should show you methods that take too long to execute (if you don't have it, get the 30 day trial for it).
Give WinDbg a try, it's a solid debugger (unlike $$anonymous$$onoDevelop :) )
I am on pro (though I am not a pro). The profiler is a good idea, thanks. Do you know if it can profile the Player as opposed to the Editor? For some reason, I don't think we've ever seen this hang-up while using the Editor. Though, it could just be because we don't let it sit and run in the Editor for long periods of time. And I'll definitely give WinDbg a try, thanks!
The profiler can run on anything i think. It uses some custom networking layer (just a guess, since you select the player to profile using some name/port combo IIRC). So, yes, you can build a player that supports profiling (needs to be selected in the export) and then profile it.
