koobas.hobune.stream
Crash in headless build, not present in headed build
I'm currently working on a networked project in Unity 2017.1.0p5. The architecture I'm shooting for is running a Linux build of the game in headless mode as a server on Amazon GameLift that clients connect to. Throughout development, I have been testing with two non-headless builds on Windows with no major problems. I've come to the point in development where I need to test the game in the target configuration and am running into some issues.
Here's how it has gone:
I tried testing directly on Amazon GameLift with a headless Linux build of the game as a server. The client can get into my Lobby scene just fine, but during LevelLoad on the server the server crashes, disconnecting the client. (Note: The way my project is configured, the server loads the level entirely before it send a message to tell the clients to load into the next scene.)
I tested with a headless Linux build locally on a Linux machine for better and faster debugging. The same server crashing problem occurred. I also tested a non-headless Linux build as the server and connected to it with a client on a Windows machine. This worked just fine, and produced the same behavior I have been seeing in testing 2 non-headless builds simultaneously on a Windows machine. By my measure, these tests show that it is not a GameLift issue and not a Linux issue.
I tested a headless Windows build of the game locally by running the game through the command line with the arguments -batchmode -nographics. This produces the same crash as seen in the above two scenarios. Again, as always, connecting to a Windows build of a non-headless server works fine.
Other important things to note:
- Loading into a blank scene works fine.
- There does not appear to be a specific"offending" object that is causing the crash in the scene. I went through the whole scene, deleting objects two or three at a time until I was able to load into the scene on the server without a crash. Restoring the scene, then deleting the last few objects I deleted and trying that as a build still produces a crash. (This, of course, does not rule out the possibility of multiple offending objects.)
- As far as I have been able to log, all Awake and Start methods on all Monobehaviours in my scene are finishing before the crash happens. I am loading the scene asynchronously and have a coroutine that is waiting for the AsyncOperation to be done that is never completing, so I know the crash is happening before the level has been loaded completely.
- There are no meaningful errors (that I can see) preceding the crash in the unity log file itself.
I have attached the crash log as it is substantial.link text
total shot in the dark here: Does the scene that runs on the headless machine have any camera? If so, perhaps the headless-ness(?) is confusing the system when trying to render camera outputs?
After a bunch of testing we have discovered the source of our issue. It was that our particle systems were on and attempting to run on our server, causing it to crash. The particle systems in unity are threaded and this was resulting in the memory access violation.
The fix for this was to disable all the particle systems(at the component level) then have the clients turn them on if/when needed. If the server does need to reference the particle emitters from time to time, this will be accomplished by turning the particle components on when they are needed.
