- Home /
Continuous Integration with Unity?
Hi,
has anyone had success creating a continuous integration setup with Unity? I.e., some server listening for changes in the repository, automatically checking out code and assets whenever such changes occur, compile the code and run unit-tests, ideally with code coverage profiling?
I'm trying to set this up currently with Teamcity as as CI-server (company standard) and SharpUnit for the unit-tests, but it seems problematic that a) the build agent runs as a service (obviously, as it cannot be dependent on a particular user being logged in) and that does not seem to play together well with starting Unity, even with -batchmode -quit -executeMethod
b) the output of SharpUnit is put into Unity's log-file which resides in a subfolder of %localappdata% which makes it somewhat hard to access by the build agent, particularly if the later is run under the system/services account; besides, it's really mangled up with all the other log output
c) regarding code coverage I don't even have any ideas or pointers - is there anything like code coverage for Unity?
Many thanks, Max Vienna, Austria
At Apex we use CruiseControl.Net as build server. a) We also have problems with getting the batch mode stable. No idea why, but if I sit at the PC and forces a build, it works, but nightly builds mostly fails, b) ditto, c) we only run tests on code not dependent on Unity-dlls using nant and nunit. The reason is that calls to Unity-dlls are stopped, if not running Unity editor or build, which kind of forces us to make most code outside Unity.
Good news! We figured out how to solve a). On windows use argument -nographics to let the build server run at night when the server's desktop is locked (no graphics device seems to be available)
pop up this, maybe somebody can add more info, since I was about to write the same question.
Answer by Tom 17 · May 30, 2011 at 08:00 AM
I use Jenkins (formerly Hudson) CI Server with that line as a build step (windows batch):
%unity_dir%\Editor\Unity.exe -batchmode -nographics -quit -createProject "%WORKSPACE%\unity_test" -projectPath "%WORKSPACE%\unity_test" -assetServerUpdate 192.168.0.2 unit_testing username password -executeMethod CIScript.start
%unity_dir% is an environment variable I set up, %workspace% is from Jenkins. The "CISript" is the entry point for all my automated building and testing. (EDIT: to be precise, the static method "start" of the class "CIScript" is the exact entry point)
SharpUnit features a XML-reporter, I beefed that up a little to resemble JUnit enough to be directly usable from Jenkins - release pending.
I have some Tests as editor scripts, they set up a new scene and add GameObjects, to add MonoBehaviour-derived classes as components, respectively.
What is currently not possible is to run a simulation (i.e. EditorApplication.isPlaying = true) while in batchmode, because you can't enter play mode, run a few or many frames and leave it graciously. The leaving part is the problem (ref:
Concurrency and editor scripts and the play mode and
Programmatically playing a scene for a single frame)
For automated smoke/application tests you will have to do the following things:
run the batch mode as prescribed above but don't use -quit
- in the static method do all of the following:
setup a scene, use throwaway ones with EditorApplication.NewScene() or load some prepared ones.
set EditorApplication.isPlaying to true
add one essential script to the scene, see below, that will enable you to end the simulation
add a static method to EditorApplication.playmodeStateChanged delegate, the respective method will be run twice, so make sure that the game actually ran before you do anything within that method.
Oh yeah, that's the part where you can do assertions enclosed in try blocks.You should do assertions through the OneFrameSignaller script before the play mode is ended. You can also use the code on this delegate to set up another scene and repeat the test cycle on the next scene.
Be aware the the static method (i.e. entry point of the batch mode execution) must terminate before the play mode can actually start, due to all unity api stuff running on the same system thread aparently. Thus don't use the -quit option for the batch mode.
The key script below is an editor script. You can't manually add that to a GameObject in the actual unity editor by drag&drop. Programmatically this works never the less, don't stop and wonder why, embrace the fact - it makes things so much easier..
using UnityEngine; using UnityEditor; using System; using System.Collections; using System.Threading;
public class OneFrameSignaller : MonoBehaviour { public static bool atLeastOneFrameRan = false; public static DateTime startTime; public static TimeSpan testDuration;
void Update()
{
atLeastOneFrameRan = true;
StartCoroutine(signalFrameEnd()); // use this to quit after this frame, you could use a counter instead
if (DateTime.Now - OneFrameSignaller.startTime > OneFrameSignaller.testDuration) // use this if you want a timed end of the test
{
StartCoroutine(signalFrameEnd());
}
// Thread.Sleep(20); //this is somewhat evil, us this to extend the execution time
// of the frames if you feel they run to fast for useful assertions
}
void Start() {
OneFrameSignaller.startTime = DateTime.Now;
OneFrameSignaller.testDuration = TimeSpan.FromSeconds(10);
Application.runInBackground = true;
}
IEnumerator signalFrameEnd()
{
yield return new WaitForEndOfFrame();
EditorApplication.isPlaying = false;
}
}
So what happens is this:
The static method exits, through Unity's internal workings the play mode starts as EditorApplication.isPlaying was set to true. The above script is executed like a normal MonoBehaviour script. If the exit-condition is met, at the end of the frame EditorApplication.isPlaying is set to false. Any assertions about the state of the existing scene should be made before isPlaying is set to false. For assertions I use Assert from SharpUnit. Be sure to enclose this in try-blocks because test setup is very costly here. So defy the principle of using one test-setup/teardown per test, just make sure that assertions don't interfere with each other. Setting isPlaying to false triggers the execution of the delegate EditorApplication.playmodeStateChanged. From there you can either quit now with EditorApplication.Exit(0) or another non-zero number to indicate test failures to the continuous integration server during build - or you go ahead and setup a new scene and repeat the cycle. Just be sure to call Exit on some point or your build will never finish.
If all Tests pass I build a new version of the game using the BuildPipeline.
Seems even more involved, after some trying out things I didn't manage to recreate the exact behaviour as described above. I would like to have some feedback by others before I jump at describing another way. I still got it working again, but it still is more involved. When the playmode starts terrible things happen to the memory ;-) Afaik it has to do with a special feature of the play mode. When it starts, the scene is backed up but also all variables of the assembly are reset?! So you need some persistence mechanism with the variables that govern the state of the test-execution.
Do you still have the X$$anonymous$$L reporter? I was trying to integrate SharpUnit to Jenkins with NUnit plugin. But the plugin can't read the resulting X$$anonymous$$L. I think this is because SharpUnit's X$$anonymous$$L output is not correct.
Hi, any restriction on type of license we can use on CI server ? or, we could use Unity personal or pro licenses ?
Answer by sebas77 · Jun 21, 2012 at 03:29 PM
I am not sure if this is still relevant, but with Jenkins/Nunitlite and its Unity plugin we have now CI without major problems.
NunitLite Runner: https://github.com/zoon/NUnitLiteUnityRunner
Hi sebas77, How do you deal with method that uses coroutine?
you mean for unit test? you cannot test them...the only way is to run them synchronously with a while
Answer by Rustam-Ganeyev · Nov 20, 2013 at 10:22 PM
Here's an asset on asset store: https://www.assetstore.unity3d.com/#/content/12695.
You can use it with any CI framework.
Answer by dan_ginovker · Feb 15 at 05:43 PM
=== 2022 Working Version ===
In case anyone finds this thread from Google and wants to run the game to test things headlessly (like me), here's all the info you'll need:
public class PlayTestsLoader : MonoBehaviour
{
public static bool PlayTestsRunning = false;
public static bool PlayTestsWantToStop = false;
// Run with (note paths are Linux syntax - replace as needed)
// ./Unity -projectPath ~/unityprojectname -executeMethod PlayTestsLoader.LoadTestScenes -logfile "~/unityprojectname/logs.txt" -batchmode
public static void LoadTestScenes()
{
try
{
EditorUserBuildSettings.SwitchActiveBuildTarget(BuildTargetGroup.Android, BuildTarget.Android); // Test Android code paths as if you were doing it manually in Editor
EditorSceneManager.OpenScene($"Assets/Scenes/mainscene.unity"); // Load the scene in Editor - Not at play time!
EditorApplication.isPlaying = true; // When this static method returns, setting isPlaying to true will start the game
TestLog($"Entered playmode");
}
catch (Exception e)
{
TestLog($"Had exception {e}");
}
TestLog($"Complete PlayTestsLoader.LoadTestScenes");
}
static void TestLog(string log)
{
Debug.Log($"=== PlayTests === {log}");
}
public void Update()
{
if (PlayTestsRunning && PlayTestsWantToStop)
{
TestLog("ALL TESTS COMPLETED -- RETURNING 0");
EditorApplication.Exit(0); // This will stop the game from running when you're done, and make the headless (invisible) editor exit
}
}
public void OnEnable()
{
PlayTestsRunning = true;
TestLog("PlayTestsLoader Monobehaviour started");
StartCoroutine(PlayTestJourneyController());
}
IEnumerator PlayTestJourneyController()
{
yield return StartPvP();
yield return SayHelloAfterPvP();
PlayTestsWantToStop = true;
yield return null;
}
IEnumerator StartPvP()
{
TestLog($"Hello from StartPvP");
yield break;
}
IEnumerator SayHelloAfterPvP()
{
TestLog($"Hello from SayHelloAfterPvP");
yield break;
}
}