Continuous Integration with Unity?

I use Jenkins (formerly Hudson) CI Server with that line as a build step (windows batch):

%unity_dir%\Editor\Unity.exe -batchmode  -nographics -quit -createProject "%WORKSPACE%\unity_test" -projectPath "%WORKSPACE%\unity_test" -assetServerUpdate 192.168.0.2 unit_testing username password -executeMethod CIScript.start

%unity_dir% is an environment variable I set up, %workspace% is from Jenkins. The "CISript" is the entry point for all my automated building and testing. (EDIT: to be precise, the static method "start" of the class "CIScript" is the exact entry point)

SharpUnit features a XML-reporter, I beefed that up a little to resemble JUnit enough to be directly usable from Jenkins - release pending.

I have some Tests as editor scripts, they set up a new scene and add GameObjects, to add MonoBehaviour-derived classes as components, respectively. What is currently not possible is to run a simulation (i.e. EditorApplication.isPlaying = true) while in batchmode, because you can't enter play mode, run a few or many frames and leave it graciously. The leaving part is the problem (ref: Concurrency and editor scripts and the play mode and Programmatically playing a scene for a single frame)

For automated smoke/application tests you will have to do the following things:

• run the batch mode as prescribed above but don't use -quit

• in the static method do all of the following:

  • setup a scene, use throwaway ones with EditorApplication.NewScene() or load some prepared ones.

  • set EditorApplication.isPlaying to true

  • add one essential script to the scene, see below, that will enable you to end the simulation

  • add a static method to EditorApplication.playmodeStateChanged delegate, the respective method will be run twice, so make sure that the game actually ran before you do anything within that method. Oh yeah, that's the part where you can do assertions enclosed in try blocks. You should do assertions through the OneFrameSignaller script before the play mode is ended. You can also use the code on this delegate to set up another scene and repeat the test cycle on the next scene.

Be aware the the static method (i.e. entry point of the batch mode execution) must terminate before the play mode can actually start, due to all unity api stuff running on the same system thread aparently. Thus don't use the -quit option for the batch mode.

The key script below is an editor script. You can't manually add that to a GameObject in the actual unity editor by drag&drop. Programmatically this works never the less, don't stop and wonder why, embrace the fact - it makes things so much easier..

using UnityEngine; using UnityEditor; using System; using System.Collections; using System.Threading;

 void Update()
 {
     atLeastOneFrameRan = true;
     StartCoroutine(signalFrameEnd()); // use this to quit after this frame, you could use a counter instead
     if (DateTime.Now - OneFrameSignaller.startTime > OneFrameSignaller.testDuration) // use this if you want a timed end of the test
     {
         StartCoroutine(signalFrameEnd());
     }
     // Thread.Sleep(20); //this is somewhat evil, us this to extend the execution time 
     // of the frames if you feel they run to fast for useful assertions
 }

 void Start() {
     OneFrameSignaller.startTime = DateTime.Now;
     OneFrameSignaller.testDuration = TimeSpan.FromSeconds(10);
     Application.runInBackground = true;
 }
 
 IEnumerator signalFrameEnd()
 {
     yield return new WaitForEndOfFrame();
     EditorApplication.isPlaying = false;
 }

So what happens is this:

The static method exits, through Unity's internal workings the play mode starts as EditorApplication.isPlaying was set to true. The above script is executed like a normal MonoBehaviour script. If the exit-condition is met, at the end of the frame EditorApplication.isPlaying is set to false. Any assertions about the state of the existing scene should be made before isPlaying is set to false. For assertions I use Assert from SharpUnit. Be sure to enclose this in try-blocks because test setup is very costly here. So defy the principle of using one test-setup/teardown per test, just make sure that assertions don't interfere with each other. Setting isPlaying to false triggers the execution of the delegate EditorApplication.playmodeStateChanged. From there you can either quit now with EditorApplication.Exit(0) or another non-zero number to indicate test failures to the continuous integration server during build - or you go ahead and setup a new scene and repeat the cycle. Just be sure to call Exit on some point or your build will never finish.

If all Tests pass I build a new version of the game using the BuildPipeline.

I am not sure if this is still relevant, but with Jenkins/Nunitlite and its Unity plugin we have now CI without major problems.

NunitLite Runner: https://github.com/zoon/NUnitLiteUnityRunner

Here's an asset on asset store: https://www.assetstore.unity3d.com/#/content/12695.

You can use it with any CI framework.

=== 2022 Working Version ===

In case anyone finds this thread from Google and wants to run the game to test things headlessly (like me), here's all the info you'll need:

 public class PlayTestsLoader : MonoBehaviour
 {
     public static bool PlayTestsRunning = false;
     public static bool PlayTestsWantToStop = false;
     // Run with (note paths are Linux syntax - replace as needed)
     // ./Unity -projectPath ~/unityprojectname -executeMethod PlayTestsLoader.LoadTestScenes -logfile "~/unityprojectname/logs.txt" -batchmode
     public static void LoadTestScenes()
     {
         try
         {
             EditorUserBuildSettings.SwitchActiveBuildTarget(BuildTargetGroup.Android, BuildTarget.Android); // Test Android code paths as if you were doing it manually in Editor
             EditorSceneManager.OpenScene($"Assets/Scenes/mainscene.unity"); // Load the scene in Editor - Not at play time!
             EditorApplication.isPlaying = true; // When this static method returns, setting isPlaying to true will start the game
             TestLog($"Entered playmode");
         }
         catch (Exception e)
         {
             TestLog($"Had exception {e}");
         }
         TestLog($"Complete PlayTestsLoader.LoadTestScenes");
     }

     static void TestLog(string log)
     {
         Debug.Log($"=== PlayTests === {log}");
     }

     public void Update()
     {
         if (PlayTestsRunning && PlayTestsWantToStop)
         {
             TestLog("ALL TESTS COMPLETED -- RETURNING 0");
             EditorApplication.Exit(0); // This will stop the game from running when you're done, and make the headless (invisible) editor exit
         }
     }

     public void OnEnable()
     {
         PlayTestsRunning = true;
         TestLog("PlayTestsLoader Monobehaviour started");
         StartCoroutine(PlayTestJourneyController());
     }
   
    IEnumerator PlayTestJourneyController()
     {
         yield return StartPvP();
         yield return SayHelloAfterPvP();
    
         PlayTestsWantToStop = true;
         yield return null;
     }
   
     IEnumerator StartPvP()
     {
         TestLog($"Hello from StartPvP");
         yield break;
     }
   
     IEnumerator SayHelloAfterPvP()
     {
         TestLog($"Hello from SayHelloAfterPvP");
         yield break;
     }
 }