koobas.hobune.stream
Why do tests run from the command line in player claim they aren't on the same platform as when run from player in editor?
I am running my unit tests in unity using the following command:
.\Unity.exe -runTests -projectPath path\to\project -testResults path\to\results.xml -testPlatform StandaloneWindows64
This successfully opens unity and the StandaloneWindows64 player, and claims the tests are being run successfully. However, the results.xml file claims that all the tests were skipped because they are not being run on the supported platform. When I run the tests from the Unity test runner, they are run successfully. I verified by adding failing unit tests and observing that they are run and reported as failing.
Each test uses this line (or similar) to restrict the platforms it can be run on:
[UnityPlatform(include = new[] {RuntimePlatform.WSAPlayerARM, RuntimePlatform.WSAPlayerX64, RuntimePlatform.WSAPlayerX86, RuntimePlatform.OSXPlayer, RuntimePlatform.Android})]
The Unity manual doesn't include platforms like WSAPlatformX64or similar in its documentation here, in the section "Running from the Command Line".
Why does running tests from the command line as above not follow the same behavior as the "Run all in Player" button in Unity? How do I get the tests to run on the platform(s) I want to from the command line?
EDIT: (More information)
I'm running on Unity 2017.1.0p4. Here's a section from the results.xml saying the test was skipped:
<test-case id="1005" ...(omitted)>
<properties>
<property name="_SKIPREASON" value="Only supported on WSAPlayerARM, WSAPlayerX64, WSAPlayerX86, OSXPlayer, Android" />
</properties>
<reason>
<message><![CDATA[Only supported on WSAPlayerARM, WSAPlayerX64, WSAPlayerX86, OSXPlayer, Android]]></message>
</reason>
</test-case>
When using "Run all in Player", the platform is StandaloneWindows.
EDIT 2:
Digging a little more, it seems I had my build settings wrong. The tests weren't running. Now they are, but the command line tests still won't. Basically I just want to use -testPlatform WSAPlayer*, but according to the documentation it doesn't seem like that's possible.
Hi @jwallra
Your arguments to Unity looks correctly, what version of Unity do you run?
UnityPlatform basically matches the include with Application.platform, and if that is not in the include list, it will be skipped.
Can you post an example of a test that gets skipped, with the Skip reason from the result.xml ? If your UnityPlatform is as you wrote above, and -testPlatform StandaloneWindows64 the tests would be skipped.
When running tests in Unity by "Run all in Player", what platform are you targeting, set in the Build settings?
I've updated my question with the details you asked about.
@HaraldNielsen - Do my edits make the problem more clear?
Hi @jwallra
How do your Test runner windows look like after run tests? They should be marked as skipped.
I just got the version you are running to test out the same setup as you where seeing.
This is my test: https://pastebin.com/PcyPZF8x
This is the test runner windows, displaying the tests skipped: http://imgur.com/doFhaA7
This is what is expected output. If you don't get the same result, let me know.
I'm not running them from the in-editor test runner. I'm running them in the player, which only displays results from failing tests, and says that all tests succeed if they're skipped (without any other diagnostic information). For example, with failing tests, run on WSAPlayer: http://imgur.com/y5vwJdq
I want to run on WSAPlayer, not the in-editor test runner, but from the command line. Is this possible?
According to @HaraldNielsen's comment, this isn't currently possible with the recommended set of arguments:
When using WSAPlayer you should get notified that it's not supported yet and the player build is terminated, so you can not, im sorry to say, run tests for that platform. Im currently adding WSA (and other platforms) support, so you would be able in a near future.
However, I've found that using this set of arguments sort of works:
.\Unity.exe -runTests -projectPath path\to\project -testResults path\to\results.xml -testPlatform WSAPlayer -batchmode
That command runs the tests in the WSAPlayer, which isn't in the list of available PlatformAttributes, but doesn't output the results to the specified file, so it's still a bit difficult to verify if the tests passed outside of using the -logFile, which doesn't have individual test results.
