Probably you’ve heard about the Performance Testing package made by Unity, but to be fair their documentation sucks and if you follow it straight you might face some issues. So I want to list the steps I did to have it running and especially how to get results from performance tests after a run on a real device because obviously, we don’t want to measure performance of the game only inside the editor.

For this tutorial, I used Unity version 2021.2.3f1.

1. Create Tests Class
Open Test Runner (Window – General – Test Runner) and press Create Test Assembly Folder and then Create Test Script in the current folder

2. Update manifest with correct Performance Testing package version
The latest working version seems to be "com.unity.test-framework.performance": "2.8.0-preview", so add this line to your manifest.json located in the Assets folder.
I tried the version suggested in the documentation 2.5.0 and then some other 2.x.0 versions, but none seem to work, looks like most of them are not out of preview. Drop a comment if 2.8 or any newer version is out of preview by the time you are reading it.

3. Add Performance Testing assembly definitions to your newly created tests assembly definition
Packages/com.unity.test-framework.performance/Runtime/Unity.PerformanceTesting.asmdef
Packages/com.unity.test-framework.performance/Editor/Unity.PerformanceTesting.Editor.asmdef

After these steps, you should be able to run a performance test inside the editor and see the results in the Test Report window (Window – Analysis – Performance Test Report).
For the sake of the setup tutorial I used this simple performance test placeholder:

[Test, Performance]
public void Test()
{
    Measure.Method(Counter).Run();
}

private static void Counter()
{
    var sum = 0;
    for (var i = 0; i < 10000000; i++)
    {
        sum += i;
    }
}

And how the report looks inside the editor:

So now let’s move on to the interesting part.

Running performance tests on a target platform

The Unity blog post says about an important requirement, however running performance tests via the editor on the target platform doesn’t produce a performance report (still true for version 2.8.0).

After testing thoroughly with different versions of the Performance Testing package and Unity editor I found the following results:
On Unity 2019.3 a report is produced when running tests on a device via the terminal. A report is also available inside the editor via the Test Report window when running tests using the Run All Tests (Android) button inside the Test Runner window.
However, on Unity 2020.3/2021.3/2022.1 no report is created using both the terminal and the editor to run tests on a target device. I created a bug report hopefully it will be resolved soon, will update the post once it is fixed.

So if you are still using 2019 LTS then you are completely fine running your performance tests and checking the results right inside the editor. But if you need to maintain some history of your test runs as well as automate this process you should prefer running them via the terminal.
To make it work we need to use another piece of advice from that blog post: to run a performance test suite via the command line and pass the path to save the test report as an argument. In addition pass -logFile as it helps to figure out what’s wrong if there are no test results after you run the command.
If you take the suggested command from the blog post

Unity.exe -runTests [-batchmode] -projectPath
C:\XRAutomatedTests-2018.2\PerformanceTests\UnityPerformanceBenchmark -testPlatform Android -buildTarget Android -playergraphicsapi=OpenGLES3 -mtRendering -scriptingbackend=mono -testResults
C:\PerfTests\results\PerfBenchmark_Android_OpenGLES3_MtRendering_Mono.xml -logfile
C:\PerfTests\logs\PerfBenchmark_Android_OpenGLES3_MtRendering_Mono_UnityLog.txt

and replace “Android” with “Standalone” for -testPlatform and -buildTarget you would get Test platform not found error. Googling hadn’t helped, so I decided to dig into their code, and here is what we can find:

static BuildTarget? SetFilterAndGetBuildTarget(string testPlatform, Filter filter)
{
    BuildTarget? buildTarget = null;
    if (testPlatform.ToLower() == "editmode")
    {
        filter.testMode = TestMode.EditMode;
    }
    else if (testPlatform.ToLower() == "playmode")
    {
        filter.testMode = TestMode.PlayMode;
    }
    else
    {
        try
        {
            buildTarget = (BuildTarget)Enum.Parse(typeof(BuildTarget), testPlatform, true);
            filter.testMode = TestMode.PlayMode;
        }
        catch (ArgumentException)
        {
            throw new SetupException(SetupException.ExceptionType.PlatformNotFound, testPlatform);
        }
    }
    return buildTarget;
}

/// <summary>
/// A flag indicating whether to run Edit Mode or Play Mode tests.
/// </summary>
[Flags]
public enum TestMode
{
    /// <summary>
    /// Run EditMode tests.
    /// </summary>
    EditMode = 1 << 0,
    /// <summary>
    /// Run PlayMode tests.
    /// </summary>
    PlayMode = 1 << 1
}

Turns out -testPlatform argument accepts “EditMode” and “PlayMode” strings, or it can fallback to BuildTarget type which doesn’t include “Standalone”. You can decompile UnityEditor.CoreModule.dll and check BuildTarget.cs to see available types:

/// <summary>
///   <para>Target build platform.</para>
/// </summary>
[NativeType("Runtime/Serialize/SerializationMetaFlags.h")]
public enum BuildTarget
{
  NoTarget = -2, // 0xFFFFFFFE
  [Obsolete("BlackBerry has been removed in 5.4")] BB10 = -1, // 0xFFFFFFFF
  [Obsolete("Use WSAPlayer instead (UnityUpgradable) -> WSAPlayer", true)] MetroPlayer = -1, // 0xFFFFFFFF
  StandaloneOSX = 2,
  [Obsolete("Use StandaloneOSX instead (UnityUpgradable) -> StandaloneOSX", true)] StandaloneOSXUniversal = 3,
  [Obsolete("StandaloneOSXIntel has been removed in 2017.3")] StandaloneOSXIntel = 4,
  StandaloneWindows = 5,
  [Obsolete("WebPlayer has been removed in 5.4", true)] WebPlayer = 6,
  [Obsolete("WebPlayerStreamed has been removed in 5.4", true)] WebPlayerStreamed = 7,
  iOS = 9,
  [Obsolete("PS3 has been removed in >=5.5")] PS3 = 10, // 0x0000000A
  [Obsolete("XBOX360 has been removed in 5.5")] XBOX360 = 11, // 0x0000000B
  Android = 13, // 0x0000000D
  [Obsolete("StandaloneLinux has been removed in 2019.2")] StandaloneLinux = 17, // 0x00000011
  StandaloneWindows64 = 19, // 0x00000013
  WebGL = 20, // 0x00000014
  WSAPlayer = 21, // 0x00000015
  StandaloneLinux64 = 24, // 0x00000018
  [Obsolete("StandaloneLinuxUniversal has been removed in 2019.2")] StandaloneLinuxUniversal = 25, // 0x00000019
  [Obsolete("Use WSAPlayer with Windows Phone 8.1 selected")] WP8Player = 26, // 0x0000001A
  [Obsolete("StandaloneOSXIntel64 has been removed in 2017.3")] StandaloneOSXIntel64 = 27, // 0x0000001B
  [Obsolete("BlackBerry has been removed in 5.4")] BlackBerry = 28, // 0x0000001C
  [Obsolete("Tizen has been removed in 2017.3")] Tizen = 29, // 0x0000001D
  [Obsolete("PSP2 is no longer supported as of Unity 2018.3")] PSP2 = 30, // 0x0000001E
  PS4 = 31, // 0x0000001F
  [Obsolete("PSM has been removed in >= 5.3")] PSM = 32, // 0x00000020
  XboxOne = 33, // 0x00000021
  [Obsolete("SamsungTV has been removed in 2017.3")] SamsungTV = 34, // 0x00000022
  [Obsolete("Nintendo 3DS support is unavailable since 2018.1")] N3DS = 35, // 0x00000023
  [Obsolete("Wii U support was removed in 2018.1")] WiiU = 36, // 0x00000024
  tvOS = 37, // 0x00000025
  Switch = 38, // 0x00000026
  Lumin = 39, // 0x00000027
  Stadia = 40, // 0x00000028
  CloudRendering = 41, // 0x00000029
  [Obsolete("GameCoreScarlett is deprecated, please use GameCoreXboxSeries (UnityUpgradable) -> GameCoreXboxSeries", false)] GameCoreScarlett = 42, // 0x0000002A
  GameCoreXboxSeries = 42, // 0x0000002A
  GameCoreXboxOne = 43, // 0x0000002B
  PS5 = 44, // 0x0000002C
  EmbeddedLinux = 45, // 0x0000002D
}

As for me, this API looks really inconvenient: not only does it accept different types of parameters, but it also accepts BuildTarget parameter of a type that differs from another command-line argument that is literally called “-buildTarget”. At least would be nice to have in the documentation the list of accepted parameters like the one I added above. BUT I must point out that to use any platform from BuildTarget.cs you must have a module for that platform installed, otherwise there would be no report produced. “PlayMode” and “EditMode” work for any platform though, because it just launches the editor and runs tests there yielding the test report for a run inside the editor disregarding the build target argument. So for edit and play mode, it is easier to just run it via Test Runner and read the report via the Test Report inspector window.

I use Windows PowerShell which goes bundled with Windows 10 and 11. I also changed the directory to one where Unity is installed so I can use Unity.exe directly. By default, Unity.exe goes to C:/Program Files/UnityHub/Editor/UNITY_VERSION/Editor, if it differs for you, you can go to Unity Hub – Installs – Menu – Show in Explorer and copy the path in the opened explorer window.

Then use this command to change the directory where Unity.exe is located, in my case:

cd "C:/Program Files/UnityHub/Editor/2021.2.3f1/Editor"

After that you can run the performance test suit on the target platform with the following command (Android in my case, but you can replace it with any other BuildTarget provided above):

./Unity.exe -runTests -batchMode -projectPath D:/Development/Unity/PerformanceTesting -testPlatform Android -buildTarget Android -mtRendering -scriptingbackend=il2cpp -testResults D:/Development/Unity/ PerformanceTesting/TestResults/TestReport.xml -logfile D:/Development/Unity/PerformanceTesting/TestLogs/TestLog.txt

As a result you would get TestReport.xml.

Reading Performance Testing Report

To make something more readable from TestReport.xml, you should use another tool made by Unity: Performance Benchmark Reporter. I would say that their wiki is much better, but for the sake of this tutorial, I will add what I did too. Download this tool from their GitHub repo here. Then download .NET Core SDK from Microsoft website here, unless you have it already. The URL leads to the version I used, but you can try to use a newer one if it is available.
Now change the directory to the one you downloaded Performance Benchmark Reporter to. In my case it is:

cd "D:/Development/Unity/UnityPerformanceBenchmarkReporter_1_2_0"

And then use this command to process your test results (again replace the results path and the report path with yours)

dotnet UnityPerformanceBenchmarkReporter.dll --results="D:/Development/UnityPerformanceTesting/TestResults" --reportdirpath="D:/Development/UnityPerformance/TestingTest/Results/Reports"

And here is the absolutely stunning report of the run:

Testables

Bonus info you might be missing from the documentation. If you add testables in manifest.json, you would be able to run tests shipped with packages. That’s how you add tests from the performance testing package:

{
  "dependencies": {
    ...
    "com.unity.test-framework.performance": "2.8.0-preview"
  },
  "testables": [
    "com.unity.test-framework.performance"
  ]
}

And now you can see them in the test runner and run them inside the editor or on a target platform.

References:

1. https://blog.unity.com/technology/performance-benchmarking-in-unity-how-to-get-started
   
2. https://docs.unity3d.com/Packages/com.unity.test-framework.performance@2.8/manual/index.html