UEST (Unreal Engine Suckless Testing)

This project aims to provide a testing framework for Unreal Engine that does not suck.

Goals

• Simple things should be simple, complex things should be possible

• Extensibility

• Async execution support

• IDE and CI first-class support

• User-visible API ergonomics is more important than internal implementation

Dependencies

This project relies on C++20 features. Tested against Unreal Engine 5.4.

Usage

Defining tests

Simple test

#include "UEST.h"

TEST(MyFancyTest)
{
    // test body goes here
    ASSERT_THAT(...);
}

Test class with multiple methods

#include "UEST.h"

TEST_CLASS(MyFancyTestClass)
{
    TEST_METHOD(Method1)
    {
        // test body goes here
        ASSERT_THAT(...);
    }

    TEST_METHOD(Method2)
    {
        // test body goes here
        ASSERT_THAT(...);
    }

    // put helper fields or methods here
}

If you want to execute a common piece of logic before and after each test method in a test class, you can do that using BEFORE_EACH/AFTER_EACH macros:

#include "UEST.h"

TEST_CLASS(MyFancyTestClass)
{
    BEFORE_EACH()
    {
        // Place code that will be executed before each test method of this class
    }

    AFTER_EACH()
    {
        // Place code that will be executed after each test method of this class
    }

    ...
}

Assertions

All UEST assertions are done through ASSERT_THAT(Expression, Matcher). Failed assertion performs return, aborting further test execution.

Available matchers

ASSERT_THAT(Value, Is::True)

Tests that Value is true.

ASSERT_THAT(Value, Is::False)

Tests that Value is false.

ASSERT_THAT(Value, Is::Null)

Tests that Value is nullptr.

ASSERT_THAT(Value, Is::EqualTo(Expected))

Tests that Value is equal to Expected.

ASSERT_THAT(Value, Is::LessThan(OtherValue))

Tests that Value is less than OtherValue.

ASSERT_THAT(Value, Is::LessThanOrEqualTo(OtherValue)) or ASSERT_THAT(Value, Is::AtMost(OtherValue)

Tests that Value is less than or equal to OtherValue.

ASSERT_THAT(Value, Is::GreaterThan(OtherValue))

Tests that Value is greater than OtherValue.

ASSERT_THAT(Value, Is::GreaterThanOrEqualTo(OtherValue)) or ASSERT_THAT(Value, Is::AtLeast(OtherValue)

Tests that Value is greater than or equal to OtherValue.

ASSERT_THAT(Value, Is::Zero)

Shortcut for ASSERT_THAT(Value, Is::EqualTo(0)).

ASSERT_THAT(Value, Is::Positive)

Shortcut for ASSERT_THAT(Value, Is::GreaterThan(0)).

ASSERT_THAT(Value, Is::Negative)

Shortcut for ASSERT_THAT(Value, Is::LessThan(0)).

ASSERT_THAT(Value, Is::InRange(From, To))

Tests that Value is greater than or equal to From and is less than or equal to To.

ASSERT_THAT(Value, Is::Empty)

Tests that Value is empty using its IsEmpty() method. Use this for FString or collections (TArray, TMap, etc).

ASSERT_THAT(Value, Is::Valid)

Tests that Value is valid using its IsValid() method. Use this for TSharedPtr, TWeakObjectPtr or TWeakPtr.

ASSERT_THAT(Value, Is::NaN)

Tests that Value is floating NaN. Supports both float and double.

❗	Because of the bug in Clang template type deduction in versions older than 19.0, matchers with parameters (LessThan, GreaterThan, EqualTo and so on) require explicit template type specification: ASSERT_THAT(0, Is::LessThan<int>(1)).

You can also negate assertions using ASSERT_THAT(Value, Is::Not::<matcher>).

Negated assertion example:

ASSERT_THAT(Value, Is::Not::Null);

TODO: Document how to write custom matchers

Running tests

UEST is seamlessly integrated into Unreal Engine testing infrastructure, so you can run them using standard Session Frontend or IDE integration plugins.

Testing game worlds

UEST provides a convenient way to test game worlds, both standalone and multiplayer.

Basic usage

TEST(MyGame, SimpleMultiplayerTest)
{
	auto Tester = FScopedGame().Create();

	// You can create a dedicated server
	UGameInstance* Server = Tester.CreateGame(EScopedGameType::Server, TEXT("/Engine/Maps/Entry"));

	// You can connect a client to it
	UGameInstance* Client = Tester.CreateClientFor(Server);
	ASSERT_THAT(Client, Is::Not::Null);

	// Actually, you can connect as many clients as you want!
	for (int32 Index = 0; Index < 10; ++Index)
	{
		Tester.CreateClientFor(Server);
	}

	// You can access game worlds
	UWorld* ServerWorld = Server->GetWorld();
	ASSERT_THAT(ServerWorld, Is::Not::Null);
	UWorld* ClientWorld = Client->GetWorld();
	ASSERT_THAT(ClientWorld, Is::Not::Null);

	// You can access actors in worlds
	APlayerController* ClientPC = ClientWorld->GetFirstPlayerController();
	ASSERT_THAT(ClientPC, Is::Not::Null);

	// You can lookup matching replicated actors in paired worlds
	APlayerController* ServerPC = Tester.FindReplicatedObjectIn(ClientPC, Server->GetWorld());
	ASSERT_THAT(ServerPC, Is::Not::Null);

	// You can advance game time
	Tester.Tick(1);

	// You can shut down individual game instances
	Tester.DestroyGame(Client);

	// You can also create standalone game worlds
	UGameInstance* Standalone = Tester.CreateGame(EScopedGameType::Client, TEXT("/Engine/Maps/Entry"));

	// Tester automatically cleans everything up when goes out of scope
}

Further development plans

• More matchers

• Add ASSERT_MULTIPLE that allows performing multiple assertions without interrupting execution between them, also known as "soft assertions".

• Add API to disable tests (with EAutomationTestFlags::Disabled under the hood)

• Add API for asynchronous/latent tests

Analysis of existing Unreal Engine solutions

As of 5.4, Unreal Engine has 4 (FOUR, that’s not a typo) APIs for writing tests and all are very far from being good for various reasons.

Let’s analyze them one-by-one.

Automation Test

IMPLEMENT_SIMPLE_AUTOMATION_TEST(FMyTest, "MyGame.MyTest", EAutomationTestFlags::ApplicationContextMask | EAutomationTestFlags::ProductFilter)
bool FMyTest::RunTest(const FString& Parameters)
{
    UTEST_TRUE_EXPR(true);

    return true;
}

The good

• VisualStudio and JetBrains Rider know how to run this.

• UTEST* macros interrupt test execution (though these macros are useless for all other test frameworks because of non-void return false;)

The bad

• Assertions do not capture expression that is being tested. You have to write descriptive messages by hand.

• Overcomplicated way to add multiple tests with common logic.

The ugly

• You need to write your test name three times as if it isn’t clear enough what test name actually is.

• Requires lots of typing. Macro could easily declare RunTest signature automatically. Also, almost nobody wants to use custom flags.

• You must return a bool from the test. If test reports an error, it should be marked as failing. If there are no errors, it should be marked as successful. This bool adds a completely useless (and even harmful) way to fail without a message.

• Nontrivial assertions (like UTEST_EQUAL_EXPR) are unable to print exact values of actual/expected.

• Inadequate support for async tests. As soon as something becomes async, test body transforms into ADD_LATENT_AUTOMATION_COMMAND monster without an easy way of passing data between commands.

Automation Spec

DEFINE_SPEC(MySpec, "MyGame.MySpec", EAutomationTestFlags::ProductFilter | EAutomationTestFlags::ApplicationContextMask)
void MyCustomSpec::Define()
{
    TestTrue(TEXT("True should be true"), true);
}

The good

• Understood by VS and Rider

• void return type

• Better async execution support, but not the best. Programming community developed much better techniques than callback hell.

• May attract people that are familiar with spec-based approach from other areas.

The ugly

• Declaring test name three times again

• Flags again

• No builtin way to interrupt test execution when assertion fails, so people have to invent their own assrtion macros.

Low Level

TEST_CASE("MyGame.MyTest", "[ApplicationContextMask][ProductFilter]")
{
    REQUIRE(true);
}

The good

• Test name is written only once…​ Well, no.

  The caveat is that TEST_CASE macro uses a very broken way to generate unique class names. They collide across compilation units and namespaces, and you end up asking yourself "why my test doesn’t register at all". Instead, Epics tell users to use TEST_CASE_NAMED, where you need to write test name twice. That way, you end up with the same test class name collision chances as other approaches.

The bad

• Not understood by Rider (RIDER-110897)

The ugly

• String tags, really? I am more than sure people will make typos and spend multiple hours trying to figure out why their test doesn’t run.

• Assertions are a joke.

  Just look at it:

  #define REQUIRE(Expr) if (!(Expr)) { FAutomationTestFramework::Get().GetCurrentTest()->AddError(TEXT("Required condition failed, interrupting test")); return; }

  Yep, you guessed it right, all you will get for failed assertion is "Required condition failed, interrupting test"

CQTest

TEST(MyTest, "MyGame")
{
    ASSERT_THAT(IsTrue(true));
}

The good

• Test name is written only once

• No more flags

• AreEqual assertion is extensible and can print arbitrary types in error messages

• void test body

• Nice way to add multiple test methods to a single test class

The bad

• Not understood by Rider (RSCPP-36039). Not sure about VS, would not be surprised if situation is the same.

• Async execution is as bad as in Automation Test style

• clang-format is unable to properly indent TEST_CLASS with nested TEST_METHOD

The ugly

• Assertions do not capture tested expression. Expected condition to be true., seriously?

• Inadequate way to add custom assertions. You need to use custom macros instead of TEST and TEST_CLASS because they hardcode FNoDiscardAsserter. And this framework claims they are about composition instead of inheritance! There was absolutely zero reason to tie test class to a single asserter. Asserter could easily be absolutely external class to the test itself, see NUnit for example.