This is a test framework based on xUnit.net v3 that allows defining a test case as a sequence of steps and sub-steps. These steps and sub-steps are based on code defined either as named methods or closures.
The framework is designed for automated tests that are written based on test cases.
Consider a test case based on the template at http://www.softwaretestinghelp.com/test-case-template-examples:
Test case ID: TC001
Test case summary: Log in to website
Precondition: The user has an account on the website
| Step | Test step | Test data | Expected result | Actual result | Status |
|------|------------------------------------------------|---------------------|-----------------------|---------------|--------|
| 1 | Open a web browser and navigate to the website | https://my.site.com | | | |
| 2 | Enter the user name | user | | | |
| 3 | Enter password | password | | | |
| 4 | Click the Login button | | The user is logged in | | |
This test case can be defined as a test class:
[TestCase("TC001")] // this is cosmetic
public class TestCase1
{
// define test data
private const string WebsiteUrl = "https://my.site.com";
private const string UserName = "user";
private const string Password = "password";
[Summary("Log in to website")]
public void LoginToWebSite()
{
// describe the test case steps
TestCase.Current.Descriptor
.AddStep(StepType.Precondition, "The user has an account on the website", CreateUserAccount)
.AddStep(StepType.Input, "The user has an account on the website", OpenWebBrowser)
.AddStep(StepType.Input, "Enter the user name", EnterUserName)
.AddStep(StepType.Input, "Enter the password", EnterPassword)
.AddStep(StepType.Input, "Click the Login button", LogIn)
.AddStep(StepType.ExpectedResult, "The user is logged in", VerifyUserIsLoggedIn)
;
}
private void CreateUserAccount() {...}
private void OpenWebBrowser() {...} // use WebsiteUrl
private void EnterUserName() {...} // use Username
private void EnterPassword() {...} // use Password
private void LogIn() {...}
private void VerifyUserIsLoggedIn() {...}
}The same test case can be defined in v1 format using step attributes. This requires using Automation.TestFramework.SourceGenerators:
[TestCase("TC001")] // this is cosmetic
public partial class TestCase1
{
// define test data
private const string WebsiteUrl = "https://my.site.com";
private const string UserName = "user";
private const string Password = "password";
[Summary("Log in to website")]
public partial void LoginToWebSite();
[Precondition(1, "The user has an account on the website")]
private void CreateUserAccount() {...}
[Input(1, "The user has an account on the website")]
private void OpenWebBrowser() {...} // use WebsiteUrl
[Input(2, "Enter the user name")]
private void EnterUserName() {...} // use Username
[Input(3, "Enter the password")]
private void EnterPassword() {...} // use Password
[Input(4, "Click the Login button")]
private void LogIn() {...}
[ExpectedResult(4, "The user is logged in")]
private void VerifyUserIsLoggedIn() {...}
}Note: this changes a fundamental concept of xUnit, where a test method is viewed as a test case and the test class is viewed as a collection of related test cases. For us, the test case is the test class.
Create a new xUnit test project as explained in https://xunit.net/docs/getting-started/v3/cmdline#create-the-unit-test-project.
Then add the NuGet package Automation.TestFramework.Dynamic.
The test framework has the same target framework as xUnit v3:
- .NET Standard 2.0
- .NET Framework 4.7.2
- .NET 6.0
The test method marked as [Summary] is automatically discovered by the xUnit test runner, as it was a [Fact].
The actual test case steps are not discovered until the test is executed, which is why they are called dynamic tests.
Run the Summary test to start discovering the actual test case steps based on closures.
These closures are executed by the test framework in the order in which they were added when describing the test case.
Each closure is wrapped inside a test result linked to the Summary test. The display names of these tests are given by the description used when adding the steps.
This way, the test report matches the test case definition as closely as possible.
The test framework uses attributes to identify test cases:
[TestCase]: identifies a test class as a test case. This is purely cosmetic and can be omitted.[Summary]: used as the 'entry point' of the test case that can be discovered by the test runner. Each test case class should have a single test method marked as Summary.
The test case is defined as a sequence of steps. A step is defined by:
- the step type
- a description (used as display name)
- the code that implements the step
The test framework defines 5 types of steps:
- Setup
- Precondition
- Input
- Expected result
- Cleanup
The code that implements the step can be anything. For example:
- a method from the test class
- a static method from another class
- a closure
Step attributes become available when using Automation.TestFramework.SourceGenerators. They correspond to the 5 types of steps mentioned above:
[Setup][Precondition][Input][ExpectedResult][Cleanup]
The steps are executed in the order they are added to the current test case.
If a step fails before other steps are executed, then the other steps are executed as skipped tests, except for the Cleanup steps. These are always executed.
Steps are added to the current test case inside the Summary method.
[Summary]
public void Summary
{
TestCase.Current.Descriptor
.AddStep(StepType.Input, "This is the input", Input)
}
private void Input() { ... }If the code that implements the step returns a Task or ValueTask, then the step is considered async.
Async steps are added using .AddAsyncStep():
[Summary]
public void Summary
{
TestCase.Current.Descriptor
.AddStep(StepType.Input, "This is the input", Input)
.AddAsyncStep(StepType.ExpectedResult, "This is the expected result", ExpectedResult);
}
private void Input() { ... }
private Task ExpectedResult() { ... }A test case can have both sync and async steps.
Each step can have sub-steps. These are dynamic tests that run during the current step execution.
private void Input()
{
int value = 1;
Step.Current.Descriptor
.AddSubStep("Phase 1", () => { value = 2; })
.AddAsyncSubStep("Phase 2 (async)", async() => { await ... })
.Execute();
Assert.Equal(2, value);
value = 3;
Step.Current.Descriptor
.ExecuteubStep("Phase 3", () => { Assert.Equal(3, value); })
.ExecuteAsyncSubStep("Phase 4 (async)", async() => { await ... });
}This is similar to the way steps are added to the current test case.
There is a difference though: the execution of the sub-steps can be mixed with the execution of the current step, by calling Execute methods. This allows executing a sequence of sub-steps or even one sub-step at a time.
If no Execute method is called after adding all sub-steps, then the test framework executes all of them in order.
If a sub-step fails before other sub-steps are executed, then the other sub-steps are executed as skipped tests (unless the failed sub-step is a verification, see below).
These are special sub-steps used in ExpectedResult steps when they consist of multiple assertions whose outcomes determine if the test step passes or fails.
For example, consider a basic test case such as:
Precondition: User logs in
Input: User goes to the Profile page
Expected result: The user display name and email are correct
The expected result verifies 2 things: the user display name and the email address. They both need to be correct for the test to pass. They both need to be visible in the test report, in case one of them fails. The failure may be considered critical, or not.
For the above test case, assume that when either of the user display name / email is incorrect then the other one does not need to be verified - the test fails anyway. This can be written as:
private void ExpectedResult()
{
ExpectedResultStep.Current.Descriptor
.Assert("Expect the user display name is correct", () => Assert.[...])
.AssertAsync("Expect the email is correct", async () =>
{
await ...
Assert.[...]
});
}This code produces two tests for the test step.
When they both pass then the test report contains:
[3/3] Expected result] 1. The user display name and email are correct - passed
[3/3] [Expected result] 1.1. Expect the user display name is correct - passed
[3/3] [Expected result] 1.2. Expect the email is correct - passed
When an assertion fails then the failure is shown in the test report, the next assertions are not executed at all, and the test step fails with a specific error. I.e. when the user display name is not correct:
[3/3] [Expected result] 1. The user display name and email are correct - failed: One or more of the expected results did not match. 1 assertion(s) were skipped.
[3/3] [Expected result] 1.1. Expect the user display name is correct - failed
For the above test case, assume that when one of the user display name / email is incorrect then the other needs to be checked too before the test fails. This can be written as:
[ExpectedResult]
private void ExpectedResult()
{
ExpectedResultStep.Current.Descriptor
.Verify("Expect the user display name is correct", () => Assert.[...])
.VerifyAsync("Expect the email is correct", async () =>
{
await ...
Assert.[...]
});
}When a verification fails, then the failure is shown in the test report and the next assertion/verification is executed. I.e. when the user display name is not correct but the email is, then:
[3/3] [Expected result] 1. The user display name and email are correct - failed: One or more of the expected results did not match
[3/3] [Expected result] 1.1. Expect the user display name is correct - failed
[3/3] [Expected result] 1.2. Expect the email is correct - passed
The [Summary] attribute supports specifying a description that is used as the test display name.
If this description is missing then the method name is used - but not as is. It is 'humanized' using https://github.com/Humanizr/Humanizer.
[Summary]
public void LoginToWebsite() {...}The name of this test as shown in the test report will be "Log in to website".
The test framework raises events that the user code can handle using EventSource.Instance.
For example, consider this scoped class:
using Automation.TestFramework.Dynamic; // needed for EventSource
public class EventHandlers : IDisposable
{
public EventHandlers()
{
EventSource.Instance.StepError += OnStepError;
}
public void Dispose()
{
EventSource.Instance.StepError -= OnStepError;
}
private void OnStepError(object sender, Exception e)
{
// semder is the instance used to invoke the step method or closure, or null if static
// e is the exception
}
}This class can be used with any of the xUnit fixtures, such as:
AssemblyFixture(Type)ICollectionFixture<>IClassFixture<>