Blogpost about support of MTP across test frameworks

[Evangelink]

Dear framework authors @thomhurst @bradwilson @OsirisTerje @rprouse @Alxandr @haf,

I would like to write a blogpost about the support of MTP (Microsoft.Testing.Platform) in the various test frameworks. Here is the draft about it and I would really like to get your feedback about it.

Reflecting on a Year of Microsoft.Testing.Platform

A year ago, we launched Microsoft.Testing.Platform as part of the MSTest Runner announcement. Our goal was to create a reliable testing platform for .NET projects, focused on extensibility and modularity. Over the past year, we've collaborated with the community and test framework authors to enhance this platform, enriching the ecosystem and providing better tools and integrations for developers.
In this post, we’ll highlight the test frameworks that have embraced Microsoft.Testing.Platform, share their unique characteristics, and provide resources for getting started.

Get Started with Microsoft.Testing.Platform

• Overview
• Comparison with VSTest
• Available extensions
• GitHub repository
• Microsoft.Testing.Platform for extension authors

Enabling Microsoft.Testing.Platform in your favorite test framework

The test frameworks are ordered alphabetically.

Expecto

Expecto aims to make it easy to test CLR based software; be it with unit tests, stress tests, regression tests or property based tests. Expecto tests are parallel and async by default, so that you can use all your cores for testing your software. This also opens up a new way of catching threading and memory issues for free using stress testing.

With the release of vTO BE DEFINED, YoloDev.Expecto.TestSdk now supports running test through the new testing platform. To opt-in, simply edit your projects to set <EnableExpectoRunner>true</EnableExpectoRunner> and <OutputType>Exe</OutputType>.

Sample Application

Contoso.fsproj:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
	<TargetFramework>net8.0</TargetFramework>
    <!-- Set the output to exe and set EnableExpectoRunner to true to enable the new platform -->
    <EnableExpectoRunner>true</EnableExpectoRunner>
    <OutputType>exe</OutputType>
    <!-- 
        To enable the Microsoft.Testing.Platform 'dotnet test' experience, add the following property:
        <TestingPlatformDotnetTestSupport>true</TestingPlatformDotnetTestSupport>
        Please carefully read https://learn.microsoft.com/dotnet/core/testing/unit-testing-platform-integration-dotnet-test
        to understand limitations around mixed test platform.
    -->
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="YoloDev.Expecto.TestSdk" VersionOverride="$(PackageVersion)" />
  </ItemGroup>

  <ItemGroup>
    <Compile Include="Test.fs" />
  </ItemGroup>

</Project>

Test.fs:

open Expecto

let tests =
    testList "Calculator Tests" [
        test "Add function returns sum" {
            let calculator = Calculator()
            let result = calculator.Add(1, 2)
            Expect.equal result 3 "Expected sum to be 3"
        }
    ]

[<EntryPoint>]
let main argv =
    runTestsWithArgs defaultConfig argv tests

MSTest

MSTest, Microsoft Testing Framework, is a fully supported, open source, and cross-platform test framework with which to write tests targeting .NET Framework, .NET Core, .NET, UWP, and WinUI on Windows, Linux, and Mac.

With the release of v3.2.0, MSTest.TestAdapter now supports running test through the new testing platform. To opt-in, simply edit your projects to set <EnableMSTestRunner>true</EnableMSTestRunner> and <OutputType>Exe</OutputType>.

Sample Application

Contoso.Tests.csproj:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>        
    <!-- Set the output to exe and set EnableMSTestRunner to true to enable the new platform -->
    <EnableMSTestRunner>true</EnableNUnitRunner>
    <OutputType>Exe</OutputType>
    <!-- 
        To enable the Microsoft.Testing.Platform 'dotnet test' experience, add the following property:
        <TestingPlatformDotnetTestSupport>true</TestingPlatformDotnetTestSupport>
        Please carefully read https://learn.microsoft.com/dotnet/core/testing/unit-testing-platform-integration-dotnet-test
        to understand limitations around mixed test platform.
    -->
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="MSTest" Version="3.7.1" />
  </ItemGroup>

</Project>

Test.cs:

[TestClass]
public class CalculatorTests
{
    [TestMethod]
    public void Add_WhenCalled_ReturnsSum()
    {
        var calculator = new Calculator();
        var result = calculator.Add(1, 2);

        Assert.AreEqual(3, result);
    }
}

NUnit

NUnit is a unit-testing framework for all .NET languages. Initially ported from JUnit, the current production release has been completely rewritten with many new features and support for a wide range of .NET platforms.

With the release of v5, NUnit3TestAdapter now supports running test through the new testing platform. To opt-in, simply edit your projects to set <EnableNUnitRunner>true</EnableNUnitRunner> and <OutputType>Exe</OutputType>.

Sample Application

Contoso.Tests.csproj:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>        
    <!-- Set the output to exe and set EnableExpectoRunner to true to enable the new platform -->
    <EnableNUnitRunner>true</EnableNUnitRunner>
    <OutputType>Exe</OutputType>
    <!-- 
        To enable the Microsoft.Testing.Platform 'dotnet test' experience, add the following property:
        <TestingPlatformDotnetTestSupport>true</TestingPlatformDotnetTestSupport>
        Please carefully read https://learn.microsoft.com/dotnet/core/testing/unit-testing-platform-integration-dotnet-test
        to understand limitations around mixed test platform.
    -->
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.12.0" />
    <PackageReference Include="NUnit" Version="4.3.2" />
    <PackageReference Include="NUnit.Analyzers" Version="4.6.0"/>
    <PackageReference Include="NUnit3TestAdapter" Version="5.0.0" />
  </ItemGroup>

</Project>

Test.cs:

public class CalculatorTests
{
    [Test]
    public void Add_WhenCalled_ReturnsSum()
    {
        var calculator = new Calculator();
        var result = calculator.Add(1, 2);

        Assert.That(result,Is.EqualTo(3));
    }
}

TUnit

TUnit is a modern, flexible and fast testing framework for C#. With Native AOT and Trimmed Single File application support included!

Sample Application

Contoso.csproj:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <OutputType>Exe</OutputType>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="TUnit" Version="0.6.117" />
  </ItemGroup>

</Project>

Test1.cs:

public class CalculatorTests
{
    [Test]
    public async Task Add_WhenCalled_ReturnsSum()
    {
        var calculator = new Calculator();
        var result = calculator.Add(1, 2);

        await Assert.That(result).IsEqualTo(3);
    }
}

xUnit.net

xUnit.net is a free, open source, community-focused unit testing tool for the .NET Framework. Written by the original inventor of NUnit v2, xUnit.net is the latest technology for unit testing C#, F#, VB.NET and other .NET languages.

Sample Application

With the release of xunit.v3, xUnit.net now supports running test through the new testing platform. To opt-in, simply edit your projects to set <UseMicrosoftTestingPlatformRunner>true</UseMicrosoftTestingPlatformRunner>.

Sample Application

Contoso.Tests.csproj:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>                
    <OutputType>Exe</OutputType>
    <!-- Set UseMicrosoftTestingPlatformRunner to true to enable the new platform -->
    <UseMicrosoftTestingPlatformRunner>true</UseMicrosoftTestingPlatformRunner>
    <!-- 
        To enable the Microsoft.Testing.Platform 'dotnet test' experience, add the following property:
        <TestingPlatformDotnetTestSupport>true</TestingPlatformDotnetTestSupport>
        Please carefully read https://learn.microsoft.com/dotnet/core/testing/unit-testing-platform-integration-dotnet-test
        to understand limitations around mixed test platform.
    -->
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="xunit.v3" Version="1.0.1" />
    <!-- 
      Optionally, to preserve compatibility with VSTest, add the following 2 packages
      <PackageReference Include="xunit.runner.visualstudio" Version="3.0.1" />
      <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.12.0" />
      -->
  </ItemGroup>

</Project>

Test.cs:

public class CalculatorTests
{
    [Fact]
    public void Add_WhenCalled_ReturnsSum()
    {
        var calculator = new Calculator();
        var result = calculator.Add(1, 2);

        Assert.Equal(3, result);
    }
}

Looking Ahead

We would like to extend our heartfelt appreciation to the framework authors we have collaborated with and continue to work closely with.

We are thrilled to witness the ongoing evolution of this platform and its ability to empower developers. We eagerly anticipate numerous contributions from the community and look forward to the innovative extensions that will be created.

If you haven't already, we encourage you to explore the platform, experiment with your preferred framework, and share your feedback.

Together, let's continue to build an outstanding .NET testing ecosystem!

[thomhurst]

Just a couple of things:

• Isn't the latest NUnit production release v4? Not v3.
• The code for TUnit looks like MSTest code - It's invalid code for TUnit.

Correct would be:

public class CalculatorTests
{
    [Test]
    public async Task Add_WhenCalled_ReturnsSum()
    {
        var calculator = new Calculator();
        var result = calculator.Add(1, 2);

        await Assert.That(result).IsEqualTo(3);
    }
}

• Did you forget to include MSTest?

[Evangelink]

Isn't the latest NUnit production release v4? Not v3.

You are right, I have blindly copied the intro text of each framework website. It might also be worth for me to mention the version adding support for MTP for each framework.

The code for TUnit looks like MSTest code - It's invalid code for TUnit.

Sorry I just tried to write from doc but didn't try to have a proper run. Thanks for the fix.

Did you forget to include MSTest?

Ahahah yeah xD

@thomhurst anything you would like to add about TUnit? Are you fine with such blog post?

[Evangelink]

I have added a csproj example using version 0.6.117

[thomhurst]

I don't really know what to say about it haha. Yeah I'm fine with a blog post 😄

[bradwilson]

My list:

• The product name is xUnit.net, not xUnit. Please do not shorten the name.

• This is not necessary:

  and <OutputType>Exe</OutputType>

  This is because all xUnit.net v3 test projects must have this. It is not a unique requirement for MTP support. Our projects are always executable. So, it should be in the sample .csproj, but not grouped under "this is needed for MTP".

  For reference, this is our default template <PropertyGroup>, which you can see already includes <OutputType>Exe</OutputType>:

  <PropertyGroup>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <OutputType>Exe</OutputType>
    <RootNamespace>tests</RootNamespace>
    <TargetFramework>net8.0</TargetFramework>
    <!--
    To enable the Microsoft Testing Platform 'dotnet test' experience, add property:
      <TestingPlatformDotnetTestSupport>true</TestingPlatformDotnetTestSupport>
  
    To enable the Microsoft Testing Platform native command line experience, add property:
      <UseMicrosoftTestingPlatformRunner>true</UseMicrosoftTestingPlatformRunner>
  
    For more information on Microsoft Testing Platform support in xUnit.net, please visit:
      https://xunit.net/docs/getting-started/v3/microsoft-testing-platform
    -->
  </PropertyGroup>

• This is not correct:

  With the release of v3, xunit.runner.visualstudio now supports running test through the new testing platform.

  MTP support is built into v3. xunit.runner.visualstudio is solely for supporting VSTest. You should either remove the reference to xunit.runner.visualstudio from your sample project, or if your intention is to support both MTP and VSTest, add the missing the required reference to Microsoft.NET.Test.Sdk.

[Evangelink]

Thanks for the feedback @bradwilson. What's your recommendation? MPT + VSTest or MTP only? I honestly prefer to go with what you would recommend/favor as you are the owner.

[bradwilson]

Today? Include VSTest. It's required for Test Explorer, and since you didn't define TestingPlatformDotnetTestSupport, it's required for dotnet test as well.

My answer might be different a year or two from now when everything supports MTP by default.

[Evangelink]

Updated! FYI we are having good results with the unfold of the MTP support in VS so hopefully 17.13 will have it on by default.

About dotnet test, I'll add a separate parapgrah to mention it for all test frameworks.

[thomhurst]

Updated! FYI we are having good results with the unfold of the MTP support in VS so hopefully 17.13 will have it on by default.

About dotnet test, I'll add a separate parapgrah to mention it for all test frameworks.

Did the VS team budge at all on allowing framework authors to have more control over the test viewer layout?

[Evangelink]

Did the VS team budge at all on allowing framework authors to have more control over the test viewer layout?

Not a priority at the moment :s

[OsirisTerje]

NUnit:

1. We will release adapter version 5 (non-beta) rather soon now, so just write 5.0.0, and not beta.5. And I believe you missed the SDK:
   In the Itemgroup section:

 <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.12.0" />
    <PackageReference Include="NUnit" Version="4.3.2" />
    <PackageReference Include="NUnit.Analyzers" Version="4.6.0"/>
    <PackageReference Include="NUnit3TestAdapter" Version="5.0.0" />
  </ItemGroup>

2. The code should be, to be kept up to current :

public class CalculatorTests
{
    [Test]
    public void Add_WhenCalled_ReturnsSum()
    {
        var calculator = new Calculator();
        var result = calculator.Add(1, 2);

        Assert.That(result,Is.EqualTo(3));
    }
}

[Evangelink]

Closing as I have moved on with internal blog post release process.