WinUI3 Test App UITestMethod failing

[DavidTynewyddBI]

Describe the bug

When trying to run tests from a WinUI3 test app, getting the following for tests marked with the [UITestMethod] attribute:

Failed TestMethod2 Error Message: An unhandled exception was thrown by the 'Execute' method. Please report this error to the author of the attribute 'Microsoft.VisualStudio.TestTools.UnitTesting.AppContainer.UITestMethodAttribute'. System.InvalidOperationException: UITestMethodAttribute.DispatcherQueue should not be null. To use UITestMethodAttribute within a WinUI Desktop App, remember to set the static UITestMethodAttribute.DispatcherQueue during the test initialization. at Microsoft.VisualStudio.TestTools.UnitTesting.AppContainer.UITestMethodAttribute.Execute(ITestMethod testMethod) in /_/src/TestFramework/TestFramework.Extensions/Attributes/WinUI_UITestMethodAttribute.cs:line 67 at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestMethodRunner.ExecuteTest(TestMethodInfo testMethodInfo) in /_/src/Adapter/MSTest.TestAdapter/Execution/TestMethodRunner.cs:line 460

Works fine if run within Visual Studio's test explorer.

Steps To Reproduce

1. Create an empty WinUI3 test app
2. Build using dotnet build
3. Run dotnet test .\bin\Debug\net8.0-windows10.0.19041.0\win-x64\App1.dll (replace with your app name)

Expected behavior

The two default tests to pass

Actual behavior

The test marked with [UITestMethod] fails with:
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Failed TestMethod2
Error Message:
An unhandled exception was thrown by the 'Execute' method. Please report this error to the author of the attribute 'Microsoft.VisualStudio.TestTools.UnitTesting.AppContainer.UITestMethodAttribute'.
System.InvalidOperationException: UITestMethodAttribute.DispatcherQueue should not be null. To use UITestMethodAttribute within a WinUI Desktop App, remember to set the static UITestMethodAttribute.DispatcherQueue during the test initialization.
at Microsoft.VisualStudio.TestTools.UnitTesting.AppContainer.UITestMethodAttribute.Execute(ITestMethod testMethod) in //src/TestFramework/TestFramework.Extensions/Attributes/WinUI_UITestMethodAttribute.cs:line 67
at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestMethodRunner.ExecuteTest(TestMethodInfo testMethodInfo) in //src/Adapter/MSTest.TestAdapter/Execution/TestMethodRunner.cs:line 460

Failed! - Failed: 1, Passed: 1, Skipped: 0, Total: 2, Duration: 115 ms - App1.dll (net8.0)

Additional context

I'm unclear if this should even work at this point. I found some old references saying it wasn't supported, but the most recent was a year ago.

[Youssef1313]

The error mentions this:

To use UITestMethodAttribute within a WinUI Desktop App, remember to set the static UITestMethodAttribute.DispatcherQueue during the test initialization.

Did you do that? If yes, can you upload a project that reproduces this?

[DavidTynewyddBI]

The empty test project does this in the UnitTestApp.xaml.cs

        /// <summary>
        /// Invoked when the application is launched.
        /// </summary>
        /// <param name="args">Details about the launch request and process.</param>
        protected override void OnLaunched(Microsoft.UI.Xaml.LaunchActivatedEventArgs args)
        {
            Microsoft.VisualStudio.TestPlatform.TestExecutor.UnitTestClient.CreateDefaultUI();

            m_window = new UnitTestAppWindow();
            m_window.Activate();

            UITestMethodAttribute.DispatcherQueue = m_window.DispatcherQueue;

            Microsoft.VisualStudio.TestPlatform.TestExecutor.UnitTestClient.Run(Environment.CommandLine);
        }

[Youssef1313]

@DavidTynewyddBI dotnet test doesn't seem to be capable of running WinUI test apps right now. What you need to do is to run the following command from VS Developer Command Prompt:

vstest.console.exe bin/Debug/net8.0-windows10.0.19041.0/win-x64/App1.build.appxrecipe

[Evangelink]

I confirm that dotnet test is not compatible with UWP nor WinUI.

[DavidTynewyddBI]

@Evangelink I know this has been asked before, but is there any plan for support on the horizon? It seems strange to me that the two are incompatible

[Youssef1313]

@nohwnd Should this move to VSTest as a feature request? Or maybe track one item for VSTest (if you want to), and one other item for MTP?

[nohwnd]

VSTest is not adding new features, and the limitation is coming from how the bits that test uwp and winui are shipped (as far as I am aware). The two special pieces that VisualStudio has (and that vstest.console.exe that is shipped WITH visual studio has), are UWP test runtime provider and uwp deployer. They depend on deployment tooling in VS and are under VS license. So unless we extract the tools, review their licensing and start shipping the provider and deployer with Microsoft.TestPlatform vstest.console and dotnet test vstest.console we will not have the functionality outside of VS shipment.

I did not look if all the things that are needed are closed, or maybe already shipped to nuget.org, we can have a look for MTP where we might need it to bring the feature.

[DavidTynewyddBI]

Thanks for the quick responses all. My main reason for trying to use dotnet test over vstest.console is that I'm trying to also include code coverage into our CI pipeline. It seems that dotnet test has more support for this. I'm exploring options but it is frustrating that a fairly straightforward use-case is so complicated (not necessarily the fault of anyone in this thread)

[nohwnd]

I do agree with you. One thing I can recommend is running your desired dotnet test command with --diag:log.txt, and grabbing the runsettings xml from there, you will need to remove the concrete paths for data collectors, but overall that should give you a full and portable way of using the same setup among dotnet test and vstest.console.