dotnet test filter or query filter not working

[JochemPalmsens]

We want to upgrade to xUnit v3 (as it's much faster we've noticed). All seems to go well locally, but on our build server we don't want to run all the tests.
The old action command-line is

dotnet test --filter "(FullyQualifiedName!~IntegrationTests)&(FullyQualifiedName!~Contoso.Application)&(FullyQualifiedName!~Contoso.ProductIntegration)&(FullyQualifiedName!~Contoso.Simulation.Recording)&(FullyQualifiedName!~BatteryTester)" /p:Platform=x64 --no-restore --no-build --configuration Debug --logger "console;verbosity=detailed" --logger "html;verbosity=detailed"

When switching to xUnit v3, both the -filter and -logger commands seems to be ignored! It tries to run all the test( assemblie)s and the logging is not verbose, but just one line per test project.

I've tried fixing the filtering using the new xunit v3 query filter.

dotnet test /p:Platform=x64 --no-restore --no-build --configuration Debug -- --filter-query "/(!Contoso.Application.Tests)&(!Contoso.ProductIntegration.Tests)&(!Contoso.Instruments.IntegrationTests)&(!Contoso.Recording.Tests)"

But that filter also doesn't seem to work: it's still running all the test asseblies.
Is there a known issue with using dotnet test with xUnit v3 or am I doing it wrong?

I've tried to simplify it down

dotnet test -- --filter-query "/Contoso.Product.Api.Tests"

which should only run the Contoso.Product.Api.Tests assembly... However it still runs all the test assemblies.

I'm using Visual studio 17.13.6
All projects have the following properties set

    <OutputType>Exe</OutputType>
    <TestingPlatformDotnetTestSupport>true</TestingPlatformDotnetTestSupport>
    <UseMicrosoftTestingPlatformRunner>true</UseMicrosoftTestingPlatformRunner>

So something doesn't seem to be working correctly. I've tried looking if I missed some step, but it's hard to find good information.

(and secondly it would also be nice to get the verbose logging back)

[bradwilson]

When switching to xUnit v3, both the -filter and -logger commands seems to be ignored!

That's a consequence of you enabling MTP for dotnet test, and the "switcharoo" the VSTest team does in Microsoft.Testing.Platform.MSBuild. Most of the old command line switches don't do anything for MTP (notably the ones related to building/selection, like --configuration and --framework, do continue to work).

You have to use the new switches, and pass them after -- so that dotnet test will send them directly to us.

The VSTest team is working on improving the dotnet test experience, but I don't know what their plans are for bridging the old command line switches and the new ones. For the time being, you should assume nothing from dotnet test -? is trustworthy when running an MTP project. Your best bet right now is to exec an xUnit.net v3 test project with -? to get a list of supported options:

  $ .\src\xunit.v3.assert.tests\bin\Debug\net472\xunit.v3.assert.tests.exe -?
xUnit.net v3 Microsoft.Testing.Platform Runner v2.0.2-pre.9-dev+176b936a21 (64-bit .NET Framework 4.8.9300.0)

Usage xunit.v3.assert.tests.exe [option providers] [extension option providers]

Execute a .NET Test Application.

Options:
    --config-file
        Specifies a testconfig.json file.

    --diagnostic
        Enable the diagnostic logging. The default log level is 'Trace'.
        The file will be written in the output directory with the name log_[yyMMddHHmmssfff].diag

    --diagnostic-filelogger-synchronouswrite
        Force the built-in file logger to write the log synchronously.
        Useful for scenario where you don't want to lose any log (i.e. in case of crash).
        Note that this is slowing down the test execution.
[...]

Note that all these command line switches must be placed after the --.

--filter-query "/(!Contoso.Application.Tests)&(!Contoso.ProductIntegration.Tests)&(!Contoso.Instruments.IntegrationTests)&(!Contoso.Recording.Tests)"

I can verify that an assembly negation query does work as expected for me:

Note that the assembly is "run" but none of the tests are run (because they've all been filtered out). However, MTP dotnet test considers a test assembly that runs no tests to be a failure. They have a command line switch to set the minimum number of tests that must run (or else the test assembly is considered to be failed), --minimum-expected-tests, but that does not accept 0 as a valid value.

Instead, you can pass the additional command line switch --ignore-exit-code 8 and it will ignore test projects that "failed" because they didn't run any tests. (MTP exit codes)

[Youssef1313]

@bradwilson Depending on how acceptable for you to take breaking changes, I think you can plug an MSBuild target that produces an error when TestingPlatformDotnetTestSupport is set to true and VSTest-specific command-line switches are used.

Most of the VSTest-specific command-line switches are transformed to MSBuild properties, so it's easy to do that in a target.

Some properties:

• VSTestSetting
• VSTestListTests
• VSTestTestCaseFilter
• VSTestTestAdapterPath
• VSTestLogger
• VSTestDiag
• VSTestResultsDirectory
• VSTestCollect
• VSTestBlame
• VSTestBlameCrash
• VSTestBlameHang

[bradwilson]

@Youssef1313 Why doesn't Microsoft.Testing.Platform.MSBuild do this?

[JochemPalmsens]

Hmm, ok. Well that's rather unwieldy, but it seems to work. Maybe add it to the filter-query documentation?

Than going to my secundary question: Is there a way to get the verbose logging back? i.e. listing all the tests that are run?

[Youssef1313]

@Youssef1313 Why doesn't Microsoft.Testing.Platform.MSBuild do this?

We can do that. But there are backcompat concerns.

@JochemPalmsens Try <TestingPlatformCaptureOutput>false</TestingPlatformCaptureOutput> and -v:detailed

[bradwilson]

@JochemPalmsens Good point. Done.