How to ExcludeFromCodeCoverageAttribute in a test framework scenario?

[jzabroski]

Description

If I run dotnet test --collect="Code Coverage", then it counts xunit classes towards code coverage statistics.

Coverlet addresses this problem by providing an ExcludeByAttribute where the user can define arbitrary attributes to use for exclusion purposes.

Steps to reproduce

1. Create a new .NET library project that has a class Foo that calculates A + B
2. Create a new xunit test project that tests class Foo
3. Run dotnet test --collect="Code Coverage"

Expected behavior

xunit assemblies not included in code coverage report.

Actual behavior

xunit assemblies included in code coverage report.

Diagnostic logs

n/a - please specify what you would want for this, and I can provide

Environment

Test SDK 17.5

[jzabroski]

Note: Brad Wilson, xunit maintainer, said this is not a real concern for him, since xunit does not collect code coverage stats for the project today.

However, I am curious what the VSTest engineers have to say about this base case.

[MarcoRossignoli]

cc: Jakub Chocholowicz (@jakubch1)

[jakubch1]

Thanks for the report. We will work on excluding xunit dlls by default.

[jzabroski]

Oops. Hit close by accident while holding my newborn.

Just wanted to say that you should also think about Coverlet approach ExcludeByAttribute. For example, what about assertion libraries like Shoudly.

[jakubch1]

In latest version https://dev.azure.com/dnceng/public/_artifacts/feed/test-tools/NuGet/Microsoft.CodeCoverage/overview/17.7.0-preview.23203.7 we added default exclusion of few libraries. We are working on some more generic approach like Coverlet has so not closing this.

[gioce90]

We are working on some more generic approach like Coverlet has so not closing this.

Hi Jakub Chocholowicz (@jakubch1) , what does this approach consist of? Can you share some information about it? It would be nice to exclude from code coverage report all about third-party libraries (without using .runsettings files)

[jakubch1]

We added some default exclusions for xunit and moq libraries. Generic solution is still not implemented.

[gioce90]

I think this is related to #72

[jakubch1]

In latest version https://dev.azure.com/dnceng/public/_artifacts/feed/test-tools/NuGet/Microsoft.CodeCoverage/overview/17.10.0-preview-24073-02 we ported from Coverlet feature to optionally exclude modules based on existence of sources.

To enable this feature in command line you can use:

dotnet test --collect "Code Coverage;ExcludeAssembliesWithoutSources=MissingAll"

If you use config/runsettings you can add:

<ExcludeAssembliesWithoutSources>MissingAll</ExcludeAssembliesWithoutSources>

under <Configuration> tag.

Remember that this is adding additional processing to check existence of sources. You will get always better performance by configuring correctly <ModulePaths> regexes.