[MTP] Better message for never finished tests?

[thomhurst]

If you report some tests as in progress, and then complete / end the test run, the result will say 0 tests run.

0 tests finished, but some were started. Could we be more informative?

[+0/x0/?0] TestPlatformRepro.dll (net9.0|arm64) - 15 tests running (15s)

[+0/x0/?0] TestPlatformRepro.dll (net9.0|arm64) - 15 tests running (18s)

[+0/x0/?0] TestPlatformRepro.dll (net9.0|arm64) - 15 tests running (21s)

[+0/x0/?0] TestPlatformRepro.dll (net9.0|arm64) - 15 tests running (24s)

[+0/x0/?0] TestPlatformRepro.dll (net9.0|arm64) - 15 tests running (27s)


Test run summary: Zero tests ran - /Users/runner/work/Microsoft.Testing.Platform.Reproductions/Microsoft.Testing.Platform.Reproductions/TestPlatformRepro/bin/Release/net9.0/TestPlatformRepro.dll (net9.0|arm64)

[nohwnd]

Sounds good, I can see writing out the tests that were in progress as being quite useful for understanding what was going on.

[thomhurst]

Probably worth failing the run too. The above would fail because of the zero tests run, but I imagine if I sent some success test nodes too, the run would pass.

[nohwnd]

I think that already happens, cancellation is translated to test run failure state. so no matter how many tests passed you will get fail.

[thomhurst]

I'm not necessarily talking about a cancellation. If you call ExecuteRequestContext.Complete() without sending a final status to a test. Granted it's probably a bug in the test framework, but I think the testing platform should still report that as a failure, because if tests never finished, something has gone wrong IMO. Plus they could fail and you just don't know.

Here's code to show what I mean:

using Microsoft.Testing.Platform.Extensions.Messages;
using Microsoft.Testing.Platform.Extensions.TestFramework;
using Microsoft.Testing.Platform.TestHost;

internal class DummyAdapter : ITestFramework, IDataProducer
{
    public string Uid => nameof(DummyAdapter);

    public string Version => string.Empty;

    public string DisplayName => string.Empty;

    public string Description => string.Empty;

    public Type[] DataTypesProduced => [typeof(TestNodeUpdateMessage)];

    public Task<CloseTestSessionResult> CloseTestSessionAsync(CloseTestSessionContext context) => Task.FromResult(new CloseTestSessionResult { IsSuccess = true });

    public Task<CreateTestSessionResult> CreateTestSessionAsync(CreateTestSessionContext context) => Task.FromResult(new CreateTestSessionResult { IsSuccess = true });

    public async Task ExecuteRequestAsync(ExecuteRequestContext context)
    {
        for (int i = 0; i < 15; i++)
        {
            await context.MessageBus.PublishAsync(this, new TestNodeUpdateMessage(new SessionUid("1"), new TestNode
            {
                Uid = $"{i}",
                DisplayName = $"Test {i}",
                Properties = new PropertyBag(i % 2 == 0 ? PassedTestNodeStateProperty.CachedInstance : InProgressTestNodeStateProperty.CachedInstance)
            }));
        }

        await Task.Delay(TimeSpan.FromSeconds(5));
        
        context.Complete();
    }

    public Task<bool> IsEnabledAsync() => Task.FromResult(true);
}

Output is:

Test run summary: Passed! - TestPlatformRepro.dll (net9.0|x64)
total: 8
failed: 0
succeeded: 8
skipped: 0
duration: 5s 083ms

But 15 tests were started. So I'd expect a failure. And like you said, some messages about which tests never completed.

[nohwnd]

Ok now it is clear 👍