dotnet-watch tests failing

[dsplaisted]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=629827
Build error leg or test failing: dotnet-watch.Tests.dll.1.WorkItemExecution
Pull request: #39888

Error Message

##[error].packages/microsoft.dotnet.helix.sdk/8.0.0-beta.24177.1/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(89,5): error : (NETCORE_ENGINEERING_TELEMETRY=Test) Work item dotnet-watch.Tests.dll.1 in job 82728a56-9430-4356-ab82-1c21670792a7 has failed

{
  "ErrorMessage": "",
  "ErrorPattern": "Work item dotnet-watch\\.Tests\\.dll\\.1 in job [a-z0-9\\-]+ has failed",
  "BuildRetry": true,
  "ExcludeConsoleLog": true
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=629827
Error message validated: [Work item dotnet-watch\.Tests\.dll\.1 in job [a-z0-9\-]+ has failed]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 4/4/2024 9:51:10 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1096885	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49818
1096317	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49806
1096679	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49697
1095795	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49746
1096725	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49814
1096543	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49810
2751390	dotnet-sdk	🟣 Run ContainerBased Tests (azureLinux30Net10BuildAmd64)	Log
1096421	dotnet/sdk	🟣 Run FullFramework Tests	Log	#49800
2751337	dotnet-sdk	🟣 Run ContainerBased Tests (azureLinux30Net10BuildAmd64)	Log
1096076	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49389
1096316	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49635
1096430	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49808
1096392	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49807
1096348	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49767
1096172	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49740
1096224	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49756
1096221	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49642
1096256	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49802
1096254	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49801
1096042	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49798
1094688	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49780
1093088	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49756
1095895	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49798
1095956	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49761
1095821	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49748
2750945	dotnet-sdk	🟣 Run ContainerBased Tests (azureLinux30Net10BuildAmd64)	Log
1095872	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49772
1095284	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49743
1095745	dotnet/sdk	🟣 Run FullFramework Tests	Log	#49611
1095315	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49389
2750910	dotnet-sdk	🟣 Run ContainerBased Tests (azureLinux30Net10BuildAmd64)	Log
1095809	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49772
1095579	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49794
1095430	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49417
1095425	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49721
1095189	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49772
1095283	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49749
1095088	dotnet/sdk	🟣 Run ContainerBased Tests	Log
1095047	dotnet/sdk	🟣 Run ContainerBased Tests	Log
1094777	dotnet/sdk	🟣 Run ContainerBased Tests	Log
1094870	dotnet/sdk	🟣 Run ContainerBased Tests	Log
1094698	dotnet/sdk	Run TestBuild Tests	Log	#49739
1094697	dotnet/sdk	Run TestBuild Tests	Log	#49738
2750215	dotnet-sdk	🟣 Run ContainerBased Tests (azureLinux30Net10BuildAmd64)	Log
1094633	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49743
2749921	dotnet-sdk	Run FullFramework Tests	Log	#51489
2749922	dotnet-sdk	Run TestBuild Tests	Log	#51522
1092507	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49746
1094042	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49772
1092494	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49697
1092816	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49749
2749593	dotnet-sdk	🟣 Run ContainerBased Tests (azureLinux30Net10BuildAmd64)	Log
1093747	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49769
1093472	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49737
2749484	dotnet-sdk	🟣 Run ContainerBased Tests (azureLinux30Net10BuildAmd64)	Log
2749398	dotnet-sdk	🟣 Run ContainerBased Tests (azureLinux30Net10BuildAmd64)	Log
1093450	dotnet/sdk	Run TestBuild Tests	Log	#49602
1093488	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49743
1091905	dotnet/sdk	Run TestBuild Tests	Log	#49619
1093316	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49761
1093258	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49761
2749162	dotnet-sdk	🟣 Run ContainerBased Tests (azureLinux30Net10BuildAmd64)	Log
1093163	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49751
1093085	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49755
1093106	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49759
1093091	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49758
2748837	dotnet-sdk	Run FullFramework Tests	Log	#51489
1092892	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49751
1092855	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49556
1092790	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#47389
1092260	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49745
1092687	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49748
1092654	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#47389
1092658	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49737
1092655	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49736
1092495	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49736
1092496	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#47389
2748265	dotnet-sdk	🟣 Run ContainerBased Tests (azureLinux30Net10BuildAmd64)	Log
1092293	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49697
1091473	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49736
1092265	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49664
1092201	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49734
2747911	dotnet-sdk	Run FullFramework Tests	Log	#51466
2747912	dotnet-sdk	Run FullFramework Tests	Log	#51465
2747895	dotnet-sdk	🟣 Run ContainerBased Tests (azureLinux30Net10BuildAmd64)	Log
1092031	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49734
1092021	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49660
1092019	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49740
1092018	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49737
1091895	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49737
1091887	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#47389
1091873	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49726
1091871	dotnet/sdk	Run FullFramework Tests	Log	#49619
1091843	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49740
2747576	dotnet-sdk	Run FullFramework Tests	Log	#51332
1091813	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49664
1091800	dotnet/sdk	Run TestBuild Tests	Log	#49657
1091803	dotnet/sdk	Run FullFramework Tests	Log	#49724
1091580	dotnet/sdk	🟣 Run TestBuild Tests	Log	#49726
2747476	dotnet-sdk	🟣 Run ContainerBased Tests (azureLinux30Net10BuildAmd64)	Log
Displaying 100 of 183 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
33	111	183

[marcpopMSFT]

@tmat we still see this fairly frequently. Any luck? Here's a recent one from a codeflow PR:
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-sdk-refs-pull-41616-merge-58511284b77c40778a/dotnet-watch.Tests.dll.1/1/console.418de7ca.log?helixlogtype=result

[tmat]

No luck yet. This might help: https://github.com/dotnet/sdk/blob/d45e40d665e647804de03d2b4eec2aae9c488cb5/test/dotnet-watch.Tests/xunit.runner.json

It will print out test info every 20 seconds a test is running.

[marcpopMSFT]

@tmat another failure this morning shows a bunch of build processes for watch apps still running... https://helixre107v0xd1eu3ibi6ka.blob.core.windows.net/dotnet-sdk-refs-pull-41958-merge-84357342ee154a5088/dotnet-watch.Tests.dll.2/1/console.3dddc463.log?helixlogtype=result

[tmat]

Looks like that's in 8.0 branch. Porting test diagnostics from 9: #41963

[Forgind]

This has been making one of my PRs fail today, so I looked into it a bit. Based on when this issue was created and which tests are failing, I suspect this PR:
#39618

I asked jasonmalinowski whether the MSBuildWorkspace is supposed to support F# because I'd found a code snippet that suggested that it shouldn't, and he said it doesn't and never did. Did that PR make us use MSBuildWorkspace in a different way than we had before?

[Forgind]

The changes to HotReloadDotNetWatcher look most suspicious to me. It looks like a substantial change to how we call MSBuild.