The Operation will be canceled. The next steps may not contain expected logs.

[tarekgh]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=696281

Build leg reported

Build / browser-wasm linux Release LibraryTests / Send to Helix

Pull Request

dotnet/runtime#103009

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The Operation will be canceled. The next steps may not contain expected logs.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=696281
Error message validated: [The Operation will be canceled. The next steps may not contain expected logs.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/4/2024 8:08:48 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
980252	dotnet/runtime	Send to Helix	Log
2663215	dotnet-extensions	Build Azure DevOps plugin	Log	#48455
2663144	dotnet-runtime	Send job to Helix (Unix)	Log
980124	dotnet/sdk	Build	Log	dotnet/sdk#47167
980084	dotnet/sdk	Build	Log	dotnet/sdk#47480
980073	dotnet/runtime	Send to Helix	Log
2662942	dotnet-runtime	Send tests to Helix (Unix)	Log	#48447
2662947	dotnet-wcf	OneLocBuild	Log
2662910	dotnet-runtime	Send job to Helix (Windows)	Log
2662865	dotnet-performance	Send job to Helix (Windows)	Log
2662841	dotnet-dotnet	Checkout dotnet-dotnet@dev/ellahathaway/signing-validation to s	Log
2662834	dotnet-dotnet	🛡 Component Governance	Log
2662830	dotnet-dotnet	Secure Supply Chain Analysis (auto-injected by policy)	Log
2662819	dotnet-dotnet	Download secrets: EngKeyVault	Log
2662782	dotnet-roslyn	Test	Log
2662777	dotnet-roslyn	Test	Log
2662645	dotnet-sdk	Run Tests in Helix	Log	#48427
2662650	dotnet-sdk	Run AoT Tests	Log	#48424
2662737	dotnet-roslyn	Test	Log
2662730	dotnet-roslyn	Test	Log
979543	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#110818
979534	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#113493
979525	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#113491
979522	dotnet/runtime	Send tests to Helix (Windows)	Log
979507	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#113490
979496	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#113184
2662664	dotnet-roslyn	Test	Log
979481	dotnet/roslyn	Test	Log
2662662	dotnet-roslyn	Test	Log
2662651	dotnet-sdk	Run AoT Tests	Log	#48432
979473	dotnet/roslyn	Test	Log
2662636	dotnet-runtime	Send job to Helix (Unix)	Log
2662644	dotnet-sdk	Run AoT Tests in Helix	Log	#48428
2662643	dotnet-sdk	Run AoT Tests in Helix	Log	#48426
2662637	dotnet-roslyn	Test	Log
979442	dotnet/roslyn	Test	Log
979438	dotnet/roslyn	Test	Log
2662620	dotnet-sdk	Run AoT Tests in Helix	Log	#48428
2662619	dotnet-sdk	Run AoT Tests in Helix	Log	#48426
2662618	dotnet-sdk	Run AoT Tests in Helix	Log	#48427
2662622	dotnet-roslyn	Test	Log
2662600	dotnet-sdk	Run TestBuild Tests	Log	#48424
2662612	dotnet-roslyn	Test	Log
2662567	dotnet-aspnetcore	Prepare BuildLogs staging directory	Log	#48416
2662558	dotnet-dotnet	Checkout dotnet-dotnet@dev/ellahathaway/realsign-nonwindows-all to s	Log
2662529	dotnet-dotnet	🛡 Guardian: ARMory	Log
2662485	dotnet-efcore	Publish Using Darc	Log
979146	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#47554
2662465	dotnet-dotnet	Build (centOSStreamContainer)	Log
2662490	dotnet-sdk	Checkout dotnet-sdk@refs/pull/48403/merge to s	Log	#48403
2662489	dotnet-sdk	Checkout dotnet-sdk@refs/pull/48401/merge to s	Log	#48401
2662488	dotnet-sdk	Checkout dotnet-sdk@refs/pull/48402/merge to s	Log	#48402
2662470	dotnet-arcade-validation	🛡 Generate SBoM Manifest	Log
979129	dotnet/sdk	🟣 Prepare dotnet/project-system for formatting	Log	dotnet/sdk#47491
2662417	dotnet-runtime	Run performance job script	Log
2662307	dotnet-runtime	Send job to Helix (Unix)	Log
2662289	dotnet-runtime	Add Properties To Pipeline Env	Log
978652	dotnet/sdk	🟣 Prepare dotnet/efcore for formatting	Log	dotnet/sdk#47480
978612	dotnet/runtime	AOT compile CoreCLR tests	Log
978623	dotnet/sdk	Build	Log	dotnet/sdk#47499
978527	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#113413
2662091	dotnet-runtime	Send job to Helix (Unix)	Log
978475	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#112937
2662065	dotnet-roslyn	Test	Log
2662063	dotnet-roslyn	Test	Log
978247	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#77580
2662050	dotnet-sdk	Run AoT Tests in Helix	Log	#48373
2662049	dotnet-sdk	Run AoT Tests in Helix	Log	#48374
2662048	dotnet-sdk	Run Tests in Helix	Log	#48375
2662046	dotnet-efcore	Send job to helix	Log	#48385
2661996	dotnet-dotnet-monitor	Build	Log
2661985	dotnet-roslyn	Test	Log
978103	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#113449
978100	dotnet/runtime	Send to Helix	Log	dotnet/runtime#113448
978063	dotnet/runtime	Send to Helix	Log	dotnet/runtime#113447
978090	dotnet/runtime	Send to Helix	Log
978059	dotnet/runtime	Send to Helix	Log	dotnet/runtime#113446
977843	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#47519
2661936	dotnet-roslyn	Test	Log
2661921	dotnet-roslyn	Test	Log
2661916	dotnet-roslyn	Test	Log
2661891	dotnet-efcore	Send job to helix	Log
2661878	dotnet-dotnet-monitor	Build	Log
978012	dotnet/machinelearning	Build Packages	Log	dotnet/machinelearning#7422
2661835	dotnet-runtime	Send job to Helix (Unix)	Log
977957	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#113445
2661852	dotnet-sdk	Run AoT Tests in Helix	Log	#48374
2661853	dotnet-sdk	Run Tests in Helix	Log	#48375
2661851	dotnet-sdk	Run AoT Tests in Helix	Log	#48373
2661696	dotnet-sdk	Run AoT Tests	Log	#48367
978092	dotnet/runtime	Send to Helix	Log
978134	dotnet/roslyn	Run Integration Tests	Log
977947	dotnet/arcade	Run Helix Tests	Log	dotnet/arcade#15511
978087	dotnet/runtime	Send job to Helix (Unix)	Log
2661789	dotnet-dotnet	Build (azurelinuxArm64MuslCrossContainer)	Log
978007	dotnet/sdk	Run AoT Tests	Log	dotnet/sdk#47525
977999	dotnet/sdk	Run TestBuild Tests	Log	dotnet/sdk#47524
977990	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#60902
978068	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#77299
977965	dotnet/roslyn	Test	Log	dotnet/roslyn#77567
Displaying 100 of 1107 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
54	291	1012

[dkurepa]

These appear to be timeout cancellations, and most of them are happening on the same runtime leg that uses OSX.1200.ARM64.Open, looking at grafana, they do indeed appear to be quite high. Will check with the team on what we can do about this

[dkurepa]

For pipelines experiencing timeouts on osx.1200.arm64.open we recommend using osx.14.arm64.open, as this queue has a lot more machines

[dkurepa]

There is also a number of Publish Using Darc timeouts, these are in 99% of cases AzDo Api issues. During the Publish Using Darc job, we queue up a Maestro Promotion pipeline, that downloads the artifacts from the build calling it, and publishes them where they need to go. While publishing is happening, the build pipeline keeps busy waiting for the promotion pipeline to be done, checking the status of it every 60 seconds. In cases where this times out, the build most likely did get promoted, the AzDo API just kept telling us the wrong thing

[ivanpovazan]

@dkurepa for context, the same timeouts are also caught with this tracking #1883
where I asked about potentially bumping the queues to osx.13.amd/arm64, would that suffice?
(bumping to osx.14 is not an option right now as we still want testing on osx.13 which is the minimum supported OS version)