restarted. Azure DevOps can't recover from restarts.

[lewing]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=784852

Build leg reported

Build / linux-x64 debug Libraries_AllConfigurations

Pull Request

dotnet/runtime#106599

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "restarted. Azure DevOps can't recover from restarts.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=784852
Error message validated: [restarted. Azure DevOps can't recover from restarts.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 8/26/2024 7:12:18 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
977821	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113442
980477	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113374
980451	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113464
980404	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113413
980337	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#100729
980309	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113429
979323	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113229
980266	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113416
980093	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113519
980065	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113517
980084	dotnet/sdk	Android_Shortstack_arm64		dotnet/sdk#47480
979999	dotnet/sdk	Browser_Shortstack_wasm		dotnet/sdk#47577
979945	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113507
979802	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113503
979556	dotnet/sdk	Android_Shortstack_arm64		dotnet/sdk#47515
979543	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#110818
979250	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113482
979190	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113437
2662466	dotnet-dotnet	AzureLinux_x64_Cross_x64
979131	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113342
979030	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113429
978996	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#107815
978926	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113471
978904	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113456
978829	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113084
978565	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113288
978535	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113456
978512	dotnet/sdk	Browser_Shortstack_wasm		dotnet/sdk#47480
978485	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113461
978115	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113451
977869	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111126
977745	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#110818
977684	dotnet/runtime	browser-wasm linux Release LibraryTests
977623	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113343
977602	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#112604
977591	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113426
977578	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113434
977542	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113342
977540	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113342
977550	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113435
977523	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113429
2661575	dotnet-dotnet	AzureLinux_x64_Cross_Shortstack_Mono_LLVMAOT_arm64
977498	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113425
977443	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113420
977399	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113422
977374	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#112074
977345	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#113333
977318	dotnet/sdk	Android_Shortstack_arm64		dotnet/sdk#47489
977175	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113339
976966	dotnet/sdk	Android_Shortstack_arm64		dotnet/sdk#47491
976371	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113399
976297	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113288
976186	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113342
976048	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113339
975889	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113385
975835	dotnet/sdk	Android_Shortstack_arm64		dotnet/sdk#47458
975664	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113339
2660070	dotnet-dotnet	Android_Shortstack_x64
975024	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading
974705	dotnet/sdk	Android_Shortstack_x64		dotnet/sdk#47397
974578	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113339
974427	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#112514
974601	dotnet/sdk	Android_Shortstack_arm64		dotnet/sdk#47414
974120	dotnet/sdk	Browser_Shortstack_wasm		dotnet/sdk#47313
973886	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113164
973371	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113306
973047	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113041
973009	dotnet/sdk	Android_Shortstack_arm64		dotnet/sdk#47389
972845	dotnet/runtime	browser-wasm linux Release LibraryTests
972820	dotnet/sdk	Android_Shortstack_arm64		dotnet/sdk#47338
972793	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113288
972783	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#112937
972141	dotnet/sdk	Android_Shortstack_arm64		dotnet/sdk#47361
971931	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113258
971912	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111934
968614	dotnet/runtime	browser-wasm linux Release AllSubsets_Mono_RuntimeTests monointerpreter	Log	dotnet/runtime#113129
2657674	dotnet-dotnet-buildtools-prereqs-docker	Windows2025_amd64 src-windowsservercore-ltsc2025-helix-graph
971793	dotnet/sdk	Browser_Shortstack_wasm		dotnet/sdk#47355
971666	dotnet/runtime	browser-wasm linux Release AllSubsets_Mono_RuntimeTests monointerpreter		dotnet/runtime#113014
971634	dotnet/runtime	browser-wasm linux Release AllSubsets_Mono_RuntimeTests monointerpreter		dotnet/runtime#112863
971632	dotnet/runtime	browser-wasm linux Release AllSubsets_Mono_RuntimeTests monointerpreter
971543	dotnet/runtime	browser-wasm linux Release AllSubsets_Mono_RuntimeTests monointerpreter		dotnet/runtime#113241
971589	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113248
971477	dotnet/runtime	browser-wasm linux Release AllSubsets_Mono_RuntimeTests monointerpreter		dotnet/runtime#113241
971299	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading
971177	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113234
971173	dotnet/runtime	browser-wasm linux Release AllSubsets_Mono_RuntimeTests monointerpreter		dotnet/runtime#111178
971164	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113232
971051	dotnet/runtime	browser-wasm linux Release AllSubsets_Mono_RuntimeTests monointerpreter
970974	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113226
970793	dotnet/runtime	browser-wasm linux Release LibraryTests
970713	dotnet/runtime	browser-wasm linux Release AllSubsets_Mono_RuntimeTests monointerpreter		dotnet/runtime#113164
970701	dotnet/runtime	browser-wasm linux Release AllSubsets_Mono_RuntimeTests monointerpreter		dotnet/runtime#112945
970605	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113211
970593	dotnet/runtime	browser-wasm linux Release AllSubsets_Mono_RuntimeTests monointerpreter		dotnet/runtime#112863
970590	dotnet/runtime	browser-wasm linux Release AllSubsets_Mono_RuntimeTests monointerpreter		dotnet/runtime#113210
967751	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113014
970518	dotnet/runtime	browser-wasm linux Release AllSubsets_Mono_RuntimeTests monointerpreter		dotnet/runtime#112945
970499	dotnet/runtime	browser-wasm linux Release AllSubsets_Mono_RuntimeTests monointerpreter		dotnet/runtime#113164
970418	dotnet/runtime	browser-wasm linux Release AllSubsets_Mono_RuntimeTests monointerpreter		dotnet/runtime#113184
Displaying 100 of 268 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
20	73	256

[am11]

It seems like all reports are pointing to linux-x64 dev-innerloop leg from this definition: https://dev.azure.com/dnceng-public/public/_build?definitionId=133. GitHub doesn't sync the status and keeps showing as if the job is running for days.. Opened dotnet/runtime#108581 to disable the leg.

[am11]

Just before the timeout we see low memory warnings like these:

https://dev.azure.com/dnceng-public/public/_build/results?buildId=829130&view=logs&j=e80acbf0-bc87-577c-4c46-0016b0794913&t=f0fa9d72-e49a-5249-4d28-1199014b9857

then it hangs for ~20 minutes or so before giving up. The build command has -allConfigurations so it builds all product+test assemblies for all platforms ({linux,win,osx,freebsd,illumos}-{x86,x64,arm,arm64,riscv64.. etc.}) all in one invocation of build (which isn't exactly efficient as we should probably group them..), which means, as it stands, this leg needs decent amount of RAM.

[steveisok]

@ilyas1974 @markwilkie I don't think our RAM consumption has increased that much to not be able to handle this configuration. Thoughts?

[ericstj]

I took a look at a few passing builds, many have the same logs where they approach 95% memory usage but eventually succeed.

Here are some samples:
https://dev.azure.com/dnceng-public/public/_build/results?buildId=836601&view=logs&j=e80acbf0-bc87-577c-4c46-0016b0794913&t=f0fa9d72-e49a-5249-4d28-1199014b9857
https://dev.azure.com/dnceng-public/public/_build/results?buildId=838110&view=logs&j=e80acbf0-bc87-577c-4c46-0016b0794913&t=f0fa9d72-e49a-5249-4d28-1199014b9857&l=3923
https://dev.azure.com/dnceng-public/public/_build/results?buildId=837745&view=logs&j=e80acbf0-bc87-577c-4c46-0016b0794913&t=f0fa9d72-e49a-5249-4d28-1199014b9857&l=4228

Near this point I see logs like this:

  initializing ChangeMakerService with capabilities: Baseline, AddMethodToExistingType, AddStaticFieldToExistingType, AddInstanceFieldToExistingType, NewTypeDefinition, ChangeCustomAttributes, UpdateParameters, GenericAddMethodToExistingType, GenericUpdateMethod, GenericAddFieldToExistingType
  baseline ready
  got a change
  parsing patch #1 from /__w/1/s/src/libraries/System.Runtime.Loader/tests/ApplyUpdate/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField/GenericAddInstanceField_v1.cs and creating delta
  Found changes in GenericAddInstanceField.cs
  change service made fa564b82-cf1c-4fb0-9d1a-f5ca4c71ff03
  wrote /__w/1/s/artifacts/bin/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField/Debug/net10.0/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField.dll.1.dmeta
  got a change
  parsing patch #2 from /__w/1/s/src/libraries/System.Runtime.Loader/tests/ApplyUpdate/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField/GenericAddInstanceField_v2.cs and creating delta
  Found changes in GenericAddInstanceField.cs
  change service made fa564b82-cf1c-4fb0-9d1a-f5ca4c71ff03
  wrote /__w/1/s/artifacts/bin/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField/Debug/net10.0/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField.dll.2.dmeta
  done

It looks to me like this is coming from https://github.com/dotnet/hotreload-utils/blob/254ec75de6127c368827d15c3af2477095b8b1b4/src/Microsoft.DotNet.HotReload.Utils.Generator/EnC/ChangeMakerService.cs#L28

Does anyone have an idea why hotreload would be running during a build?? I could imagine that if some hot reload service was runnign during a build or if tests were running while the product was building that could explain high memory usage.

[snakex64]

This seems to be happening consistently in PR #109320 . Should I just try re-running a third time or is it never gonna work?