Checkout failure: "Git fetch failed with exit code 128"

[carlossanlop]

Build

https://dev.azure.com/dnceng/public/_build/results?buildId=1706886&view=results

Build leg reported

Libraries Test Run release coreclr Linux_musl x64 Release

Pull Request

Rolling build for runtime

Action required for the engineering services team

To triage this issue (First Responder / @dotnet/dnceng):

• Open the failing build above and investigate
• Add a comment explaining your findings

If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:

1. Add the label "Known Build Error"
2. Edit this issue and add an error string in the Json below that can help us match this issue with future build breaks. You should use the known issues documentation

{
   "ErrorMessage" : "Git fetch failed with exit code: 128",
   "BuildRetry": true
}

Additional information about the issue reported

The Checkout dotnet/runtime@main to s step is the one that failed. Here is the end of the log output:

remote: Compressing objects:  29% (30070/103687)        
remote: Compressing objects:  30% (31107/103687)        
remote: Compressing objects:  31% (32143/103687)        
remote: Compressing objects:  31% (32322/103687)        
remote: Compressing objects:  32% (33180/103687)        
remote: Compressing objects:  33% (34217/103687)        
remote: Compressing objects:  33% (35247/103687)        
remote: Compressing objects:  34% (35254/103687)        
remote: Compressing objects:  35% (36291/103687)        
remote: Compressing objects:  36% (37328/103687)        
remote: Compressing objects:  37% (38365/103687)        
remote: Compressing objects:  37% (38926/103687)        
remote: Compressing objects:  38% (39402/103687)        
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: protocol error: bad pack header
##[error]Git fetch failed with exit code: 128
Finishing: Checkout dotnet/runtime@main to s

Report

Build	Definition	Step Name	Console log	Pull Request
2753551	dotnet-efcore	Checkout dotnet-efcore@refs/pull/51725/merge to s	Log	#51725
1100721	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62822/merge to s	Log	dotnet/aspnetcore#62822
1100723	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62822/merge to s	Log	dotnet/aspnetcore#62822
1100725	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62822/merge to s	Log	dotnet/aspnetcore#62822
1100634	dotnet/runtime	Checkout dotnet/runtime@refs/pull/117679/merge to s	Log	dotnet/runtime#117679
1099035	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62806/merge to s	Log	dotnet/aspnetcore#62806
1099051	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62808/merge to s	Log	dotnet/aspnetcore#62808
1099052	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62808/merge to s	Log	dotnet/aspnetcore#62808
1098416	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62791/merge to s	Log	dotnet/aspnetcore#62791
1098411	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62790/merge to s	Log	dotnet/aspnetcore#62790
1098410	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62790/merge to s	Log	dotnet/aspnetcore#62790
1098415	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62791/merge to s	Log	dotnet/aspnetcore#62791
1098267	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62788/merge to s	Log	dotnet/aspnetcore#62788
1097443	dotnet/installer	Checkout dotnet/installer@refs/pull/20505/merge to s	Log	dotnet/installer#20505
1097438	dotnet/installer	Checkout dotnet/installer@refs/pull/20508/merge to s	Log	dotnet/installer#20508
1091127	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62660/merge to s	Log	dotnet/aspnetcore#62660
1091126	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62660/merge to s	Log	dotnet/aspnetcore#62660
1091128	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62660/merge to s	Log	dotnet/aspnetcore#62660
1088499	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62585/merge to s	Log	dotnet/aspnetcore#62585
1088500	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62585/merge to s	Log	dotnet/aspnetcore#62585
1088226	dotnet/runtime	Checkout dotnet/runtime@refs/pull/116916/merge to s	Log	dotnet/runtime#116916
1087960	dotnet/sdk	Checkout dotnet/sdk@refs/pull/49673/merge to s	Log	dotnet/sdk#49673
1085286	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62561/merge to s	Log	dotnet/aspnetcore#62561
1085288	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62561/merge to s	Log	dotnet/aspnetcore#62561
1085282	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62560/merge to s	Log	dotnet/aspnetcore#62560
1085601	dotnet/runtime	Checkout dotnet/runtime@refs/pull/117297/merge to s	Log	dotnet/runtime#117297
1082711	dotnet/runtime	Checkout dotnet/runtime@refs/pull/117190/merge to s	Log	dotnet/runtime#117190
1082319	maestro-auth-test/helix-queue-insights-test	Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/1059/merge to s	Log	maestro-auth-test/helix-queue-insights-test#1059
1082245	maestro-auth-test/helix-queue-insights-test	Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/1056/merge to s	Log	maestro-auth-test/helix-queue-insights-test#1056
1081365	dotnet/runtime	Checkout dotnet/runtime@refs/pull/117103/merge to s	Log	dotnet/runtime#117103
2738860	dotnet-aspnetcore	Checkout dotnet-aspnetcore@refs/pull/51251/merge to s	Log	#51251
2738694	dotnet-aspnetcore	Checkout dotnet-aspnetcore@refs/pull/51231/merge to s	Log	#51231
1080053	dotnet/sdk	Checkout dotnet/sdk@refs/pull/49585/merge to s	Log	dotnet/sdk#49585
1079912	dotnet/installer	Checkout dotnet/installer@refs/pull/20488/merge to s	Log	dotnet/installer#20488
1079455	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62488/merge to s	Log	dotnet/aspnetcore#62488
1079456	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62488/merge to s	Log	dotnet/aspnetcore#62488
1079449	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62487/merge to s	Log	dotnet/aspnetcore#62487
1079448	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62487/merge to s	Log	dotnet/aspnetcore#62487
1079457	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62488/merge to s	Log	dotnet/aspnetcore#62488
1079450	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62487/merge to s	Log	dotnet/aspnetcore#62487
1079402	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62486/merge to s	Log	dotnet/aspnetcore#62486
1079401	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62486/merge to s	Log	dotnet/aspnetcore#62486
1079396	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62485/merge to s	Log	dotnet/aspnetcore#62485
1079395	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62485/merge to s	Log	dotnet/aspnetcore#62485
1079403	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62486/merge to s	Log	dotnet/aspnetcore#62486
1079397	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62485/merge to s	Log	dotnet/aspnetcore#62485
1079322	dotnet/xharness	Checkout dotnet/xharness@refs/pull/1426/merge to s	Log	dotnet/xharness#1426
1079323	dotnet/xharness	Checkout dotnet/xharness@refs/pull/1426/merge to s	Log	dotnet/xharness#1426
1078634	dotnet/runtime	Checkout dotnet/runtime@refs/pull/116795/merge to s	Log	dotnet/runtime#116795
1078410	maestro-auth-test/helix-queue-insights-test	Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/1052/merge to s	Log	maestro-auth-test/helix-queue-insights-test#1052
2737099	dotnet-runtime	Checkout dotnet-runtime@main to s	Log
1076957	maestro-auth-test/helix-queue-insights-test	Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/1049/merge to s	Log	maestro-auth-test/helix-queue-insights-test#1049
1076534	dotnet/runtime	Checkout dotnet/runtime@refs/pull/116634/merge to s	Log	dotnet/runtime#116634
1076387	maestro-auth-test/helix-queue-insights-test	Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/1047/merge to s	Log	maestro-auth-test/helix-queue-insights-test#1047

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
4	15	54

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=323960
Error message validated: Git fetch failed with exit code: 128
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/28/2023 9:15:14 PM UTC

[ChadNedzlek]

So this is... interesting. I agree this seems like essentially a random network failure that is not the fault of the code. Also... it doesn't seem like there is anything to be done other than retry the build.

@ulisesh and @AlitzelMendez. This seems like another instance of a "known issue" that would really benefit from the ability to get retried automatically. I think Carlos did the right thing by opening the known issue, but his PR (and any future PR that hits this) is still going to be blocked, since the build doesn't run at all in this case, because the checkout failed. And it would be great to help folks that hit this.

[ulisesh]

I created the issue to track how often it was happening and measure the impact of the problem. It might be hard to get traction on networking problems but having the number of impacted builds has great value in my opinion.

In the last 24 hours, we had 37 builds failed with the same error. cc @markwilkie who might find this data interesting.

I think automatic retries for known issues will be the next feature after we are done with analyzing builds retroactively, we can discuss this if people think we should change priorities.

[markwilkie]

So 37 builds were caught with the known issues feature @ulisesh ? Could I get the query please? (super exciting)

[ulisesh]

I saw 37 hits in the office but now I only see 33 🤔. There are 33 failed jobs in 2 different builds. Sorry if I misused the terms but I still think it is great that we can track these build breaks 😄

This is the query:

KnownIssues
| where IssueId == 9009 and IssueRepository == "dotnet/arcade"

[ViktorHofer]

This happened 36 times in the last 24 hours. Should we add a retry (assuming that's possible)?