WinHttpHandler test UseClientCertOnHttp2_OSSupportsIt_Success failing

[stephentoub]

This fails for me locally 100% of the time (dotnet msbuild /t:test).
cc: @alnikola

      System.Net.Http.WinHttpHandlerFunctional.Tests.ClientCertificateTest.UseClientCertOnHttp2_OSSupportsIt_Success [FAIL]
        System.Exception : HTTP 1.1 request received.
        Stack Trace:
          d:\repos\runtime\src\libraries\Common\tests\System\Net\Http\Http2LoopbackConnection.cs(74,0): at System.Net.Test.Common.Http2LoopbackConnection..ctor(Socket socket, Http2Options httpOptions)
          d:\repos\runtime\src\libraries\Common\tests\System\Net\Http\Http2LoopbackServer.cs(89,0): at System.Net.Test.Common.Http2LoopbackServer.AcceptConnectionAsync()
          d:\repos\runtime\src\libraries\Common\tests\System\Net\Http\Http2LoopbackServer.cs(108,0): at System.Net.Test.Common.Http2LoopbackServer.EstablishConnectionGetSettingsAsync(SettingsEntry[] settingsEntries)
          d:\repos\runtime\src\libraries\Common\tests\System\Net\Http\Http2LoopbackServer.cs(102,0): at System.Net.Test.Common.Http2LoopbackServer.EstablishConnectionAsync(SettingsEntry[] settingsEntries)
          d:\repos\runtime\src\libraries\System.Net.Http.WinHttpHandler\tests\FunctionalTests\ClientCertificateTest.cs(80,0): at System.Net.Http.WinHttpHandlerFunctional.Tests.ClientCertificateTest.<>c__DisplayClass4_0.<<UseClientCertOnHttp2_OSSupportsIt_Success>b__1>d.MoveNext()
          --- End of stack trace from previous location ---
          d:\repos\runtime\src\libraries\Common\tests\System\Threading\Tasks\TaskTimeoutExtensions.cs(83,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
          d:\repos\runtime\src\libraries\Common\tests\System\Threading\Tasks\TaskTimeoutExtensions.cs(111,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
          d:\repos\runtime\src\libraries\Common\tests\System\Threading\Tasks\TaskTimeoutExtensions.cs(71,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks, Int32 millisecondsTimeout)
          d:\repos\runtime\src\libraries\Common\tests\System\Net\Http\Http2LoopbackServer.cs(196,0): at System.Net.Test.Common.Http2LoopbackServer.CreateClientAndServerAsync(Func`2 clientFunc, Func`2 serverFunc, Http2Options http2Options, Int32 timeout)
          d:\repos\runtime\src\libraries\System.Net.Http.WinHttpHandler\tests\FunctionalTests\ClientCertificateTest.cs(62,0): at System.Net.Http.WinHttpHandlerFunctional.Tests.ClientCertificateTest.UseClientCertOnHttp2_OSSupportsIt_Success()
          --- End of stack trace from previous location ---

[jaredpar]

Confirmed this is blocking CI, generating a report

[jaredpar]

Console Log Summary

   System.Net.Http.Functional.Tests.PlatformHandlerTest_Cookies_Http2.GetAsync_ReceiveSetCookieHeader_CookieAdded(cookieName: "ABC", cookieValue: "123", useCookies: False) [FAIL]
      System.Net.Http.HttpRequestException : Error while copying content to a stream.
      ---- System.IO.IOException : The read operation failed, see inner exception.
      -------- System.Net.Http.WinHttpException : Error 12030 calling WINHTTP_CALLBACK_STATUS_REQUEST_ERROR, 'The connection with the server was terminated abnormally'.
      Stack Trace:
        /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs(446,0): at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
        /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs(551,0): at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts, CancellationToken callerToken, Int64 timeoutTime)
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(83,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(111,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(71,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks, Int32 millisecondsTimeout)
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Cookies.cs(344,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Cookies.<>c__DisplayClass18_0.<<GetAsync_ReceiveSetCookieHeader_CookieAdded>b__0>d.MoveNext()
        --- End of stack trace from previous location ---
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(38,0): at System.Threading.Tasks.TaskTimeoutExtensions.TimeoutAfter(Task task, TimeSpan timeout)
        /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs(243,0): at System.Net.Test.Common.Http2LoopbackServerFactory.CreateServerAsync(Func`3 funcAsync, Int32 millisecondsTimeout, GenericLoopbackOptions options)
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Cookies.cs(334,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Cookies.GetAsync_ReceiveSetCookieHeader_CookieAdded(String cookieName, String cookieValue, Boolean useCookies)
        --- End of stack trace from previous location ---
        ----- Inner Stack Trace -----
        /_/src/libraries/Common/src/System/Threading/Tasks/RendezvousAwaitable.cs(64,0): at System.Threading.Tasks.RendezvousAwaitable`1.GetResult()
        /_/src/libraries/System.Net.Http.WinHttpHandler/src/System/Net/Http/WinHttpResponseStream.cs(127,0): at System.Net.Http.WinHttpResponseStream.CopyToAsyncCore(Stream destination, Byte[] buffer, CancellationToken cancellationToken)
        /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs(440,0): at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
        ----- Inner Stack Trace -----
        /_/src/coreclr/src/System.Private.CoreLib/src/System/Environment.CoreCLR.cs(95,0): at System.Environment.get_StackTrace()
        /_/src/libraries/Common/src/System/Runtime/ExceptionServices/ExceptionStackTrace.cs(24,0): at System.Runtime.ExceptionServices.ExceptionStackTrace.AddCurrentStack(Exception exception)
        /_/src/libraries/System.Net.Http.WinHttpHandler/src/System/Net/Http/WinHttpException.cs(59,0): at System.Net.Http.WinHttpException.CreateExceptionUsingError(Int32 error, String nameOfCalledFunction)
        /_/src/libraries/System.Net.Http.WinHttpHandler/src/System/Net/Http/WinHttpRequestCallback.cs(333,0): at System.Net.Http.WinHttpRequestCallback.OnRequestError(WinHttpRequestState state, WINHTTP_ASYNC_RESULT asyncResult)
        /_/src/libraries/System.Net.Http.WinHttpHandler/src/System/Net/Http/WinHttpRequestCallback.cs(104,0): at System.Net.Http.WinHttpRequestCallback.RequestCallback(IntPtr handle, WinHttpRequestState state, UInt32 internetStatus, IntPtr statusInformation, UInt32 statusInformationLength)
        /_/src/libraries/System.Net.Http.WinHttpHandler/src/System/Net/Http/WinHttpRequestCallback.cs(47,0): at System.Net.Http.WinHttpRequestCallback.WinHttpCallback(IntPtr handle, IntPtr context, UInt32 internetStatus, IntPtr statusInformation, UInt32 statusInformationLength)
           at Interop.WinHttp.WinHttpQueryDataAvailable(SafeWinHttpHandle requestHandle, IntPtr parameterIgnoredAndShouldBeNullForAsync)
        /_/src/libraries/System.Net.Http.WinHttpHandler/src/System/Net/Http/WinHttpResponseStream.cs(122,0): at System.Net.Http.WinHttpResponseStream.CopyToAsyncCore(Stream destination, Byte[] buffer, CancellationToken cancellationToken)
        /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs(42,0): at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
           at System.Net.Http.WinHttpResponseStream.CopyToAsyncCore(Stream destination, Byte[] buffer, CancellationToken cancellationToken)
        /_/src/libraries/System.Net.Http.WinHttpHandler/src/System/Net/Http/WinHttpResponseStream.cs(106,0): at System.Net.Http.WinHttpResponseStream.CopyToAsync(Stream destination, Int32 bufferSize, CancellationToken cancellationToken)
        /_/src/libraries/Common/src/System/Net/Http/NoWriteNoSeekStreamContent.cs(49,0): at System.Net.Http.NoWriteNoSeekStreamContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
        /_/src/libraries/Common/src/System/Net/Http/NoWriteNoSeekStreamContent.cs(31,0): at System.Net.Http.NoWriteNoSeekStreamContent.SerializeToStreamAsync(Stream stream, TransportContext context)
        /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs(325,0): at System.Net.Http.HttpContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
        /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs(425,0): at System.Net.Http.HttpContent.LoadIntoBufferAsync(Int64 maxBufferSize, CancellationToken cancellationToken)
        /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs(551,0): at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts, CancellationToken callerToken, Int64 timeoutTime)
        /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs(290,0): at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
        /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs(172,0): at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
        /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs(329,0): at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
        /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs(307,0): at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
        /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs(784,0): at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
        /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs(3236,0): at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
        /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs(3208,0): at System.Threading.Tasks.Task.FinishContinuations()
        /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Future.cs(398,0): at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
        /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskCompletionSource.cs(266,0): at System.Threading.Tasks.TaskCompletionSource`1.TrySetResult(TResult result)
        /_/src/libraries/System.Net.Http.WinHttpHandler/src/System/Net/Http/WinHttpHandler.cs(898,0): at System.Net.Http.WinHttpHandler.StartRequestAsync(WinHttpRequestState state)
        /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs(290,0): at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
        /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs(172,0): at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
        /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs(329,0): at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
        /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs(307,0): at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
        /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs(2364,0): at System.Threading.Tasks.Task.InnerInvoke()
        /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs(2352,0): at System.Threading.Tasks.Task.<>c.<.cctor>b__274_0(Object obj)
        /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs(289,0): at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
        /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs(2314,0): at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
        /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs(2246,0): at System.Threading.Tasks.Task.ExecuteEntryUnsafe(Thread threadPoolThread)
        /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs(2237,0): at System.Threading.Tasks.Task.ExecuteFromThreadPool(Thread threadPoolThread)
        /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPool.cs(659,0): at System.Threading.ThreadPoolWorkQueue.Dispatch()
        /_/src/coreclr/src/System.Private.CoreLib/src/System/Threading/ThreadPool.CoreCLR.cs(29,0): at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
        --- End of stack trace from AddCurrentStack ---

Builds

Build	Pull Request	Test Failure Count
#565812	#33758	1
#567106	#33860	2
#567452	#33841	1
#568318	#33911	2
#569240	#33874	1
#569624	#33961	3
#569691	#33820	2
#569694	#32587	3
#569775	#33967	1
#569778	#33946	1
#569904	#33970	1
#569926	#33959	1
#569968	#33175	2
#569980	Rolling	1
#570254	#33459	1
#570266	#33984	1
#570305	Rolling	3
#570426	#33485	1
#570477	#32587	2
#570525	#33990	1
#570532	Rolling	1
#570537	#33991	1
#570549	#33994	1
#570584	#33984	1
#570613	#33945	2
#570636	Rolling	1
#570652	#33485	2
#570690	Rolling	2
#570708	#32592	1
#570741	#33459	1
#570746	#34006	1
#570760	#32865	1
#570763	Rolling	2
#570791	#33970	1
#570809	Rolling	1
#570811	#33953	2
#570869	#33881	1
#570901	Rolling	1
#570910	#33749	2
#570932	#33953	2
#570952	#32587	1

Configurations

• netcoreapp5.0-Windows_NT-Debug-x64-CoreCLR_checked-Windows.10.Amd64.Open
• netcoreapp5.0-Windows_NT-Debug-x64-CoreCLR_release-(Windows.Nano.1809.Amd64.Open)windows.10.amd64.serverrs5.open@mcr.microsoft.com/dotnet-buildtools/prereqs:nanoserver-1809-helix-amd64-08e8e40-20200107182504
• netcoreapp5.0-Windows_NT-Debug-x64-CoreCLR_release-Windows.10.Amd64.Server19H1.ES.Open
• netcoreapp5.0-Windows_NT-Debug-x64-Mono_release-(Windows.Nano.1809.Amd64.Open)windows.10.amd64.serverrs5.open@mcr.microsoft.com/dotnet-buildtools/prereqs:nanoserver-1809-helix-amd64-08e8e40-20200107182504
• netcoreapp5.0-Windows_NT-Debug-x86-CoreCLR_release-Windows.10.Amd64.Server19H1.Open
• netcoreapp5.0-Windows_NT-Release-x64-CoreCLR_release-(Windows.Nano.1809.Amd64.Open)windows.10.amd64.serverrs5.open@mcr.microsoft.com/dotnet-buildtools/prereqs:nanoserver-1809-helix-amd64-08e8e40-20200107182504
• netcoreapp5.0-Windows_NT-Release-x64-CoreCLR_release-Windows.10.Amd64.Server19H1.Open
• netcoreapp5.0-Windows_NT-Release-x64-CoreCLR_release-Windows.10.Amd64.ServerRS5.Open
• netcoreapp5.0-Windows_NT-Release-x86-CoreCLR_release-Windows.10.Amd64.Server19H1.ES.Open
• netcoreapp5.0-Windows_NT-Release-x86-CoreCLR_release-Windows.10.Amd64.Server19H1.Open
• netcoreapp5.0-Windows_NT-Release-x86-CoreCLR_release-Windows.10.Amd64.ServerRS5.Open

Helix Logs

Build	Pull Request	Console	Core	Test Results
#565812	#33758	console.3b55a635.log		testResults.xml
#567106	#33860	console.35442594.log		testResults.xml
#567106	#33860	console.558c9de6.log		testResults.xml
#567452	#33841	console.e852ee4a.log		testResults.xml
#568318	#33911	console.531601a9.log		testResults.xml
#568318	#33911	console.b7094c9a.log		testResults.xml
#569240	#33874	console.778b6ce6.log		testResults.xml
#569624	#33961	console.caa70bbc.log		testResults.xml
#569624	#33961	console.3d5253fa.log		testResults.xml
#569624	#33961	console.e89e11e3.log		testResults.xml
#569691	#33820	console.e76e8138.log		testResults.xml
#569691	#33820	console.3e95bd35.log		testResults.xml
#569694	#32587	console.d610294d.log		testResults.xml
#569694	#32587	console.ac9623b9.log		testResults.xml
#569694	#32587	console.aa612ef9.log		testResults.xml
#569775	#33967	console.f9cd56b4.log		testResults.xml
#569778	#33946	console.d3090095.log		testResults.xml
#569904	#33970	console.30eb6b28.log		testResults.xml
#569926	#33959	console.c6631a59.log		testResults.xml
#569968	#33175	console.6af85bdd.log		testResults.xml
#569968	#33175	console.0f846131.log		testResults.xml
#569980	Rolling	console.6f54e640.log		testResults.xml
#570254	#33459	console.3386c112.log		testResults.xml
#570266	#33984	console.895ef3a3.log		testResults.xml
#570305	Rolling	console.2f502ada.log		testResults.xml
#570305	Rolling	console.cd74224e.log		testResults.xml
#570305	Rolling	console.afdf118a.log		testResults.xml
#570426	#33485	console.b348c7cc.log		testResults.xml
#570477	#32587	console.dc4eff19.log		testResults.xml
#570477	#32587	console.39e40f1f.log		testResults.xml
#570525	#33990	console.51fba67a.log		testResults.xml
#570532	Rolling	console.ceb1f32b.log		testResults.xml
#570537	#33991	console.8388cb31.log		testResults.xml
#570549	#33994	console.82ff7d64.log		testResults.xml
#570584	#33984	console.9cff2320.log		testResults.xml
#570613	#33945	console.2d490471.log		testResults.xml
#570613	#33945	console.710c64a6.log		testResults.xml
#570636	Rolling	console.3494e63d.log		testResults.xml
#570652	#33485	console.7dbcd842.log		testResults.xml
#570652	#33485	console.b10f9e51.log		testResults.xml
#570690	Rolling	console.957e50f2.log		testResults.xml
#570690	Rolling	console.c3d44da8.log		testResults.xml
#570708	#32592	console.899a02e1.log
#570741	#33459	console.f8dbd632.log		testResults.xml
#570746	#34006	console.03e5bffc.log		testResults.xml
#570760	#32865	console.b5d5f1eb.log		testResults.xml
#570763	Rolling	console.43112406.log		testResults.xml
#570763	Rolling	console.560d31c2.log		testResults.xml
#570791	#33970	console.a5bbaa20.log		testResults.xml
#570809	Rolling	console.80e19be6.log		testResults.xml
#570811	#33953	console.a46c6705.log		testResults.xml
#570811	#33953	console.6ebf110e.log		testResults.xml
#570869	#33881	console.1fa44965.log		testResults.xml
#570901	Rolling	console.97152cf9.log		testResults.xml
#570910	#33749	console.1c8bb746.log		testResults.xml
#570910	#33749	console.607475c0.log		testResults.xml
#570932	#33953	console.92bb5ee2.log		testResults.xml
#570932	#33953	console.61330cdc.log		testResults.xml
#570952	#32587	console.cf9b84af.log		testResults.xml

runfo tests -d runtime -c 100 -pr -n "System.Net.Http.WinHttpHandler.Functional.Tests Work Item" -m -g tests

[alnikola]

@jaredpar @stephentoub I believe it's a mistake. CI is failing due to a different test System.Net.Http.Functional.Tests.PlatformHandlerTest_Cookies_Http2.GetAsync_ReceiveSetCookieHeader_CookieAdded tracked by this issue #33930

Current issue is about UseClientCertOnHttp2_OSSupportsIt_Succes test which cannot be run on CI because requires Win 10 OS version 2004 or higher.

[alnikola]

Just to clarify. This issue is valid, but only for a local test run. In contrast, the other issue #33930 affects the whole CI.