Test failure: System.Net.WebSockets.Client.Tests.CancelTest / ReceiveAsync_Cancel_Success (server: ws://corefx-net.cloudapp.net/WebSocket/EchoWebSocket.ashx)

Opened on behalf of @jiangzeng

The test System.Net.WebSockets.Client.Tests.CancelTest/ReceiveAsync_Cancel_Success(server: ws://corefx-net.cloudapp.net/WebSocket/EchoWebSocket.ashx) has failed.

Assert.Equal() Failure
? (pos 4)
Expected: The 'System.Net.WebSockets.InternalClientWebS···
Actual: The remote party closed the WebSocket connect···
? (pos 4)

    Stack Trace:

       at System.Net.WebSockets.Client.Tests.ClientWebSocketTestBase.<TestCancellation>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at System.Net.WebSockets.Client.Tests.CancelTest.<ReceiveAsync_Cancel_Success>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

Build : Master - 20161102.03 (Core Tests)
Failing configurations:

• Suse.132.Amd64
  • AnyCPU-Debug

Details:
https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fcli~2F/build/20161102.03/workItem/System.Net.WebSockets.Client.Tests/analysis/xunit/System.Net.WebSockets.Client.Tests.CancelTest~2FReceiveAsync_Cancel_Success(server:%20ws:~2F~2Fcorefx-net.cloudapp.net~2FWebSocket~2FEchoWebSocket.ashx)

[KristinXie1]

This issue doesn't happen for more than 2 months, so close it.

[KristinXie1]

This issue happens on Outer Loop CentOS 7.1 Debug (build#74), detail: https://ci.dot.net/job/dotnet_corefx/job/master/job/outerloop_centos7.1_debug/lastFailedBuild/consoleText

Message:

System.Net.WebSockets.Client.Tests.CancelTest.ReceiveAsync_Cancel_Success(server: ws://corefx-net.cloudapp.net/WebSocket/EchoWebSocket.ashx) [FAIL]
        Assert.Equal() Failure
                      ↓ (pos 4)
        Expected: The 'System.Net.WebSockets.InternalClientWebS···
        Actual:   The remote party closed the WebSocket connect···
                      ↑ (pos 4)
        Stack Trace:

[KristinXie1]

This issue is repro on OuterLoop_Ubuntu14.04_debug (build#113), detail: https://ci.dot.net/job/dotnet_corefx/job/master/job/outerloop_ubuntu14.04_debug/113/testReport/System.Net.WebSockets.Client.Tests/CancelTest/ReceiveAsync_Cancel_Success_server__ws___corefx_net_cloudapp_net_WebSocket_EchoWebSocket_ashx_/

So added "blocking-clean-ci" label.

[Priya91]

cc @steveharter looking at sockets tests..

[steveharter]

This test uses the Echo servers, so some fallibility expected. We need to determine if that is the case and whether retry should be applied.

List of CI failures since 1\1\2017:

Test Case: System.Net.WebSockets.Client.Tests.CancelTest.ReceiveAsync_Cancel_Success(server: ws://corefx-net.cloudapp.net/WebSocket/EchoWebSocket.ashx)
Failed Jenkins Jobs
Build Number 	Machine Name 	Date
dotnet_corefx/master/outerloop_opensuse13.2_debug 61 	suse132-20160315-outer84c770 	01/07 05:18 PM
dotnet_corefx/master/outerloop_centos7.1_debug 74 	centos71-20160609-1-outercf8ce0 	01/21 07:54 AM
dotnet_corefx/master/outerloop_opensuse13.2_release 74 	suse132-20160315-outer6597a0 	01/24 12:46 PM
dotnet_corefx/master/outerloop_win10_debug 81 	win2016-d6b040 	02/02 01:30 PM
dotnet_corefx/master/outerloop_opensuse13.2_release 83 	suse132-20160315-outer595450 	02/02 12:45 PM
dotnet_corefx/master/outerloop_centos7.1_debug 87 	centos71-20160609-1-outer88b8c0 	02/04 07:53 AM
dotnet_corefx/master/outerloop_rhel7.2_release 90 	rhel72-20160412-1-outer9b9960 	02/05 07:37 AM
dotnet_corefx/master/outerloop_ubuntu14.04_debug 113 	ubuntu1404-20160206-outer6dbc00 	02/22 07:49 PM
dotnet_corefx/master/outerloop_windows_nt_release 113 	win2012-fd8ae0 	03/01 11:07 AM

with detail from last two failures:
(windows)

System.Net.WebSockets.Client.Tests.CancelTest.ReceiveAsync_Cancel_Success(server: ws://corefx-net.cloudapp.net/WebSocket/EchoWebSocket.ashx) (from (empty))
Failing for the past 1 build (Since Failed#113 )
Took 0.11 sec.
Stacktrace

MESSAGE:
Assert.Equal() Failure\r\n ↓ (pos 4)\r\nExpected: The 'System.Net.WebSockets.InternalClientWebS···\r\nActual: The WebSocket is in an invalid state ('Aborte···\r\n ↑ (pos 4)
+++++++++++++++++++
STACK TRACE:
at System.Net.WebSockets.Client.Tests.ClientWebSocketTestBase.<TestCancellation>d__8.MoveNext() in D:\j\workspace\outerloop_win---65ef7d78\src\System.Net.WebSockets.Client\tests\ClientWebSocketTestBase.cs:line 81 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.WebSockets.Client.Tests.CancelTest.<ReceiveAsync_Cancel_Success>d__3.MoveNext() in D:\j\workspace\outerloop_win---65ef7d78\src\System.Net.WebSockets.Client\tests\CancelTest.cs:line 57 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

(Ubuntu)

System.Net.WebSockets.Client.Tests.CancelTest.ReceiveAsync_Cancel_Success(server: ws://corefx-net.cloudapp.net/WebSocket/EchoWebSocket.ashx) (from (empty))
Failing for the past 1 build (Since Failed#113 )
Took 0.28 sec.
Stacktrace

MESSAGE:
Assert.Equal() Failure\n ↓ (pos 4)\nExpected: The 'System.Net.WebSockets.InternalClientWebS···\nActual: The remote party closed the WebSocket connect···\n ↑ (pos 4)
+++++++++++++++++++
STACK TRACE:
at System.Net.WebSockets.Client.Tests.ClientWebSocketTestBase.<TestCancellation>d__8.MoveNext() in /mnt/j/workspace/dotnet_corefx/master/outerloop_ubuntu14.04_debug/src/System.Net.WebSockets.Client/tests/ClientWebSocketTestBase.cs:line 75 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.WebSockets.Client.Tests.CancelTest.<ReceiveAsync_Cancel_Success>d__3.MoveNext() in /mnt/j/workspace/dotnet_corefx/master/outerloop_ubuntu14.04_debug/src/System.Net.WebSockets.Client/tests/CancelTest.cs:line 57 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

[KristinXie1]

This issue is repro on build OuterLoop_Windows8.1_release (build#113), detail: https://ci.dot.net/job/dotnet_corefx/job/master/job/outerloop_windows_nt_release/113/testReport/System.Net.WebSockets.Client.Tests/CancelTest/ReceiveAsync_Cancel_Success_server__ws___corefx_net_cloudapp_net_WebSocket_EchoWebSocket_ashx_/

[steveharter]

See also this #17542 for analogous issue on S.N.Sockets

[steveharter]

Disabling the test for now; either:

1. The test needs to updated to account for additional WebSocketException strings (for receive case at least)
2. The test needs to remove the check for exact WebSocketException string (for receive case at least)
3. A product issue where wrong WebSocketException string is being used

[steveharter]

Moving to future; this is considered a test bug; the test needs to be updated to account for a WebSocketException that contains a string that is not hard-coded in the test. Doesn't appear to be a product issue, or at least a significant product issue, as the test is expected failure case.

[karelz]

Just curious if you were able to repro it in your "run in a loop" runs?