Investigate flaky test-http-agent

[mscdex]

Example failure on pi2-raspbian-wheezy #1
Example failure on pi2-raspbian-wheezy #2

Similar issue here as in #5938?

Output:

# TIMEOUT
#0 200
#1 200
#2 200
#3 200
#4 200
#5 200
#6 200
#7 200
#8 200
#9 200
#10 200
#11 200
#12 200
#13 200
#14 200

[mscdex]

Ref: #5346

[Trott]

While looking at this test being flaky in February, it was discovered to fail always on a connection in the low 20s. In other words, if it failed, it failed on connection 22 or 23 or 24, but never on 16 or 36 or 46 or 59 or...

Which was very odd...

And now it seems that we're seeing that number creep lower resulting in increased flakiness on Pi 2 devices. I have no explanation. It's really weird.

/cc @nodejs/build @rvagg

[Trott]

Again: https://ci.nodejs.org/job/node-test-binary-arm/1681/RUN_SUBSET=3,nodes=pi2-raspbian-wheezy/tapTestReport/test.tap-53/

Ugh. Not sure what to do. Bringing it down to 12 or so would probably fix it, but it would seem that in another month, we'd likely be facing the same issue again.

[Trott]

cc @nodejs/testing

[mscdex]

Again: https://ci.nodejs.org/job/node-test-binary-arm/1727/RUN_SUBSET=2,nodes=pi2-raspbian-wheezy/console

[santigimeno]

While looking at this test being flaky in February, it was discovered to fail always on a connection in the low 20s. In other words, if it failed, it failed on connection 22 or 23 or 24, but never on 16 or 36 or 46 or 59 or...

@Trott Would it be worth stress testing the test with the source code we had in February, just to check whether it's a problem in the code or in the raspberry bots?

[Trott]

@santigimeno I suppose that's worth a shot.

And, another one: https://ci.nodejs.org/job/node-test-binary-arm/1728/RUN_SUBSET=2,nodes=pi2-raspbian-wheezy/tapTestReport/test.tap-53/

[Trott]

And another: https://ci.nodejs.org/job/node-test-binary-arm/1723/RUN_SUBSET=2,nodes=pi2-raspbian-wheezy/console

I suppose it's superfluous to document them all here as the test is now failing so often.

[Trott]

Per @santigimeno's suggestion, here are three CI stress tests:

• the previous version that uses 100 connections
• the current version in master that uses 16 connections
• a version with the connections reduced to 9

[Trott]

And one more, also at @santigimeno's suggestion:

• reverting everything since the commit in February that last made this test not flaky

[Trott]

Confounding results: Previous version of the test (that used 100 connections) failed, of course.

But no failures on any of the other stress tests, including current master.

Maybe its flakiness is dependent on other things going on on the network or something? I mean, it shouldn't be, right? Maybe confirm that the test is using localhost and not something odd like the machine's networked IP or (would this next one even work?) a broadcast address or something...

[Trott]

Continues to fail with alarming frequency. Running CI stress test against master. https://ci.nodejs.org/job/node-stress-single-test/596/nodes=pi2-raspbian-wheezy/console