Investigate flaky tests on FreeBSD

[mscdex]

• Version: master
• Platform: freebsd
• Subsystem: child_process, http, timers

I saw these three unrelated tests fail recently on CI: https://ci.nodejs.org/job/node-test-commit-freebsd/4650/nodes=freebsd10-64/console

• parallel/test-child-process-fork-dgram

not ok 129 parallel/test-child-process-fork-dgram
# TIMEOUT
  ---
  duration_ms: 60.55

• parallel/test-http-server-consumed-timeout

not ok 546 parallel/test-http-server-consumed-timeout
# 
# assert.js:85
#   throw new assert.AssertionError({
#   ^
# AssertionError: Request timeout should not fire
#     at Object.exports.fail (/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd10-64/test/common.js:443:10)
#     at IncomingMessage.req.setTimeout (/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd10-64/test/parallel/test-http-server-consumed-timeout.js:13:12)
#     at emitOne (events.js:96:13)
#     at IncomingMessage.emit (events.js:188:7)
#     at Socket.<anonymous> (_http_server.js:310:50)
#     at emitNone (events.js:86:13)
#     at Socket.emit (events.js:185:7)
#     at Socket._onTimeout (net.js:342:8)
#     at ontimeout (timers.js:365:14)
#     at tryOnTimeout (timers.js:237:5)
  ---
  duration_ms: 1.943

• parallel/test-timers-same-timeout-wrong-list-deleted

not ok 970 parallel/test-timers-same-timeout-wrong-list-deleted
# 
# assert.js:85
#   throw new assert.AssertionError({
#   ^
# AssertionError: Elapsed time does not include second timer's timeout.
#     at process.<anonymous> (/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd10-64/test/parallel/test-timers-same-timeout-wrong-list-deleted.js:26:10)
#     at emitOne (events.js:101:20)
#     at process.emit (events.js:188:7)
  ---
  duration_ms: 1.69

[jasnell]

Seeing this also: https://ci.nodejs.org/job/node-test-commit-freebsd/4673/nodes=freebsd10-64/console

not ok 970 parallel/test-timers-same-timeout-wrong-list-deleted
# 
# assert.js:85
#   throw new assert.AssertionError({
#   ^
# AssertionError: Elapsed time does not include second timer's timeout.
#     at process.<anonymous> (/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd10-64/test/parallel/test-timers-same-timeout-wrong-list-deleted.js:26:10)
#     at emitOne (events.js:101:20)
#     at process.emit (events.js:188:7)
  ---
  duration_ms: 1.71
  ...

[Trott]

And this: https://ci.nodejs.org/job/node-test-commit-freebsd/4670/nodes=freebsd10-64/console

not ok 856 parallel/test-net-write-after-close
# TIMEOUT
# about to try to write
  ---
  duration_ms: 60.135

[jbergstroem]

Might be worth mentioning that I've seen increased amounts of processes lingering. Usually a bad sign as well as increasing the failure rate of future tests.

[Trott]

Yeah, I kind of suspect that when we see three tests failing, that the first one failed and didn't get cleaned up, and that's what's making the subsequent ones fail. Not always the case, but often enough...

[jbergstroem]

I cleaned a host and ran a test against master. Here's the failed tests:

not ok 131 parallel/test-child-process-fork-dgram
not ok 247 parallel/test-dgram-send-callback-multi-buffer
not ok 454 parallel/test-http-client-timeout-with-data
not ok 545 parallel/test-http-server-consumed-timeout
not ok 962 parallel/test-timers-blocking-callback

stalled processes after that run:

parallel/test-child-process-fork-dgram.js child