investigate flaky test-timers-blocking-callback

[Trott]

• Version: 11.0.0-pre (master)
• Platform: osx1012
• Subsystem: test timers

https://ci.nodejs.org/job/node-test-commit-osx/19771/nodes=osx1012/console

13:09:28 not ok 2336 sequential/test-timers-blocking-callback
13:09:28   ---
13:09:28   duration_ms: 2.270
13:09:28   severity: fail
13:09:28   exitcode: 1
13:09:28   stack: |-
13:09:28     assert.js:80
13:09:28       throw new AssertionError(obj);
13:09:28       ^
13:09:28     
13:09:28     AssertionError [ERR_ASSERTION]: timeout delayed by more than 100% (230ms)
13:09:28         at blockingCallback (/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1012/test/sequential/test-timers-blocking-callback.js:62:14)
13:09:28         at ontimeout (timers.js:457:11)
13:09:28         at tryOnTimeout (timers.js:327:5)
13:09:28         at listOnTimeout (timers.js:301:5)
13:09:28         at processTimers (timers.js:258:5)
13:09:28   ...

[Trott]

Happened again today.

https://ci.nodejs.org/job/node-test-commit-osx/nodes=osx1012/19834/console

20:24:08 not ok 2340 sequential/test-timers-blocking-callback
20:24:08   ---
20:24:08   duration_ms: 2.301
20:24:08   severity: fail
20:24:08   exitcode: 1
20:24:08   stack: |-
20:24:08     assert.js:80
20:24:08       throw new AssertionError(obj);
20:24:08       ^
20:24:08     
20:24:08     AssertionError [ERR_ASSERTION]: timeout delayed by more than 100% (250ms)
20:24:08         at blockingCallback (/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1012/test/sequential/test-timers-blocking-callback.js:62:14)
20:24:08         at ontimeout (timers.js:457:11)
20:24:08         at tryOnTimeout (timers.js:327:5)
20:24:08         at listOnTimeout (timers.js:301:5)
20:24:08         at processTimers (timers.js:258:5)
20:24:08   ...

[apapirovski]

@Trott Given that this only fails on macOS 10.12, I wonder if there's either something weird about the setup of that system or a bug in the OS itself. FWIW I'm on 10.12.6 and can't get this test to fail even in parallel with -j 96 --repeat 960.

Also, the duration of the run when it fails (2.3s) indicates that something weird is happening on that machine. This test normally takes significantly less time.

[Trott]

@apapirovski For what it's worth, since the test is in sequential, the -j 96 won't really affect it. If you want to test under load, move it to parallel and run that version with -j 96 --repeat 960 (or whatever).

[apapirovski]

@Trott yes, that is what I did to test it.

[Trott]

@Trott yes, that is what I did to test it.

Oooooohhhhhh...yeah, in that case...I'm not seeing it fail that way either, so this is probably something other than mere load (like maybe one of the two possibilities you suggest, of course)

[Trott]

Again today: https://ci.nodejs.org/job/node-test-commit-osx/20095/nodes=osx1012/console

04:20:50 not ok 2354 sequential/test-timers-blocking-callback
04:20:50   ---
04:20:50   duration_ms: 1.755
04:20:50   severity: fail
04:20:50   exitcode: 1
04:20:50   stack: |-
04:20:50     assert.js:84
04:20:50       throw new AssertionError(obj);
04:20:50       ^
04:20:50     
04:20:50     AssertionError [ERR_ASSERTION]: timeout delayed by more than 100% (250ms)
04:20:50         at blockingCallback (/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1012/test/sequential/test-timers-blocking-callback.js:62:14)
04:20:50         at ontimeout (timers.js:454:11)
04:20:50         at tryOnTimeout (timers.js:326:5)
04:20:50         at listOnTimeout (timers.js:300:5)
04:20:50         at processTimers (timers.js:257:5)
04:20:50   ...

[Trott]

FWIW, the 10.12 machines in CI are relatively small machines (I believe 3GHz 2-core 4GB RAM) so maybe that's part of what's going on?

[Trott]

@nodejs/platform-macos

[apapirovski]

I'm still completely stumped by this. I might have to request access to one of these machines to make further progress.