Benchmarks in CI produce erroneous results

[Trott]

Running benchmark CI jobs using PRs that only change the README file, I've found at least two that consistently report significant differences. I am not able to reproduce these erroneous results locally. Something is wrong with the benchmark CI setup, but I have no idea what.

Ref: nodejs/node#27380 (comment) and nodejs/node#27380 (comment)

Ref: nodejs/node#27081 (comment)

[github-actions]

This issue is stale because it has been open many days with no activity. It will be closed soon unless the stale label is removed or a comment is made.

[github-actions]

This issue is stale because it has been open many days with no activity. It will be closed soon unless the stale label is removed or a comment is made.

[Trott]

Checking to see if timers benchmark is still affected: https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/977/

(That's a PR doing a timers benchmark comparison using a PR that only changes a single .md file. So there should be no statistically significant difference in performance. Previously, I had consistently seen results indicating a statistically significant difference. That basically means we can't trust the benchmarks.)

[Trott]

Checking to see if timers benchmark is still affected: https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/977/

(That's a PR doing a timers benchmark comparison using a PR that only changes a single .md file. So there should be no statistically significant difference in performance. Previously, I had consistently seen results indicating a statistically significant difference. That basically means we can't trust the benchmarks.)

I've also queued a dns benchmark for the same purpose: https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/978/

[Trott]

Seems like this is no longer an issue, but two last CI runs just to make sure.

dns: https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/984/

events: https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/985/

[Trott]

Looks good. Closing.