Reduce CoreMark benchmark noise in CI - #124
Conversation
The benchmark job measured the PR and main once each and reported the difference to three decimal places, which reads as a precise result but is not one. Across the 41 PRs that carry a bot comment, the reported delta ranges from -13.23% to +13.39%, and PRs that only touch docs, the README, the logo or CI config still move it by as much as 3.80%. Two things make the measurement worse than it needs to be. A single CoreMark run scores itself from wall-clock time over roughly fifteen seconds on a shared runner, so one sample carries several percent of noise on its own. On top of that main was always measured second: the mean delta over those 41 PRs is -1.13% and 27 of them are negative, which a sign test puts at p = 0.03 against an even split. Build both bench binaries up front, stage each one next to the wasm it loads, then run them alternately and report the median of five samples per side. Swapping which side goes first on every other sample keeps the ordering from favoring either one. The comment now carries the observed sample range and marks a delta that falls inside it as noise. This does not make scores comparable between runs and cannot: baseline scores for main span 230 to 498 depending on which CPU model the runner lands on. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Welcome, new contributor! Please make sure you've read our contributing guide, as well as our policy regarding AI usage, and we look forward to reviewing your pull request shortly |
CoreMark Benchmark ResultsCurrent Score: 265.252 |
Code Coverage ReportCurrent Coverage: 95.38% |
|
The main thing CoreMark benchmarks should measure is how much extra work a PR adds. Instruction counts should be a good enough stand-in that isn't affected by the hypervisor, and you should be able to get them via |
A CoreMark score mostly measures how fast the runner is. What a PR can change is how much work the interpreter does, and instructions retired measures that directly: it does not move with the CPU model the runner landed on or with whoever else is on the machine, so one run per side resolves a difference that no number of timed samples can. Hardware counters are not an option here. GitHub runners are Azure VMs whose hypervisor does not expose the PMU, so perf reports cycles, instructions and branches as unsupported, and reading a counter through ptrace is out for the same reason. Count under callgrind instead, which is pure user-space instrumentation and needs no counters at all. Counting first needs a workload that does not depend on the clock. CoreMark sizes itself by timing ten iterations, multiplying by ten until that takes at least a second, then settling on iterations * (1 + 10 / floor(seconds)). The divisor is an integer, so a calibration round of 1.9s and one of 2.1s differ by nearly 2x in the work that follows. Counting that without pinning it would be worse than timing it. COREMARK_FIXED_CLOCK feeds the module a fixed table of timestamps instead, which holds it at 110 iterations with an eleven second measured window: still a valid CoreMark run, and one that scores exactly 10.0 every time. The bench asserts that score, so a module that starts timing itself differently fails the job rather than producing two counts that describe different work. The score stays, at three samples per side rather than five. It is still worth reporting what the runner managed, but it is no longer the number a regression has to be read out of. This also increments CLOCK_CALL_COUNT, which nothing ever incremented before, since the fixed clock indexes its table by it. Co-authored-by: Cursor <cursoragent@cursor.com>
A bench binary built before COREMARK_FIXED_CLOCK existed ignores it and sizes itself from the clock, so counting it against one that pins its workload compares two different amounts of work. That is the situation for this PR's own baseline, and for any branch that predates it, so the step probes each side first: a pinned run scores exactly 10.000 and costs a fraction of a second, which is a cheaper way to find out than the callgrind run it would otherwise spoil. Skip the counts with a warning in that case rather than fail, leaving the timed comparison to stand on its own as it did before. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Good call, thanks. I've pushed 07a2d16, which adds an instruction count and moves the timed score down to being context. Two things got in the way of doing it quite the way you suggested, so I'll write them down here. I couldn't get at hardware counters from these runners. They're Azure VMs and the hypervisor doesn't expose the PMU, so perf reports cycles, instructions and branches as The other thing is that the workload isn't fixed today, and counting an unfixed workload would have been worse than timing it. CoreMark picks its own iteration count from the clock. It times ten iterations, multiplies by ten until that takes at least a second, then settles on So the bench now has a Two caveats I'd rather state up front than have you find. Ir says nothing about cache or branch behaviour, and for a dispatch loop that's a fair part of what decides how long the work takes. I'd read it as a gate on work added rather than a measure of speed. Adding Also, a bench binary built before this lands ignores the variable and can't be pinned, so the step checks each side first and skips the counts with a warning instead of comparing two different workloads. That covers this PR's own baseline from On cost, AI usage, per AI_POLICY.md. As with the rest of this PR, Cursor's agent wrote the implementation and a draft of this comment. I checked the runner PMU limitation and the calibration behaviour myself before writing them down. |
|
@suthat I prepared a focused follow-up for this PR: suthat#1 issues in PR #124:
followup pull request to complete fix from PR #124It gives both builds the same benchmark, fails invalid comparisons, and reports timed results without calling them a confidence limit. Both builds produced the expected Callgrind counts instructions for the fixed workload, but it does not measure cache or branch performance. AI usage disclosure: OpenAI Codex was used to analyze specific PR #124 escape issues and devise testing protocol for fix. It was also used in implement the two workflow changes. Manually validated and reviewed the commit. AI assisted draft cross-fork PR and draft of this comment before alterations. The diff is submitted for author and maintainer review. Validationincludes Actionlint 1.7.12, workflow YAML/Bash/JavaScript syntax, formatting, bench-target Cargo check and Clippy, and an actual bootstrap check: both the PR build and current The contribution is a draft for your review. It contains one commit and changes only |
Building main against this branch's harness is what makes the two instruction counts comparable, but it can fail on its own: a harness written against an interface main does not have yet will not compile there, and the first PR to change the interpreter and the bench together would fail a job it had nothing to do with. A baseline that cannot be fetched, built or pinned warns and skips the counts, leaving the timed comparison in place. A pr side that cannot pin is this branch's own doing and still fails. Also keeps the flag on an instruction difference over 1%, reworded. The count is exact, so the figure is a reading aid rather than a threshold, but a large difference is worth putting in front of a reviewer rather than leaving it to be picked out of the numbers. Co-authored-by: arthurianresolve <268402532+arthurianresolve@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>
|
Thanks @arthurianresolve this was a real gap and I've taken most of it. The counts never firing on this PR's own run had been nagging at me, and I'd talked myself into "they start from the next PR" instead of fixing it. Copying the harness across is the obvious answer once someone says it out loud. I checked it before taking it: Two things I changed on the way in. The baseline stays best effort. Grafting a PR's harness onto an older I also kept the flag on an instruction diff over 1%, reworded. The count is exact so the threshold is only a reading aid, but I'd rather a large difference sat in front of a reviewer than be left to be picked out of the numbers. The rest went in as you wrote it are the harness copy, the executable check in Your commit is in as 64e1852 with mine on top, and your Codex disclosure is carried into the description. @h313 net effect is that the instruction counts show up on this PR's own run now, rather than starting from the next one. |
The x86 runners are Azure VMs whose hypervisor hides the PMU, so perf reports instructions, cycles and branches as unsupported and callgrind was the only way to count. It costs 29s per side, which is more than the measurement is worth. The ARM runners expose armv8_pmuv3_0, where the same count is read straight off the counter in 1.1s for both sides, and agrees with callgrind to within 0.0007%. With an exact count of the work done, timing each side repeatedly no longer buys anything, so BENCH_SAMPLES is gone along with the median, the sample range and the alternating order. The comment reports both scores without a percentage between them, since a single timed run on a shared runner is worth a few percent either way. WABT also comes out of this job, which builds and runs the bench fine without it. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Ran the same scrape independently before finding this PR, on a wider window: 98 PRs (#7–#154) against the 75 here. Every figure that overlaps matches exactly: Two things the wider window adds:
For what it's worth, I went down the more-samples road first. Against an A/A run (same binary on both sides) best-of-N was the only estimator that behaved, and even it still carried ~1.5% error at N=12. That matches the call here to stop quoting a percentage from timed runs and count the work instead. Happy to share the raw table if it's useful. AI usage per AI_POLICY.md: AI-assisted (Claude Code) for the scrape and this comment; no source modified. I ran the commands myself and every number comes from those runs. |
4 participants
Closes #76.
Problem
The benchmark job measures the PR and
mainonce each, then reports the difference to three decimal places. One CoreMark run cannot support that kind of precision. The score is worked out inside the wasm module fromclock_mswall clock time over roughly fifteen seconds, on a runner shared with other tenants.To get a sense of how big the effect is, I scraped the bot's comment from every PR that has one, 75 of them, and worked out the reported delta.
The clearest cases are the PRs that cannot have changed the code the benchmark exercises at all, so docs, the README, the logo and CI configuration.
There is a bias in the ordering as well.
mainis always measured second, and 48 of the 75 deltas are negative, which a two-sided sign test puts at p = 0.02 against an even split. That is not proof on its own, but it is what you would expect if the second run tends to land on a warmer machine.@Robbepop's point about runner hardware shows up in the same data. Baseline scores for
mainrange from 223 to 498, a factor of 2.23, in clusters that look like distinct CPU models. It does not affect the delta, since both sides are already measured in the same job, but it does mean the absolute score is meaningless outside its own run, which the comment never said.What this changes
Counting the work
Following @h313's review, the figure the comment leads with is now the number of instructions each side executes, counted once per side. It does not move with the CPU model the runner landed on, or with whoever else is on the machine, so a difference of any size in it is work this PR added or removed.
An earlier revision of this branch counted under
valgrind --tool=callgrind, on the assumption that a GitHub runner cannot read hardware counters at all. That assumption is half right, and rather than leave it sitting in the workflow I put a probe on my fork and read the answer off the machines.On
ubuntu-latestthe guest sees onlybreakpoint,kprobe,msr,software,tracepointanduprobe.instructions,cycles,branchesandbranch-missesall come back as<not supported>, even withperf_event_paranoiddropped to -1. These are Azure VMs and the hypervisor does not expose the PMU, which is runner-images#4974.ubuntu-24.04-armis a different machine. It exposesarmv8_pmuv3_0, andperf stat -e instructions:ureturns a real figure.Callgrind is also more expensive than I estimated in an earlier revision of this description.
So the benchmark job now runs on
ubuntu-24.04-armand reads the count off the counter. Valgrind is gone, and counting both sides takes 1.1s including the probe and the sysctl below.Before relying on a hardware counter I checked how steady it is, since a counter that wanders is worse than no counter. Ten runs of the pinned workload on one runner spread 1,860 instructions out of 5.59 billion, about 0.000033%, and ten runs on a second runner land inside the same window. Callgrind on the same binary reports 5,588,959,698, which is within 0.0007% of what the counter gives. Two methods that share no code landing that close together is what I wanted to see before trusting either.
The workflow's own run is the end to end version of the same check. Both sides built and measured in one job, the count came out 6,270,756,712 against 6,270,756,228 for the baseline, a gap of 484 instructions, while the timed score of those same two builds differed by 0.32%.
Two operational notes. The image ships
perf_event_paranoidat 4, which refuses to count even a process the job started itself, so the step lowers it to 1. And if the PMU ever stops being exposed on those images, the step warns and skips the counts rather than failing a PR over it, the same way it already handles a baseline that cannot be built.Counting also needs a workload that does not depend on the clock, and this one did. CoreMark picks its own iteration count by timing ten iterations, multiplying by ten until that takes at least a second, then settling on
iterations * (1 + 10 / floor(seconds)). The divisor is an integer, so a calibration round that takes 1.9s and one that takes 2.1s end up doing nearly twice as much work as each other. Counted unpinned, a PR that changed nothing could have come out +80%.COREMARK_FIXED_CLOCK=1hands the module a fixed table of timestamps instead, which holds it at 110 iterations with an eleven second measured window. That is still a valid CoreMark run, since it clears the ten second minimum and the CRC checks pass, and it scores exactly 10.000 every time. The bench asserts that score, so a module that starts timing itself differently fails the job rather than quietly yielding two counts taken from different work.A binary built before that variable existed ignores it and sizes itself from the clock, which is every baseline until this lands. So the baseline is built from
main's interpreter with this branch's bench source copied over it, holding the measuring apparatus fixed and varying only the thing being measured. That is @arthurianresolve's suggestion, from suthat#1, and it is why the counts appear on this PR's own run rather than starting from the next one.The graft can fail on its own terms, since a harness written against an interface
maindoes not have yet will not compile there. A baseline that cannot be fetched, built or pinned warns and skips the counts rather than failing a job it had nothing to do with, and the timed comparison stands. Aprside that cannot pin is this branch's own doing and does fail. The pinned run and the counted run are the same run, so that check costs nothing.One limit worth stating, and the comment states it too. Instructions retired says nothing about cache or branch behaviour, and for a dispatch loop that is a fair part of what decides how long the work takes. I would read it as a gate on work added rather than a measure of speed.
The timed score
Both bench binaries are built up front and staged next to the wasm they load, so neither side needs its revision checked out again at measurement time. Each side is then timed once.
An earlier revision of this branch took several samples per side and reported the median with the range beside it. With an exact count of the work done, that no longer buys anything, so
BENCH_SAMPLES, the median, the sample range and the alternating order are all gone. The comment shows both scores with no percentage between them, since with one run each that percentage is noise, and printing a figure while telling people not to trust it is worse than leaving it out.The job came out at 1m08s on this branch's own run, against 57s to 70s for the same job on
maintoday. WABT comes out of this job as well, since the bench builds and runs without it, which I confirmed on a clean runner.One consequence worth flagging. The timed score is measured on ARM now, so its absolute value shifts, roughly 280 where x86 gave 255. Both sides are still measured on one machine in one job so the comparison holds, and the comment already says the score means nothing outside its own run. If the timing is wanted on x86 specifically, the count can move into a small ARM job of its own, at the cost of a second job.
Two things I deliberately did not do. The score stays wall clock based, matching the reference wasm3 implementation, rather than switching
clock_msto CPU time, since that changes what the benchmark means and is worth deciding separately. And the baseline staysmainrather than the merge base, to keep this change to the measurement itself.Testing
On real runners
The probe workflows are the main evidence here, since the question was about a machine I do not own. They ran on my fork, on
ubuntu-latestandubuntu-24.04-arm, and the numbers above are read from their logs.What each architecture exposes, and what callgrind costs. Counter repeatability, ten runs each on two separate ARM runners, with callgrind on the same binary for comparison. The reworked job end to end on a pull request, which is where the 1m08s and the two counts above come from.
What the timed score can and cannot resolve
Ten runs collected while this branch touched only
.github/, so the bench binary built from this branch and the one built frommainwere byte identical and I verified the sha256. That made the workflow's own comparison an A/A test whose true delta is exactly zero, so whatever it reported is measurement error and nothing else.Taking one sample per side and applying that to each pair in turn would have reported anywhere from -3.75% to +3.57% for a change of exactly zero. That is the reason the comment no longer quotes a percentage for the score.
The pinned workload
COREMARK_FIXED_CLOCK=1scores exactly 10.000 with fourclock_mscalls, twice in a row, and runs in 0.229s locally against 20.473s for the same binary without it. Two orders of magnitude less work, and the same amount of it every time, which is what makes it affordable to count and meaningful to compare. The graft was checked the same way, somain's interpreter built with this branch'scoremark.rscompiles and scores exactly 10.000 with fourclock_mscalls.Failure paths
perfdoes not exist on macOS, so the step bodies were pulled verbatim out ofci.ymland driven under bash withRUNNER_TEMPandGITHUB_OUTPUTpointed at a scratch directory, with stubs standing in forperfand the bench binary. Eleven cases. Both sides counted, no baseline staged, a baseline that cannot pin, aprside that cannot pin, a runner with no hardware counter, noperfbinary at all, aperfrun that writes no figure, a bench that exits non-zero, the timed step with and without a baseline, and a timed run that produces no score. The warnings, the skips and the two non-zero exits all land where they should.On the comment side I rendered the artifact through the
github-scriptbody fromcomment.ymlwith the REST calls stubbed out, for a normal comparison, an instruction difference over 1%, no baseline, counts skipped with only scores present, and an artifact in the pre-change format, which is what a comment run picking up a CI run started before this merges would see.Toolchain and lint
cargo test --workspacepasses, 391 tests, withwast2jsonpresent for the spec suite.cargo fmt --checkandcargo clippy --workspace --all-targetsare clean.actionlintover the two workflows drops from 9 findings onmainto 5. The four that go were in the benchmark job's WABT install, which this PR removes. The five left are pre-existing, four in the coverage job and one on the Miri job's toolchain action, and this PR touches neither.What testing could not cover
The bot comment rendered on my fork's test PR came out in the old format, because
workflow_runalways uses the workflow file from the default branch. The new rendering is therefore only exercised locally, through the harness described above.Total clock_ms calls: 0, which I flagged earlier as out of scope, is fixed here as a side effect, since the fixed clock indexes its table byCLOCK_CALL_COUNTand so has to increment it.AI usage disclosure, per AI_POLICY.md. I used Cursor's agent for the parts where it saves real time and I can check what came back. It scraped and tabulated the 75 historical bot comments, wrote the workflow changes and the fixed clock bench mode, built the stub harnesses, and drafted this description.
The judgement calls are mine and so is the verification. I re-derived the statistics from the raw scraped scores rather than take them on trust, wrote the probe workflows because the perf question deserved a measurement rather than an argument, ran every failure path locally before pushing, and read the diff line by line. Where the agent's first answer was wrong, which is what the callgrind approach turned out to be, the fix came from going and measuring.
The commit that grafts the harness onto the baseline is @arthurianresolve's, written with OpenAI Codex and disclosed as such on suthat#1. I reproduced its result before taking it and changed how it handles a baseline that cannot build.
Scope is
.github/workflows/ci.yml,.github/workflows/comment.ymlandcrates/spacewasm_std/benches/coremark.rs. Nothing undersrc/is touched.