Deferred from the review of #194.
docs/knowledge/measurement.md defines an arm as a cache reset, a cold invocation, then a warm invocation against the cache the cold run just wrote, and requires a thermal washout before each arm.
That washout equalizes the start of an arm. It cannot equalize the warm baseline, because the warm run follows that arm's own cold run — and the cold runs differ in length between the binaries under comparison. On the pin measurement, OLD's cold invocation ran roughly 7 seconds longer than NEW's, so OLD's warm run starts from a hotter machine by construction. The ≤ +5% warm limit can therefore pass or fail on carryover rather than on the change.
The bias has a known direction: it makes OLD's warm number worse, which flatters NEW. The warm results recorded on #195 (worst pair −16.5%, an improvement against a regression bound) are therefore not endangered by it, but a future comparison where warm is closer to the bound would be.
What a fix looks like
Preserve the cache the cold run produced, but insert a washout between the cold and warm invocations rather than only at the arm boundary. That changes the structure of an arm, which is why it was deferred rather than patched during review.
Open question worth settling at the same time: whether a fixed cooldown is sufficient, or whether the washout should be defined by an observed return to a thermal or frequency baseline. The protocol currently says "fixed cooling time, or measured return to baseline" without choosing.
Why it was deferred
It is a real gap and not a defect in anything measured so far. The correction is a new procedural step rather than a clause, and #194 had reached the point where each round produced another clause-sized improvement to a protocol document — the kind of scope growth that does not terminate on its own.
Deferred from the review of #194.
docs/knowledge/measurement.mddefines an arm as a cache reset, a cold invocation, then a warm invocation against the cache the cold run just wrote, and requires a thermal washout before each arm.That washout equalizes the start of an arm. It cannot equalize the warm baseline, because the warm run follows that arm's own cold run — and the cold runs differ in length between the binaries under comparison. On the pin measurement, OLD's cold invocation ran roughly 7 seconds longer than NEW's, so OLD's warm run starts from a hotter machine by construction. The
≤ +5%warm limit can therefore pass or fail on carryover rather than on the change.The bias has a known direction: it makes OLD's warm number worse, which flatters NEW. The warm results recorded on #195 (worst pair −16.5%, an improvement against a regression bound) are therefore not endangered by it, but a future comparison where warm is closer to the bound would be.
What a fix looks like
Preserve the cache the cold run produced, but insert a washout between the cold and warm invocations rather than only at the arm boundary. That changes the structure of an arm, which is why it was deferred rather than patched during review.
Open question worth settling at the same time: whether a fixed cooldown is sufficient, or whether the washout should be defined by an observed return to a thermal or frequency baseline. The protocol currently says "fixed cooling time, or measured return to baseline" without choosing.
Why it was deferred
It is a real gap and not a defect in anything measured so far. The correction is a new procedural step rather than a clause, and #194 had reached the point where each round produced another clause-sized improvement to a protocol document — the kind of scope growth that does not terminate on its own.