Audit the gates themselves, not just the product

[Alex Weininger (alexweininger)]

The story

A gate in a sibling suite failed 16 times in a row. Sixteen runs, sixteen red marks, zero passes. Everyone read it as "the product can't do this yet."

The product was fine. The gate's storage probe was signing its requests with a corrupted account key, so the storage emulator answered 403 to every single request. No generated app could ever have passed it, no matter how good. Ten percent of the corpus was being charged for a bug in the measuring instrument.

Nobody noticed, because nothing was watching the gates themselves. Everything was watching the product.

This PR adds the thing that watches the gates.

What it does

npm run gate-health reads runs that already happened — free, no model tokens, no new run — and asks one question per gate: has this gate ever actually done its job?

GATE                                            pass  fail  n/att   n/a  runs   rate  VERDICT
requirements                                       5     1      1     0     7    83%  healthy
Agent should have opened the requirements webv…   12     0      2     0    14   100%  never-failed
model gpt-5 resolved and the agent responded       0     0      1     0     1    n/a  never-attempted

Then, underneath, the part that matters — WHAT TO GO AND LOOK AT, with the specific gate, the reason, and the command to reproduce it:

NEVER PASSED — a gate that ran and never once succeeded is more likely broken than
the product is to be uniformly incapable of exactly that one thing.
  * requirements
      1 failure(s), 0 passes across 1 run(s)
      e.g. FAIL: requirements.json satisfies the requirements contract — Expected "No datastore…
      reproduce: npm run regrade -- 2026082582510393

How to read the five verdicts

Verdict	In plain terms	Why you'd care
never-passed	It ran, and never once succeeded	The gate is probably broken, not the product
never-failed	It has never gone red	It may not be measuring anything
always-not-applicable	It never rendered a verdict at all	It's reporting a perfect score while testing nothing
never-attempted	It never got the chance to run	Something upstream is broken; this gate is innocent
healthy	Has both passed and failed	Nothing to do

Limits — please read these before quoting anything

None of these verdicts proves a defect. Every one of them is a reason to look, not a finding. A gate can be legitimately never-failed because the product is genuinely good at that thing.

Three runs minimum. Below --min-runs (default 3) a verdict is printed but marked (low confidence) and never fails the build. Two runs agreeing is a coincidence with a label on it.

A young suite looks "broken" by these metrics and isn't. Right now 25 of 30 gates are never-failed. That is what a three-week-old suite with 1–14 observations per gate looks like. The number worth watching is whether that ratio survives the corpus growing — not its value today.

Only never-passed, with enough runs behind it, sets exit 1.

The porting problem, and two things found on the way

#1669 reads a cor-validation.json with a per-gate status and an explicit notAttempted flag. That file does not exist in our world. MSBench records only passed: true|false plus a nullable error, so all five distinctions are reconstructed from four artifacts: eval.json, the exec table in session.sqlite (which is where exit 1 and exit 3 stay distinct), error.json, and final-agent-config.json (so a run that produced no eval.json can still name the gates that never ran).

Two findings came out of that, and both are bigger than the tool.

1. Our historical results are wrong in both directions

#1669 models cascade failures — downstream gates recorded as failed when they never executed. We have that too, but we have something it doesn't model: void runs also manufacture fake passes.

Run	What happened	What was recorded
2026082583236973	Rate limited. The agent produced literally nothing.	1/4 — including a PASS for Agent should not fall back to the chat question tool
2026082467189297	Extension never activated.	4/7 — where all four "passes" are the negative assertions

A COUNT(*) = 0 assertion is trivially true against an empty table. So a run where the agent never ran collects free passes on every negative check. 7 of 26 instances in the corpus are void. This tool discards everything from a void instance, passes included.

The liveness sentinel (#1706) fixes this going forward. It does nothing for what's already recorded — so any run predating #1706 may contain vacuous passes, and anyone trend-plotting historical runs should know the older half of the corpus is contaminated in both directions.

2. Not-applicable results must never be counted as verdicts

The incoming fidelity and runtime gates emit NOT_APPLICABLE gate=… class=… reason=… on stderr and exit 3, which MSBench records as passed: false. So an N/A is scored as a failure, and a gate that's N/A across the whole corpus reads 0-for-16 — the story at the top of this PR, except the gate is fine and the environment is the problem. Not hypothetical: the five runtime-* gates emit functionsHostUnavailable on every current stimulus, because all four are Azure Functions and the container has no func binary.

This convention was reversed mid-review, and the reversal is the interesting part. Exit 0 was ruled first, explicitly on the grounds that this tool's always-not-applicable verdict made it safe. That premise was false, and this PR is what showed it: MSBench writes exitCode = 0 as passed: true, resolved derives from it, and the run-analysis site, msbench-cli report and Kusto all publish that number. This report could say "not applicable" while the headline said green — and nobody investigates green. Observing inflation is not the same as being able to undo it. Exit 3 is pessimistic and recoverable; exit 0 was optimistic and unrecoverable.

Because detection keys off the marker, not the exit code, the reversal needed no code change to the bucketing. Three behaviours are a contract:

1. N/A is its own bucket. Never folded into passed — and under exit 3, never into failed, which is now the live risk and would charge the product for a missing binary.
2. Every rate excludes N/A from both numerator and denominator. The rate column prints n/a, meaning nothing was judged — not 0%, not 100%.
3. Grouped by reason code. This is what turns five red gates into one install command.

class= splits the two kinds, and the mapping is mechanical rather than a lookup table here, so a new reason code can't silently default into the wrong bucket:

class=	Tallied as	Reading
outOfScope	notApplicable	The gate was wired to a stack it can't answer for — a wiring bug with an owner. "Delete it" is only right if it's out of scope for every stack, which this report can't tell you alone.
environmentGap	notAttempted	The gate applies; the machine can't run it. Correctly stays loud — we genuinely aren't testing something we claim to.

Absent or unrecognised class= reads as environmentGap — the safe direction, since noProjectManifestFound most likely means the tree was never staged, and calling that dead weight would invite deleting a gate to fix a staging bug.

Verified against fixtures, since no grader emits the marker yet and this branch is load-bearing. All four cases discriminate correctly, and MSBench recorded every one of them as passed: false:

Fixture	Tallied
N/A, class=environmentGap, exit 3	notAttempted, reason functionsHostUnavailable — not a failure
N/A, class=outOfScope, exit 3	notApplicable — not a failure
Genuine crash: exit 3, no marker	notAttempted, GRADER_EXIT_3 — not a product failure
Real product failure: exit 1	failed

The five-gate Functions case collapses to exactly one actionable line instead of five mystery failures:

NEVER ATTEMPTED — these never got the chance to run. This indicts whatever is
upstream of them, not the gates, and it says nothing at all about the product.

  * functionsHostUnavailable — 5 gate(s) blocked, 0 real verdicts between them
      runtime-app-starts, runtime-crud, runtime-frontend,
      runtime-frontend-api, runtime-health

Without the grouping, that reads as five gates failing on every run — the fastest route to somebody switching them off.

Where the data lives

The worry going in was that this would only ever see one laptop's runs. It's better than that, and the detail is worth knowing:

• Run data is remote. msbench-cli extract is backend-served, so any run id you have access to can be audited from any machine. The local cache is a cache, not the source.
• Run discovery is local-only today. The CLI already has list runs --kusto, but MSBench User doesn't appear to grant Kusto read: Principal 'aaduser=…' is not authorized to read database 'ces_telemetry_prod' (and 'msbench' on AME). That's a filable access gap and the whole fix for discovery.
• Kusto couldn't answer this even with access. The ingested views carry run/instance status, timings and resolved rate. Per-assertion details[] is ingested nowhere.

So the tool is run-id-driven and indifferent to provenance. The day Kusto read lands, list runs --kusto piped into npm run gate-health works with no change to the tool.

Two things found by interrogating the tool, not by admiring it

A prediction that was wrong, and the real bug it exposed

It was predicted that the sentinel gate would be mis-detected: the stimulus YAML declares SELECT COUNT(*) > 0 FROM llm_responses, the harness appends WHERE stepIndex = :stepIndex, so any declared-vs-stored comparison done by equality on query text would never match and the gate would report never-attempted against runs that genuinely carry it.

Tested, and the prediction was false — that code path was never in play. The declared-gate catalogue matches on the assertion comment, never on query text, so the appended filter is irrelevant to it. That retires the equality-vs-substring concern properly: not "we think it's fine", but "here is why it cannot arise".

The test exposed a different, real defect. The sentinel did not flip to healthy, for a cause nobody had predicted — sibling stimuli word the same gate differently:

Stimulus family	Sentinel comment
plan	Sentinel; session data must exist or the negative checks below are vacuous
scaffold	Sentinel; session data must exist or every check below is vacuous

Run 2026082618693091 carries the second and passes it, so the sentinel is executing. What remains never-seen is the first wording — correctly, since those stimuli haven't been re-run since #1706. Two wordings of one gate, two histories, and the report was right both times.

The two scaffold runs show the same drift on two further gates:

2026082618693091	2026082619460117
Agent should not open the plan view to approve the plan itself	Agent should not take over planning by opening the plan view
Agent should not fall back to the chat question tool	Agent should refuse with a message, not by asking a chat question

This is the comment-identity fragility documented below, live, three commits after it was written down as theoretical — and it is outside what filename identity can fix, because these are SQL assertions with no grader file. The fix belongs upstream: one canonical string per shared gate, plus a drift check. This report is the evidence that the divergence is real.

A missing run — chased to ground, and the guard it produced

Auditing two scaffold runs, 2026082619460117 extracted with only run_metadata.json and no instances, while being reported elsewhere as 7/7 resolved: true. A run reporting green while its artifact is absent is the recurring failure class here, so it was chased rather than assumed.

Two hypotheses were on the table: reconciliation lag (the blob hadn't landed when it was read) and an order-dependent reader (the archives differ in member order — output.zip is first in the run that read fine, third in the one that didn't).

It was lag, and it is settled by three independent checks:

Check	Result
Timestamps	results.zip written 22:28:22; the read was ~22:23:30 — five minutes before the file existed
Pre-registered test — load 2026082620153444, output.zip at index 4, complete artifact	Reads fine: 1 instance, 6 pass / 1 fail
Multi-instance regression — 2026082603685803, output blobs at indices 4, 6, 10	All 3 instances loaded, not one

findInstances is a total readdirSync scan — no break, no first-match, no index assumption — and was separately verified against a directory whose -output entries are interleaved with decoys in adversarial readdir order. It also never opens an archive; msbench-cli extract does that.

The clinching detail is in the original observation: the extraction produced run_metadata.json, which is member 4, behind output.zip at member 3. A reader that bailed on a non-matching member could not have reached member 4.

The full corpus was re-run under both readers. Zero verdicts moved — 25 runs, 30 instances, 46 gates, identical tallies.

The guard is in anyway, because the underlying criticism was correct. "No instances" previously meant two different things in one quiet log line: we failed to load data that exists and there is no data. Those need opposite responses. The reader now cross-checks the cached results.zip:

• Archive contains *-output.zip but the extraction yielded nothing → READER FAULT, loud, in the preamble, naming the runs and stating that no tally below should be quoted.
• Archive has no output member → pending or missing. A corpus fact, not a bug.

A silently dropped run under-reports coverage invisibly, and it does so towards never-executed — this report's loudest verdict. That is this tool's own version of the failure class it was built to find, so it is now checked rather than assumed.

What the tool did right is the durable point. Under a transient partial artifact it printed no instances found and audited one run, instead of quietly auditing one while reporting as though it had seen two. A tool that averaged over missing data would have published a clean bill of health for half a corpus.

Known limitation: gate identity

MSBench carries no gate id — an assertion is identified only by its comment. That's unstable: requirements.json should be valid JSON carrying a questions array and requirements.json satisfies the requirements contract are the same gate before and after it moved from SQL to exec:, so by comment they look like two gates with 7 and 3 runs instead of one with 10. A gate can silently reset its own history by being reworded.

The identity fix, measured

Same 21 runs, same data, two identity schemes:

--identity comment (the naive view)	pass	fail	runs	verdict
requirements.json satisfies the requirements contract	2	1	3	healthy
requirements.json describes an API-only Cosmos + Blob project	1	0	1	never-failed
requirements.json should decompose into exactly three services	1	0	2	never-failed
requirements.json should recommend no datastore at all	1	0	1	never-failed

--identity gate (default)	pass	fail	runs	verdict
requirements	5	1	7	healthy

Those are four invocations of the same grader, validate-requirements.ts, differing only in flags. Under comment identity they fragment into four rows, three of which look like never-failed gates resting on one or two runs — which is exactly the "gate silently resets its own history" defect. Only the second table is true.

Default --identity gate mitigates it by keying exec: gates on the grader filename — the same id gate= is derived from — which also recovers stable identity for runs recorded before the convention existed. It's deliberately coarser (all validate-requirements.ts invocations are one gate); --identity comment gives the raw view. It only helps exec: gates — SQL assertions have no stderr and stay comment-keyed. Retrofitting existing stimuli is a separate follow-up.

Scope

Three files: evals/msbench/gate-health.ts, a package.json script, and a README section. Nothing in config/, graders/, src/artifacts/ or grader-certification/, which sibling sessions are editing. npm run typecheck passes. Verified against all 21 runs / 26 instances in the local corpus, plus a synthetic fixture for the N/A path.

[Copilot]

Pull request overview

This PR adds a new MSBench utility (npm run gate-health) that audits gate health across past runs to detect “instrument failures” (e.g., gates that never pass, gates that never fail, always-N/A gates being mis-scored as passes, and gates that are never attempted), plus documentation describing how to use and interpret the report.

Changes:

• Add evals/msbench/gate-health.ts to extract and analyze prior MSBench runs, classify per-gate health verdicts, and optionally emit machine-readable JSON (failing only on confident never-passed).
• Add an npm run gate-health script in evals/package.json.
• Document gate-health usage, verdict semantics, data sources, and flags in evals/msbench/README.md.

Show a summary per file

File	Description
evals/package.json	Adds a gate-health npm script to run the new auditor.
evals/msbench/README.md	Adds a detailed “Auditing the gates themselves” section explaining verdicts, inputs, and flags.
evals/msbench/gate-health.ts	Implements the gate-health auditor (run discovery/extraction, per-instance parsing, per-gate tallies/verdicts, report + JSON mode).

Review details

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Suppressed comments (1)

evals/msbench/gate-health.ts:45

• The header comment hard-codes a specific void-instance count ("Six of twenty-six...") which is already inconsistent with the README section added in this PR ("Seven of twenty-six...") and will drift over time as the corpus grows. Consider removing the exact count (or at least keeping it in sync with the README) to avoid stale documentation.

 * passes it never earned. Six of twenty-six instances in the current corpus are void.

• Files reviewed: 3/3 changed files
• Comments generated: 2
• Review effort level: Lite