PR-level AI gate and decision-cost perf gate have not passed once since 2026-07-31 (58 runs, 0 successes) — CLAUDE.md's mandated AI evidence is unobtainable

[matthewevans]

Summary

The PR-level Paired-seed AI gate and Decision-cost perf gate jobs in .github/workflows/ai-gate.yml have not produced a passing run since at least 2026-07-31. Every run either hits the timeout-minutes: 60 wall and is cancelled, or fails outright.

CLAUDE.md requires these for AI work — "AI behavior changes must run cargo ai-gate and refresh baselines only with the paired-seed report attached." That requirement is currently unsatisfiable in CI: the gate produces no verdict at all, so an AI-behavior PR merges with no gate evidence rather than with a green one. The gate is not failing loudly; it is silently producing nothing.

Evidence

Sampled the 58 most recent completed AI gate workflow runs (gh run list --workflow="AI gate" --limit 60), oldest in sample 2026-07-31T22:39:27Z:

conclusion	count
cancelled (60-min timeout)	49
failure	9
success	0

Zero successes. Affected branches are unrelated to each other and include main, so this is not one bad change:

main, card/captain-marvel-apex-avenger, claude/jetfire-characters-3b456b, claude/phase-developer-track-card-o1e67k, feat/combo-detector-player-feedback, fix/5237-all-will-be-one-targets, fix/6507-depletion-sacrifice-rider, fix/6640-serpent-society-ward-poison-cost, fix/6942-ai-empty-selection, fix/ai-target-sourced-damage-lethality, fix/prune-connections-map-740b, mech/static-forced-attack, ship/ai-reliability-fixes, ship/canonicalize-delayed-trigger-lifecycle

Recent cancelled runs (all ~61 min wall clock):

run	branch	start	end
30821722308	feat/combo-detector-player-feedback	08-03T14:14	08-03T15:15
30821719213	fix/6942-ai-empty-selection	08-03T14:14	08-03T15:14
30816542439	feat/combo-detector-player-feedback	08-03T13:08	08-03T14:10
30814076784	feat/combo-detector-player-feedback	08-03T12:33	08-03T13:34
30786917498	card/captain-marvel-apex-avenger	08-03T05:21	08-03T06:23
30786622542	mech/static-forced-attack	08-03T05:14	08-03T06:15

Where the time goes

Step timings from run 30821719213 show setup is healthy and the gate step itself consumes the entire budget:

Paired-seed AI gate

Restore AI card data caches   14:14:37 → 14:14:42   (5s)
Generate card data            14:14:42 → 14:22:47   (8m05s)
Run quick AI gate             14:22:47 → 15:14:25   (51m38s, CANCELLED)

Decision-cost perf gate

Generate card data            14:14:35 → 14:22:15   (7m40s)
Run decision-cost perf gate   14:22:15 → 15:14:23   (52m08s, CANCELLED)

So ~8 minutes of the 60 is card-data generation and ~52 minutes is the gate binary, which never terminates within budget. Both jobs are timeout-minutes: 60 (ai-gate.yml:32 and :121).

Open questions for whoever picks this up

1. Did the gate get slower, or did it start hanging? A gate that legitimately needs >52 min is a different bug from one that livelocks. Worth capturing whether it is making progress when killed — the cancellation discards the step log tail, which is itself part of why this has gone unnoticed.
2. Is the 8-minute "Generate card data" step a cache miss? ai-gate.yml cardgen cache key never sees AtomicCards.json (restores before download): stale card-data + phantom sharing with ci.yml #5456 (closed) was a cardgen cache-key ordering bug in this same workflow; worth confirming that fix is still effective before assuming the budget is the problem.
3. Should the gate fail loudly instead of being cancelled? A cancelled job reads as neutral in the checks UI and does not block anything, which is how three days of no coverage went unremarked. A hard failure, or a much shorter timeout with an explicit "gate did not converge" message, would surface this immediately.

Related

• Nightly AI gate + decision-cost monitors fail on a redirect, so the gate never runs #6816 — Nightly AI gate + decision-cost monitors fail on a redirect, so the gate never runs. Distinct but same family: that issue covers the two Nightly * jobs, which are skipped on PR runs. This issue covers the two PR-level jobs. Both halves of the AI gate are currently non-functional, by different mechanisms.
• ai-gate.yml cardgen cache key never sees AtomicCards.json (restores before download): stale card-data + phantom sharing with ci.yml #5456 (closed) — prior cardgen cache-key defect in this same workflow.

Confidence

• Confirmed by direct query of the GitHub Actions API: the 58/0 success rate, the branch list, and the step timings above.
• Not established: the root cause of the 52-minute gate step. I have not run the gate locally and am not claiming to know whether it is slow, hung, or livelocked.

[matthewevans]

Profiling result: where the 60 minutes goes, and when it broke

All figures below are computed from the GitHub Actions job/step API and the job logs. Commands are given so each number can be re-derived.

1. The gate is not a test suite — it is 30 self-play games in a debug build

.github/workflows/ai-gate.yml:49 runs cargo ai-gate --games 10, and .cargo/config.toml:16 defines ai-gate = "run --bin ai-gate --" — no --release, no --profile. The logs confirm it:

Running `target/debug/ai-gate --games 10`

Three matchups (red-mirror, affinity-mirror, enchantress-mirror) × 10 games = 30 games, played by a debug-compiled engine.

2. Compile is NOT the problem — game execution is

Splitting the step at cargo's Finished line:

run	date	compile	play
30045434134	07-23	9.2 min	38.0 min
30089373210	07-24	9.0 min	33.7 min
30199759810	07-26	9.4 min	48.4 min
30473302314	07-29	2.2 min (warm)	41.4 min

Compile is a stable ~9 min cold, ~2 min warm. Every minute of growth is in playing games. Build caching will not fix this.

3. Matchups run in PARALLEL — wall time is the slowest matchup, not the sum

crates/phase-ai/src/duel_suite/run.rs:323 run_matchups_parallel spawns available_parallelism().min(run_total) workers — 3 on a 4-vCPU hosted runner — one matchup each. So the [N/3] prefix is completion order, and total wall time = max over matchups.

Per-matchup durations (gate start → that matchup's done line):

gate start	red-mirror	affinity-mirror	enchantress-mirror	total (=max)	completed
07-23 21:29	21.0m	29.1m	37.9m	37.9m	3/3
07-24 11:32	12.7m	32.7m	33.6m	33.6m	3/3
07-26 11:24	21.4m	25.1m	48.4m	48.4m	3/3
07-29 01:08	25.1m	41.5m	44.1m	44.1m	3/3
07-29 17:04	21.5m	36.8m	41.3m	41.3m	3/3
07-30 20:21	33.8m	—	—	timeout	1/3
07-30 23:50	37.9m	—	29.4m	timeout	2/3
07-31 01:32	44.0m	—	—	timeout	1/3
08-01 00:17	—	—	—	timeout	0/3
08-02 00:08	—	—	—	timeout	0/3
08-03 00:29	41.6m	—	—	timeout	1/3
08-03 14:24	43.2m	—	—	timeout	1/3

enchantress-mirror was the critical path in every run that completed. It is the matchup that decides the gate's wall time.

4. When it broke

red-mirror is the useful tracer: it is the fastest matchup and therefore still completes in most timed-out runs.

• Through 07-29T17:04: red-mirror runs 12.7–25.1 min, and 3/3 matchups finish.
• From 07-30T20:21 onward: red-mirror runs 33.8–44.0 min, and only 0–2 of 3 matchups finish.

The last successful AI gate run in the entire recorded history is 30473302314, 2026-07-29T16:59 (minion_vehicles_axis, ec7601a408). Everything after is cancelled or failure.

So the break lies between 07-29T17:04 and 07-30T20:21 — roughly a 1.7–2× slowdown in per-matchup game execution. 27 commits landed in that window (git log --since=2026-07-29T17:00 --until=2026-07-30T21:00).

5. Why there is no safety valve

PlannerServices::with_deadline force-overrides the search wall-clock budget to Deadline::none() whenever execution_mode.is_measurement(), and cargo ai-gate runs in measurement mode. Every deadline.expired() guard is a dead branch on this path. A change that deepens or widens the search therefore translates directly into gate wall-clock with nothing to bound it.

run.rs:341-345 corroborates that the search runs deep here — workers are given a 32 MiB stack specifically because "the determinized Hard+ search recurses deep" under "a debug + measurement build with no wall-clock bail".

6. Candidate commits — NOT asserted as the cause

I could not run a local A/B (this machine is at 99% disk; building ai-gate at two SHAs in isolated target dirs would risk exhausting it). So the following are ranked by prior, not proven:

commit	when	why suspected
bbb176eb75	07-29 17:33	feat(phase-ai): add vehicles deck-feature axis + VehicleDeploymentPolicy — adds a policy to the per-candidate scoring loop. Lands 29 min after the last fast run and before every slow one. Best positional fit.
e6c6bb07c4	07-30 20:02	fix(ai): bind AI actions to engine-issued proposals — interaction binding adds per-simulated-action work (semantic_slots, slot-vec clone, a format! per bound slot) inside the search hot path.
d05ee575d6	07-30 18:44	perf(ai,engine) — mimalloc + incremental layers flush; a perf change that could have regressed this workload.
a2b2c311cb, 3f276a584a	07-29 17:46 / 18:55	repeated-optional-payment modals — more decision states for the search to enumerate.

The decisive experiment is cheap for anyone with disk headroom: build ai-gate at ec7601a408 (last known-fast) and at 73fd7f6de9 (end of window), run --games 10 --suite-filter red-mirror --seed <fixed> on both, compare wall time. Bisect within the window from there.

7. The 82 failure runs are a separate issue

Sampled 30776424365: it is a compile error, not a gate failure —

error: function `delayed_trigger_payload_matches` is never used
error: could not compile `phase-engine` (lib) due to 1 previous error

Branch code failing -D warnings. Unrelated to the timeout; noted so it is not conflated.

Recommended fixes, in order of leverage

1. Stop running the gate in a debug build. This is the largest single lever and — critically — it is verdict-neutral:

• The paired-seed verdict is behavioural: the report compares baseline p0% / current p0% / flips W→L / flips L→W / sign p. No timing term.
• The decision-cost gate is counter-based, and ai-gate.yml:138-140 already states it outright: "debug profile (authoritative): counter VALUES are profile-independent."

So an optimized profile changes wall-clock only. [profile.server-release] already exists in the workspace for exactly this class of native compute binary (added because fat-LTO [profile.release] — which is tuned for WASM size, opt-level='z' — pushed a build to ~34 min against a 35-min timeout).

⚠️ Coupling to check first: both jobs share cache-shared-key: rust-ai-gate, and the perf gate's comment notes it relies on the win-rate job to keep the debug cache warm. Re-pointing only the win-rate job would leave the perf gate cold. Move both, or give them separate cache keys.

2. Parallelize at the game level, not the matchup level. Currently the unit of parallelism is the matchup — 3 work items on 3 workers — so wall time is pinned to enchantress-mirror while the other two workers idle. On 07-26, red finished at 21.4 min and the job ran to 48.4: one core sat idle for 27 minutes. With 30 games as the work unit, the same hardware finishes in roughly total_work / cores. The seed derivation is already per-(matchup, game) and results are documented as byte-identical regardless of scheduling, so this preserves determinism.

3. Only then, chase the 07-29→07-30 regression. Fixing the profile buys the headroom to investigate without every run costing an hour.

4. Do not simply raise timeout-minutes. The trend is monotone; 90 minutes buys weeks, not a fix, and doubles the billed failure case.

Reproduction commands

gh run list --workflow="AI gate" --limit 300 --json databaseId,headBranch,headSha,conclusion,createdAt,updatedAt
gh api repos/phase-rs/phase/actions/runs/<run>/jobs --jq '.jobs[]|select(.name|test("Paired-seed"))|.id'
gh api repos/phase-rs/phase/actions/jobs/<job>/logs | grep -aE "Running .target/|done \(games"

Confidence

• Confirmed: the debug invocation, the parallel matchup structure, the compile/play split, all per-matchup timings, the completion counts, the last-success timestamp, the deadline nulling under measurement mode, and both gates' verdicts being profile-independent.
• Not established: which commit caused the regression. Section 6 is a ranked hypothesis list, and the A/B in that section is what would settle it.

[mike-theDude]

Triage analysis

Clarified summary: The PR-level AI workflow did not reliably produce both required gate verdicts, preventing AI changes from obtaining the project's mandated paired-seed and decision-cost evidence.

Expected behavior: Both PR jobs finish within their budgets, upload their evidence, and return a meaningful pass/fail verdict.

Observed behavior: The original sample had no successful workflow runs. Subsequent work in #6969 and #6998 improved throughput, diagnostics, build-profile consistency, and measurement determinism. A current run (30964302403) shows that the paired-seed gate now completes successfully, but the decision-cost perf gate still completes with failure.

Investigation findings: The original “neither gate has passed” symptom is partially resolved, but the workflow remains unusable as a fully green PR gate. The remaining failure is localized to the decision-cost perf job rather than the prior blanket timeout/silence behavior.

Likely scope: AI gate harness, performance baselines/provenance, and the AI workflow configuration. #6969 and #6998 are directly related implementation history; #6816 and #5456 describe narrower, non-duplicate gate failures.

Acceptance criteria: On representative PR and scheduled runs, both jobs finish, publish complete evidence, and report accurate verdicts; a real decision-cost regression fails while an unchanged baseline passes.

[lgray]

🤖 AI text below 🤖

@matthewevans — I ran a local counter-level investigation alongside your
profiling comment. Your
analysis has since been overtaken by events in a good way, so I'll lead with that, then offer the one
piece our instrument can still contribute.

1. The 60-minute wall looks fixed — wall-clock evidence only

Scoped to timing deliberately; I'm making no claim here about counter verdicts. Run
31050404179 completed both PR-level
jobs — real completions, not skips:

job	duration
Decision-cost perf gate	21:51:21 → 22:02:00 (10.6 min)
Paired-seed AI gate	21:51:21 → 22:02:44 (11.4 min)

Against the 60-minute wall, and against the 41–48 min of game play alone in your §2/§3 tables.
Durations are independent of any baseline, which is why I'm citing this run for timing and
nothing else: it is on ship/storm-ci-review-followups (#7044), which also updates
perf-baseline.json, so its counter verdict reflects that update rather than mainline's state.

Two limits, so this isn't read as more than it shows:

• There is no main-scope PR-level run to point at. Every main run of this workflow is a
  schedule event — the nightly jobs — so the PR-level jobs don't execute on main at all. The
  claim is "completes on PR branches carrying fix(ai): gate projection wall-clock cap on measurement mode + unify AI gate build profiles #6998", not "green on main". Each head is verifiably
  downstream of fix(ai): gate projection wall-clock cap on measurement mode + unify AI gate build profiles #6998 (GET /repos/phase-rs/phase/compare/c75fda9e6...<head> → status: ahead).
• The most recent scheduled run,
  31000694717 (2026-08-05T11:14Z), was
  cancelled — ~15 h after fix(ai): gate projection wall-clock cap on measurement mode + unify AI gate build profiles #6998 landed. That is nightly scope (Nightly AI gate + decision-cost monitors fail on a redirect, so the gate never runs #6816's family), a heavier workload
  than this issue covers, and I have not investigated it.

Your top two recommendations both landed in #6998 (c75fda9e6, 2026-08-04T20:09Z), the day after
you posted: "gate projection wall-clock cap on measurement mode + unify AI gate build profiles" —
your §5 (the Deadline::none() override leaving no safety valve) and your recommendation 1 (stop
running in debug), in one commit.

One consequence for anyone reading the thread top-to-bottom: your §1 debug-build finding is
superseded rather than current. .cargo/config.toml:31-32 now reads

ai-gate      = "run --profile server-release --bin ai-gate --"
ai-perf-gate = "run --profile server-release --bin ai-perf-gate --"

Your §1 was accurate for the logs you sampled; it just no longer describes the tree.

On the counter side, I gather #7044 already carries the baseline update, so I'll defer to it rather
than duplicate the analysis.

2. Confirming your window, from a different instrument

Your §4 localizes the break to 07-29T17:04 → 07-30T20:21 and counts 27 commits. I reproduce exactly
27 independently:

git rev-list --count --first-parent --since=2026-07-29T17:00 --until=2026-07-30T21:00
27

3. What our differential instrument says inside your window

Everything below is differential — adjacent-commit comparisons of engine counters (layers_full_eval,
state_clone_for_legality, per-SBA-pass counts, etc.), each run against a fixed card-data root on
both sides. I'm making no claim about absolute values, and none of this is a comparison against the
committed baseline.

Two of my sample points sit inside your window: d05ee575d (your §6 suspect 3) and 73fd7f6de (your
window end). The span between them is exactly two commits, and it contains e6c6bb07c4, your §6
suspect 2:

counter	raw	per-pass	reading
game length (twin-validated: two independent per-SBA-pass counters at 0.982 / 0.983)	0.98	—	flat
crew_eligibility_scans	×1.18	×1.21	per-decision regression
restriction_static_mode_gate_scans	×1.13	×1.15	mixed
attackable_player_sweeps	×1.10	×1.13	mixed
state_clone_for_legality	×1.08	×1.10	mixed
mana_aura_trigger_scans	×1.09	×1.11	mixed
five counters (auto-tap / priority-probe family)	×0.75–0.82	×0.77	per-decision improvements

13 of 29 counters moved; the rest were identical.

A ~10–21% per-decision step with flat game length does not look like your 1.7–2× slowdown. Being
exact about what that rules out — your §6 lists five commits across four rows:

• The span is exclusive of d05ee575d, so it tests only e6c6bb07c4 and #6830. It leaves
  bbb176eb75, d05ee575d itself, and the two repeated-optional-payment modal commits
  (a2b2c311cb, 3f276a584a) untested — one of five removed, not four.
• The measurement is an aggregate over a two-commit span, so it bounds the pair, not each member.
  A flat pair removes e6c6bb07c4 as sole cause only barring a compensating ~40% improvement in
  #6830 masking a ~40% regression in it. Nothing in either diff suggests such a compensation, but
  this span cannot exclude it; splitting the pair would take one more sample point.

Caveat on the proxy. These counters come from ai-perf-gate's three scenarios at
action_cap=3000, not from ai-gate --games 10. Different workload, and the proxy's fidelity to your
wall clock is unverified. What makes it plausible is your own §5: with Deadline::none() forced in
measurement mode, work per decision translates into wall clock with nothing bounding it, and these
counters measure that work.

4. Context on where decision cost grew, if useful

Same differential caveat. Bisecting c9daf66e3..5c8d12bf5fb1 (36 commits, first-parent, sample points
partitioning the range exactly — 5+2+1+6+1+4+1+16):

• 31c260f50 (fix(ai): evaluate target-powered damage and fights #6826) dominates. Game length ×1.50, layers_full_eval ×2.49 raw / ×1.66
  per-pass, restriction_static_mode_gate_scans ×1.93 / ×1.29, state_clone_for_legality ×1.80 —
  alongside six per-decision improvements. It moves both axes at once.
• Confirmed on two independent measurement paths (dev-profile bisect and a server-release run)
  agreeing within ~3% on every counter above.
• 28 of 36 commits are provably counter-neutral — identical across all 29 counters. Not diffuse
  drift; a small number of identifiable changes.
• #6973 is the largest improvement measured. #6977 and #6998 are disclosed with their authors'
  own numbers.

5. Offer

Your §6 marks the candidates explicitly unproven and your §7 notes you couldn't run the A/B locally
(disk). If commit-level attribution still matters now that the runs complete, our counter
instrument bisects your 27-commit window locally at roughly 1/100th the cost per point of an
--games 10 wall-time A/B, and I have ~1 TB free plus a warm bisect worktree and isolated
CARGO_HOME. Your §6 experiment proper — building ai-gate at ec7601a408 and 73fd7f6de9 and
comparing --games 10 --suite-filter red-mirror on a fixed seed — needs two ai-gate builds I don't
currently have, but the disk that blocked you isn't a constraint here.

I haven't started either, because it's your investigation and #6998 may have made it moot. Say the
word.

6. Methods

• Same build recipe, same frozen card-data root, same three scenarios on both sides of every
  comparison. One 4-commit window moved zero of 29 counters — that negative control is what licenses
  reading any non-zero delta elsewhere as a real behaviour difference rather than noise.
• A frozen card-data root distorts counter magnitudes, so read ratios rather than values. Against
  a frozen snapshot my runs produced systematically shorter games than CI does against generated card
  data, and several counters move with game length, so absolute counts are not comparable across the
  two. Differential ratios within one root are unaffected — both sides share it, so it cancels —
  which is why every figure I quote is a ratio and none is an absolute count.
• Length normalisation is validated in place: sba_battlefield_snapshot_builds and
  legend_rule_mode_gate_scans are independent per-SBA-pass counters, and every normalised figure is
  suppressed unless they agree.
• wall_clock_ms is the median across the K samples, not a suite total
  (duel_suite/perf.rs: wall_clock_ms: wall[wall.len() / 2]), and the source labels it "recorded
  for human triage only; it is never compared". Reading it as a total understates the gate ~5×;
  noting it because it's an easy trap.
• No baseline was written at any point: every run used --current-output to a scratch path and never
  --refresh-baseline. crates/phase-ai/baselines/perf-baseline.json is byte-identical to
  origin/main throughout.

[matthewevans]

Resolved. Current GitHub Actions evidence shows repeated successful PR-level AI gate runs (for example 31526494844, 31526102891, 31521852642), completing in about 20 minutes. The latest failure, 31556782561, is a meaningful Decision-cost perf gate threshold verdict (four counters over baseline), while its paired-seed gate succeeded; this is no longer the reported timeout/no-verdict systemic failure and belongs with the affected PR/baseline review.