fix(review): stop lease cancellation storms

[steipete]

Summary

Stop the exact-review cancellation storm observed on 2026-07-28 by fixing both sides of the failure chain:

• Worker/Worker state hydration now checks out only the small git-projected trees it still consumes, with a blobless sparse checkout. It no longer materializes the roughly 42 GB records/ledger/assets projection immediately before replacing those trees from Worker storage.
• A successful review generation explicitly transitions its durable queue lease from review to finalizing. Review heartbeats retain the 20-minute crash detector; finalization retains the 130-minute execution fence and terminal-run reconciliation backstop.

The repository-variable reverter incident is intentionally out of scope.

Incident attribution

All six sampled cancelled runs had a Review exact event item job with zero executed steps. They were pending, not stalled in Codex or hydration. GitHub Actions permits one running and one pending job per concurrency group; a new same-item dispatch automatically cancels the old pending job even with cancel-in-progress: false.

Cancelled run	Item/revision	Queued	Cancelled	Attribution
30382854769	openclaw/openclaw#115308 rev 2	17:25:59Z	17:32:07Z	Replaced by same-revision dispatch 30383308463 after the six-minute dispatch lease recycled
30384016348	openclaw/openclaw#115318 rev 3	17:41:16Z	17:47:17Z	Replaced by same-revision dispatch 30384471862
30384226794	openclaw/openclaw#115307 rev 3	17:44:02Z	17:50:03Z	Replaced by same-revision dispatch 30384679786
30384244283	openclaw/openclaw#115250 rev 4	17:44:15Z	17:50:17Z	Replaced by same-revision dispatch 30384699070
30384626569	openclaw/openclaw#112808 rev 6	17:49:16Z	17:55:22Z	Replaced by same-revision dispatch 30385076875
30381694767	openclaw/clawscan#35 rev 4	17:10:33Z	17:12:55Z	Expected content supersession: rev 5 with a different head arrived while rev 4 was still pending

There is no exact-review cancellation API call in the queue or reconciler. The only repository workflow cancellation call is the unrelated state-materializer scheduled-run dedupe. The org audit log had no cancellation action matching these samples.

openclaw/openclaw#115308 lifecycle

The PR entered the durable queue as revision 2 and run 30381116855 claimed it at 17:03:23Z. The durable status comment was created at 17:04:44Z. Codex completed successfully at 17:06:32Z, the exact-review bundle completed at 17:06:34Z, and the run then stalled in setup-state from 17:06:35Z onward.

That step was still checking out the full generated-state tree before Worker hydration. The review heartbeat stopped when Codex completed, so the queue's effective lease became the last heartbeat plus 20 minutes instead of the 130-minute execution expiry. Around 17:26Z the Durable Object reclaimed the apparently stale lease and dispatched the first retry. The original GitHub job was still running, so each retry remained the single pending same-item job; every six-minute dispatch-lease recycle dispatched another copy and GitHub cancelled the prior pending copy. The PR head and review revision did not change during this loop.

The maintainer merged the PR at 18:28:03Z after posting the manual-landing explanation at 18:24:12Z. The queue's live-item probe subsequently removed the now-terminal item. The hydration source flip-flopping did not stall this run: it had already entered setup-state at 17:06Z, more than 30 minutes before those variable changes began.

Reconciler findings

The reconciler observed terminal workflow results; it did not cancel healthy runs. In the incident window, 809 reconciler runs succeeded and three scheduled sweeps failed. All three failures happened later in Recover orphaned review placeholders because the placeholder backlog exceeded its alert threshold; terminal claimed-run reconciliation had already completed. Those failures did not cause the sampled cancellations.

Changes

• Split setup-state checkout behavior:
  • Worker records + Worker ledger/assets: blobless sparse checkout of jobs, results, notifications, and root reports.
  • Any explicit git source or caller-provided sparse pattern: existing git-projected checkout behavior.
• Added the tuple-authenticated finalizing lease phase.
• Kept review-phase heartbeat expiry and newer-source generation fences unchanged.
• Added workflow and queue regression coverage plus a changelog entry.

The existing semantic edit dedupe from #857 remains intact. Unchanged retries in this incident were lease redispatches, not duplicate pull_request.edited ingress, so broadening that dedupe would not address the dominant cause.

Proof

• Focused source/workflow suite: 367 passed, 0 failed.
• pnpm run check: green (format, all TypeScript builds, four lint lanes, full unit/coverage suite).
• Local sparse-checkout proof against openclaw/clawsweeper-state: repository reports roughly 42 GB; blobless no-checkout clone was 36 MB, the Worker projection checkout was 170 MB, and clone plus checkout completed in about 30 seconds.
• /Users/steipete/.claude/skills/autoreview/scripts/autoreview --mode local: clean, no accepted/actionable findings.

Risk

Explicit git hydration remains the deliberate full-projection rollback path. A finalizer that genuinely hangs now holds its lease until the 130-minute execution expiry or terminal workflow reconciliation instead of spawning duplicate reviews after 20 minutes; newer source revisions still fence it immediately.