Skip to content

fix(nodeagent): blank-sheet write deadlock + accrued doom-loop breaker#215

Merged
HomenShum merged 2 commits into
mainfrom
fix/live-agent-write-path
Jul 18, 2026
Merged

fix(nodeagent): blank-sheet write deadlock + accrued doom-loop breaker#215
HomenShum merged 2 commits into
mainfrom
fix/live-agent-write-path

Conversation

@HomenShum

Copy link
Copy Markdown
Owner

Live incident this fixes

Fresh authenticated prod room, first spreadsheet command -> 11 model turns, every
update_sheet blocked, zero cells written, credits burned until manual cancel.
Full telemetry: .qa/evidence/20260718-live-agent/ (room NRNXCFYJK5B).

Two root causes, two fixes

  1. Blank-sheet preflight deadlock. verify_workbook preflight demanded an existing
    version for every target cell; first writes target cells that don't exist, re-reading
    can never version them, so preflight failed forever and the guardrail blocked all
    writes behind it. Missing version now = CREATE approved at baseVersion 0 (matching
    the write path's own contract); stale versions on existing cells still fail closed.
  2. Breaker defeated by interleaving. The consecutive identical-failure cap resets on
    any interleaved success; the doom cycle (write blocked -> inspect ok -> verify
    needs_repair) kept it at 1 forever. Added an accrued per-failure-key cap (6) that
    terminates with an honest protocol_stall handoff naming the blocked tool.

Verification

  • tests/verifyWorkbookBlankSheet.test.ts — 4 scenarios incl. fail-closed stale-version
  • tests/deterministicFailureAccrual.test.ts — interleaved loop terminates at exactly 6
  • npm run floor: 2,515/2,515, exit 0

Process disclosure

The convex functions on this branch reached zealous-goshawk-766 pre-review via
npx convex dev --once (the CLI reads .env.local directly; env -u does not
isolate — memory updated). Prod failure logs verified clean post-push. Merging
reconciles git with what prod already runs; a protocol-correct convex:deploy
from main follows the merge. Live re-verification of the fix (fresh room, same
command, cells must land) is the close-out gate.

🤖 Generated with Claude Code

…oop cap

Live incident (room NRNXCFYJK5B, 2026-07-18): a first-user "create a Q3
variance sheet" command in a fresh authenticated prod room looped 11 model
turns with every managed write blocked, wrote zero cells, and burned credits
until manually cancelled.

Root cause 1 — blank-sheet preflight deadlock: verify_workbook preflight
required an existing integer version for every target cell, but first writes
target cells that DO NOT EXIST yet; re-reading can never version them, so
preflight returned needs_repair forever and the workflow guardrail blocked
every write behind it. Fix: a missing version is a CREATE (the managed write
path already defaults creates to baseVersion 0) — preflight passes and
approves those ops at baseVersion 0; stale versions on existing cells still
fail closed. (tests/verifyWorkbookBlankSheet.test.ts, 4 scenarios)

Root cause 2 — breaker defeated by interleaving: the consecutive
deterministic-failure cap resets when any unrelated call succeeds; the live
loop alternated write(blocked) -> inspect(ok) -> verify(needs_repair) so the
count never exceeded 1. Fix: accrued per-failure-key cap (6) that survives
interleaving and terminates with an honest protocol_stall handoff naming the
tool. (tests/deterministicFailureAccrual.test.ts)

Floor: 2,515/2,515, typechecks clean (exit 0).

PROCESS NOTE: this branch's convex functions were unintentionally pushed to
zealous-goshawk-766 pre-review by `npx convex dev --once` (the CLI reads
.env.local directly; `env -u CONVEX_DEPLOYMENT` does not isolate). Prod
failure logs verified clean post-push; merging this PR reconciles git with
what prod already runs, followed by a protocol-correct convex:deploy from
main.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@vercel

vercel Bot commented Jul 18, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
noderoom Ready Ready Preview, Comment Jul 18, 2026 10:16am

Request Review

@github-actions

github-actions Bot commented Jul 18, 2026

Copy link
Copy Markdown

Scaffold Handoff — For Your Coding Agent

Your coding agent (Codex, Claude Code, etc.) should apply the accepted
scaffold proposals below. Do NOT touch any immutable files.

Immutability Check

Mode: advisory

✅ No immutable files were modified in this branch.

Changed Files

  • .proofloop/proofs/persona-dogfood/analyst-progress.json
  • .proofloop/proofs/persona-dogfood/analyst-proof-bundle.json
  • .proofloop/proofs/persona-dogfood/analyst-result.xlsx
  • .proofloop/proofs/persona-dogfood/analyst.png
  • .proofloop/proofs/persona-dogfood/finance-operator-progress.json
  • .proofloop/proofs/persona-dogfood/finance-operator-proof-bundle.json
  • .proofloop/proofs/persona-dogfood/finance-operator-result.xlsx
  • .proofloop/proofs/persona-dogfood/finance-operator.png
  • .proofloop/proofs/persona-dogfood/founder-progress.json
  • .proofloop/proofs/persona-dogfood/researcher-progress.json
  • .proofloop/proofs/persona-dogfood/researcher-proof-bundle.json
  • .proofloop/proofs/persona-dogfood/researcher-result.json
  • .proofloop/proofs/persona-dogfood/researcher.png
  • .proofloop/proofs/visual-parity/compact-1180x800.json
  • .proofloop/proofs/visual-parity/compact-1180x800/chat.png
  • .proofloop/proofs/visual-parity/compact-1180x800/deck.png
  • .proofloop/proofs/visual-parity/compact-1180x800/graph-agent.png
  • .proofloop/proofs/visual-parity/compact-1180x800/graph-controls.png
  • .proofloop/proofs/visual-parity/compact-1180x800/graph.png
  • .proofloop/proofs/visual-parity/compact-1180x800/notebook-editor.png
  • .proofloop/proofs/visual-parity/compact-1180x800/notebook-kernel.png
  • .proofloop/proofs/visual-parity/compact-1180x800/spreadsheet.png
  • .proofloop/proofs/visual-parity/compact-1180x800/trace.png
  • .proofloop/proofs/visual-parity/desktop-1456x940.json
  • .proofloop/proofs/visual-parity/desktop-1456x940/chat.png
  • .proofloop/proofs/visual-parity/desktop-1456x940/deck.png
  • .proofloop/proofs/visual-parity/desktop-1456x940/graph-agent.png
  • .proofloop/proofs/visual-parity/desktop-1456x940/graph-controls.png
  • .proofloop/proofs/visual-parity/desktop-1456x940/graph.png
  • .proofloop/proofs/visual-parity/desktop-1456x940/notebook-editor.png
  • .proofloop/proofs/visual-parity/desktop-1456x940/notebook-kernel.png
  • .proofloop/proofs/visual-parity/desktop-1456x940/spreadsheet.png
  • .proofloop/proofs/visual-parity/desktop-1456x940/trace.png
  • .proofloop/proofs/visual-parity/phone-390x844.json
  • .proofloop/proofs/visual-parity/phone-390x844/chat.png
  • .proofloop/proofs/visual-parity/phone-390x844/deck.png
  • .proofloop/proofs/visual-parity/phone-390x844/graph-agent.png
  • .proofloop/proofs/visual-parity/phone-390x844/graph-controls.png
  • .proofloop/proofs/visual-parity/phone-390x844/graph.png
  • .proofloop/proofs/visual-parity/phone-390x844/notebook-editor.png
  • .proofloop/proofs/visual-parity/phone-390x844/notebook-kernel.png
  • .proofloop/proofs/visual-parity/phone-390x844/spreadsheet.png
  • .proofloop/proofs/visual-parity/phone-390x844/trace.png
  • .proofloop/proofs/visual-parity/tablet-820x1180.json
  • .proofloop/proofs/visual-parity/tablet-820x1180/chat.png
  • .proofloop/proofs/visual-parity/tablet-820x1180/deck.png
  • .proofloop/proofs/visual-parity/tablet-820x1180/graph-agent.png
  • .proofloop/proofs/visual-parity/tablet-820x1180/graph-controls.png
  • .proofloop/proofs/visual-parity/tablet-820x1180/graph.png
  • .proofloop/proofs/visual-parity/tablet-820x1180/notebook-editor.png
  • .proofloop/proofs/visual-parity/tablet-820x1180/notebook-kernel.png
  • .proofloop/proofs/visual-parity/tablet-820x1180/spreadsheet.png
  • .proofloop/proofs/visual-parity/tablet-820x1180/trace.png
  • .qa/evidence/20260717-postmerge/a11y-final.mjs
  • .qa/evidence/20260717-postmerge/aria-instrument.mjs
  • .qa/evidence/20260717-postmerge/aria-instrument2.mjs
  • .qa/evidence/20260717-postmerge/aria-keepchain.mjs
  • .qa/evidence/20260717-postmerge/aria-probe.mjs
  • .qa/evidence/20260717-postmerge/aria-target.mjs
  • .qa/evidence/20260717-postmerge/before-after-shots.mjs
  • .qa/evidence/20260717-postmerge/before-after/after-composer.png
  • .qa/evidence/20260717-postmerge/before-after/after-hero-rescaled.png
  • .qa/evidence/20260717-postmerge/before-after/after-landing-hero.png
  • .qa/evidence/20260717-postmerge/before-after/after-mobile-recents.png
  • .qa/evidence/20260717-postmerge/before-after/after-room-sheethead.png
  • .qa/evidence/20260717-postmerge/before-after/after-room-steadystate.png
  • .qa/evidence/20260717-postmerge/before-after/after-signature-cite.png
  • .qa/evidence/20260717-postmerge/before-after/after-signature-commit.png
  • .qa/evidence/20260717-postmerge/before-after/after-signature-lock.png
  • .qa/evidence/20260717-postmerge/before-after/after-signature-smart-merge.png
  • .qa/evidence/20260717-postmerge/before-after/after-ssr-mobile-header.png
  • .qa/evidence/20260717-postmerge/before-after/after-trust-chips.png
  • .qa/evidence/20260717-postmerge/before-after/after-trust-line.png
  • .qa/evidence/20260717-postmerge/before-after/before-composer.png
  • .qa/evidence/20260717-postmerge/before-after/before-landing-hero.png
  • .qa/evidence/20260717-postmerge/before-after/before-ssr-mobile-header.png
  • .qa/evidence/20260717-postmerge/before-after/before-trust-line.png
  • .qa/evidence/20260717-postmerge/clip-probe.mjs
  • .qa/evidence/20260717-postmerge/clip/check-0.6.png
  • .qa/evidence/20260717-postmerge/clip/check-4.4.png
  • .qa/evidence/20260717-postmerge/clip/check-9.9.png
  • .qa/evidence/20260717-postmerge/clip/final-0.3.png
  • .qa/evidence/20260717-postmerge/clip/final-9.9.png
  • .qa/evidence/20260717-postmerge/clip/raw.webm
  • .qa/evidence/20260717-postmerge/clip/v2-0.3.png
  • .qa/evidence/20260717-postmerge/clip/v2-5.0.png
  • .qa/evidence/20260717-postmerge/clip/v2-9.7.png
  • .qa/evidence/20260717-postmerge/crop-before-header.mjs
  • .qa/evidence/20260717-postmerge/design-after-shots.mjs
  • .qa/evidence/20260717-postmerge/live-enum.mjs
  • .qa/evidence/20260717-postmerge/local-landing-desktop.png
  • .qa/evidence/20260717-postmerge/local-landing-mobile.png
  • .qa/evidence/20260717-postmerge/local-landing-tablet.png
  • .qa/evidence/20260717-postmerge/local-mobile-home-reduced.png
  • .qa/evidence/20260717-postmerge/local-mobile-home.png
  • .qa/evidence/20260717-postmerge/local-room-desktop.png
  • .qa/evidence/20260717-postmerge/local-room-reduced-motion.png
  • .qa/evidence/20260717-postmerge/prettify-audit.json
  • .qa/evidence/20260717-postmerge/prod-landing-desktop.png
  • .qa/evidence/20260717-postmerge/prod-landing-mobile.png
  • .qa/evidence/20260717-postmerge/record-signature.mjs
  • .qa/evidence/20260717-postmerge/room-clean-shot.mjs
  • .qa/evidence/20260717-postmerge/signature-frames.mjs
  • .qa/evidence/20260717-postmerge/trust-verify.mjs
  • .qa/evidence/20260717-postmerge/variance-color-probe.mjs
  • .qa/evidence/20260718-live-agent/gate-peek.mjs
  • .qa/evidence/20260718-live-agent/gate-peek.png
  • .qa/evidence/20260718-live-agent/peek2.mjs
  • .qa/evidence/20260718-live-agent/peek2.png
  • .qa/evidence/20260718-live-agent/stream-probe.mjs
  • .qa/memory/findings.jsonl
  • .qa/memory/runs.jsonl
  • convex/_generated/api.d.ts
  • docs/eval/OFFICIAL_BENCHMARK_READINESS.md
  • docs/eval/OFFICIAL_BENCHMARK_TASK_COVERAGE.md
  • docs/eval/OPENROUTER_CONVEX_BENCHMARK.md
  • docs/eval/agent-improvement-loop.md
  • docs/eval/agent-improvement-loop.svg
  • docs/eval/agent-improvement-loop/20260718T101550Z.json
  • docs/eval/agent-improvement-loop/latest.json
  • docs/eval/agent-workspace-sandbox-smoke.json
  • docs/eval/algorithm-artifact-smoke.json
  • docs/eval/bankertoolbench-official-contract.json
  • docs/eval/docker-sandbox-probe.json
  • docs/eval/eval-runs.jsonl
  • docs/eval/halo-convex-context-telemetry.json
  • docs/eval/halo-self-improvement-smoke.json
  • docs/eval/halo-variant-selection.json
  • docs/eval/nodemem-benchmark-report.json
  • docs/eval/official-benchmark-readiness.json
  • docs/eval/official-benchmark-task-coverage.json
  • docs/eval/openrouter-convex-benchmark.json
  • docs/eval/professional-catalog-proofs.json
  • docs/eval/professional-proof-ledger.json
  • docs/eval/spreadsheetbench-chart-visual-probe.json
  • docs/eval/traces/credit/20260718T101557826Z-2ee44939_dirty.c3677cfff6a65125/cascade-healthy.json
  • docs/eval/traces/credit/20260718T101557826Z-2ee44939_dirty.c3677cfff6a65125/delta-incomplete.json
  • docs/eval/traces/credit/20260718T101557826Z-2ee44939_dirty.c3677cfff6a65125/mapping-correct.json
  • docs/eval/traces/credit/20260718T101557826Z-2ee44939_dirty.c3677cfff6a65125/mapping-misbind.json
  • docs/eval/traces/credit/20260718T101557826Z-2ee44939_dirty.c3677cfff6a65125/summit-stressed.json
  • docs/eval/traces/ladder/20260718T101557465Z-2ee44939_dirty.01bd8b55a0c45cf9/ladder_L1_read_scripted.json
  • docs/eval/traces/ladder/20260718T101557465Z-2ee44939_dirty.01bd8b55a0c45cf9/ladder_L2_edit_scripted.json
  • docs/eval/traces/ladder/20260718T101557465Z-2ee44939_dirty.01bd8b55a0c45cf9/ladder_L3_conflict_scripted.json
  • docs/eval/traces/ladder/20260718T101557465Z-2ee44939_dirty.01bd8b55a0c45cf9/ladder_L4_blocked_scripted.json
  • docs/eval/traces/ladder/20260718T101557465Z-2ee44939_dirty.01bd8b55a0c45cf9/ladder_L5_large_range_scripted.json
  • docs/eval/traces/ladder/20260718T101557465Z-2ee44939_dirty.01bd8b55a0c45cf9/ladder_L6_long_horizon_scripted.json
  • docs/eval/traces/ladder/20260718T101557465Z-2ee44939_dirty.01bd8b55a0c45cf9/ladder_L7_resume_scripted.json
  • docs/eval/workflow-previews/app-ask-reconcile.gif
  • docs/eval/workflow-previews/app-proposals-review.gif
  • docs/eval/workflow-previews/app-research-enrich.gif
  • docs/eval/workflow-previews/app-variance-fill.gif
  • docs/qa/state-captures/chat/composer-empty--dark--1860.png
  • docs/qa/state-captures/chat/composer-focus--dark--1860.png
  • docs/qa/state-captures/chat/composer-typed--dark--1860.png
  • docs/qa/state-captures/chat/message-hover--dark--1860.png
  • docs/qa/state-captures/landing/default--dark--1860.png
  • docs/qa/state-captures/landing/default--light--1860.png
  • docs/qa/state-captures/landing/join-focus--dark--1860.png
  • docs/qa/state-captures/leftrail/default--dark--1860.png
  • docs/qa/state-captures/leftrail/file-hover--dark--1860.png
  • docs/qa/state-captures/leftrail/static-hover--dark--1860.png
  • docs/qa/state-captures/manifest.json
  • docs/qa/state-captures/research/default--dark--1860.png
  • docs/qa/state-captures/research/import-empty-error--dark--1860.png
  • docs/qa/state-captures/research/import-open--dark--1860.png
  • docs/qa/state-captures/sheet/cell-editing--dark--1860.png
  • docs/qa/state-captures/sheet/empty-cells--dark--1860.png
  • docs/qa/state-captures/sheet/proposal-accept-hover--dark--1860.png
  • docs/qa/state-captures/sheet/proposal-pending--dark--1860.png
  • docs/qa/state-captures/sheet/proposal-reject-hover--dark--1860.png
  • docs/qa/state-captures/topbar/autoallow-off--dark--1860.png
  • docs/qa/state-captures/topbar/autoallow-on--dark--1860.png
  • docs/qa/state-captures/topbar/default--dark--1860.png
  • docs/qa/state-captures/topbar/default--light--1860.png
  • docs/qa/state-captures/topbar/roomcode-copied--dark--1860.png
  • docs/qa/state-captures/topbar/roomcode-hover--dark--1860.png
  • docs/qa/state-captures/topbar/roomcode-rest--dark--1860.png
  • src/nodeagent/core/runtime.ts
  • src/nodeagent/skills/spreadsheet/cellMutator.ts
  • src/nodeagent/skills/spreadsheet/workbookTaskIntelligence.ts
  • tests/deterministicFailureAccrual.test.ts
  • tests/verifyWorkbookBlankSheet.test.ts

Needs Adversarial Review — Do NOT Apply Yet

These proposals passed the reject check but have not been approved by
an adversarial reviewer. A human or frozen LLM judge must approve them first.

  • scaf-001 (AGENTS.md): Add explicit instruction for step spreadsheetbench-runner-fixture: Step spreadsheetbench-runner-fixture failed — scaffold may need explicit instruction or evidence assertion.
  • scaf-002 (AGENTS.md): Add explicit instruction for step convex-boundaries: Step convex-boundaries failed — scaffold may need explicit instruction or evidence assertion.

Safety Boundary

Agent may improve the scaffold.
Agent may NOT weaken the proof gate.

Immutable files (never modify):

  • scripts/proofloop.mjs
  • scripts/agent-improvement-loop.ts
  • tests/harnessChangeEval.test.ts
  • .github/workflows/
  • src/eval/evalTrustPolicy.ts
  • src/eval/architectureBudget.ts
  • evals/evalStore.ts

Scaffold files (safe to modify):

  • AGENTS.md
  • CLAUDE.md
  • proofloop/scenarios/*.yaml
  • proofloop/rubrics/*.yaml
  • proofloop/subagents/*.md
  • proofloop/adapters/*.js
  • .proofloop/memory.jsonl
  • src/nodeagent/models/prompts/systemPrompt.ts

…coverage

Live re-verify of the blank-sheet fix (room NR44HTRXVCD) surfaced deadlock
layer 2: instruction-derived required targets parse as A1 (Sheet!B6) while
agent plans write structured room ids (r6__B), so missing_target_coverage
returned needs_repair on every preflight regardless of plan quality — the
model even tried writing "uses the fully qualified elementId ...!B6" into its
own instruction to appease the checker.

normalizeAddress now maps structured letter-column ids (r6__B, R12__aa) onto
their A1 twin at the key-derivation choke point, so every target set and
coverage comparison lands in one address space. Semantic column ids
(r_rev__variance) have no A1 twin and pass through untouched.

70/70 across verifyWorkbookBlankSheet, workbookTaskIntelligence(+V2), and
deterministicFailureAccrual; typecheck clean.

Changed areas:
- src - workbookTaskIntelligence structured-id A1 canonicalization
- tests - verifyWorkbookBlankSheet target-coverage cases

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@HomenShum
HomenShum force-pushed the fix/live-agent-write-path branch from 5c2f2f6 to d22b5d7 Compare July 18, 2026 10:15
@HomenShum
HomenShum merged commit 29b726e into main Jul 18, 2026
10 checks passed
@HomenShum
HomenShum deleted the fix/live-agent-write-path branch July 18, 2026 10:23
HomenShum added a commit that referenced this pull request Jul 18, 2026
…ebius (#216)

The live-run probe (room NRNXCFYJK5B) showed every fresh room landing on the
GLM-Nebius orchestrator route. This promotes moonshotai/kimi-k3 (1M context) to
the front of the openrouter agent/chat/coding/judge/analysis/deepResearch
preference ladders; z-ai/glm-5.2 is DEMOTED to a fallback, never dropped.

- modelPricing: kimi-k3 added ($3/$15 per 1M, 1M ctx) so cost/receipts are
  non-zero and honest (getProviderForModel already routes moonshotai/* to
  OpenRouter, whose key ships in prod).
- The live default is also env-driven (AGENT_ORCHESTRATOR_MODEL); that convex
  env flip is a separate, cost-bearing prod change staged after the P0
  write-path fix (#215) deploys — NOT included here.

Changed areas:
- src - modelCatalog kimi-k3 pricing + openrouter preference reorder
- tests - kimiK3Default pins priced/provider/lead-of-ladder
- docs - ORCHESTRATOR_WORKER_ROUTING orchestrator model
- README - routing prose + honest cost framing (caps, not stale $/job)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant