fix(redesign): honor title-only requests behind any determiner#565
Merged
Conversation
detectRequestedResponseShape matched "return only the title" but not "return only its title": the determiner group listed articles and no possessives, so the possessive phrasing fell through to the five-section memo and silently overrode an explicit user output constraint on a paid run. The determiner set now lives in one TITLE_DETERMINER constant shared by every title pattern. It was previously duplicated across two regexes, which is how the two could disagree in the first place. Found by re-verifying the 2026-07-16 production audit against main instead of trusting it. Four of its five P1s were already fixed by #550/#561/#564 within hours of the run; only this one survived. The suite stayed green because every title case it asserted used the one phrasing that worked, so the new tests use the audit's verbatim production prompts and cover the negative cases the wider pattern could now over-match. Both test files are added to the CI runtime-smoke allowlist, which is an explicit file list rather than a full-suite run. Without that, the regression guard would never execute in CI and would repeat the same blind spot one level up. Also pins DEFAULT_TIERS to the runtime's modelForTier with a parity test. The two were hand-maintained mirrors with no test binding them; drift would make the paid preflight advertise a model the runtime never runs. The composer header comment still claimed "provider names appear only in the trace", the opposite of the disclosure contract shipped in #550. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
HomenShum
enabled auto-merge (squash)
July 17, 2026 10:34
✅ Dogfood Visual QA Gate: PASSED
ArtifactsDownload the Generated by Dogfood QA Gate |
|
Demo: walkthrough of the surfaces this PR changed is available as a workflow artifact ( |
This was referenced Jul 17, 2026
HomenShum
added a commit
that referenced
this pull request
Jul 17, 2026
…571) applyDeterministicResponsePolicy only rewrote unsupported "best/strongest source|claim" superlatives when the run had zero URL-backed evidence rows. In a mixed run - one real URL plus several cached section labels like Setup or Rising Action - the superlative passed through, so an answer could claim "strongest supported claim with its best source" while the rendered best source was a label with no URL. On a paid runtime path that is false confidence, not honest degradation. The gate is now sentence-level: a superlative stands only if its own sentence carries a [N] citation resolving to a URL-backed evidence row; otherwise that sentence is rewritten and the source-needed limitation is appended. A superlative citing a cached label, or citing nothing, is rewritten even when a URL-backed row exists elsewhere in the run. The best/strongest pattern now lives in one shared constant used by both the sanitizer and the gate so they cannot drift. Closes the residual left open by the 2026-07-16 audit follow-up (#565 fixed four of five P1s; this is the fifth). Closes #566 Co-authored-by: hshum <hshum@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
HomenShum
added a commit
that referenced
this pull request
Jul 17, 2026
* fix(scratchnode): repair red handoff tests and close CI allowlist gap Three ScratchnodeEventsSurface tests failed on clean main with a TypeError because the api mock predated ImportRecapButton's domains.product.scratchnodeImport query, and CI never saw the red because the runtime-smoke job runs an explicit allowlist that omitted the file. The api mock is now a Proxy that degrades unknown function references to unresolved queries (the surface's honest gates render nothing) instead of crashing at property access, the convex/react mock gains the useMutation the component already imports, two scenarios cover the import affordance's published/unresolved gates, and the file joins the runtime-smoke allowlist so future drift turns CI red. Closes #567 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(composer): make run cancellation spatially stable, not timer-only The submit->Stop same-slot swap meant a double-click slower than the 400ms arming window (Windows accessibility settings allow ~900ms) could still cancel the run it just created. Stop and submit are now both always rendered: Stop holds a reserved slot (visibility:hidden while idle) and submit stays at identical coordinates while streaming, merely disabled — so the second click of a double-click lands on an inert control at every interval. The arming delay remains as defense in depth. Closes #568 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(chat): honor sentence, paragraph, and word-limit response shapes detectRequestedResponseShape recognized only title-only and bullets, so "in one sentence", "a single paragraph", and "under 50 words" silently rendered the five-section memo despite the user's explicit constraint. Three new shapes are detected, instructed (via an exhaustive responseShapeSystemInstructions switch replacing the ternary chain), and deterministically enforced in applyDeterministicResponsePolicy: first sentence extraction, prose collapse, and sentence-accumulating word budgets. Honesty survives compaction — unsupported runs carry a "Source needed" prefix or inline limitation instead of busting the user's limit. JSON/table shapes stay model-side deliberately: the memo render path is markdown prose, and a deterministic policy cannot reconstruct valid structured output from parsed memo fields. Closes #569 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(e2e): pin 390px chat geometry by computed grid, seal audit lane The audit's mobile collapse (grid 70px 320px, chat clipped to a 70px strip) was invisible to both the CSS-source string guard and the document-overflow boolean. one-flow-regression (Tier B, per-PR) now asserts the computed grid-template-columns of .rd-shell__main resolves to one track and main#main-content stays >300px at 390x844. Reversion- proved by reinjecting a higher-specificity two-column rule: computed geometry returns exactly the audited 70px/320px shape, which the new assertions reject. Also seals the audit-residual work in the redesign-chat changelog lane and backfills the #565/#571 entries' pending merge markers. Closes #570 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: hshum <hshum@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
detectRequestedResponseShapematchedreturn only the titlebut notreturn only its title. The determiner group listed articles (a/the) and no possessives, so the possessive phrasing fell through to the five-section memo and silently overrode an explicit user output constraint on a paid run.The determiner set now lives in one
TITLE_DETERMINERconstant shared by every title pattern — it was duplicated across two regexes, which is how they could disagree in the first place.Why this is the only P1 left from the 2026-07-16 audit
The audit listed five P1s. Re-verifying each against
mainrather than trusting the doc:.rd-shell→ (0,4,0). Cause was misattributed: the rule was inagent-workspace.css, notprimitives.css, and no "persisted wide-mode flag" exists (nb-wideis prototype-only) — the collapse was unconditional at ≤760px.CANCEL_ARM_DELAY_MS+cancelArmed. The audit captured a ~66-minute window between #548 and #550.exactly two bulletsalready worked;return only its titledid not.Provider names appear in the trace, not here.and now rendersPaid · {provider} · {model}pre-submit.sanitizeUnsupportedSuperlativesalready rewrites the superlative and appends a source-needed limitation. One residual remains, see below.The audit was taken against production hours before #550 landed, so it reads as stale against
main.Why the suite stayed green while production failed
responseShape.test.ts:29asserted"Return only the title."— the one phrasing that works. The test encoded the bug's blind spot. The new tests therefore use the audit's verbatim production prompts, plus negative cases ("explain why its title is misleading","job titles") to pin the false-positive risk the wider pattern introduces.Both test files are added to the CI runtime-smoke allowlist, which is an explicit file list, not a full-suite run. Without that, this guard would never execute in CI — repeating the same blind spot one level up.
Also included
DEFAULT_TIERSpinned to the runtime'smodelForTierwith a parity test. They were hand-maintained mirrors with no test binding them; drift would make the paid preflight advertise a model the runtime never runs (and never prices).UniversalComposerheader comment corrected — it still claimed "no provider names in UI / provider names appear only in the trace", the opposite of the contract shipped in fix(redesign): harden live agent chat contracts #550, so a future reader could "restore" it and re-open that finding.Verification
npx tsc --noEmit --pretty false→ 0 errorsnpx tsc -p convex --noEmit --pretty false→ 0 errorsnpm run build→ cleanexpected { kind: 'memo' } to deeply equal { kind: 'title_only' }(2 failed / 11 passed); restored → 16 passedScratchnodeEventsSurface.test.tsxconfirmed pre-existing on clean main with all edits stashed, and unrelatedNo visual proof: no rendered surface changed — the response body is runtime-produced and the composer edit is a comment.
Known gaps, not fixed here
sanitizeUnsupportedSuperlativesfires only when zero supported URLs exist run-wide. A mixed run (one valid URL + five cached labels) still lets "strongest supported claim with its best source" through unsanitized. Fixing it is a product-design question, not a bug fix.ScratchnodeEventsSurface.test.tsxis red on main (ImportRecapButton:448reads.productof undefined) and CI structurally cannot see it — it is not in the runtime-smoke allowlist.AgentWorkspaceResponsiveCss.guard.test.tsis areadFileSync+toContainstring match and cannot catch a new higher-specificity competitor.detectRequestedResponseShapecovers onlytitle_onlyandbullets."one sentence","a table","JSON","under 50 words"still fall through to the memo — a feature gap, not this bug.🤖 Generated with Claude Code