Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,7 @@ jobs:
scripts/__tests__/releaseWorkflowContracts.test.ts
convex/domains/redesign/chatRuns.responseShape.test.ts
src/features/redesign/components/UniversalComposer.test.tsx
src/features/redesign/surfaces/ScratchnodeEventsSurface.test.tsx

scratchnode-launch-gates:
name: ScratchNode launch gates
Expand Down
50 changes: 46 additions & 4 deletions CHANGELOG/pages/redesign-chat.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,48 @@
Append-only lane for the public redesign chat, reproducible answer receipts, and their
transition into an authenticated live conversation. Newest entries first.

## 2026-07-17 - Resolve the four hardening residuals from the production audit

The 2026-07-16 audit cycle left four filed residuals (#567-#570); all four land here.

1. **Red tests CI could not see (#567)** - three ScratchnodeEventsSurface tests failed on
clean main (`TypeError: reading 'product'`) because the test's api mock predated
ImportRecapButton's `domains.product.scratchnodeImport` query, and the runtime-smoke
allowlist never ran the file. The api mock is now a Proxy that degrades unknown
function refs to unresolved queries, the missing `useMutation` mock is added, two
scenarios cover the import affordance's published/unresolved gates, and the file joins
the CI allowlist.
2. **Timer-only cancel guard (#568)** - Stop and submit are both always rendered: Stop in
a reserved `visibility:hidden` slot, submit staying at identical coordinates while
streaming (disabled). A double-click's second click lands on an inert control at ANY
user double-click interval; the 400ms arming stays as defense in depth.
3. **Shape detector gap (#569)** - "in one sentence", "a single paragraph", and
"under N words" are now detected, instructed via an exhaustive
`responseShapeSystemInstructions` switch, and deterministically enforced (first-sentence
extraction, prose collapse, sentence-accumulating word budget). Honesty survives
compaction: unsupported runs carry `Source needed:` within the requested shape.
JSON/table stay model-side deliberately - the render path is markdown prose.
4. **No computed-geometry regression capture (#570)** - one-flow-regression (Tier B,
per-PR) gains a 390x844 test asserting `.rd-shell__main` resolves to ONE grid track and
`main#main-content` stays >300px wide - the audit's 70px collapse was invisible to both
the CSS-source string guard and the document-overflow boolean.

**PR / canonical main commit**: `PENDING #NNN MAIN SHA / FINAL QA`.

**Evidence state**:
- Source: `pending`
- Checks: `npx tsc --noEmit` -> 0 errors. Full runtime-smoke allowlist locally ->
312 passed / 25 skipped (integration skips without env keys, matching CI). Adjacent
guards (ChatResponseShape, ChatRunLifecycle, ChatContinuation, sourceVerificationPolicy)
-> 12 passed.
- Visual proof: `not recorded` - behavior changes are runtime/composer mechanics.
- Preview: Tier B one-flow-regression includes the new 390px geometry test on this PR.
- Production live: `not recorded` at entry time.

**Author**: Homen Shum + Claude Fable 5.
**Touches**: [components/fast-agent-panel.md](../components/fast-agent-panel.md) is NOT
touched - the composer change is redesign-chat's UniversalComposer, not FastAgentPanel.

## 2026-07-17 - Ground "best source" superlatives on their own citation

`applyDeterministicResponsePolicy` only rewrote unsupported "best/strongest source|claim"
Expand All @@ -22,10 +64,10 @@ constant used by both the sanitizer and the gate so they cannot disagree.
This closes the residual left open by the 2026-07-16 audit follow-up (#565 fixed four of
the five P1s; this is the fifth). Tracked as #566.

**PR / canonical main commit**: `PENDING #NNN MAIN SHA / FINAL QA`.
**PR / canonical main commit**: `#571` / `6198dc39`.

**Evidence state**:
- Source: `pending`
- Source: `merged`
- Checks: `npx tsc -p convex --noEmit --pretty false` -> 0 errors. `npx vitest run
convex/domains/redesign/chatRuns.responseShape.test.ts` -> 16 passed (3 new: grounded
citation kept, cached-label citation rewritten, uncited superlative rewritten). Gate
Expand Down Expand Up @@ -63,10 +105,10 @@ names in UI / provider names appear only in the trace" - the opposite of the dis
contract shipped in #550 - and pinned `DEFAULT_TIERS` to the runtime's `modelForTier` with
a parity test, since the two were hand-maintained mirrors with no test binding them.

**PR / canonical main commit**: `PENDING #NNN MAIN SHA / FINAL QA`.
**PR / canonical main commit**: `#565` / `b2d12ae0`.

**Evidence state**:
- Source: `pending`
- Source: `merged`
- Checks: `npx tsc --noEmit --pretty false` -> 0 errors. `npx vitest run
convex/domains/redesign/chatRuns.responseShape.test.ts
src/features/redesign/components/UniversalComposer.test.tsx` -> 16 passed. Fix reverted
Expand Down
85 changes: 85 additions & 0 deletions convex/domains/redesign/chatRuns.responseShape.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,91 @@ describe("redesign chat runtime response policy", () => {
}
});

// Issue #569 — every explicit shape beyond title/bullets used to fall
// through to the five-section memo. These scenarios model an analyst
// pasting exact-output constraints into the composer.
it("detects single-sentence, single-paragraph, and word-limit requests", () => {
expect(detectRequestedResponseShape("Summarize the brief in one sentence.")).toEqual({ kind: "sentence" });
expect(detectRequestedResponseShape("Give me just one sentence on the risk.")).toEqual({ kind: "sentence" });
expect(detectRequestedResponseShape("Write a one-sentence summary.")).toEqual({ kind: "sentence" });
expect(detectRequestedResponseShape("Explain the funding round in a single paragraph.")).toEqual({ kind: "paragraph" });
expect(detectRequestedResponseShape("One-paragraph answer please.")).toEqual({ kind: "paragraph" });
expect(detectRequestedResponseShape("Summarize in under 50 words.")).toEqual({ kind: "word_limit", limit: 50 });
expect(detectRequestedResponseShape("Answer in 30 words or fewer.")).toEqual({ kind: "word_limit", limit: 30 });
expect(detectRequestedResponseShape("No more than 80 words.")).toEqual({ kind: "word_limit", limit: 80 });
});

it("keeps memo for incidental or degenerate shape phrasing", () => {
// The memo template itself says "one sentence" per heading — user prose
// that merely mentions sentences/paragraphs must not flip the shape.
expect(detectRequestedResponseShape("The first sentence of their pitch is misleading.")).toEqual({ kind: "memo" });
expect(detectRequestedResponseShape("Compare the opening paragraph across both reports.")).toEqual({ kind: "memo" });
// Degenerate or absurd limits never bind.
expect(detectRequestedResponseShape("Summarize in under 2 words.")).toEqual({ kind: "memo" });
expect(detectRequestedResponseShape("Summarize in under 90000 words.")).toEqual({ kind: "memo" });
// Bullets outrank a word limit when both appear.
expect(
detectRequestedResponseShape("Give me exactly 3 bullets, under 50 words."),
).toEqual({ kind: "bullets", count: 3 });
});

it("returns exactly one sentence while omitting memo-only fields", () => {
// idx binds the [1] marker to the URL-backed row so the #571 superlative
// gate recognizes the claim as grounded and leaves the sentence intact.
const shaped = applyDeterministicResponsePolicy(
"Summarize the brief in one sentence.",
memo,
[{ idx: 1, source: "https://example.com/acme", quote: "Acme launched a new product." }],
);
expect(shaped.shortAnswer).toBe("Acme is the strongest supported claim in the current packet. [1]");
expect(shaped.whyItMatters).toBe("");
expect(shaped.risks).toEqual([]);
expect(shaped.nextAction).toBe("");
});

it("collapses the memo into one paragraph and keeps the source-needed limitation inline when unsupported", () => {
const supported = applyDeterministicResponsePolicy(
"Explain it in a single paragraph.",
memo,
[{ source: "https://example.com/acme", quote: "Acme launched a new product." }],
);
expect(supported.shortAnswer).not.toContain("\n");
expect(supported.whyItMatters).toBe("");

const unsupported = applyDeterministicResponsePolicy(
"Explain it in a single paragraph.",
memo,
[{ source: "Setup" }],
);
// Honesty survives the compact shape: the limitation rides inside the
// paragraph because compact renders hide the risks section entirely.
expect(unsupported.shortAnswer).toContain("Source needed");
expect(unsupported.risks).toEqual([]);
});

it("enforces an explicit word limit deterministically, including the honesty prefix", () => {
const longMemo = {
...memo,
shortAnswer:
"Acme closed the round. The filing lists twelve investors across three continents and two follow-on commitments.",
};
const shaped = applyDeterministicResponsePolicy(
"Summarize in under 8 words.",
longMemo,
[{ source: "https://example.com/acme", quote: "Acme closed the round." }],
);
expect(shaped.shortAnswer.split(/\s+/).length).toBeLessThanOrEqual(8);
expect(shaped.whyItMatters).toBe("");

const unsupported = applyDeterministicResponsePolicy(
"Summarize in under 8 words.",
longMemo,
[{ source: "Setup" }],
);
expect(unsupported.shortAnswer.startsWith("Source needed:")).toBe(true);
expect(unsupported.shortAnswer.split(/\s+/).length).toBeLessThanOrEqual(8);
});

it("does not mistake incidental prose about titles for a shape request", () => {
expect(
detectRequestedResponseShape("Summarize the report and explain why its title is misleading."),
Expand Down
Loading
Loading