feat(llm): Prism-style model router + wire the /ask path through it#460
Conversation
…gh it
Introduces one deterministic entry point for "which model serves THIS turn?"
(shared/llm/router.ts), adapting Augment Prism's planner-on-a-pool routing, and
wires the highest-volume single-shot path (event /ask synthesis) through it.
Why: a 5-subagent sweep found ~7 uncoordinated model-selection mechanisms and
the pricing/eval data a router needs, but no unifying planner, and a
"production-first" default (kimi-always) that ignores task complexity. Prism's
finding — the top 10% of turns consume ~57% of LLM rounds, most turns are light
but billed at frontier rates — applies directly to NodeBench's huge
complexity variance.
shared/llm/router.ts:
- routeLLM(taskClass, signals) -> { model, provider, tier, score, escalated,
reason }. PURE + DETERMINISTIC (no Date.now/random) so routes are replay-safe.
- Fixed, quality-targeted pools per task class (ask_answer / classify / extract
/ synthesize / agent_reason / judge). Floors pinned to CURRENT production
behavior, so routing a call site through it is a no-op until signals say
otherwise.
- computeComplexityScore: additive + capped (monotonic) from cheap signals
(input length, source count, multi-entity, analytical intent).
- askAnswerSignals(question, sourceCount): derives /ask routing signals.
- Env overrides (SCRATCHNODE_ASK_MODEL_LIGHT / _HEAVY) let ops pin/retune
without a deploy.
NodeBench adaptation vs Prism: most of our surface is SINGLE-SHOT (no prompt
cache to evict between requests), so Prism's ~10x switch penalty does NOT apply
and we route per-request with a pure heuristic planner at zero added latency.
Cache-aware stickiness — Prism's hard part — is reserved for the multi-turn
agent loops (next layer). V1 ships ESCALATE-UP only (always quality-safe);
eval-gated DEMOTE-DOWN for over-provisioned paths is the documented next step.
convex/events.ts (/ask wiring, additive + behavior-preserving):
- generateProviderAnswer now takes the routed model; the ask_answer pool floors
at the exact prior Haiku id and escalates to Sonnet (claude-sonnet-4-6, the
confirmed Anthropic sdkId) on long / analytical / multi-entity questions.
- legacy SCRATCHNODE_ASK_MODEL still force-pins (disables routing) for ops.
- emits a `model_route` trace step (visible in "Show trace"); modelId +
estimatedCostCents already stored per answer, so escalations + cost are
auditable. The cost estimator already prices Haiku vs Sonnet correctly.
Tests: shared/llm/router.test.ts — scenario-based (short factual -> Haiku floor;
long analytical multi-entity -> Sonnet; many sources -> escalate; forceTarget ->
target; determinism; monotonic score; single-candidate pools never escalate;
env overrides).
Docs: docs/architecture/LLM_ROUTER.md (prior art + roadmap for the remaining
task classes).
Verification: tsc --noEmit clean; vitest router + /ask boundary + scratchnode
events regression = 90 passed / 1 skip; npm run build clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
PR size advisoryThis PR adds 498 lines of substantive change. CONTRIBUTING.md defines a soft limit of ~400 LOC. If the PR is genuinely cohesive (e.g. an architecture map, a generated migration, a deletion of a dead module), no action is needed. Otherwise consider:
This is advisory — it does not block the merge. |
🤖 Augment PR SummarySummary: Introduces a deterministic Prism-style LLM router and wires the high-volume Changes:
Technical Notes: V1 is escalation-only (quality-safe), uses no randomness/time for routing decisions, and keeps per-request routing fast for single-shot paths. 🤖 Was this summary useful? React with 👍 or 👎 |
![augmentcode[bot]](https://avatars.githubusercontent.com/in/1027498?s=60&v=4){kind=small}
| }); | ||
|
|
||
| describe("LLM router — ops env override", () => { | ||
| afterEach(() => { |
There was a problem hiding this comment.
afterEach deletes SCRATCHNODE_ASK_MODEL_LIGHT/HEAVY rather than restoring prior values, which can make this suite mutate the global process env and potentially break other tests when those vars are set in a developer/CI environment.
Severity: medium
🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.
| | Task class | Pool (floor → heavy) | Wired call site | | ||
| |---|---|---| | ||
| | `ask_answer` | Haiku `claude-haiku-4-5-20251001` → Sonnet `claude-sonnet-4-6` | ✅ `convex/events.ts` `generateProviderAnswer` (PR: this one) | | ||
| | `classify` | gemini-3.1-flash-lite (single) | ⬜ `server/routes/search.ts` query classify | |
There was a problem hiding this comment.
The “Status” table lists model names like gemini-3.1-flash-lite / flash-lite → flash, but shared/llm/router.ts uses the exact wire IDs with -preview suffixes (e.g. gemini-3.1-flash-lite-preview, gemini-3-flash-preview); this mismatch could confuse ops/debugging when comparing telemetry to docs.
Severity: low
🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.
|
Demo: walkthrough of the surfaces this PR changed is available as a workflow artifact ( |
2 participants
What
One deterministic
routeLLM(taskClass, signals)entry point for "which model serves THIS turn?" (shared/llm/router.ts), adapting Augment Prism's planner-on-a-pool routing, and the first call site wired through it: the highest-volume single-shot path, event/asksynthesis.This is the foundation from the LLM-routing analysis — it collapses NodeBench's ~7 uncoordinated model-selection mechanisms into one planner, starting with
/ask.Why (Prism findings → NodeBench)
A 5-subagent sweep found NodeBench has all the pieces a router needs (pricing tables,
assessComplexity, fallback chains, eval judges) but no unifying planner, and a "production-first" default (kimi-always) that ignores task complexity. Prism's core finding — the top 10% of turns consume ~57% of LLM rounds; most turns are light but billed at frontier rates — maps directly onto NodeBench's huge complexity variance (a one-line/asklookup vs. a deep-diligence run).NodeBench's edge over Augment: most of our surface is single-shot (no prompt cache to evict between requests), so Prism's ~10× switch penalty doesn't apply — we route per-request with a pure heuristic planner at zero added latency. Cache-aware stickiness (Prism's hard part) is reserved for the multi-turn agent loops (next layer).
shared/llm/router.tsrouteLLM→{ model, provider, tier, score, escalated, reason }. Pure + DETERMINISTIC (noDate.now/random) → replay-safe routes.ask_answer/classify/extract/synthesize/agent_reason/judge). Floors pinned to current production behavior → wiring a call site through it is a no-op until signals say otherwise.computeComplexityScore— additive + capped (monotonic) from cheap signals (length, source count, multi-entity, analytical intent).SCRATCHNODE_ASK_MODEL_LIGHT/_HEAVY) let ops pin/retune without a deploy./askwiring (convex/events.ts, additive + behavior-preserving)generateProviderAnswertakes the routed model;ask_answerfloors at the exact prior Haiku id and escalates to Sonnet (claude-sonnet-4-6— the confirmed Anthropic sdkId, used in the repo's direct-API test scripts) on long / analytical / multi-entity questions.SCRATCHNODE_ASK_MODELstill force-pins (disables routing) for ops.model_routetrace step (visible in "Show trace");modelId+estimatedCostCentsalready stored per answer, so escalations + cost are auditable. The cost estimator already prices Haiku vs Sonnet correctly.Tests
shared/llm/router.test.ts— scenario-based: short factual → Haiku floor; long analytical multi-entity → Sonnet; many sources → escalate;forceTarget→ target; determinism; monotonic score; single-candidate pools never escalate; env overrides. (18 cases.)Verification
npx tsc --noEmit— cleannpx vitest run(router +/askboundary + scratchnode events regression) — 90 passed / 1 skipnpm run build— clean (exit 0)Roadmap (next PRs, per
docs/architecture/LLM_ROUTER.md)classify/extract/synthesizeinserver/routes/search.ts.agent_reason(FastAgent + Convex agents).agentRunJudge/ dogfood scores.🤖 Generated with Claude Code