feat(demo): grammar-triggered KV branch swap on setType - #9
Merged
Conversation
…retry When the retry loop fires (orphan detection or runtime error), the new conversation's tokenIds share a long prefix with the failed attempt's prefilled tokens — the chat template emits the same bytes up through the first '<|turn>model\n' marker, only diverging where the assistant-code turn starts. Before: restoreCache → full re-prefill of [user, assistant, error, model_prompt] from position 3005 (system cache end). After: rollbackKV to systemCacheEnd + LCP and prefill only the diff. Measured on an ER-diagram prompt that retried twice: retry 2 reused 626/1253 tokens (50%), saving ~500ms of re-prefill. Retry 1 fell below the 32-token floor because the user prompt was only ~26 tokens — those savings (<50ms) aren't worth rollback bookkeeping. The eager-prefill path captures its token sequence into lastEagerPrefilledTokens so a retry after an eagerly-prefilled attempt can still LCP without re-tokenising. currentAttemptPrefilled scopes the retry source to one generate() call; cachedUserTokens continues to serve cross-call reuse unchanged. Unique to the retry-loop architecture — no other browser LLM stack has an error-feedback retry step to apply this to.
After phase A ends, dump the accumulated thinking buffer (with <channel|>
marker stripped) so the curious user can inspect what the model reasoned
about before the code phase ran. Kept behind a silent log — no UI impact.
At each retry, log the full user-feedback message that the next attempt's
tokenIds will include. Previously only a one-line summary hit the console
('attempt 1 failed: X is not defined'); now the exact prompt the model is
being asked to fix is visible too.
Retry-LCP test with the ER-diagram prompt showed retry 1 fell below the 32-token floor: user prompt tokenized to ~26, so the ~25-token reusable prefix was discarded and the retry took the full restoreCache path. Rollback cost is dominated by one worker postMessage round-trip (~1ms) plus length-counter truncation (no GPU work). Per-token prefill saves ~0.8ms, so breakeven is ~8 tokens. Below 8 the math goes negative; at 8+ it's pure win. Lowering the floor catches short-prompt retries without regressing anything above the old floor.
S_IN_THINK's grammar mask has every bit set (grammar.ts line 82-87: every token returns allowed=true in that state), so the mask kernel is a full 262k-lane sweep that performs zero semantic work. Disable it while grammar.state == S_IN_THINK and re-enable the moment we transition out (the <channel|> token flips state to S_FREE, next iteration sets the real code-syntax mask). Measured on TCP state-machine prompt (200 thinking + 370 code tokens, 20s total wall): within run-to-run noise, so savings are <0.3ms per token — smaller than estimated but correct and free. Also trims the per-generation dispatch count by ~200-800 entries, which makes chrome://tracing more readable.
…l-and-restart
Three UX / robustness fixes bundled:
1. stripMarkdown(): the thinking channel emits markdown (bold, headers,
bullets, fenced code), but Excalidraw text elements render plain text —
raw `**word**` and `## Header` showed up verbatim. Strip the syntax
before showThinkingCloud so the cloud stays readable. Simple regex
stripper covers the cases the model actually emits.
2. Web Locks singleton: a second tab running the same engine would
allocate its own WebGPU device, duplicate the 3 GB model pointer, and
race for OPFS — effectively guaranteed to OOM. Acquire
`navigator.locks.request("turboquant-draw-engine", { ifAvailable: true })`
at init; if unavailable, block the tab with a "close the other tab"
card instead of spinning up a doomed worker. Lock is held for the
page lifetime and releases on unload.
3. generate() cancel-and-restart: previously a second click only aborted
the in-flight run — you had to click THREE times to actually retry
(abort, then nothing, then new). Now: abort + await
currentGeneration (the in-flight run's finally-resolved promise) +
fall through to start a fresh run with the current prompt. Single
click, single restart.
Gemma 4 E2B layers 15..34 reuse KV cache from layers 13/14 (kvReuseLayer() returns the source layer). Step 5 (tq_encode) already guards on `reuseLayer < 0`, but steps 2-4 (Q/K/V matmul, QK/V norm, Q/K RoPE) were computing K and V outputs that got immediately discarded. For 20 of 35 layers × 2 matmuls per layer, that's 40 discarded matmul dispatches per decode step — each ~1/4 the work of the Q projection (kvSize=512 vs qSize=2048). Gate the K and V projections, norms, and K RoPE on `reuseLayer < 0`. Q stays unconditional because every layer's attention needs fresh Q. Measured 29.1-29.7 tok/s across 3 prompts (cold M1), up from ~28.9 baseline — consistent +0.5 tok/s. Output token counts unchanged, so model argmax path is identical; quality preserved.
Companion to f2554c2 (decode-path skip). runLayerBatched had the same gap: computing K+V matmul, K-norm, V-norm, K RoPE for layers 15..34 whose outputs are immediately discarded (tq_encode at step 5 already guards on reuseLayer). Gate those steps symmetrically. Affects prefill speed only. Decode tok/s unchanged (separate code path). All 3 bench prompts still compile to the same diagrams.
Apply the same subgroupShuffleXor pattern the matmul kernels use to argmax's two-phase reduction. Max + argmax pair is order-invariant (max(a,b) == max(b,a) regardless of visit order), so unlike rms-norm's sum-of-squares this rewrite carries no FP-reorder quality risk. argmax runs twice per decode step (pass1 over 262144 logits, pass2 over the 256 per-workgroup partials). Savings per call are below measurement noise — keeping the change for consistency with the other subgroup-reduce kernels and to simplify future kernels that copy this pattern. No regression in 3-prompt bench (30.7-31.2 tok/s, same as before within noise).
Same pattern as argmax: 5-stride butterfly inside each 32-wide subgroup followed by 3-stride cross-subgroup butterfly over 8 stashed maxes. Max is order-invariant, so unlike phase 3's sum (which keeps its tree) there's no FP-reorder risk. Each softmax call drops 7 of the 8 barriers from the old tree reduce for phase 2. 35 layers × 7 barriers ≈ 245 fewer sync points per decode. Below measurement noise individually — kept for consistency with argmax and the other subgroup-reduce kernels.
Apply the argmax/softmax max-reduction pattern to tq_encode's max_r reduction. Order-invariant (max), so it's safe unlike phases 4/4b which keep shared-memory sum trees because reordering their FP sums would change the compressed K/V values and propagate quality drift through all 35 layers' encodes. Scope: 15 encode calls per decode step (K + V for 15 own-KV layers post shared-KV skip), 2 barriers saved × 2 (K,V) × 15 = 60 fewer sync points. Below measurement noise individually — kept for consistency. Output token counts unchanged across 3-prompt bench.
Session 2's Hi-Z pattern applied at the loop level inside mh_softmax. Phases 2 (max), 3 (exp+sum), 4 (normalize) now iterate only the in-window slice [win_start, scan_end) instead of scanning 0..n. For sliding-window layers (28/35, window=512) with n > 3000 positions, the old scan covered ~6× more slots than necessary. Phase 1 used to write -1e30 to out-of-window slots so downstream reads were harmless. With tighter phase 2-4 bounds (and phase 5's compact-list path that wsum_p1 follows), those slots are never read. Drop phase 1 entirely. Correctness preserved: wsum_p1 iterates the active_list built by phase 5, which starts at win_start. No path reads data[row_start + i] for i < win_start or i >= scan_end. Measured 31.0-31.7 tok/s across 3 prompts, same range as prior commit's 31.1-31.9 — marginal but all 3 still compile first-try with unchanged token counts.
WeInfer (ICLR '26) identified uniform-buffer setup as a hidden WebGPU cost — many dispatches within a single decode step post identical uniform blocks. rmsNorm's [dim, eps, 0, 0] is identical across all 35 layers on a given width; matmul's [n_rows, n_cols] repeats for Q/K/V/ attn_out within a layer and across shape-matched layers. UniformMega now keys each slot by exact byte content (string key, not a hash — a collision would silently bind a dispatch to some other kernel's data). Repeat requests return the existing offset instead of allocating a fresh 256-byte-aligned slot + appending another writeBuffer byte range. Correctness: string keys guarantee no aliasing. All 3 bench prompts still compile first-try with identical output token counts to the pre-dedup run (263, 418, 385). Speed 30.6-31.5 tok/s, same as baseline within noise — the writeBuffer upload size drops but that's not currently the bottleneck. Kept for the memory-traffic reduction on low-end GPUs where WeInfer measured 1.2-2× gains.
Before queuing a full model regen on orphan-node detection, try the
mechanical codemod: strip the `const VAR = addXxx("LABEL", ...)` lines
whose LABEL is in the orphan list, AND drop any connect/message/etc.
lines that reference those dead VARs (those would throw at runtime).
If the trimmed code compiles + the remaining diagram meets the orphan
threshold, accept it and skip the retry.
Saves ~15 s per would-be retry when the model over-declares (a common
small-model tax — the ablation notes logged orphan-driven retries on
OAuth, Swimlane, K8s in no-thinking mode). Unique to our stack: no
other browser LLM has (a) a compile gate in-page, (b) a semantic
orphan detector, and (c) the ability to mechanically patch the
model's output. Equivalent to what an IDE's "quick fix: remove unused
declaration" does — applied automatically at generation time.
Quality: if the strip produces a degenerate diagram (too small, still
has orphans), the code falls through to the existing retry path. The
fix only commits when the post-strip diagram is objectively better.
Tested on 3-prompt bench — none triggered orphans under thinking
mode, so auto-fix path stayed dormant. Speed unchanged (30.7-31.4
tok/s). The path will activate on orphan-prone prompts.
iPhone user hit a generic "enable chrome unsafe webgpu" message that didn't apply to iOS Safari. Detect iOS / Android via user agent (including the iPad-pretending-to-be-Mac case via maxTouchPoints) and give OS-specific guidance: - iOS without WebGPU → Settings → Safari → Advanced → Feature Flags, plus the reality that the 3 GB model exceeds typical iPhone tab memory caps. - iOS with WebGPU but missing subgroups/shader-f16 → explain Apple hasn't shipped subgroups in Safari yet; desktop browser required. - Android Chrome missing WebGPU → the chrome://flags instructions that ARE relevant on that platform. - Android Chrome with WebGPU but missing features → flag names + memory caveat. - Mobile with full WebGPU feature set → non-blocking console.warn about the 3 GB model size; user can still try. Doesn't enable mobile support at the rendering level (still needs subgroup-free fallback shaders for mobile GPUs that lack the extension), but the error the user sees is now informative enough to act on.
Page title, OG tags, Twitter card, visible subtitle, and the
WebGPU support check all now say desktop Chrome 134+ first, rather
than burying the requirement under a load failure. Safari/iOS/Android
users now see the real answer ("use desktop Chrome") instead of
flag-toggle instructions that don't actually resolve the underlying
blockers (subgroups extension not shipped in Safari; mobile per-tab
memory caps below 3 GB).
HN/LinkedIn-ready title reads "Gemma 4 E2B in desktop Chrome
(WebGPU)" — sets expectations before the click.
- types.ts: Zod v4 requires key type — z.record(z.unknown()) → z.record(z.string(), z.unknown()) at 3 sites - engine.ts / model-loader.ts: WebGPU writeBuffer no longer accepts Uint8Array<ArrayBufferLike> due to SharedArrayBuffer union; cast typed arrays to <ArrayBuffer> generic at 4 upload sites - vite-env.d.ts: declare Vite client types plus *.wgsl?raw and *.ts?worker module shapes so import.meta.env + raw/worker imports type-check - package.json: add @types/react and @types/react-dom (v19) bun x tsc --noEmit now reports 0 errors.
The subgroupShuffleXor butterfly was added in 46c5758 but the directive was missing — Chrome rejects the shader at compile time with: cannot call built-in function 'subgroupShuffleXor' without extension 'subgroups' All other shaders using subgroup ops already enable the extension.
Prompt sources changed; cache regen ran via the vite plugin and produced the same KV bytes, so only the hash file needs updating.
Grammar-triggered mid-generation KV swap using pre-baked branches. Router prompt + per-mode specialised prompts baked offline; on setType() grammar transition, mount the matching branch onto the live cache at zero runtime prefill cost. Retry rolls back to router. Covers problem statement, prior art gap, file format, mount mechanics (v1 flat, v2 nested with RoPE shift), grammar extension, build pipeline, edge cases, and open questions.
New module demo/src/draw/prompts/: preamble.ts shared boilerplate router.ts small mode-picker prompt (~200 tok) sequence.ts full sequence docs + SDK_TYPES (~800 tok) architecture.ts full arch/flowchart/class/ER/swimlane docs (~2200 tok) index.ts BRANCHES map for the build pipeline Additive only. main.ts's SYSTEM_PROMPT is untouched — these new sources are used by the upcoming build pipeline (#110) and mount primitive (#107) but don't change runtime behaviour yet. Reality check noted in docs: the runtime DiagramType is only 'architecture' | 'sequence', so v1 is router + 2 modes, not 3. 'class', 'swimlane', 'er' are shape sub-types inside 'architecture'.
Wraps N TQKV KV-cache blobs into one file keyed by branch name. Format: 4B magic 'TQKC' 2B version 2B branch_count per branch: 1B name_len + name + 4B offset + 4B len + 4B token_count then each branch's TQKV blob, concatenated packContainer() / unpackContainer() are pure — no engine, no GPU. parseSystemCache() auto-detects TQKV (single blob, back-compat) vs TQKC and returns a unified branches map either way. 7 unit tests cover round-trip, back-compat, empty/oversize names, bad magic. All pass.
Adds orthogonal SDK state (UNSET / SEQUENCE / ARCHITECTURE) next to the
existing char-level grammar states. A ModeTracker side-channel scans
decoded token text for completed setType('sequence'|'architecture')
calls and flips state on first match.
Chose side-channel over nesting into the char-level transition table:
nesting would 3x the mask + transitions footprint for a single
fire-once feature. Pattern match is O(emitted text len) and runs
until the first hit, then becomes a no-op.
9 unit tests: fragmented-stream accumulation, whitespace tolerance,
thinking-text false-positive rejection, reset, unknown-arg rejection.
Adds the hook API that grammar-triggered mount actions will use:
tracker.onEnter(SDK_MODE_SEQUENCE, () => engine.mountKV('sequence'))
Handlers fire synchronously on transition (before observe returns), in
registration order. Handlers persist across reset() so a fresh
generation re-arms them automatically.
onEnter(UNSET) throws — UNSET is never 'entered', it's the initial /
post-reset state.
5 new tests cover: fires on transition, multiple handlers in order,
no-fire on ignored args, UNSET rejection, persistence across reset.
Adds the multi-branch mount primitive on InferenceEngine. A branch is registered once (CPU-resident blob + tokenIds) and then mountKV(name) swaps it in as the active system cache via existing loadCache(). v1 constraint documented in-code: flat mount only — a branch IS the system cache, not an append onto it. Mounting at position 0 means the branch's RoPE-encoded K vectors match their phase, no re-RoPE needed. After mount, caller re-prefills the user's turn on top of the new system cache. Integration with worker IPC + wiring to ModeTracker comes in #108.
Build-cache spec now iterates over [router, sequence, architecture], prefilling each branch in its own fresh page, dumping per-branch KV, and packing all three into a TQKC container at public/system-cache.bin. Runtime changes (minimal v1a support): - main.ts reads ?branch=<name> query param to force a specific branch's prompt for end-to-end testing. Falls back to legacy SYSTEM_PROMPT when no branch is specified. - When fetching system-cache.bin, auto-detects TQKV vs TQKC magic: - TQKV: load as-is (legacy path) - TQKC + ?branch set: extract matching branch, use its blob - TQKC + no ?branch: skip cache, worker prefills from scratch (the v1a regression: default path pays one prefill cost until the mid-stream mount wiring (#108) lands and swaps at runtime) vite.config.ts prompt-hash now includes all prompts/ sources so a change to any branch invalidates the cache. Preamble gets a 'think briefly, no markdown' instruction — applies across all branches to cut thinking verbosity.
End-to-end wiring for the dynamic context window:
**Prompts**
- router.ts: reshaped into a real menu — brief-intro + 5-line example
per mode, explicit 'setType(...) first, context specialises after'
framing. ~500 tokens.
- architecture.ts: mirrors today's monolithic prompt content verbatim
(minus sequence-specific sections), restoring the quality baseline
that the shorter split was regressing on 15-node infra diagrams.
- preamble.ts: 'think briefly' is now a NAMED tail that router +
sequence opt into; architecture skips it (needs thorough planning).
**Runtime**
- main.ts: removes the ?branch= URL hack. At boot, tokenizes all three
branches and passes sequence + architecture as extraBranches to the
worker (router becomes the active KV via loadCache / prefill). The
cache fetch path now parses TQKC only — single-blob TQKV is no
longer a supported runtime path on this branch.
- engine-worker.ts: init accepts extraBranches[] and calls
engine.registerBranch for each. Adds 'mountBranchAndPrefill' message
that swaps active KV to the named branch and prefills user-turn
tokens on top, returning the first predicted token.
- engine.ts: snapshot/restore pair now also round-trips activeBranch
bookkeeping (_snapshotBranch), so a retry after a mid-stream mount
cleanly returns to the router.
**Generate loop**
- Phase B token callback runs ModeTracker.observe on every decoded
code token. On setType('sequence'|'architecture') fire, abort the
stream, discard all pre-mount output (was generated under router —
untrusted), mountBranchAndPrefill the target branch, and run a
SECOND thinking+code pass under the specialised KV. ModeTracker's
fire-once semantics cap this at one mount per attempt.
- #113 retry protocol: each attempt's implicit restoreCache puts the
KV back to router (captured at snapshot time). activeBranch follows.
Retry re-enters the attempt loop and can re-detect setType fresh.
DiagramType (types.ts) expands from 2 values to the full union the SDK
and prompts actually use:
architecture | sequence | flowchart | state | orgchart | er | class | swimlane
The sub-types within the non-sequence category are presentational —
same Graphviz layout, different shape preferences and rules. Listed
exhaustively so the type system matches every setType value the prompts
can produce. sdk-types.ts (shown to the LLM) mirrors the union.
ModeTracker routes all 8 values: 'sequence' → SDK_MODE_SEQUENCE,
everything else → SDK_MODE_ARCHITECTURE. Both branches' KV is already
pre-baked, so the seven non-sequence sub-types share the architecture
branch (adding per-sub-type branches is queued as v2 in #103).
Router prompt updated to list all 8 setType values explicitly in its
menu. Architecture prompt now uses sub-type-appropriate setType in each
example (setType('swimlane'), setType('class'), etc.) rather than
normalising everything to setType('architecture') — the runtime passes
the literal string through to the viewer, so being faithful to the
intent the model is expressing is the right behaviour.
**Build pipeline fix.** Restored a test-only ?buildBranch=<name> query
param that tells main.ts which branch's prompt to prefill during the
build-cache run. Without it, every branch in the TQKC container was
just the router prompt (bug visible in the just-rebuilt cache: all 3
entries reported 794 tokens). Clearly named so it can't be mistaken
for a user-facing feature.
Tests: expanded mode-tracker.spec.ts to cover routing of all 8 values
(22 tests pass).
Each of the 8 DiagramType values now has its own dedicated branch with type-specific docs, rules, and examples — no content overlap between sub-types. Dynamic context window mounts the exact branch the model requested, not a catch-all. New prompt files under demo/src/draw/prompts/: flowchart.ts — addBox + addEllipse + addDiamond, yes/no decision labels state.ts — addEllipse states + transition labels, LR default orgchart.ts — addBox hierarchy, parent→child edges er.ts — addTable + columns with PK/FK + cardinality edges class.ts — addClass with attributes/methods + relation opt swimlane.ts — addBox(col) + addLane containers + cross-lane arrows architecture.ts trimmed to ONLY infrastructure content — no swimlane example, no UML class example, no flowchart rules. The broad-scope monolith was diluting per-type focus. ModeTracker gets one SDK_MODE_* constant per DiagramType value (8 total vs the prior 3). Each maps 1:1 to a branch name in prompts/index.ts, so onEnter handlers set pendingMount = name directly. Router fires nothing (UNSET → any named mode). main.ts: - tokenizes all 8 branch prompts up-front - registers onEnter for all 7 non-router modes in generate() - router is still the boot branch; mid-stream swap picks from 7 Build pipeline BRANCH_ORDER updated to all 9 branches (router + 8, but router IS one of the 8 in BRANCHES so the test iterates 9 entries). Wait, rereading: BRANCHES has 9 keys (router + 8 types); BRANCH_ORDER in build-cache.spec.ts also lists 9 — router first for boot, then the 7 non-sequence types, then sequence last. The test builds each in a fresh page and packs all into one TQKC container. 22 unit tests pass (expanded routing test now validates 8 distinct mode → branch mappings; type system enforces BranchName = keyof typeof BRANCHES so main's handler registration exhausts all cases).
After mounting the specialised branch, the do-over runs phase A+B
fresh under the new KV. The model's first code line under the branch
is typically setType(...) again (the branch's own examples show that
pattern).
Previously I called modeTracker.reset() right before the second pass,
which cleared the fire-once lock. When the second pass emitted
setType("sequence") as its first code token, the tracker re-fired,
set pendingMount, aborted the stream, and truncated the code at 17ch
of `setType("sequence` before the string even closed.
First attempt failed with 'Invalid or unexpected token'; retry
succeeded only because retry feedback primed the model to continue
mid-stream (with 'const user = addActor(...)') instead of re-emitting
setType. That masked the bug.
Fix: keep the tracker's fired state across the do-over. One fire per
attempt; reset happens only when the outer retry loop creates a fresh
tracker at the next attempt's start.
The router's only job is to emit setType(...). Full planning happens under the specialised branch during the post-mount do-over. Running a full thinking phase under the router (shallow context) is ~5s of wasted decode that the do-over's second thinking pass redoes anyway. Previously observed: router thinking 285ch, then mount, then branch thinking 447ch for the same prompt. Fix: after the standard user-turn prefill (which ends in `<|channel>thought\n` via the chat template), immediately prefill `<channel|>` + the code-mode reminder. This closes the thinking channel inside the KV before the model generates anything, so the router run goes straight to code-phase (where setType fires, mount triggers, do-over does the real thinking under the branch). New runCodePhaseOnly helper handles the router run (code only with mount detection). runThinkingAndCode stays for the do-over under the specialised branch (full planning with branch-specific docs). Tradeoff: one forward pass of wasted compute on the nextToken from the original user prefill (it was a thinking token we no longer use). Cheap compared to a full phase A.
… re-emit
After a mid-stream mount, the pre-mount setType(...) is wiped by
setCode("") in the do-over, and the mounted branch's prompt tells the
model "setType has been called" — so it typically does NOT re-emit
setType as its first code line. Without setType in the executed code,
the SDK's diagramType stays at the default ('architecture'), which
renders sequence / class / er / etc. as generic architecture — wrong
layout.
Fix: when mountTarget was set (mount fired during this attempt) AND
the generated code doesn't already start with setType(, prepend
setType("<target>");\n before executeCode.
Defensive both ways: if the model DOES re-emit setType, we skip the
prepend. If it doesn't, we add it. SDK's diagramType matches the
branch we mounted.
Previously, on mount we called setCode('') which wiped the router's
partial 'setType("seq' from the editor. The setType line then didn't
come back until the very end when setCode(finalCode) ran — so users
saw setType appear, disappear, and only reappear after generation
completed.
Fix: on mount, REPLACE generatedCode with the canonical
'setType("<target>");\n' and setCode it immediately. The partial
setType becomes the completed setType line in-place. The branch's
subsequent output streams in below.
Dedupe step at end-of-attempt: if the branch ALSO emits setType(...)
as its first code line (possible depending on how the branch prompt
reads), we strip the duplicate so executeCode sees exactly one
setType at the top.
Replaces the partial-setType-then-overwrite approach with: the router
phase observes tokens purely (for mount detection) and never touches
the code editor. On mount, the editor is seeded with the canonical
setType line and branch tokens append below it from there.
Symptoms this fixes: the setType string flickered / disappeared from
the editor mid-stream because the router's partial output
('setType("seq') got written, then overwritten with the canonical
form, then sometimes cleared when the branch re-emitted. Users saw
the header blink out during streaming.
Cleaner flow:
1. On generate start: setCode('')
2. Router run: observeForMount per token — ticks counter, feeds
decoded text to ModeTracker, nothing else. Editor stays empty.
3. Mount fires: setCode('setType("<target>");\n'). This is now
the only write to the editor for the setType line.
4. Branch run: renderCodeToken appends decoded chunks via
appendCode() below the seeded setType. Statement-boundary
partial executor works off generatedCode as before.
5. End-of-attempt dedupe logic kept for the case where the branch
re-emits setType (defensive — branches' prompts say 'setType
has been called' but the model sometimes re-emits anyway).
Removed from the mount do-over: tokenCount=0, lastStmtCount=0,
thinkingText='', clearThinkingCloud(). These were resetting counters
that should carry across the attempt (token counter shouldn't jump
backwards; thinkingText wasn't populated during router since the
router phase is thinking-skipped).
…e in editor
Two bugs from the 2026-04-18 swimlane test session:
1. Lane header rectangles reported as orphan nodes. The lane header-bg
(sdk.ts:2322) is a 'structural' rectangle with no edges pointing
at it, same as the lane background at 2280. The background already
carried customData._group=true so detectOrphanNodes skipped it;
the header was missing the flag so it tripped orphan detection
on every swimlane diagram. Added the flag — orphan count should
go from 'lanes+1' down to 'real unconnected boxes only'.
2. Editor showed two setType lines during streaming. On mount we seed
the editor with 'setType("<target>");\n'; the branch often
re-emits setType as its first code line because its examples
start with it, so the editor ended up with two. Dedupe at
end-of-attempt scrubbed before execute, but the visible streaming
flashed both lines.
Fix: branch-phase leading setType suppressor. On mount arm
suppressLeadingSetType=true. renderCodeToken buffers decoded
chunks until the first newline, inspects the first line:
- matches /^\s*setType\s*\(/ → drop the line, flush rest
- otherwise → flush the whole buffer
Flag auto-disarms after the decision. Defensive 256-char cap so
a pathologically long no-newline first line can't swallow the
whole response.
…cache.bin Pre-TQKC, the worker's post-prefill auto-dump of the system-prompt KV was a convenience: main POSTed it to /api/write-system-cache so the dev server overwrote public/system-cache.bin, making the next reload fast. With the multi-branch TQKC format that convenience is destructive — the worker only dumps the ACTIVE branch (router by default), so a single-branch TQKV write blows away the other seven branches users built with bun run rebuild-cache. Observed: after an interactive test session, public/system-cache.bin went from 33 MB TQKC / 9 branches down to 8.87 MB TQKV / 0 branches. Every page load then hit the no-cache fallback and prefilled router from scratch. Fix: main ignores cacheRebuilt — the worker still dumps internally (the build-cache playwright test reads it back via engine.dumpSystemCache + packContainer, which is the authoritative path), but main never POSTs. Interactive dev now requires running bun run rebuild-cache after prompt edits — the convenience is gone but the TQKC on disk stays intact across normal use.
File: 88.32 MB, 9 branches router 766 tok 5.92 MB sequence 1202 tok 9.29 MB architecture 1662 tok 12.84 MB flowchart 1219 tok 9.42 MB state 1148 tok 8.87 MB orgchart 1189 tok 9.19 MB er 1438 tok 11.11 MB class 1405 tok 10.86 MB swimlane 1399 tok 10.81 MB
…Needed Follow-up to #6756491 (PR #8). The original validatePayloadSizes duplicated the polar/qjl encoded-size formulas inline: const polar_bits = num_pairs * 7; // BITS_PER_PAIR const expected_polar = (polar_bits + 7) / 8 + 1; const expected_qjl = (dim + 7) / 8; Both modules already export the same math as pub helpers (polar.polarBytesNeeded, qjl.qjlBytesNeeded). Calling them directly keeps the validator in sync automatically if either format ever changes its internal widths (e.g. R_BITS/THETA_BITS retuning, QJL widening from 1-bit). Removes the silent drift risk. Also adds symmetric test coverage for the qjl_bytes validation path — the original tests only corrupted polar_bytes (offset 6..10), leaving the qjl_bytes check at offset 10..14 untested. Two new tests (decode + dot) corrupt qjl_bytes and assert the same InvalidPayload / 0.0 behaviour. 63 tests total, all pass.
- Rewrite design doc: removes 'not implemented' status, fixes 2-mode claim (now 9 branches), corrects u64→u32 offsets in the format spec, updates sizes to match the actual 88.3 MB cache, moves done items into the shipped checklist, removes the 'Names' padding section. - Bump to 0.4.1 for PR #8 (security fixes: u32 overflow guards in tq_dot_batch + format.slicePayload, validatePayloadSizes for decode/dot) + follow-up refactor that delegates to polarBytesNeeded/qjlBytesNeeded.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Boots with a small "router" system prompt listing the 8 diagram types.
When the model emits
setType("..."), the grammar detects it and theengine swaps the active KV to that type's pre-baked branch (full docs
at build time and live in a
TQKCmulti-branch container file.Design doc:
demo/docs/dynamic-context-window.md.Branches under
demo/src/draw/prompts/:router— menu, ~800 tokarchitecture/sequence/flowchart/state/orgchart/er/class/swimlane— per-type docs, ~1100-1700 tok eachTouches:
ModeTrackeringrammar.ts(firesonEnteron first setType)engine.mountKV(name)+registerBranchfor pre-baked swapsengine-workermessagemountBranchAndPrefillto do mount + user-turn prefill atomicallyDiagramTypeunion expanded to all 8 values intypes.ts+sdk-types.tsRolls in the demo improvements that accumulated on this branch:
subgroup-butterfly reductions (argmax, mh_softmax, tq_encode);
window-bound softmax; shared-KV skip in batched prefill; uniform-buffer
dedup; mobile-aware WebGPU error; markdown-stripped thinking cloud;
cancel-and-restart; singleton lock; orphan auto-fix before retry;
retry-LCP rollback.
Also a follow-up to #8:
validatePayloadSizesnow callspolar.polarBytesNeeded/qjl.qjlBytesNeededinstead of re-derivingthe formulas, plus symmetric tests for the qjl_bytes corruption path.
Package bumped to 0.4.1 for the PR #8 security fixes + follow-up.
Cache regenerated:
demo/public/system-cache.binis a 9-branch TQKC(88.3 MB).
bun run rebuild-cachereproduces.Test plan
mode-tracker,system-cache-container)bun x tsc --noEmitcleanzig fmt --checkcleansetType(...)once