fix(cache): keep the per-turn billing line out of the cached prefix (+3 attribution fixes) - #161
Open
qbq-leo-martens wants to merge 4 commits into
Open
fix(cache): keep the per-turn billing line out of the cached prefix (+3 attribution fixes)#161qbq-leo-martens wants to merge 4 commits into
qbq-leo-martens wants to merge 4 commits into
Conversation
…essages[0] `historyImageSha8` hashed the image blocks on `messages[0]`, described in its own docstring as "the synthetic history message". It is not: whenever a slab anchor exists, `collapseHistory` returns `[...head, syntheticUser, ...tail]` and transform.ts passes `protectedPrefix = slabAnchorIdx + 1`, so on every Claude Code request `messages[0]` is the protected slab message. The field that exists to prove history-image byte-identity was therefore reporting SLAB stability under the history name — a drifting collapse boundary still showed a rock-steady `history_image_sha8` in events.jsonl, which is precisely the signal teamchong#11 attribution relies on. Locate the synthetic message by its banner, the same way `cachePrefixDigest` does. Also split the pinned-prefix digest per layer. `cache_prefix_sha8` proves THAT the cacheable prefix moved but never WHICH layer moved, and the layers fail for different reasons and need different fixes: tools drift when the client loads a deferred tool, system drifts when volatile text (env, git status) rides inside the pinned span, and the imaged head drifts when a collapse boundary or marker placement moves. Emit `cache_prefix_tools_sha8` / `cache_prefix_system_sha8` / `cache_prefix_head_sha8` alongside the aggregate so one session of telemetry names the culprit instead of narrowing it to "somewhere in the prefix". Motivation: a production session showed 530/530 requests with a unique `cache_prefix_sha8` at a constant `cache_prefix_bytes`, 0 cache_read across 87M cache_create tokens — with the aggregate hash alone the cause could not be attributed, and `history_image_sha8` looked stable because it was measuring the wrong message.
…t attribution `cachePrefixDigest` hashed every block up to and including the message that carries pxpipe's imaged prefix. That is not the span Anthropic caches: caching ends at the LAST cache_control marker, and after a history collapse the two differ by construction. The synthetic message's newest freeze chunk re-renders on every turn BY DESIGN (append-only rendering: only completed chunks are frozen) and it sits AFTER the pinned marker — so the boundary-scoped digest changes every single turn even when the cached span is byte-stable, and `cache_prefix_sha8` reads "pxpipe busted its own cache" on healthy traffic. Emit `cache_prefix_marked_sha8` / `_marked_bytes` / `_marker_pos` alongside: the digest through the breakpoint, its size, and where the breakpoint sits as `m<msg>.b<block>`. The marked digest is the one that must hold turn over turn; the marker position makes a roaming breakpoint — a bust cause in its own right, previously invisible — a one-field diagnosis. Tests pin the contract that decides whether cache_read happens at all: two consecutive turns of one session keep the marked span byte-identical while the live tail grows.
… text Claude Code folds `x-anthropic-billing-header: <...>` into the *first* system block, and that line carries a per-turn request id. Anything inside the cache_control-anchored prefix has to be byte-identical across turns, so this one line rewrote the pinned prefix on every single request and cost us the whole prompt cache — every turn paid cache_create instead of cache_read. transform now strips the line out of the system text (leaving no empty block behind) and hands it back on TransformInfo; proxy sets it as the outbound header it always claimed to be. Set PXPIPE_KEEP_BILLING_LINE=1 to fall back to the old in-prompt placement. Also records the two digests that made this diagnosable: a text-only system digest and per-block `index[*]:len:hash4` fingerprints, which is what pinned the churn to block 0 rather than to the collapse boundary.
…xists renderStatsTableFragment read s.eventsWithBaseline, but stats.ts emits eventsWithUsage; the missing field made the comparison false for every session, so the "cache hit (by events)" row silently rendered "-" no matter what the cache did. The type carried the stale name too, so tsc had no chance to catch it. Test pins the value to its own row (the table renders as one line, so a bare toContain passes on a neighbouring number) and fails with '-' against the old expression.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Running Claude Code through pxpipe, my prompt-cache hit rate sat at ~0% for a whole
working day while the dashboard cheerfully reported ">95%". That turned out to be two
separate things: one real cache buster, and three places where the telemetry could not
have shown it.
All four are small, independent commits.
1.
fix(cache): the per-turn billing line lived inside the cached prefixClaude Code (>= 2.1.x) folds an
x-anthropic-billing-header: cc_version=...line intoits system text, as an unmarked trailing block. That value is not session-stable —
it carries a per-turn id — so the tail of the cached prefix changed on every single
request: guaranteed full cache miss, every turn, every session.
stripBillingLine()already existed, but it only ever ran on the concatenatedrawSysText, and by then the block Claude Code actually sends had already beenclassified as part of the prefix, so the regex never matched what mattered.
Fix:
liftBillingLineFromSystem()pulls the line out of every system block beforethe static/dynamic split, and the proxy re-attaches it as a real request header
(
x-anthropic-billing-header) — where it belongs, and where it cannot invalidateanything. A block that held nothing but the billing line disappears instead of leaving an
empty shell behind.
Escape hatch:
PXPIPE_KEEP_BILLING_LINE=1restores the old in-prompt placement.Effect on my host: cache reads went from ~0 to the expected 90%+ of input tokens per
session, immediately and across restarts.
2.
fix(cache): bust attribution digested the boundary message, not the MARKED spancachePrefixMarkedSha/cachePrefixMarkedByteswere computed over the boundary messagerather than the span that actually carries the
cache_controlmarker — i.e. the bytesAnthropic really caches. The diagnostics therefore could never point at what changed
inside the pinned prefix (which is exactly what I needed to find #1). Now the digest
covers the marked span and the marker position is recorded alongside it, with per-layer
digests (tools / system / head) so a bust can be attributed to one layer.
3.
fix(cache): history-image hash was read frommessages[0]history_image_sha8was taken frommessages[0]instead of the message that actuallycarries the imaged history, so a perfectly stable history looked like it churned every
turn (and real churn could hide).
4.
fix(dashboard): the by-events cache-hit rate divided by a field that doesn't existrenderStatsTableFragment()dividescacheHitEventsbys.eventsWithBaseline, butstats.tsemits that number aseventsWithUsage.undefined > 0is false, so theevent-based hit rate silently rendered
-forever. Renamed the field inFullStatsSummaryto match the producer.Tests
tests/cache-bust-attribution.test.ts(10 cases), including the regression thatmatters: two turns differing ONLY in the billing id keep a byte-identical cached
prefix, plus per-layer digest movement and marked-span stability while the live tail
grows.
tests/dashboard-api.test.tsandtests/render.test.ts.tsc --noEmitclean,npm run buildclean.Notes
Found while debugging a real ~2h token burn on a self-hosted box; happy to adjust naming,
split the commits differently, or drop #4 into its own PR if you prefer.