Skip to content

components.md

Latest commit

 

History

History
346 lines (286 loc) · 19.7 KB

components.md

File metadata and controls

346 lines (286 loc) · 19.7 KB
Raw

Components

Every registered component, exactly as it behaves in code. All operate on tool-output messages (role:"tool"; for Anthropic, tool_result blocks normalized to that shape by apply). Reformat = lossless. Offload = drops bytes, stashes the original, leaves a <<cg:HASH>> marker recoverable via context_guru_expand / GET /expand.

Summary

Component Kind What it drops Recoverable Fires on Key config (default)
format Reformat nothing (compacts JSON) n/a (lossless) pretty-printed JSON tool output min_tokens (50)
toon Reformat nothing (re-encodes JSON arrays as TOON) n/a (lossless) uniform flat JSON object-arrays min_tokens (50)
cacheinject Reformat nothing (adds cache_control) n/a (lossless) Anthropic-family requests —
skeleton Offload function/method bodies via expand fenced ```lang code blocks min_tokens (80)
dedup Offload later byte-identical tool outputs via expand repeated identical outputs min_tokens (100)
collapse Offload middle of an oversized output via expand any large tool output (fallback) max_tokens (2000), head_lines (20), tail_lines (20)
failed_run Offload earlier superseded test/build runs via expand ≥2 run-like outputs min_tokens (100)
cmdfilter Offload lines per declarative DSL filter via expand output matching a filter filters ([]), disable_builtins (false)
extract Offload obvious noise (repeated lines/blocks, blank runs, progress bars) via expand any large output min_tokens (300), trigger
extract_llm Offload (LLM) query-irrelevant content via an LLM-written sandboxed filter via expand large output in a large request strategy (code), model.source, trigger, rewrite, skip_file_reads
smartcrush Offload middle items of a JSON array via expand JSON-array tool output min_items (5), min_tokens (200), keep_first (3), keep_last (2)
mask Offload older tool outputs (age-based) via expand more than keep_recent outputs keep_recent (3), min_tokens (100), keep_head_chars (96)
summarize Offload (LLM) the middle of the transcript → one summary via expand long trajectories summary_level (regular), keep_last (3), min_tokens (500), resummarize_tokens (6000), model.source, trigger

Presets (config): off [] · safe [format, cacheinject] · balanced [format, dedup, failed_run, cmdfilter, cacheinject] · aggressive adds smartcrush, extract · coding [format, skeleton, cmdfilter, cacheinject] · mcp [format, smartcrush, cacheinject] · agent [format, dedup, failed_run, mask, extract, cacheinject] — for long agentic sessions; mask is the biggest lever there (~27–30% content-token savings, no reward loss — see RESULTS.md) · general [format, toon, dedup, failed_run, cmdfilter, mask, extract, collapse, cacheinject] — the recommended all-round pipeline: the reward-neutral levers of agent plus the situational shrinkers (toon/cmdfilter/collapse) that cost nothing when they don't fire. balanced is not recommended for agentic traffic — it omits mask, so it barely helps (6% vs 31% in the Terminal-Bench replay) · summarize [summarize] (run alone — it restructures the whole transcript).

Dynamic, model-aware triggers. Trigger thresholds can be expressed as fractions of the model's context window (resolved dynamically via LiteLLM's public model map, no hand-maintained list): min_request_frac, min_output_frac, and a hard huge_output_frac ("huge tool call" — act regardless of the request-level gate). collapse.max_frac scales its size budget likewise. Absolutes (min_request_tokens, etc.) still win; when the window is unknown, fractions are ignored and absolutes apply (backward compatible). This lets one config generalize across models/benchmarks.

Reversibility in practice. The context_guru_expand tool is advertised on outgoing requests (INJECT_EXPAND=auto|always|never, default auto = only when the request already declares tools and the store persists), so Offload markers are genuinely recoverable — not just described in marker text. Every offloader also applies a marker-inclusive never-worse check per message, so a rewrite never grows a message by the marker's tokens.

LLM-based components (extract with strategy: code/rlm, and summarize) call a model, chosen by model.source: incoming (default — reuse the proxied request's own model + key) or config (a dedicated cheap model set via CHEAP_MODEL* env / the gateway's CheapModel). When no model is available they degrade — extract to its deterministic projection, summarize to a no-op. See design.md.

Common gates every Offload respects: skip non-text (Rewritable) messages, skip content already carrying a marker (no double-offload), and skip if the rewrite (marker + hint included) isn't actually smaller.

Reformat (lossless)

format

Re-encodes a pretty-printed JSON tool output as compact JSON — same value, fewer whitespace tokens. Only acts on tool messages whose trimmed text starts with {/[, is valid JSON, is ≥ min_tokens, and gets smaller. v1 is json-compact only (a TOON encoder is planned).

before:  { "id": 1,           after:  {"id":1,"name":"ada","tags":["x","y"]}
           "name": "ada",
           "tags": [ "x", "y" ] }
  • Lossiness: none — nothing stashed. Shines: verbose pretty-printed JSON/MCP payloads. Inert: already-compact JSON, non-JSON text, small outputs.

toon

Re-encodes a JSON array of uniform, flat objects as TOON (Token-Oriented Object Notation): one header listing the field names once, then one comma-separated row per element. It drops the braces, repeated keys, and quotes that dominate a JSON array's token cost. It's a Reformat (repack in place, nothing stashed): every scalar value is preserved, with one small representational simplification — JSON null renders as an empty cell (indistinguishable from ""). Only arrays whose elements share one key set and hold scalar values are encoded; anything nested, ragged, or non-array is left untouched, and the pipeline's never-worse guard reverts any case that fails to shrink.

before:  [{"id":1,"name":"Alice"},{"id":2,"name":"Bob"}]
after:   [2]{id,name}:
         1,Alice
         2,Bob
  • Config: min_tokens (50). Lossiness: none — nothing stashed (JSON null → empty cell). Shines: long homogeneous JSON arrays (the llm-d TOON config). Inert: nested/ragged/non-array output, or not smaller.

cacheinject

Places an Anthropic cache_control: {type: ephemeral} breakpoint on the last content block of the message just before the newest turn (a stable prefix boundary), so the provider KV cache hits across turns. Adds a control directive, changes no model-visible content.

  • Lossiness: none. Shines: Anthropic/Bedrock/Vertex agents that don't self-cache (the savings lever is provider-side cache hits, invisible to /stats token counts). Inert: non-cache-aware providers, string-content messages (can't carry a block breakpoint), a breakpoint already present. /stats will list it under top_passthrough since it saves no content tokens — that's expected, not dead weight.

Offload (lossy, reversible)

skeleton

Parses fenced ```lang code blocks with tree-sitter and replaces function/method/constructor bodies with a placeholder, keeping signatures, imports, types, and class bodies (so method signatures survive). Stashes the whole original message.

flowchart LR
  A["go fenced block<br/>full func bodies"] --> B{"tree-sitter parse<br/>lang known? body ≥ min_tokens?"}
  B -->|no| A
  B -->|yes| C["signatures + { … }<br/>+ <<cg:HASH>> marker"]
  C --> D[(Store: original)]
Loading 
before:  func Add(a, b int) int {          after:  func Add(a, b int) int { … }
             return a + b                           func Sub(a, b int) int { … }
         }                                          <<cg:9f2a…>> [full source: call context_guru_expand]
  • Config: min_tokens (80, per body). Grammars: go, python, js/ts/tsx, rust, java, c/cpp, ruby, php, c#, kotlin, swift, scala. Shines: the coding preset — the agent reads big source files but mostly needs the shape. Inert: no fenced blocks, unfenced file reads, unknown language, skeleton not smaller than the body.

dedup

Replaces a tool output byte-identical to an earlier one in the same request with a short pointer + marker. Exact match only (near-duplicate is deferred).

before:  <big config dump>  … (later, identical) <same big config dump>
after:   <big config dump>  … [identical to an earlier tool output] <<cg:1c8e…>>
  • Config: min_tokens (100). Shines: agents that re-read the same file/command output repeatedly. Inert: no exact repeats, small outputs.

collapse

Content-agnostic fallback for an oversized tool output nothing more specific handled: keep a head_lines + tail_lines window, stash the full original. Runs late (after cmdfilter/format); skips content already marked.

before:  <2,000-line log>
after:   <first 20 lines>
         ... (1960 lines omitted) <<cg:44ab…>> [full output: call context_guru_expand]
         <last 20 lines>
  • Config: max_tokens (2000 threshold), head_lines (20), tail_lines (20). Shines: a catch-all last stage for huge outputs. Inert: output ≤ max_tokens, or too few lines for head/tail to help.

failed_run

Recognizes test/build run output (regex: N passed/failed, BUILD SUCCESS/FAIL, Traceback, FAILED, panic:, npm ERR!, pytest session banners). Keeps the most recent run in full, collapses every earlier run to a pointer + marker — a superseded run is safely recoverable.

before:  [run 1] 3 failed, 5 passed …   [run 2 after fix] 8 passed
after:   [superseded by a later run] <<cg:7d1c…>> [full output: …]   [run 2] 8 passed
  • Config: min_tokens (100). Needs ≥2 run-like outputs. Shines: iterative fix→re-run loops. Inert: <2 runs detected, small outputs. False positives cost only an expand round-trip, never data.

cmdfilter

Shrinks tool output with declarative DSL filters (see below). Matches a filter on the output's first non-empty line, applies its 8-stage pipeline, stashes the original, and appends a recovery hint only when the filter was actually lossy. Ships builtin pytest / npm-install / make filters.

before:  pytest … 100 lines of PASSED + warnings + 1 failure
after:   <failures + summary, passing noise stripped, ≤80 lines> <<cg:…>> [full output: …]
  • Config: filters (inline filter YAML docs, added with no recompile), disable_builtins. Enabled only when ≥1 filter is loaded. Shines: noisy but structured command/log output (test runners, package managers, build tools). Inert: output whose first line matches no filter, or where filtering doesn't shrink it.

extract

Deterministic, no-LLM. Collapses only obvious, provably redundant noise: consecutively repeated lines/blocks (up to 12 lines), runs of blank lines, and progress-bar/spinner churn — keeping every unique informative line verbatim. Runs cheaply on every request; stashes the original.

before:  resolved 200 packages                after:  resolved 200 packages
         warning: peer dependency unmet                warning: peer dependency unmet
         warning: peer dependency unmet   (×15)        build complete in 4.2s
         …                                             <<cg:40b571fdebccdcd4>> [full output: …]
         build complete in 4.2s

(captured live: 15 identical warnings → 1, blank runs collapsed.)

  • Config: min_tokens (300), trigger, marker_mode. Shines: build/install logs, package-manager output, anything with repeated warnings/progress bars. Inert: below floor, nothing obviously redundant, or not smaller once the marker is added. Full page: extract.

extract_llm (LLM)

The relevance-aware counterpart to extract: a cheap model writes a sandboxed Starlark filter (no imports/IO, step + 2s limits) specific to that output, deleting the irrelevant lines/records and — in rewrite mode — rewording/collapsing spans, while keeping ids/paths/errors verbatim. It sees the full output (bounded ~32k chars). JSON bodies are filtered structurally.

before:  2024 GET /users/0 200 12ms   (×60)   after:  2024 GET /users/58 200 12ms
         ERROR auth timeout on token refresh          2024 GET /users/59 200 12ms
         2024 GET /items/0 200 8ms    (×60)            ERROR auth timeout on token refresh
                                                       2024 GET /items/0 200 8ms
                                                       2024 GET /items/1 200 8ms
                                                       [auth timeout error + context; repetitive
                                                        successful requests elided] <<cg:9233…>>

(captured live via aws/claude-haiku-4-5; query: "find the auth timeout error and nearby context".)

  • Guarantee: rewrite: false accepts a result only if it is an in-order character subsequence of the input (deletion-only, provably no fabrication/reorder). Default rewrite: true is the more powerful mode (sanity + strictly-smaller only; ids/paths/errors still required verbatim).
  • Model: model.source = incoming (proxied model+key) or config (CHEAP_MODEL*). No model → no-op.
  • Throttled + reused: gated by trigger and throttled per session (llm_every_n_requests) / per request (llm_max_per_request); a reduced output is checkpointed per session and reused byte-for-byte on later turns (no new call, prefix stays KV-cache stable). skip_file_reads (auto) leaves prompt-cached source dumps verbatim since they already bill cheap.
  • Config: strategy (code), min_tokens, model.source, trigger, rewrite, llm_every_n_requests, llm_max_per_request, skip_file_reads, marker_mode. Full page: extract_llm.

smartcrush

Statistical JSON-array compressor: parse the array, keep keep_first + keep_last items plus any item whose raw JSON carries an error signal, drop the rest, stash the full original. Kept items are verbatim (schema-preserving).

before:  [ {…}, {…}, … 200 items … ]
after:   [ item0, item1, item2, item198, item199 ] [5 of 200 items shown; full array: call …] <<cg:…>>
  • Config: min_items (5), min_tokens (200), keep_first (3), keep_last (2). Shines: long homogeneous JSON arrays (list endpoints, search hits) — the mcp preset. Inert: non-array output, fewer than min_items, nothing to drop. v1 uses fixed anchors (headroom's Kneedle adaptive-K is a documented refinement).

mask

Age-based garbage collection: keep the newest keep_recent tool outputs verbatim, replace older ones (≥ min_tokens) with a short marker + stash. Complementary to the content-based offloaders.

after (older):  [older tool output masked; starts: 700 701 def __rmul__(self, m): 702 …] <<cg:…>> [full output: call context_guru_expand]
  • Config: keep_recent (3), min_tokens (100), keep_head_chars (96). Shines: long agent trajectories where old tool results are unlikely to matter (top lever on terminal/code traffic: 27.5% on Terminal-Bench, 12.5% on SWE-bench; scales down to ~4% on small structured customer-service outputs). Inert: ≤ keep_recent tool outputs, small outputs.
  • keep_head_chars leaves a one-line head-peek of the hidden output inside the marker (see above) so the model knows what was masked without a blind expand round-trip — evidence showed a bare marker on a masked source-file read forces needless expands. Set 0 for the opaque marker (≈2pp more savings).

summarize (LLM)

Compresses the middle of the trajectory into one LLM-written summary (ported from CE-Manager's ReSum-style summarizer). Restructures the message list to [msg0, <summary system message>, last-K]; the replaced span is stashed under a marker carried in the summary message, so expand restores the full earlier trajectory. This is the one component that changes the message count — apply.Body rebuilds the body keeping the retained messages byte-identical.

before:  [system, u1, tool, a1, tool, u2, … 30 turns …, uN-1, uN]
after:   [system, "=== History Summary === … <summary> … <<cg:…>>", uN-1, uN]

The summarizer is grounded in the current task (first user turn + recent turns are passed as "summarize toward this"), not a blind digest of the middle.

  • Config: summary_level (concise|regular|highly_detailed), keep_last (3), min_tokens (500 — span floor), include_tool_calls (false → tool outputs masked in the trajectory), model.source, trigger, resummarize_tokens (6000).
  • Gating + reuse: a trigger (min_request_tokens, min_messages; legacy start_from_message folds into min_messages) gates the first summary so it fires only on a large/deep transcript. After that, the summary is checkpointed per session and reused verbatim (no model call, and byte-identical so the prefix stays KV-cache stable) until the un-summarized tail grows past resummarize_tokens, when the checkpoint rolls forward with a fresh summary. This is what stops it re-summarizing every turn.
  • Shines: long agentic sessions where the bulk is stale middle context. Inert: transcript below trigger, span below min_tokens, or no model available (no-op). Run it alone (its own preset) — it restructures the whole transcript.

The DSL filter engine

components/dsl is a declarative, user-extensible text-filter engine (adapted from rtk), wrapped by cmdfilter. Filters are authored in YAML (no recompile), matched first-by-sorted-name, and each runs a fixed 8-stage pipeline. Because filters drop lines they are lossy, which is why the wrapping cmdfilter component is an Offload (it stashes the original first).

flowchart LR
  I[input] --> S1[1 strip_ansi] --> S2["2 replace[]"] --> S3["3 match_output[] + unless"]
  S3 --> S4[4 strip / keep lines] --> S5[5 truncate_lines_at] --> S6[6 head / tail]
  S6 --> S7[7 max_lines] --> S8[8 on_empty] --> O[output + Lossiness]
Loading

Filter fields (all optional except match): match (regex vs the selector = first non-empty line), strip_ansi, replace (chained pattern→replacement, $1 backrefs), match_output (whole-blob short-circuit: pattern/message/unless), strip_lines_matching xor keep_lines_matching, truncate_lines_at (per-line char cap), head_lines/tail_lines, max_lines (absolute cap with omission marker), on_empty (replacement when output is blank).

Lossiness reported back to cmdfilter (drives whether a recovery hint is appended): None (nothing dropped / reversible reformat), Tail (a clean contiguous tail dropped), Whole (non-contiguous or whole-blob loss).

schema_version: 1
filters:
  pytest:
    description: keep failures + summary, drop passing noise
    match: "(pytest|=+ test session starts)"
    strip_lines_matching: ["^\\s*$", " PASSED", "^\\.+$"]
    max_lines: 80
    on_empty: "pytest: all passed"
tests:                       # inline; run via dsl.RunTests (a `verify` command)
  pytest:
    - name: all-green
      input: "pytest\n....\n"
      expected: "pytest: all passed"

Documents load with schema_version: 1 and strict unknown-field rejection. Inline tests (input → expected) run via dsl.RunTests, so a filter ships with its own regression check.