Skip to content

feat: add /compact endpoint and llm-d compaction service example#8

Merged
OsherElhadad merged 1 commit into
mainfrom
feat/llm-d-compaction-service
Jul 20, 2026
Merged

feat: add /compact endpoint and llm-d compaction service example#8
OsherElhadad merged 1 commit into
mainfrom
feat/llm-d-compaction-service

Conversation

@OsherElhadad

Copy link
Copy Markdown
Collaborator

What

Makes context-guru usable as a standalone compaction service for
llm-d-router's
request-inline-compaction step (PR #1). That step POSTs the raw inference
request body to a compaction server and, on 200 + non-empty JSON, swaps that
body in. Contract: request body in → smaller body of the same shape out.

The proxy was forward-only (routes proxy to OpenAI/Anthropic and return a
completion), so this adds a compact-and-return endpoint plus the pieces needed
to run it cleanly without the store.

Changes

  • POST /compact (proxy/): runs the pipeline over the request messages
    and returns the rewritten body without forwarding upstream. OpenAI by default,
    ?provider=anthropic; per-request overrides via ?preset= and
    x-context-guru-pipeline. Fail-open (bad input → original body, 200).
  • Store enable/disable flag: store.enabled in YAML, plus --store /
    STORE. Adds a Nop store and Store.Persists(), so a full marker_mode
    automatically degrades to an irreversible drop when the store can't persist —
    instead of leaving an unresolvable <<cg:HASH>> marker (applied in
    mark/summarize/cmdfilter).
  • toon component (components/reformat/): deterministic re-encoding of
    uniform JSON object-arrays into tabular TOON. Lossless repack; tests included.
  • Config-driven model creds: the model: block on extract/summarize now
    takes provider/base_url/api_key/model/auth; empty api_key falls
    back to env keys (incl. ANTHROPIC_AUTH_TOKEN for bearer gateways).
  • examples/llm-d-service/: three configs (toon, extract-code, summarize;
    store off, marker_mode: off), build.sh, a dependency-free Go client
    mirroring the router's call, a self-contained README, and a captured
    before/after sample run.

Testing

  • CGO_ENABLED=1 go build ./..., go vet ./..., go test ./... — all pass;
    gofmt clean. New tests: toon encoder + the store-disabled marker degrade.
  • End-to-end: ran the service for each config against a shared context.
    Deterministic toon reduced a sample request ~46%. The two LLM configs ran a
    real claude-haiku-4-5 call through a bearer gateway (extract filtered to
    the relevant lines deletion-only; summarize collapsed the transcript ~81%),
    all with the store disabled and no markers in the output.

Note

The commit skips the repo's dnephin go-vet pre-commit hook (SKIP=go-vet): it
passes individual filenames to go vet, which errors on a multi-package commit
("no Go files in ..."). go vet ./... is clean; all other hooks (gofmt,
go-mod-tidy, gitlint, DCO) ran.

Turn context-guru into a standalone compaction service for llm-d-router's
request-inline-compaction step: request body in, smaller body of the same
shape out, no upstream forward, fail-open.

- proxy: POST /compact runs the pipeline and returns the rewritten body
  without forwarding; OpenAI default, ?provider=anthropic; per-request
  overrides via ?preset= and x-context-guru-pipeline.
- store: configurable enable/disable flag (store.enabled, --store/STORE) and
  a Nop store; add Store.Persists() so a full marker_mode degrades to an
  irreversible drop when the store cannot persist, instead of leaving an
  unresolvable marker (applied in mark, summarize, cmdfilter).
- reformat: new deterministic toon component (uniform JSON object-arrays to
  tabular TOON) with tests.
- offload: config-driven model endpoint and credentials in the model block
  (provider, base_url, api_key, model, auth), shared by extract and
  summarize; empty api_key falls back to env keys including bearer tokens.
- examples/llm-d-service: three configs, build.sh, a dependency-free Go
  client, a README, and a captured before/after sample run.

Assisted-By: Claude (Opus 4.8)
Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
@OsherElhadad
OsherElhadad merged commit ec46469 into main Jul 20, 2026
4 checks passed
@github-project-automation github-project-automation Bot moved this from New/ToDo to Done in Rossoctl Issue Prioritization Jul 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants