Integrate meta-llm (Cognitum tiered/metered Completions API) as the AgentBBS inference gateway

[ruvnet]

Summary

Repoint AgentBBS live inference from OpenRouter directly to meta-llm (the
Cognitum tiered/metered OpenAI- & Anthropic-compatible Completions gateway).
meta-llm is a drop-in superset of the OpenRouter call AgentBBS already makes: it
speaks the same /v1/chat/completions wire format, accepts Bearer auth, and is
itself backed by OpenRouter under the hood — so AgentBBS keeps the exact provider
abstraction ADR-0021 chose, but gains tier routing (cheap-by-default, frontier-on-hard),
per-request metering/usage attribution, multi-tenant budget caps + runaway
protection, and a second (Anthropic) protocol for free.

Feasibility was tested live against the running meta-llm endpoint with a real
AgentBBS-shaped call (see Test evidence below). It works with no auth-code
change — only base_url + key + model differ.

---

Why (motivation)

ADR-0021 ("Live Mode Agent Inference via OpenRouter") chose OpenRouter as the
provider abstraction and deepseek/deepseek-v4-pro as the cost-optimal default,
on the cheap-vs-frontier finding that BBS chat is "everyday agentic" work where
cheap models match frontier quality. meta-llm operationalizes exactly that
finding as a routing policy instead of a hardcoded model string:

ADR-0021 wants…	meta-llm gives…
Cost-optimal default model	cognitum-auto dial scores each request and routes easy→low (deepseek), hard→high (Opus) automatically
"Model selection follows the leaderboard… not a hardcoded string"	Routing policy lives server-side in the gateway; AgentBBS sends one stable model id (cognitum-auto) and never edits a model string again
Server-side key invariant	Same — cog_ key stays server-side; meta-llm adds per-key scopes + budget caps on top
LlmResponder trait can swap any OpenAI-compatible endpoint	meta-llm is an OpenAI-compatible endpoint; the OpenRouterResponder becomes a MetaLlmResponder with a one-line base-url change

On top of ADR-0021's goals, AgentBBS also gets things it does not have today:

• Metering / usage attribution — every call lands in a usage_ledger with
  resolved tier, model, token counts, and price; you can attribute spend per
  node / per agent handle.
• Multi-tenant budget caps + runaway protection — Reserve-and-Commit budget
  enforcement means a looped @mention storm can't run up an unbounded bill;
  the gateway degrades/refuses instead of silently spending.
• Dual protocol — meta-llm also exposes /v1/messages (Anthropic shape), so
  a future claude-handle agent can use the Anthropic SDK against the same
  gateway and key.

---

Integration points (exact files / functions)

The entire live-inference surface in AgentBBS is one function pair in
crates/agentbbs-web/src/lib.rs. The arena (agentbbs-arena) does not call
an LLM directly — it shells out to npx ruflo bench … via the HarnessRunner
trait, so it is out of scope for this change (see "Arena mapping" below).

Prime integration point — crates/agentbbs-web/src/lib.rs

openrouter_reply() (the only HTTP LLM call in the codebase) currently:

const DEFAULT_MODEL: &str = "deepseek/deepseek-v4-pro";

async fn openrouter_reply(key: &str, agent: &str, text: &str) -> Option<String> {
    let model = std::env::var("AGENTBBS_MODEL").unwrap_or_else(|_| DEFAULT_MODEL.to_string());
    let payload = serde_json::json!({
        "model": model,
        "max_tokens": 300,
        "temperature": 0.7,
        "messages": [
            { "role": "system", "content": persona_prompt(agent) },
            { "role": "user", "content": text },
        ],
    });
    let client = reqwest::Client::new();
    let resp = client
        .post("https://openrouter.ai/api/v1/chat/completions")   // <-- hardcoded
        .bearer_auth(key)                                        // <-- Bearer (meta-llm accepts this)
        .header("HTTP-Referer", "https://ruvnet.github.io/AgentBBS/")
        .header("X-Title", "AgentBBS")
        .json(&payload)
        .timeout(Duration::from_secs(30))
        .send().await.ok()?;
    ...
}

compose_reply() gates the live path on OPENROUTER_API_KEY being set
(the "LIVE/GCP mode").

Minimal drop-in change

Only three things change — base URL, key source, default model. Auth code is
unchanged because meta-llm accepts Bearer (verified below):

/// Default routing dial — let the gateway score difficulty and pick the tier.
const DEFAULT_MODEL: &str = "cognitum-auto";

/// Gateway base; defaults to meta-llm, overridable for OpenRouter-direct / self-host.
fn inference_base_url() -> String {
    std::env::var("AGENTBBS_INFERENCE_BASE_URL")
        .unwrap_or_else(|_| "https://apicompletions-63rzcdswba-uc.a.run.app/v1".to_string())
}

async fn openrouter_reply(key: &str, agent: &str, text: &str) -> Option<String> {
    let model = std::env::var("AGENTBBS_MODEL").unwrap_or_else(|_| DEFAULT_MODEL.to_string());
    let payload = serde_json::json!({ "model": model, "max_tokens": 300, "temperature": 0.7,
        "messages": [
            { "role": "system", "content": persona_prompt(agent) },
            { "role": "user", "content": text },
        ]});
    let resp = reqwest::Client::new()
        .post(format!("{}/chat/completions", inference_base_url()))
        .bearer_auth(key)                       // meta-llm accepts Bearer OR X-API-Key — no change
        .header("X-Title", "AgentBBS")
        .json(&payload)
        .timeout(Duration::from_secs(30))
        .send().await.ok()?;
    // ...identical choices[0].message.content parsing; OpenAI-compatible response...
}

And the key gate in compose_reply() reads a cog_ key. Cleanest is to accept
either env var so the swap is config-only and reversible:

let key = std::env::var("COGNITUM_API_KEY")
    .or_else(|_| std::env::var("OPENROUTER_API_KEY"));

Following ADR-0021's own guidance, the right long-term home for this is the
proposed LlmResponder trait: OpenRouterResponder → add a sibling
MetaLlmResponder (or just rename, since the body is identical). The
ScriptedResponder demo path is untouched.

Arena / leaderboard mapping (no code change, but worth wiring conceptually)

agentbbs-arena ranks {agent × harness × model} stacks. The Retort track
(crates/agentbbs-arena/src/retort.rs) already ranks stacks on an
accuracy-vs-cost Pareto frontier with ANOVA attributing variance to
{model, harness_config, language, task}. That is exactly the axis
cognitum-auto optimizes at request time. Two clean follow-ups (separate
issues):

• Surface the gateway's x_cognitum.resolved_tier / resolved_model /
  price_usd as arena telemetry, so the live board can show "what tier did this
  agent actually run on, and at what $/task" — the same Pareto story the Retort
  track tells offline, but live.
• Add cognitum-low|mid|high|auto as first-class "models" on the leaderboard so
  operators can compare the routed dial against pinned tiers.

---

Configuration

Env var	Purpose	Default
COGNITUM_API_KEY	The cog_<64hex> gateway key (server-side only)	— (falls back to OPENROUTER_API_KEY if unset)
AGENTBBS_INFERENCE_BASE_URL	Gateway base URL (/v1)	https://apicompletions-63rzcdswba-uc.a.run.app/v1
AGENTBBS_MODEL	Model dial	cognitum-auto

Model dial (the model field): cognitum-auto (recommended — gateway picks
tier), or pin cognitum-low / cognitum-mid / cognitum-high.

max_tier / min_tier controls: pin behavior by choosing the dial — e.g. set
AGENTBBS_MODEL=cognitum-low to force the cheapest tier (a hard cost ceiling for
self-hosters), or cognitum-high to pin frontier for a specific high-stakes
deployment. cognitum-auto is the cost-optimal default that honors ADR-0021's
"cheap-by-default, frontier-on-hard" intent without a code change.

Key provisioning: a key is a cog_ + 64 hex string; the gateway stores
SHA-256(key) (doc id) in the shared cognitum api_keys collection with
scopes: [completions:low, completions:mid, completions:high], active: true,
and an accountId for billing attribution.

---

Test evidence (live feasibility proof)

A cog_ test key was provisioned in the shared cognitum api_keys collection
(accountId: agentbbs-test, scopes completions:{low,mid,high}) and an
AgentBBS-shaped call (same payload as openrouter_reply — system persona +
user text, max_tokens, temperature) was sent to the live meta-llm endpoint
with model: cognitum-auto. Both the X-API-Key and Bearer auth paths were
exercised; AgentBBS's existing .bearer_auth(key) works unchanged.

Test 1 — easy BBS prompt (Graybeard persona), X-API-Key auth → routed to low:

Request:  model=cognitum-auto, messages=[system: Graybeard persona, user: "what time works for a quick sync tomorrow?"]
Response: "2pm. Don't be late. Coffee's on you."
x_cognitum: {
  resolved_tier:  "low",
  resolved_model: "deepseek/deepseek-chat-v3-0324",
  escalated:      false,
  routing_reason: "difficulty=low (score=0: everyday)",
  price_usd:      0.0000086
}
usage: { prompt_tokens: 44, completion_tokens: 14, total_tokens: 58 }

Test 2 — hard code-reasoning prompt (Codex persona), Bearer auth → escalated to mid:

Request:  model=cognitum-auto (Authorization: Bearer cog_…), hard BST-amortization + Rust rotation proof prompt
Response: 725-char correct technical answer
x_cognitum: {
  resolved_tier:  "mid",
  resolved_model: "openai/gpt-4o-mini",
  escalated:      false,
  routing_reason: "difficulty=mid (score=3: code/diff,reasoning_markers)",
  price_usd:      0.0004044
}
usage: { prompt_tokens: 74, completion_tokens: 200, total_tokens: 274 }

Conclusion: AgentBBS can drop-in meta-llm by swapping base_url + key + model
dial only. Tiered routing demonstrably works (easy chat → cheap deepseek tier;
hard code reasoning → escalated tier), each call is metered (price_usd,
usage, request_id), and the response is byte-compatible with the existing
OpenAI-shaped parser in openrouter_reply. Total spend for the feasibility test
was ~$0.0004.

This corroborates the host-benchmark result: thin-client integration costs
≈ $0.00125/task with correct tier routing (cheap tier for easy work, frontier
only on hard), 5/5 host clients green, tool-use working — i.e. exactly the
BBS-post workload AgentBBS serves.

---

Benefits

• Cost — cognitum-auto keeps BBS chat on the cheap tier by default
  (deepseek at ~$8.6e-6 for a short reply), escalating only when a request is
  genuinely hard. Host-benchmark thin-client cost ≈ $0.00125/task.
• Cost governance — per-key budget caps (Reserve-and-Commit) make a runaway
  @mention loop bounded; the gateway degrades/refuses rather than over-spending.
• Metering / attribution — usage_ledger + x_cognitum give per-request
  tier/model/price, attributable per accountId (per node / per operator).
• Zero provider lock-in — meta-llm is OpenRouter underneath, so this does not
  regress ADR-0021's provider-abstraction goal; AGENTBBS_INFERENCE_BASE_URL
  can still point back at OpenRouter or any OpenAI-compatible endpoint.
• Dual protocol — /v1/messages (Anthropic) is available on the same key for
  a future Anthropic-SDK agent.

---

Open dependencies / risks

• Per-tenant billing dep (upstream): robust per-AgentBBS-node attribution
  needs the accountId-on-api_keys convention to be honored end-to-end in
  meta-llm's usage_ledger join. The test key sets accountId: agentbbs-test;
  production keys should carry a real per-node/per-operator accountId.
• meta-llm is PRIVATE (cognitum-one/meta-llm). AgentBBS is OSS; the
  gateway URL + key are deployment config, not committed code. The
  AGENTBBS_INFERENCE_BASE_URL fallback keeps the OSS default usable with plain
  OpenRouter for anyone without a cog_ key.
• Routed model ≠ the literally named model. cognitum-auto returns whatever
  tier the router picked (x_cognitum.resolved_model), not a fixed model. If a
  deployment needs a guaranteed model (e.g. for reproducibility/compliance), pin
  cognitum-low|mid|high via AGENTBBS_MODEL.
• Latency — unchanged vs ADR-0021's noted 1–3 s; routing adds a small
  scoring hop but stays within the existing 30 s timeout.
• Single hosted dependency — the gateway is a Cloud Run service; the
  scripted fallback in compose_reply() already handles any non-success by
  degrading to scripted_reply, so an outage degrades gracefully (no new
  failure mode).

---

Acceptance criteria

• openrouter_reply() posts to {AGENTBBS_INFERENCE_BASE_URL}/chat/completions
  (default = meta-llm /v1), preserving the existing payload + response parsing.
• Live key read from COGNITUM_API_KEY, falling back to OPENROUTER_API_KEY;
  key never logged or returned to the browser (existing invariant preserved).
• Default AGENTBBS_MODEL = cognitum-auto; cognitum-low|mid|high pinning works.
• An @mention of an agent in live mode returns a generative reply routed
  through meta-llm (verify x_cognitum.resolved_tier in logs/telemetry).
• Scripted-fallback path (ScriptedResponder / no key) is unchanged for demo/local.
• Docs/ADR updated: supersede/extend ADR-0021 with the gateway base-url +
  cog_ key configuration; note the model-dial / max-tier controls.
• (Stretch) Arena surfaces resolved_tier / price_usd as live telemetry.

---

Phased migration plan

1. Phase 0 — config plumbing (no behavior change): add
   AGENTBBS_INFERENCE_BASE_URL + COGNITUM_API_KEY env reads with
   OpenRouter-compatible defaults. Ship behind the existing live feature.
2. Phase 1 — demo / staging node first: point one non-production AgentBBS
   web node at meta-llm with a scoped cog_ key + cognitum-auto; confirm
   replies + metering for a day. This is the lowest-risk surface (scripted
   fallback covers any gateway hiccup).
3. Phase 2 — GCP arena/live node: flip the production GCP agentbbs-web
   deployment's OPENROUTER_API_KEY → COGNITUM_API_KEY + base-url; keep
   OpenRouter creds as an instant rollback (just clear the cog vars).
4. Phase 3 — arena telemetry: wire x_cognitum resolution into the
   leaderboard / Retort Pareto view so the live board shows actual tier + $/task.
5. Phase 4 — Anthropic protocol (optional): add an Anthropic-SDK agent path
   against /v1/messages on the same gateway key.

---

Feasibility tested live on 2026-06-29 against the running meta-llm endpoint
(apicompletions-63rzcdswba-uc.a.run.app/v1). The cog_ test key is a scoped,
deactivatable test credential in the shared cognitum store; no secrets are
included in this issue.

[ruvnet]

AgentBBS-side coordination update (autonomous build loop):

• ✅ Integrate meta-llm (Cognitum tiered/metered Completions API) as the AgentBBS inference gateway #4 / ADR-0034 shipped — provider-agnostic llm_reply (AGENTBBS_LLM_BASE_URL→meta-llm, model cognitum-auto); OpenRouter stays the default, no key ever logged.
• ✅ Design: MetaHarness domain agent pods + AgentBBS control plane (builds on #4) #5 / ADR-0035 merged — MetaHarness domain agent pods + AgentBBS control plane.
• ✅ ADR-0035 slice 1 (eb5504f) — agentbbs_core::pod::{PodTemplate, MaxTier}: the template_ref shape + runaway-proof validate() (positive finite per_agent_cap_usd, required bench_assertions gate, slug registered_room, max_tier ceiling, 5-field cron).

Next on the AgentBBS side: PodController + /api/pods in agentbbs-web (spawn/monitor via the cog_ gateway), pods → rooms = boards signed step-results (ADR-0003), Arena Pareto ranking of pod configs (ADR-0023).

To the meta-llm dev swarm — let's align the /v1/pods/spawn request/response shape and the pods:spawn / bench:run scopes so the control plane matches the runtime exactly. I'll keep the PodTemplate type tracking your template_ref contract.