epic: Session Lifetime Cap Plugin

[evaline-ju]

Problem

Autonomous agents have no platform-enforced ceiling on token spend per session. A runaway or unexpectedly deep run exhausts quota silently with no cost visibility and no circuit breaker. Session lifetime budgets give platform operators a transparent guardrail — enforced at the sidecar layer with no agent code changes.

This evolves two existing implementations: llm-budget-proxy (standalone proxy, per-session token caps via sync Postgres, requires agents to route explicitly) and litellm-budget-track (same sidecar plugin slot, but daily USD ceiling from a local file with no cross-pod state).

Why a Cortex plugin

Cortex already sits on every inference call as part of the pod's auth and policy pipeline — it sees token counts in LLM responses without any agent cooperation. It also tracks per-session state, so budget counters are naturally scoped to each agent session without additional plumbing. Enforcing budgets here means uniform coverage across agent frameworks, no per-agent instrumentation, and workload-level isolation. The alternative — a standalone proxy or per-framework integration — requires agents to opt in and breaks when they don't.

User Interaction

An operator sets per-agent limits in sidecar config:

- name: token-budget
  config:
    max_tokens: 50000
    max_calls: 100
    max_duration_seconds: 1800
    on_exceed: deny   # or "observe" for shadow mode

When a session crosses a limit, the next inference request gets a 429:

{"error": "budget_exceeded", "reason": "token limit reached", "spent": 45000, "limit": 50000}

User Stories

1. As a platform operator, I can see how many tokens each agent session is consuming — so I can identify expensive sessions and establish baselines before setting limits
2. As a platform operator, I can set budget thresholds in shadow mode and see which sessions would have been denied — so I can calibrate limits from real traffic without risk
3. As a platform operator, I can enforce a hard token/call/time ceiling per session — so a runaway agent is stopped before it exhausts quota
4. As a platform operator, I can query historical spend across sessions — so I can report on agent cost by team or workload over time

design

Implementation Breakdown

1. Token count extraction — Wire inference-parser to emit per-session token totals to session API

2. Storage abstraction — General-purpose plugin storage interface for cross-pod persistent state. Initial implementation backed by Redis, isolated as a separate module to avoid coupling.

   Operation	Purpose
   Get / Set (with TTL)	Caching (tokens, decisions)
   Incr	Atomic counters (rate limiting)
   HashIncr / HashGet / HashSetNX	Structured per-session state (budget counters)
   Expire	TTL management for any key

3. Counter accumulation — Async write path, key expiration, background refresh loop, fail-open behavior

4. Budget evaluation engine — Check max_tokens, max_calls, max_duration_seconds against cached counters

5. Shadow mode — Log would-have-denied events; no blocking

6. Enforcement mode — Structured 429 response with spend/limit/reason

7. Per-agent configuration — Plugin config in sidecar YAML, workload identity scoping

8. Audit/reporting — Optional durable spend log for historical queries (storage TBD)

9. Testing — Unit tests for evaluation logic; integration tests for pod restart, storage failure, multi-replica accuracy, and overshoot bounds

10. Documentation — Plugin config reference, shadow→enforce rollout runbook, failure mode guide, orchestrator 429 handling guide

Open Questions

Storage scope — The interface covers token-budget and basic rate limiting. Are there other planned plugin use cases that would require expanding it (e.g. pub/sub for session fan-out, atomic composite operations)?

Connection sharing — Should plugins share a single storage connection (fewer connections, requires binary-level wiring) or each manage their own (simpler plugin code, more connections)? [Leaning toward independent per-plugin connections since plugins are generally self-contained lifecycles and connection overhead should be minimal.]

Success criteria

• Users of the plugin can observe and enforce bounds on tokens, calls, and time per agent session

Stories to be associated with this epic

TODO

[huang195]

Stopping a long running agent due to exceeded budget seems excessive if it's still making reasonable progress. I'm wondering if HITL would be a possible alternative to just denying additional inference calls? As Cortex could potentially hook onto HITL events, we could have another plugin that decides if it is worth letting the agent continue by granting it a temporary budget increase.