diff --git a/CLAUDE.md b/CLAUDE.md index 74b8cdc..fa3f781 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -24,7 +24,8 @@ COVENANT_BEHAVIOR_DRIFT=1 covenant check # body-only drift (schema identical); p | 1 proxy + quarantine | `covenant/proxy/` | `fastapi`/`httpx`/`uvicorn` are the `[proxy]` extra, lazily imported in `cli.py`. | | 2 store | `covenant/store/` | `asyncpg` is the `[store]` extra. Store writes are best-effort: log and swallow, never fail the request path. | | 3 probes + judge | `covenant/fingerprint.py`, `covenant/judge/` | Probes *execute* tools at snapshot/check time (list read-only tools only). `anthropic`/`google-genai` are the `[judge]` extra, imported on use; the model-name prefix picks the provider. | -| 4 observability | `covenant/proxy/metrics.py`, `deploy/` | `prometheus-client` rides the `[proxy]` extra. One `CollectorRegistry` per app, never the global registry (tests create many apps). Metric writes are in-process and non-throwing, never on the store path. | +| 4 observability | `covenant/proxy/metrics.py`, `deploy/` | `prometheus-client` rides the `[proxy]` extra. One `CollectorRegistry` per app, never the global registry (tests create many apps). Metric writes are in-process and non-throwing, never on the store path. Tool labels are clamped to baseline names (cardinality guard). | +| 5 k8s operator | `covenant/operator/`, `deploy/helm/` | `kopf`/`kubernetes` are the `[operator]` extra. `reconcile.py` is pure (no kopf/k8s imports — tests run cluster-free); `handlers.py` is glue only. A failed check is `status.result: error`, never a crash-loop. | Design specs (rationale, rule tables, named decisions) live in `docs/superpowers/specs/` — read the relevant spec before changing classifier or proxy behavior. diff --git a/DEVELOPER-GUIDE.md b/DEVELOPER-GUIDE.md new file mode 100644 index 0000000..865acb7 --- /dev/null +++ b/DEVELOPER-GUIDE.md @@ -0,0 +1,234 @@ +# Covenant — developer's guide + +This is the owner's map of the codebase: what each part does, why it's built that way, +how to demo it, and what to say (and not say) when presenting it. The README sells the +tool to users; this file explains it to *you*. + +## The one-paragraph version + +MCP servers expose tools; agents consume them. When a server changes a tool — renames an +output field, tightens an input, rescales a value — nothing throws. The agent keeps +calling, reads a field that no longer exists, and confidently reports a wrong answer. +Covenant makes the tool contract **explicit** (`covenant snapshot` → committed +`covenant.lock.json`), **checked** (`covenant check` diffs the live server against the +baseline and classifies every change), and **enforced** (`covenant proxy` quarantines +drifted tools at runtime; the K8s operator does this declaratively for a fleet). + +## The intellectual core: the direction principle + +If you explain only one thing, explain this. Severity is not "how big is the change" — +it's **which side of the tool the change is on**, because the consumer is an LLM agent +that re-reads tool definitions every run: + +- **Input-side changes fail loud.** The server rejects a bad call, or the agent re-reads + the schema and adapts. Recoverable → **DEGRADED** (warn; fails CI only under `--strict`). +- **Output-side changes fail silent.** The agent reads a value that is gone, retyped, or + now intermittently `null` — and proceeds confidently. Unrecoverable → **BREAKING** + (fails CI; quarantined at the proxy). + +This inverts REST intuition (where a tightened input is the classic break) and it is the +justification for every row of the rule table in `covenant/diff.py`. The full rationale +lives in `docs/superpowers/specs/2026-07-01-covenant-layer0-contract-core-design.md`. + +## Layer map + +Dependency-ordered: each layer ships alone and reuses the ones below it. Optional +dependencies are real extras in `pyproject.toml` — the core installs with nothing but +`mcp`, `typer`, `rich`. + +### Layer 0 — contract core (`covenant/*.py`) + +| File | Job | +|---|---| +| `introspect.py` | Connects to an MCP server (stdio subprocess or streamable-HTTP), lists tools, runs probes. The only file that talks MCP. | +| `contract.py` | Tool → canonical contract record; read/write/parse the lock file. The lock is **deterministic**: sorted keys, no timestamp — re-snapshotting an unchanged server is byte-identical, so the lock diffs cleanly in git. | +| `diff.py` | The classifier. **Pure** — no I/O, takes two contract sets, returns `Change` records with `tier` ∈ breaking/degraded/compatible. Walks nested schemas recursively (`balance.currency`, `items[].sku`), handles type unions/nullability, and deliberately refuses to guess at `$ref`/`allOf`/`oneOf` composition (flags DEGRADED for manual review instead). | +| `report.py` | Renders the drift table (rich) or `--json`, and maps tiers → exit codes. | +| `cli.py` | Typer app: `snapshot`, `check`, `proxy`. Wires config → introspection → diff → report. | +| `config.py` | Parses `covenant.toml` (server target, `[[probes]]`, `[judge]`). | +| `errors.py` | `CovenantError`: every expected failure becomes **one clean line and exit 2** — never a stack trace, never swallowed. | +| `fingerprint.py` | Response → type-shape fingerprint (see Layer 3). | + +**Exit codes are the CI contract**: 0 clean, 1 breaking (or degraded under `--strict`), +2 config/connection error. `schema_hash` covers schemas only — a description typo must +not read as an identity change. + +### Layer 1 — proxy + quarantine (`covenant/proxy/`) + +`server.py` is a FastAPI reverse-proxy that forwards every JSON-RPC exchange +byte-for-byte (SSE passthrough included) — the client cannot tell it's there. A +`tools/call` to a **quarantined** tool is short-circuited with a clean MCP `isError` +result and never forwarded: the agent sees "tool unavailable" instead of hallucinating +around drifted output. `detect.py` re-checks the upstream; `quarantine.py` holds state. + +The key design decision: **drift detection is proxy-owned**. `POST /covenant/refresh` +re-reads the baseline from disk (so a re-snapshot or updated ConfigMap takes effect +without a restart), then makes the proxy re-list the upstream *itself*. Enforcement +never depends on the client's `tools/list` timing, because the SDK can list *after* +the call it should have protected. + +### Layer 2 — store (`covenant/store/`) + +Optional Postgres persistence (`asyncpg`): quarantine survives restarts; calls and drift +events are logged. The invariant to quote: **store writes are best-effort — log and +swallow, never fail the request path**. A firewall must not drop traffic because its own +telemetry hiccuped. `memory.py` is the default in-process implementation, `base.py` the +interface, so the proxy code has one code path. + +### Layer 3 — behavioral probes + LLM judge (`fingerprint.py`, `judge/`) + +Schema checking can't see a server that *lies* (schema unchanged, body different), and +most real MCP tools declare no `outputSchema` at all. Probes cover both: you commit safe, +read-only example calls in `covenant.toml`; `snapshot` runs them and stores each +response's **fingerprint** — the type shape of what actually came back — plus one sample +response. `check` re-runs the probes and classifies shape drift with the same severity +model, at location `behavior`. + +The judge (`--judge`, `[judge]` extra) catches drift a fingerprint can't: same shape, +changed meaning (a balance quietly rescaled dollars→cents). The model name picks the +provider — `claude-*` → Anthropic, `gemini-*` → Gemini. + +The invariant to quote: **judge verdicts are advisory — DEGRADED, never BREAKING.** A +probabilistic detector must not trigger quarantine. Shape drift is judged on the +fingerprint alone because values legitimately change between runs. + +### Layer 4 — observability (`covenant/proxy/metrics.py`, `deploy/`) + +Prometheus metrics at `GET /covenant/metrics`: per-tool call counters (ok/error/blocked), +latency histograms, drift events, a quarantine gauge. `docker compose up -d prometheus +grafana` gives a provisioned dashboard — the quarantine stat flips green→red within one +scrape of a drift. + +Two decisions worth naming: +- **One `CollectorRegistry` per app instance**, never the global registry — tests create + many apps and the global one collides. +- **Label-cardinality guard**: tool labels are clamped to the baseline name set (unknown + tool names become `"unknown"`), so an attacker calling ten thousand made-up tool names + can't mint ten thousand Prometheus series. + +### Layer 5 — K8s operator + Helm (`covenant/operator/`, `deploy/helm/covenant/`) + +Declarative contract conformance: an `MCPContract` CR names a server, a baseline +ConfigMap, and an interval; a kopf operator runs the existing Layer 0/3 check on that +schedule, writes the verdict into `.status` (printer columns: `kubectl get mcpcontracts` +→ RESULT / BREAKING / LAST CHECK), and POSTs the proxy's `/covenant/refresh` so +quarantine follows drift. + +Decisions worth naming: +- **Purity split**: `reconcile.py` holds all logic and has **zero kopf/kubernetes + imports** — the whole layer unit-tests without a cluster (`tests/test_operator.py`). + `handlers.py` is glue only. +- **In-operator checks, not Jobs** (a deliberate deviation from the original pitch): a + check takes milliseconds; running it as a Job means log-scraping or per-Job RBAC just + to get status back. Revisit Jobs when a check becomes long or needs isolation. +- **Per-CR scheduling via due-gating**: kopf's timer interval is fixed at decoration time + (30s poll); each CR keeps its own `spec.intervalSeconds` enforced by `due()` against + `status.lastCheckTime`. +- **A failed check is status, not an exception**: unreachable server, malformed + baseline, or missing ConfigMap key → `status.result: error`, with tier counts zeroed + so kopf's merge patch can't leave stale numbers behind. The operator never raises + from the timer and never crash-loops on one bad contract. +- **RBAC is least-privilege**: mcpcontracts (+status) list/watch/get/patch, configmaps + get, events create. Nothing else. + +One Dockerfile serves both roles (proxy and operator); the Helm chart ships CRD + proxy +Deployment/Service + operator Deployment. + +## How the pieces talk + +``` +covenant.toml ──snapshot──▶ covenant.lock.json (committed, deterministic) + │ + ┌──────────────────────────┼───────────────────────────┐ + ▼ ▼ ▼ + covenant check (CI) covenant proxy K8s operator + diff + probes + judge quarantine on drift same check on a schedule, + exit 0/1/2 /covenant/refresh verdict → CR status, + /covenant/metrics nudges proxy refresh +``` + +One classifier (`diff.py`), one baseline format (`contract.py`), three enforcement +surfaces (CI, runtime proxy, cluster). That reuse is the architecture argument: higher +layers add *where* the check runs, never *what* the check means. + +## Demo script + +```bash +# 1. The linter catches a lie in the schema AND the body +covenant check # OK, exit 0 +COVENANT_DRIFT=1 covenant check # BREAKING (output + behavior rows), exit 1 + +# 2. Schema-identical drift — only probes catch it +COVENANT_BEHAVIOR_DRIFT=1 covenant check # schema clean, behavior BREAKING, exit 1 + +# 3. Semantic drift — same shape, changed meaning; only the judge catches it +COVENANT_SEMANTIC_DRIFT=1 covenant check --judge --strict # DEGRADED, exit 1 (strict) + +# 4. Runtime containment +covenant proxy --upstream http://localhost:8000/mcp --port 9000 +curl -X POST localhost:9000/covenant/refresh # proxy re-checks, quarantines +curl localhost:9000/covenant/status + +# 5. Observability +docker compose up -d prometheus grafana # dashboard at localhost:3000 + +# 6. Fleet (needs a cluster + docker build) +helm install covenant deploy/helm/covenant --set proxy.upstream=http://my-server:8000/mcp +kubectl create configmap covenant-baseline --from-file=covenant.lock.json +kubectl apply -f examples/mcpcontract.yaml +kubectl get mcpcontracts -w # RESULT flips clean -> breaking +``` + +CI runs the drift injections against the repo's own example server on every push +(`.github/workflows/ci.yml`) — the project eats its own dog food. + +## Verification status — be precise about this + +- 129 tests (+3 Postgres-skipped without a DB), `ruff`, strict `mypy` — green locally + and in CI. Postgres tests run against a real container when `COVENANT_TEST_DB` is set. +- **Cluster-verified end-to-end** (Docker Desktop Kubernetes): image builds and both + container roles run; `helm lint` clean; `helm install` deploys all 7 objects; the + operator reconciles a live `MCPContract` through the full lifecycle — clean → + breaking (BREAKING=2: schema + probe rows) with the proxy quarantining the exact + tool, then back to clean with the quarantine released. The live run surfaced and + fixed two integration bugs no unit test could see: kopf needs cluster-scoped + CRD + namespace list rights at startup, and the MCP SDK's DNS-rebinding guard + 421s `host.docker.internal` unless the demo server allowlists it + (`COVENANT_ALLOWED_HOSTS`). +- **Observability verified live** (compose stack): Prometheus scrapes the proxy's + `/covenant/metrics` (target `up`), and the full quarantine flip was watched through + the pipeline — `covenant_quarantined_tools` 0 → 1 on injected drift → 0 on restore, + queried both directly in Prometheus and through Grafana's provisioned datasource. + Every layer has now been exercised against real infrastructure. + +## Honest limitations (know these before someone finds them) + +- **Probes are hand-committed, not LLM-generated.** The original pitch (Project.md) had a + ReAct agent RAG-generating probe suites; what shipped is `[[probes]]` you write + yourself. Honest framing: committed probes are deterministic and side-effect-safe by + construction; generated probes are the roadmap. +- **The judge is advisory by design** — it cannot quarantine. This is a feature (no + probabilistic quarantine), but it means semantic drift alone never blocks anything. +- **Composed schemas (`$ref`/`allOf`/`anyOf`/`oneOf`) are not resolved** — changes there + flag DEGRADED for manual review. Deliberate: never silently pass what you can't parse. +- **A clean `check` means "no schema/behavior drift on what we probed"**, not "contract + safe". Coverage is exactly your probe list. +- **No OTel spans** (deferred; Prometheus only). No value-distribution fingerprints — + shape only, plus the judge at the margins. + +## Presenting it — three framings, same system + +- **Technical reviewer**: lead with the direction principle and the purity discipline + (pure `diff.py`, pure `reconcile.py`, deterministic lock). The claim is a working + severity *theory* for MCP drift plus three enforcement surfaces reusing one classifier + — not novel algorithms, novel application: OpenAPI-diff/Pact discipline ported to a + protocol that has none, with an agent-aware twist (input-loud/output-silent). +- **Pitch**: "REST got OpenAPI diffing and contract testing a decade ago; MCP — the way + every agent gets its tools now — has nothing. Covenant is the contract firewall for + MCP: it catches the drift before your agents hallucinate around it, and quarantines the + tool so they fail safe." The wow moment is demo #2: schema identical, body changed, + caught anyway. +- **Job-portfolio (platform/reliability roles)**: point at the layer table — reverse + proxy (FastAPI/async), Postgres store, Prometheus/Grafana, kopf operator + CRD + Helm + with least-privilege RBAC — each an independently shippable increment with its own + design spec under `docs/superpowers/specs/`. diff --git a/Dockerfile b/Dockerfile new file mode 100644 index 0000000..3c591c1 --- /dev/null +++ b/Dockerfile @@ -0,0 +1,14 @@ +# Covenant: proxy + operator in one image; give the full command per role. +# proxy: docker run covenant-mcp covenant proxy --upstream http://... --host 0.0.0.0 +# operator: docker run covenant-mcp kopf run -m covenant.operator.handlers --all-namespaces +FROM python:3.12-slim + +WORKDIR /app +COPY pyproject.toml README.md ./ +COPY covenant ./covenant +RUN pip install --no-cache-dir ".[proxy,operator]" + +RUN useradd --create-home --uid 1000 covenant +USER covenant + +CMD ["covenant", "--help"] diff --git a/README.md b/README.md index e42cf62..c930b1b 100644 --- a/README.md +++ b/README.md @@ -131,7 +131,7 @@ covenant proxy --upstream http://localhost:8000/mcp --port 9000 # point your MCP client at http://127.0.0.1:9000/mcp ``` -- `POST /covenant/refresh` — Covenant re-lists the upstream itself and re-checks. Detection is proxy-owned by design: a client's `tools/list` can arrive *after* the call it should have protected, so enforcement never depends on client behavior. +- `POST /covenant/refresh` — Covenant re-reads the baseline from disk (picking up a re-snapshot or an updated ConfigMap mount), then re-lists the upstream itself and re-checks. Detection is proxy-owned by design: a client's `tools/list` can arrive *after* the call it should have protected, so enforcement never depends on client behavior. - `GET /covenant/status` — currently quarantined tools and why. - `GET /covenant/calls` — recent call log with latency and outcomes. - `GET /covenant/metrics` — Prometheus metrics: per-tool call counters (ok/error/blocked), latency histograms, drift events, quarantine gauge. `docker compose up -d prometheus grafana` gives a provisioned dashboard at `http://localhost:3000` — the quarantine stat flips green→red within one scrape of a drift. @@ -151,6 +151,20 @@ covenant proxy --upstream http://localhost:8000/mcp \ Store failures are logged and never break the request path — a firewall must not drop traffic because its own telemetry hiccuped. Demo: [examples/demo_layer2.py](examples/demo_layer2.py). +## Kubernetes: the `MCPContract` operator + +Declare contract conformance instead of scripting it. The Helm chart ships the proxy, a kopf operator, and an `MCPContract` CRD — the operator re-runs the contract check on each CR's own schedule, writes the verdict into status, and nudges the proxy to quarantine on drift: + +```bash +docker build -t covenant-mcp:0.1.0 . +helm install covenant deploy/helm/covenant --set proxy.upstream=http://my-server:8000/mcp +kubectl create configmap covenant-baseline --from-file=covenant.lock.json +kubectl apply -f examples/mcpcontract.yaml +kubectl get mcpcontracts -w # RESULT flips clean -> breaking when the server drifts +``` + +Design decisions (in-operator checks vs Jobs, per-CR scheduling, error-as-status): [Layer 5 design spec](docs/superpowers/specs/2026-07-03-covenant-layer5-k8s-operator-design.md). + ## Architecture Covenant is built in dependency-ordered layers; each ships alone and each higher layer reuses the contract core. @@ -162,7 +176,7 @@ Covenant is built in dependency-ordered layers; each ships alone and each higher | 2 | Postgres contract store (call log, drift events, durable quarantine) | ✅ shipped | | 3 | Behavioral probes — response fingerprints + LLM judge for semantic drift | ✅ shipped | | 4 | Observability — Prometheus metrics + Grafana dashboard (OTel deferred) | ✅ shipped | -| 5 | K8s operator + Helm — `MCPContract` CRD, probes as Jobs | roadmap | +| 5 | K8s operator + Helm — `MCPContract` CRD, scheduled in-operator checks | ✅ shipped | Design specs for the shipped layers live in [docs/superpowers/specs](docs/superpowers/specs). @@ -170,7 +184,7 @@ Design specs for the shipped layers live in [docs/superpowers/specs](docs/superp ```bash pip install -e ".[dev]" -pytest # 112 tests; Postgres-backed tests skip without a DB +pytest # 132 tests; Postgres-backed tests skip without a DB ruff check . && mypy covenant ``` diff --git a/covenant/cli.py b/covenant/cli.py index bcc4466..5cd200f 100644 --- a/covenant/cli.py +++ b/covenant/cli.py @@ -174,7 +174,7 @@ def proxy( err.print(f"[red]error:[/red] {e}") raise typer.Exit(2) from e - fastapi_app = create_app(upstream, base_tools, store=store) + fastapi_app = create_app(upstream, base_tools, store=store, baseline_path=baseline) persistence = "postgres" if store else "in-memory (no persistence)" console.print(f"[green]Covenant proxy[/green] guarding [cyan]{upstream}[/cyan] " f"at [cyan]http://{host}:{port}/mcp[/cyan]") diff --git a/covenant/contract.py b/covenant/contract.py index 6a5a316..dfa9657 100644 --- a/covenant/contract.py +++ b/covenant/contract.py @@ -90,12 +90,22 @@ def write_baseline( def read_baseline(path: str | Path) -> tuple[str, list[JsonDict], list[JsonDict]]: """Read a baseline file; return (server, wire-shape tool dicts, probe records).""" p = Path(path) - if not p.exists(): - raise BaselineError(f"baseline not found: {p} (run `covenant snapshot` first)") try: - data = json.loads(p.read_text(encoding="utf-8")) + text = p.read_text(encoding="utf-8") + except (OSError, UnicodeDecodeError) as e: + raise BaselineError( + f"cannot read baseline: {p} ({e}) - run `covenant snapshot` first") from e + return parse_baseline(text, source=str(p)) + + +def parse_baseline(text: str, source: str) -> tuple[str, list[JsonDict], list[JsonDict]]: + """Parse baseline JSON text (a file or a ConfigMap value) into wire-shape parts.""" + try: + data = json.loads(text) except json.JSONDecodeError as e: - raise BaselineError(f"baseline is not valid JSON: {p} ({e})") from e + raise BaselineError(f"baseline is not valid JSON: {source} ({e})") from e + if not isinstance(data, dict): + raise BaselineError(f"baseline is not a JSON object: {source}") tools = [ { diff --git a/covenant/introspect.py b/covenant/introspect.py index 41fe81b..7cef518 100644 --- a/covenant/introspect.py +++ b/covenant/introspect.py @@ -82,7 +82,8 @@ def _resolve_result(result: Any) -> tuple[object, bool, str | None]: return None, False, None -async def _introspect(config: Config) -> list[JsonDict]: +async def introspect_async(config: Config) -> list[JsonDict]: + """List the server's tools in MCP wire shape (async; the proxy reuses this).""" async with _session(config) as session: result = await session.list_tools() return [_tool_to_dict(t) for t in result.tools] @@ -110,7 +111,7 @@ async def _run_probes(config: Config, probes: list[Probe]) -> list[JsonDict]: def introspect(config: Config) -> list[JsonDict]: """Introspect the configured server; return wire-shape tool dicts.""" try: - return asyncio.run(_introspect(config)) + return asyncio.run(introspect_async(config)) except CovenantError: raise except Exception as e: # noqa: BLE001 - surface any transport failure as one clean error diff --git a/covenant/operator/__init__.py b/covenant/operator/__init__.py new file mode 100644 index 0000000..4e894c8 --- /dev/null +++ b/covenant/operator/__init__.py @@ -0,0 +1,6 @@ +"""Layer 5: the Kubernetes operator for ``MCPContract`` resources. + +``reconcile.py`` is pure logic (no kopf, no kubernetes client) — fully testable +without a cluster. ``handlers.py`` is the thin kopf glue: run it with +``kopf run -m covenant.operator.handlers``. +""" diff --git a/covenant/operator/handlers.py b/covenant/operator/handlers.py new file mode 100644 index 0000000..65ceb3f --- /dev/null +++ b/covenant/operator/handlers.py @@ -0,0 +1,79 @@ +"""kopf glue for the MCPContract operator: ``kopf run -m covenant.operator.handlers``. + +A timer fires every POLL_S per resource; ``reconcile.due`` gates it to the CR's own +``spec.intervalSeconds`` so each contract keeps its own schedule. Every failure mode +lands in ``status.result`` — the operator never crash-loops on one bad contract. +""" + +from __future__ import annotations + +import logging +from datetime import UTC, datetime +from typing import Any + +import httpx +import kopf +from kubernetes import client as k8s +from kubernetes import config as k8s_config + +from ..config import DEFAULT_BASELINE +from ..errors import CovenantError +from . import reconcile + +log = logging.getLogger("covenant.operator") + +GROUP, VERSION, PLURAL = "covenant.dev", "v1alpha1", "mcpcontracts" +POLL_S = 30 +_REFRESH_TIMEOUT_S = 10.0 +# Sync handlers share one thread pool; a check blocks its slot for up to the MCP +# transport timeout, so the default pool (cpus+4) would let a few hung servers +# starve every other CR's timer. Sized for hundreds of CRs; tunable. +_MAX_WORKERS = 32 + + +@kopf.on.startup() +def configure(settings: kopf.OperatorSettings, **_: Any) -> None: + settings.execution.max_workers = _MAX_WORKERS + try: + k8s_config.load_incluster_config() + except k8s_config.ConfigException: + k8s_config.load_kube_config() # local development against a kubeconfig + + +def _baseline_text(spec: kopf.Spec, namespace: str) -> str: + ref = spec["baselineConfigMap"] + cm = k8s.CoreV1Api().read_namespaced_config_map(ref["name"], namespace) + key = ref.get("key", DEFAULT_BASELINE) + text = (cm.data or {}).get(key) + if text is None: + raise CovenantError(f"configmap {ref['name']} has no key {key!r}") + return str(text) + + +@kopf.timer(GROUP, VERSION, PLURAL, interval=POLL_S) +def check(*, spec: kopf.Spec, status: kopf.Status, namespace: str | None, + patch: kopf.Patch, **_: Any) -> None: + interval = int(spec.get("intervalSeconds", reconcile.DEFAULT_INTERVAL_S)) + now = datetime.now(UTC) + if not reconcile.due(status.get("lastCheckTime"), interval, now): + return + + try: + # MCPContract is namespaced; kopf types namespace optional for cluster scope. + baseline = _baseline_text(spec, namespace or "default") + server_url = str(spec["server"]) # required by the CRD; guarded anyway + except Exception as e: # noqa: BLE001 - a misconfigured CR is status, not a crash-loop + patch.status.update( + reconcile.error_status(now, f"misconfigured MCPContract: {e}")) + return + + result = reconcile.check_contract(server_url, baseline, now) + patch.status.update(result) + + # Nudge the proxy to re-check and quarantine; best-effort, like store writes. + refresh_url = spec.get("proxyRefreshUrl") + if refresh_url and result.get("result") != "error": + try: + httpx.post(str(refresh_url), timeout=_REFRESH_TIMEOUT_S) + except Exception as e: # noqa: BLE001 + log.warning("proxy refresh failed (%s): %s", refresh_url, e) diff --git a/covenant/operator/reconcile.py b/covenant/operator/reconcile.py new file mode 100644 index 0000000..d7549ec --- /dev/null +++ b/covenant/operator/reconcile.py @@ -0,0 +1,64 @@ +"""Pure reconcile logic for an MCPContract: is a check due, and what did it find. + +Everything here is cluster-free — kopf and the kubernetes client stay in +``handlers.py``. A check reuses the Layer 0/3 pipeline verbatim: introspect the +server, diff against the baseline, re-run baselined probes. Errors never +propagate: a failed check becomes ``result: error`` in the CR status, because an +operator must not crash-loop on one unreachable server. +""" + +from __future__ import annotations + +from datetime import datetime + +from .._types import JsonDict +from ..config import Config, Probe +from ..contract import parse_baseline +from ..diff import diff_probes, diff_tools +from ..errors import CovenantError +from ..introspect import introspect, run_probes +from ..report import summarize + +DEFAULT_INTERVAL_S = 300 + + +def due(last_check_iso: str | None, interval_s: int, now: datetime) -> bool: + """True when the contract has never been checked or its interval has elapsed.""" + if not last_check_iso: + return True + try: + last = datetime.fromisoformat(last_check_iso) + except ValueError: + return True # unreadable timestamp: re-check rather than stall forever + if last.tzinfo is None: + return True # naive timestamp (not ours): unusable against an aware now + return (now - last).total_seconds() >= interval_s + + +def error_status(now: datetime, message: str) -> JsonDict: + """Status patch for a failed check. Counts are zeroed explicitly: kopf applies + status as a JSON merge patch, so omitting them would leave a previous check's + counts on display next to ``result: error``.""" + return {"lastCheckTime": now.isoformat(), "result": "error", "message": message, + "breaking": 0, "degraded": 0, "compatible": 0} + + +def check_contract(server_url: str, baseline_text: str, now: datetime) -> JsonDict: + """Run one contract check; always return a status patch, never raise.""" + try: + _, base_tools, base_probes = parse_baseline(baseline_text, source="configmap") + cfg = Config(server_command=None, server_url=server_url, baseline_path="") + changes = diff_tools(base_tools, introspect(cfg)) + if base_probes: + probes = [Probe(tool=p["tool"], args=p.get("args") or {}) for p in base_probes] + changes += diff_probes(base_probes, run_probes(cfg, probes)) + except CovenantError as e: + return error_status(now, str(e)) + except Exception as e: # noqa: BLE001 - a malformed baseline must not crash-loop the operator + return error_status(now, f"{type(e).__name__}: {e}") + + result, counts = summarize(changes) + return { + "lastCheckTime": now.isoformat(), "result": result, **counts, + "message": "; ".join(c.message for c in changes[:5]) or "contract matches the baseline", + } diff --git a/covenant/proxy/server.py b/covenant/proxy/server.py index 0f66918..bd49c3d 100644 --- a/covenant/proxy/server.py +++ b/covenant/proxy/server.py @@ -28,6 +28,10 @@ from fastapi.responses import StreamingResponse from .._types import JsonDict +from ..config import Config +from ..contract import read_baseline +from ..errors import CovenantError +from ..introspect import introspect_async from ..store.base import Store from ..store.memory import InMemoryStore from .detect import detect @@ -82,24 +86,13 @@ async def _list_upstream(app: FastAPI) -> list[JsonDict]: lister: Lister | None = app.state.lister if lister is not None: return await lister() - from mcp import ClientSession - from mcp.client.streamable_http import streamablehttp_client - - async with ( - streamablehttp_client(app.state.upstream) as (read, write, _), - ClientSession(read, write) as session, - ): - await session.initialize() - result = await session.list_tools() - return [ - { - "name": t.name, - "description": t.description, - "inputSchema": t.inputSchema, - "outputSchema": getattr(t, "outputSchema", None), - } - for t in result.tools - ] + cfg = Config(server_command=None, server_url=app.state.upstream, baseline_path="") + return await introspect_async(cfg) + + +def _label(app: FastAPI, tool: str) -> str: + """Metric label for a tool name, clamped to the baseline set (cardinality guard).""" + return tool if tool in app.state.baseline_names else "unknown" def _is_error(resp_json: object) -> bool: @@ -115,15 +108,12 @@ async def _proxy(app: FastAPI, request: Request) -> Response: metrics: Metrics = app.state.metrics body = await request.body() - rpc = None + parsed: object = None if body: - try: - rpc = json.loads(body) - except json.JSONDecodeError: - rpc = None - method = rpc.get("method") if isinstance(rpc, dict) else None - rpc_id = rpc.get("id") if isinstance(rpc, dict) else None - params = rpc.get("params") if isinstance(rpc, dict) else None + with contextlib.suppress(json.JSONDecodeError): + parsed = json.loads(body) + rpc: JsonDict = parsed if isinstance(parsed, dict) else {} + method, rpc_id, params = rpc.get("method"), rpc.get("id"), rpc.get("params") tool = params.get("name") if isinstance(params, dict) else None # Quarantine enforcement: block a call to a flagged tool, never forward it. @@ -133,7 +123,7 @@ async def _proxy(app: FastAPI, request: Request) -> Response: f"tool unavailable - '{tool}' quarantined by Covenant " f"(contract drift: {q.reason(tool)})", ) - metrics.record_call(tool, "blocked") + metrics.record_call(_label(app, tool), "blocked") await _safe(store.record_call(tool, method, 0, True, True)) return Response(content=json.dumps(blocked), media_type="application/json") @@ -176,7 +166,7 @@ async def _passthrough() -> AsyncIterator[bytes]: if method == "tools/call" and isinstance(tool, str): is_err = _is_error(resp_json) - metrics.record_call(tool, "error" if is_err else "ok", latency_ms / 1000) + metrics.record_call(_label(app, tool), "error" if is_err else "ok", latency_ms / 1000) await _safe(store.record_call(tool, method, latency_ms, is_err, False)) return Response( @@ -193,6 +183,7 @@ def create_app( http_client: httpx.AsyncClient | None = None, lister: Lister | None = None, store: Store | None = None, + baseline_path: str | None = None, ) -> FastAPI: @asynccontextmanager async def lifespan(app: FastAPI) -> AsyncIterator[None]: @@ -214,6 +205,10 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]: app.state.lister = lister app.state.store = store or InMemoryStore() app.state.metrics = Metrics() + # Clamp the metric label to known tools: a client-supplied name must not be able + # to mint unbounded Prometheus timeseries (label-cardinality DoS). + app.state.baseline_names = {t["name"] for t in baseline_tools} + app.state.baseline_path = baseline_path @app.get("/covenant/status") async def status() -> JsonDict: @@ -225,6 +220,16 @@ async def calls(limit: int = 20) -> JsonDict: @app.post("/covenant/refresh") async def refresh() -> JsonDict: + # Re-read the baseline first: a re-snapshotted lock (or an updated ConfigMap + # mount) must not be diffed against the copy parsed at startup, or an + # intentional contract update reads as drift and quarantines a healthy tool. + if app.state.baseline_path: + try: + _, base_tools, _ = read_baseline(app.state.baseline_path) + except CovenantError as e: + raise HTTPException(status_code=500, detail=f"baseline reload failed: {e}") from e + app.state.baseline = base_tools + app.state.baseline_names = {t["name"] for t in base_tools} try: tools = await asyncio.wait_for(_list_upstream(app), timeout=_UPSTREAM_LIST_TIMEOUT) except TimeoutError as e: diff --git a/covenant/report.py b/covenant/report.py index b3c57ee..d70a519 100644 --- a/covenant/report.py +++ b/covenant/report.py @@ -18,13 +18,18 @@ _TIER_ORDER = {"breaking": 0, "degraded": 1, "compatible": 2} +def summarize(changes: list[Change]) -> tuple[str, dict[str, int]]: + """Worst tier ('clean' when none) plus per-tier counts — the one severity ladder.""" + counts = {"breaking": 0, "degraded": 0, "compatible": 0} + for c in changes: + counts[c.tier] += 1 + result = "breaking" if counts["breaking"] else "degraded" if counts["degraded"] else "clean" + return result, counts + + def exit_code(changes: list[Change], strict: bool) -> int: - tiers = {c.tier for c in changes} - if "breaking" in tiers: - return 1 - if strict and "degraded" in tiers: - return 1 - return 0 + result, _ = summarize(changes) + return 1 if result == "breaking" or (strict and result == "degraded") else 0 def to_json(changes: list[Change]) -> str: @@ -51,8 +56,8 @@ def render(changes: list[Change], strict: bool, console: Console | None = None) console.print(table) - breaking = sum(c.tier == "breaking" for c in changes) - degraded = sum(c.tier == "degraded" for c in changes) + _, counts = summarize(changes) + breaking, degraded = counts["breaking"], counts["degraded"] if breaking: console.print(f"[bold red]x {breaking} breaking change(s)[/bold red] - " "downstream agents would fail silently. Fix or quarantine.") diff --git a/covenant/store/base.py b/covenant/store/base.py index da34a5b..e8db91d 100644 --- a/covenant/store/base.py +++ b/covenant/store/base.py @@ -16,7 +16,6 @@ class Store(Protocol): async def connect(self) -> None: ... async def close(self) -> None: ... - async def set_status(self, tool: str, status: str, reason: str | None) -> None: ... async def sync_quarantine(self, breaking: dict[str, str]) -> None: ... async def load_quarantine(self) -> dict[str, str]: ... diff --git a/covenant/store/memory.py b/covenant/store/memory.py index d4422af..d425bce 100644 --- a/covenant/store/memory.py +++ b/covenant/store/memory.py @@ -6,14 +6,18 @@ from __future__ import annotations +from collections import deque + from .._types import JsonDict +_LOG_CAP = 1000 # the proxy runs indefinitely; an unbounded in-memory log is a leak + class InMemoryStore: def __init__(self) -> None: - self._status: dict[str, tuple[str, str | None]] = {} - self._calls: list[JsonDict] = [] - self._drift: list[JsonDict] = [] + self._quarantine: dict[str, str] = {} + self._calls: deque[JsonDict] = deque(maxlen=_LOG_CAP) + self._drift: deque[JsonDict] = deque(maxlen=_LOG_CAP) async def connect(self) -> None: return None @@ -21,25 +25,11 @@ async def connect(self) -> None: async def close(self) -> None: return None - async def set_status(self, tool: str, status: str, reason: str | None) -> None: - self._status[tool] = (status, reason) - async def sync_quarantine(self, breaking: dict[str, str]) -> None: - # Match PostgresStore: clear only quarantined entries, preserve any other - # statuses set via set_status, then apply the new quarantine set. - self._status = { - tool: st for tool, st in self._status.items() if st[0] != "quarantined" - } - self._status.update( - {tool: ("quarantined", reason) for tool, reason in breaking.items()} - ) + self._quarantine = dict(breaking) async def load_quarantine(self) -> dict[str, str]: - return { - tool: (reason or "") - for tool, (status, reason) in self._status.items() - if status == "quarantined" - } + return dict(self._quarantine) async def record_call( self, tool: str | None, method: str | None, latency_ms: int, is_error: bool, blocked: bool diff --git a/covenant/store/postgres.py b/covenant/store/postgres.py index 4d57c41..e711692 100644 --- a/covenant/store/postgres.py +++ b/covenant/store/postgres.py @@ -38,15 +38,6 @@ async def close(self) -> None: await self._pool.close() self._pool = None - async def set_status(self, tool: str, status: str, reason: str | None) -> None: - async with self._p.acquire() as c: - await c.execute( - """INSERT INTO tool_status (tool, status, reason, since) - VALUES ($1, $2, $3, now()) - ON CONFLICT (tool) DO UPDATE SET status = $2, reason = $3, since = now()""", - tool, status, reason, - ) - async def sync_quarantine(self, breaking: dict[str, str]) -> None: async with self._p.acquire() as c, c.transaction(): await c.execute("DELETE FROM tool_status WHERE status = 'quarantined'") diff --git a/covenant/store/schema.sql b/covenant/store/schema.sql index 471fda9..c48d5f1 100644 --- a/covenant/store/schema.sql +++ b/covenant/store/schema.sql @@ -2,7 +2,7 @@ CREATE TABLE IF NOT EXISTS tool_status ( tool TEXT PRIMARY KEY, - status TEXT NOT NULL, -- 'ok' | 'quarantined' + status TEXT NOT NULL, -- 'quarantined' (the only status Covenant writes) reason TEXT, since TIMESTAMPTZ NOT NULL DEFAULT now() ); diff --git a/deploy/helm/covenant/Chart.yaml b/deploy/helm/covenant/Chart.yaml new file mode 100644 index 0000000..3e6c5ab --- /dev/null +++ b/deploy/helm/covenant/Chart.yaml @@ -0,0 +1,6 @@ +apiVersion: v2 +name: covenant +description: MCP contract-and-drift firewall - proxy, quarantine, and the MCPContract operator +type: application +version: 0.1.0 +appVersion: "0.1.0" diff --git a/deploy/helm/covenant/templates/crd.yaml b/deploy/helm/covenant/templates/crd.yaml new file mode 100644 index 0000000..a34acb1 --- /dev/null +++ b/deploy/helm/covenant/templates/crd.yaml @@ -0,0 +1,55 @@ +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: mcpcontracts.covenant.dev +spec: + group: covenant.dev + scope: Namespaced + names: + kind: MCPContract + plural: mcpcontracts + singular: mcpcontract + shortNames: [mcpc] + versions: + - name: v1alpha1 + served: true + storage: true + additionalPrinterColumns: + - name: Result + type: string + jsonPath: .status.result + - name: Breaking + type: integer + jsonPath: .status.breaking + - name: Last Check + type: string + jsonPath: .status.lastCheckTime + schema: + openAPIV3Schema: + type: object + required: [spec] + properties: + spec: + type: object + required: [server, baselineConfigMap] + properties: + server: + type: string + description: MCP server URL to check (streamable-HTTP). + baselineConfigMap: + type: object + required: [name] + properties: + name: { type: string } + key: { type: string } # defaults in code (config.DEFAULT_BASELINE) + intervalSeconds: + type: integer # defaults in code (reconcile.DEFAULT_INTERVAL_S) + minimum: 30 + proxyRefreshUrl: + type: string + description: Optional Covenant proxy /covenant/refresh URL to nudge after each check. + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {} diff --git a/deploy/helm/covenant/templates/operator.yaml b/deploy/helm/covenant/templates/operator.yaml new file mode 100644 index 0000000..51b3b8c --- /dev/null +++ b/deploy/helm/covenant/templates/operator.yaml @@ -0,0 +1,61 @@ +{{- if .Values.operator.enabled }} +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ .Release.Name }}-operator +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: {{ .Release.Name }}-operator +rules: + # kopf resolves the CRD and (running --all-namespaces) lists namespaces at startup. + - apiGroups: [apiextensions.k8s.io] + resources: [customresourcedefinitions] + verbs: [list, watch, get] + - apiGroups: [""] + resources: [namespaces] + verbs: [list, watch, get] + - apiGroups: [covenant.dev] + resources: [mcpcontracts, mcpcontracts/status] + verbs: [list, watch, get, patch] + - apiGroups: [""] + resources: [configmaps] + verbs: [get] + - apiGroups: [""] + resources: [events] + verbs: [create] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: {{ .Release.Name }}-operator +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: {{ .Release.Name }}-operator +subjects: + - kind: ServiceAccount + name: {{ .Release.Name }}-operator + namespace: {{ .Release.Namespace }} +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ .Release.Name }}-operator + labels: { app: covenant-operator, release: {{ .Release.Name }} } +spec: + replicas: 1 + selector: + matchLabels: { app: covenant-operator, release: {{ .Release.Name }} } + template: + metadata: + labels: { app: covenant-operator, release: {{ .Release.Name }} } + spec: + serviceAccountName: {{ .Release.Name }}-operator + containers: + - name: operator + image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" + imagePullPolicy: {{ .Values.image.pullPolicy }} + command: [kopf, run, -m, covenant.operator.handlers, --all-namespaces] +{{- end }} diff --git a/deploy/helm/covenant/templates/proxy.yaml b/deploy/helm/covenant/templates/proxy.yaml new file mode 100644 index 0000000..b531dc9 --- /dev/null +++ b/deploy/helm/covenant/templates/proxy.yaml @@ -0,0 +1,46 @@ +{{- if .Values.proxy.enabled }} +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ .Release.Name }}-proxy + labels: { app: covenant-proxy, release: {{ .Release.Name }} } +spec: + replicas: 1 + selector: + matchLabels: { app: covenant-proxy, release: {{ .Release.Name }} } + template: + metadata: + labels: { app: covenant-proxy, release: {{ .Release.Name }} } + spec: + containers: + - name: proxy + image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" + imagePullPolicy: {{ .Values.image.pullPolicy }} + command: [covenant] + args: + - proxy + - --upstream={{ required "proxy.upstream is required" .Values.proxy.upstream }} + - --baseline=/covenant/{{ .Values.proxy.baselineKey }} + - --host=0.0.0.0 + - --port={{ .Values.proxy.port }} + ports: + - containerPort: {{ .Values.proxy.port }} + volumeMounts: + - name: baseline + mountPath: /covenant + readOnly: true + volumes: + - name: baseline + configMap: + name: {{ .Values.proxy.baselineConfigMap }} +--- +apiVersion: v1 +kind: Service +metadata: + name: {{ .Release.Name }}-proxy +spec: + selector: { app: covenant-proxy, release: {{ .Release.Name }} } + ports: + - port: {{ .Values.proxy.port }} + targetPort: {{ .Values.proxy.port }} +{{- end }} diff --git a/deploy/helm/covenant/values.yaml b/deploy/helm/covenant/values.yaml new file mode 100644 index 0000000..4303f50 --- /dev/null +++ b/deploy/helm/covenant/values.yaml @@ -0,0 +1,16 @@ +# Image built from the repo Dockerfile: docker build -t covenant-mcp:0.1.0 . +image: + repository: covenant-mcp + tag: "0.1.0" + pullPolicy: IfNotPresent + +proxy: + enabled: true + upstream: "" # REQUIRED: upstream MCP server URL, e.g. http://my-server:8000/mcp + port: 9000 + # Name of a ConfigMap holding the committed covenant.lock.json under this key. + baselineConfigMap: covenant-baseline + baselineKey: covenant.lock.json + +operator: + enabled: true diff --git a/docker-compose.yml b/docker-compose.yml index 43fdb6b..afa905d 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -13,7 +13,7 @@ services: POSTGRES_PASSWORD: covenant POSTGRES_DB: covenant ports: - - "5432:5432" + - "127.0.0.1:5432:5432" # loopback only - static credentials must not face the LAN volumes: - pgdata:/var/lib/postgresql/data healthcheck: @@ -25,7 +25,7 @@ services: prometheus: image: prom/prometheus:latest ports: - - "9090:9090" + - "127.0.0.1:9090:9090" # loopback only - a local dev stack must not listen on the LAN volumes: - ./deploy/prometheus.yml:/etc/prometheus/prometheus.yml:ro extra_hosts: @@ -34,10 +34,10 @@ services: grafana: image: grafana/grafana:latest ports: - - "3000:3000" + - "127.0.0.1:3000:3000" # loopback only environment: GF_AUTH_ANONYMOUS_ENABLED: "true" - GF_AUTH_ANONYMOUS_ORG_ROLE: Admin + GF_AUTH_ANONYMOUS_ORG_ROLE: Viewer # anonymous can view the provisioned dashboard, nothing more volumes: - ./deploy/grafana/provisioning:/etc/grafana/provisioning:ro - ./deploy/grafana/dashboards:/var/lib/grafana/dashboards:ro diff --git a/docs/superpowers/specs/2026-07-01-covenant-layer2-contract-store-design.md b/docs/superpowers/specs/2026-07-01-covenant-layer2-contract-store-design.md index 22d935e..ae20237 100644 --- a/docs/superpowers/specs/2026-07-01-covenant-layer2-contract-store-design.md +++ b/docs/superpowers/specs/2026-07-01-covenant-layer2-contract-store-design.md @@ -37,7 +37,6 @@ class Store(Protocol): async def close(self) -> None: ... async def record_call(self, tool, method, latency_ms, is_error, blocked) -> None: ... async def record_drift(self, tool, severity, changes: list[dict]) -> None: ... - async def set_status(self, tool, status, reason) -> None: ... async def load_quarantine(self) -> dict[str, str]: ... # tool -> reason, for restart async def recent_calls(self, limit) -> list[dict]: ... ``` diff --git a/docs/superpowers/specs/2026-07-03-covenant-layer4-observability-design.md b/docs/superpowers/specs/2026-07-03-covenant-layer4-observability-design.md index 0cd5ac6..d3406f0 100644 --- a/docs/superpowers/specs/2026-07-03-covenant-layer4-observability-design.md +++ b/docs/superpowers/specs/2026-07-03-covenant-layer4-observability-design.md @@ -41,7 +41,16 @@ real multi-hop fleet exists to trace. calls-by-outcome rate, p95 latency (`histogram_quantile` over buckets), quarantined-tools stat (green 0 / red ≥1), drift events. Compose runs Prometheus (scrapes the host-run proxy via `host.docker.internal`, 5s interval) -and anonymous-admin Grafana on :3000. +and anonymous-Viewer Grafana on :3000 — both bound to loopback only. + +## Security decisions + +- **Metric labels are clamped to baseline tool names** (else `"unknown"`): a + client-supplied tool name must not mint unbounded Prometheus timeseries + (label-cardinality DoS). The store call log is unaffected — an append-only + log records rows, not timeseries. +- Compose binds Prometheus and Grafana to `127.0.0.1`; anonymous Grafana is + Viewer, not Admin. ## Demo diff --git a/docs/superpowers/specs/2026-07-03-covenant-layer5-k8s-operator-design.md b/docs/superpowers/specs/2026-07-03-covenant-layer5-k8s-operator-design.md new file mode 100644 index 0000000..a7bfc50 --- /dev/null +++ b/docs/superpowers/specs/2026-07-03-covenant-layer5-k8s-operator-design.md @@ -0,0 +1,66 @@ +# Layer 5 — Kubernetes operator + Helm (`MCPContract` CRD) + +## Scope + +Declarative contract conformance in a cluster: an `MCPContract` CR names an MCP +server, a baseline ConfigMap, and a check interval; a kopf operator runs the +existing Layer 0/3 check on that schedule, writes the verdict into `.status` +(printer columns: `kubectl get mcpc` → RESULT / BREAKING / LAST CHECK), and +optionally POSTs the proxy's `/covenant/refresh` so quarantine follows drift. +One Helm chart ships the CRD, the proxy Deployment/Service, and the operator +Deployment with least-privilege RBAC. One Dockerfile serves both roles. + +## Named decisions + +- **In-operator checks, not Jobs.** The roadmap said "probes as Jobs"; running + each check as a Job means building status feedback from Job pods (log + scraping or per-Job RBAC to patch the CR) — heavy machinery for a check that + takes milliseconds. The operator runs the check in-process on a kopf timer. + Revisit Jobs when a check becomes long or needs isolation. +- **Purity split.** `reconcile.py` holds all logic (due-gating, check, status + shaping) with zero kopf/kubernetes imports — the whole layer unit-tests + without a cluster. `handlers.py` is glue only. +- **Per-CR scheduling via due-gating.** kopf's timer interval is fixed at + decoration time (30s poll); each CR keeps its own `spec.intervalSeconds` + (default 300) enforced by `due()` against `status.lastCheckTime`. Cheap + polls, per-contract schedules, no custom scheduler. +- **A failed check is status, not an exception.** Unreachable server, malformed + baseline, or missing ConfigMap key → `result: error` in status; the operator + never raises from the timer. (An earlier draft raised `kopf.PermanentError` + for a missing key, but kopf timers re-fire regardless and the CR showed no + status at all — error-as-status is strictly more visible.) Error patches + zero the tier counts: kopf merge-patches status, so omitted counts would + leave a previous check's numbers on display next to `result: error`. +- **Sync handlers on a sized executor.** kopf runs sync handlers in one shared + thread pool (default cpus+4); a check holds its slot for up to the MCP + transport timeout, so a few hung servers could starve every other CR's + timer. The operator pins `settings.execution.max_workers = 32`. +- **Baseline from a ConfigMap** — the same committed `covenant.lock.json`, + mounted by the proxy and read by the operator (`parse_baseline` extracted + from `read_baseline` for string input). One artifact, both consumers — and + `/covenant/refresh` re-reads it from disk before diffing, so an updated + ConfigMap (or re-snapshotted lock) takes effect without a proxy restart and + the operator and proxy cannot split-brain on which baseline is current. +- **Probes run when the baseline has them.** Probe records carry tool + args, + so the operator re-runs and diffs them with the Layer 3 pipeline — no extra + CR config. +- **RBAC is least-privilege — at kopf's floor, found on a live cluster**: + mcpcontracts (+status) list/watch/get/patch, configmaps get, events create, + plus customresourcedefinitions and namespaces list/watch/get — kopf resolves + the CRD and (under `--all-namespaces`) lists namespaces at startup, and + 403s on both before serving a single contract. Nothing else. + +## Demo + +```bash +docker build -t covenant-mcp:0.1.0 . +helm install covenant deploy/helm/covenant --set proxy.upstream=http://my-server:8000/mcp +kubectl create configmap covenant-baseline --from-file=covenant.lock.json +kubectl apply -f examples/mcpcontract.yaml +kubectl get mcpcontracts -w # RESULT flips clean -> breaking when the server drifts +``` + +## Deferred + +- Jobs-based probe execution (see above), multi-server fleets per CR, + metrics from the operator itself (the proxy already exposes drift metrics). diff --git a/examples/mcp_server.py b/examples/mcp_server.py index 8b0beb5..969060c 100644 --- a/examples/mcp_server.py +++ b/examples/mcp_server.py @@ -19,16 +19,24 @@ import os from mcp.server.fastmcp import FastMCP +from mcp.server.transport_security import TransportSecuritySettings from pydantic import BaseModel # json_response + stateless keep HTTP responses as single JSON bodies (no SSE), so # the proxy path is deterministic; these are ignored under stdio (Layer 0). +# COVENANT_ALLOWED_HOSTS extends the SDK's DNS-rebinding allowlist beyond localhost — +# needed when a cluster reaches this server via host.docker.internal (K8s demo). +_extra_hosts = [h for h in os.environ.get("COVENANT_ALLOWED_HOSTS", "").split(",") if h] mcp = FastMCP( "covenant-example-bank", host="127.0.0.1", port=int(os.environ.get("PORT", "8000")), json_response=True, stateless_http=True, + # always explicit: mcp 1.10.0 left protection OFF when settings were None + transport_security=TransportSecuritySettings( + allowed_hosts=["127.0.0.1:*", "localhost:*", *_extra_hosts], + ), ) DRIFT = os.environ.get("COVENANT_DRIFT") == "1" diff --git a/examples/mcpcontract.yaml b/examples/mcpcontract.yaml new file mode 100644 index 0000000..efa4c89 --- /dev/null +++ b/examples/mcpcontract.yaml @@ -0,0 +1,18 @@ +# Example MCPContract: the operator checks this server every 60s against the +# baseline in the covenant-baseline ConfigMap, writes the verdict into status +# (kubectl get mcpc), and nudges the proxy to quarantine on drift. +# +# kubectl create configmap covenant-baseline --from-file=covenant.lock.json +# kubectl apply -f examples/mcpcontract.yaml +# kubectl get mcpcontracts -w +apiVersion: covenant.dev/v1alpha1 +kind: MCPContract +metadata: + name: example-server +spec: + server: http://example-mcp:8000/mcp + baselineConfigMap: + name: covenant-baseline + key: covenant.lock.json + intervalSeconds: 60 + proxyRefreshUrl: http://covenant-proxy:9000/covenant/refresh diff --git a/pyproject.toml b/pyproject.toml index d5e19fe..e93c9ce 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -50,18 +50,17 @@ judge = [ "anthropic>=0.40", "google-genai>=1.0", ] +operator = [ + "kopf>=1.37", + "kubernetes>=29", + "httpx>=0.27", +] dev = [ + "covenant-mcp[proxy,store,judge,operator]", # one source of truth for the extras' pins "pytest>=8", "anyio>=4", "ruff>=0.6", "mypy>=1.11", - "fastapi>=0.115", - "uvicorn>=0.30", - "httpx>=0.27", - "prometheus-client>=0.20", - "asyncpg>=0.29", - "anthropic>=0.40", - "google-genai>=1.0", ] [tool.hatch.build.targets.wheel] @@ -80,9 +79,9 @@ strict = true warn_unused_ignores = true disallow_untyped_defs = true -# asyncpg ships no py.typed marker; ignore its missing stubs (not our code). +# asyncpg/kubernetes ship no py.typed marker; ignore their missing stubs (not our code). [[tool.mypy.overrides]] -module = "asyncpg.*" +module = ["asyncpg.*", "kubernetes.*"] ignore_missing_imports = true [tool.pytest.ini_options] diff --git a/tests/test_memory_store.py b/tests/test_memory_store.py index 2f08e2e..8a70082 100644 --- a/tests/test_memory_store.py +++ b/tests/test_memory_store.py @@ -9,16 +9,15 @@ def run(coro): return asyncio.run(coro) -def test_set_status_and_load_only_quarantined(): +def test_sync_quarantine_then_load(): s = InMemoryStore() - run(s.set_status("get_account", "quarantined", "output field 'balance_usd' removed")) - run(s.set_status("ping", "ok", None)) + run(s.sync_quarantine({"get_account": "output field 'balance_usd' removed"})) assert run(s.load_quarantine()) == {"get_account": "output field 'balance_usd' removed"} def test_sync_quarantine_replaces_the_set(): s = InMemoryStore() - run(s.set_status("a", "quarantined", "x")) + run(s.sync_quarantine({"a": "x"})) run(s.sync_quarantine({"b": "y"})) assert run(s.load_quarantine()) == {"b": "y"} diff --git a/tests/test_metrics.py b/tests/test_metrics.py index ebe02e3..fa3a0b4 100644 --- a/tests/test_metrics.py +++ b/tests/test_metrics.py @@ -84,6 +84,16 @@ async def lister(): assert 'covenant_drift_total{severity="breaking"} 1.0' in text +def test_unknown_tool_name_is_clamped_to_one_label(): + app = create_app("http://up/mcp", BASE, http_client=mock_client(ok_handler)) + client = TestClient(app) + for name in ("evil-1", "evil-2", "evil-3"): # attacker-minted names must not fan out + client.post("/mcp", json=rpc_call(name)) + text = client.get("/covenant/metrics").text + assert 'covenant_calls_total{outcome="ok",tool="unknown"} 3.0' in text + assert "evil" not in text + + def test_two_apps_do_not_share_a_registry(): a = create_app("http://up/mcp", BASE, http_client=mock_client(ok_handler)) b = create_app("http://up/mcp", BASE, http_client=mock_client(ok_handler)) diff --git a/tests/test_operator.py b/tests/test_operator.py new file mode 100644 index 0000000..fe25bdc --- /dev/null +++ b/tests/test_operator.py @@ -0,0 +1,109 @@ +"""Operator reconcile logic — pure functions, no kopf, no cluster.""" + +import json +from datetime import UTC, datetime, timedelta + +from covenant.operator import reconcile + +NOW = datetime(2026, 7, 3, 12, 0, 0, tzinfo=UTC) + + +def baseline(props): + return json.dumps({ + "covenant_version": "0.1.0", + "server": "http://up/mcp", + "tools": { + "get_account": { + "description": "d", + "inputSchema": None, + "outputSchema": {"type": "object", "properties": props}, + "schema_hash": "sha256:x", + } + }, + }) + + +def live(props): + return [{ + "name": "get_account", "description": "d", + "inputSchema": None, + "outputSchema": {"type": "object", "properties": props}, + }] + + +# --- due() ----------------------------------------------------------------- + +def test_never_checked_is_due(): + assert reconcile.due(None, 300, NOW) + + +def test_not_due_before_interval(): + last = (NOW - timedelta(seconds=100)).isoformat() + assert not reconcile.due(last, 300, NOW) + + +def test_due_after_interval(): + last = (NOW - timedelta(seconds=301)).isoformat() + assert reconcile.due(last, 300, NOW) + + +def test_garbage_timestamp_is_due(): + assert reconcile.due("not-a-date", 300, NOW) + + +def test_naive_timestamp_is_due(): + # parses fine but can't be compared with an aware now — must not raise + assert reconcile.due("2026-07-03T11:00:00", 300, NOW) + + +# --- check_contract() -------------------------------------------------------- + +def test_clean_check(monkeypatch): + monkeypatch.setattr(reconcile, "introspect", + lambda cfg: live({"balance_usd": {"type": "number"}})) + status = reconcile.check_contract( + "http://up/mcp", baseline({"balance_usd": {"type": "number"}}), NOW) + assert status["result"] == "clean" + assert status["breaking"] == 0 + assert status["lastCheckTime"] == NOW.isoformat() + + +def test_breaking_check(monkeypatch): + monkeypatch.setattr(reconcile, "introspect", + lambda cfg: live({"renamed": {"type": "number"}})) + status = reconcile.check_contract( + "http://up/mcp", baseline({"balance_usd": {"type": "number"}}), NOW) + assert status["result"] == "breaking" + assert status["breaking"] >= 1 + assert "balance_usd" in status["message"] + + +def test_unreachable_server_is_error_not_raise(monkeypatch): + def boom(cfg): + from covenant.errors import ConnectionError + raise ConnectionError("could not introspect MCP server") + + monkeypatch.setattr(reconcile, "introspect", boom) + status = reconcile.check_contract("http://down/mcp", baseline({}), NOW) + assert status["result"] == "error" + assert "could not introspect" in status["message"] + assert status["breaking"] == 0 # zeroed: a merge patch must not show stale counts + + +def test_bad_baseline_is_error_not_raise(): + status = reconcile.check_contract("http://up/mcp", "{not json", NOW) + assert status["result"] == "error" + + +def test_non_object_baseline_is_error_not_raise(): + # valid JSON, wrong shape — must land in status, not escape as AttributeError + status = reconcile.check_contract("http://up/mcp", "[]", NOW) + assert status["result"] == "error" + + +def test_malformed_probe_record_is_error_not_raise(monkeypatch): + monkeypatch.setattr(reconcile, "introspect", lambda cfg: live({})) + b = json.loads(baseline({})) + b["probes"] = [{"args": {}}] # no 'tool' key + status = reconcile.check_contract("http://up/mcp", json.dumps(b), NOW) + assert status["result"] == "error" diff --git a/tests/test_proxy.py b/tests/test_proxy.py index 06dad81..1c98b9c 100644 --- a/tests/test_proxy.py +++ b/tests/test_proxy.py @@ -4,6 +4,8 @@ assert forwarding, quarantine short-circuiting, refresh, and status precisely. """ +import json + import httpx from fastapi.testclient import TestClient @@ -109,3 +111,29 @@ async def lister(): assert client.post("/covenant/refresh").json()["quarantined"] assert client.post("/covenant/refresh").json()["quarantined"] == {} + + +def lock_text(props): + return json.dumps({ + "covenant_version": "0.1.0", "server": "http://up/mcp", + "tools": {"get_account": {"description": "d", "inputSchema": None, + "outputSchema": obj(props), "schema_hash": "sha256:x"}}, + }) + + +def test_refresh_reloads_baseline_from_disk(tmp_path): + # An intentional contract update: server changes AND the lock is re-snapshotted. + # Refresh must diff against the lock on disk, not the copy parsed at startup — + # otherwise the updated tool reads as drift and a healthy tool is quarantined. + lock = tmp_path / "covenant.lock.json" + lock.write_text(lock_text({"balance_usd": {"type": "number"}}), encoding="utf-8") + + async def lister(): + return [tool("get_account", out=obj({"balance_cents": {"type": "integer"}}))] + + app = create_app("http://up/mcp", BASE, lister=lister, baseline_path=str(lock), + http_client=mock_client(lambda req: httpx.Response(200))) + client = TestClient(app) + + lock.write_text(lock_text({"balance_cents": {"type": "integer"}}), encoding="utf-8") + assert client.post("/covenant/refresh").json()["quarantined"] == {}