From c76c9e75c49dbc4f067136a73c474dd24580fdf0 Mon Sep 17 00:00:00 2001 From: Mike Date: Sun, 15 Mar 2026 19:58:56 +0000 Subject: [PATCH] docs: comprehensive documentation update for evolution specs (spec 21) Rewrite README with competitive positioning (Mem0/Zep/Cognee/Letta/Engram), add trust scoring, entity resolution, temporal validity, briefing roles/scope, and configurable auto-linker rules across all docs. Create new concept docs (self-organizing memory, trust scoring, entity resolution), agent-type guides (coding, research, browser), and honest competitor comparison blog posts. Co-Authored-By: Claude Opus 4.6 --- ARCHITECTURE.md | 63 ++++++++- CHANGELOG.md | 33 +++++ CLAUDE.md | 13 ++ README.md | 152 ++++++++++++-------- cortex.example.toml | 20 +++ docs/blog/cortex-vs-mem0.md | 89 ++++++++++++ docs/blog/cortex-vs-zep.md | 91 ++++++++++++ docs/concepts/auto-linker.md | 83 +++++++++-- docs/concepts/briefings.md | 84 +++++++++-- docs/concepts/decay-and-memory.md | 65 +++++++-- docs/concepts/entity-resolution.md | 82 +++++++++++ docs/concepts/graph-model.md | 46 +++++- docs/concepts/hybrid-search.md | 31 +++++ docs/concepts/self-organizing-memory.md | 92 ++++++++++++ docs/concepts/trust-scoring.md | 89 ++++++++++++ docs/guides/browser-agent.md | 129 +++++++++++++++++ docs/guides/coding-agent.md | 178 ++++++++++++++++++++++++ docs/guides/multi-agent.md | 49 +++++++ docs/guides/research-agent.md | 141 +++++++++++++++++++ docs/reference/cli.md | 29 +++- docs/reference/config.md | 71 ++++++++-- docs/reference/grpc-api.md | 50 +++++++ docs/reference/http-api.md | 68 +++++++++ 23 files changed, 1637 insertions(+), 111 deletions(-) create mode 100644 docs/blog/cortex-vs-mem0.md create mode 100644 docs/blog/cortex-vs-zep.md create mode 100644 docs/concepts/entity-resolution.md create mode 100644 docs/concepts/self-organizing-memory.md create mode 100644 docs/concepts/trust-scoring.md create mode 100644 docs/guides/browser-agent.md create mode 100644 docs/guides/coding-agent.md create mode 100644 docs/guides/research-agent.md diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index edbf9834..d42d1e05 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -23,7 +23,12 @@ Cortex is a self-organizing graph memory engine built in Rust. It combines tradi **Design**: - Primary table: `nodes` (NodeId → Node) + - Includes `valid_from`, `valid_until` (temporal validity window) + - Includes `expires_at` (lifecycle GC) + - Includes `embedding_model` (vector provenance tracking) - Edge table: `edges` (EdgeId → Edge) + - Includes `metadata` (extensible HashMap for edge context) + - `EdgeProvenance::Custom` variant for forward-compatible linking mechanisms - Secondary indexes for filtering: - `nodes_by_kind` (NodeKind → Set) - `nodes_by_source` (SourceAgent → Set) @@ -121,13 +126,15 @@ score = α × vector_similarity + (1-α) × graph_proximity └─────────────────────────────────────┘ ``` -2. **Link Rules**: - - **Similarity**: Vector similarity > 0.85 → Similar edge - - **Temporal**: Events in sequence → Precedes edge - - **Source**: Same session → PartOf edge - - **Causality**: Decision before event → Causes edge - - **Support**: Fact near decision → Supports edge - - **Reference**: Title mentions → References edge +2. **Configurable Rules** (ConfigRule): + - Rules define from_kind, to_kind, relation, weight, and a condition + - Wildcard kinds (`"*"`) match any node kind + - Condition types: min_similarity, shared_tags, temporal_proximity, newer_than, same_agent, body_field_ref, tag_references_title, negation_detected + - Legacy hardcoded rules (similarity, temporal, source, causality, support, reference) are automatically disabled when config rules are defined + + **Entity features**: + - **Entity co-occurrence**: nodes from different agents referencing the same entity get `shared_entity` edges + - **Entity promotion**: periodic scan promotes entity strings referenced by 2+ agents to first-class entity nodes 3. **Contradiction Detection**: - Finds semantically similar nodes with opposite polarity @@ -188,6 +195,48 @@ score = α × vector_similarity + (1-α) × graph_proximity - Deduplicates by title+session - Event types: stage.advanced, item.completed, evidence.submitted, etc. +### Trust Engine (`cortex-core/trust`) + +**Purpose**: Compute trust scores from graph topology at query time. + +Trust is never stored as a field. It's derived from the neighbourhood of a node, like PageRank derives authority from link structure. + +**Five signals**: +1. **Corroboration** -- how many independent agents stored similar facts +2. **Contradiction penalty** -- unresolved `contradicts` edges reduce trust +3. **Source track record** -- historical accuracy of the authoring agent +4. **Access reinforcement** -- frequently retrieved, never corrected +5. **Freshness** -- recency of last access + +**Combination**: configurable weighted sum, defaults in `[trust]` config. + +**Caching**: source reliability is cached per-agent, refreshed every N auto-linker cycles. All other signals are computed per-query from live graph state. + +### Entity Layer + +**Purpose**: Cross-agent discovery via shared entity resolution. + +Entity nodes are regular nodes with `kind: "entity"` and `metadata.entity_type`. They serve as hub nodes: all knowledge about "Company X" from every agent converges on a single entity node via `references` edges. + +**Entity extraction**: `metadata.entities` array and `entity-` prefixed tags. + +**Auto-promotion**: when 2+ agents mention the same normalised entity string, the auto-linker promotes it to a first-class entity node. + +**Agent nodes**: deprecated `kind: "agent"` migrates to `kind: "entity"` with `metadata.entity_type: "agent"`. + +### Briefing Engine (`cortex-core/briefing`) + +**Purpose**: Generate structured context documents for agents. + +Briefings use a **role-based** configuration rather than hardcoded sections. Each role (identity, persistent, trackable, temporal, reviewable, superseding) defines a retrieval strategy. Kinds are mapped to roles in config. + +**Scope parameter**: +- **Agent** (default): only the requesting agent's knowledge +- **Shared**: agent's knowledge plus cross-agent context about shared entities (via two-hop entity traversal) +- **Unified**: multi-agent briefing for orchestrators, spans multiple agents + +**Trust-aware ranking**: when trust scoring is enabled, nodes are ranked by `0.6 * importance + 0.4 * trust_score`. + ## Data Flow ### Ingestion Flow diff --git a/CHANGELOG.md b/CHANGELOG.md index 5b417fe4..9ab46eb0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,39 @@ All notable changes to Cortex are documented in this file. +## [0.3.0] - YYYY-MM-DD + +### Added +- **Temporal validity** -- Node fields: `valid_from`, `valid_until` for truth windows. + Query with `valid_at()` filter. +- **Lifecycle expiry** -- Node field: `expires_at`. Retention engine auto-sweeps + expired nodes. +- **Embedding model tracking** -- Node field: `embedding_model`. Tracks which + model generated each vector for migration detection. +- **Edge metadata** -- Extensible HashMap on edges for contextual data. +- **Custom provenance** -- `EdgeProvenance::Custom` variant for forward-compatible + linking mechanisms. +- **Trust scoring** -- Compute trust from graph topology: corroboration, + contradiction, source reliability, access reinforcement, freshness. +- **Entity layer** -- Entity nodes (`kind: "entity"` + `metadata.entity_type`), + `authored_by`/`references` relations, auto-promotion from co-occurrence. +- **Briefing roles** -- Configurable mapping from node kinds to briefing roles + (identity, persistent, trackable, temporal, reviewable, superseding). +- **Briefing scope** -- Agent (default), Shared (cross-agent), Unified + (orchestrator) scope parameter. +- **Metadata query filter** -- `NodeFilter.with_metadata(key, value)` for + querying by metadata values. +- **Legacy rule deprecation** -- Hardcoded structural rules replaced by + configurable `[[auto_linker.rules]]` with wildcard kind support. +- **Metadata conventions** -- Well-known metadata keys documented for + interoperability (`entity_type`, `aliases`, `parent_agent`, `task_id`, etc). + +### Changed +- Briefing engine uses role-based config instead of hardcoded section kinds. +- Auto-linker supports entity co-occurrence and entity promotion. +- Retention engine respects `expires_at` field. +- `cortex init` accepts `--template` flag for agent-type presets. + ## [0.2.0] - 2026-03-14 ### Added diff --git a/CLAUDE.md b/CLAUDE.md index 216059ad..ab120404 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -48,6 +48,10 @@ cortex shell # Interactive REPL - `Cortex::open(path)` — library mode, no server needed - `Storage` trait — `RedbStorage` implements it - `IngestAdapter` trait — pluggable event sources +- `TrustEngine` -- computes trust from graph topology, no stored confidence +- `BriefingRoleConfig` -- maps node kinds to briefing roles (identity, persistent, trackable, temporal, reviewable, superseding) +- `BriefingScope` -- Agent (default), Shared (cross-agent), Unified (orchestrator) +- Entity convention: `kind: "entity"` + `metadata.entity_type` - Config: `cortex.toml` with `#[serde(default)]` on all structs ## Architecture decisions @@ -57,6 +61,10 @@ cortex shell # Interactive REPL - **gRPC** (tonic) for production API, **HTTP** (axum) for debug/viz - **warren-adapter** is optional (`--features warren`), cortex-core has zero network deps - Auto-linker runs background loop: similarity rules → edges, decay → prune, dedup → merge +- **Trust from topology** -- confidence is computed at query time from graph structure (corroboration, contradiction, source reliability, access, freshness). Never stored as a field. +- **Entities as convention** -- entity nodes are regular nodes with kind: "entity" and metadata.entity_type. Not a separate primitive. Relations: authored_by, references. +- **Shared graph model** -- one graph, all agents write to it. Isolation is deployment (run two instances), not data layer scoping. +- **Temporal validity** -- valid_from/valid_until for epistemic truth windows. expires_at for lifecycle GC. Distinct concerns. ## Config @@ -99,6 +107,11 @@ order, adding fields mid-struct, or removing fields **silently corrupts all exis The `test_node_schema_golden` test in `redb_storage.rs` will fail immediately if the bincode format changes without these steps being followed. +**Fields added in evolution (specs 09-10):** +- Node: `valid_from`, `valid_until`, `expires_at`, `embedding_model` (all Option, serde default None) +- Edge: `metadata` (HashMap, serde default empty) +- EdgeProvenance: `Custom { kind, detail }` variant + ## Common pitfalls - Port 9090 may conflict with existing services — override in `cortex.toml` diff --git a/README.md b/README.md index f5a6b0b4..ccc049e0 100644 --- a/README.md +++ b/README.md @@ -1,31 +1,31 @@ # Cortex -**Embedded graph memory for AI agents. One binary. One file. Zero dependencies.** +**Self-organizing graph memory for AI agents. One binary. Zero dependencies.** -Cortex is a local knowledge graph that stores what your AI agents know, automatically discovers relationships between knowledge, and synthesises context briefings on demand. Think SQLite, but for agent memory. +Cortex is an embedded temporal graph memory that auto-links knowledge, decays what's irrelevant, detects contradictions, and synthesises context briefings on demand. Think SQLite, but for agent memory. ## Why Cortex? -Your agent's memory shouldn't be a text file. It should be a living graph that wires itself, forgets what's irrelevant, and tells your agent exactly what it needs to know. +Your agents each know something. Cortex discovers what they know together. -- **Graph-native** — typed nodes and edges, not just vectors -- **Auto-linking** — relationships discovered via embedding similarity -- **Decay** — unused knowledge fades, important knowledge persists -- **Briefings** — "what do I need to know?" → tailored context document -- **Hybrid search** — vector similarity × graph proximity -- **Query DSL** — filter expressions: `kind:decision AND importance>0.7` -- **SSE Events** — real-time `GET /events/stream` for graph change notifications -- **Schema Validation** — per-kind metadata constraints with type checking -- **Mutation Hooks** — callback system for node/edge write events -- **Embedded** — single file, no external dependencies -- **Fast** — Rust, HNSW index, mmap'd storage +Most agent memory systems are glorified key-value stores with a vector index. Cortex is a self-organizing knowledge graph: it wires itself, forgets what's stale, resolves contradictions, computes trust from topology, and tells each agent exactly what it needs to know. + +**Self-organizing** -- the auto-linker runs in the background, discovering relationships via embedding similarity, shared entities, temporal proximity, and configurable structural rules. You store facts; Cortex finds the connections. + +**Temporal** -- every node carries validity windows (`valid_from`, `valid_until`) and lifecycle expiry (`expires_at`). Query "facts true on January 15th" or "facts that expired before June." Stale knowledge is filtered, not deleted. + +**Trust from topology** -- confidence is not a field you set. It's computed from the graph structure: corroboration across agents, contradiction count, source track record, access reinforcement, and freshness. Like PageRank for knowledge. + +**Briefing synthesis** -- "what does my agent need to know?" generates a structured context document from the graph. Configurable roles, agent or cross-agent scope, contradiction alerts. + +**Embedded** -- single Rust binary, redb storage (ACID, mmap), HNSW vector index. No external dependencies. No server to manage. `cargo install` and go. ## Quick Start ### Install ```bash -# Linux / macOS — one line, no Rust or protoc needed +# Linux / macOS -- one line, no Rust or protoc needed curl -sSf https://raw.githubusercontent.com/MikeSquared-Agency/cortex/main/install.sh | sh # Via Cargo @@ -47,11 +47,21 @@ cortex serve # Store some knowledge cortex node create --kind fact --title "The API uses JWT auth" --importance 0.7 +# Store with temporal validity +cortex node create --kind fact --title "Rate limit is 1000/min" \ + --valid-from 2026-01-01T00:00:00Z + # Search cortex search "authentication" # Get a briefing for your agent cortex briefing my-agent + +# Get a cross-agent briefing +cortex briefing my-agent --scope shared + +# Check trust score for a node +cortex trust ``` ### Prompt Management @@ -93,6 +103,12 @@ cx.store("decision", "Use FastAPI", body="Async + type hints", importance=0.8) results = cx.search("backend framework") print(cx.briefing("my-agent")) + +# Cross-agent briefing +print(cx.briefing("my-agent", scope="shared")) + +# Store an entity +cx.store_entity("Anthropic", entity_type="company", aliases=["anthropic-ai"]) ``` ### Embedded in Rust @@ -116,25 +132,43 @@ let results = cx.search("authentication", 5)?; - **[Prompt System](docs/concepts/prompt-system.md)** - **[Architecture](docs/concepts/architecture.md)** -## Why Not a Vector DB? - -| Feature | Cortex | Mem0 | Zep | Chroma | pgvector | -|---------|--------|------|-----|--------|----------| -| Embedded (no server) | ✅ | ❌ | ❌ | ✅ | ❌ | -| Graph relationships | ✅ native | ❌ | ❌ | ❌ | ❌ | -| Auto-linking | ✅ | ❌ | ❌ | ❌ | ❌ | -| Edge decay | ✅ | ❌ | ❌ | ❌ | ❌ | -| Contradiction detection | ✅ | ❌ | ❌ | ❌ | ❌ | -| Briefing synthesis | ✅ | ❌ | ❌ | ❌ | ❌ | -| Hybrid search (vector+graph) | ✅ | ❌ | ❌ | ❌ | ❌ | -| Local embeddings | ✅ | ❌ | ❌ | ✅ | ❌ | -| Single binary | ✅ | ❌ | ❌ | ❌ | ❌ | -| Prompt versioning | ✅ | ❌ | ❌ | ❌ | ❌ | -| Query DSL | ✅ | ❌ | ❌ | ❌ | ✅ | -| Real-time events (SSE) | ✅ | ❌ | ❌ | ❌ | ❌ | -| Schema validation | ✅ | ❌ | ❌ | ❌ | ❌ | - -**Our moat:** Graph-native memory with auto-linking and decay. Nobody else does this. +### Concepts + +- **[Self-Organizing Memory](docs/concepts/self-organizing-memory.md)** -- how Cortex discovers what your agents know together +- **[Graph Model](docs/concepts/graph-model.md)** -- nodes, edges, temporal validity, entities +- **[Trust Scoring](docs/concepts/trust-scoring.md)** -- PageRank for knowledge +- **[Entity Resolution](docs/concepts/entity-resolution.md)** -- cross-agent discovery via shared entities +- **[Briefings](docs/concepts/briefings.md)** -- role-based context synthesis with scope +- **[Auto-Linker](docs/concepts/auto-linker.md)** -- configurable rules, entity promotion +- **[Decay and Memory](docs/concepts/decay-and-memory.md)** -- edge decay vs temporal validity +- **[Metadata Conventions](docs/reference/metadata-conventions.md)** -- well-known metadata keys + +### Guides + +- **[Coding Agent](docs/guides/coding-agent.md)** -- memory for coding agents +- **[Research Agent](docs/guides/research-agent.md)** -- memory for research agents +- **[Browser Agent](docs/guides/browser-agent.md)** -- memory for browser agents +- **[Multi-Agent](docs/guides/multi-agent.md)** -- shared memory, entities, scoped briefings + +## How Cortex Compares + +| | Cortex | Mem0 | Zep/Graphiti | Cognee | Letta | Engram | +|---|---|---|---|---|---|---| +| Embedded (no server) | Yes | No | No | No | No | Yes | +| Self-organizing graph | Yes | No | Partial | Partial | No | No | +| Auto-linking | Yes | No | No | No | No | No | +| Temporal validity | Yes | No | Yes | No | No | No | +| Knowledge decay | Yes | No | No | Partial | No | Yes | +| Contradiction detection | Yes | No | Yes | No | No | No | +| Trust from topology | Yes | No | No | No | No | No | +| Briefing synthesis | Yes | No | No | No | No | No | +| Entity resolution | Yes | Yes | Yes | Yes | No | No | +| Cross-agent discovery | Yes | Yes | No | Yes | No | No | +| Single binary | Yes | No | No | No | No | Yes | +| Rust (performance) | Yes | No | No | No | No | No (Go) | +| Open source (MIT) | Yes | Partial | Partial | Yes | Yes | Yes | + +Cortex occupies a unique position: embedded AND self-organizing. Every competitor requires external infrastructure or manual curation. Cortex does neither. ## Graph Visualisation @@ -149,34 +183,34 @@ Cortex ships a live graph explorer. Start the server and open [http://localhost: ## Architecture ``` -┌────────────────────────────────────────────┐ -│ Your Application │ -│ AI Agent SDK / gRPC client │ -└─────────────────┬──────────────────────────┘ - │ -┌─────────────────▼──────────────────────────┐ -│ Cortex │ -│ gRPC :9090 HTTP :9091 │ -│ ┌──────────┐ ┌───────────┐ ┌─────────┐ │ -│ │ Storage │ │ Graph │ │ HNSW │ │ -│ │ (redb) │ │ Engine │ │ Index │ │ -│ └──────────┘ └───────────┘ └─────────┘ │ -│ ┌──────────┐ ┌───────────┐ ┌─────────┐ │ -│ │Auto-Link │ │ Briefing │ │ Ingest │ │ -│ │(background)│ │ Engine │ │ Pipeline│ │ -│ └──────────┘ └───────────┘ └─────────┘ │ -│ ┌──────────┐ ┌───────────┐ ┌─────────┐ │ -│ │ Prompt │ │ Selection │ │ Observe │ │ -│ │ Resolver│ │ Engine │ │ Scoring │ │ -│ └──────────┘ └───────────┘ └─────────┘ │ -└────────────────────────────────────────────┘ +┌────────────────────────────────────────────────┐ +│ Your Application │ +│ AI Agent(s) SDK / gRPC / MCP │ +└──────────────────┬─────────────────────────────┘ + │ +┌──────────────────▼─────────────────────────────┐ +│ Cortex │ +│ gRPC :9090 HTTP :9091 │ +│ ┌───────────┐ ┌────────────┐ ┌──────────┐ │ +│ │ Storage │ │ Graph │ │ HNSW │ │ +│ │ (redb) │ │ Engine │ │ Index │ │ +│ └───────────┘ └────────────┘ └──────────┘ │ +│ ┌───────────┐ ┌────────────┐ ┌──────────┐ │ +│ │ Auto-Link │ │ Briefing │ │ Trust │ │ +│ │ + Entities │ │ + Scope │ │ Engine │ │ +│ └───────────┘ └────────────┘ └──────────┘ │ +│ ┌───────────┐ ┌────────────┐ ┌──────────┐ │ +│ │ Prompt │ │ Selection │ │ Retention│ │ +│ │ Resolver │ │ Engine │ │ + Expiry │ │ +│ └───────────┘ └────────────┘ └──────────┘ │ +└────────────────────────────────────────────────┘ ``` ## Integration Guides -- **[LangChain](docs/guides/langchain.md)** — Use Cortex as a LangChain memory backend -- **[CrewAI](docs/guides/crewai.md)** — Share memory across a multi-agent team -- **[OpenClaw / Warren](docs/guides/openclaw.md)** — Native integration with Warren +- **[LangChain](docs/guides/langchain.md)** -- Use Cortex as a LangChain memory backend +- **[CrewAI](docs/guides/crewai.md)** -- Share memory across a multi-agent team +- **[OpenClaw / Warren](docs/guides/openclaw.md)** -- Native integration with Warren ## Examples @@ -195,4 +229,4 @@ See [CONTRIBUTING.md](CONTRIBUTING.md). All contributions welcome. ## License -MIT — see [LICENSE](LICENSE). +MIT -- see [LICENSE](LICENSE). diff --git a/cortex.example.toml b/cortex.example.toml index 291a4c6d..0a57dfa6 100644 --- a/cortex.example.toml +++ b/cortex.example.toml @@ -29,6 +29,8 @@ enabled = true interval_seconds = 60 # How often to run similarity_threshold = 0.75 # Min cosine similarity for auto-edges max_edges_per_node = 20 # Cap outgoing similarity edges +entity_promote_every_n_cycles = 60 # How often to run entity promotion (cycles) +entity_promote_min_agents = 2 # Min agents referencing an entity before promotion # Default structural rules (equivalent to legacy hardcoded rules). # Remove or modify these for non-personal-assistant agents. @@ -82,6 +84,24 @@ relation = "supersedes" weight = 0.9 condition = { type = "newer_than" } +# ─── Trust Scoring ─────────────────────────────────────── +# Trust is computed from graph topology at query time (like PageRank for knowledge). +# These settings tune the five trust signals. +[trust] +corroboration_saturation = 3 # Max independent agents before corroboration maxes out +corroboration_min_weight = 0.6 # Minimum edge weight to count as corroboration +contradiction_weight = 0.3 # How much an unresolved contradiction reduces trust +access_saturation = 20 # Access count at which reinforcement maxes out +freshness_halflife = 90 # Days until freshness signal halves + +# Relative importance of each trust signal. Must sum to 1.0. +[trust.weights] +corroboration = 0.30 +contradiction = 0.25 +source = 0.20 +access = 0.15 +freshness = 0.10 + # ─── Embedding ──────────────────────────────────────────── [embedding] # model = "all-MiniLM-L6-v2" # Local embedding model diff --git a/docs/blog/cortex-vs-mem0.md b/docs/blog/cortex-vs-mem0.md new file mode 100644 index 00000000..f9b45f3d --- /dev/null +++ b/docs/blog/cortex-vs-mem0.md @@ -0,0 +1,89 @@ +# Cortex vs Mem0 + +Both Cortex and Mem0 solve agent memory. They take fundamentally different approaches. + +Cortex is an embedded temporal graph engine that self-organizes knowledge. Mem0 is a memory layer backed by external vector and graph databases with a managed cloud offering. + +This is an honest comparison to help you decide which fits your architecture. + +## What Mem0 Does Well + +Mem0 has earned its position in the agent memory space: + +- **Established ecosystem.** 47.8K GitHub stars and an active community. When you hit a problem, someone has probably solved it before. The integrations list is long and growing. + +- **Managed cloud offering.** Mem0 Platform handles infrastructure for you. No databases to provision, no scaling to worry about. You get an API key and start storing memories immediately. + +- **Built-in entity extraction.** Mem0 pulls entities from conversations automatically and stores them as structured memories. You pass in raw text and get back organized facts. + +- **Multi-user memory isolation.** First-class support for separating memories by user, session, or agent. This is well-tested in production across many tenants. + +- **Simple API.** `m.add()` and `m.search()` get you surprisingly far. The learning curve is minimal. Most developers are productive within minutes, not hours. + +These are genuine strengths. If they match your requirements, Mem0 is a solid choice. + +## What Cortex Does Differently + +Cortex makes a different set of trade-offs: + +- **Embedded.** Cortex is a single binary with no external dependencies. It uses redb, an embedded ACID database with zero-copy mmap reads. Mem0 requires a vector database (Qdrant, Pinecone, ChromaDB, or others) and optionally a graph database (Neo4j). For many agent deployments, that infrastructure overhead is unnecessary. + +- **Self-organizing.** Cortex does not just store knowledge -- it discovers structure. The auto-linker runs continuously, applying configurable rules to create edges between nodes based on semantic similarity, temporal proximity, shared tags, and other conditions. Mem0 stores memories as independent records. Relationships between them are not discovered automatically. + +- **Temporal validity.** Every Cortex node carries a truth window: `valid_from` and `valid_until`. You can query what was true at any point in time. "What did we know about this customer in Q3?" is a first-class query. Mem0 does not model temporal truth. A memory is either present or deleted. + +- **Trust from topology.** Cortex computes confidence scores from graph structure. When multiple independent sources corroborate a fact, trust increases. When sources contradict each other, trust decreases and contradictions are surfaced. Mem0 does not have a trust model -- all memories are treated equally regardless of how well-supported they are. + +- **Briefing synthesis.** Cortex generates structured context documents (briefings) tailored to a specific agent's role and scope. Instead of returning a list of search results, it produces a coherent summary of what an agent needs to know right now. Mem0 returns ranked search results that the calling agent must interpret and assemble into context. + +- **Contradiction detection.** Cortex identifies conflicting knowledge and surfaces it explicitly. If two nodes assert incompatible facts, the graph marks the contradiction and exposes it in briefings. Mem0 can store contradictory memories side by side without detection. + +## Quick Reference + +| Capability | Cortex | Mem0 | +|---|---|---| +| Deployment model | Embedded single binary | Cloud service or self-hosted with external DBs | +| Storage backend | redb (embedded, ACID) | Qdrant, Pinecone, ChromaDB, etc. | +| Auto-linking | Configurable rules, continuous | Not available | +| Temporal validity | `valid_from` / `valid_until` per node | Not available | +| Trust scoring | Graph topology-based | Not available | +| Contradiction detection | Built-in | Not available | +| Briefing synthesis | Role-based, configurable scope | Not available (search results only) | +| Multi-tenant isolation | Via separate data directories | First-class, built-in | +| Managed cloud | Not available | Mem0 Platform | +| Entity extraction | Via auto-linker promotion rules | Built-in from conversations | + +## When to Choose Mem0 + +Mem0 is the better choice when: + +- You want a managed cloud service with no infrastructure to operate. +- You need multi-tenant memory isolation that has been battle-tested in production. +- You prefer a larger ecosystem and community for support and integrations. +- Your use case is straightforward: store memories from conversations, retrieve them later. +- You are already running a vector database and want to add memory on top of it. +- You need to get started quickly and `m.add()` / `m.search()` covers your requirements. + +## When to Choose Cortex + +Cortex is the better choice when: + +- You want embedded deployment with no external dependencies. One binary, one data directory. +- Your agents need self-organizing memory that discovers relationships between facts without explicit extraction pipelines. +- You need temporal validity. Your domain requires knowing when facts were true, not just what is true now. +- You want trust scoring derived from graph topology, not just recency or similarity ranking. +- You need briefing synthesis -- structured context documents, not raw search results. +- You want to run on-premise or air-gapped with zero network dependencies. Cortex-core has no network code. +- You need contradiction detection to surface conflicting knowledge rather than silently storing both sides. + +## Can You Use Both? + +They are not mutually exclusive. A reasonable architecture uses Cortex as the local agent memory engine -- fast, embedded, self-organizing -- and Mem0 as a shared cloud memory layer for cross-agent or cross-session knowledge that needs to be centrally managed. + +Cortex handles the graph intelligence: linking, trust, temporal queries, briefings. Mem0 handles the shared persistence and multi-tenant isolation. Each does what it does best. + +The choice is not "which memory system" but "which layer of your memory architecture does each serve." + +--- + +*Last updated: March 2026. If anything here is inaccurate or outdated, open an issue. We want this comparison to be fair.* diff --git a/docs/blog/cortex-vs-zep.md b/docs/blog/cortex-vs-zep.md new file mode 100644 index 00000000..efd5dbdb --- /dev/null +++ b/docs/blog/cortex-vs-zep.md @@ -0,0 +1,91 @@ +# Cortex vs Zep (Graphiti) + +Zep's Graphiti is the closest architectural peer to Cortex. Both are temporal knowledge graphs designed for AI agents. Both model entities, relationships, and time. + +The differences are in deployment model, intelligence layer, and philosophy. This comparison is honest about where each excels. + +## What Zep Does Well + +Zep and its Graphiti framework have significant strengths: + +- **Temporal knowledge graph backed by research.** Graphiti has a published arXiv paper detailing its temporal knowledge graph approach. The academic foundation is solid and the design decisions are well-documented. + +- **94.8% Dialogue Memory Recall.** Zep reports a 94.8% score on the DMR benchmark, which measures how well a system recalls facts from prior conversations. That is a strong result and they publish their methodology. + +- **Enterprise-ready.** Zep Cloud is a managed offering with support, SLAs, and the operational maturity that enterprise customers require. If you need a vendor relationship and uptime guarantees, this matters. + +- **Proven graph database backends.** Zep uses Neo4j or FalkorDB under the hood. These are mature, well-understood graph databases with large ecosystems, extensive tooling, and known scaling characteristics. + +- **Episodic and semantic memory distinction.** Graphiti separates raw episodes (what happened) from semantic entities and relationships (what it means). This is a clean conceptual model that maps well to how conversations generate knowledge. + +These are real advantages, particularly for teams that need enterprise support or already operate Neo4j infrastructure. + +## What Cortex Does Differently + +Cortex makes fundamentally different architectural choices: + +- **Embedded.** Cortex uses redb, an embedded ACID database with zero-copy mmap. There is no external database to deploy or manage. Zep requires Neo4j or FalkorDB, which means running a separate database process, managing its storage, and handling its failure modes. For many agent deployments, that operational overhead is not justified by the workload. + +- **Self-organizing.** Cortex's auto-linker discovers relationships automatically through configurable rules. You define conditions -- semantic similarity thresholds, temporal proximity windows, shared metadata tags -- and the auto-linker continuously creates and maintains edges. Graphiti relies on an explicit extraction pipeline: LLM calls extract entities and relationships from episodes. This works well but couples the graph structure to the quality and cost of those LLM calls. + +- **Trust from topology.** Cortex computes trust scores from graph structure. Corroboration from independent sources increases confidence. Contradictions decrease it. Source reliability propagates through edges. Zep does not have a trust model. All extracted entities and relationships are treated with equal confidence regardless of how many sources support them. + +- **Briefing synthesis with roles.** Cortex generates role-based briefings with configurable scope. An agent requests a briefing for its role and receives a structured context document -- not search results, but synthesised knowledge relevant to its current task. Briefing scope can include cross-agent knowledge or restrict to a single agent's graph. Zep provides search and retrieval, but the calling agent must assemble context from the results itself. + +- **Single binary.** `cargo install cortex-memory` and you are running. No Docker Compose file, no database provisioning, no connection strings. The entire engine -- storage, graph, vector search, auto-linker -- runs in-process. Zep's quickstart involves standing up multiple services. + +- **Configurable auto-linker rules.** Cortex lets you define linking rules declaratively: match conditions, edge types, weight functions. You control how knowledge self-organizes. Zep's relationship discovery is built into the entity extraction pipeline and is less configurable at the graph level. + +## Quick Reference + +| Capability | Cortex | Zep / Graphiti | +|---|---|---| +| Deployment model | Embedded single binary | Service (Neo4j/FalkorDB + Zep) | +| Storage backend | redb (embedded, ACID) | Neo4j or FalkorDB | +| Relationship discovery | Auto-linker with configurable rules | LLM extraction pipeline | +| Temporal model | `valid_from` / `valid_until` per node | Temporal edges with timestamps | +| Trust scoring | Graph topology-based | Not available | +| Briefing synthesis | Role-based, configurable scope | Not available (search only) | +| Contradiction detection | Built-in | Not available | +| Dialogue recall benchmark | Not published | 94.8% DMR | +| Enterprise cloud | Not available | Zep Cloud with SLAs | +| Episodic memory | Nodes with metadata | First-class episode type | + +## When to Choose Zep + +Zep is the better choice when: + +- You need enterprise support, SLAs, and a managed cloud offering. +- You already run Neo4j or FalkorDB and want to build on that infrastructure. +- You need the highest possible dialogue recall and the DMR benchmark matters to your evaluation. +- You want a managed service where extraction, storage, and retrieval are handled for you. +- Your team has experience with graph databases and prefers explicit schema control. + +## When to Choose Cortex + +Cortex is the better choice when: + +- You want embedded deployment with no external databases. One process, one data directory, zero network dependencies. +- You need self-organizing memory that discovers relationships through configurable rules rather than LLM extraction pipelines. +- You want trust scoring derived from graph topology -- corroboration, contradiction, and source reliability. +- You need configurable briefing roles and cross-agent scope for multi-agent systems. +- You prefer a single binary over a multi-service architecture. +- You want to run on-premise or in air-gapped environments where external database services are not available. + +## Architectural Comparison + +The fundamental difference is this: Zep is a service. Cortex is an engine. + +Zep's pipeline is **extract, store, query**. Episodes go in, an LLM extracts entities and relationships, they are stored in a graph database, and you query them later. The intelligence is in the extraction step. + +Cortex's pipeline is **store, auto-organize, synthesise**. Knowledge goes in as nodes, the auto-linker discovers structure continuously, trust propagates through the graph, and briefings synthesise what matters. The intelligence is in the graph itself. + +This leads to different cost profiles. Zep's extraction step requires LLM calls per episode, which adds latency and cost proportional to input volume. Cortex's auto-linker runs locally with no LLM dependency, though you can optionally use LLMs for embedding generation. + +Zep is better if you need a production-grade cloud service with enterprise support and proven dialogue recall benchmarks. Cortex is better if you want embedded intelligence that self-organizes, computes trust, and generates briefings -- all without leaving the process. + +Neither approach is universally superior. They reflect different beliefs about where intelligence should live in an agent memory system. + +--- + +*Last updated: March 2026. If anything here is inaccurate or outdated, open an issue. We want this comparison to be fair.* diff --git a/docs/concepts/auto-linker.md b/docs/concepts/auto-linker.md index 9d6bab53..40830130 100644 --- a/docs/concepts/auto-linker.md +++ b/docs/concepts/auto-linker.md @@ -6,11 +6,74 @@ The auto-linker is a background process that automatically discovers relationshi Every `interval_seconds` (default: 60), the auto-linker: -1. **Processes the backlog** — newly created or modified nodes since the last cycle -2. **Similarity scan** — for each backlog node, searches the vector index for similar nodes -3. **Edge creation** — creates `similar_to` edges for pairs above `similarity_threshold` -4. **Dedup** — removes redundant or conflicting edges -5. **Contradiction detection** — identifies nodes that contradict existing knowledge and creates `contradicts` edges +1. **Processes the backlog** -- newly created or modified nodes since the last cycle +2. **Similarity scan** -- for each backlog node, searches the vector index for similar nodes +3. **Edge creation** -- creates `similar_to` edges for pairs above `similarity_threshold` +4. **Structural rules** -- applies configurable rules (see below) to create typed edges +5. **Contradiction detection** -- identifies nodes that contradict existing knowledge and creates `contradicts` edges +6. **Entity co-occurrence** -- when two nodes from different agents reference the same entity, creates a `shared_entity` edge +7. **Entity promotion** -- when an entity string appears across 2+ agents, promotes it to a first-class entity node (periodic, configurable interval) +8. **Dedup** -- removes redundant or conflicting edges + +## Configurable Rules + +Rules define how the auto-linker creates typed edges beyond basic similarity. Each rule specifies source/target kinds, the relation to create, and a condition that must be met. + +```toml +[[auto_linker.rules]] +name = "decision-leads-to-event" +from_kind = "decision" +to_kind = "event" +relation = "led_to" +weight = 0.8 +condition = { type = "temporal_proximity", window_minutes = 60 } +``` + +### Wildcard Kinds + +Use `"*"` to match any kind: + +```toml +[[auto_linker.rules]] +name = "same-agent-linking" +from_kind = "*" +to_kind = "*" +relation = "related_to" +weight = 0.6 +condition = { type = "same_agent" } +``` + +### Condition Types + +| Type | Parameters | Description | +|------|-----------|-------------| +| `min_similarity` | `threshold` | Vector similarity above threshold | +| `shared_tags` | `min_shared` | Minimum shared tags between nodes | +| `temporal_proximity` | `window_minutes` | Created within N minutes of each other | +| `newer_than` | -- | Source node is newer than target | +| `same_agent` | -- | Both nodes from the same agent | +| `body_field_ref` | `field` | Body contains a reference to target's field | +| `body_field_contains` | `field`, `value` | Body field contains a specific value | +| `tag_references_title` | -- | Source has a tag matching target's title | +| `negation_detected` | -- | Source negates target's content | + +### Legacy Rule Deprecation + +The hardcoded structural rules (causality, temporal sequencing, support detection, reference matching) have been replaced by configurable rules. When any `[[auto_linker.rules]]` are defined in config, legacy rules are automatically disabled. + +To replicate legacy behaviour, use the default rules in `cortex.example.toml`. To customise, modify or remove rules as needed. + +## Entity Features + +### Entity Co-occurrence + +When two nodes from different agents reference the same entity (via `entity-` prefixed tags or `metadata.entities`), the auto-linker creates a `shared_entity` edge between them with the entity name in edge metadata. + +### Entity Promotion + +Every `entity_promote_every_n_cycles` cycles (default: 60), the auto-linker scans for entity strings referenced by `entity_promote_min_agents` or more agents (default: 2). Matching strings are promoted to first-class entity nodes with `kind: "entity"` and `metadata.entity_type`. + +See [Entity Resolution](./entity-resolution.md) for the full model. ## Configuration @@ -20,6 +83,8 @@ enabled = true interval_seconds = 60 similarity_threshold = 0.75 max_edges_per_node = 20 +entity_promote_every_n_cycles = 60 +entity_promote_min_agents = 2 ``` ## Manual Trigger @@ -36,16 +101,16 @@ curl -X POST http://localhost:9091/auto-linker/trigger ```bash cortex stats -# Shows: cycles, nodes_processed, edges_created, edges_pruned +# Shows: cycles, nodes_processed, edges_created, edges_pruned, entities_promoted ``` ## Similarity Threshold The threshold controls how similar two nodes must be (cosine similarity over their embeddings) before an edge is created. -- `0.90+` — near-identical content only -- `0.75` — related concepts (recommended default) -- `0.60` — broad associations +- `0.90+` -- near-identical content only +- `0.75` -- related concepts (recommended default) +- `0.60` -- broad associations ## Deduplication diff --git a/docs/concepts/briefings.md b/docs/concepts/briefings.md index 4886ee23..c8b4211e 100644 --- a/docs/concepts/briefings.md +++ b/docs/concepts/briefings.md @@ -13,34 +13,88 @@ briefing = cx.briefing("my-agent") # Inject into LLM system prompt ``` -## Sections +## Roles -Briefings are composed of ordered sections. Section order matters: graph-based sections (Patterns, Goals, Unresolved) run before Active Context so they can stake claims on the most important nodes before the general-purpose section fills the remaining budget. +Briefing sections are driven by roles, not hardcoded kinds. Each role defines a retrieval strategy. Kinds are mapped to roles in `cortex.toml`: -### Identity -Agent identity node — who this agent is, its role and capabilities. +| Role | Strategy | Default kinds | +|------|----------|---------------| +| identity | Always present, first section | agent | +| persistent | Always present, standing context | preference | +| trackable | Graph traversal from agent node | goal | +| temporal | Time-windowed recent items | event | +| reviewable | Ranked by access + importance | pattern | +| superseding | Newer facts replace older ones | fact, decision | -### Goals -Active goal nodes associated with the agent. Ordered by importance. +Plus two automatic sections: **contradictions** (always runs, surfaces unresolved conflicts) and **auto-discovered** (catch-all for kinds not mapped to any role). -### Patterns -Recurring patterns from the graph — things that have been observed multiple times. +### Custom Role Mappings -### Unresolved -Knowledge marked as unresolved, contradicted, or requiring follow-up. +Override roles for your agent type: -### Active Context -Recent nodes involving this agent, ranked by `importance x recency`. This is the most general section and runs last. +```toml +# Coding agent +[briefing.roles] +identity = ["agent"] +persistent = ["constraint", "architecture-decision"] +trackable = ["task", "milestone"] +temporal = ["commit", "deployment", "incident"] +reviewable = ["pattern", "anti-pattern", "code-smell"] +superseding = ["dependency", "api-version"] +``` + +See the [Coding Agent](../guides/coding-agent.md), [Research Agent](../guides/research-agent.md), and [Browser Agent](../guides/browser-agent.md) guides for complete examples. + +## Scope + +Briefings support three scope levels: + +- **Agent** (default): only the requesting agent's knowledge +- **Shared**: agent's knowledge plus cross-agent context about shared entities +- **Unified**: multi-agent briefing for orchestrators, spans multiple agents + +```bash +# Agent scope (default) +cortex briefing kai + +# Shared scope -- includes cross-agent entity context +cortex briefing kai --scope shared + +# Unified scope -- multi-agent briefing for an orchestrator +cortex briefing --agents kai,scout,lily +``` + +API: `GET /briefing/kai?scope=shared` + +Shared scope works via two-hop entity traversal: agent -> entity -> other agents' knowledge. See [Entity Resolution](./entity-resolution.md) for how this works. + +## Trust-Aware Ranking + +When trust scoring is enabled, briefing nodes are ranked by a combination of importance (0.6 weight) and trust score (0.4 weight). Nodes with unresolved contradictions are flagged. + +Trust/importance weights are configurable in `[briefing]` config. See [Trust Scoring](./trust-scoring.md) for the full model. ## Configuration ```toml [briefing] max_tokens = 2000 -sections = ["identity", "goals", "patterns", "unresolved", "active_context"] + +[briefing.roles] +identity = ["agent"] +persistent = ["preference"] +trackable = ["goal"] +temporal = ["event"] +reviewable = ["pattern"] +superseding = ["fact", "decision"] + +# Optional section title overrides +[briefing.titles] +persistent = "Standing constraints" +temporal = "Recent activity" ``` -Remove sections you don't need. Reduce `max_tokens` for tighter context budgets. +Remove roles you don't need. Reduce `max_tokens` for tighter context budgets. ## Caching @@ -54,5 +108,7 @@ rpc GetBriefing(GetBriefingRequest) returns (BriefingResponse); message GetBriefingRequest { string agent_id = 1; uint32 max_tokens = 2; + BriefingScope scope = 3; // AGENT, SHARED, or UNIFIED + repeated string agent_ids = 4; // For UNIFIED scope } ``` diff --git a/docs/concepts/decay-and-memory.md b/docs/concepts/decay-and-memory.md index 250c18b5..ded8579f 100644 --- a/docs/concepts/decay-and-memory.md +++ b/docs/concepts/decay-and-memory.md @@ -1,24 +1,29 @@ # Decay and Memory -Cortex models the natural fading of knowledge over time using edge weight decay. +Cortex has two mechanisms for handling knowledge staleness: -## How Decay Works +1. **Edge weight decay** -- relationship strength fades over time unless reinforced +2. **Temporal validity** -- nodes carry truth windows (`valid_from`/`valid_until`) and lifecycle expiry (`expires_at`) + +These are distinct concerns. Decay is about *relevance* (how strongly connected). Temporal validity is about *truth* (when a fact was actually true). + +## Edge Weight Decay Edge weights start at 1.0 when created. Over time, weights decrease based on: -- **Time since last access** — unused edges decay faster -- **Relation type** — some relations (e.g. `fact`) decay slower than others (e.g. `similar_to`) -- **Node importance** — edges connected to high-importance nodes decay more slowly +- **Time since last access** -- unused edges decay faster +- **Relation type** -- some relations (e.g. `fact`) decay slower than others (e.g. `similar_to`) +- **Node importance** -- edges connected to high-importance nodes decay more slowly The decay function is exponential: ``` -weight(t) = weight₀ × e^(-λt) +weight(t) = weight_0 * e^(-lambda * t) ``` -Where `λ` is the decay rate (configurable per relation) and `t` is time since last access. +Where `lambda` is the decay rate (configurable per relation) and `t` is time since last access. -## Why Decay? +### Why Decay? Without decay, a graph accumulated over months becomes noisy. Old, irrelevant knowledge crowds out recent, relevant knowledge in briefings and search results. @@ -27,12 +32,52 @@ Decay ensures that: - Stale relationships fade away naturally - Important knowledge (high `importance` score) persists longer -## Reinforcement +### Reinforcement -Accessing a node or edge (via search, traversal, or briefing generation) reinforces it — resets or slows its decay. This mirrors how memory works: things you think about stay sharp, things you ignore fade. +Accessing a node or edge (via search, traversal, or briefing generation) reinforces it -- resets or slows its decay. This mirrors how memory works: things you think about stay sharp, things you ignore fade. The auto-linker also reinforces similarity edges it re-observes in each cycle. +## Temporal Validity + +Nodes can declare when their content was true: + +- `valid_from`: when the fact became true (None = always true) +- `valid_until`: when the fact stopped being true (None = still true) + +Query with: `NodeFilter::new().valid_at(Utc::now())` + +```bash +# Store a fact with temporal bounds +cortex node create --kind fact --title "Rate limit is 1000/min" \ + --valid-from 2026-01-01T00:00:00Z + +# Search for facts true at a specific time +cortex search "rate limit" --valid-at 2026-01-15T00:00:00Z +``` + +Nodes past their `valid_until` remain in the graph for historical queries but are excluded from briefings and default search results. + +## Lifecycle Expiry + +`expires_at` is distinct from `valid_until`: + +- `valid_until` = epistemic ("this stopped being true") +- `expires_at` = lifecycle ("delete this from the graph") + +Use cases: +- **Sub-agent working memory**: expires after task completion +- **GDPR compliance**: expires on data retention deadline +- **Ephemeral context**: temporary notes that auto-clean + +```bash +# Store a node that expires in 7 days +cortex node create --kind fact --title "Sprint goal: fix auth bug" \ + --expires-at 2026-03-22T00:00:00Z +``` + +The retention engine sweeps nodes past their `expires_at` during each cycle. + ## Retention Policies Hard retention limits are separate from decay. See [configuration](../getting-started/configuration.md) for `[retention]` settings. diff --git a/docs/concepts/entity-resolution.md b/docs/concepts/entity-resolution.md new file mode 100644 index 00000000..51af33fa --- /dev/null +++ b/docs/concepts/entity-resolution.md @@ -0,0 +1,82 @@ +# Entity Resolution + +Entities are the bridge between agents. When Agent A knows about "Anthropic" and Agent B knows about "Anthropic PBC", Cortex recognizes they are talking about the same thing and connects their knowledge through a shared entity node. This is how siloed agents discover what each other knows -- without direct communication. + +## Entity Nodes + +Entity nodes are regular nodes with `kind: "entity"`. They are not a separate storage primitive -- the same node, edge, and query infrastructure applies. Two well-known metadata keys define the convention: + +- **`entity_type`** (string): Classifies the entity. Well-known values: `agent`, `company`, `person`, `technology`, `project`, `location`, `product`. Custom types are allowed. +- **`aliases`** (array of strings): Alternative names for the entity, used during entity matching. + +See [Metadata Conventions](../reference/metadata-conventions.md) for the full list. + +## How Entities Are Created + +Entities enter the graph through two paths. + +**Explicit creation.** You create an entity node directly via the SDK, CLI, or API with `kind: "entity"`. This is the preferred path when you know upfront which entities matter. + +**Auto-promotion.** When two or more agents reference the same normalized entity string, the auto-linker promotes it to a first-class entity node. Entity references are extracted from two sources: + +1. **Tags** prefixed with `entity-` (e.g., `entity-anthropic`) +2. **`metadata.entities`** -- a JSON array of entity name strings + +Normalization lowercases the string, strips leading articles ("the", "a", "an"), and joins words with hyphens. "The Company X" and "company-x" resolve to the same entity. + +Promotion is configurable: + +- `entity_promote_every_n_cycles` (default: 60) -- how often the promotion scan runs, in auto-linker cycles. +- `entity_promote_min_agents` (default: 2) -- minimum distinct agents referencing an entity before promotion. + +When an entity is promoted, the auto-linker creates `references` edges from every mentioning node to the new entity node. + +## Relations + +Three relation types connect entities to the rest of the graph: + +| Relation | Direction | Meaning | +|----------|-----------|---------| +| `references` | knowledge node -> entity | This node mentions or is about this entity | +| `shared_entity` | node -> node | Two nodes from different agents reference the same entity | +| `authored_by` | entity (agent) -> knowledge node | The agent entity that created this knowledge | + +The `shared_entity` relation is created by the auto-linker's entity co-occurrence rule. It only fires across different agents -- same-agent co-occurrences are not flagged as cross-agent discoveries. + +## Cross-Agent Discovery (Two-Hop Traversal) + +The real power of entity nodes is two-hop traversal. Starting from any node, you can follow `references` edges to reach entity nodes, then follow other agents' `references` edges back to their knowledge. The path looks like: + +``` +Agent A's node --[references]--> Entity <--[references]-- Agent B's node +``` + +This is how briefings with `scope=shared` work. The briefing engine follows reference edges to entities, then follows other agents' reference edges back to discover cross-agent knowledge. Entities are the hub nodes that connect siloed information -- like a Palantir graph where everything links through the entities. + +## Agent Nodes + +The deprecated `kind: "agent"` migrates to `kind: "entity"` with `entity_type: "agent"`. Agent identity nodes are just entity nodes. The migration function (`migrate_agent_to_entity`) converts the kind and adds an `entity-type-agent` tag. Existing `kind: "agent"` nodes continue to work but will be migrated automatically during auto-linker cycles. + +## Configuration + +```toml +[auto_linker] +entity_promote_every_n_cycles = 60 # How often to scan for promotable entities +entity_promote_min_agents = 2 # Min agents before auto-promotion triggers +``` + +Lower `entity_promote_min_agents` to 1 if you want single-agent entities (useful for solo-agent setups). Increase `entity_promote_every_n_cycles` if promotion scans are too expensive on large graphs. + +## CLI + +```bash +# Create an entity node +cortex node create --kind entity --title "Anthropic" \ + --metadata '{"entity_type": "company", "aliases": ["anthropic-ai", "Anthropic PBC"]}' + +# List all entity nodes +cortex node list --kind entity + +# Find nodes referencing a specific entity +cortex search "Anthropic" --kind entity +``` diff --git a/docs/concepts/graph-model.md b/docs/concepts/graph-model.md index df0c6a83..c94ba9da 100644 --- a/docs/concepts/graph-model.md +++ b/docs/concepts/graph-model.md @@ -17,6 +17,10 @@ A node represents a discrete piece of knowledge. | `source.agent` | string | Which agent created this node | | `created_at` | timestamp | Creation time | | `metadata` | map | Arbitrary key-value pairs | +| `valid_from` | Option\ | When this fact became true in the real world | +| `valid_until` | Option\ | When this fact stopped being true | +| `expires_at` | Option\ | When to garbage-collect this node (lifecycle expiry) | +| `embedding_model` | Option\ | Which embedding model generated the vector | ### Node Kinds @@ -31,7 +35,8 @@ Node kinds are validated lowercase strings. Built-in kinds: | `observation` | A performance observation recording interaction outcomes | | `pattern` | A recurring behaviour or structure | | `preference` | A stated preference | -| `agent` | An agent identity node | +| `agent` | An agent identity node (migrating to `entity` with `entity_type: "agent"`) | +| `entity` | A resolved entity (person, company, technology, etc.) | | `prompt` | A versioned prompt template with sections and metadata | Custom kinds are allowed — any lowercase alphanumeric string with hyphens (e.g. `project-milestone`). @@ -48,6 +53,17 @@ An edge represents a typed, weighted relationship between two nodes. | `relation` | Relation | Relationship type (see below) | | `weight` | f32 0–1 | Relationship strength | | `created_at` | timestamp | Creation time | +| `metadata` | map | Arbitrary key-value pairs (edge context) | + +### Edge Provenance + +Each edge records how it was created: + +| Variant | Description | +|---------|-------------| +| `Manual` | Created explicitly by a user or agent | +| `AutoLinker` | Created by the auto-linker (similarity, structural rules) | +| `Custom { kind, detail }` | Forward-compatible variant for plugins and adapters | ### Relations @@ -71,9 +87,37 @@ Relations are validated lowercase strings with underscores. Built-in relations: | `informed_by` | Observation was informed by a prompt variant | | `rolled_back` | Version was rolled back due to degradation | | `rolled_back_to` | Target version of a rollback | +| `authored_by` | Agent entity authored this knowledge node | +| `references` | Knowledge node references this entity | +| `shared_entity` | Two nodes share a reference to the same entity | +| `led_to` | A decision led to an event (configurable rule) | +| `instance_of` | An observation is an instance of a pattern | Custom relations are allowed — any lowercase alphanumeric string with underscores. +## Entity Nodes + +Entity nodes are regular nodes with `kind: "entity"` and `metadata.entity_type`. They serve as hub nodes: all knowledge about "Company X" from every agent converges on a single entity node via `references` edges. + +**Entity extraction**: `metadata.entities` array and `entity-` prefixed tags. + +**Auto-promotion**: When 2+ agents mention the same normalised entity string, the auto-linker promotes it to a first-class entity node. + +**Agent nodes**: The deprecated `kind: "agent"` migrates to `kind: "entity"` with `metadata.entity_type: "agent"`. + +See [Entity Resolution](./entity-resolution.md) for the full model. + +## Temporal Validity + +Cortex supports a bi-temporal model for knowledge: + +- `valid_from` / `valid_until` -- **epistemic truth windows**. When was this fact actually true in the real world? Query with `NodeFilter::new().valid_at(timestamp)`. +- `expires_at` -- **lifecycle expiry**. When should this node be garbage-collected? The retention engine sweeps expired nodes. + +These are distinct from edge decay. Decay is about *relevance* (how strongly connected). Temporal validity is about *truth* (when a fact was actually true). See [Decay and Memory](./decay-and-memory.md) for the distinction. + +Nodes past their `valid_until` remain in the graph for historical queries but are excluded from briefings and default search results. + ## Decay Edge weights decay over time when edges are not accessed. This models the natural fading of relevance: knowledge that hasn't been touched recently becomes less strongly connected. The auto-linker reinforces edges that remain relevant by re-observing similarity. diff --git a/docs/concepts/hybrid-search.md b/docs/concepts/hybrid-search.md index 9b4eeb6a..e81b7af6 100644 --- a/docs/concepts/hybrid-search.md +++ b/docs/concepts/hybrid-search.md @@ -50,3 +50,34 @@ cortex search "authentication" --hybrid --alpha 0.7 --hops 2 ```python results = cx.search("authentication", limit=10, hybrid=True, alpha=0.7) ``` + +## Temporal Filtering + +Hybrid search accepts an optional `valid_at` parameter. When set, candidates are filtered to only include nodes that were valid at the requested time, after the vector pass but before the merge. + +This lets you ask questions like "what did we know about authentication in January?" and get only the facts that were true at that time. + +### CLI + +```bash +cortex search "authentication" --hybrid --valid-at 2025-01-15T00:00:00Z +``` + +### gRPC + +```protobuf +message HybridSearchRequest { + string query = 1; + uint32 limit = 2; + float alpha = 3; + uint32 graph_hops = 4; + google.protobuf.Timestamp valid_at = 5; // optional temporal filter +} +``` + +### Python SDK + +```python +from datetime import datetime +results = cx.search("authentication", hybrid=True, valid_at=datetime(2025, 1, 15)) +``` diff --git a/docs/concepts/self-organizing-memory.md b/docs/concepts/self-organizing-memory.md new file mode 100644 index 00000000..3219104a --- /dev/null +++ b/docs/concepts/self-organizing-memory.md @@ -0,0 +1,92 @@ +# Self-Organizing Memory + +Most agent memory systems are passive storage. You put data in, you get data out, and nothing happens in between. Cortex is different. It is an active memory engine that continuously organizes, connects, and curates the knowledge your agents produce. + +You store facts. Cortex discovers what your agents know together. + +## What Makes Memory Self-Organizing? + +Five capabilities work together in the background to turn a pile of nodes into a structured, trustworthy knowledge graph. + +### 1. Auto-Linking + +The auto-linker is a background process that runs every `interval_seconds` (default: 60 seconds). Each cycle, it processes newly created or modified nodes and discovers relationships through four mechanisms: + +- **Embedding similarity** -- cosine similarity over vector embeddings, with a configurable threshold (default: 0.75). +- **Shared entity references** -- two nodes that mention the same entity (via tags or metadata) get a `shared_entity` edge. +- **Temporal proximity** -- nodes created within a configurable time window are candidates for structural linking. +- **Configurable rules** -- declarative rules in `cortex.toml` that match on node kind, relation type, and custom conditions. + +When two nodes are sufficiently related, the auto-linker creates a typed, weighted edge between them. You do not wire the graph by hand. You store knowledge. Cortex finds the connections. + +See [Auto-Linker](./auto-linker.md) for the full configuration reference. + +### 2. Contradiction Detection + +When two nodes express conflicting information, the auto-linker creates a `contradicts` edge and flags both nodes for review. Contradictions are not buried or silently overwritten. They surface in agent briefings under the Unresolved section, so agents can reason about conflicts rather than ignoring them. + +Resolved contradictions (where one side is superseded) stop penalizing trust scores. Unresolved contradictions continue to reduce trust until an agent or operator addresses them. + +### 3. Knowledge Decay + +Edge weights start at 1.0 and fade over time unless reinforced by access. The decay function is exponential: + +``` +weight(t) = weight_0 * e^(-lambda * t) +``` + +Accessing a node through search, traversal, or briefing generation reinforces its edges -- resets or slows the decay clock. Important knowledge (high `importance` score) decays more slowly thanks to importance shielding. The result: stale associations dissolve naturally, while knowledge that agents actively use persists. This mirrors how human memory works: things you think about stay sharp, things you ignore fade. + +See [Decay and Memory](./decay-and-memory.md) for decay rates and retention policies. + +### 4. Trust from Topology + +Confidence in Cortex is not a stored field. It is computed from the graph structure at query time, drawing on five signals: corroboration across independent agents, contradiction count, source agent track record, access reinforcement, and freshness. This works like PageRank for knowledge -- authority is derived from structure, not from a static label. + +New corroboration immediately increases trust. A new contradiction immediately decreases it. No migration, no stale confidence values. + +See [Trust Scoring](./trust-scoring.md) for the combination formula and configuration. + +### 5. Entity Promotion + +When multiple agents reference the same real-world entity (a person, company, technology, or project), the auto-linker promotes it to a first-class entity node with `kind: "entity"`. Entity nodes become hub nodes in the graph, connecting all related knowledge across agents. Two agents that have never communicated can discover each other's knowledge through a shared entity. + +Promotion is configurable: `entity_promote_min_agents` (default: 2) controls how many independent agents must reference an entity before it is promoted. + +See [Entity Resolution](./entity-resolution.md) for the full entity model. + +## The Feedback Loop + +These five capabilities form a virtuous cycle. Auto-linking creates structure from unstructured knowledge. Decay removes noise from that structure. Contradiction detection surfaces conflicts that auto-linking discovers. Trust scoring ranks the surviving knowledge by reliability. Entity promotion connects knowledge across agent boundaries, feeding more context back into the next auto-linker cycle. + +Each cycle makes the graph more useful than the last. Early cycles produce broad, low-confidence associations. Over time, decay prunes the weak links, corroboration strengthens the strong ones, and entity hubs emerge as the connective tissue between agents. + +## What This Means for Your Agent + +Consider two agents: a researcher that gathers technical facts and an architect that records design decisions. Each stores knowledge independently. Neither knows the other exists. + +The auto-linker discovers that the researcher's fact about a library's performance characteristics is relevant to the architect's decision to adopt that library. It creates a `similar_to` edge. Both agents reference "PostgreSQL" in their tags, so entity promotion creates a shared entity node linking their knowledge. The architect's briefing now includes the researcher's performance data under the cross-agent section. The researcher's briefing flags a contradiction between two performance benchmarks. + +No manual wiring. No shared schema negotiation. No message passing between agents. The graph organized itself. + +This is the core value proposition of self-organizing memory: agents that share a Cortex instance automatically benefit from each other's knowledge, without any explicit coordination. + +## Further Reading + +Each self-organizing capability has its own concept page with full configuration details: + +- [Auto-Linker](./auto-linker.md) -- background linking process, rules, similarity thresholds +- [Decay and Memory](./decay-and-memory.md) -- exponential decay, reinforcement, retention policies +- [Trust Scoring](./trust-scoring.md) -- five signals, combination formula, API endpoints +- [Entity Resolution](./entity-resolution.md) -- entity nodes, auto-promotion, cross-agent discovery +- [Briefings](./briefings.md) -- how self-organized knowledge is synthesized for agents + +## Getting Started + +To see self-organizing memory in action: + +1. Follow the [Quickstart](../getting-started/quickstart.md) to set up a Cortex instance. +2. Review the `[auto_linker]` and `[trust]` sections in [`cortex.example.toml`](../../cortex.example.toml) for tuning options. +3. Store nodes from two or more agents and watch the auto-linker create structure. + +Run `cortex stats` to monitor auto-linker cycles, edges created, and entities promoted. diff --git a/docs/concepts/trust-scoring.md b/docs/concepts/trust-scoring.md new file mode 100644 index 00000000..de48fca1 --- /dev/null +++ b/docs/concepts/trust-scoring.md @@ -0,0 +1,89 @@ +# Trust Scoring + +Trust in Cortex is not a field you set. It is computed from the graph structure at query time, like PageRank derives authority from link structure. A node's trust score reflects how well-corroborated, how uncontested, and how actively used that knowledge is across the entire graph. + +## Five Signals + +Trust is composed of five independent signals, each capturing a different dimension of reliability. + +### 1. Corroboration + +How many independent agents stored similar facts. More independent sources means higher trust. Corroboration saturates at `corroboration_saturation` (default: 3) agents -- the marginal value of additional sources diminishes. Only edges with weight >= `corroboration_min_weight` (default: 0.6) count. Weak similarity edges are excluded. + +### 2. Contradiction Penalty + +Unresolved `contradicts` edges reduce trust. Each active contradiction applies a penalty scaled by `contradiction_weight` (default: 0.3). Resolved contradictions (where one side is superseded) do not count. This incentivizes agents to resolve conflicts rather than leaving them open. + +### 3. Source Track Record + +Historical accuracy of the authoring agent. Computed from the ratio of corroborated to contradicted nodes for that agent across the entire graph. An agent whose facts are frequently corroborated earns a high source score; one whose facts are frequently contradicted earns a low one. Cached per-agent, refreshed every N auto-linker cycles. + +### 4. Access Reinforcement + +Frequently retrieved nodes that are never corrected gain trust. Each access (search hit, briefing inclusion, explicit retrieval) increments the counter. Saturates at `access_saturation` (default: 20) accesses. The intuition: if agents keep retrieving a node and never contradict it, it is probably accurate. + +### 5. Freshness + +Recently created or accessed nodes receive a freshness boost. Exponential decay with `freshness_halflife` (default: 90 days). Prevents the graph from being dominated by old, possibly outdated knowledge even when that knowledge has high corroboration. + +## Combination Formula + +The five signals are combined as a weighted sum. Each signal is normalized to 0.0-1.0 before weighting: + +``` +trust = 0.30 * corroboration + + 0.25 * (1 - contradiction_penalty) + + 0.20 * source_track_record + + 0.15 * access_reinforcement + + 0.10 * freshness_boost +``` + +The weights must sum to 1.0. Override them in `[trust.weights]`. + +## Configuration + +```toml +[trust] +corroboration_saturation = 3 # Max agents before corroboration maxes out +corroboration_min_weight = 0.6 # Minimum edge weight to count as corroboration +contradiction_weight = 0.3 # Penalty per unresolved contradiction +access_saturation = 20 # Access count at which reinforcement maxes out +freshness_halflife = 90 # Days until freshness signal halves + +[trust.weights] +corroboration = 0.30 +contradiction = 0.25 +source = 0.20 +access = 0.15 +freshness = 0.10 +``` + +All values have sensible defaults. You only need to add `[trust]` to your `cortex.toml` if you want to override them. + +## API + +| Interface | Command / Endpoint | Description | +|-----------|-------------------|-------------| +| CLI | `cortex trust ` | Trust score for a single node | +| CLI | `cortex trust --agent ` | Aggregate trust metrics for an agent | +| HTTP | `GET /trust/:node_id` | Single node trust score | +| HTTP | `POST /trust/batch` | Batch scores (JSON array of node IDs) | +| HTTP | `GET /trust/agents` | Trust summary per agent | +| gRPC | `GetTrustScore` | Single node trust score | +| gRPC | `BatchGetTrustScore` | Batch trust scores | + +## Trust in Briefings + +When trust scoring is enabled, the briefing engine uses trust to rank and annotate nodes. Briefing nodes are ranked by a blended score: + +``` +briefing_rank = 0.6 * importance + 0.4 * trust_score +``` + +Nodes with unresolved contradictions are flagged in the Unresolved section of the briefing, regardless of their trust score. This ensures agents are always aware of contested knowledge. + +## Design Decision: Computed, Not Stored + +Trust is deliberately not stored as a field on nodes. Stored confidence goes stale the moment the graph changes. A new corroborating edge from another agent should immediately increase trust. A new contradiction should immediately decrease it. + +By computing trust at query time from the live graph topology, Cortex ensures that trust scores always reflect the current state of knowledge. There is no migration when the trust formula changes. There is no drift between stored scores and reality. The graph is the source of truth, and trust is a view over it. diff --git a/docs/guides/browser-agent.md b/docs/guides/browser-agent.md new file mode 100644 index 00000000..6eb2c605 --- /dev/null +++ b/docs/guides/browser-agent.md @@ -0,0 +1,129 @@ +# Browser Agent Memory + +Cortex gives browser agents a persistent memory for navigating websites, filling forms, and executing web tasks across sessions. The key capability: procedural memory. When an agent learns how to complete a checkout flow on a specific site, that procedure is stored and recalled the next time the agent visits. When the site layout changes, the old procedure is superseded, not lost. + +## Configuration + +Add a `[briefing.roles]` section to your `cortex.toml` for browser agent workflows: + +```toml +[briefing.roles] +identity = ["agent"] +persistent = ["procedure", "site-config"] +trackable = ["task", "workflow"] +temporal = ["navigation", "interaction"] +reviewable = ["selector-pattern", "failure-pattern"] +superseding = ["site-layout", "form-structure"] +``` + +Role mapping for browser agents: + +- **persistent** -- step-by-step procedures and site configurations that the agent needs every time it visits a domain. +- **trackable** -- active tasks and multi-step workflows in progress. +- **temporal** -- individual navigations and interactions, providing a session log. +- **reviewable** -- CSS selector patterns and failure patterns that surface for re-evaluation when they start breaking. +- **superseding** -- site layouts and form structures where only the latest version matters. When a site redesigns, the new layout supersedes the old one. + +## Procedural Memory + +Store step-by-step procedures for specific sites. Tag each procedure with the domain so the briefing engine can surface the right procedure when the agent visits that site: + +```python +from cortex_memory import Cortex + +cx = Cortex("localhost:9090") + +cx.store( + "procedure", + "Complete checkout on shop.example.com", + body="1. Click cart icon (selector: .cart-badge)\n" + "2. Click 'Proceed to Checkout' (selector: #checkout-btn)\n" + "3. Fill shipping form: name, address, city, postal code\n" + "4. Select shipping method (selector: input[name=shipping])\n" + "5. Click 'Place Order' (selector: .submit-order)", + importance=0.85, + source_agent="browser-agent", + tags=["shop.example.com", "checkout"], +) +``` + +When the site redesigns and selectors change, store the updated procedure. The auto-linker creates a `supersedes` edge between the old and new versions because they share the same domain tag and high semantic similarity. The briefing shows only the current procedure. + +## Cross-Domain Transfer + +Techniques learned on one site often apply to others. Cookie consent banners, CAPTCHA flows, and login patterns are structurally similar across domains. The auto-linker discovers this: + +```python +cx.store( + "selector-pattern", + "Cookie consent dismiss pattern", + body="Most cookie banners use a button with text 'Accept', " + "'Accept All', or 'I Agree'. Common selectors: " + "#accept-cookies, .cookie-accept, [data-action=accept].", + importance=0.7, + source_agent="browser-agent", + tags=["cookie-consent", "cross-domain"], +) +``` + +When the agent stores a site-specific interaction that matches this pattern, the auto-linker creates an edge linking the specific interaction to the general pattern. The next time the agent encounters a cookie banner on a new site, the briefing surfaces the general pattern as relevant context. + +## Failure Patterns + +Store failures as reviewable nodes. When the same failure appears across multiple sites, the briefing engine surfaces the pattern so the agent can develop a general workaround: + +```python +cx.store( + "failure-pattern", + "Dynamic content loading breaks click targets", + body="On 3 sites, clicking a button immediately after page load " + "failed because AJAX content shifted the layout. Fix: wait " + "for network idle or use explicit element-visible checks.", + importance=0.75, + source_agent="browser-agent", + tags=["dynamic-content", "race-condition"], +) +``` + +The reviewable role means this failure pattern resurfaces periodically in briefings, reminding the agent to apply the workaround proactively rather than waiting for the failure to recur. + +## Temporal Validity + +Site layouts change without warning. When a selector pattern breaks, mark it with `valid_until` so the briefing engine stops recommending stale procedures: + +```python +# Original selector pattern +cx.store( + "site-layout", + "shop.example.com navigation structure", + body="Main nav uses .nav-primary with dropdown menus on hover. " + "Product pages at /products/{slug}.", + importance=0.7, + source_agent="browser-agent", + tags=["shop.example.com", "navigation"], + metadata={"valid_from": "2025-11-01"}, +) + +# When the site redesigns, store the new layout and expire the old one +cx.store( + "site-layout", + "shop.example.com navigation structure (2026 redesign)", + body="Main nav moved to hamburger menu (.nav-mobile). " + "Product pages now at /shop/{category}/{slug}.", + importance=0.7, + source_agent="browser-agent", + tags=["shop.example.com", "navigation"], + metadata={"valid_from": "2026-02-15"}, +) +``` + +The briefing engine filters out nodes outside their validity window. The agent sees only the current site layout without manually cleaning up old records. + +## Quick Setup + +```bash +cortex init --template browser +cortex serve +``` + +This generates a `cortex.toml` with browser agent roles and a shorter temporal window suited to the fast-changing nature of web content. diff --git a/docs/guides/coding-agent.md b/docs/guides/coding-agent.md new file mode 100644 index 00000000..baa6b64f --- /dev/null +++ b/docs/guides/coding-agent.md @@ -0,0 +1,178 @@ +# Coding Agent Memory + +Cortex gives coding agents (Cursor, Copilot, Claude Code, or custom) a persistent memory layer that survives across sessions. The key insight: coding agents need to remember constraints, track architectural decisions, and know when those decisions get superseded by newer ones. Without this, agents rediscover the same context every session or, worse, act on stale decisions. + +## Configuration + +Add a `[briefing.roles]` section to your `cortex.toml` to tell the briefing engine how to categorize knowledge for a coding agent: + +```toml +[briefing.roles] +identity = ["agent"] +persistent = ["constraint", "architecture-decision"] +trackable = ["task", "milestone"] +temporal = ["commit", "deployment", "incident"] +reviewable = ["pattern", "anti-pattern", "code-smell"] +superseding = ["dependency", "api-version"] +``` + +Each role controls how nodes appear in the agent's briefing: + +- **identity** -- the agent's own node. Always shown at the top of a briefing. +- **persistent** -- constraints and architecture decisions that the agent must always know. These appear in every briefing, regardless of age. +- **trackable** -- active tasks and milestones. Shown while in progress, removed when completed. +- **temporal** -- time-ordered events like commits and deployments. Shown in reverse chronological order with a rolling window. +- **reviewable** -- patterns and anti-patterns that surface for periodic review. The briefing engine rotates these so the agent re-evaluates them over time. +- **superseding** -- dependencies and API versions where only the latest version matters. When a newer node supersedes an older one, the briefing shows only the current version. + +## Storing Architectural Decisions + +Store architecture decision records (ADRs) as `architecture-decision` nodes with the rationale in the body and high importance so they persist in briefings: + +```python +from cortex_memory import Cortex + +cx = Cortex("localhost:9090") + +# Store the decision +adr_id = cx.store( + "architecture-decision", + "Use PostgreSQL for transactional data", + body="We evaluated DynamoDB, CockroachDB, and PostgreSQL. " + "PostgreSQL wins on operational simplicity, tooling maturity, " + "and the team's existing expertise. DynamoDB was ruled out due " + "to vendor lock-in constraints.", + importance=0.9, + source_agent="coding-agent", + tags=["database", "adr-001"], +) + +# Link it to the constraint that drove the decision +constraint_id = cx.store( + "constraint", + "No vendor lock-in for core infrastructure", + body="All core infrastructure must be portable across cloud providers.", + importance=0.95, + source_agent="coding-agent", + tags=["infrastructure"], +) +``` + +The auto-linker detects the semantic overlap between the decision body (which mentions vendor lock-in) and the constraint, and creates an edge between them automatically. + +## Tracking Patterns + +Store observed patterns and anti-patterns as `reviewable` nodes. The briefing engine periodically resurfaces them so the agent can verify they still hold: + +```python +cx.store( + "pattern", + "Repository pattern for database access", + body="All database queries go through repository classes. " + "Direct SQL in service layers is prohibited.", + importance=0.7, + source_agent="coding-agent", + tags=["architecture", "data-access"], +) + +cx.store( + "anti-pattern", + "God service classes", + body="Service classes exceeding 500 lines indicate missing abstractions. " + "Split by bounded context.", + importance=0.7, + source_agent="coding-agent", + tags=["architecture", "maintainability"], +) +``` + +When the agent later stores an observation like "OrderService is 800 lines and growing", the auto-linker connects it to the god-service anti-pattern via semantic similarity. The next briefing surfaces both the observation and the anti-pattern together. + +## Superseding Decisions + +When a dependency changes or an architecture decision is revised, store the new version. The auto-linker creates `supersedes` edges between nodes in the `superseding` role that share tags or high semantic similarity: + +```python +# Original dependency record +cx.store( + "dependency", + "React 18 for frontend", + body="Using React 18 with concurrent features enabled.", + importance=0.8, + source_agent="coding-agent", + tags=["frontend", "react"], +) + +# Later, when upgrading: +cx.store( + "dependency", + "React 19 for frontend", + body="Upgraded to React 19. Server Components now available. " + "Removed legacy Suspense workarounds.", + importance=0.8, + source_agent="coding-agent", + tags=["frontend", "react"], +) +``` + +The briefing engine shows only "React 19 for frontend" in the superseding section. The older node is still in the graph (and reachable via search or traversal) but no longer clutters the briefing. + +## Temporal Context + +Use `valid_from` and `valid_until` metadata for facts that are true only during a specific period. The briefing engine filters out nodes outside their validity window: + +```python +cx.store( + "api-version", + "Payment API v2 is the current version", + body="v2 introduced idempotency keys and webhook signatures. " + "v1 is deprecated but still operational until 2026-06-01.", + importance=0.8, + source_agent="coding-agent", + tags=["payments", "api"], + metadata={"valid_from": "2025-09-15"}, +) +``` + +When v3 ships, store the new version with its own `valid_from` and update the v2 node's `valid_until`. The briefing engine automatically stops showing v2 and starts showing v3. + +## Example Briefing Output + +A coding agent calling `cx.briefing("coding-agent", compact=True)` with the roles above receives output structured like this: + +``` +## Identity +- coding-agent: Full-stack development assistant for the payments platform. + +## Constraints (persistent) +- No vendor lock-in for core infrastructure +- All public endpoints require idempotency keys + +## Architecture Decisions (persistent) +- Use PostgreSQL for transactional data [adr-001] +- Event sourcing for payment state transitions [adr-007] + +## Active Tasks (trackable) +- Migrate checkout flow to React 19 Server Components [in-progress] + +## Recent Events (temporal) +- 2026-03-14: Deployed payment-service v2.4.1 +- 2026-03-12: Incident — webhook retry storm (resolved) + +## Patterns for Review (reviewable) +- Repository pattern for database access +- Anti-pattern: God service classes + +## Current Versions (superseding) +- React 19 for frontend (supersedes React 18) +- Payment API v2 is the current version +``` + +## Quick Setup + +```bash +cortex init --template coding +cortex serve +``` + +This generates a `cortex.toml` pre-configured with the coding agent roles shown above. Start the server, connect your agent, and begin storing decisions. diff --git a/docs/guides/multi-agent.md b/docs/guides/multi-agent.md index c8c547bc..af75ed29 100644 --- a/docs/guides/multi-agent.md +++ b/docs/guides/multi-agent.md @@ -104,6 +104,55 @@ cortex agent history writer --limit 10 Weights update automatically via EMA — well-performing prompts rise, underperforming ones fade. No manual tuning needed. +## Entity Hub Traversal + +Entities are the bridge between agents. When multiple agents store knowledge about the same entity (person, company, technology), Cortex connects them via entity nodes. + +``` +Agent A's fact ──[references]──► "Anthropic" entity ◄──[references]── Agent B's decision +``` + +This two-hop pattern is how cross-agent discovery works: follow `references` edges from your knowledge to entities, then follow other agents' `references` edges back to their knowledge. + +```python +# Store knowledge that references an entity +cx.store("fact", "Anthropic released Claude 4", + source_agent="researcher", tags=["entity-anthropic"]) + +# The auto-linker will connect this to other agents' knowledge about Anthropic +``` + +See [Entity Resolution](../concepts/entity-resolution.md) for the full model. + +## Briefing Scope + +Multi-agent setups unlock two additional briefing scopes: + +### Shared Scope + +Includes the requesting agent's knowledge plus cross-agent context about shared entities. Use when an agent needs awareness of what other agents know about the same topics. + +```bash +cortex briefing researcher --scope shared +``` + +### Unified Scope + +Multi-agent briefing for orchestrators. Spans multiple agents and synthesises a combined view. + +```bash +cortex briefing --agents researcher,architect,writer +``` + +## Philosophy: Agents Are Visitors, the Graph Is Persistent + +Agents come and go. They crash, restart, get replaced. The graph persists. Design your multi-agent system with this in mind: + +- Store knowledge in the graph, not in agent state +- Use briefings to bootstrap agent context on startup +- Let the auto-linker discover cross-agent connections -- don't wire them manually +- Use entities as the coordination mechanism between agents + ## Configuration No special configuration is needed for multi-agent setups. Run a single Cortex server and point all agents at it. diff --git a/docs/guides/research-agent.md b/docs/guides/research-agent.md new file mode 100644 index 00000000..5bd10612 --- /dev/null +++ b/docs/guides/research-agent.md @@ -0,0 +1,141 @@ +# Research Agent Memory + +Cortex gives research agents a structured memory for tracking hypotheses, experiments, and findings over long investigations. The key capability: when two findings contradict each other, Cortex detects the conflict and surfaces both in the agent's briefing so nothing gets silently overwritten. + +## Configuration + +Add a `[briefing.roles]` section to your `cortex.toml` tuned for research workflows: + +```toml +[briefing.roles] +identity = ["agent"] +persistent = ["hypothesis", "methodology"] +trackable = ["research-question", "objective"] +temporal = ["experiment", "observation"] +reviewable = ["finding", "pattern"] +superseding = ["claim", "measurement", "citation"] +``` + +Role mapping for research: + +- **persistent** -- hypotheses and methodologies that frame the entire investigation. Always visible. +- **trackable** -- active research questions and objectives. Cleared when answered or abandoned. +- **temporal** -- experiments and observations in chronological order. Provides an audit trail of what was tried and when. +- **reviewable** -- findings and patterns that need periodic re-evaluation as new evidence arrives. +- **superseding** -- claims, measurements, and citations where only the latest version matters. When a newer measurement replaces an older one, the briefing shows only the current value. + +## Tracking Hypotheses + +Store hypotheses as persistent nodes so they remain visible throughout the research: + +```python +from cortex_memory import Cortex + +cx = Cortex("localhost:9090") + +hyp_id = cx.store( + "hypothesis", + "Retrieval-augmented generation reduces hallucination rates", + body="RAG with domain-specific corpora should reduce factual " + "hallucination by >50% compared to parametric-only generation.", + importance=0.9, + source_agent="research-agent", + tags=["rag", "hallucination", "nlp"], +) +``` + +When an experiment supports or contradicts this hypothesis, store the result as a temporal `experiment` node. The auto-linker detects semantic overlap between the hypothesis and the experiment, creating `supports` or `contradicts` edges based on the relationship: + +```python +cx.store( + "experiment", + "RAG hallucination benchmark — medical QA", + body="Tested RAG vs. baseline on 500 medical questions. " + "RAG reduced hallucination rate from 23% to 8% (65% reduction). " + "Supports hypothesis.", + importance=0.8, + source_agent="research-agent", + tags=["rag", "hallucination", "medical"], +) +``` + +## Contradiction Detection + +When two findings conflict, Cortex flags them in the briefing. This happens automatically when findings share tags or high semantic similarity but contain opposing claims. + +```python +# Finding A +cx.store( + "finding", + "Treatment X is effective for condition Y", + body="Meta-analysis of 12 RCTs shows statistically significant " + "improvement (p<0.01, effect size 0.45).", + importance=0.85, + source_agent="research-agent", + tags=["treatment-x", "condition-y", "efficacy"], +) + +# Finding B (contradicts A) +cx.store( + "finding", + "Treatment X shows no significant effect on condition Y", + body="Large-scale replication study (n=5000) found no statistically " + "significant improvement (p=0.34, effect size 0.03).", + importance=0.85, + source_agent="research-agent", + tags=["treatment-x", "condition-y", "efficacy"], +) +``` + +The auto-linker detects the shared tags and opposing language, creating a `contradicts` edge. The next briefing surfaces both findings with a contradiction alert, prompting the agent to reconcile them rather than silently using whichever it encountered last. + +## Entity Graphs + +Build entity graphs of papers, authors, and institutions to cross-reference findings by shared entities: + +```python +from cortex_memory import EntityType + +cx.store_entity("Dr. Jane Smith", entity_type=EntityType.PERSON, + aliases=["J. Smith", "Smith J."], + source_agent="research-agent") + +cx.store_entity("Stanford NLP Group", entity_type=EntityType.COMPANY, + aliases=["Stanford NLP"], + source_agent="research-agent") +``` + +When findings reference the same author or institution, the auto-linker connects them through the shared entity. This lets the agent answer questions like "What has Dr. Smith published that relates to our hypothesis?" through a single graph traversal. + +## Cross-Agent Research + +Multiple research agents specializing in different domains can share a single Cortex instance. Each agent writes to its own `source_agent` namespace, but the shared graph means the auto-linker connects findings across agents automatically. + +```python +# Biology agent stores a finding +cx.store("finding", "Protein X binds to receptor Y", + source_agent="bio-agent", importance=0.8, + tags=["protein-x", "receptor-y"]) + +# Chemistry agent stores a related finding +cx.store("finding", "Compound Z inhibits receptor Y binding", + source_agent="chem-agent", importance=0.8, + tags=["compound-z", "receptor-y"]) +``` + +Use `scope=shared` in briefings to include cross-agent findings: + +```python +briefing = cx.briefing("bio-agent", compact=True) +# Includes the chemistry agent's finding about receptor Y +# because the shared entity connects them +``` + +## Quick Setup + +```bash +cortex init --template research +cortex serve +``` + +This generates a `cortex.toml` with research-oriented roles and retention settings tuned for long-running investigations. diff --git a/docs/reference/cli.md b/docs/reference/cli.md index cb55c732..9c7d7f29 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -19,7 +19,13 @@ Initialise a new Cortex project in the current directory. cortex init [--path ] ``` -Creates `cortex.toml` with sensible defaults. +Creates `cortex.toml` with sensible defaults. Use `--template` for agent-type presets: + +```bash +cortex init --template coding # Coding agent preset +cortex init --template research # Research agent preset +cortex init --template browser # Browser agent preset +``` ### `cortex serve` @@ -34,7 +40,8 @@ cortex serve [--config cortex.toml] Manage nodes. ```bash -cortex node create --kind --title [--body <body>] [--importance 0.7] [--tags tag1,tag2] +cortex node create --kind <kind> --title <title> [--body <body>] [--importance 0.7] [--tags tag1,tag2] \ + [--valid-from <ISO8601>] [--valid-until <ISO8601>] [--expires-at <ISO8601>] cortex node get <id> cortex node list [--kind <kind>] [--limit 50] cortex node delete <id> @@ -64,9 +71,15 @@ cortex search <query> [--limit 10] [--kind <kind>] [--hybrid] [--alpha 0.7] Generate a briefing for an agent. ```bash -cortex briefing <agent-id> [--format text|json] [--max-tokens 2000] +cortex briefing <agent-id> [--format text|json] [--max-tokens 2000] [--scope agent|shared] +cortex briefing --agents <id1,id2,...> [--format text|json] # Unified scope ``` +| Flag | Default | Description | +|------|---------|-------------| +| `--scope` | `agent` | Briefing scope: `agent` (own knowledge), `shared` (cross-agent entities) | +| `--agents` | -- | Comma-separated agent IDs for unified (multi-agent) briefing | + ### `cortex traverse` Traverse the graph from a starting node. @@ -117,6 +130,16 @@ Security utilities. cortex security generate-key # Generate an AES-256-GCM encryption key ``` +### `cortex trust` + +Show trust score breakdown for a node or agent reliability. + +```bash +cortex trust <node-id> # Trust breakdown for a node +cortex trust --agent <agent-id> # Agent reliability score +cortex trust --batch <id1>,<id2>,... # Batch trust scores +``` + ### `cortex stats` Show server statistics. diff --git a/docs/reference/config.md b/docs/reference/config.md index 52670569..c978b386 100644 --- a/docs/reference/config.md +++ b/docs/reference/config.md @@ -6,14 +6,17 @@ Cortex is configured via `cortex.toml` in your project directory. Run `cortex in See **[Configuration Guide](../getting-started/configuration.md)** for the complete `cortex.toml` reference with all sections: -- `[server]` — ports, data directory -- `[auto_linker]` — background linking settings -- `[briefing]` — section ordering, token budget -- `[retention]` — TTL, max nodes, eviction strategy -- `[security]` — encryption at rest -- `[ingest.nats]` — NATS subscription -- `[write_gate]` — write quality checks configuration -- `[schemas.*]` — per-kind metadata schemas +- `[server]` -- ports, data directory +- `[auto_linker]` -- background linking settings, entity promotion, configurable rules +- `[briefing]` -- token budget, role-to-kind mapping +- `[briefing.roles]` -- map node kinds to briefing roles +- `[trust]` -- trust scoring parameters +- `[trust.weights]` -- relative importance of trust signals +- `[retention]` -- TTL, max nodes, eviction strategy +- `[security]` -- encryption at rest +- `[ingest.nats]` -- NATS subscription +- `[write_gate]` -- write quality checks configuration +- `[schemas.*]` -- per-kind metadata schemas ## Schema Validation @@ -55,6 +58,58 @@ allowed_values = ["proposed", "accepted", "rejected"] Schema violations produce a 422 response with `gate.check == "schema"` and details about each violated constraint. +## Trust Scoring + +Compute trust from graph topology. See [Trust Scoring](../concepts/trust-scoring.md) for the full model. + +```toml +[trust] +corroboration_saturation = 3 +corroboration_min_weight = 0.6 +contradiction_weight = 0.3 +access_saturation = 20 +freshness_halflife = 90 + +[trust.weights] +corroboration = 0.30 +contradiction = 0.25 +source = 0.20 +access = 0.15 +freshness = 0.10 +``` + +## Briefing Roles + +Map node kinds to briefing roles. See [Briefings](../concepts/briefings.md) for role descriptions. + +```toml +[briefing.roles] +identity = ["agent"] +persistent = ["preference"] +trackable = ["goal"] +temporal = ["event"] +reviewable = ["pattern"] +superseding = ["fact", "decision"] +``` + +## Auto-Linker Rules + +Configurable structural rules for the auto-linker. See [Auto-Linker](../concepts/auto-linker.md) for condition types. + +```toml +[auto_linker] +entity_promote_every_n_cycles = 60 +entity_promote_min_agents = 2 + +[[auto_linker.rules]] +name = "decision-leads-to-event" +from_kind = "decision" +to_kind = "event" +relation = "led_to" +weight = 0.8 +condition = { type = "temporal_proximity", window_minutes = 60 } +``` + ## Environment Variables | Variable | Description | diff --git a/docs/reference/grpc-api.md b/docs/reference/grpc-api.md index 170ea3c2..de7fb2ac 100644 --- a/docs/reference/grpc-api.md +++ b/docs/reference/grpc-api.md @@ -17,6 +17,9 @@ message CreateNodeRequest { repeated string tags = 5; string source_agent = 6; map<string, string> metadata = 7; + google.protobuf.Timestamp valid_from = 8; // optional temporal validity + google.protobuf.Timestamp valid_until = 9; // optional temporal validity + google.protobuf.Timestamp expires_at = 10; // optional lifecycle expiry } ``` @@ -90,6 +93,7 @@ message CreateEdgeRequest { string to_id = 2; string relation = 3; float weight = 4; + map<string, string> metadata = 5; // optional edge metadata } ``` @@ -101,6 +105,14 @@ rpc GetBriefing(GetBriefingRequest) returns (BriefingResponse); message GetBriefingRequest { string agent_id = 1; uint32 max_tokens = 2; + BriefingScope scope = 3; // AGENT, SHARED, or UNIFIED + repeated string agent_ids = 4; // For UNIFIED scope +} + +enum BriefingScope { + AGENT = 0; + SHARED = 1; + UNIFIED = 2; } message BriefingResponse { @@ -109,6 +121,44 @@ message BriefingResponse { } ``` +### GetTrustScore + +```protobuf +rpc GetTrustScore(GetTrustScoreRequest) returns (TrustScoreResponse); + +message GetTrustScoreRequest { + string node_id = 1; +} + +message TrustScoreResponse { + string node_id = 1; + float trust_score = 2; + TrustSignals signals = 3; +} + +message TrustSignals { + float corroboration = 1; + float contradiction = 2; + float source_reliability = 3; + float access_reinforcement = 4; + float freshness = 5; +} +``` + +### BatchGetTrustScore + +```protobuf +rpc BatchGetTrustScore(BatchTrustRequest) returns (BatchTrustResponse); + +message BatchTrustRequest { + repeated string node_ids = 1; +} + +message BatchTrustResponse { + repeated TrustScoreResponse scores = 1; +} +``` + ### Traverse ```protobuf diff --git a/docs/reference/http-api.md b/docs/reference/http-api.md index 90b5e6bc..28605480 100644 --- a/docs/reference/http-api.md +++ b/docs/reference/http-api.md @@ -48,6 +48,14 @@ Query params: `q` (query string, required), `limit`, `kind`. Get a briefing for an agent. +Query params: `scope` (agent|shared, default: agent), `max_tokens`. + +## GET /briefing + +Get a unified (multi-agent) briefing. + +Query params: `agents` (comma-separated agent IDs, required), `max_tokens`. + ## GET /graph/export Export the full graph as JSON. @@ -70,6 +78,66 @@ Open the interactive graph visualiser in your browser. Trigger an immediate auto-linker cycle. +--- + +## Trust Scoring API + +### GET /trust/:node_id + +Get the trust score breakdown for a single node. + +```json +{ + "success": true, + "data": { + "node_id": "019...", + "trust_score": 0.82, + "signals": { + "corroboration": 0.90, + "contradiction": 0.95, + "source_reliability": 0.75, + "access_reinforcement": 0.60, + "freshness": 0.88 + }, + "weights": { + "corroboration": 0.30, + "contradiction": 0.25, + "source": 0.20, + "access": 0.15, + "freshness": 0.10 + } + } +} +``` + +### POST /trust/batch + +Batch trust scores for multiple nodes. + +```bash +curl -X POST http://localhost:9091/trust/batch \ + -H "Content-Type: application/json" \ + -d '{"node_ids": ["019...", "019..."]}' +``` + +### GET /trust/agents + +Agent reliability scores (source track record). + +```json +{ + "success": true, + "data": { + "agents": [ + { "agent_id": "kai", "reliability": 0.85, "corroborated": 42, "contradicted": 3 }, + { "agent_id": "scout", "reliability": 0.72, "corroborated": 28, "contradicted": 7 } + ] + } +} +``` + +--- + ## GET /auto-linker/status Get auto-linker metrics.