diff --git a/README.md b/README.md index e39578c..2c3abc9 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ [![CI](https://github.com/davidgut1982/lore-mcp/actions/workflows/ci.yml/badge.svg)](https://github.com/davidgut1982/lore-mcp/actions/workflows/ci.yml) [![Python](https://img.shields.io/badge/python-3.11+-green)](https://python.org) [![MCP](https://img.shields.io/badge/MCP-compatible-purple)](https://modelcontextprotocol.io) -[![Hybrid Search](https://img.shields.io/badge/search-hybrid%20%2B%20semantic-blueviolet)](https://github.com/davidgut1982/lore-mcp#semantic-search-v060) +[![Hybrid Search](https://img.shields.io/badge/search-hybrid%20%2B%20semantic-blueviolet)](https://github.com/davidgut1982/lore-mcp#semantic-search) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE) ![Lore demo](docs/demo.gif) @@ -82,29 +82,30 @@ Your agents know: result 1 is production-safe. Result 2, spot-check before actin --- -## Semantic Search (v0.6.0+) +## Semantic Search Lore finds entries by meaning, not just keywords. Search "DNS broken in containers" and it returns an entry titled "LXC containers inherit resolv.conf from the host" — no keyword overlap required. -Powered by local sentence-transformers embeddings (no API key, no external calls), combined with FTS5 lexical search and Reciprocal Rank Fusion. The same model used by mcp-memory-service, fully self-hosted. +Powered by local sentence-transformers embeddings (no API key, no external calls), combined with lexical full-text search and Reciprocal Rank Fusion. The same model used by mcp-memory-service, fully self-hosted. On SQLite the lexical leg uses FTS5; on PostgreSQL it uses a GIN full-text index plus pgvector for the semantic leg. ### Enable it -> **Heads up:** Semantic search is feature-complete and shipping in a future release. The v0.6.0 release that included it was yanked from PyPI on 2026-05-24 while we set up a proper staging and end-to-end testing pipeline. You can run it from source today by cloning the repo and running `pip install -e ".[semantic]"`. - ```bash -# Once a stable release is published: pip install lore-knowledge-mcp[semantic] LORE_SEMANTIC_SEARCH=true lore-mcp ``` -### What you get +### Search modes + +`kb_search` resolves its mode from (in order): an explicit `search_mode`/`semantic`/`hybrid` argument, then `LORE_SEARCH_MODE_DEFAULT`, then the built-in default of `hybrid`. Every search response echoes `requested_mode` (the caller's intent) alongside `search_mode` (the mode actually executed, after any degradation). | Mode | When to use | |---|---| -| `fts` | Exact term matches (default when semantic is off) | -| `semantic` | Meaning-based retrieval, no keyword overlap needed | -| `hybrid` | Best of both — FTS5 + vector via RRF (recommended) | +| `fts` | Exact term matches. | +| `semantic` | Meaning-based retrieval, no keyword overlap needed. | +| `hybrid` | Best of both — lexical + vector via RRF (default). | + +> The `summary` mode was removed — passing `search_mode="summary"` now returns a validation error. Lore is LLM-free by design; summarisation is the caller's responsibility. ### Backfill existing KB @@ -119,14 +120,30 @@ kb_embedding_status() # check coverage | Variable | Default | Notes | |---|---|---| -| `LORE_SEMANTIC_SEARCH` | `false` | Master switch — off = current behavior unchanged | -| `LORE_EMBEDDING_MODEL` | `all-MiniLM-L6-v2` | 384d, ~90MB, English-optimized | -| `LORE_RRF_K` | `10` | Increase to 30–60 for corpora >10k entries | +| `LORE_SEMANTIC_SEARCH` | `false` | Master switch — off = lexical-only behaviour. | +| `LORE_SEARCH_MODE_DEFAULT` | `hybrid` | Default mode for `kb_search` when no mode is passed (`fts`, `semantic`, or `hybrid`). | +| `LORE_EMBEDDING_MODEL` | `all-MiniLM-L6-v2` | 384d, ~90MB, English-optimized. | +| `LORE_RRF_K` | `10` | Increase to 30–60 for corpora >10k entries. | For multilingual content, set `LORE_EMBEDDING_MODEL=paraphrase-multilingual-MiniLM-L12-v2` (same 384d, no schema change). --- +## Automatic Memory Extraction + +Lore can extract durable memories from agent conversations automatically. At the end of a session, conversation turns are sent asynchronously to a fast LLM, which extracts facts, preferences, goals, events, and system facts — then deduplicates them against the existing KB before writing. + +- **Opt-in** — disabled by default (`auto_extract.enabled: false`). +- **Two providers** — OpenRouter (default, simple setup) or Cerebras direct API (gpt-oss-120b, 300+ TPS, high prompt-cache hit rate). +- **Graceful degradation** — a missing API key, HTTP error, or bad JSON returns an empty result silently; it never raises and never blocks the session. +- **Auditable** — every auto-extracted entry is tagged `source:auto-extracted`, with an optional review queue (`topic="auto-memory-pending"`) for human approval. + +Set `OPENROUTER_API_KEY` (or `CEREBRAS_API_KEY`) and enable it in your plugin config. + +→ **[Full setup guide: docs/auto-extraction-setup.md](docs/auto-extraction-setup.md)** — API keys, provider config, tuning thresholds, review mode, and inspecting or removing extracted entries. + +--- + ## Automating Lore in Your Workflow Add one line to every agent's system prompt and one entry to `~/.mcp.json` — that's the entire integration. Each phase of your engineering workflow reads prior knowledge from Lore and writes its findings back, so nothing is re-discovered from scratch. @@ -145,15 +162,13 @@ Add one line to every agent's system prompt and one entry to `~/.mcp.json` — t pip install lore-knowledge-mcp ``` -### Optional: semantic search - -> **Note:** The v0.6.0 PyPI release was yanked — see [Semantic Search](#semantic-search-v060) for current install status. +#### Optional: semantic search ```bash pip install lore-knowledge-mcp[semantic] ``` -Then set `LORE_SEMANTIC_SEARCH=true`. See [Semantic Search](#semantic-search-v060) for details. +Then set `LORE_SEMANTIC_SEARCH=true`. See [Semantic Search](#semantic-search) for details. ### 2. Start the server @@ -236,11 +251,14 @@ That’s it. Lore is ready. | Tool | What it does | |---|---| | `kb_add` | Add an entry. Accepts `author`, `source_type` for attribution. | -| `kb_search` | Semantic search with optional topic filter. | +| `kb_search` | Semantic / hybrid / FTS search with optional topic filter. | | `kb_get` | Fetch full entry by ID. | +| `kb_get_batch` | Fetch multiple entries by ID in a single call (re-keyed by `kb_id`). | | `kb_list` | List entries, filter by topic. | | `kb_update` | Update content, tags, or set `verified` flag. | | `kb_delete` | Delete entry (requires `confirm=true`). | +| `kb_backfill_embeddings` | Generate embeddings for existing entries (idempotent). | +| `kb_embedding_status` | Report embedding coverage across the KB. | ### Investigations | Tool | What it does | @@ -250,6 +268,8 @@ That’s it. Lore is ready. | `investigation_get` | Fetch full investigation by ID. | | `investigation_log_experiment` | Log a structured hypothesis → result → conclusion. | | `investigation_list_experiments` | List all logged experiments. | +| `investigation_delete_note` | Hard-delete a note (requires `confirm=true`; production guard). | +| `investigation_delete_experiment` | Hard-delete an experiment (requires `confirm=true`; production guard). | ### Journal | Tool | What it does | @@ -257,12 +277,13 @@ That’s it. Lore is ready. | `journal_append` | Add a milestone, decision, or reflection. | | `journal_list` | List recent entries (default 20). | | `journal_get` | Fetch entry by ID. | +| `journal_delete` | Hard-delete an entry (requires `confirm=true`; production guard). | | `snapshot_config` | Snapshot a config object to the journal. | ### Document Ingestion | Tool | What it does | |---|---| -| `kb_ingest_doc` | Ingest a markdown file into the KB. | +| `kb_ingest_doc` | Ingest a markdown file into the KB (`strategy`: `full` or `chunked`). | | `kb_ingest_dir` | Batch-ingest a directory, with change detection. | | `kb_sync_status` | Check what's changed since last sync. | @@ -293,6 +314,7 @@ That’s it. Lore is ready. | Best for | Solo developers, local use | Teams, shared agents, production | | Config | `DB_BACKEND=sqlite` (default) | `DB_BACKEND=postgres` + connection vars | | Data location | `./knowledge-data/` (override with `KNOWLEDGE_DATA_DIR`) | Your database | +| Semantic search | sqlite-vec + FTS5 | pgvector + GIN full-text index | **SQLite is the default.** No configuration needed — just install and run. The SQLite database and any local-file search corpus live under @@ -303,7 +325,9 @@ path — set it to an absolute path for a stable on-disk location). multiple machines or agents simultaneously. `DB_BACKEND=postgres` (and the `postgresql` alias) select the bundled local PostgreSQL client — the same path as `DB_BACKEND=local`. Connection defaults are generic (`DB_NAME=lore`, -`DB_USER=lore_user`); override them with the connection variables below. +`DB_USER=lore_user`); override them with the connection variables below. On +PostgreSQL, hybrid search uses a combined title+content GIN full-text index for +the lexical leg and pgvector for the semantic leg. ```bash # PostgreSQL setup @@ -324,6 +348,10 @@ lore-mcp | `KNOWLEDGE_DATA_DIR` | `./knowledge-data` | Root for the SQLite DB and local-file search. Portable by default — no `/srv` paths. | | `DB_NAME` | `lore` | PostgreSQL database name. | | `DB_USER` | `lore_user` | PostgreSQL user. | +| `LORE_SEMANTIC_SEARCH` | `false` | Master switch for semantic/hybrid search (requires the `[semantic]` extra). | +| `LORE_SEARCH_MODE_DEFAULT` | `hybrid` | Default `kb_search` mode when none is passed (`fts`, `semantic`, `hybrid`). | +| `OPENROUTER_API_KEY` | _(unset)_ | API key for automatic memory extraction via OpenRouter. See [Automatic Memory Extraction](#automatic-memory-extraction). | +| `CEREBRAS_API_KEY` | _(unset)_ | API key for automatic memory extraction via the Cerebras direct API. | | `LATVIAN_LEARNING_ROOT` | _(unset)_ | Optional corpus root for `search_local`. Unset → that source is skipped. | | `LATVIAN_XTTS_ROOT` | _(unset)_ | Optional transcript root for `search_transcripts`. Unset → returns a clean "not configured" result. | | `INGEST_ROOT` | _(unset)_ | Optional corpora root for `search_corpora`. Unset → returns a clean "not configured" result. | @@ -342,7 +370,7 @@ works out of the box. A Hermes agent memory provider plugin that backs conversation memory with Lore is available as a separate package: -**[hermes-lore-plugin](https://github.com/davidgut1982/hermes-lore-plugin)** — drop-in memory provider for the [Hermes agent](https://github.com/NousResearch/Hermes). Stores KB entries in Lore, prefetches relevant context on session start, and deduplicates before storing. +**[hermes-lore-plugin](https://github.com/davidgut1982/hermes-lore-plugin)** — drop-in memory provider for the [Hermes agent](https://github.com/NousResearch/Hermes). Stores KB entries in Lore, prefetches relevant context on session start, and deduplicates before storing. It also drives the [automatic memory extraction](#automatic-memory-extraction) pipeline. ---