Skip to content

docs: refresh README to reflect 0.9.0 #44

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
davidgut1982 merged 1 commit into from
Jun 2, 2026
Merged

docs: refresh README to reflect 0.9.0 #44

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 49 additions & 21 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
[![CI](https://github.com/davidgut1982/lore-mcp/actions/workflows/ci.yml/badge.svg)](https://github.com/davidgut1982/lore-mcp/actions/workflows/ci.yml)
[![Python](https://img.shields.io/badge/python-3.11+-green)](https://python.org)
[![MCP](https://img.shields.io/badge/MCP-compatible-purple)](https://modelcontextprotocol.io)
[![Hybrid Search](https://img.shields.io/badge/search-hybrid%20%2B%20semantic-blueviolet)](https://github.com/davidgut1982/lore-mcp#semantic-search-v060)
[![Hybrid Search](https://img.shields.io/badge/search-hybrid%20%2B%20semantic-blueviolet)](https://github.com/davidgut1982/lore-mcp#semantic-search)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

![Lore demo](docs/demo.gif)
Expand Down Expand Up @@ -82,29 +82,30 @@ Your agents know: result 1 is production-safe. Result 2, spot-check before actin

---

## Semantic Search (v0.6.0+)
## Semantic Search

Lore finds entries by meaning, not just keywords. Search "DNS broken in containers" and it returns an entry titled "LXC containers inherit resolv.conf from the host" — no keyword overlap required.

Powered by local sentence-transformers embeddings (no API key, no external calls), combined with FTS5 lexical search and Reciprocal Rank Fusion. The same model used by mcp-memory-service, fully self-hosted.
Powered by local sentence-transformers embeddings (no API key, no external calls), combined with lexical full-text search and Reciprocal Rank Fusion. The same model used by mcp-memory-service, fully self-hosted. On SQLite the lexical leg uses FTS5; on PostgreSQL it uses a GIN full-text index plus pgvector for the semantic leg.

### Enable it

> **Heads up:** Semantic search is feature-complete and shipping in a future release. The v0.6.0 release that included it was yanked from PyPI on 2026-05-24 while we set up a proper staging and end-to-end testing pipeline. You can run it from source today by cloning the repo and running `pip install -e ".[semantic]"`.

```bash
# Once a stable release is published:
pip install lore-knowledge-mcp[semantic]
LORE_SEMANTIC_SEARCH=true lore-mcp
```

### What you get
### Search modes

`kb_search` resolves its mode from (in order): an explicit `search_mode`/`semantic`/`hybrid` argument, then `LORE_SEARCH_MODE_DEFAULT`, then the built-in default of `hybrid`. Every search response echoes `requested_mode` (the caller's intent) alongside `search_mode` (the mode actually executed, after any degradation).

| Mode | When to use |
|---|---|
| `fts` | Exact term matches (default when semantic is off) |
| `semantic` | Meaning-based retrieval, no keyword overlap needed |
| `hybrid` | Best of both — FTS5 + vector via RRF (recommended) |
| `fts` | Exact term matches. |
| `semantic` | Meaning-based retrieval, no keyword overlap needed. |
| `hybrid` | Best of both — lexical + vector via RRF (default). |

> The `summary` mode was removed — passing `search_mode="summary"` now returns a validation error. Lore is LLM-free by design; summarisation is the caller's responsibility.

### Backfill existing KB

Expand All @@ -119,14 +120,30 @@ kb_embedding_status() # check coverage

| Variable | Default | Notes |
|---|---|---|
| `LORE_SEMANTIC_SEARCH` | `false` | Master switch — off = current behavior unchanged |
| `LORE_EMBEDDING_MODEL` | `all-MiniLM-L6-v2` | 384d, ~90MB, English-optimized |
| `LORE_RRF_K` | `10` | Increase to 30–60 for corpora >10k entries |
| `LORE_SEMANTIC_SEARCH` | `false` | Master switch — off = lexical-only behaviour. |
| `LORE_SEARCH_MODE_DEFAULT` | `hybrid` | Default mode for `kb_search` when no mode is passed (`fts`, `semantic`, or `hybrid`). |
| `LORE_EMBEDDING_MODEL` | `all-MiniLM-L6-v2` | 384d, ~90MB, English-optimized. |
| `LORE_RRF_K` | `10` | Increase to 30–60 for corpora >10k entries. |

For multilingual content, set `LORE_EMBEDDING_MODEL=paraphrase-multilingual-MiniLM-L12-v2` (same 384d, no schema change).

---

## Automatic Memory Extraction

Lore can extract durable memories from agent conversations automatically. At the end of a session, conversation turns are sent asynchronously to a fast LLM, which extracts facts, preferences, goals, events, and system facts — then deduplicates them against the existing KB before writing.

- **Opt-in** — disabled by default (`auto_extract.enabled: false`).
- **Two providers** — OpenRouter (default, simple setup) or Cerebras direct API (gpt-oss-120b, 300+ TPS, high prompt-cache hit rate).
- **Graceful degradation** — a missing API key, HTTP error, or bad JSON returns an empty result silently; it never raises and never blocks the session.
- **Auditable** — every auto-extracted entry is tagged `source:auto-extracted`, with an optional review queue (`topic="auto-memory-pending"`) for human approval.

Set `OPENROUTER_API_KEY` (or `CEREBRAS_API_KEY`) and enable it in your plugin config.

→ **[Full setup guide: docs/auto-extraction-setup.md](docs/auto-extraction-setup.md)** — API keys, provider config, tuning thresholds, review mode, and inspecting or removing extracted entries.

---

## Automating Lore in Your Workflow

Add one line to every agent's system prompt and one entry to `~/.mcp.json` — that's the entire integration. Each phase of your engineering workflow reads prior knowledge from Lore and writes its findings back, so nothing is re-discovered from scratch.
Expand All @@ -145,15 +162,13 @@ Add one line to every agent's system prompt and one entry to `~/.mcp.json` — t
pip install lore-knowledge-mcp
```

### Optional: semantic search

> **Note:** The v0.6.0 PyPI release was yanked — see [Semantic Search](#semantic-search-v060) for current install status.
#### Optional: semantic search

```bash
pip install lore-knowledge-mcp[semantic]
```

Then set `LORE_SEMANTIC_SEARCH=true`. See [Semantic Search](#semantic-search-v060) for details.
Then set `LORE_SEMANTIC_SEARCH=true`. See [Semantic Search](#semantic-search) for details.

### 2. Start the server

Expand Down Expand Up @@ -236,11 +251,14 @@ That’s it. Lore is ready.
| Tool | What it does |
|---|---|
| `kb_add` | Add an entry. Accepts `author`, `source_type` for attribution. |
| `kb_search` | Semantic search with optional topic filter. |
| `kb_search` | Semantic / hybrid / FTS search with optional topic filter. |
| `kb_get` | Fetch full entry by ID. |
| `kb_get_batch` | Fetch multiple entries by ID in a single call (re-keyed by `kb_id`). |
| `kb_list` | List entries, filter by topic. |
| `kb_update` | Update content, tags, or set `verified` flag. |
| `kb_delete` | Delete entry (requires `confirm=true`). |
| `kb_backfill_embeddings` | Generate embeddings for existing entries (idempotent). |
| `kb_embedding_status` | Report embedding coverage across the KB. |

### Investigations
| Tool | What it does |
Expand All @@ -250,19 +268,22 @@ That’s it. Lore is ready.
| `investigation_get` | Fetch full investigation by ID. |
| `investigation_log_experiment` | Log a structured hypothesis → result → conclusion. |
| `investigation_list_experiments` | List all logged experiments. |
| `investigation_delete_note` | Hard-delete a note (requires `confirm=true`; production guard). |
| `investigation_delete_experiment` | Hard-delete an experiment (requires `confirm=true`; production guard). |

### Journal
| Tool | What it does |
|---|---|
| `journal_append` | Add a milestone, decision, or reflection. |
| `journal_list` | List recent entries (default 20). |
| `journal_get` | Fetch entry by ID. |
| `journal_delete` | Hard-delete an entry (requires `confirm=true`; production guard). |
| `snapshot_config` | Snapshot a config object to the journal. |

### Document Ingestion
| Tool | What it does |
|---|---|
| `kb_ingest_doc` | Ingest a markdown file into the KB. |
| `kb_ingest_doc` | Ingest a markdown file into the KB (`strategy`: `full` or `chunked`). |
| `kb_ingest_dir` | Batch-ingest a directory, with change detection. |
| `kb_sync_status` | Check what's changed since last sync. |

Expand Down Expand Up @@ -293,6 +314,7 @@ That’s it. Lore is ready.
| Best for | Solo developers, local use | Teams, shared agents, production |
| Config | `DB_BACKEND=sqlite` (default) | `DB_BACKEND=postgres` + connection vars |
| Data location | `./knowledge-data/` (override with `KNOWLEDGE_DATA_DIR`) | Your database |
| Semantic search | sqlite-vec + FTS5 | pgvector + GIN full-text index |

**SQLite is the default.** No configuration needed — just install and run. The
SQLite database and any local-file search corpus live under
Expand All @@ -303,7 +325,9 @@ path — set it to an absolute path for a stable on-disk location).
multiple machines or agents simultaneously. `DB_BACKEND=postgres` (and the
`postgresql` alias) select the bundled local PostgreSQL client — the same path
as `DB_BACKEND=local`. Connection defaults are generic (`DB_NAME=lore`,
`DB_USER=lore_user`); override them with the connection variables below.
`DB_USER=lore_user`); override them with the connection variables below. On
PostgreSQL, hybrid search uses a combined title+content GIN full-text index for
the lexical leg and pgvector for the semantic leg.

```bash
# PostgreSQL setup
Expand All @@ -324,6 +348,10 @@ lore-mcp
| `KNOWLEDGE_DATA_DIR` | `./knowledge-data` | Root for the SQLite DB and local-file search. Portable by default — no `/srv` paths. |
| `DB_NAME` | `lore` | PostgreSQL database name. |
| `DB_USER` | `lore_user` | PostgreSQL user. |
| `LORE_SEMANTIC_SEARCH` | `false` | Master switch for semantic/hybrid search (requires the `[semantic]` extra). |
| `LORE_SEARCH_MODE_DEFAULT` | `hybrid` | Default `kb_search` mode when none is passed (`fts`, `semantic`, `hybrid`). |
| `OPENROUTER_API_KEY` | _(unset)_ | API key for automatic memory extraction via OpenRouter. See [Automatic Memory Extraction](#automatic-memory-extraction). |
| `CEREBRAS_API_KEY` | _(unset)_ | API key for automatic memory extraction via the Cerebras direct API. |
| `LATVIAN_LEARNING_ROOT` | _(unset)_ | Optional corpus root for `search_local`. Unset → that source is skipped. |
| `LATVIAN_XTTS_ROOT` | _(unset)_ | Optional transcript root for `search_transcripts`. Unset → returns a clean "not configured" result. |
| `INGEST_ROOT` | _(unset)_ | Optional corpora root for `search_corpora`. Unset → returns a clean "not configured" result. |
Expand All @@ -342,7 +370,7 @@ works out of the box.

A Hermes agent memory provider plugin that backs conversation memory with Lore is available as a separate package:

**[hermes-lore-plugin](https://github.com/davidgut1982/hermes-lore-plugin)** — drop-in memory provider for the [Hermes agent](https://github.com/NousResearch/Hermes). Stores KB entries in Lore, prefetches relevant context on session start, and deduplicates before storing.
**[hermes-lore-plugin](https://github.com/davidgut1982/hermes-lore-plugin)** — drop-in memory provider for the [Hermes agent](https://github.com/NousResearch/Hermes). Stores KB entries in Lore, prefetches relevant context on session start, and deduplicates before storing. It also drives the [automatic memory extraction](#automatic-memory-extraction) pipeline.

---

Expand Down
Loading