Release/v2.36.0

CHANGELOG.md

@@ -1,6 +1,39 @@
 # Changelog
 
-## Version 2 - MARM Protocol to Universal MCP Server Evolution
+<details>
+<summary><strong>August 2nd, 2026: Automatic Concept Graph Indexing (v2.36.0)</strong></summary>
+
+### Added: Memories Become Graph Nodes on Their Own
+
+- The concept graph only grew when someone clicked Build Concept Graph, so it was stale until a human remembered to refresh it. Storing a memory now queues it for indexing, and a background worker turns it into a node about 30 seconds later on both transports. Nothing to click.
+- The queue is a durable table in the memory database, written in the same transaction as the memory itself, so a memory cannot exist without its indexing task. A server killed mid-extraction loses nothing: the task is still there on the next start and shutdown never waits for extraction to finish.
+- Extraction failures retry with a growing delay and record the reason. A memory that fails three times is parked with its error rather than blocking the queue behind it. A failure never affects the memory itself, which stores and recalls normally throughout.
+- Turn it off with `CONCEPT_AUTO_INDEX=false` (or `0`, `no`, `off`). That stops the worker, not the queue: writes keep recording indexing tasks, so re-enabling it indexes everything written while it was off. Pacing is `CONCEPT_INDEX_DEBOUNCE_SECONDS` (30), `CONCEPT_INDEX_BATCH_SIZE` (20, capped at 500), `CONCEPT_INDEX_BATCH_PAUSE_MS` (250), `CONCEPT_INDEX_LEASE_SECONDS` (300), and `CONCEPT_INDEX_MAX_ATTEMPTS` (3).
+- Clearing a backlog is not free, and the numbers are published rather than estimated. On a real 768-memory corpus, recall during a drain moves from ~8ms to ~16ms median (p95 ~12ms to ~31ms) while writes are unaffected; `scripts/benchmarking/performance/bench_concept_worker.py` reproduces it. Entity extraction is CPU-bound, so this is contention for cores, not lock waiting, and tuning the encoder would not help. The pause between batches exists for the tail: it cuts worst-case recall during indexing from roughly 270ms to 80ms in exchange for about 18% longer drains. Steady-state indexing of a few new memories is idle almost all the time and none of this applies.
+- Running HTTP and STDIO at once is safe. Both take a leased lock in the memory database before touching the graph, so a rebuild in one cannot drop tables while the other is writing to them. A build that finds the graph busy reports `build_in_progress` and can be run again rather than colliding.
+- The Console's Knowledge Explorer picks new nodes up while it is open. It polls a small change marker rather than the graph, so an idle Explorer costs a few counts per check, and it stops entirely when the tab is not showing.
+
+### Fixed: Builds Silently Ignored Everything Past the Newest 500 Memories
+
+- Every build ended with a hard limit of 500 rows. On a corpus larger than that, the older memories were not slow to reach, they were unreachable: no scope, no setting, and no number of rebuilds would ever index them.
+- Builds now page through the whole scope. `CONCEPT_BUILD_ROW_CAP` still exists and still defaults to 500, but it is a page size now, not a ceiling. Anyone who lowered it to bound build cost gets more, smaller pages instead of a truncated graph, and a full build on a large corpus is genuinely long-running as a result.
+
+### Changed: Compacted Sessions Index Their Sources, Not Their Summary
+
+- When a session compacts, its original memories are kept as sources and a summary is written alongside them. Builds used to skip the sources and index the summary, which is backwards for a graph: the summary restates concepts the sources already stated, so every entity in a compacted session was attributed to a paraphrase rather than to where it was actually said.
+- Sources are now indexed and generated summaries are not. This is also what lets a memory reach the graph the moment it is written instead of waiting for its session to compact.
+
+### Upgrade Note
+
+This release requires one graph rebuild. Existing graphs contain entities extracted from compaction summaries that the new rule would never produce, and there is no way to remove only those. MARM detects the old graph and reports `rebuild_required` until you run:
+
+```
+marm_concept_build(search_all=True)
+```
+
+The old graph is backed up next to the database first. The build clears the queue it just covered, so the background worker does not immediately re-extract the same corpus.
+
+</details>
 
 <details>
 <summary><strong>July 31st, 2026: Chunk Durability and Repair (v2.35.0)</strong></summary>

CONTRIBUTORS.md

@@ -12,7 +12,7 @@ Thank you to everyone helping push local-first, persistent AI memory forward.
 - **sarvesh1327** ([@sarvesh1327](https://github.com/sarvesh1327)) — Fixed runtime preset handling so an explicit `COMPACTION_TRIGGER_COUNT` environment override is preserved instead of being clobbered by default/swarm/trusted presets. Added regression coverage for operator/Docker tuning paths around compaction trigger configuration ([#43](https://github.com/Lyellr88/marm-memory/pull/43)).
 - **zza-830** ([@zza-830](https://github.com/zza-830)) — Hardened configuration parsing with bounds checks and clamping warnings across server ports, rate limits, queue sizes, recall limits, compaction settings, and search weights. Also added `MARM_RECALL_DEBUG` observability so recall lane selection, fallback behavior, and candidate breakdowns can be inspected safely through stderr without adding new MCP tools ([#54](https://github.com/Lyellr88/marm-memory/pull/54)).
 - **Vaishnavi Desai** ([@vaishnavidesai09](https://github.com/vaishnavidesai09)) — Added the exact retrieval lane for code, config, command, and API-contract queries. The work introduced syntax-heavy query detection, the `exact_mode` control surface, deterministic FTS/BM25 recall with LIKE fallback, full HTTP/STDIO/service/core parameter wiring, project/platform scoping in exact recall, and regression coverage for routing, ranking, fallback behavior, and response compatibility ([#71](https://github.com/Lyellr88/marm-memory/pull/71)).
-- **Muneeb Ahmad** ([@Mxneeb](https://github.com/Mxneeb)) — Proposed replacing min-max fusion with Reciprocal Rank Fusion on the hybrid recall path, with a complete implementation and a documented mathematical rationale ([#112](https://github.com/Lyellr88/marm-memory/pull/112)). The change was not merged, but the implementation became the reference for a controlled bake-off: a weighted RRF variant preserving MARM's shipped lexical weight, so fusion was the only variable. RRF measured 6.8-7.3pp below min-max across all five LoCoMo categories, which settled a question that had been open on reasoning alone and produced the fusion decision record in `docs/current/`. The experiment also corrected MARM's estimate of its own benchmark noise from ~0.1pp to 0.56pp, a methodology fix that outlives the experiment. Separately reported a real consolidation defect, fixed in 2.33.1 and first released in v2.34.0 ([#113](https://github.com/Lyellr88/marm-memory/issues/113)): `CONSOLIDATION_THRESHOLD` is documented as a cosine threshold but was compared against a blended ranking score, which proved to be live under min-max as well, not introduced by the RRF proposal. Also surfaced the chunk-count bias in `_score_chunk_aware`'s max-over-chunks pooling, since measured and confirmed, and independently identified the recall risk the FTS candidate filter carries for low-token-overlap paraphrases.
+- **Muneeb Ahmad** ([@Mxneeb](https://github.com/Mxneeb)) — Proposed replacing min-max fusion with Reciprocal Rank Fusion on the hybrid recall path, with a complete implementation and a documented mathematical rationale ([#112](https://github.com/Lyellr88/marm-memory/pull/112)). The change was not merged, but the implementation became the reference for a controlled bake-off: a weighted RRF variant preserving MARM's shipped lexical weight, so fusion was the only variable. RRF measured 6.8-7.3pp below min-max across all five LoCoMo categories, which settled a question that had been open on reasoning alone and produced the fusion decision record in `docs/current/`. The experiment also corrected MARM's estimate of its own benchmark noise from ~0.1pp to 0.56pp, a methodology fix that outlives the experiment. Separately reported a real consolidation defect, fixed in 2.36.0 and first released in v2.36.0 ([#113](https://github.com/Lyellr88/marm-memory/issues/113)): `CONSOLIDATION_THRESHOLD` is documented as a cosine threshold but was compared against a blended ranking score, which proved to be live under min-max as well, not introduced by the RRF proposal. Also surfaced the chunk-count bias in `_score_chunk_aware`'s max-over-chunks pooling, since measured and confirmed, and independently identified the recall risk the FTS candidate filter carries for low-token-overlap paraphrases.
 
 ## Security Acknowledgments