Lyellr88 · Lyellr88 · Aug 4, 2026 · Aug 4, 2026 · Aug 4, 2026 · coderabbitai
diff --git a/AGENTS.md b/AGENTS.md
@@ -13,6 +13,9 @@ MARM is a local-first MCP memory server: Python FastAPI in `marm-mcp-server/`, p
 - **Storage**: SQLite WAL at `~/.marm/marm_memory.db` (connection pool, FTS5 external-content index `memories_fts`, `memory_chunks` for long-memory chunking). The concept graph uses its own database `~/.marm/index/marm_index.db` with its own pool. Never share connections between the two.
 - **Write path**: all memory writes go through the serialized async write queue (one worker). Do not add write paths that bypass it. `marm_log_entry` dual-writes: a `log_entries` row plus a semantic memory in `memories` (via the queue); a semantic-store failure must never fail the log write.
 - **Code graph**: a pinned external binary (codebase-memory-mcp) supervised as a child process over newline-delimited JSON-RPC (`core/graph_supervisor.py`, `core/graph_client.py`). It starts lazily and runs degraded on failure. Graph or concept failures must never break the 7 core memory tools.
+- **Both graphs index themselves**, on by default, one background worker each on both transports. Concept extraction is queue-driven: a write enqueues a durable outbox row in the same transaction as the memory (`core/concept_worker.py`). Code indexing is poll-driven: a git signature per indexed repo, re-indexing on a commit and every cycle while the tree is dirty (`core/graph_index_worker.py`). Neither may be made to block a write, a recall, or startup.
+- **Cross-process serialization is a leased DB row, never an asyncio lock.** `core/lease_lock.py` owns the mechanics; `concept_build_lock` and `graph_index_lock` are its two bindings, deliberately separate rows. HTTP and STDIO are separate processes, so an in-process lock protects nothing. Every code-index call AND `delete_project` take the graph gate. Release is driven by the engine call's completion, not the awaiting task: `asyncio.to_thread` cancellation cancels the await and leaves the thread writing.
+- **Runtime switches live in the DB, not just the environment** (`core/runtime_flags.py`). A saved override beats the env var so a Dockerfile cannot silently re-enable what a user turned off, and both workers re-read per cycle so no restart is needed. Any new background worker follows this: read the flag every cycle, and start the loop even when off so it can be turned on from another process.
 - **Graph-aware recall**: `marm_smart_recall` keeps primary memory ranking authoritative and may add bounded `graph_context` from the isolated concept database. Graph enrichment is read-only and fail-open; trim graph details before primary results when enforcing response limits.
 - **Embeddings**: one fastembed `jinaai/jina-embeddings-v2-small-en` encoder (512 dimensions), lazy-loaded and serialized behind a lock. Writes must succeed even when the encoder is unavailable. Existing data requires `marm-mcp-server --migrate-embeddings` before restart when upgrading from MiniLM.
 
@@ -75,9 +78,10 @@ Semver: MAJOR = breaking (schema renames, parameter removals), MINOR = new tools
 - Dev setup: `cd marm-mcp-server && pip install -e ".[dev]" && python scripts/bundle-concept-model.py`
 - Benchmarks live in `scripts/benchmarking/`: `preformance/bench_hotpath.py` for hot-path performance, `accuracy/locomo/run_eval.py` for LoCoMo retrieval accuracy. Do not publish performance claims neither script can back.
 
-## Current Stats (v2.28.0)
+## Current Stats (v2.37.0)
 
 - 14 MCP tools over HTTP + STDIO
-- 2 isolated SQLite databases (memory + concept graph)
+- 3 isolated SQLite databases (memory + concept graph + analytics), no shared pools. The code graph engine owns its own store outside all three
 - Hybrid recall: FTS5 BM25 exact lane + bounded semantic rerank
 - Bundled concept extraction: spaCy plus the `en_core_web_sm` pipeline, both loaded lazily; Docker image includes the graph engine
+- Two background indexers, both on by default: concept extraction from a durable outbox, code re-indexing from a git-signature poll
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,29 @@
 # Changelog
 
+<details>
+<summary><strong>August 4th, 2026: Automatic Code Graph Indexing (v2.37.0)</strong></summary>
+
+### Added: Indexed Repositories Refresh Themselves
+
+- A code graph was only ever as fresh as the last time someone called `marm_graph_index`. This repo's own index was four days and a full release behind when the work started, and nothing surfaced that. A stale graph does not fail loudly, it answers confidently from deleted code. Indexed repositories are now re-indexed in the background on both transports, on by default, with nothing to click.
+- Changes are detected with a git signature computed outside the engine, so an idle check costs no engine lock. A commit moves `HEAD` and triggers a re-index. While the working tree is dirty the repo is re-indexed every cycle instead, because `git status` reports which files changed and not what is in them: the second and every later edit to one file produce byte-identical output, so any cheaper fingerprint stops noticing after the first save. Indexing is incremental, so an unchanged dirty repo costs a few hundred milliseconds.
+- Non-git directories have no cheap signature at all, so they get an unconditional re-index on a slower lane (`GRAPH_AUTO_INDEX_FULL_INTERVAL`, 300s) rather than the fast one.
+- Every index call in MARM, automatic or manual, now passes through one leased row in the memory database. HTTP and STDIO are separate processes with separate engine children over one shared engine store, and the previous in-process lock could not see across that boundary. A manual index that arrives during an automatic one reports `index_in_progress` instead of running alongside it.
+- Turn it off with `marm-mcp-server projects auto off`, or from an agent with `marm_graph_index(action="auto_off")`. `knowledge auto off` does the same for concept extraction. Both take effect on the next cycle with no restart, both survive one, and both work with the graph engine stopped. A saved switch beats the environment variable, so a `GRAPH_AUTO_INDEX=true` in a Dockerfile cannot silently re-enable something you turned off; `auto status` names which one won.
+- Deleting a project records a durable suppression, so a poller holding a cached project list cannot recreate what you just deleted. An explicit manual index re-enrolls it.
+- Pacing is `GRAPH_AUTO_INDEX_INTERVAL` (30s), `GRAPH_AUTO_INDEX_MODE` (`moderate`), `GRAPH_AUTO_INDEX_LEASE_SECONDS` (120), and `GRAPH_AUTO_INDEX_PROJECT_TTL` (300s). 30 seconds rather than the engine's own 5: a git signature measures ~103ms per project on Windows where process spawn dominates, and a code graph does not need sub-minute freshness.
+- Nothing is downloaded on your behalf. Auto-indexing is on by default, but the poller stays dormant until the graph engine binary is already on disk, so a fresh install does not pull ~269MB at first boot for a user who never calls a graph tool.
+
+### Upgrade Note
+
+No action required. Two tables are added to the memory database on first start. If you would rather index only on request, run:
+
+```
+marm-mcp-server projects auto off
+```
+
+</details>
+
 <details>
 <summary><strong>August 2nd, 2026: Automatic Concept Graph Indexing (v2.36.0)</strong></summary>
 

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -53,6 +53,7 @@ python -m marm_mcp_server --generate-key
 ```text
 marm-mcp-server/
   marm_mcp_server/
+    __main__.py                # `python -m marm_mcp_server` entry, delegates to cli
     cli.py                     # HTTP CLI, dependency checks, and server factory
     server.py                  # FastAPI HTTP composition root
     server_stdio.py            # STDIO bootstrap and core-tool registration
@@ -63,14 +64,25 @@ marm-mcp-server/
       memory_db.py             # SQLite schema, connection pool, and DB maintenance routines
       memory_scoring.py        # Semantic, FTS, temporal, and chunk-aware recall scoring
       memory_ops.py            # Store/update/recall/delete/list memory operations
+      memory_recall.py         # Recall orchestration across the scoring lanes
+      memory_delete.py         # Delete paths and their cascade handling
       write_queue.py           # Serialized write queue for SQLite writer stability
       consolidation.py         # Content-hash and semantic write-time consolidation
       compaction.py            # Background compaction candidate detection and nudges
       compaction_scheduler.py  # Optional compaction maintenance scheduler
+      docs_db.py               # Indexed copy of the shipped docs served to agents
       concept_db.py            # Concept graph schema and isolated SQLite pool
       concept_extraction.py    # spaCy entity/relationship extraction (bundled model)
+      concept_queue.py         # Durable outbox: one indexing task per stored memory
+      concept_worker.py        # Background worker draining that queue into the graph
+      concept_build_lock.py    # Concept-graph binding of the cross-process lease
+      lease_lock.py            # Leased-row mutual exclusion, shared by both graphs
       graph_supervisor.py      # Lazy singleton supervisor for the embedded graph engine
       graph_client.py          # Concept graph's in-process link into the code graph
+      graph_index_lock.py      # The one gate every code-graph store mutation takes
+      graph_index_worker.py    # Git-signature poller that keeps code graphs current
+      runtime_flags.py         # Persisted on/off switches and watch suppressions
+      runtime_manager.py       # Local runtime discovery and background start/stop
       protocol_delivery_state.py  # Bounded HTTP protocol-delivery state
       models.py                # Shared Pydantic request/response models
       events.py                # Internal event hooks
@@ -100,16 +112,34 @@ marm-mcp-server/
       notebook.py              # Notebook dispatch service
       recall.py                # Shared smart-recall response logic
       summary.py               # Shared session summary formatting
+      graph_context.py         # Bounded read-only concept context for recall
+      log_entry.py             # Shared log-entry/notebook data ops, both transports
       compaction_apply.py      # Atomic compaction apply transaction
       compaction_summarize.py  # Compaction cluster summarization helpers
       stdio_entry_tools.py     # STDIO log entry/show/delete workflow bodies
       stdio_graph_tools.py     # STDIO graph/concept bodies and registration helper
+      cli_parser.py            # Argument parsers for the product and legacy CLIs
+      cli_output.py            # Human-readable status/doctor/maintenance rendering
+      product_help.py          # Terminal-aware root help rendering
+      product_workflows.py     # High-level local workflows (start, upgrade, uninstall)
+      product_logs.py          # Bounded managed-runtime log display
+      runtime_status.py        # Read-only status aggregation for diagnostics
+      projects_cli.py          # `projects` code-index commands
+      graph_auto_cli.py        # `projects auto` / `knowledge auto` on-off switches
+      key_management.py        # Persistent local API-key operations
+      package_management.py    # Installer detection and registry checks
+      skill_install.py         # Installs the bundled marm-init skill into agent folders
+      docker_cli.py            # Docker parser registration and dispatch
+      docker_commands.py       # Safe Docker command planning and execution
     utils/
       dependency_check.py      # Runtime dependency validation
       helpers.py               # Shared helpers
       logging_filters.py       # Process logging noise filters
       multiprocess_guard.py    # Unsupported multi-worker runtime warning
       security.py              # API key generation
+      embedding_state.py       # Inspect persisted embedding compatibility, no runtime init
+      embedding_migration.py   # Resumable stopped-server embedding vector migration
+      chunk_backfill.py        # Stopped-server backfill of memory_chunks after config change
   marm_graph/                  # Embedded marm-graph wrapper: subprocess JSON-RPC client,
                                #   tool router, and backend verification for the pinned
                                #   codebase-memory-mcp binary
@@ -132,6 +162,8 @@ STDIO mode lives in `marm_mcp_server/server_stdio.py` and uses the official MCP
 
 If a tool behavior changes, check whether the HTTP endpoint and STDIO tool both need the same update.
 
+Separate transports means separate processes. Both run the same background indexers (concept extraction and code re-indexing) against the same databases, so anything they touch needs mutual exclusion that reaches across processes: a leased row in the memory database, not an `asyncio.Lock` or a module-level `threading.Lock`. `core/lease_lock.py` is that primitive, and a run of the test suite is not enough to catch a mistake here, since one interpreter never exercises the boundary. The two-process tests in `tests/test_concept_two_process.py` and `tests/test_graph_auto_index.py` spawn a real second interpreter for exactly that reason.
+
 **Docker HTTP requires an API key**
 
 Docker HTTP binds inside the container with `SERVER_HOST=0.0.0.0`, and host requests arrive through Docker bridge networking rather than `127.0.0.1`. Always pass `MARM_API_KEY` for Docker HTTP.

diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md
@@ -12,7 +12,12 @@ Thank you to everyone helping push local-first, persistent AI memory forward.
 - **sarvesh1327** ([@sarvesh1327](https://github.com/sarvesh1327)) — Fixed runtime preset handling so an explicit `COMPACTION_TRIGGER_COUNT` environment override is preserved instead of being clobbered by default/swarm/trusted presets. Added regression coverage for operator/Docker tuning paths around compaction trigger configuration ([#43](https://github.com/Lyellr88/marm-memory/pull/43)).
 - **zza-830** ([@zza-830](https://github.com/zza-830)) — Hardened configuration parsing with bounds checks and clamping warnings across server ports, rate limits, queue sizes, recall limits, compaction settings, and search weights. Also added `MARM_RECALL_DEBUG` observability so recall lane selection, fallback behavior, and candidate breakdowns can be inspected safely through stderr without adding new MCP tools ([#54](https://github.com/Lyellr88/marm-memory/pull/54)).
 - **Vaishnavi Desai** ([@vaishnavidesai09](https://github.com/vaishnavidesai09)) — Added the exact retrieval lane for code, config, command, and API-contract queries. The work introduced syntax-heavy query detection, the `exact_mode` control surface, deterministic FTS/BM25 recall with LIKE fallback, full HTTP/STDIO/service/core parameter wiring, project/platform scoping in exact recall, and regression coverage for routing, ranking, fallback behavior, and response compatibility ([#71](https://github.com/Lyellr88/marm-memory/pull/71)).
-- **Muneeb Ahmad** ([@Mxneeb](https://github.com/Mxneeb)) — Proposed replacing min-max fusion with Reciprocal Rank Fusion on the hybrid recall path, with a complete implementation and a documented mathematical rationale ([#112](https://github.com/Lyellr88/marm-memory/pull/112)). The change was not merged, but the implementation became the reference for a controlled bake-off: a weighted RRF variant preserving MARM's shipped lexical weight, so fusion was the only variable. RRF measured 6.8-7.3pp below min-max across all five LoCoMo categories, which settled a question that had been open on reasoning alone and produced the fusion decision record in `docs/current/`. The experiment also corrected MARM's estimate of its own benchmark noise from ~0.1pp to 0.56pp, a methodology fix that outlives the experiment. Separately reported a real consolidation defect, fixed in 2.36.0 and first released in v2.36.0 ([#113](https://github.com/Lyellr88/marm-memory/issues/113)): `CONSOLIDATION_THRESHOLD` is documented as a cosine threshold but was compared against a blended ranking score, which proved to be live under min-max as well, not introduced by the RRF proposal. Also surfaced the chunk-count bias in `_score_chunk_aware`'s max-over-chunks pooling, since measured and confirmed, and independently identified the recall risk the FTS candidate filter carries for low-token-overlap paraphrases.
+- **Muneeb Ahmad** ([@Mxneeb](https://github.com/Mxneeb)) — Proposed replacing min-max fusion with Reciprocal Rank Fusion on the hybrid recall path, with a complete implementation and a documented mathematical rationale ([#112](https://github.com/Lyellr88/marm-memory/pull/112)). The change was not merged, but the implementation became the reference for a controlled bake-off: a weighted RRF variant preserving MARM's shipped lexical weight, so fusion was the only variable. RRF measured 6.8-7.3pp below min-max across all five LoCoMo categories, which settled a question that had been open on reasoning alone and produced the fusion decision record in `docs/current/`. The experiment also corrected MARM's estimate of its own benchmark noise from ~0.1pp to 0.56pp, a methodology fix that outlives the experiment. Separately reported a real consolidation defect, fixed in v2.33.1 ([#113](https://github.com/Lyellr88/marm-memory/issues/113)): `CONSOLIDATION_THRESHOLD` is documented as a cosine threshold but was compared against a blended ranking score, which proved to be live under min-max as well, not introduced by the RRF proposal. Also surfaced the chunk-count bias in `_score_chunk_aware`'s max-over-chunks pooling, since measured and confirmed, and independently identified the recall risk the FTS candidate filter carries for low-token-overlap paraphrases.
+
+## Test & Documentation Contributors
+
+- **Aditya** ([@adity982](https://github.com/adity982)) — Made the zero-temporal-weight regression test verify what it claimed. The old assertion only compared the two candidates' positions when the second one happened to surface, so it could pass without ever exercising the ordering relationship it described. The test now stubs the lexical lane to return both candidates with fixed scores, pins `HYBRID_SEARCH_TEXT_WEIGHT`, and asserts the exact result order ([#124](https://github.com/Lyellr88/marm-memory/pull/124)). Test-only, with no change to production behavior.
+- **tomatotomata** ([@tomatotomata](https://github.com/tomatotomata)) — Replaced a stale version-history and internal audit block in the compaction endpoint's module docstring with the lifecycle the code actually implements: pending candidates, staged review, then apply or discard, noting that final writes go through the single-writer queue while staging stays per-candidate ([#126](https://github.com/Lyellr88/marm-memory/pull/126)). Documentation only.
 
 ## Security Acknowledgments