Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ __pycache__/
# MCP server database
marm-mcp-server/*.db
marm-mcp-server/marm_memory.db
marm-mcp-server/pip/

# Logs and runtime data
logs/
Expand Down Expand Up @@ -110,13 +111,18 @@ docs/future/
.impeccable
.cursor
.pytest_cache/
.pytest-review-*/
.pytest-smoke-*/
.pytest_tmp*/
.pytest_tmp_dashboard/
.pytest_tmp_review/
.pytest_tmp_protocol/
marm-mcp-server/.pytest_cache/
marm-mcp-server/.pytest_tmp/
marm-mcp-server/.pytest_tmp*/
marm-mcp-server/.pytest-review-*/
marm-mcp-server/.pytest-smoke-*/
marm-pytest-*/
marm_usage_analytics.db
dump.md
dump2.md
Expand Down
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ Semver: MAJOR = breaking (schema renames, parameter removals), MINOR = new tools
## Testing

- Tests live in `marm-mcp-server/tests/`; run with `pytest` from `marm-mcp-server/`.
- Run `python scripts/test-scripts/smoke-commands.py` from the repo root for the local CLI smoke suite. It uses `smoke`, `smoke_lifecycle`, `smoke_docker`, and `smoke_destructive` markers. `--docker` and `--destructive` are explicit opt-ins; destructive mode uses a disposable virtual environment rather than the active package.
- Hit real FastAPI endpoints and real SQLite. Mock only when it meaningfully speeds the test AND matches real behavior with at least 95% fidelity.
- Every new MARM Console API route needs at least one happy-path FastAPI response-contract test with the MCP adapter stubbed. This verifies the actual response model without requiring a live graph backend.
- No existence-check or coded-to-pass tests. Deep tests that exercise real paths beat broad shallow coverage.
Expand Down
23 changes: 23 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,29 @@

## Version 2 - MARM Protocol to Universal MCP Server Evolution

<details>
<summary><strong>July 24th, 2026: Hybrid Recall Fusion, Windows Key Fix, and Command Smoke Suite (v2.29.0)</strong></summary>

### Hybrid Recall Now Fuses Lexical Relevance

- Semantic recall now blends the FTS5 BM25 keyword score into ranking instead of using it only as a candidate pre-filter and then discarding it. On the hybrid path, relevance combines semantic similarity (65%) with the normalized BM25 score (35%) before temporal weighting, so exact-term matches such as identifiers, config keys, and error strings surface more reliably. This changes recall ordering; it is backward compatible and needs no migration.
- The chunk-aware scorer was vectorized into a single matrix operation. Results are identical to the previous per-chunk loop, with less per-query work on large scans.
- Temporal decay is now applied consistently on the text-search fallback lane, so newer results are preferred when the semantic model is unavailable. The deterministic exact/lexical lane still returns matches in BM25 order, unaffected by age.

### Windows Managed-Key Reliability

- The managed API-key file (`~/.marm/.env`) is now locked to the executing Windows identity (resolved via `whoami`) rather than an environment-derived username. This fixes a case where a key created under one resolved identity could not be read back by the same process. Both `key init` and automatic HTTP key creation use the same tested helper.

### Local Command Smoke Suite

- Added `scripts/test-scripts/smoke-commands.py` and a pytest module that exercise the entire `marm-memory` command surface: every help route, safe read-only dispatches, an isolated HTTP start/health/stop lifecycle, and a managed-key round trip. A real Docker lifecycle (`--docker`) and an uninstall/reinstall inside a disposable virtual environment (`--destructive`) are explicit opt-ins that never touch the active install. A static inventory check fails if a newly added command has no smoke coverage.

### Benchmark Integrity

- The recall-scaling benchmark now times only shipped code paths (`recall_similar` and `_fetch_and_score_embedding_rows`); the previous benchmark-local reimplementations were removed so published numbers reflect what a caller actually runs. Both compared paths use the same async dispatch and exclude the constant query-encode cost. The README performance tables were refreshed from a single run.

</details>

<details>
<summary><strong>July 24th, 2026: Restored PyPI And Registry Publishing (v2.28.2)</strong></summary>

Expand Down
49 changes: 27 additions & 22 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
width="900"
height="250">
</picture>
<h1 align="center">MARM: Local-First Persistent Multi-Agent Memory Layer for MCP Clients v2.28.2</h1>
<h1 align="center">MARM: Local-First Persistent Multi-Agent Memory Layer for MCP Clients v2.29.0</h1>

[![License](https://img.shields.io/badge/license-Apache--2.0-blue)](https://github.com/Lyellr88/marm-memory/blob/MARM-main/LICENSE)
[![Python](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/)
Expand Down Expand Up @@ -159,7 +159,6 @@ When a managed key is active, `marm-memory console --import-key` opens a local C

Use `marm-memory upgrade --check` to compare the installed package with PyPI. `marm-memory upgrade` previews a safe native upgrade; `--yes` performs it only where the active installer can be replaced safely. `marm-memory uninstall` similarly previews package removal and always preserves `~/.marm`, including memory databases, graph indexes, keys, logs, and configuration. On Windows, editable installs, or pipx installs, MARM prints the exact manual command rather than attempting to replace an active launcher.


### Upgrade Existing Embeddings

The Jina v2 Small default uses 512-dimensional embeddings; older `all-MiniLM-L6-v2` data is 384-dimensional and must be re-embedded after upgrading. Stop every MARM HTTP and STDIO process, then run:
Expand All @@ -174,40 +173,46 @@ The command refuses to continue when it detects a live HTTP server, but STDIO pr

MARM is tuned for fast recall first, even as memory grows and long memories are chunked behind the scenes.

These measurements use the fastembed-backed `jinaai/jina-embeddings-v2-small-en` encoder and a throwaway local SQLite database.
These measurements use the fastembed-backed `jinaai/jina-embeddings-v2-small-en` encoder and a throwaway local SQLite database. Every timed path calls the shipped `MARMMemory` code, not a benchmark-local reimplementation. All numbers below come from a single run of [`scripts/benchmarking/performance/bench_hotpath.py`](scripts/benchmarking/performance/bench_hotpath.py) on local hardware; absolute milliseconds vary by machine, so treat the scaling shape as the signal.

### 1. Retrieval Latency Scaling

End-to-end `recall_similar` latency (includes query encoding).

| Session Size ($N$) | Min Latency | Median Latency | p95 Latency |
| :--- | :--- | :--- | :--- |
| **N = 100** | 6.6 ms | 7.4 ms | 8.0 ms |
| **N = 500** | 7.1 ms | 8.1 ms | 10.0 ms |
| **N = 1,000** | 7.6 ms | 8.5 ms | 9.0 ms |
| **N = 2,000** | 9.3 ms | 10.5 ms | 11.6 ms |
| **N = 4,000** | 11.5 ms | 12.1 ms | 13.4 ms |
| **N = 100** | 6.3 ms | 6.5 ms | 8.1 ms |
| **N = 500** | 7.2 ms | 7.4 ms | 8.0 ms |
| **N = 1,000** | 8.0 ms | 8.2 ms | 9.9 ms |
| **N = 2,000** | 9.2 ms | 9.7 ms | 10.8 ms |
| **N = 4,000** | 11.4 ms | 12.0 ms | 14.8 ms |

### 2. Encoder + Concurrency

- **Cold model load:** `887ms`
- **Warm encode:** median `4.0ms`, p95 `4.4ms`
- **Concurrent recall:** 10 gathered recalls completed in `616.3ms` vs `440.7ms` serial. The current path is intentionally serialized around shared encoder/SQLite work, so gathering calls does not create parallel speedup.
- **Cold model load:** `934ms`
- **Warm encode:** median `4.2ms`, p95 `4.8ms`
- **Concurrent recall:** 10 gathered recalls completed in `609.5ms` vs `411.3ms` serial. The current path is intentionally serialized around shared encoder/SQLite work, so gathering calls does not create parallel speedup.

### 3. Write-Time Ingestion Cost

- **Consolidation off:** median `6.8ms`, p95 `7.9ms`
- **Consolidation on:** median `51.3ms`, p95 `93.7ms`
- **Tradeoff:** write-time dedupe/clustering adds `7.6x` median cost so recall stays fast and cleaner over time.
- **Consolidation off:** median `5.9ms`, p95 `7.5ms`
- **Consolidation on:** median `61.2ms`, p95 `105.3ms`
- **Tradeoff:** write-time dedupe/clustering adds `10.3x` median cost so recall stays fast and cleaner over time.

### 4. Hybrid Search Scaling
### 4. Recall Scaling: Full Scan vs Production Hybrid

| Session Size ($N$) | Pure Semantic | Production Hybrid | FTS Filter -> Rerank | Speedup vs Pure |
| :--- | :--- | :--- | :--- | :--- |
| **N = 100** | 2.3 ms | 7.9 ms | 1.9 ms | 1.2x |
| **N = 1,000** | 23.1 ms | 9.2 ms | 2.5 ms | 9.2x |
| **N = 4,000** | 106.5 ms | 13.2 ms | 4.7 ms | 22.8x |
| **N = 10,000** | 267.1 ms | 13.6 ms | 5.3 ms | 50.4x |
Why recall stays roughly flat as memory grows: instead of scoring every stored vector, production recall uses an FTS keyword pre-filter to a bounded candidate set, then re-ranks that set by semantic + BM25 + temporal score. Both columns are real code paths (`_fetch_and_score_embedding_rows` for the full scan, `recall_similar` for hybrid), dispatched through the same async path and timed with the query vector precomputed so the constant encode cost from section 1 is excluded from both. Each iteration alternates which path runs first so neither one consistently benefits from the other's warmed cache.

These Jina v2 Small benchmarks used a throwaway real SQLite database and the live configured encoder on local hardware. Reproduce them: [`scripts/benchmarking/performance/bench_hotpath.py`](scripts/benchmarking/performance/bench_hotpath.py)
| Session Size ($N$) | Full Semantic Scan | Production Hybrid | Speedup |
| :--- | :--- | :--- | :--- |
| **N = 100** | 3.5 ms | 3.8 ms | 0.9x |
| **N = 500** | 15.9 ms | 4.2 ms | 3.8x |
| **N = 1,000** | 34.4 ms | 4.9 ms | 7.0x |
| **N = 2,000** | 67.6 ms | 5.7 ms | 11.9x |
| **N = 4,000** | 132.5 ms | 7.3 ms | 18.0x |
| **N = 10,000** | 330.4 ms | 8.4 ms | 39.4x |

The full scan grows roughly linearly with $N$ while hybrid recall stays near-flat, so the advantage widens with session size. At very small $N$ the pre-filter overhead is not yet worth it (hybrid is marginally slower at N = 100); the win appears once there is enough to skip. Reproduce with [`scripts/benchmarking/performance/bench_hotpath.py`](scripts/benchmarking/performance/bench_hotpath.py).

### 5. LoCoMo Retrieval Accuracy

Expand Down
2 changes: 1 addition & 1 deletion docs/INSTALL-DOCKER.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Universal Memory Intelligence Platform for AI Agents

**MARM v2.28.2** - Memory Accurate Response Mode
**MARM v2.29.0** - Memory Accurate Response Mode
*Docker deployment guide for Windows, Mac, and Linux*

---
Expand Down
4 changes: 2 additions & 2 deletions docs/INSTALL-LINUX.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Universal Memory Intelligence Platform for AI Agents

**MARM v2.28.2** - Memory Accurate Response Mode
**MARM v2.29.0** - Memory Accurate Response Mode
*Complete Linux installation guide*

---
Expand Down Expand Up @@ -320,7 +320,7 @@ curl -s http://localhost:8001/health
{
"status": "healthy",
"service": "MARM MCP Server",
"version": "2.28.2",
"version": "2.29.0",
"timestamp": "2026-01-01T00:00:00+00:00",
"database": "connected",
"semantic_search": "available"
Expand Down
2 changes: 1 addition & 1 deletion docs/INSTALL-PLATFORMS.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# MARM v2.28.2 MCP Server - Platform Integration Guide
# MARM v2.29.0 MCP Server - Platform Integration Guide

## Table of Contents

Expand Down
4 changes: 2 additions & 2 deletions docs/INSTALL-WINDOWS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Universal Memory Intelligence Platform for AI Agents

**MARM v2.28.2** - Memory Accurate Response Mode
**MARM v2.29.0** - Memory Accurate Response Mode
*Complete Windows installation guide*

---
Expand Down Expand Up @@ -294,7 +294,7 @@ Invoke-WebRequest -Uri http://localhost:8001/health
{
"status": "healthy",
"service": "MARM MCP Server",
"version": "2.28.2",
"version": "2.29.0",
"timestamp": "2026-01-01T00:00:00+00:00",
"database": "connected",
"semantic_search": "available"
Expand Down
2 changes: 1 addition & 1 deletion docs/TECHNICAL-OVERVIEW.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# MARM Technical Overview

> Current implementation: MARM MCP Server v2.28.2
> Current implementation: MARM MCP Server v2.29.0

This document explains what MARM is, why it is built this way, and how information moves through the system from an agent writing something to that information being recalled later. It is intended as a technical product overview, not a source-code reference.

Expand Down
2 changes: 1 addition & 1 deletion marm-mcp-server/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \

LABEL org.opencontainers.image.title="MARM Universal MCP Server"
LABEL org.opencontainers.image.description="Production-ready Universal MCP Server with advanced AI memory capabilities, semantic search, and professional-grade architecture"
LABEL org.opencontainers.image.version="2.28.2"
LABEL org.opencontainers.image.version="2.29.0"
LABEL org.opencontainers.image.authors="Ryan Lyell - marm-memory"
LABEL org.opencontainers.image.url="https://marmsystems.com"
LABEL org.opencontainers.image.source="https://github.com/Lyellr88/marm-memory"
Expand Down
2 changes: 1 addition & 1 deletion marm-mcp-server/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ mcp-name: io.github.Lyellr88/marm-mcp-server
width="900"
height="250">
</picture>
<h1 align="center">MARM: Local-First Persistent Multi-Agent Memory Layer for MCP Clients v2.28.2</h1>
<h1 align="center">MARM: Local-First Persistent Multi-Agent Memory Layer for MCP Clients v2.29.0</h1>

[![License](https://img.shields.io/badge/license-Apache--2.0-blue)](https://github.com/Lyellr88/marm-memory/blob/MARM-main/LICENSE)
[![Python](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/)
Expand Down
2 changes: 1 addition & 1 deletion marm-mcp-server/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ services:
environment:
- SERVER_HOST=0.0.0.0
- SERVER_PORT=8001
- SERVER_VERSION=2.28.2
- SERVER_VERSION=2.29.0

- ENVIRONMENT=production
- LOG_LEVEL=INFO
Expand Down
2 changes: 1 addition & 1 deletion marm-mcp-server/marm-docs/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# MARM: Local-First Persistent Multi-Agent Memory Layer for MCP Clients v2.28.2</h1>
# MARM: Local-First Persistent Multi-Agent Memory Layer for MCP Clients v2.29.0</h1>


## Table of Contents
Expand Down
4 changes: 2 additions & 2 deletions marm-mcp-server/marm_mcp_server/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,10 @@
- Production-grade performance

Author: Ryan Lyell - marm-memory
Version: 2.28.2
Version: 2.29.0
"""

__version__ = "2.28.2"
__version__ = "2.29.0"
__author__ = "Ryan Lyell"
__email__ = "lyell@marmsystems.com"

Expand Down
30 changes: 9 additions & 21 deletions marm-mcp-server/marm_mcp_server/config/settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
import sys
from pathlib import Path

from ..utils.security import generate_api_key
from ..utils.security import generate_api_key, restrict_windows_file_to_current_user


def _safe_int(env_key: str, default: int) -> int:
Expand Down Expand Up @@ -122,7 +122,7 @@ def get_analytics_db_path():
f"WARNING: SERVER_PORT={_raw_port} out of [1, 65535], clamped to {SERVER_PORT}",
file=sys.stderr,
)
SERVER_VERSION = "2.28.2"
SERVER_VERSION = "2.29.0"

GRAPH_ENABLED = os.environ.get("GRAPH_ENABLED", "true").lower() != "false"

Expand Down Expand Up @@ -397,25 +397,13 @@ def _load_key_from_file() -> str:
_MARM_ENV_PATH.chmod(0o600)
except OSError:
pass
if sys.platform == "win32":
try:
import getpass
import subprocess

user = getpass.getuser()
subprocess.run(
[
"icacls",
str(_MARM_ENV_PATH),
"/inheritance:r",
"/grant:r",
f"{user}:(F)",
],
check=False,
capture_output=True,
)
except Exception:
pass
if sys.platform == "win32" and not restrict_windows_file_to_current_user(
_MARM_ENV_PATH
):
print(
f"WARNING: Could not restrict API key file: {_MARM_ENV_PATH}",
file=sys.stderr,
)
except Exception as _e:
print(f"WARNING: Could not save API key to {_MARM_ENV_PATH}: {_e}")

Expand Down
Loading
Loading