✨ WP2: mem0 comparison arm (--arm mem0) + live local run - #22
Merged
Conversation
`--arm mem0` runs the same ten scenarios against mem0's public API via a `Mem0Arm` adapter (Step.text → Memory.add, Scenario.query → search, one mem0 session id and one on-disk store per scenario). Assertion names and row order are shared with the default arm (`assertion_names()`), so the two markdown tables compare cell-for-cell. - Missing `mem0ai` exits 2 with the install hint — never a silent skip. - The header pins `mem0.__version__` plus the full configured backend (llm provider/model, embedder provider/model + dims, vector store, base url) and the interpreter. - No assertion is relaxed for one arm or tightened for the other. Where mem0 has no equivalent concept, the adapter probes the installed library (`add(timestamp=…)`, `search(reference_date=…)`) and renders `n/a (unsupported)` with the library's own refusal as the Detail; those rows are excluded from the PASS tally. - Substring matching is case-insensitive in both arms (mem0 rewrites a turn to third person); the default arm's output is unchanged — byte-identical to main in both --markdown and plain modes. - Per-scenario wall-clock budget (`--mem0-timeout`, default 600s) and a stderr ingest trace, so a stalled or empty scenario degrades to an aligned FAIL row instead of killing the run. Tests run without mem0 installed and without network: arm-flag parsing, the exit-2 path via a failing import, the shared assertion-row contract for both arms (mem0 driven by a stub module), n/a table semantics, and the pinned header. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B1GNoTx3qTPrb8GQcxooWj
Both arms run on the same machine, interpreter and day, tables pasted
verbatim under their pinned headers: lean-memory 0.2.4 ALL PASS 26/26;
mem0 2.0.17 (llm ollama/qwen2.5:3b, embedder ollama/nomic-embed-text,
local qdrant) 8/20 graded assertions with 6 rendered `n/a (unsupported)`
— identical verdicts across two consecutive runs.
Includes the API mapping table and caveats separating architecture from
backbone: mem0 2.0.17 OSS rejects both temporal parameters at the API
boundary (the six n/a rows), while qwen2.5:3b returns `{"memory": []}`
for an isolated single-sentence turn, which explains the empty-store
FAILs. Reproduced, versioned behaviour only.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B1GNoTx3qTPrb8GQcxooWj
…, honest tallies The published mem0 arm ran against a mem0 whose non-semantic retrieval halves were silently disabled, and the artifact did not say so. mem0 2.x scores `semantic + BM25 + entity-boost`; without `fastembed` the Qdrant store's `keyword_search()` returns nothing, and without `spacy`+`en_core_web_sm` queries are not lemmatised and entity boosts are empty. Both fail with a log warning and nothing else. Arm B is re-run with both extras installed (fastembed 0.8.0, spacy 3.8.13 + en_core_web_sm). Every verdict is unchanged — 26/26 rows identical to the handicapped run — so the ranking-dependent rows are measured, not assumed, to be independent of the handicap. The state is now pinned in the emitted header (`hybrid_bm25=on/OFF`, `lemmatizer+entity_boost=on/OFF`) so a table can never again be read without knowing which mem0 it describes; the reproduce recipe and the missing-dependency hint install the extras. Also from the review: - Pin the sampling config the run depends on. `temperature` was hard-coded inside `memory_config()`; it plus `top_p`, `max_tokens` and the `search()` `threshold`/`rerank` knobs are now `Mem0Config` fields that land in the header, replacing the prose "verdict-identical across two runs" with a config a reader can re-derive. `top_p`/`max_tokens` restate mem0's own defaults, so the run is described, not changed. - Disclose the selection bias. The ten scenario texts were authored against lean-memory's offline extractor lexicon and never re-tuned for mem0, and arm A uses rule-based stub backends with no LLM call while arm B uses mem0's LLM extraction. New caveat 1 plus a sentence in the Fairness rule state that 26/26 vs 8/20 is not an apples-to-apples extraction comparison. - Fix the retirement predicate. `t != top1` assumed top-1 is the current fact, which holds in the lean arm but not in mem0's — where top-1 was the stale memory. The published Detail therefore blamed the consolidated memory rather than the actually-stale one, and a stale top-1 could have scored a false PASS. Both arms now share `_carries_retired_value()`; arm A's table is byte-identical. - Sweep the history of every live memory, not only ids `add()` echoed back, so "no retirement record" is an observation about mem0 rather than an adapter blind spot. - Choose the mem0 call shape by signature inspection instead of a blanket `except TypeError`, which could swallow a genuine internal TypeError and publish the legacy retry's error message in its place. - Add an `error (harness)` row state. Timeouts and dead backends were counted as FAILs in the published tally; they are now excluded from both sides of it, reported in their own summary line, and still fail the run. - Correct the `_contains` docstring (the needles are mixed-case, not lower-case) and pin lean-memory's verbatim casing with a case-sensitive test, so the case-insensitive matcher does not cost a regression guard. - Split the API-mapping row that credited `latest-set-exact` to `get_all()`; it comes from mem0's relevance-thresholded `search()`. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B1GNoTx3qTPrb8GQcxooWj
End-to-end verification of the mem0 arm re-ran the live comparison and found the appendix pinned the wrong OS. The run machine reports macOS 26.5.2 (Darwin 25.5.0, arm64); the appendix said "macOS 15". Every other pinned knob verified against the venv that produced the table: mem0ai 2.0.17, fastembed 0.8.0, spacy 3.8.13 + en_core_web_sm 3.8.0, ollama 0.6.2, qdrant-client 1.19.0, Python 3.14.6. The appendix stakes its credibility on "the reproducibility is re-derivable rather than merely asserted", so a wrong OS pin is a real defect: it sends anyone reconciling a divergent run after the wrong variable. Also records the architecture, which matters more than most of the knobs already listed. No table cell changes. Re-verified on this machine: - offline suite: 363 passed with all sockets blocked (network egress denied at socket.socket/create_connection/getaddrinfo), import resolved to the worktree's src - `--arm mem0` without mem0 -> exit 2 + install hint; default arm -> exit 0 - default arm byte-identical to main (md5 6bfe357c…) - arm A and arm B blocks both reproduce the committed appendix byte for byte; arm B identical across five fresh runs here (md5 c5a69a0f…) on top of the four already recorded - warm arm-B wall-clock 28.4-30.9 s incl. ~3.8 s import warm-up, consistent with the "~27 s of scenario wall-clock" claim Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B1GNoTx3qTPrb8GQcxooWj
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #15. Implements the frozen Task 5 design (
docs/superpowers/plans/2026-07-29-wp2-update-integrity.md) and publishes a reproduced live run; user go-ahead granted 2026-08-07 (fully local via Ollama).Design conformance:
--arm mem0onmain(); default arm byte-identical to main in both output modes (proven by diff);Mem0Armadapter mapsStep.text→mem0.Memory.add, queries → mem0search; assertion names/order shared between arms via a testedassertion_names()contract; missing mem0 → exit 2 with install hint; header pinsmem0.__version__, LLM/embedder/vector-store config, sampling knobs (temperature/top_p, no seed), and whether mem0's BM25 + entity-boost retrieval halves are installed.The run (2026-08-07, fully local): mem0 2.0.17 on Ollama qwen2.5:3b + nomic-embed-text, local on-disk qdrant,
MEM0_TELEMETRY=false, full retrieval backend (mem0ai[extras]+[nlp]— the review caught the first run silently measuring a handicapped mem0 with BM25 disabled; re-run on the full backend produced identical verdicts, now measured rather than assumed). Result table in thedocs/competitive-landscape.mdappendix; byte-identical across nine runs (zero variance, stated). Honest caveats recorded: qwen2.5:3b backbone confound (three FAILs are pure backbone artefacts and are marked as such), scenario-text selection bias (texts were authored for lean-memory's extractor lexicon), and mem0 OSS having noas_of/timestamp surface (renderedn/a (unsupported)with mem0's own error).Offline discipline, measured: suite green without mem0 installed and with egress hard-blocked via a socket-raising shim — 363 passed (319 + 44 new). Exit codes verified: 0 default, 1 failures-present, 2 missing-dep. Adversarial review: 1 critical + 2 important + 6 minor, all independently re-verified and fixed.
🤖 Generated with Claude Code
https://claude.ai/code/session_01B1GNoTx3qTPrb8GQcxooWj