feat(v3.0): Phase 53 Benchmark Suite (closes v3.0)#30
Merged
Conversation
- 53-03: fix wave 2->3, add 53-02 to depends_on (dependency_correctness blocker) - 53-02: add compute_compression_ratio + estimate_raw_tokens to scorer.rs task (task_completeness warning) - 53-03: add .github/workflows/ci.yml benchmark-smoke job with continue-on-error: true (requirement_coverage BENCH-07) Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
- Add memory-bench crate to workspace with clap, serde, toml deps - Create minimal binary entrypoint and lib.rs with fixture module - Add benchmarks/baselines.toml with MemMachine and Mem0 scores - Create download-locomo.sh script for LOCOMO dataset - Add 7 stub JSONL session files for benchmark fixtures - Gitignore locomo-data/ directory Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Define Fixture and TestCase structs with serde deserialization - Implement Fixture::load with id/query validation - Implement Fixture::load_dir to collect test cases from directory - Add 4 unit tests: parse valid, empty id, empty query, load_dir - Create 3 TOML fixture files: temporal, multisession, compression Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Add 53-01-SUMMARY.md with execution results - Update STATE.md position to plan 2 of 3 - Update ROADMAP.md progress for phase 53 - Mark BENCH-01 and BENCH-06 requirements complete Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Scorer: accuracy, recall@k, percentile, compression ratio, token estimation - Runner: shells out to memory binary for search queries and session ingestion - Report: JSON and markdown table output with optional competitor comparison - Baseline: TOML loader for competitor scores - 18 tests passing, clippy clean Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- CLI: temporal, multisession, compression, all, locomo subcommands - Global --memory-bin flag for binary path override - All subcommand: --compare flag for baseline side-by-side table - Full pipeline: load fixtures, ingest sessions, run queries, score, report Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Add locomo.rs with LocomoConversation, Turn, Question structs - Implement load_dataset to read JSON files from directory - Implement score_conversation with case-insensitive substring matching - Implement aggregate_results with per-type breakdown - Add 6 unit tests covering parsing, scoring, aggregation, and loading - Register pub mod locomo in lib.rs Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
…l QA - Wire Commands::Locomo in main.rs with dataset ingestion and scoring - Move tempfile from dev-dependencies to dependencies for runtime use - Add benchmark-smoke job to CI with continue-on-error: true - Format all files to pass cargo fmt check - Full pr-precheck passes (fmt, clippy, test, doc) Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Create 53-03-SUMMARY.md with execution results - Update STATE.md: phase 53 complete, 148 plans total - Update ROADMAP.md: phase 53 progress to complete - Mark BENCH-04 and BENCH-07 requirements complete Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
RichardHightower
had a problem deploying
to
e2e-cli
GitHub Actions
Failure
RichardHightower
had a problem deploying
to
e2e-cli
GitHub Actions
Failure
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 53 of v3.0 — the final phase of the Competitive Parity & Benchmarks milestone. Adds a benchmark harness for measuring retrieval quality, including a LOCOMO adapter that produces publishable scores.
What ships
A new
memory-benchcrate (binary namememory-bench) with:temporalmultisessioncompressionalllocomoModule breakdown
fixture.rs— TOML fixture loader with schema validationrunner.rs— drives amemory-clibinary against the fixture's setup/queriesscorer.rs— produces per-query scores (precision/recall/MRR depending on suite)report.rs— markdown + JSON output, baseline-comparison diffsbaseline.rs— store and load reference scores for regression detectionlocomo.rs— LOCOMO dataset adapter (parses the public eval format)Design decisions
memory(Phase 52's CLI) over the daemon's gRPC, not the daemon directly. Lets us benchmark the actual user-facing surface.--dataset-path. Avoids licensing/size issues.memory-bench all --dry-runto ensure the harness compiles and starts; it does NOT run the full LOCOMO scoring (would take minutes and require external data).--baseline path/to/baseline.jsonmakes the runner emit a diff report, ready for regression CI later.Files added
crates/memory-bench/— new binary crate, 9 source files.planning/phases/53-benchmark-suite/— 3 plans + 3 summaries + context/research/validation/verificationTests
task test): 58 test blocks, 0 failures (up from 55 pre-Phase-53 — picked up 3 new memory-bench test blocks)task clippy): zero warnings on--workspace --all-targets --all-features -- -D warningstask doc): zero rustdoc warnings underRUSTDOCFLAGS=-D warningstask fmtpassesRebase notes (2026-05-14)
Rebased from
gsd/phase-53-benchmark-suite(local stack, accumulated through stacked phases) onto post-Phase-53.5origin/main. Usedgit rebase --onto origin/main 2c7e1fe 2733233to drop all Phase 51 + 52 commits already on main. 13 commits in this PR are Phase 53's own work plus one final clean planning-doc update.What this closes
v3.0 Competitive Parity & Benchmarks ships on merge — Phases 51, 51.5, 52, 53.5 already on main; Phase 53 is the final piece. After merge, the milestone is shipped (5 phases, 8 plans counting decimals).
Next
v3.1 work is on
gsd/phase-{54..56}locally (export/backup/import). v3.2 work is ongsd/phase-{57..58}(runtime registration metadata). Both stacks are backed up to origin; landing strategy still TBD.