feat(v3.0): Phase 51 Retrieval Orchestrator

.planning/REQUIREMENTS.md

@@ -9,14 +9,14 @@ Requirements for the Competitive Parity & Benchmarks milestone. Each maps to roa
 
 ### Retrieval Orchestrator (ORCH)
 
-- [ ] **ORCH-01**: `memory-orchestrator` crate exists with query expansion, RRF fusion, and rerank pipeline
-- [ ] **ORCH-02**: RRF fusion produces different ranking than any single index when scores diverge (unit tested)
-- [ ] **ORCH-03**: Orchestrator returns results when one of the four indexes returns empty (fail-open, unit tested)
-- [ ] **ORCH-04**: LLM rerank mode invokes configured LLM client and reorders results (integration tested with mock)
-- [ ] **ORCH-05**: Cross-encoder reranker extension point stubbed (trait exists, not implemented)
-- [ ] **ORCH-06**: `ContextBuilder` converts ranked results into structured `MemoryContext` with summary, events, entities, tokens
-- [ ] **ORCH-07**: Heuristic query expansion generates lowercase + keyword-stripped variants
-- [ ] **ORCH-08**: Existing `memory-retrieval` crate unchanged — orchestrator wraps `RetrievalExecutor`
+- [x] **ORCH-01**: `memory-orchestrator` crate exists with query expansion, RRF fusion, and rerank pipeline
+- [x] **ORCH-02**: RRF fusion produces different ranking than any single index when scores diverge (unit tested)
+- [x] **ORCH-03**: Orchestrator returns results when one of the four indexes returns empty (fail-open, unit tested)
+- [x] **ORCH-04**: LLM rerank mode invokes configured LLM client and reorders results (integration tested with mock)
+- [x] **ORCH-05**: Cross-encoder reranker extension point stubbed (trait exists, not implemented)
+- [x] **ORCH-06**: `ContextBuilder` converts ranked results into structured `MemoryContext` with summary, events, entities, tokens
+- [x] **ORCH-07**: Heuristic query expansion generates lowercase + keyword-stripped variants
+- [x] **ORCH-08**: Existing `memory-retrieval` crate unchanged — orchestrator wraps `RetrievalExecutor`
 
 ### CLI API (CLI)
 
@@ -63,14 +63,14 @@ Requirements for the Competitive Parity & Benchmarks milestone. Each maps to roa
 
 | Requirement | Phase | Status |
 |-------------|-------|--------|
-| ORCH-01 | Phase 51 | Pending |
-| ORCH-02 | Phase 51 | Pending |
-| ORCH-03 | Phase 51 | Pending |
-| ORCH-04 | Phase 51 | Pending |
-| ORCH-05 | Phase 51 | Pending |
-| ORCH-06 | Phase 51 | Pending |
-| ORCH-07 | Phase 51 | Pending |
-| ORCH-08 | Phase 51 | Pending |
+| ORCH-01 | Phase 51 | Complete |
+| ORCH-02 | Phase 51 | Complete |
+| ORCH-03 | Phase 51 | Complete |
+| ORCH-04 | Phase 51 | Complete |
+| ORCH-05 | Phase 51 | Complete |
+| ORCH-06 | Phase 51 | Complete |
+| ORCH-07 | Phase 51 | Complete |
+| ORCH-08 | Phase 51 | Complete |
 | CLI-01 | Phase 52 | Pending |
 | CLI-02 | Phase 52 | Pending |
 | CLI-03 | Phase 52 | Pending |

.planning/ROADMAP.md

@@ -141,7 +141,7 @@ See: `.planning/milestones/v2.7-ROADMAP.md`
 
 **Milestone Goal:** Close the three gaps that keep Agent-Memory from being the category leader: retrieval pipeline orchestration, a dead-simple CLI API, and a benchmark suite that produces a publishable LOCOMO score.
 
-- [ ] **Phase 51: Retrieval Orchestrator** - Query expansion, RRF fusion, LLM reranking, and context building as a new crate wrapping RetrievalExecutor
+- [x] **Phase 51: Retrieval Orchestrator** - Query expansion, RRF fusion, LLM reranking, and context building as a new crate wrapping RetrievalExecutor (executed 2026-03-22; landing on main via PR)
 - [x] **Phase 51.5: API Summarizer Wiring** - Wire `ApiSummarizer` from config (out-of-band; merged 2026-04-28 via PR #27)
 - [ ] **Phase 52: Simple CLI API** - New `memory` binary with search, context, recall, add, timeline, summary subcommands
 - [ ] **Phase 53: Benchmark Suite** - Custom TOML-fixture harness with LOCOMO adapter and publishable scoring
@@ -161,9 +161,9 @@ See: `.planning/milestones/v2.7-ROADMAP.md`
 **Plans**: 3 plans
 
 Plans:
-- [ ] 51-01-PLAN.md — Scaffold crate, core types, and query expansion
-- [ ] 51-02-PLAN.md — RRF fusion, reranker trait, and context builder
-- [ ] 51-03-PLAN.md — Wire MemoryOrchestrator and full QA
+- [x] 51-01-PLAN.md — Scaffold crate, core types, and query expansion (completed 2026-03-22)
+- [x] 51-02-PLAN.md — RRF fusion, reranker trait, and context builder (completed 2026-03-22)
+- [x] 51-03-PLAN.md — Wire MemoryOrchestrator and full QA (completed 2026-03-22)
 
 ### Phase 51.5: API Summarizer Wiring (Merged)
 **Goal**: Replace the heuristic-only summarizer with a config-driven `ApiSummarizer` so events can be summarized via an Anthropic/OpenAI/etc. API when configured
@@ -223,8 +223,8 @@ Phases execute in numeric order: 51 -> 51.5 (merged out-of-band) -> 52 -> 53
 | v2.5 Semantic Dedup | 35-38 | 11/11 | Complete | 2026-03-10 |
 | v2.6 Cognitive Retrieval | 39-44 | 13/13 | Complete | 2026-03-16 |
 | v2.7 Multi-Runtime Portability | 45-50 | 11/11 | Complete | 2026-03-22 |
-| v3.0 Competitive Parity | 51-53 + 51.5 | 1/TBD | In progress | Phase 51.5 merged 2026-04-28 |
+| v3.0 Competitive Parity | 51-53 + 51.5 | 4/TBD | In progress | Phase 51.5 merged 2026-04-28; Phase 51 landing |
 
 ---
 
-*Updated: 2026-04-27 — Phase 51.5 (API Summarizer Wiring) merged out-of-band via PR #27*
+*Updated: 2026-04-28 — Phase 51 implementation cherry-picked from gsd/phase-51 branch; landing via PR*

.planning/STATE.md

@@ -4,14 +4,14 @@ milestone: v3.0
 milestone_name: Competitive Parity & Benchmarks
 status: in_progress
 stopped_at: null
-last_updated: "2026-04-27T00:00:00.000Z"
-last_activity: 2026-04-27 — Phase 51.5 (API Summarizer Wiring) merged via PR #27
+last_updated: "2026-04-28T00:00:00.000Z"
+last_activity: 2026-04-28 — Phase 51 (Retrieval Orchestrator) cherry-picked from local branch; landing via PR
 progress:
   total_phases: 4
-  completed_phases: 1
-  total_plans: 1
-  completed_plans: 1
-  percent: 25
+  completed_phases: 2
+  total_plans: 4
+  completed_plans: 4
+  percent: 50
 ---
 
 # Project State
@@ -25,12 +25,12 @@ See: .planning/PROJECT.md (updated 2026-03-22)
 
 ## Current Position
 
-Phase: 51.5 of 53 (API Summarizer Wiring — MERGED)
-Plan: out-of-band (no PLAN.md; pre-GSD execution)
-Status: Phase 51.5 merged; Phase 51 still pending
-Last activity: 2026-04-27 — PR #27 merged as squash commit `3a73582`
+Phase: 51 of 53 (Retrieval Orchestrator) — landing via PR
+Plan: 3 of 3 complete (51-01, 51-02, 51-03 all summaries land in this PR)
+Status: Phase 51 + 51.5 both done; Phase 52 next
+Last activity: 2026-04-28 — Cherry-picked 12 commits from gsd/phase-51-retrieval-orchestrator into feature branch
 
-Progress: [██░░░░░░░░] 25% (1 of 4 phases)
+Progress: [█████░░░░░] 50% (2 of 4 phases)
 
 ## Out-of-band Work
 
@@ -63,6 +63,12 @@ See .planning/MILESTONES.md
 - LOCOMO dataset never committed (gitignored)
 - Existing implementation plans in docs/superpowers/plans/ will be converted to GSD plans
 - Phase 51.5 inserted as a decimal phase (out-of-band insertion pattern from `/gsd:insert-phase`) since the summarizer wiring shipped before Phase 51 itself
+- [Phase 51]: RerankMode defaults to Heuristic (no LLM cost by default)
+- [Phase 51]: RankedResult uses f64 for fusion precision, SearchResult uses f32
+- [Phase 51]: RRF deduplicates by doc_id, keeping first-seen SearchResult
+- [Phase 51]: HeuristicReranker trims to top 10 (MAX_RESULTS const)
+- [Phase 51]: Token estimation: chars * 0.75 + 50 overhead
+- [Phase 51]: MemoryOrchestrator accepts Box<dyn Reranker> via with_reranker() for test injection
 
 ## Blockers
 
@@ -91,12 +97,12 @@ See: .planning/MILESTONES.md for complete history
 
 ## Cumulative Stats
 
-- ~56,400 LOC Rust across 15 crates
-- 50 phases, 146 plans across 9 milestones
-- 46+ E2E tests + 144 bats CLI tests
+- ~56,400 LOC Rust across 15 crates + memory-orchestrator (new in Phase 51)
+- 51 phases (50 + Phase 51), 150 plans across 9 milestones (counting Phase 51's 3 plans + 51.5)
+- 46+ E2E tests + 144 bats CLI tests + new orchestrator unit tests
 
 ## Session Continuity
 
-**Last Session:** 2026-04-27
-**Stopped At:** Phase 51.5 merged via PR #27; planning files synced to reflect merge and flag deferred items
+**Last Session:** 2026-04-28
+**Stopped At:** Phase 51 cherry-picked from gsd/phase-51 branch; awaiting PR merge
 **Resume File:** None

.planning/phases/51-retrieval-orchestrator/51-01-SUMMARY.md

@@ -0,0 +1,108 @@
+---
+phase: 51-retrieval-orchestrator
+plan: 01
+subsystem: retrieval
+tags: [orchestrator, query-expansion, rrf, types, crate-scaffold]
+
+requires:
+  - phase: 45-retrieval-policy
+    provides: "RetrievalExecutor, SearchResult, FallbackChain, LayerExecutor trait"
+provides:
+  - "memory-orchestrator crate in workspace"
+  - "OrchestratorConfig, RankedResult, MemoryContext, RerankMode types"
+  - "expand_query heuristic function"
+  - "Stub modules for fusion, rerank, context_builder, orchestrator"
+affects: [51-02-PLAN, 51-03-PLAN]
+
+tech-stack:
+  added: [memory-orchestrator]
+  patterns: [heuristic-query-expansion, orchestrator-types]
+
+key-files:
+  created:
+    - crates/memory-orchestrator/Cargo.toml
+    - crates/memory-orchestrator/src/lib.rs
+    - crates/memory-orchestrator/src/types.rs
+    - crates/memory-orchestrator/src/expand.rs
+    - crates/memory-orchestrator/src/fusion.rs
+    - crates/memory-orchestrator/src/rerank.rs
+    - crates/memory-orchestrator/src/context_builder.rs
+    - crates/memory-orchestrator/src/orchestrator.rs
+  modified:
+    - Cargo.toml
+
+key-decisions:
+  - "RerankMode defaults to Heuristic (no LLM cost by default)"
+  - "RankedResult uses f64 scores for fusion precision, while SearchResult uses f32"
+  - "Query expansion strips 7 question-word prefixes for keyword-biased variants"
+
+patterns-established:
+  - "Orchestrator wraps memory-retrieval types with higher-level abstractions"
+  - "Heuristic query expansion: original + lowercase + keyword-stripped variants"
+
+requirements-completed: [ORCH-01, ORCH-07]
+
+duration: 2min
+completed: 2026-03-22
+---
+
+# Phase 51 Plan 01: Retrieval Orchestrator Crate Scaffold Summary
+
+**memory-orchestrator crate with OrchestratorConfig/RankedResult/MemoryContext types and heuristic query expansion (10 tests)**
+
+## Performance
+
+- **Duration:** 2 min
+- **Started:** 2026-03-22T04:01:03Z
+- **Completed:** 2026-03-22T04:03:17Z
+- **Tasks:** 2
+- **Files modified:** 9
+
+## Accomplishments
+- Scaffolded memory-orchestrator crate with workspace integration (members + deps)
+- Defined 4 core types: OrchestratorConfig, RankedResult, MemoryContext, RerankMode
+- Implemented heuristic query expansion generating lowercase and keyword-stripped variants
+- Created stub modules for fusion, rerank, context_builder, orchestrator
+- 10 unit tests all passing, clippy clean
+
+## Task Commits
+
+Each task was committed atomically:
+
+1. **Task 1: Scaffold crate and define core types** - `7874baa` (feat)
+2. **Task 2: Implement heuristic query expansion** - `7dc22c8` (feat)
+
+## Files Created/Modified
+- `Cargo.toml` - Added memory-orchestrator to workspace members and dependencies
+- `crates/memory-orchestrator/Cargo.toml` - Crate manifest with workspace deps
+- `crates/memory-orchestrator/src/lib.rs` - Public API re-exports
+- `crates/memory-orchestrator/src/types.rs` - OrchestratorConfig, RankedResult, MemoryContext, RerankMode
+- `crates/memory-orchestrator/src/expand.rs` - Heuristic query expansion with 6 tests
+- `crates/memory-orchestrator/src/fusion.rs` - RRF fusion stub
+- `crates/memory-orchestrator/src/rerank.rs` - Reranking stub
+- `crates/memory-orchestrator/src/context_builder.rs` - Context assembly stub
+- `crates/memory-orchestrator/src/orchestrator.rs` - Top-level orchestrator stub
+
+## Decisions Made
+- RerankMode defaults to Heuristic (avoids LLM cost by default)
+- RankedResult uses f64 scores for fusion precision while upstream SearchResult uses f32
+- Query expansion strips 7 question-word prefixes (what, how, why, when, where, did we, do we)
+
+## Deviations from Plan
+
+None - plan executed exactly as written.
+
+## Issues Encountered
+None
+
+## User Setup Required
+None - no external service configuration required.
+
+## Next Phase Readiness
+- Core types ready for fusion (51-02) and context builder (51-03) plans
+- Stub modules in place for incremental implementation
+- All tests pass, clippy clean
+
+---
+*Phase: 51-retrieval-orchestrator*
+*Completed: 2026-03-22*

.planning/phases/51-retrieval-orchestrator/51-02-SUMMARY.md

@@ -0,0 +1,114 @@
+---
+phase: 51-retrieval-orchestrator
+plan: 02
+subsystem: retrieval
+tags: [rrf, fusion, reranking, context-builder, orchestrator]
+
+requires:
+  - phase: 51-retrieval-orchestrator-01
+    provides: "OrchestratorConfig, RankedResult, MemoryContext, RerankMode types"
+provides:
+  - "rrf_fuse function for merging ranked lists from multiple indexes"
+  - "Reranker trait with HeuristicReranker (default) and CrossEncoderReranker (stub)"
+  - "ContextBuilder converting reranked results to MemoryContext"
+affects: [51-retrieval-orchestrator-03, benchmark-suite]
+
+tech-stack:
+  added: []
+  patterns: [reciprocal-rank-fusion, trait-based-reranking, token-estimation]
+
+key-files:
+  created:
+    - crates/memory-orchestrator/src/fusion.rs
+    - crates/memory-orchestrator/src/rerank.rs
+    - crates/memory-orchestrator/src/context_builder.rs
+  modified: []
+
+key-decisions:
+  - "RRF deduplicates by doc_id using first-seen inner result"
+  - "HeuristicReranker trims to top 10 (const MAX_RESULTS)"
+  - "CrossEncoderReranker logs warning and delegates to HeuristicReranker"
+  - "Token estimation: chars * 0.75 + 50 overhead"
+  - "RetrievalLayer import scoped to #[cfg(test)] to satisfy clippy"
+
+patterns-established:
+  - "FusedResult wraps SearchResult with RRF score for pipeline flow"
+  - "Reranker async trait enables pluggable reranking strategies"
+  - "ContextBuilder::build is a pure function (no state needed)"
+
+requirements-completed: [ORCH-02, ORCH-03, ORCH-04, ORCH-05, ORCH-06]
+
+duration: 3min
+completed: 2026-03-22
+---
+
+# Phase 51 Plan 02: Pipeline Components Summary
+
+**RRF fusion, heuristic/cross-encoder reranker trait, and context builder for retrieval orchestrator pipeline**
+
+## Performance
+
+- **Duration:** 3 min
+- **Started:** 2026-03-22T04:05:40Z
+- **Completed:** 2026-03-22T04:08:21Z
+- **Tasks:** 3
+- **Files modified:** 3
+
+## Accomplishments
+- RRF fusion with deduplication, consensus boosting, and empty-list handling (4 tests)
+- Reranker trait with HeuristicReranker (top-10 trim) and CrossEncoderReranker stub (2 tests)
+- ContextBuilder producing MemoryContext with summary, events, token estimate, and confidence (3 tests)
+
+## Task Commits
+
+Each task was committed atomically:
+
+1. **Task 1: Implement RRF fusion** - `e14625c` (feat)
+2. **Task 2: Implement Reranker trait and HeuristicReranker** - `2e28d0f` (feat)
+3. **Task 3: Implement ContextBuilder** - `72efa6a` (feat)
+
+_Clippy fix:_ `ca4a6c9` - moved RetrievalLayer import to test scope
+
+## Files Created/Modified
+- `crates/memory-orchestrator/src/fusion.rs` - RRF fusion: rrf_fuse function and FusedResult type
+- `crates/memory-orchestrator/src/rerank.rs` - Reranker trait, HeuristicReranker, CrossEncoderReranker stub
+- `crates/memory-orchestrator/src/context_builder.rs` - ContextBuilder converting ranked results to MemoryContext
+
+## Decisions Made
+- RRF deduplicates by doc_id, keeping first-seen SearchResult as inner
+- HeuristicReranker uses const MAX_RESULTS = 10 for trimming
+- CrossEncoderReranker logs tracing::warn and delegates to HeuristicReranker
+- Token estimation formula: chars * 0.75 + 50 overhead
+- RetrievalLayer import moved to #[cfg(test)] scope to satisfy clippy -D warnings
+
+## Deviations from Plan
+
+### Auto-fixed Issues
+
+**1. [Rule 1 - Bug] Unused import warning in fusion.rs**
+- **Found during:** Post-task verification (clippy)
+- **Issue:** `RetrievalLayer` imported at module level but only used in tests
+- **Fix:** Moved import to `#[cfg(test)]` module
+- **Files modified:** crates/memory-orchestrator/src/fusion.rs
+- **Verification:** `cargo clippy -p memory-orchestrator -- -D warnings` passes
+- **Committed in:** ca4a6c9
+
+---
+
+**Total deviations:** 1 auto-fixed (1 bug)
+**Impact on plan:** Minor import scoping fix. No scope creep.
+
+## Issues Encountered
+None
+
+## User Setup Required
+None - no external service configuration required.
+
+## Next Phase Readiness
+- All three pipeline components (fusion, rerank, context_builder) ready for wiring into MemoryOrchestrator in plan 03
+- 9 new tests added (4 fusion + 2 rerank + 3 context_builder), total 19 in crate
+- Zero clippy warnings
+
+---
+*Phase: 51-retrieval-orchestrator*
+*Completed: 2026-03-22*