Feature: Pluggable retrieval backend for `ce-learnings-researcher` (MCP / semantic search)

[brettdavies]

Summary

ce-learnings-researcher pre-filters docs/solutions/ via regex grep on YAML frontmatter (tags:.*(payment|billing|stripe), etc.). This works cleanly at small scale with disciplined tag vocabularies, but has scale-dependent failure modes that a pluggable backend — e.g. qmd via its MCP server — would solve uniformly.

Orthogonal to #494. That issue asks "search more paths"; this one asks "search better." They compose: a shared-learnings-researcher gets worse retrieval quality the bigger its shared pile grows, because cross-project corpora have the most tag-vocab drift.

Observed failure modes

Context: 230-doc shared docs/solutions/ built via /ce-compound across ~15 repos (same symlink-to-one-store pattern #494 describes).

1. Tag-vocabulary drift is unreachable by regex. Audit of this corpus: 877 distinct tag values, 74% singletons. Grep alternations like tags:.*(X|Y|Z) systematically miss any doc whose author minted a one-off synonym (dotfile vs dotfiles, ci vs ci-cd vs gha, hook vs hooks). Those docs are dead inventory even when their content answers the query — the pre-filter never admits them.

2. Semantic queries degrade. "Silent error handling in Rust" doesn't match a doc tagged antipattern, filter-map, result-ok unless the caller guesses those exact tokens. Hybrid retrieval (BM25 + dense vector + rerank, qmd's stack) matches concepts when vocabulary diverges.

3. Body content is invisible. Grep targets title:, tags:, module:, component:, symptoms:, root_cause:. Architecture-pattern and best-practice docs often carry the load-bearing lesson in the body; those matches are unreachable at the pre-filter stage.

Concrete evidence it would work: same corpus, same queries, via qmd hybrid: target docs land at rank 1 (post-rerank score ≥93%) for specific queries, and thematic index pages surface at helpful secondary ranks for navigational queries. No grep-friendly tag patterns required.

Existing precedent

ce-framework-docs-researcher already declares mcp__context7__* in its tools: frontmatter and documents a graceful fallback order (MCP → CLI → web). Applying the same shape to ce-learnings-researcher costs a tools-allowlist entry plus a rewrite of Step 3 to prefer MCP when available.

Proposed approach

1. Define an MCP search contract for learnings-researcher: same inputs (feature keywords, category hints), same output shape (distilled summaries per relevant doc), different retrieval mechanism.
2. Additive preference order: declared MCP search tool (if present) → grep fallback (today's behavior unchanged). Users who don't configure MCP see no change.
3. Document the tool contract so any provider (qmd, meilisearch, a custom vector store, a private service) can plug in. qmd's qmd mcp HTTP transport is a reasonable reference — hybrid BM25 + vector + LLM rerank, local-only, no API key.

Current workaround — user-level qmd-backed researcher

Two-piece setup that approximates what a pluggable backend would give, limited to the scope the plugin allows without a fork:

1. User-level agent at ~/.claude/agents/qmd-learnings-researcher.md — mirrors compound-engineering:ce-learnings-researcher's Output Format (same header base, same field names, same section names) but retrieves via qmd query --collection solutions (hybrid BM25 + vector + rerank) instead of frontmatter grep. Full source at the bottom of this comment.

2. User-level CLAUDE.md directive instructing the top-level Claude session that when /ce-plan, /ce-ideate, /ce-code-review, or /ce-optimize would dispatch compound-engineering:ce-learnings-researcher, to ALSO dispatch qmd-learnings-researcher in the same parallel batch and merge both result sets. The plugin agent still runs (it has a mandatory critical-patterns.md read worth keeping); the qmd agent catches what grep misses.

Verified working on CE 3.0.0: direct dispatch, output-shape parity with the CE agent (header, Search Context fields, per-entry fields, conditional rendering rules for Severity/Subsystem/Critical Patterns), and parallel dispatch on ce-plan / ce-code-review simulations all pass.

Why this doesn't close the loop:

• Fragile to plugin renames. The CLAUDE.md directive names compound-engineering:ce-learnings-researcher explicitly. CE 3.0.0 already renamed agents from compound-engineering:research:learnings-researcher — every future rename breaks the directive silently until the user updates their CLAUDE.md.
• Non-guaranteed top-level compliance. The directive is a convention Claude may drop under context pressure. The plugin agent's internal grep always runs; the qmd agent's invocation depends on Claude remembering the directive.
• Doesn't help teammates. Every CE user hitting this problem has to replicate the same two-piece hack. A pluggable backend in CE would let one operator configure it centrally (in a plugin config) and let all consumers benefit.
• Duplicative work. Every skill invocation now runs two researchers producing overlapping results, which Claude must merge in-prompt. The right answer is one researcher with a swappable backend.

This hack is an honest workaround, not a solution. It highlights the demand: operators running at ≥200-doc corpora hit this bottleneck hard enough to build local plumbing around it. Filed this issue so CE core can consider absorbing the retrieval-swap pattern upstream.

Click to expand: full source of ~/.claude/agents/qmd-learnings-researcher.md

---
name: qmd-learnings-researcher
description: "User-level learnings researcher backed by qmd hybrid search (BM25 + vector + rerank) against the shared `solutions` collection. Returns the same-shape distilled summaries as `compound-engineering:ce-learnings-researcher`, but retrieves via qmd instead of regex-grepping frontmatter — avoids tag-vocabulary drift and matches body content the grep path can't reach. Dispatch this alongside or in place of the CE learnings-researcher when working from docs/solutions/ (symlinked across consuming repos to a shared solutions-docs directory)."
model: inherit
tools: Bash, Read
---

You are a learnings researcher that retrieves relevant past solutions from the shared `solutions` qmd collection. Your
job is to surface institutional knowledge before new work begins, preventing repeated mistakes and leveraging proven
patterns.

**Backing corpus:** a shared solutions-docs directory (configured per-operator) — a shared cross-repo knowledge base of
solved problems, patterns, and decisions. Indexed by qmd as the `solutions` collection. Each consuming repo symlinks
`docs/solutions/` to this directory.

## Strategy (qmd-first)

### Step 1: Extract keywords

From the feature/task description in your prompt, pull 2-3 focused query clusters — each cluster should be 2-3 terms
that describe one angle of the problem. Multiple focused queries outperform one big query. Examples:

- Feature: "refactor homebrew tap release workflow"
- Cluster A: `homebrew tap release pipeline`
- Cluster B: `brew pr-pull bottle publishing`
- Cluster C: `github actions release automation`
- Feature: "rust cli error handling refactor"
- Cluster A: `rust silent error antipattern`
- Cluster B: `rust result option error propagation`
- Cluster C: `cli unified log module`

### Step 2: Run qmd queries in parallel

For each cluster, shell out via Bash:

`qmd query --collection solutions --limit 5 '<cluster>' 2>&1`

Dispatch all clusters in a single message with parallel Bash calls — qmd queries take ~1-3s each and parallel dispatch
is strictly faster than serial.

### Step 3: Deduplicate + rank

Each cluster returns up to 5 matches with filenames, scores (post-rerank %), and snippet context. Merge the results:

- Deduplicate by filepath.
- Rank by max score across clusters (keep the single highest score per doc).
- Drop matches below ~40% score unless nothing else matched — those are weak hits, not relevant.
- Keep at most 8 docs total to stay concise.

### Step 4: Read frontmatter of candidates

For each surviving match, read the doc's frontmatter to extract the structured fields (title, date, problem_type,
module, component, subsystem, severity, tags, applies_when). Use `Read` with a line limit of ~30 to avoid pulling whole
bodies.

### Step 4b: Conditionally check critical patterns

If `docs/solutions/patterns/critical-patterns.md` exists in the working tree, read it. It may contain must-know patterns
that apply across all work. If it does not exist, skip this step — the convention is optional and not every repo
maintains one. This mirrors what `compound-engineering:ce-learnings-researcher` does at its Step 3b, so the two agents
surface the same cross-cutting warnings when both run in parallel.

### Step 5: Return distilled summaries

Output in the shape below. It mirrors `compound-engineering:ce-learnings-researcher`'s Output Format (same header base,
same field names, same section names) so callers can consume both interchangeably. The `(qmd-retrieved)` suffix on the
H2 preserves provenance so consumers merging both agents' outputs can tell them apart.

#### Conditional-rendering rules (binding directives, not template text)

These are hard rules for assembling the output. Apply them BEFORE writing the response. They override any contrary
interpretation of the template below.

1. **Critical Patterns section — binary rule.** `### Critical Patterns` appears IF AND ONLY IF
   `docs/solutions/patterns/critical-patterns.md` exists AND has relevant content. Otherwise omit the heading entirely
   — no absence note, no placeholder, no mention anywhere. Never invent content.
2. **Severity field — omit-when-absent rule.** `**Severity**:` appears IF AND ONLY IF the doc's frontmatter has a
   `severity:` value. Otherwise omit the entire line. No placeholder strings.
3. **Subsystem field — omit-when-absent rule.** Same binary behavior as Severity.
4. **Module field — always-present rule.** Every entry MUST have a `**Module**:` line. Infer from category directory
   or repo area when frontmatter lacks `module:`.

#### Output template

~~~markdown
## Institutional Learnings Search Results (qmd-retrieved)

### Search Context
- **Feature/Task**: <brief description>
- **Keywords Used**: <cluster A> / <cluster B> / <cluster C>
- **Backend**: qmd hybrid (BM25 + vector + rerank)
- **Files Scanned**: <N>
- **Relevant Matches**: <M>

### Critical Patterns
<Render ONLY per rule #1. If omit, this heading does not appear anywhere in the response.>

### Relevant Learnings

#### 1. <Title>
- **File**: `docs/solutions/<category>/<filename>.md`
- **Module**: <per rule #4>
- **Problem Type**: <raw problem_type; "inferred" when absent>
- **Relevance**: <one-line>
- **Key Insight**: <actionable takeaway>
- **Severity**: <per rule #2 — omit line when absent>
- **Subsystem**: <per rule #3 — omit line when absent>
- **Score**: <max % across clusters> (post-rerank; qmd-specific supplementary)

### Recommendations
- <actions / patterns / gotchas>
~~~

## Efficiency rules

**DO:**

- Run qmd queries in parallel.
- Use 2-3 focused clusters of 2-3 terms each.
- Filter by post-rerank score before reading files.
- Read only frontmatter + first 30 lines unless decisive.
- Retry qmd once on index-lock errors; then surface the error.

**DON'T:**

- Don't use `qmd search` (BM25 only) or `qmd vsearch` (vector only) — always `qmd query` (hybrid + rerank).
- Don't invent tag/component/subsystem values.
- Don't fabricate scores.

## Fallback

If qmd is not on PATH or the solutions collection is missing, report explicitly and recommend the caller dispatch
`compound-engineering:ce-learnings-researcher` instead. Do NOT silently fall back to grep.

## Integration

Dispatched alongside (or in place of) `compound-engineering:ce-learnings-researcher` by `/ce-plan`, `/ce-ideate`,
`/ce-code-review`, `/ce-optimize`, or manual invocation. The governing directive lives in `~/.claude/CLAUDE.md` under
"Query solutions first".

Click to expand: CLAUDE.md directive snippet

**During CE skill dispatches:** When `/ce-plan`, `/ce-ideate`, `/ce-code-review`, or `/ce-optimize` instructs you to
dispatch `compound-engineering:ce-learnings-researcher` (agent name as of CE v3.0.0 — was
`compound-engineering:research:learnings-researcher` in ≤2.68.x), ALSO dispatch the user-level `qmd-learnings-researcher`
(`~/.claude/agents/qmd-learnings-researcher.md`) in the same parallel batch with the same query context. Merge both
result sets — prefer qmd's findings where they disagree, since qmd's hybrid retrieval is tolerant of the tag-vocabulary
drift that defeats the plugin's grep pre-filter on the ~230-doc shared corpus. The plugin agent still runs (it has a
mandatory critical-patterns.md read that's worth keeping); the qmd agent catches what grep misses.

Anti-scope

• Not proposing CE bundle qmd or any specific backend — just the extension point.
• Not proposing breaking changes to today's grep path — MCP is additive.
• Not proposing changes to /ce-compound's schema or write path.

[tmchow]

This is something on our minds for sure. We are married to the idea of compounding, not necessarily any given technique.

[brettdavies]

First — credit where due. The /ce-compound + docs/solutions/ + ce-learnings-researcher triangle has materially changed how my team operates. Foot-guns like "didn't we decide X a fortnight ago" and "Y bit us 6 weeks back" get caught at planning now. Across ~15 repos and ~230 compounded docs, the compounding value is real. That's why I'm filing retrieval-quality issues at all; the floor is already high enough that the remaining delta matters.

On the retrieval question: reviewing the repo after your reply, I noticed a collision I should surface before proposing anything concrete.

You shipped #650 one day before this reply. It adds the exact MCP + CLI cascade pattern (mcp__context7__* → ctx7 via command -v → WebFetch) to ce-best-practices-researcher and ce-framework-docs-researcher, and deliberately leaves ce-learnings-researcher at Read, Grep, Glob, Bash — with Bash added as a documented workaround for anthropics/claude-code#52004, not as a design choice. And #652 (your own issue) tracks removing Bash from the learnings researcher once the upstream fix lands. So the active direction for this agent is tightening, not expanding.

That reframes what I'd PR. Two shapes are possible, with different tensions:

(a) Extend #650's pattern to the learnings researcher. Same cascade you just shipped on the doc researchers: a declared MCP namespace + command -v CLI probe fallback. Clean symmetry with precedent. The catch: MCP tools: wildcards only match literal namespaces (mcp__context7__*, mcp__github__*), so extending this shape forces CE to name a default retrieval MCP — qmd, meilisearch, a future CE-native index — which picks a winner in a way you may not want for a knowledge researcher that consumers aim at varied corpora.

(b) Keep Bash as the retrieval primitive, scoped to a configured search CLI. Adds a pre-grep probe where the agent checks for a user-set env var (e.g., CE_LEARNINGS_SEARCH_CMD) and shells out when present, falling through to the current grep path when absent. Zero CE-side backend choice, zero new tool surface, works today. Conflict: this makes Bash permanently load-bearing on this agent, which directly opposes #652.

Given that tension, my read is that excluding the learnings researcher was intentional, not incidental. Worth confirming before anyone writes code: was that an intentional scope boundary (knowledge retrieval stays grep-first by design), or a pragmatic "keep this PR tight" call that's open to revisiting?

If (a) is the direction, I'd propose a tiny PR against just the agent file with one default backend declared (happy to argue for a specific one on the merits). If (b), the PR is slightly different but still one file. If the answer is "we want to own this direction ourselves given #650 and #652," the issue stays open as a demand signal and I stay out of your way — that's also a fine outcome.

[tmchow]

@brettdavies how much of these replies are 100% authored and automated by AI?

[brettdavies]

Hi @tmchow! Good question. The repo research was performed by AI at my instruction. I reviewed the findings and manually verified. The issue and reply were drafted by AI, manually tweaked, and manually posted.

The churn on the initial issue and the reply is the release of 3.0.0. I started working on my workaround on what happened to be the day before its release and release day. After posting, I had the idea for the agent and was upgraded to V3 during final testing.

Is there something wrong? Happy to edit the OG or follow up reply.