Pathfinder: synthetic snippet scores make the heuristic-confidence fallback near-constant HIGH

[jerelvelarde]

Spun out of the #112 review (non-blocking).

parseSnippets synthesizes score: Math.max(0.5, 1 - i * 0.05), so results[0].score is always 1.0. ConfidenceScorer.heuristicScore derives from topScore * 0.4 + avgScore * 0.4 + min(count*0.03, 0.2); with the default 8 server-filtered results this lands ~0.93 → HIGH essentially whenever any results return, regardless of real match quality.

Impact is bounded — the primary Claude scorer (reads generated text + snippet content) is unaffected; this only drives the degraded/fallback heuristic path and any pre-filter. But it's a behavior change from the old JSON path, where genuine similarity (0.3–0.92) produced graduated heuristics. The tradeoff was made consciously (results are already server-filtered by min_score), so this is a "revisit if it matters" item, not a bug.

Options: (a) if copilotkit-docs-mcp exposes a numeric relevance anywhere in the payload, thread it through instead of the synthetic rank; (b) otherwise, damp the heuristic's reliance on topScore when scores are synthetic, or flag synthetic-scored results so heuristicScore can treat them differently.

Files: packages/outpost/ai/src/pathfinder.ts (parseSnippets), packages/outpost/ai/src/confidence.ts (heuristicScore).

[copilotkit-outpost-bot]

🎫 Ticket TKT-CLRDBRZ2 created. Our AI assistant is reviewing your issue...

[copilotkit-outpost-bot]

⚠️ This is an AI-generated response and may be incomplete. We've escalated this to our engineering team — someone will follow up in this thread shortly.

Synthetic Score Inflation in the Heuristic Path

This is a well-scoped analysis. Let me break down the problem and the concrete options.

---

The Core Issue

Code example (typescript)

// pathfinder.ts — parseSnippets
score: Math.max(0.5, 1 - i * 0.05)
// results[0].score = 1.0, results[7].score = 0.65 — always, regardless of actual match quality

// confidence.ts — heuristicScore (reconstructed)
topScore * 0.4 + avgScore * 0.4 + Math.min(count * 0.03, 0.2)
// With 8 results: ~1.0*0.4 + ~0.83*0.4 + 0.20 = ~0.93 → HIGH

The heuristic was designed assuming topScore carries semantic signal. It doesn't anymore — it carries rank, which is already implicit in count.

---

Option A — Thread Real Relevance Through (Preferred if Available)

Check whether copilotkit-docs-mcp returns anything usable in the payload — even a coarse tier or a normalized BM25 score:

</details>typescript
// pathfinder.ts
function parseSnippets(rawResults: McpResult[]): Snippet[] {
  return rawResults.map((r, i) => ({
    content: r.content,
    source: r.source,
    // Use server-provided relevance if present, fall back to rank-based synthetic
    score: r.relevance ?? r.score ?? Math.max(0.5, 1 - i * 0.05),
    scoreIsSynthetic: r.relevance == null && r.score == null,
  }));
}
<details><summary>Code example</summary>

Then in heuristicScore, branch on the flag:

// confidence.ts
heuristicScore(snippets: Snippet[]): number {
  const allSynthetic = snippets.every(s => s.scoreIsSynthetic);

  if (allSynthetic) {
    // Scores carry no semantic signal — rely only on count
    const countComponent = Math.min(snippets.length * 0.03, 0.2);
    return 0.55 + countComponent; // anchored lower, max ~0.75 → MEDIUM
  }

  const topScore = snippets[0]?.score ?? 0;
  const avgScore = snippets.reduce((s, r) => s + r.score, 0) / snippets.length;
  return topScore * 0.4 + avgScore * 0.4 + Math.min(snippets.length * 0.03, 0.2);
}

---

Option B — Damp topScore Weight When Scores Are Synthetic

If you don't want to add the flag field, a simpler structural fix: when scores are synthetic, topScore and avgScore are correlated by construction, so their combined 0.8 weight is redundant with count. Redistribute:

Code example (typescript)

// confidence.ts — synthetic-aware rebalancing
heuristicScore(snippets: Snippet[], synthetic = false): number {
  const topScore = snippets[0]?.score ?? 0;
  const avgScore = snippets.reduce((s, r) => s + r.score, 0) / snippets.length;
  const countComponent = Math.min(snippets.length * 0.03, 0.2);

  if (synthetic) {
    // topScore is always 1.0 — don't let it anchor the result high
    return avgScore * 0.5 + countComponent * 2.5; // max ~0.83, more graduated
  }

  return topScore * 0.4 + avgScore * 0.4 + countComponent;
}

This preserves the formula shape while reducing the ceiling from ~0.93 to ~0.83 and making it more sensitive to count variation.

---

Option C — Normalize Synthetic Scores Against a Softer Ceiling

Alternatively, keep parseSnippets as-is but cap synthetic topScore before it enters the formula:

Code example (typescript)

// confidence.ts
const effectiveTop = snippets[0]?.scoreIsSynthetic
  ? 0.7  // treat "ranked first" as moderately confident, not certain
  : (snippets[0]?.score ?? 0);

This is the smallest diff but least principled — the 0.7 constant needs justification.

---

Recommendation

Option	Invasiveness	Signal quality	Notes
A (real relevance)	Medium	Best	Requires MCP payload audit first
B (damp weights)	Low	Good	Safe default if A isn't available
C (cap topScore)	Minimal	Adequate	Magic constant is a smell

Start with A — audit the MCP payload for any numeric relevance field. If nothing is there, ship B as the interim fix. The scoreIsSynthetic flag from A is worth adding regardless, since it makes the behavior explicit and testable:

</details>typescript
// Easy to assert in tests
expect(snippets.every(s => s.scoreIsSynthetic)).toBe(true);
expect(scorer.heuristicScore(snippets)).toBeLessThan(0.80); // no longer locks to HIGH

The key invariant to restore: zero results → LOW, few weak results → MEDIUM, many strong results → HIGH. Right now the middle tier is collapsed.

---

_{🤖 Generated by CopilotKit AI Support · Documentation}

---

Was this helpful? React with 👍 or 👎