Add task-aware quality routing

CHANGELOG.md

@@ -7,6 +7,11 @@ All notable changes to this project are documented here. The format is based on
 ## [Unreleased]
 
 ### Added
+- Provider-neutral task-aware quality routing for grounded Markdown reading,
+  including a local classifier, explicit CLI/MCP/proxy task hints, a faithful
+  reader role, sanitized versioned regression fixture, and exact-identity
+  multi-trial evidence that preserves existing ordering when no candidate is
+  measured.
 - Deterministic, quota-bounded per-model protocol conformance canaries for
   chat, streaming, tools, JSON object/schema, vision, Responses, and Anthropic Messages,
   with sanitized persisted evidence, model/status visibility, protected

README.md

@@ -236,13 +236,15 @@ freellmpool ask -m gpt-4o-mini "hi"      # routed to a free model
 ### Roles
 
 `freellmpool roles` lists ask-role presets (`coder`, `critic`, `summarizer`,
-`long-context`, `cheap`, `fast`, `second-opinion`, ...). Each role sets routing,
-token budget, temperature, and system-prompt hints without inventing a second
-routing engine. Explicit flags (`--model`, `--providers`, `--routing`, `--max-tokens`)
-win over role defaults, and the verbose output shows when an override happened.
+`grounded-reader`, `long-context`, `cheap`, `fast`, `second-opinion`, ...). Each
+role sets routing, token budget, temperature, task intent, and system-prompt hints
+without inventing a second routing engine. Explicit flags (`--model`, `--providers`,
+`--routing`, `--task`, `--max-tokens`) win over role defaults.
 
 ```bash
 freellmpool ask --role coder "write a pytest for this function"
+freellmpool ask --role grounded-reader "read this Markdown file"
+freellmpool ask --routing quality --task general "ignore automatic task classification"
 FREELLMPOOL_MODE=wise freellmpool ask --role cheap "summarize this patch"
 ```
 
@@ -562,9 +564,21 @@ have the smallest daily caps, so a naive pool gets weaker as the day fills. Qual
 routing matches each prompt's *difficulty* to each model's *capability*: hard
 prompts (long input, code, reasoning cues) go to the strongest available model, and
 easy ones go to lightweight models — which rations scarce strong-model quota so the
-pool stays sharp for longer. Capability is grounded in real benchmark data, not
-guessed from names; models that no benchmark lists cover fall back to a name
-heuristic.
+pool stays sharp for longer. It also recognizes high-confidence grounded Markdown
+reading/extraction requests locally. When repeated evidence from the versioned,
+sanitized fixture exists for a candidate's exact model identity, a bounded task-fit
+term influences quality ordering. Unmeasured models remain reachable, and if no
+candidate has current evidence the ordering is unchanged. Capability is grounded in
+real benchmark data, not guessed from names; models that no benchmark lists cover
+fall back to a name heuristic.
+
+Use `--task grounded-reading` to declare that intent, `--task general` to suppress
+automatic classification, or `--task auto` to classify locally. OpenAI-compatible,
+Responses, and Anthropic proxy clients can send the body extension
+`"task": "grounded-reading"` or the `X-Freellmpool-Task` header. Explicit intent
+wins over automatic classification. Task evidence stores aggregate pass counts,
+fixture hashes, and scores only—never prompts, documents, responses, or provider
+secrets.
 
 The bundled, offline scores come from [LMArena](https://lmarena.ai/) Elo (an
 MIT-licensed snapshot) and the [Aider](https://aider.chat/) code-editing

docs/AGENTS.md

@@ -154,7 +154,9 @@ freellmpool profile doctor metaswarm --dry-run
 ## Workflows that help agents
 
 - `freellmpool roles` shows role presets (`coder`, `critic`, `summarizer`,
-  `second-opinion`, ...).
+  `grounded-reader`, `second-opinion`, ...).
+- `freellmpool ask --role grounded-reader` requests faithful extraction, while
+  explicit `--task general` suppresses automatic grounded-reading classification.
 - `freellmpool ask --role coder --second-opinion` can review an implementation
   plan before a long agent run.
 - `freellmpool battle "which prompt version is clearer?"` compares model answers

docs/MCP.md

@@ -16,7 +16,7 @@ and `stdout` is reserved for the protocol.
 
 | Tool | What it does |
 |---|---|
-| `free_llm_ask` | Ask a free model (`prompt`, optional `system` / `model` / `provider` / `routing` / `max_tokens`). The reply names the serving model. |
+| `free_llm_ask` | Ask a free model (`prompt`, optional `system` / `model` / `provider` / `routing` / `task` / `max_tokens`). `task` accepts `auto`, `general`, or `grounded-reading`. The reply names the serving model. |
 | `free_llm_panel` | Ask the **same** prompt to 2-5 different free models at once and compare. Optional `synthesize` merges them into one best answer; synthesis failure leaves the individual answers visible. |
 | `free_llm_second_opinion` | Agent-facing second-opinion surface. Same panel behavior as `free_llm_panel` — exposed as its own tool so callers can declare intent (`prompt`, `n`, `synthesize`, `routing`, `max_tokens`). |
 | `free_llm_battle` | Bounded multi-model comparison rendered as a Markdown table (`prompt`, `n`, `synthesize`, `routing`, `max_tokens`). Per-model failures stay visible in the output. |
@@ -25,7 +25,7 @@ and `stdout` is reserved for the protocol.
 | `free_llm_tailnet_info` | Show safe Tailscale Tailnet connection instructions for serving the proxy on another machine. Output NEVER contains a real local bearer token (uses a `<proxy-key>` placeholder) and never leaks provider API keys. Degrades cleanly when `tailscale` is absent. Optional `port` (default 8080). |
 | `free_llm_quota_wise` | Local quota-mode / headroom advice from local counters only. Output NEVER recommends account rotation, rate-limit bypass, or automatic paid fallback — only "wait for UTC reset", "lower fan-out/token budget", or an explicit paid choice outside the default flow. |
 | `tokenmax` | 🌈 Gloriously excessive: blast the prompt to **every** free model across **every** provider at once, then the **calling** model synthesizes them all. Emits live `notifications/progress` (`🌈 TOKENMAXXING ▸ 47/168 models…`) so hosts like Claude Code show it ticking up, and a colorful rainbow banner in the result. Tongue-in-cheek, genuinely useful for hard questions. |
-| `free_llm_route` | Explain where a prompt **would** route (estimated difficulty + ranked candidate models) **without spending a token**. |
+| `free_llm_route` | Explain where a prompt **would** route (estimated difficulty, resolved task, and ranked candidate models/evidence) **without spending a token**. |
 | `free_llm_models` | List available `provider/model` ids. |
 | `free_llm_quota` | Today's per-provider usage + daily-limit headroom, plus session totals and estimated cost avoided. |
 | `free_llm_stats` | Lifetime tokens served free + estimated cost avoided vs Claude Opus 4.8 (persists across restarts). |

pyproject.toml

@@ -89,6 +89,7 @@ packages = ["src/freellmpool"]
 [tool.hatch.build.targets.wheel.force-include]
 "src/freellmpool/providers.toml" = "freellmpool/providers.toml"
 "src/freellmpool/capability_scores.json" = "freellmpool/capability_scores.json"
+"src/freellmpool/task_evidence.json" = "freellmpool/task_evidence.json"
 "src/freellmpool/recipes/second-opinion.json" = "freellmpool/recipes/second-opinion.json"
 "src/freellmpool/recipes/pr-review.json" = "freellmpool/recipes/pr-review.json"
 "src/freellmpool/recipes/repo-summary.json" = "freellmpool/recipes/repo-summary.json"

src/freellmpool/aio.py

@@ -40,6 +40,7 @@
 from .observe import emit
 from .router import Pool, _is_account_quota_exhaustion, _is_health_failure
 from .routing_modes import normalize_routing_mode
+from .task_quality import TASK_GENERAL, resolve_task, validate_task
 
 #: An async transport: ``await apost(url, headers, json_body, timeout) -> HTTPResult``.
 AsyncPostFn = Callable[[str, dict, dict, float], Awaitable["_client.HTTPResult"]]
@@ -382,6 +383,7 @@ async def achat(
         response_format=None,
         protocol: str | None = None,
         routing: str | None = None,
+        task: str | None = None,
     ) -> Reply:
         """Async failover completion — same routing/cache/metrics as :meth:`Pool.chat`.
 
@@ -392,6 +394,11 @@ async def achat(
             raise NoProvidersConfigured("no provider has an API key set")
         provider_list = list(providers) if providers else None
         eff = normalize_routing_mode(routing, p.routing)
+        if eff == "quality":
+            resolved_task = resolve_task(messages, task)
+        else:
+            validate_task(task)
+            resolved_task = TASK_GENERAL
         features = required_features(
             messages,
             tools=tools,
@@ -421,6 +428,7 @@ async def achat(
                 eff,
                 response_format=response_format,
                 protocol=protocol,
+                task=resolved_task,
             )
             hit = await asyncio.to_thread(p._cache.get, cache_key)  # blocking sqlite off-loop
             feature_cache_eligible = (
@@ -457,6 +465,7 @@ async def achat(
             candidates,
             difficulty,
             eff,
+            resolved_task,
         )
         if not targets:
             raise NoProvidersConfigured("no candidate (provider, model) matched the given filters")

src/freellmpool/cache.py

@@ -4,8 +4,8 @@
 or ``[settings] cache_ttl`` in config.toml. Handy for dev/test loops where the
 same prompts run repeatedly: it saves quota and answers instantly.
 
-Keyed on a hash of (messages, model, providers, max_tokens, temperature, tools),
-so only *identical* requests hit the cache. Standard-library sqlite3, no deps.
+Keyed on a hash of the request and routing/task intent, so only *identical*
+requests hit the cache. Standard-library sqlite3, no deps.
 """
 
 from __future__ import annotations
@@ -75,6 +75,7 @@ def make_key(
         routing=None,
         response_format=None,
         protocol=None,
+        task=None,
     ) -> str | None:
         try:
             payload = json.dumps(
@@ -91,6 +92,9 @@ def make_key(
                     "routing": routing,
                     "response_format": response_format,
                     "protocol": protocol,
+                    # Explicit task intent can select a different model under the
+                    # same routing mode and therefore needs its own cache bucket.
+                    "task": task,
                 },
                 sort_keys=True,
             )

src/freellmpool/cli.py

@@ -55,6 +55,7 @@
 from .router import Pool
 from .routing_modes import PUBLIC_ROUTING_ALIASES, routing_override
 from .savings import format_saved
+from .task_quality import TASK_HINTS
 
 
 def _read_stdin() -> str:
@@ -139,6 +140,7 @@ def cmd_ask(args: argparse.Namespace) -> int:
     if args.routing is None and routing is None and role is None and args.mode == "normal":
         if not has_routing_config:
             routing = "fair"
+    task = args.task if args.task is not None else (role.task if role is not None else None)
 
     second_opinion = bool(args.second_opinion or (role is not None and role.name == "second-opinion"))
     if second_opinion:
@@ -156,6 +158,7 @@ def cmd_ask(args: argparse.Namespace) -> int:
             max_tokens=max_tokens,
             timeout=args.timeout,
             synthesize=args.synthesize,
+            task=task,
         )
         if not result.answers:
             print("freellmpool: no providers configured", file=sys.stderr)
@@ -168,7 +171,13 @@ def cmd_ask(args: argparse.Namespace) -> int:
         if system:
             messages.append({"role": "system", "content": system})
         messages.append({"role": "user", "content": prompt})
-        targets = pool.rank_targets(messages, routing=routing, model=model_filter, providers=provider_filter)
+        targets = pool.rank_targets(
+            messages,
+            routing=routing,
+            model=model_filter,
+            providers=provider_filter,
+            task=task,
+        )
         snapshot = pool.quota.snapshot()
         if declared_quota_exhausted(targets, snapshot):
             print(
@@ -192,6 +201,7 @@ def cmd_ask(args: argparse.Namespace) -> int:
             temperature=temperature,
             timeout=args.timeout,
             routing=routing,
+            task=task,
         )
     except NoProvidersConfigured as exc:
         print(f"freellmpool: {exc}", file=sys.stderr)
@@ -2002,6 +2012,11 @@ def build_parser() -> argparse.ArgumentParser:
         choices=PUBLIC_ROUTING_ALIASES,
         help="routing mode override (auto uses the pool default)",
     )
+    p_ask.add_argument(
+        "--task",
+        choices=TASK_HINTS,
+        help="task hint for quality routing (auto classifies locally)",
+    )
     p_ask.add_argument(
         "--mode",
         choices=["normal", "wise"],

src/freellmpool/mcp_server.py

@@ -64,6 +64,13 @@
     generate_session_token_simple,
     safe_base_url,
 )
+from .task_quality import (
+    TASK_GENERAL,
+    TASK_HINTS,
+    model_task_score,
+    task_evidence_table,
+    task_resolution,
+)
 from .tokenmax import HARD_CAP, RAINBOW_BANNER, fan_out, select_targets
 
 _DEFAULT_PROTOCOL = "2025-06-18"
@@ -116,6 +123,11 @@
                     "enum": list(PUBLIC_ROUTING_ALIASES),
                     "description": "How to pick the model: agent (strongest healthy tier with quota spreading), quality (capability matched to prompt), spread (whole-pool breadth), fast (lowest latency), fair (provider balance), or auto (server default).",
                 },
+                "task": {
+                    "type": "string",
+                    "enum": list(TASK_HINTS),
+                    "description": "Optional task hint for quality routing; auto classifies locally.",
+                },
                 "max_tokens": {
                     "type": "integer",
                     "description": "Max output tokens (default 1024).",
@@ -378,6 +390,11 @@
                     "enum": list(PUBLIC_ROUTING_ALIASES),
                     "description": "Routing mode to explain (default: the server's mode).",
                 },
+                "task": {
+                    "type": "string",
+                    "enum": list(TASK_HINTS),
+                    "description": "Optional task hint to explain.",
+                },
             },
             "required": ["prompt"],
         },
@@ -505,6 +522,7 @@ def _tool_ask(pool: Pool, args: dict) -> dict:
             providers=providers,
             routing=routing,
             max_tokens=_max_tokens(args.get("max_tokens"), 1024),
+            task=args.get("task"),
         )
     except Exception as exc:  # noqa: BLE001 — surface as a tool error
         return _text(f"{type(exc).__name__}: {exc}", is_error=True)
@@ -776,17 +794,38 @@ def _tool_route(pool: Pool, args: dict) -> dict:
     routing = _routing_arg(args.get("routing")) or pool.routing
     msgs = _messages(None, prompt)
     difficulty = prompt_difficulty(msgs)
-    targets = pool.rank_targets(msgs, routing=routing)
+    try:
+        resolution = task_resolution(msgs, args.get("task"))
+        targets = pool.rank_targets(msgs, routing=routing, task=resolution.task)
+    except ValueError as exc:
+        return _text(str(exc), is_error=True)
     table = capability_table()
+    task_table = (
+        task_evidence_table(resolution.task)
+        if resolution.task != TASK_GENERAL
+        else {}
+    )
     lines = [
         f"routing mode: {routing}",
         f"estimated prompt difficulty: {difficulty:.2f}  (0 = trivial, 1 = hardest)",
+        f"resolved task: {resolution.task} ({resolution.source})",
         "",
         f"top candidates (in failover order){' — strongest-tier first' if routing == 'agent' else ' — strongest-fit first' if routing == 'quality' else ''}:",
     ]
     for i, t in enumerate(targets[:8], 1):
         cap = model_capability(t.model, table)
-        lines.append(f"  {i:>2}. {t.provider.id}/{t.model}  (capability {cap:.2f})")
+        task_score = model_task_score(t.model, task_table)
+        task_text = (
+            ""
+            if resolution.task == TASK_GENERAL
+            else f", task evidence {task_score:.2f}"
+            if task_score is not None
+            else ", task evidence unmeasured"
+        )
+        lines.append(
+            f"  {i:>2}. {t.provider.id}/{t.model}  "
+            f"(capability {cap:.2f}{task_text})"
+        )
     if not targets:
         lines.append("  (no configured candidates)")
     return _text("\n".join(lines))

src/freellmpool/panel.py

@@ -96,6 +96,7 @@ def select_panel_targets(
     routing: str | None = DEFAULT_ROUTING,
     model: str | None = None,
     providers: Iterable[str] | None = None,
+    task: str | None = None,
 ) -> list[Any]:
     """Pick a small, diverse set of targets for a second-opinion panel.
 
@@ -106,7 +107,13 @@ def select_panel_targets(
     """
 
     limit = clamp_panel_count(n)
-    candidates = pool.rank_targets(messages, routing=routing, model=model, providers=providers)
+    candidates = pool.rank_targets(
+        messages,
+        routing=routing,
+        model=model,
+        providers=providers,
+        task=task,
+    )
     if not candidates:
         return []
 
@@ -165,6 +172,7 @@ def run_panel(
     max_tokens: object = DEFAULT_MAX_TOKENS,
     timeout: float = DEFAULT_TIMEOUT,
     synthesize: bool = False,
+    task: str | None = None,
 ) -> PanelResult:
     requested_count = _int_or_default(n, DEFAULT_PANEL_COUNT)
     selected_count = clamp_panel_count(n)
@@ -178,6 +186,7 @@ def run_panel(
         routing=routing,
         model=model,
         providers=providers,
+        task=task,
     )
     if not picks:
         return PanelResult(
@@ -200,6 +209,7 @@ def ask_one(target: Any) -> PanelAnswer:
                 providers=[provider_id],
                 max_tokens=token_limit,
                 timeout=timeout,
+                task=task,
             )
             latency_ms = round((time.monotonic() - started) * 1000)
             label = f"{reply.provider_id}/{reply.model}"