diff --git a/CHANGELOG.md b/CHANGELOG.md index 056b9e7..cb5a4c7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,11 @@ All notable changes to this project are documented here. The format is based on ## [Unreleased] ### Added +- Provider-neutral task-aware quality routing for grounded Markdown reading, + including a local classifier, explicit CLI/MCP/proxy task hints, a faithful + reader role, sanitized versioned regression fixture, and exact-identity + multi-trial evidence that preserves existing ordering when no candidate is + measured. - Deterministic, quota-bounded per-model protocol conformance canaries for chat, streaming, tools, JSON object/schema, vision, Responses, and Anthropic Messages, with sanitized persisted evidence, model/status visibility, protected diff --git a/README.md b/README.md index 0f9a7c9..04f6196 100644 --- a/README.md +++ b/README.md @@ -236,13 +236,15 @@ freellmpool ask -m gpt-4o-mini "hi" # routed to a free model ### Roles `freellmpool roles` lists ask-role presets (`coder`, `critic`, `summarizer`, -`long-context`, `cheap`, `fast`, `second-opinion`, ...). Each role sets routing, -token budget, temperature, and system-prompt hints without inventing a second -routing engine. Explicit flags (`--model`, `--providers`, `--routing`, `--max-tokens`) -win over role defaults, and the verbose output shows when an override happened. +`grounded-reader`, `long-context`, `cheap`, `fast`, `second-opinion`, ...). Each +role sets routing, token budget, temperature, task intent, and system-prompt hints +without inventing a second routing engine. Explicit flags (`--model`, `--providers`, +`--routing`, `--task`, `--max-tokens`) win over role defaults. ```bash freellmpool ask --role coder "write a pytest for this function" +freellmpool ask --role grounded-reader "read this Markdown file" +freellmpool ask --routing quality --task general "ignore automatic task classification" FREELLMPOOL_MODE=wise freellmpool ask --role cheap "summarize this patch" ``` @@ -562,9 +564,21 @@ have the smallest daily caps, so a naive pool gets weaker as the day fills. Qual routing matches each prompt's *difficulty* to each model's *capability*: hard prompts (long input, code, reasoning cues) go to the strongest available model, and easy ones go to lightweight models — which rations scarce strong-model quota so the -pool stays sharp for longer. Capability is grounded in real benchmark data, not -guessed from names; models that no benchmark lists cover fall back to a name -heuristic. +pool stays sharp for longer. It also recognizes high-confidence grounded Markdown +reading/extraction requests locally. When repeated evidence from the versioned, +sanitized fixture exists for a candidate's exact model identity, a bounded task-fit +term influences quality ordering. Unmeasured models remain reachable, and if no +candidate has current evidence the ordering is unchanged. Capability is grounded in +real benchmark data, not guessed from names; models that no benchmark lists cover +fall back to a name heuristic. + +Use `--task grounded-reading` to declare that intent, `--task general` to suppress +automatic classification, or `--task auto` to classify locally. OpenAI-compatible, +Responses, and Anthropic proxy clients can send the body extension +`"task": "grounded-reading"` or the `X-Freellmpool-Task` header. Explicit intent +wins over automatic classification. Task evidence stores aggregate pass counts, +fixture hashes, and scores only—never prompts, documents, responses, or provider +secrets. The bundled, offline scores come from [LMArena](https://lmarena.ai/) Elo (an MIT-licensed snapshot) and the [Aider](https://aider.chat/) code-editing diff --git a/docs/AGENTS.md b/docs/AGENTS.md index 477a633..15ef11a 100644 --- a/docs/AGENTS.md +++ b/docs/AGENTS.md @@ -154,7 +154,9 @@ freellmpool profile doctor metaswarm --dry-run ## Workflows that help agents - `freellmpool roles` shows role presets (`coder`, `critic`, `summarizer`, - `second-opinion`, ...). + `grounded-reader`, `second-opinion`, ...). +- `freellmpool ask --role grounded-reader` requests faithful extraction, while + explicit `--task general` suppresses automatic grounded-reading classification. - `freellmpool ask --role coder --second-opinion` can review an implementation plan before a long agent run. - `freellmpool battle "which prompt version is clearer?"` compares model answers diff --git a/docs/MCP.md b/docs/MCP.md index f721c93..e23df4f 100644 --- a/docs/MCP.md +++ b/docs/MCP.md @@ -16,7 +16,7 @@ and `stdout` is reserved for the protocol. | Tool | What it does | |---|---| -| `free_llm_ask` | Ask a free model (`prompt`, optional `system` / `model` / `provider` / `routing` / `max_tokens`). The reply names the serving model. | +| `free_llm_ask` | Ask a free model (`prompt`, optional `system` / `model` / `provider` / `routing` / `task` / `max_tokens`). `task` accepts `auto`, `general`, or `grounded-reading`. The reply names the serving model. | | `free_llm_panel` | Ask the **same** prompt to 2-5 different free models at once and compare. Optional `synthesize` merges them into one best answer; synthesis failure leaves the individual answers visible. | | `free_llm_second_opinion` | Agent-facing second-opinion surface. Same panel behavior as `free_llm_panel` — exposed as its own tool so callers can declare intent (`prompt`, `n`, `synthesize`, `routing`, `max_tokens`). | | `free_llm_battle` | Bounded multi-model comparison rendered as a Markdown table (`prompt`, `n`, `synthesize`, `routing`, `max_tokens`). Per-model failures stay visible in the output. | @@ -25,7 +25,7 @@ and `stdout` is reserved for the protocol. | `free_llm_tailnet_info` | Show safe Tailscale Tailnet connection instructions for serving the proxy on another machine. Output NEVER contains a real local bearer token (uses a `` placeholder) and never leaks provider API keys. Degrades cleanly when `tailscale` is absent. Optional `port` (default 8080). | | `free_llm_quota_wise` | Local quota-mode / headroom advice from local counters only. Output NEVER recommends account rotation, rate-limit bypass, or automatic paid fallback — only "wait for UTC reset", "lower fan-out/token budget", or an explicit paid choice outside the default flow. | | `tokenmax` | 🌈 Gloriously excessive: blast the prompt to **every** free model across **every** provider at once, then the **calling** model synthesizes them all. Emits live `notifications/progress` (`🌈 TOKENMAXXING ▸ 47/168 models…`) so hosts like Claude Code show it ticking up, and a colorful rainbow banner in the result. Tongue-in-cheek, genuinely useful for hard questions. | -| `free_llm_route` | Explain where a prompt **would** route (estimated difficulty + ranked candidate models) **without spending a token**. | +| `free_llm_route` | Explain where a prompt **would** route (estimated difficulty, resolved task, and ranked candidate models/evidence) **without spending a token**. | | `free_llm_models` | List available `provider/model` ids. | | `free_llm_quota` | Today's per-provider usage + daily-limit headroom, plus session totals and estimated cost avoided. | | `free_llm_stats` | Lifetime tokens served free + estimated cost avoided vs Claude Opus 4.8 (persists across restarts). | diff --git a/pyproject.toml b/pyproject.toml index fb117cf..7c25c9d 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -89,6 +89,7 @@ packages = ["src/freellmpool"] [tool.hatch.build.targets.wheel.force-include] "src/freellmpool/providers.toml" = "freellmpool/providers.toml" "src/freellmpool/capability_scores.json" = "freellmpool/capability_scores.json" +"src/freellmpool/task_evidence.json" = "freellmpool/task_evidence.json" "src/freellmpool/recipes/second-opinion.json" = "freellmpool/recipes/second-opinion.json" "src/freellmpool/recipes/pr-review.json" = "freellmpool/recipes/pr-review.json" "src/freellmpool/recipes/repo-summary.json" = "freellmpool/recipes/repo-summary.json" diff --git a/src/freellmpool/aio.py b/src/freellmpool/aio.py index b4ea4d3..d09845c 100644 --- a/src/freellmpool/aio.py +++ b/src/freellmpool/aio.py @@ -40,6 +40,7 @@ from .observe import emit from .router import Pool, _is_account_quota_exhaustion, _is_health_failure from .routing_modes import normalize_routing_mode +from .task_quality import TASK_GENERAL, resolve_task, validate_task #: An async transport: ``await apost(url, headers, json_body, timeout) -> HTTPResult``. AsyncPostFn = Callable[[str, dict, dict, float], Awaitable["_client.HTTPResult"]] @@ -382,6 +383,7 @@ async def achat( response_format=None, protocol: str | None = None, routing: str | None = None, + task: str | None = None, ) -> Reply: """Async failover completion — same routing/cache/metrics as :meth:`Pool.chat`. @@ -392,6 +394,11 @@ async def achat( raise NoProvidersConfigured("no provider has an API key set") provider_list = list(providers) if providers else None eff = normalize_routing_mode(routing, p.routing) + if eff == "quality": + resolved_task = resolve_task(messages, task) + else: + validate_task(task) + resolved_task = TASK_GENERAL features = required_features( messages, tools=tools, @@ -421,6 +428,7 @@ async def achat( eff, response_format=response_format, protocol=protocol, + task=resolved_task, ) hit = await asyncio.to_thread(p._cache.get, cache_key) # blocking sqlite off-loop feature_cache_eligible = ( @@ -457,6 +465,7 @@ async def achat( candidates, difficulty, eff, + resolved_task, ) if not targets: raise NoProvidersConfigured("no candidate (provider, model) matched the given filters") diff --git a/src/freellmpool/cache.py b/src/freellmpool/cache.py index f53d3ff..2c43dbc 100644 --- a/src/freellmpool/cache.py +++ b/src/freellmpool/cache.py @@ -4,8 +4,8 @@ or ``[settings] cache_ttl`` in config.toml. Handy for dev/test loops where the same prompts run repeatedly: it saves quota and answers instantly. -Keyed on a hash of (messages, model, providers, max_tokens, temperature, tools), -so only *identical* requests hit the cache. Standard-library sqlite3, no deps. +Keyed on a hash of the request and routing/task intent, so only *identical* +requests hit the cache. Standard-library sqlite3, no deps. """ from __future__ import annotations @@ -75,6 +75,7 @@ def make_key( routing=None, response_format=None, protocol=None, + task=None, ) -> str | None: try: payload = json.dumps( @@ -91,6 +92,9 @@ def make_key( "routing": routing, "response_format": response_format, "protocol": protocol, + # Explicit task intent can select a different model under the + # same routing mode and therefore needs its own cache bucket. + "task": task, }, sort_keys=True, ) diff --git a/src/freellmpool/cli.py b/src/freellmpool/cli.py index 280e12b..17bac22 100644 --- a/src/freellmpool/cli.py +++ b/src/freellmpool/cli.py @@ -55,6 +55,7 @@ from .router import Pool from .routing_modes import PUBLIC_ROUTING_ALIASES, routing_override from .savings import format_saved +from .task_quality import TASK_HINTS def _read_stdin() -> str: @@ -139,6 +140,7 @@ def cmd_ask(args: argparse.Namespace) -> int: if args.routing is None and routing is None and role is None and args.mode == "normal": if not has_routing_config: routing = "fair" + task = args.task if args.task is not None else (role.task if role is not None else None) second_opinion = bool(args.second_opinion or (role is not None and role.name == "second-opinion")) if second_opinion: @@ -156,6 +158,7 @@ def cmd_ask(args: argparse.Namespace) -> int: max_tokens=max_tokens, timeout=args.timeout, synthesize=args.synthesize, + task=task, ) if not result.answers: print("freellmpool: no providers configured", file=sys.stderr) @@ -168,7 +171,13 @@ def cmd_ask(args: argparse.Namespace) -> int: if system: messages.append({"role": "system", "content": system}) messages.append({"role": "user", "content": prompt}) - targets = pool.rank_targets(messages, routing=routing, model=model_filter, providers=provider_filter) + targets = pool.rank_targets( + messages, + routing=routing, + model=model_filter, + providers=provider_filter, + task=task, + ) snapshot = pool.quota.snapshot() if declared_quota_exhausted(targets, snapshot): print( @@ -192,6 +201,7 @@ def cmd_ask(args: argparse.Namespace) -> int: temperature=temperature, timeout=args.timeout, routing=routing, + task=task, ) except NoProvidersConfigured as exc: print(f"freellmpool: {exc}", file=sys.stderr) @@ -2002,6 +2012,11 @@ def build_parser() -> argparse.ArgumentParser: choices=PUBLIC_ROUTING_ALIASES, help="routing mode override (auto uses the pool default)", ) + p_ask.add_argument( + "--task", + choices=TASK_HINTS, + help="task hint for quality routing (auto classifies locally)", + ) p_ask.add_argument( "--mode", choices=["normal", "wise"], diff --git a/src/freellmpool/mcp_server.py b/src/freellmpool/mcp_server.py index 484dc20..5aaf644 100644 --- a/src/freellmpool/mcp_server.py +++ b/src/freellmpool/mcp_server.py @@ -64,6 +64,13 @@ generate_session_token_simple, safe_base_url, ) +from .task_quality import ( + TASK_GENERAL, + TASK_HINTS, + model_task_score, + task_evidence_table, + task_resolution, +) from .tokenmax import HARD_CAP, RAINBOW_BANNER, fan_out, select_targets _DEFAULT_PROTOCOL = "2025-06-18" @@ -116,6 +123,11 @@ "enum": list(PUBLIC_ROUTING_ALIASES), "description": "How to pick the model: agent (strongest healthy tier with quota spreading), quality (capability matched to prompt), spread (whole-pool breadth), fast (lowest latency), fair (provider balance), or auto (server default).", }, + "task": { + "type": "string", + "enum": list(TASK_HINTS), + "description": "Optional task hint for quality routing; auto classifies locally.", + }, "max_tokens": { "type": "integer", "description": "Max output tokens (default 1024).", @@ -378,6 +390,11 @@ "enum": list(PUBLIC_ROUTING_ALIASES), "description": "Routing mode to explain (default: the server's mode).", }, + "task": { + "type": "string", + "enum": list(TASK_HINTS), + "description": "Optional task hint to explain.", + }, }, "required": ["prompt"], }, @@ -505,6 +522,7 @@ def _tool_ask(pool: Pool, args: dict) -> dict: providers=providers, routing=routing, max_tokens=_max_tokens(args.get("max_tokens"), 1024), + task=args.get("task"), ) except Exception as exc: # noqa: BLE001 — surface as a tool error return _text(f"{type(exc).__name__}: {exc}", is_error=True) @@ -776,17 +794,38 @@ def _tool_route(pool: Pool, args: dict) -> dict: routing = _routing_arg(args.get("routing")) or pool.routing msgs = _messages(None, prompt) difficulty = prompt_difficulty(msgs) - targets = pool.rank_targets(msgs, routing=routing) + try: + resolution = task_resolution(msgs, args.get("task")) + targets = pool.rank_targets(msgs, routing=routing, task=resolution.task) + except ValueError as exc: + return _text(str(exc), is_error=True) table = capability_table() + task_table = ( + task_evidence_table(resolution.task) + if resolution.task != TASK_GENERAL + else {} + ) lines = [ f"routing mode: {routing}", f"estimated prompt difficulty: {difficulty:.2f} (0 = trivial, 1 = hardest)", + f"resolved task: {resolution.task} ({resolution.source})", "", f"top candidates (in failover order){' — strongest-tier first' if routing == 'agent' else ' — strongest-fit first' if routing == 'quality' else ''}:", ] for i, t in enumerate(targets[:8], 1): cap = model_capability(t.model, table) - lines.append(f" {i:>2}. {t.provider.id}/{t.model} (capability {cap:.2f})") + task_score = model_task_score(t.model, task_table) + task_text = ( + "" + if resolution.task == TASK_GENERAL + else f", task evidence {task_score:.2f}" + if task_score is not None + else ", task evidence unmeasured" + ) + lines.append( + f" {i:>2}. {t.provider.id}/{t.model} " + f"(capability {cap:.2f}{task_text})" + ) if not targets: lines.append(" (no configured candidates)") return _text("\n".join(lines)) diff --git a/src/freellmpool/panel.py b/src/freellmpool/panel.py index 01af5d9..7932493 100644 --- a/src/freellmpool/panel.py +++ b/src/freellmpool/panel.py @@ -96,6 +96,7 @@ def select_panel_targets( routing: str | None = DEFAULT_ROUTING, model: str | None = None, providers: Iterable[str] | None = None, + task: str | None = None, ) -> list[Any]: """Pick a small, diverse set of targets for a second-opinion panel. @@ -106,7 +107,13 @@ def select_panel_targets( """ limit = clamp_panel_count(n) - candidates = pool.rank_targets(messages, routing=routing, model=model, providers=providers) + candidates = pool.rank_targets( + messages, + routing=routing, + model=model, + providers=providers, + task=task, + ) if not candidates: return [] @@ -165,6 +172,7 @@ def run_panel( max_tokens: object = DEFAULT_MAX_TOKENS, timeout: float = DEFAULT_TIMEOUT, synthesize: bool = False, + task: str | None = None, ) -> PanelResult: requested_count = _int_or_default(n, DEFAULT_PANEL_COUNT) selected_count = clamp_panel_count(n) @@ -178,6 +186,7 @@ def run_panel( routing=routing, model=model, providers=providers, + task=task, ) if not picks: return PanelResult( @@ -200,6 +209,7 @@ def ask_one(target: Any) -> PanelAnswer: providers=[provider_id], max_tokens=token_limit, timeout=timeout, + task=task, ) latency_ms = round((time.monotonic() - started) * 1000) label = f"{reply.provider_id}/{reply.model}" diff --git a/src/freellmpool/proxy.py b/src/freellmpool/proxy.py index a22a0ca..e84af87 100644 --- a/src/freellmpool/proxy.py +++ b/src/freellmpool/proxy.py @@ -53,6 +53,7 @@ from .router import Pool from .routing_modes import PUBLIC_ROUTING_ALIASES, routing_override from .savings import usd_saved +from .task_quality import task_resolution _MAX_BODY = 16 * 1024 * 1024 # 16 MB cap on request bodies # Audio uploads are larger than JSON; Groq's free tier accepts up to 25 MB, so cap audio @@ -396,6 +397,12 @@ def _routing_and_model(headers, requested: str) -> tuple[str | None, str]: return override, requested +def _task_hint(headers, req: dict) -> object: + """Header intent wins over an optional OpenAI-compatible body extension.""" + header = headers.get("X-Freellmpool-Task") + return header if header is not None else req.get("task") + + def make_handler(pool: Pool, api_key: str | None = None): # Ring buffer of recently-served (provider, model). Appended from worker # threads and snapshotted by /status, so guard it: a deque append is atomic, @@ -782,6 +789,9 @@ def _handle_messages(self, req: dict) -> None: resolved = resolve_alias(model_str, pool.env) provider_filter, model_filter = _parse_model(resolved, {p.id for p in pool.providers}) try: + task = task_resolution( + chat["messages"], _task_hint(self.headers, req) + ).task reply = pool.chat( chat["messages"], model=model_filter, @@ -792,7 +802,11 @@ def _handle_messages(self, req: dict) -> None: tool_choice=chat["tool_choice"], protocol="anthropic_messages", routing=routing_override, + task=task, ) + except ValueError as exc: + self._anthropic_error(400, str(exc)) + return except NoProvidersConfigured as exc: self._anthropic_error(503, str(exc), "no_providers") return @@ -952,11 +966,14 @@ def _resolve( max_tokens = int(_max_tokens_value(req, 1024)) temp_raw = req.get("temperature") temperature = 0.0 if temp_raw is None else float(temp_raw) + task = task_resolution( + messages, _task_hint(self.headers, req) + ).task except (TypeError, ValueError): self._error( 400, "'max_tokens'/'max_completion_tokens'/'max_output_tokens'/" - "'temperature' must be numbers", + "'temperature' must be numbers and 'task' must be valid", "invalid_request_error", ) return None @@ -982,6 +999,7 @@ def _resolve( response_format=response_format, protocol=protocol, routing=routing_override, + task=task, ) except NoProvidersConfigured as exc: self._error(503, str(exc), "no_providers") @@ -1057,11 +1075,12 @@ def _stream_chat(self, req: dict, norm: list[dict]) -> None: max_tokens = int(_max_tokens_value(req, 1024)) temp_raw = req.get("temperature") temperature = 0.0 if temp_raw is None else float(temp_raw) + task = task_resolution(norm, _task_hint(self.headers, req)).task except (TypeError, ValueError): self._error( 400, "'max_tokens'/'max_completion_tokens'/'max_output_tokens'/" - "'temperature' must be numbers", + "'temperature' must be numbers and 'task' must be valid", "invalid_request_error", ) return @@ -1080,6 +1099,7 @@ def _stream_chat(self, req: dict, norm: list[dict]) -> None: temperature=temperature, timeout=upstream_timeout, routing=routing_override, + task=task, ) meta = next(gen) # provider/model chosen, or raises before any bytes except NoProvidersConfigured as exc: diff --git a/src/freellmpool/roles.py b/src/freellmpool/roles.py index 885034b..bbfa26c 100644 --- a/src/freellmpool/roles.py +++ b/src/freellmpool/roles.py @@ -24,6 +24,7 @@ class RoleSpec: max_tokens: int | None = None temperature: float | None = None system_prefix: str | None = None + task: str | None = None _ROLE_SPECS: tuple[RoleSpec, ...] = ( @@ -55,6 +56,18 @@ class RoleSpec: "Summarize the following text concisely, preserving key facts and trade-offs." ), ), + RoleSpec( + name="grounded-reader", + description="Quality routing for faithful document reading and extraction.", + routing="quality", + max_tokens=1024, + temperature=0.0, + system_prefix=( + "Read the supplied material faithfully. Distinguish stated facts from " + "inference and do not invent details." + ), + task="grounded-reading", + ), RoleSpec( name="long-context", description="Quality routing with larger token budget for detailed answers.", @@ -126,5 +139,7 @@ def format_roles() -> str: extras.append(f"max_tokens={role.max_tokens}") if role.temperature is not None: extras.append(f"temperature={role.temperature}") + if role.task is not None: + extras.append(f"task={role.task}") lines.append(f" {role.name:<16} {role.description} ({', '.join(extras)})") return "\n".join(lines) diff --git a/src/freellmpool/router.py b/src/freellmpool/router.py index 77fba87..f70a173 100644 --- a/src/freellmpool/router.py +++ b/src/freellmpool/router.py @@ -57,6 +57,13 @@ ) from .routing_modes import normalize_routing_mode from .stats import StatsStore +from .task_quality import ( + TASK_GENERAL, + model_task_score, + resolve_task, + task_evidence_table, + validate_task, +) # A parsed "context limit" below this is treated as garbled/implausible and not # learned, so one bad provider error can't poison routing pool-wide. @@ -72,6 +79,8 @@ # breaks ties among models that already clear the difficulty bar. _QUALITY_SLOW_S = 8.0 _QUALITY_UNKNOWN_LAT = 0.34 +_QUALITY_UNKNOWN_TASK = 0.5 +_QUALITY_TASK_WEIGHT = 1.0 # "spread" routing groups providers into coarse usage tiers of this many requests, so the # least-used tier is served first (spreading load across the WHOLE pool to avoid one # provider hitting its rate limit), while within a tier the fastest/healthiest is preferred. @@ -381,6 +390,7 @@ def rank_targets( routing: str | None = None, model: str | None = None, providers: Iterable[str] | None = None, + task: str | None = None, ) -> list[Target]: """Public: the ordered candidate (provider, model) targets for ``messages``, using the same ordering the failover loop would. Powers the MCP route @@ -389,10 +399,16 @@ def rank_targets( provider_list = list(providers) if providers else None eff = normalize_routing_mode(routing, self.routing) difficulty = prompt_difficulty(messages) if eff == "quality" else None + if eff == "quality": + resolved_task = resolve_task(messages, task) + else: + validate_task(task) + resolved_task = TASK_GENERAL return self._order( self._all_targets(include=provider_list, model=model), difficulty=difficulty, routing=eff, + task=resolved_task, ) def _mark_cooldown(self, provider_id: str, now: float) -> None: @@ -674,7 +690,11 @@ def _feature_targets( return self.conformance.verified_targets(targets, wanted) def _order( - self, targets: list[Target], difficulty: float | None = None, routing: str | None = None + self, + targets: list[Target], + difficulty: float | None = None, + routing: str | None = None, + task: str = TASK_GENERAL, ) -> list[Target]: """Order candidate targets for failover. @@ -690,9 +710,10 @@ def _order( fastest/healthiest within a tier. Best for sustained agentic loops on free tiers: the breadth of ``fair`` with the speed of ``fast``. - ``quality``: match the request's ``difficulty`` (0–1) to each model's - benchmark-scored capability — strong models for hard prompts, light models - for easy ones (rationing scarce strong-model quota). Ordered globally, not + ``quality``: match the request's ``difficulty`` (0–1) and bounded validated + task evidence to each model's benchmark-scored capability — strong models + for hard prompts, proven task fits when available, and light models for easy + ones (rationing scarce strong-model quota). Ordered globally, not provider-grouped, since capability match is the opted-in intent. ``agent``: keep every turn on the strongest available benchmark capability @@ -773,6 +794,11 @@ def agent_key( if mode == "quality": table = capability_table() need = difficulty if difficulty is not None else 0.5 + task_table = task_evidence_table(task) if task != TASK_GENERAL else {} + has_task_evidence = any( + model_task_score(target.model, task_table) is not None + for target in targets + ) def lat_pen(t: Target) -> float: # Latency penalty in [0,1]: a measured target's smoothed latency @@ -793,11 +819,19 @@ def lat_pen(t: Target) -> float: def quality_key(t: Target) -> tuple[int, int, float, int]: # over-budget, then known-failing sink to the back (still reachable); - # then capability-fit blended with latency; then least-used. + # then capability/task fit blended with latency; then least-used. + # With no valid task evidence the added term is exactly zero. + task_penalty = 0.0 + if has_task_evidence: + measured = model_task_score(t.model, task_table) + task_score = _QUALITY_UNKNOWN_TASK if measured is None else measured + task_penalty = (1.0 - task_score) * _QUALITY_TASK_WEIGHT return ( over_of(t), 1 if failing_of(t) else 0, - fit_penalty(model_capability(t.model, table), need) + lat_pen(t), + fit_penalty(model_capability(t.model, table), need) + + task_penalty + + lat_pen(t), used_of(t), ) @@ -898,6 +932,7 @@ def ask( tools: list | None = None, tool_choice=None, routing: str | None = None, + task: str | None = None, ) -> Reply: """Send ``prompt`` to the first provider that succeeds. @@ -920,6 +955,7 @@ def ask( tools=tools, tool_choice=tool_choice, routing=routing, + task=task, ) def chat( @@ -936,6 +972,7 @@ def chat( response_format=None, protocol: str | None = None, routing: str | None = None, + task: str | None = None, ) -> Reply: """Like :meth:`ask` but takes raw OpenAI-style ``messages``. @@ -965,6 +1002,11 @@ def chat( # per-request override (fast/quality/fair/…) keys its own cache bucket and # never serves a reply produced under a different mode's intent. eff = normalize_routing_mode(routing, self.routing) + if eff == "quality": + resolved_task = resolve_task(messages, task) + else: + validate_task(task) + resolved_task = TASK_GENERAL cache_key = None if self._cache is not None: cache_key = self._cache.make_key( @@ -978,6 +1020,7 @@ def chat( eff, response_format=response_format, protocol=protocol, + task=resolved_task, ) hit = self._cache.get(cache_key) feature_cache_eligible = ( @@ -1013,6 +1056,7 @@ def chat( candidates, difficulty=difficulty, routing=eff, + task=resolved_task, ) if not targets: raise NoProvidersConfigured("no candidate (provider, model) matched the given filters") @@ -1191,6 +1235,7 @@ def stream_chat( temperature: float = 0.0, timeout: float = 90.0, routing: str | None = None, + task: str | None = None, ): """Stream content deltas with token-level streaming. @@ -1204,6 +1249,11 @@ def stream_chat( raise NoProvidersConfigured("no provider has an API key set") eff = normalize_routing_mode(routing, self.routing) difficulty = prompt_difficulty(messages, max_tokens) if eff == "quality" else None + if eff == "quality": + resolved_task = resolve_task(messages, task) + else: + validate_task(task) + resolved_task = TASK_GENERAL provider_list = list(providers) if providers else None candidates = self._all_targets(include=provider_list, model=model) candidates = self._feature_targets( @@ -1215,6 +1265,7 @@ def stream_chat( candidates, difficulty=difficulty, routing=eff, + task=resolved_task, ) targets = [t for t in targets if t.provider.adapter != "gemini"] if not targets: diff --git a/src/freellmpool/task_evidence.json b/src/freellmpool/task_evidence.json new file mode 100644 index 0000000..bee168f --- /dev/null +++ b/src/freellmpool/task_evidence.json @@ -0,0 +1,17 @@ +{ + "version": 1, + "fixture_sha256": { + "grounded-reading": "68256d842856b9a81c5d3f93efa873e23689b69febd8d245a1c0eead7aedff82" + }, + "scores": { + "grounded-reading": { + "llama-3.3-70b-versatile": { + "fixture_sha256": "68256d842856b9a81c5d3f93efa873e23689b69febd8d245a1c0eead7aedff82", + "passed": 3, + "score": 1.0, + "source": "synthetic-grounded-v1 live free-tier validation 2026-07-29", + "trials": 3 + } + } + } +} diff --git a/src/freellmpool/task_quality.py b/src/freellmpool/task_quality.py new file mode 100644 index 0000000..e3b07c2 --- /dev/null +++ b/src/freellmpool/task_quality.py @@ -0,0 +1,211 @@ +"""Local task classification and validated task-specific model evidence. + +Task fit is orthogonal to generic benchmark capability. The first deliberately +narrow task is grounded document reading: requests that ask a model to extract +or summarize supplied Markdown without inventing details. + +Evidence is provider-neutral and exact-model only. It must come from repeated +runs of the current sanitized fixture; production prompts are never learned +from or persisted. +""" + +from __future__ import annotations + +import json +import math +import os +import re +from collections.abc import Mapping +from dataclasses import dataclass +from functools import lru_cache +from pathlib import Path +from types import MappingProxyType + +TASK_AUTO = "auto" +TASK_GENERAL = "general" +TASK_GROUNDED_READING = "grounded-reading" +TASK_HINTS = (TASK_AUTO, TASK_GENERAL, TASK_GROUNDED_READING) + +GROUNDED_FIXTURE_SHA256 = ( + "68256d842856b9a81c5d3f93efa873e23689b69febd8d245a1c0eead7aedff82" +) +_BUNDLED_EVIDENCE = Path(__file__).with_name("task_evidence.json") +_MAX_CLASSIFIER_CHARS = 65_536 +_MIN_TRIALS = 3 + +_GROUNDED_INTENT_RE = re.compile( + r"\b(?:read|summari[sz]e|extract|tell me what|what (?:it|this) " + r"contains|according to|based (?:only )?on)\b", + re.IGNORECASE, +) +_MARKDOWN_STRUCTURE_RE = re.compile( + r"(?m)^(?:#{1,6}\s+\S|[-*]\s+\S|\|.+\||```)" +) + + +@dataclass(frozen=True, slots=True) +class TaskResolution: + task: str + source: str + + +def _routing_text(messages: object) -> str: + """Bounded user/tool text only; system and prior assistant prose cannot steer it.""" + parts: list[str] = [] + if not isinstance(messages, list): + return "" + for message in messages: + if not isinstance(message, dict) or message.get("role") not in {"user", "tool"}: + continue + content = message.get("content") + if isinstance(content, str): + parts.append(content) + elif isinstance(content, list): + parts.extend( + part["text"] + for part in content + if isinstance(part, dict) and isinstance(part.get("text"), str) + ) + text = "\n".join(parts) + if len(text) <= _MAX_CLASSIFIER_CHARS: + return text + half = _MAX_CLASSIFIER_CHARS // 2 + return f"{text[:half]}\n{text[-half:]}" + + +def classify_task(messages: object) -> str: + """Return a high-confidence local task class without making a model call.""" + text = _routing_text(messages) + if not _GROUNDED_INTENT_RE.search(text): + return TASK_GENERAL + structures = _MARKDOWN_STRUCTURE_RE.findall(text) + if len(structures) >= 2: + return TASK_GROUNDED_READING + return TASK_GENERAL + + +def task_resolution(messages: object, task: str | None = None) -> TaskResolution: + """Resolve explicit intent before automatic classification.""" + validate_task(task) + if task is None or task == TASK_AUTO: + return TaskResolution(classify_task(messages), "auto") + if task in {TASK_GENERAL, TASK_GROUNDED_READING}: + return TaskResolution(task, "explicit") + raise ValueError(f"unknown task {task!r}; expected one of: {', '.join(TASK_HINTS)}") + + +def validate_task(task: str | None) -> None: + if task is not None and task not in TASK_HINTS: + raise ValueError( + f"unknown task {task!r}; expected one of: {', '.join(TASK_HINTS)}" + ) + + +def resolve_task(messages: object, task: str | None = None) -> str: + return task_resolution(messages, task).task + + +def user_task_evidence_path() -> Path: + override = os.environ.get("FREELLMPOOL_TASK_EVIDENCE_FILE") + if override: + return Path(override).expanduser() + return Path.home() / ".config" / "freellmpool" / "task_evidence.json" + + +def _fixture_for(task: str) -> str | None: + if task == TASK_GROUNDED_READING: + return GROUNDED_FIXTURE_SHA256 + return None + + +def _read_evidence(path: Path, task: str) -> dict[str, float]: + try: + data = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + return {} + if not isinstance(data, dict) or data.get("version") != 1: + return {} + scores = data.get("scores") + raw = scores.get(task, {}) if isinstance(scores, dict) else {} + fixture = _fixture_for(task) + if not isinstance(raw, dict) or fixture is None: + return {} + out: dict[str, float] = {} + for model, entry in raw.items(): + if not isinstance(model, str) or not model or not isinstance(entry, dict): + continue + source = entry.get("source") + trials = entry.get("trials") + passed = entry.get("passed") + if ( + not isinstance(source, str) + or not source.strip() + or entry.get("fixture_sha256") != fixture + or not isinstance(trials, int) + or isinstance(trials, bool) + or trials < _MIN_TRIALS + or not isinstance(passed, int) + or isinstance(passed, bool) + or not 0 <= passed <= trials + ): + continue + raw_score = entry.get("score") + if not isinstance(raw_score, (int, float, str)) or isinstance( + raw_score, bool + ): + continue + try: + score = float(raw_score) + except ValueError: + continue + if not math.isfinite(score) or not 0.0 <= score <= 1.0: + continue + if not math.isclose(score, passed / trials, abs_tol=1e-4): + continue + out[model] = score + return out + + +@lru_cache(maxsize=24) +def _evidence_cached( + user_str: str, _user_mtime: int, task: str +) -> Mapping[str, float]: + table = _read_evidence(_BUNDLED_EVIDENCE, task) + table.update(_read_evidence(Path(user_str), task)) + return MappingProxyType(table) + + +def task_evidence_table(task: str) -> Mapping[str, float]: + """Return exact model identities with current, repeated task evidence.""" + if task == TASK_GENERAL: + return MappingProxyType({}) + if task not in TASK_HINTS or task == TASK_AUTO: + raise ValueError(f"unknown resolved task {task!r}") + user = user_task_evidence_path() + try: + mtime = user.stat().st_mtime_ns + except OSError: + mtime = 0 + return _evidence_cached(str(user), mtime, task) + + +def model_task_score( + model: str, table: Mapping[str, float] +) -> float | None: + """Exact identity lookup: no family, provider, or semantic alias borrowing.""" + return table.get(model) + + +def grounded_answer_passes(answer: str, case: Mapping[str, object]) -> bool: + """Deterministic fixture rubric used to aggregate bounded benchmark trials.""" + lowered = answer.casefold() + must_include = case.get("must_include") + must_not_invent = case.get("must_not_invent") + if not isinstance(must_include, list) or not isinstance(must_not_invent, list): + raise ValueError("grounded fixture needs must_include and must_not_invent lists") + return all( + isinstance(fact, str) and fact.casefold() in lowered for fact in must_include + ) and all( + isinstance(claim, str) and claim.casefold() not in lowered + for claim in must_not_invent + ) diff --git a/tests/fixtures/grounded_reading.json b/tests/fixtures/grounded_reading.json new file mode 100644 index 0000000..ffd0c81 --- /dev/null +++ b/tests/fixtures/grounded_reading.json @@ -0,0 +1,19 @@ +{ + "name": "lunar-finch-api-guide", + "prompt": "Read this Markdown document and tell me what it contains. Stay faithful to the document.", + "document": "# Lunar Finch Search API\n\nThe base URL is `https://api.lunarfinch.example/v2`.\n\n## Authentication\n\nSend the API key in the `X-Finch-Token` header. Keys are created in the web dashboard.\n\n## Search modes\n\n| Mode | Purpose |\n| --- | --- |\n| `precise` | Exact factual lookup |\n| `broad` | Exploratory discovery |\n\nThe `max_results` parameter defaults to **17** and cannot exceed **80**.\n\n## Python example\n\n```python\nresponse = client.search(query=\"ice shelf\", mode=\"precise\", max_results=17)\n```\n\n## Troubleshooting\n\nA `429` response means the recurring free allowance is exhausted. Wait **23 seconds** before retrying. The client never writes credentials to disk.", + "must_include": [ + "X-Finch-Token", + "precise", + "broad", + "17", + "80", + "23 seconds" + ], + "must_not_invent": [ + "npm package", + "interactive API key prompt", + "generates an API key with curl", + "writes credentials to .env" + ] +} diff --git a/tests/test_cache.py b/tests/test_cache.py index fec1665..222ff35 100644 --- a/tests/test_cache.py +++ b/tests/test_cache.py @@ -84,6 +84,15 @@ def test_make_key_includes_routing(): assert Cache.make_key(*args, routing="fair") == Cache.make_key(*args, routing="fair") +def test_make_key_includes_task_hint(): + args = ([{"role": "user", "content": "read this"}], None, None, 1024, 0.0, None, None) + assert Cache.make_key(*args, routing="quality", task="general") != Cache.make_key( + *args, + routing="quality", + task="grounded-reading", + ) + + def test_make_key_includes_response_format_and_protocol(): args = ([{"role": "user", "content": "hi"}], None, None, 1024, 0.0, None, None) plain = Cache.make_key(*args, routing="fair") diff --git a/tests/test_cli.py b/tests/test_cli.py index 759d7b4..08811eb 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -406,6 +406,57 @@ def ask(self, prompt, **kwargs): assert capsys.readouterr().out.strip() == "ok" +def test_cli_explicit_task_beats_role_task_default(monkeypatch, capsys): + from freellmpool.cli import main + from freellmpool.router import Pool + + captured = {} + + class FakePool: + def ask(self, prompt, **kwargs): + captured.update(kwargs) + return Reply(text="ok", provider_id="fake", model="fake-model", raw={}) + + monkeypatch.setattr(Pool, "from_default_config", classmethod(lambda cls: FakePool())) + monkeypatch.setattr("freellmpool.cli._read_stdin", lambda: "") + + assert ( + main( + [ + "ask", + "Read this Markdown", + "--role", + "grounded-reader", + "--task", + "general", + ] + ) + == 0 + ) + + assert captured["task"] == "general" + assert capsys.readouterr().out.strip() == "ok" + + +def test_cli_grounded_reader_role_passes_its_task_default(monkeypatch, capsys): + from freellmpool.cli import main + from freellmpool.router import Pool + + captured = {} + + class FakePool: + def ask(self, prompt, **kwargs): + captured.update(kwargs) + return Reply(text="ok", provider_id="fake", model="fake-model", raw={}) + + monkeypatch.setattr(Pool, "from_default_config", classmethod(lambda cls: FakePool())) + monkeypatch.setattr("freellmpool.cli._read_stdin", lambda: "") + + assert main(["ask", "Read this", "--role", "grounded-reader"]) == 0 + assert captured["task"] == "grounded-reading" + assert capsys.readouterr().out.strip() == "ok" + + def test_cli_ask_second_opinion_prints_two_answers(providers, env, quota, monkeypatch, capsys): from helpers import make_post diff --git a/tests/test_mcp.py b/tests/test_mcp.py index bb6b834..f04136e 100644 --- a/tests/test_mcp.py +++ b/tests/test_mcp.py @@ -6,6 +6,7 @@ from helpers import make_post, openai_body from freellmpool.mcp_server import handle_message +from freellmpool.models import Reply from freellmpool.router import Pool @@ -76,6 +77,10 @@ def test_tool_schemas_expose_expected_fields(providers, env, quota): pool = _pool(providers, env, quota) resp = handle_message(pool, {"jsonrpc": "2.0", "id": 2, "method": "tools/list"}) by_name = {t["name"]: t for t in resp["result"]["tools"]} + ask_props = by_name["free_llm_ask"]["inputSchema"]["properties"] + assert ask_props["task"]["enum"] == ["auto", "general", "grounded-reading"] + route_props = by_name["free_llm_route"]["inputSchema"]["properties"] + assert route_props["task"]["enum"] == ["auto", "general", "grounded-reading"] # Panel-style tools share the same shape (n clamp 2-5, max_tokens, synthesize, routing enum). for tool_name in ("free_llm_panel", "free_llm_second_opinion", "free_llm_battle"): @@ -139,6 +144,32 @@ def test_tools_call_ask(providers, env, quota): assert resp["result"]["isError"] is False +def test_tools_call_ask_forwards_task_hint(providers, env, quota, monkeypatch): + pool = _pool(providers, env, quota) + captured = {} + + def fake_chat(messages, **kwargs): + captured.update(kwargs) + return Reply(text="ok", provider_id="alpha", model="alpha-small", raw={}) + + monkeypatch.setattr(pool, "chat", fake_chat) + resp = handle_message( + pool, + { + "jsonrpc": "2.0", + "id": 31, + "method": "tools/call", + "params": { + "name": "free_llm_ask", + "arguments": {"prompt": "read this", "task": "grounded-reading"}, + }, + }, + ) + + assert resp["result"]["isError"] is False + assert captured["task"] == "grounded-reading" + + def test_tools_call_panel(providers, env, quota): pool = _pool(providers, env, quota) # all providers return "ok" resp = handle_message( @@ -352,6 +383,7 @@ def test_tools_call_route_is_zero_token(providers, env, quota): ) text = resp["result"]["content"][0]["text"] assert "difficulty" in text.lower() + assert "resolved task: general (auto)" in text.lower() assert "alpha/" in text # a ranked candidate assert pool.stats_snapshot()["requests"] == 0 # explained without spending a token diff --git a/tests/test_proxy.py b/tests/test_proxy.py index 5f49671..b6e87cd 100644 --- a/tests/test_proxy.py +++ b/tests/test_proxy.py @@ -1473,6 +1473,28 @@ def test_header_routing_override_accepted(server): assert "x_freellmpool" in body +def test_task_hint_header_and_body_extension_are_validated(server): + payload = { + "model": "quality", + "messages": [{"role": "user", "content": "read this"}], + } + status, _body = _post_json_with_headers( + server + "/v1/chat/completions", + payload, + {"X-Freellmpool-Task": "grounded-reading"}, + ) + assert status == 200 + + with pytest.raises(urllib.error.HTTPError) as exc_info: + _post_json( + server + "/v1/chat/completions", + {**payload, "task": "not-a-model-name"}, + ) + assert exc_info.value.code == 400 + body = json.load(exc_info.value) + assert body["error"]["type"] == "invalid_request_error" + + def test_parse_multipart_form_unit(): from freellmpool.proxy import _parse_multipart_form diff --git a/tests/test_roles.py b/tests/test_roles.py index d4b4490..5238d47 100644 --- a/tests/test_roles.py +++ b/tests/test_roles.py @@ -13,6 +13,7 @@ "conserve", "fast", "second-opinion", + "grounded-reader", } @@ -89,3 +90,11 @@ def test_second_opinion_role_uses_panel_defaults(): assert role.routing == "quality" assert role.max_tokens == 512 assert role.system_prefix is not None + + +def test_grounded_reader_role_declares_explicit_task_hint(): + role = get_role("grounded-reader") + assert role is not None + assert role.routing == "quality" + assert role.task == "grounded-reading" + assert "faithful" in role.system_prefix.lower() diff --git a/tests/test_routing.py b/tests/test_routing.py index a033242..d1240b9 100644 --- a/tests/test_routing.py +++ b/tests/test_routing.py @@ -5,8 +5,10 @@ from helpers import make_post, make_stream_post from freellmpool import capability as _capability +from freellmpool import task_quality as _task_quality from freellmpool.client import HTTPResult from freellmpool.models import Model, Provider +from freellmpool.quota import QuotaStore from freellmpool.router import Pool _EASY = [{"role": "user", "content": "hi"}] @@ -23,7 +25,15 @@ def _names(pool, **kw): return [t.name for t in pool._order(pool._all_targets(**kw))] -def _quality_pool(tmp_path, monkeypatch, quota, *, scores, models): +def _quality_pool( + tmp_path, + monkeypatch, + quota, + *, + scores, + models, + task_scores=None, +): """A quality-routing pool over ``models`` with an injected capability table. All providers succeed (200), so whichever target quality routing puts first is @@ -33,11 +43,41 @@ def _quality_pool(tmp_path, monkeypatch, quota, *, scores, models): cap_file = tmp_path / "cap.json" cap_file.write_text( - json.dumps({"scores": {k: {"score": v, "source": "arena"} for k, v in scores.items()}}), + json.dumps( + {"scores": {k: {"score": v, "source": "arena"} for k, v in scores.items()}} + ), encoding="utf-8", ) monkeypatch.setenv("FREELLMPOOL_CAPABILITY_FILE", str(cap_file)) _capability._table_cached.cache_clear() + if task_scores: + evidence_file = tmp_path / "task-evidence.json" + evidence_file.write_text( + json.dumps( + { + "version": 1, + "scores": { + task: { + model: { + **entry, + "fixture_sha256": _task_quality.GROUNDED_FIXTURE_SHA256, + "trials": 20, + "passed": round(entry["score"] * 20), + } + for model, entry in entries.items() + } + for task, entries in task_scores.items() + }, + } + ), + encoding="utf-8", + ) + monkeypatch.setenv( + "FREELLMPOOL_TASK_EVIDENCE_FILE", str(evidence_file) + ) + else: + monkeypatch.delenv("FREELLMPOOL_TASK_EVIDENCE_FILE", raising=False) + _task_quality._evidence_cached.cache_clear() provider = Provider( id="x", label="X", @@ -428,6 +468,175 @@ def test_chat_routing_override_end_to_end(tmp_path, monkeypatch, quota): assert pool.routing == "fast" # default untouched +def test_quality_grounded_reading_prefers_validated_task_evidence( + tmp_path, monkeypatch, quota +): + task = _task_quality.TASK_GROUNDED_READING + pool = _quality_pool( + tmp_path, + monkeypatch, + quota, + scores={"generalist": 0.9, "faithful": 0.6, "unfaithful": 0.8}, + models=[Model("generalist"), Model("faithful"), Model("unfaithful")], + task_scores={ + task: { + "faithful": { + "score": 0.95, + "source": "synthetic-grounded-v1", + }, + "unfaithful": { + "score": 0.2, + "source": "synthetic-grounded-v1", + }, + } + }, + ) + messages = [ + { + "role": "user", + "content": ( + "Read this Markdown document and tell me what it contains.\n\n" + "# API guide\n\n## Authentication\n\nUse X-Finch-Token.\n\n" + "## Search modes\n\n| mode | use |\n| --- | --- |\n| precise | facts |" + ), + } + ] + + assert pool.rank_targets(messages)[0].model == "faithful" + assert pool.chat(messages).model == "faithful" + + +def test_quality_without_valid_task_evidence_preserves_existing_order( + tmp_path, monkeypatch, quota +): + pool = _quality_pool( + tmp_path, + monkeypatch, + quota, + scores={"model-a": 0.55, "model-b": 0.9}, + models=[Model("model-a"), Model("model-b")], + ) + targets = pool._all_targets() + + assert pool._order( + targets, + difficulty=0.5, + routing="quality", + task=_task_quality.TASK_GROUNDED_READING, + ) == pool._order( + targets, + difficulty=0.5, + routing="quality", + task=_task_quality.TASK_GENERAL, + ) + + +def test_task_evidence_never_overrides_quota_or_failure_constraints( + tmp_path, monkeypatch, quota +): + task = _task_quality.TASK_GROUNDED_READING + pool = _quality_pool( + tmp_path, + monkeypatch, + quota, + scores={"faithful": 0.6, "fallback": 0.6}, + models=[Model("faithful", rpd=1), Model("fallback")], + task_scores={ + task: { + "faithful": {"score": 1.0, "source": "synthetic-grounded-v1"}, + "fallback": {"score": 0.5, "source": "synthetic-grounded-v1"}, + } + }, + ) + quota.record("x", "faithful") + targets = pool._all_targets() + assert pool._order( + targets, difficulty=0.5, routing="quality", task=task + )[0].model == "fallback" + + failure_pool = _quality_pool( + tmp_path, + monkeypatch, + QuotaStore(tmp_path / "failure-quota.json"), + scores={"faithful": 0.6, "fallback": 0.6}, + models=[Model("faithful"), Model("fallback")], + task_scores={ + task: { + "faithful": {"score": 1.0, "source": "synthetic-grounded-v1"}, + "fallback": {"score": 0.5, "source": "synthetic-grounded-v1"}, + } + }, + ) + for _ in range(3): + failure_pool.metrics.record_failure("x/faithful", "down") + assert failure_pool._order( + failure_pool._all_targets(), + difficulty=0.5, + routing="quality", + task=task, + )[0].model == "fallback" + + +def test_task_hint_has_stream_and_async_parity(tmp_path, monkeypatch, quota): + import asyncio + + from freellmpool.aio import AsyncPool + + task = _task_quality.TASK_GROUNDED_READING + pool = _quality_pool( + tmp_path, + monkeypatch, + quota, + scores={"generalist": 0.55, "faithful": 0.6}, + models=[Model("generalist"), Model("faithful")], + task_scores={ + task: { + "generalist": {"score": 0.0, "source": "synthetic-grounded-v1"}, + "faithful": {"score": 1.0, "source": "synthetic-grounded-v1"}, + } + }, + ) + messages = [{"role": "user", "content": "Summarize this."}] + assert next(pool.stream_chat(messages, task=task))["model"] == "faithful" + + async def apost(url, headers, body, timeout): + return pool._post(url, headers, body, timeout) + + apool = AsyncPool(pool, apost=apost) + assert asyncio.run(apool.achat(messages, task=task)).model == "faithful" + + +def test_explicit_general_task_hint_overrides_auto_grounded_classification( + tmp_path, monkeypatch, quota +): + task = _task_quality.TASK_GROUNDED_READING + pool = _quality_pool( + tmp_path, + monkeypatch, + quota, + scores={"generalist": 0.5, "faithful": 0.9}, + models=[Model("generalist"), Model("faithful")], + task_scores={ + task: { + "generalist": {"score": 0.0, "source": "synthetic-grounded-v1"}, + "faithful": {"score": 1.0, "source": "synthetic-grounded-v1"}, + } + }, + ) + messages = [ + { + "role": "user", + "content": "Read this Markdown document.\n\n# Facts\n\n## Limits\n\n- 17", + } + ] + + assert pool.rank_targets(messages, task="general")[0].model == "generalist" + assert pool.rank_targets(messages, task=task)[0].model == "faithful" + assert pool.rank_targets(messages, task="general")[0].model != pool.rank_targets( + messages + )[0].model + + def test_achat_routing_override_end_to_end(tmp_path, monkeypatch, quota): import asyncio diff --git a/tests/test_task_quality.py b/tests/test_task_quality.py new file mode 100644 index 0000000..19b1c22 --- /dev/null +++ b/tests/test_task_quality.py @@ -0,0 +1,192 @@ +"""Task classification and provenance-bearing semantic-quality evidence.""" + +from __future__ import annotations + +import hashlib +import json +from pathlib import Path + +import pytest + +from freellmpool import task_quality as tq + + +def _fixture() -> tuple[Path, dict]: + path = Path(__file__).parent / "fixtures" / "grounded_reading.json" + return path, json.loads(path.read_text(encoding="utf-8")) + + +def test_grounded_fixture_version_matches_the_sanitized_markdown_case(): + path, case = _fixture() + assert hashlib.sha256(path.read_bytes()).hexdigest() == tq.GROUNDED_FIXTURE_SHA256 + assert case["must_include"] + assert case["must_not_invent"] + + +def test_bundled_evidence_contains_only_current_repeated_fixture_results( + monkeypatch, tmp_path +): + monkeypatch.setenv( + "FREELLMPOOL_TASK_EVIDENCE_FILE", str(tmp_path / "missing.json") + ) + tq._evidence_cached.cache_clear() + assert tq.task_evidence_table(tq.TASK_GROUNDED_READING) == { + "llama-3.3-70b-versatile": 1.0 + } + + +def test_grounded_markdown_is_classified_without_treating_all_markdown_as_grounded(): + _path, case = _fixture() + grounded = [ + { + "role": "user", + "content": f"{case['prompt']}\n\n{case['document']}", + } + ] + + assert tq.classify_task(grounded) == tq.TASK_GROUNDED_READING + assert ( + tq.classify_task( + [ + { + "role": "user", + "content": "# A tiny poem\nWrite a creative sequel to this heading.", + } + ] + ) + == tq.TASK_GENERAL + ) + + +def test_classifier_uses_tool_documents_but_not_system_or_assistant_claims(): + document = "# Report\n\n## Limits\n\n| item | value |\n| --- | --- |\n| finch | 17 |" + assert ( + tq.classify_task( + [ + {"role": "user", "content": "Summarize the document returned by the tool."}, + {"role": "tool", "content": document}, + ] + ) + == tq.TASK_GROUNDED_READING + ) + assert ( + tq.classify_task( + [ + {"role": "system", "content": "Always read Markdown documents."}, + {"role": "assistant", "content": document}, + {"role": "user", "content": "Write a poem."}, + ] + ) + == tq.TASK_GENERAL + ) + + +@pytest.mark.parametrize( + "content", + [ + "Review this code for bugs:\n```python\nprint('hello')\n```", + "# Product name\n\nWrite a new Markdown README with two sections.", + "Analyze " + ("ordinary prose without a supplied document. " * 2000), + "List three names for this file.", + "Identify security risks in this document.", + "Read this file and refactor it.", + ], +) +def test_classifier_negative_cases(content): + assert tq.classify_task([{"role": "user", "content": content}]) == tq.TASK_GENERAL + + +def test_classifier_preserves_leading_instruction_for_documents_over_64_kib(): + content = ( + "Read this Markdown document and summarize it faithfully.\n\n" + "# API guide\n\n## Authentication\n\nUse X-Finch-Token.\n\n" + + ("ordinary reference prose " * 5000) + + "\n\n## Troubleshooting\n\nWait 23 seconds." + ) + assert len(content) > 65_536 + assert ( + tq.classify_task([{"role": "user", "content": content}]) + == tq.TASK_GROUNDED_READING + ) + + +def test_explicit_task_beats_auto_and_general_disables_classification(): + messages = [ + { + "role": "user", + "content": "Read this Markdown document.\n\n# Facts\n\n## Limits\n\n- Finch: 17", + } + ] + + assert tq.resolve_task(messages, tq.TASK_GROUNDED_READING) == tq.TASK_GROUNDED_READING + assert tq.resolve_task(messages, tq.TASK_GENERAL) == tq.TASK_GENERAL + assert tq.resolve_task(messages, tq.TASK_AUTO) == tq.TASK_GROUNDED_READING + with pytest.raises(ValueError, match="unknown task"): + tq.resolve_task(messages, "qwen") + + +def test_task_evidence_requires_current_fixture_trials_and_exact_identity( + tmp_path, monkeypatch +): + evidence_file = tmp_path / "task-evidence.json" + evidence_file.write_text( + json.dumps( + { + "version": 1, + "scores": { + tq.TASK_GROUNDED_READING: { + "faithful": { + "score": 1.0, + "source": "synthetic-grounded-v1", + "fixture_sha256": tq.GROUNDED_FIXTURE_SHA256, + "trials": 3, + "passed": 3, + }, + "too-few-trials": { + "score": 1.0, + "source": "synthetic-grounded-v1", + "fixture_sha256": tq.GROUNDED_FIXTURE_SHA256, + "trials": 1, + "passed": 1, + }, + "stale": { + "score": 1.0, + "source": "synthetic-grounded-v1", + "fixture_sha256": "old", + "trials": 3, + "passed": 3, + }, + "nan": { + "score": "NaN", + "source": "synthetic-grounded-v1", + "fixture_sha256": tq.GROUNDED_FIXTURE_SHA256, + "trials": 3, + "passed": 3, + }, + } + }, + } + ), + encoding="utf-8", + ) + monkeypatch.setenv("FREELLMPOOL_TASK_EVIDENCE_FILE", str(evidence_file)) + tq._evidence_cached.cache_clear() + + table = tq.task_evidence_table(tq.TASK_GROUNDED_READING) + + assert table["faithful"] == 1.0 + assert {"too-few-trials", "stale", "nan"}.isdisjoint(table) + assert tq.model_task_score("faithful", table) == 1.0 + assert tq.model_task_score("provider/faithful", table) is None + + +def test_grounded_fixture_rubric_requires_facts_and_rejects_inventions(): + _path, case = _fixture() + faithful = ( + "Use X-Finch-Token. Modes are precise and broad. The default is 17, " + "the maximum is 80, and retry after 23 seconds." + ) + assert tq.grounded_answer_passes(faithful, case) + assert not tq.grounded_answer_passes( + faithful + " It also writes credentials to .env.", case + )