Add cross-provider protocol conformance canaries

[0xzr]

Summary

• add bounded, deterministic per-model canaries for chat, streaming, tools, JSON object, JSON schema, vision, Responses, and Anthropic Messages
• gate feature-specific automatic routing on current provider/model evidence while preserving exact-pin overrides and safe cache provenance
• expose machine-readable capabilities in CLI/proxy model and status surfaces, including plugin providers
• add protected catalog-sentinel automation, cross-platform atomic evidence storage, and SDK-compatible Responses tool streaming

Verification

• pytest --cov=freellmpool --cov-branch --cov-report=json:.coverage.json
• line coverage 86.47%; branch coverage 74.51%
• ruff check .
• focused strict mypy
• catalog/count/release/proxy-stress gates
• Bandit, pip-audit, and zizmor
• OpenAI Python SDK 2.48.0 consumed both text and function-call Responses streams
• Codex Sol 5.6 xhigh approved exact tree 56099172d493138210b62a88c160136ebdf75a9c

Fixes #71

Summary by Sourcery

Introduce per-model protocol conformance canaries and wire their evidence into routing, proxy/CLI surfaces, and CI workflows.

New Features:

• Add protocol conformance store and deterministic feature canaries for chat, streaming, tools, JSON object/schema, vision, Responses, and Anthropic Messages.
• Expose per-model capabilities and verified_features in CLI models JSON output and proxy /status and /models responses, including plugin providers.
• Add CLI conformance subcommands to run bounded feature probes and inspect sanitized evidence state JSON.

Enhancements:

• Update routing, async routing, and cache keying to be feature-aware, restricting automatic routing and cache hits to verified targets while preserving exact provider/model pins.
• Extend Responses and Anthropic Messages shims to support tools, function-call history, structured streaming events, and protocol-aware provenance.
• Forward OpenAI response_format through client, async pool, and proxy while keeping token bounds and adapter compatibility.

CI:

• Expand catalog-sentinel GitHub workflow with longer authenticated timeout and a bounded protocol conformance canaries step whose artifacts and state are cached and uploaded.

Documentation:

• Document protocol conformance canaries, routing contract, protected automation, and operator privacy guarantees in PROTOCOL_CONFORMANCE.md and reference it from README and CONTRIBUTING.

Tests:

• Add comprehensive conformance store, feature inference, routing, proxy, client, cache, CLI, and workflow tests to validate bounded behavior, evidence sanitization, and protocol streaming invariants.

[sourcery-ai]

Sorry @0xzr, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

[sourcery-ai]

Reviewer's Guide

Add a protocol conformance subsystem (store, canaries, routing, CLI/proxy surfaces, and CI workflow hooks) that records bounded per-model feature evidence and uses it to gate feature-specific auto-routing, caching, and streaming across OpenAI-style, Anthropic Messages, and Responses APIs.

Sequence diagram for feature-aware chat routing with protocol conformance

sequenceDiagram
    actor User
    participant Proxy
    participant Pool
    participant ConformanceStore
    participant Provider

    User->>Proxy: POST /v1/chat/completions
    Proxy->>Proxy: _handle_chat(req)
    Proxy->>Proxy: _normalize_messages(messages)
    Proxy->>Pool: chat(messages, tools, response_format, protocol)
    Pool->>ConformanceStore: required_features(messages, tools, response_format, protocol)
    Pool->>ConformanceStore: verified_targets(all_targets, features, exact_pin)
    ConformanceStore-->>Pool: filtered_targets
    Pool->>Pool: cache.make_key(..., response_format, protocol)
    Pool->>Pool: cache.get(key)
    alt cache_hit and feature_cache_eligible
        Pool-->>Proxy: Reply(from cache)
        Proxy-->>User: HTTP 200 (cached reply)
    else cache_miss or cache_not_eligible
        loop over candidates
            Pool->>Provider: call(provider, model, messages, tools, tool_choice, response_format)
            Provider-->>Pool: Reply
        end
        Pool-->>Proxy: Reply
        Proxy-->>User: HTTP 200 (live reply)
    end

Loading

File-Level Changes

Change	Details	Files
Expose per-model protocol conformance evidence on proxy/CLI status and model listings.	Augment OpenAI and Anthropic /models payloads to include capabilities and verified_features derived from the conformance store. Extend /status provider model rows with capabilities and verified_features using a shared conformance snapshot. Ensure snapshots are taken once and reused to avoid inconsistent reads and extra IO.	src/freellmpool/proxy.py src/freellmpool/router.py tests/test_proxy.py tests/test_conformance.py
Introduce a bounded, cross-process-safe protocol conformance store and deterministic canary runner, and integrate them with routing and caching.	Add ConformanceStore for JSON-backed per-target evidence with size limits, schema validation, locking, and automatic eviction of oldest targets at capacity. Implement required_features and run_target_canaries to infer required protocol features and run fixed synthetic canary prompts/tools/vision across features like chat, streaming, tools, JSON, JSON schema, vision, Responses, and Anthropic Messages. Wire ConformanceStore into Pool/AsyncPool construction, routing target selection, and cache key eligibility to ensure feature-specific requests hit only verified targets and invalidate feature cache hits on regression.	src/freellmpool/conformance.py src/freellmpool/router.py src/freellmpool/aio.py src/freellmpool/client.py src/freellmpool/cache.py tests/test_conformance.py tests/test_cache.py
Extend the CLI with conformance commands and plugin-aware catalogs, including bounded secret import for workflow automation.	Add _runtime_catalog merging built-in and registered plugin providers and use it in providers/models commands. Introduce freellmpool conformance run and status subcommands that run a bounded feature matrix, persist sanitized evidence, and emit machine-readable JSON. Implement _conformance_env to import a JSON key map from FREELLMPOOL_CONFORMANCE_KEYS_JSON restricted to catalog-declared env names, with strict size/value bounds. Update CLI models JSON output to include capabilities and verified_features per provider/model and add tests covering plugin providers, bounded runs, secret handling, and sanitized status JSON.	src/freellmpool/cli.py tests/test_cli.py
Enhance proxy behavior for Responses, Anthropic Messages, multimodal chat, and structured response_format while keeping cache provenance and routing safe.	Normalize messages to preserve multimodal content arrays for vision routing while still handling legacy text content. Extend proxy chat handler to accept and validate response_format, forward it into router/Pool, and disable plain token streaming for structured responses. Implement Responses shim enhancements: translate Responses tools/tool_choice into Chat tools, map tool_calls and function_call history to Responses items, and generate SDK-compatible SSE streams with sequence_number and typed events for both text and function calls. Tag routed requests with protocol (responses/anthropic_messages) so required_features can gate routing appropriately and ensure cache keys include response_format and protocol to avoid cross-protocol cache collisions.	src/freellmpool/proxy.py src/freellmpool/router.py src/freellmpool/aio.py src/freellmpool/cache.py tests/test_proxy.py scripts/stress_proxy.py
Update CI/catalog sentinel workflow and documentation to account for protocol conformance artifacts and operator contract.	Extend catalog-sentinel GitHub Actions workflow to run freellmpool conformance run with a bounded feature matrix, share its state via cache, and upload conformance artifacts alongside probe artifacts, with an increased timeout to cover combined budgets. Add tests that compute the theoretical network budget for probe and conformance steps and assert workflow timeout is safely higher. Document the protocol conformance operator contract, routing behavior, environment variables, and privacy limits in a new PROTOCOL_CONFORMANCE.md and reference it from README and CONTRIBUTING. Update CHANGELOG to describe the new deterministic per-model protocol conformance canaries and routing changes.	.github/workflows/catalog-sentinel.yml tests/test_catalog_sentinel.py docs/PROTOCOL_CONFORMANCE.md README.md CONTRIBUTING.md CHANGELOG.md
Ensure client adapters and tests correctly handle response_format forwarding and Gemini limitations.	Update client.call and adapter functions to accept and forward response_format for OpenAI-compatible adapters while rejecting it for Gemini, keeping max_tokens budgets unchanged. Add tests verifying response_format forwarding and the non-relaxation of token bounds, plus additional Responses streaming shape checks. Minor stress and client test adjustments to assert new event sequences and behavior.	src/freellmpool/client.py src/freellmpool/aio.py tests/test_client.py scripts/stress_proxy.py

Assessment against linked issues

Issue	Objective	Addressed	Explanation
#71	Implement deterministic, quota-bounded per-model protocol conformance canaries for chat, streaming, tools, JSON object/schema, vision, Responses API, and Anthropic Messages, including semantic validation of outputs and re-run support after model/adapter changes.	✅
#71	Integrate protocol conformance evidence into routing and status/model surfaces so that feature-specific requests are routed only to targets that have passed required features (or are explicitly pinned), while preserving per-model failures and serving provenance and ensuring unsupported capability errors are not treated as availability failures.	✅
#71	Expose machine-readable, sanitized protocol capability results via CLI and automation (including plugin providers and catalog-sentinel workflow), with bounded canary execution, separate feature-level verification from basic health, and documentation of the operator/privacy contract.	✅

Possibly linked issues

• Add cross-provider protocol conformance canaries #71: PR adds bounded per-model protocol conformance canaries, machine-readable capability outputs, and feature-aware routing as requested.
• Add cross-provider protocol conformance canaries #71: PR implements the protocol conformance canaries, evidence storage, CLI/proxy surfaces, and workflow changes requested by issue.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.