Complete tinyagents inference migration (Motion B): crate-native provider clients, delete duplicated provider stack

[senamakel]

Summary

Complete the tinyagents inference migration ("Motion B") by registering crate-native providers::openai clients directly in the turn ModelRegistry, eliminating the remaining openhuman Provider trait objects in the agent harness and deleting the duplicated in-house provider stack. This is the follow-up to Motion A (PR #4726), which put every agent turn path on a seam-owned TurnModelSource so that no harness struct holds Arc<dyn Provider>.

Problem / Context

After Motion A, the harness holds TurnModelSource everywhere, but the source still wraps openhuman's Box<dyn Provider> stack (inference/provider/compatible*.rs, reliable.rs, router.rs — ~30k lines) that re-implements what tinyagents 1.7 ships natively (harness::providers::openai, harness::retry, harness::model::ModelRegistry). The crate ModelRegistry + RunPolicy.fallback + capability gating are already wired in assemble_turn_harness (see docs/tinyagents-phase3-router-registry-design.md) — the registered models are just ProviderModel wrappers rather than crate-native clients. Consolidating removes the double-maintenance and the isomorphic type-conversion seam.

Context/links:

• Motion A PR: refactor(agent): hold TurnModelSource across all turn paths (tinyagents Motion A, #4249) #4726
• Anchor: Improve agent harness with LangGraph-style state machine or Polaris-like architecture #4249
• Design + tracking: docs/tinyagents-phase3-router-registry-design.md, docs/tinyagents-drift-ledger.md (rows P1-8/P1-9), docs/tinyagents-inference-migration-plan.md (Phases 2–4).

Scope

In scope — the remaining dyn Provider boundaries in agent/ (all provider-resolution/build sites that Motion A intentionally left):

• inference/provider/factory.rs — build crate providers::openai clients (via ProviderSpec) for BYOK slugs, Ollama, LM Studio, the cloud slug, instead of CompatibleProvider; keep the provider-string grammar.
• subagent_runner/ops/provider.rs::resolve_subagent_provider — return a TurnModelSource/registry entry instead of Arc<dyn Provider> (this churns its 9 unit tests — deferred from Motion A on purpose).
• tools/delegate.rs, triage/routing.rs — build sources/crate clients instead of Box<dyn Provider>.
• Builder .provider()/.provider_arc() setters and the TurnModelSource::provider() escape hatch — remove once nothing needs a raw provider (incl. the streaming session cap-hit checkpoint in session/turn/session_io.rs, which currently uses source.provider().chat(...) because crate ChatModel::invoke has no delta sink — needs the crate stream API).
• Bespoke providers stay as host ChatModel impls (openhuman_backend.rs, claude_code/, claude_agent_sdk/, openai_codex.rs) — Phase 4.
• Delete compatible*.rs, reliable.rs, router.rs after cutover.

Smaller stragglers (can land independently, ahead of the big cutover):

• tinyflows/caps.rs — one-shot .chat() caller still on create_chat_provider (round-trips tool_calls + reasoning into a JSON envelope; needs the seam message converter exposed).
• src/bin/inference_probe.rs — debug binary still names create_chat_provider.

Not in scope — the inference.* RPC surface, provider-string grammar, local-runtime process management, voice, /v1 server endpoint (all host-owned; unchanged).

Acceptance criteria

• Crate-native clients registered — the turn ModelRegistry holds providers::openai clients built from config for BYOK/Ollama/LM Studio/cloud; ProviderModel no longer wraps the OpenAI-compat path.
• Provider stack deleted — inference/provider/compatible*.rs, reliable.rs, router.rs removed; the 207-test compatible_tests.rs behavioral cases re-expressed against the crate providers::openai fixtures (upstream where generic).
• No dyn Provider in agent/ — grep -rl 'dyn Provider' src/openhuman/agent/ --include='*.rs' returns only bespoke host ChatModel impls (if any); the TurnModelSource::provider() escape hatch and the builder provider setters are gone.
• Streaming checkpoint on the crate API — session_io.rs cap-hit checkpoint streams via the crate ChatModel stream path, not source.provider().chat.
• Stragglers migrated — tinyflows/caps.rs and inference_probe.rs off create_chat_provider.
• Behavior parity — streaming deltas, cost footer (charged-USD + cached tokens), tool-call timeline, and the same-family fallback ordering are byte-parity vs. the current stack (golden mock-backend transcripts + per-provider live smoke); regressions Streamed model calls run in detached spawn: thread_id/task-locals lost, cancellation doesn't stop billed provider call #4460 / $0-turn are not reintroduced.
• Diff coverage ≥ 80% — the implementing PR(s) meet the changed-lines coverage gate (Vitest + cargo-llvm-cov, enforced by .github/workflows/ci-lite.yml).

Related

• Motion A: refactor(agent): hold TurnModelSource across all turn paths (tinyagents Motion A, #4249) #4726 (refactor(agent): hold TurnModelSource across all turn paths)
• Improve agent harness with LangGraph-style state machine or Polaris-like architecture #4249 (tinyagents migration)
• docs/tinyagents-phase3-router-registry-design.md, docs/tinyagents-drift-ledger.md

https://claude.ai/code/session_018MnMVgnzxchtcs1DDUxTfF

[senamakel]

Progress update

Landed / in review:

• ✅ tinyflows/caps.rs agent-node completion → crate ChatModel — PR fix(flows): preserve literal tool-call markup in tool-less completions #4750 (standalone; independent of Motion A). Adds reusable seam converters chat_messages_to_model_messages + model_response_to_chat_response and factory::create_chat_model_pinned for the node raw/BYOK override.

Scoped out (no migration):

• ⏭️ src/bin/inference_probe.rs — on inspection its run_raw mode deliberately exercises the raw provider.chat() + provider.supports_native_tools() as a provider-layer diagnostic. Migrating it to ChatModel would defeat its purpose; it should stay raw (or, post-cutover, gain a separate crate-client probe mode). Removing it from the straggler list.

Still open — the core of this issue (the big cutover):

• Register crate-native providers::openai clients directly in the turn ModelRegistry (built from config for BYOK / Ollama / LM Studio / cloud) instead of ProviderModel-wrapped host providers.
• Delete inference/provider/compatible*.rs, reliable.rs, router.rs; re-express the 207-test compatible_tests.rs behavioral cases against crate providers::openai fixtures.
• Retire the remaining provider-resolution boundaries (factory::create_chat_provider/create_routed_provider, resolve_subagent_provider, tools/delegate.rs, triage/routing.rs, builder setters) and the TurnModelSource::provider() escape hatch (incl. moving the session_io.rs streaming cap-hit checkpoint onto the crate stream API).
• Bespoke providers (openhuman_backend, claude_code, claude_agent_sdk, openai_codex) stay as host ChatModel impls (Phase 4).

This remainder is a large, behavior-parity-critical effort (golden mock-backend transcripts + per-provider live smoke; #4460 / $0-turn safety) that depends on the crate providers::openai / ProviderSpec API — best sequenced as its own PR series after Motion A (#4726) merges.

https://claude.ai/code/session_018MnMVgnzxchtcs1DDUxTfF

[senamakel]

Progress update — crate-side foundation started

The big cutover (replacing the host OpenAiCompatibleProvider with crate-native providers::openai clients + deleting compatible*.rs) is hard-blocked on the crate being Bearer-only — OpenAiModel set auth in exactly one place (transport.rs, Authorization: Bearer only), so deleting the host client would drop auth for Gemini (x-goog-api-key), any XApiKey/Custom BYOK provider, and the managed backend. That deletion cannot land in the host PR (#4726) without regressing those providers.

Started the actual unblocking step — crate PR: tinyhumansai/tinyagents#44

• Adds a configurable AuthStyle to OpenAiModel: None / Bearer (default) / XApiKey / Anthropic (x-api-key + anthropic-version) / Custom(header), plus a repeatable with_header(..) for static attribution headers (OpenRouter HTTP-Referer/X-Title, etc.).
• Backward compatible (defaults to Bearer); pure unit-tested auth_headers mapping; 6 new tests + existing 40 openai tests green.

Sequencing from here (the multi-PR series this issue tracks):

1. ✅ Crate auth styles — tinyagents#44 (this).
2. ⏳ Crate: per-tier registry helper + any remaining wire-parity gaps (usage/cost fields, prompt-guided tool folding) so a single provider can back the workload-tier fallback set.
3. ⏳ Release tinyagents + re-pin in both host Cargo worlds.
4. ⏳ Host: factory builds crate OpenAiModel (with the right AuthStyle) for BYOK/local/cloud; build_turn_models/routing move onto the crate ModelRegistry.
5. ⏳ Host: rewrite bespoke providers (openhuman_backend, claude_code, claude_agent_sdk, openai_codex) as host ChatModel impls.
6. ⏳ Host: delete compatible*.rs/reliable.rs/router.rs; re-express compatible_tests.rs (207 tests) against crate fixtures; retire the TurnModelSource::provider() escape hatch.

Steps 4–6 (the host deletions) require step 3 (a crate release), so they stay out of #4726 by construction — #4726 remains the clean Motion A.

https://claude.ai/code/session_018MnMVgnzxchtcs1DDUxTfF

[senamakel]

Crate-feature series — progress (host cutover is gated on these)

Landing the crate features OpenAiModel needs before the host can drop its bespoke wire client, one PR at a time:

1. ✅ Auth styles — tinyagents#44 (merged). None/Bearer/XApiKey/Anthropic/Custom + static headers.
2. ✅ Re-pin — openhuman bumped to crate main; both Cargo worlds build green, no seam adaptation (the other ~10 crate commits were additive).
3. 🔵 Temperature suppression + override — tinyagents#47 (open). with_temperature_unsupported_models (*-glob → omit temperature) + with_temperature_override; pure effective_temperature helper, 4 tests.

Remaining crate gaps found by diffing the host OpenAiCompatibleProvider builder against OpenAiModel (each a follow-up crate PR):
4. ⏳ merge_system_into_user — fold the system turn into the first user message (multimodal-content-aware).
5. ⏳ ollama_num_ctx — local context sizing (options.num_ctx).
6. ⏳ extra_query_param — provider-specific query params on the endpoint URL.
7. ⏳ /v1/responses primary/fallback (OpenAI Responses API) — larger; distinct endpoint + wire shape.

Once #4–7 land + release, the host side is one PR: factory builds crate OpenAiModel (auth + temp + extras) for BYOK/local/cloud; managed backend + claude-code/codex become host ChatModel impls; per-tier routing → crate ModelRegistry; delete compatible*.rs/reliable.rs/router.rs and re-express the 207-test suite against crate fixtures; retire the TurnModelSource::provider() escape hatch.

https://claude.ai/code/session_018MnMVgnzxchtcs1DDUxTfF

[senamakel]

Crate-feature series — essentially complete (re-scoped)

Digging into the remaining host OpenAiCompatibleProvider features showed most are already covered or host-side, not crate gaps:

Host feature	Disposition
Auth styles + headers	✅ crate #44 (merged)
Temperature suppression/override	✅ crate #47 (open)
merge_system_into_user	✅ crate #48 (open)
ollama_num_ctx	✅ already works via ModelRequest.provider_options → {"options":{"num_ctx":N}} (crate test translates_provider_options_for_local_openai_compatible_models proves it)
extra_query_param + /v1/responses	➡️ host-side — used only by openai_codex (factory.rs:1935 + the Responses-API path), a bespoke provider that stays a host ChatModel impl
with_openhuman_thread_id, session-JWT, openhuman.usage/billing envelope	➡️ host-side — managed backend stays a host ChatModel impl
with_native_tool_calling(false) (prompt-guided, local runtimes)	➡️ seam-side — folded at ProviderModel/dispatcher via supports_native_tools(); a middleware concern, not the wire client

So once #47 + #48 merge and the crate cuts a release, OpenAiModel can faithfully serve every generic BYOK/cloud OpenAI-compatible provider.

Remaining work = one host-side PR (large, parity-critical)

1. Factory builds a crate OpenAiModel for BYOK/cloud, mapping host config → with_auth_style + with_temperature_unsupported_models/with_temperature_override + with_merge_system_into_user + provider_options (num_ctx).
2. Rewrite the 4 bespoke providers as host ChatModel impls: openhuman_backend (JWT + billing envelope — note it currently builds on OpenAiCompatibleProvider, so this gates the deletion), claude_code, claude_agent_sdk, openai_codex (keeps its query-param + Responses-API logic).
3. Local runtimes: keep prompt-guided tool folding at the seam (or keep a host client for local).
4. Per-tier routing → crate ModelRegistry.
5. Delete compatible*.rs/reliable.rs/router.rs; re-express the 207-test suite against crate fixtures; retire TurnModelSource::provider().
6. Per-provider wire-parity validation (golden transcripts + live smoke) before each deletion.

https://claude.ai/code/session_018MnMVgnzxchtcs1DDUxTfF

[senamakel]

Host cutover PR — scaffolding started

Branch feat/inference-crate-provider-cutover (stacked on #4726's Motion A + the re-pin), submodule pinned to the openai-host-compat integration (tinyagents #44 merged + #47 + #48) so the host builds against all three crate features.

Landed (scaffold foundation):

• inference/provider/crate_openai.rs — the single boundary that turns resolved OpenAI-compatible provider config into a crate-native OpenAiModel (ChatModel): 1:1 host→crate AuthStyle map + with_temperature_* + with_merge_system_into_user + static headers (num_ctx etc. ride provider_options). Additive, unit-tested, both Cargo worlds green.

Remaining in this PR (the large tail):

1. Factory wiring — refactor provider-string/credential resolution to emit a CrateOpenAiConfig and call build_crate_openai_model for BYOK/cloud (replacing the OpenAiCompatibleProvider construction), behind the route projection.
2. Rewrite the 4 bespoke providers as host ChatModel impls: openhuman_backend (JWT + openhuman.usage/billing envelope — currently builds on OpenAiCompatibleProvider, so this gates the deletion), claude_code, claude_agent_sdk, openai_codex (keeps /v1/responses + query-param auth).
3. TurnModelSource/build_turn_models: hold crate ChatModels; per-tier routes → crate ModelRegistry.
4. Delete compatible*.rs/reliable.rs/router.rs; re-express compatible_tests.rs (207 tests) against crate fixtures; retire TurnModelSource::provider().
5. Per-provider wire-parity validation (golden transcripts + live smoke) before each deletion.

Gating: this PR can't merge until #47/#48 merge and tinyagents cuts a release (so the vendor pin references published API instead of the integration branch), and until the parity suite runs.

https://claude.ai/code/session_018MnMVgnzxchtcs1DDUxTfF

[senamakel]

Cutover branch — crate merged, re-pinned, scaffold builders ready

#47 and #48 merged → crate main (95fa263) now carries all three host-parity OpenAiModel features (auth #44, temperature #47, merge-system #48).

Landed on feat/inference-crate-provider-cutover:

• Re-pin to released crate main (10e719c9) — vendor pin now references published API (no longer the local integration branch); both Cargo worlds green.
• crate_openai scaffold (c…, 8d2321d0):
  • build_crate_openai_model(CrateOpenAiConfig) — the single boundary turning resolved OpenAI-compat config → a crate ChatModel (1:1 host→crate AuthStyle, temperature suppression/override, merge-system, static headers).
  • make_crate_openai_chat_model(...) — the drop-in parallel to the host make_openai_compatible_provider_with_config, same resolved inputs, returns a crate ChatModel. This is the ready swap point for each generic OpenAI-compat construction site.
  • Unit-tested (auth map, profile, factory-level builder, local-none shape).

Remaining = the coupled architectural flip (large, parity-gated):

1. TurnModelSource/build_turn_models: build crate ChatModels per workload tier instead of ProviderModels → per-tier routes onto the crate ModelRegistry (the routing rework).
2. Swap the factory's generic OpenAI-compat sites from make_openai_compatible_provider_with_config → make_crate_openai_chat_model; create_chat_provider/create_chat_model return crate ChatModels.
3. Rewrite the 4 bespoke providers as host ChatModel impls: openhuman_backend (note: dynamic session-JWT + openhuman.usage/billing envelope — the crate's static-key OpenAiModel can't serve it, so it needs its own host client), claude_code, claude_agent_sdk, openai_codex.
4. Delete compatible*.rs/reliable.rs/router.rs; re-express compatible_tests.rs (207) against crate fixtures; retire TurnModelSource::provider().
5. Per-provider wire-parity validation (golden transcripts + live smoke) before each deletion.

Steps 1–3 are tightly coupled (the return-type flip cascades create_chat_provider → TurnModelSource → build_turn_models → routing) and behavior-changing per provider, so they need the parity harness — which can't run in this environment — before the step-4 deletions.

https://claude.ai/code/session_018MnMVgnzxchtcs1DDUxTfF

[senamakel]

Coupled flip — the two hardest provider pieces landed

On feat/inference-crate-provider-cutover:

• crate_openai builders — build_crate_openai_model / make_crate_openai_chat_model: generic BYOK/cloud OpenAI-compat → crate ChatModel (auth map + temperature + merge-system + headers). Tested.
• openhuman_backend_model::OpenHumanBackendModel — the managed backend as a host ChatModel (this is the piece that gates deleting compatible*.rs). Key insight that made it tractable: the crate preserves the full response on ModelResponse.raw, so the openhuman.usage/billing charged-USD envelope still round-trips to the seam's usage_info_from_response. It resolves the dynamic session JWT per call → fresh crate OpenAiModel (Bearer), and injects the ambient thread_id via provider_options (crate flattens it to the body field — parity with with_openhuman_thread_id). Tested; core lib green.

Remaining coupled flip:

1. Bespoke ChatModel impls for claude_code / claude_agent_sdk (subprocess) and openai_codex (OAuth + /v1/responses + query-param) — analogous host rewrites, but non-HTTP so they don't reuse crate_openai.
2. The return-type flip — create_chat_provider/create_chat_model return crate ChatModels; swap factory sites to make_crate_openai_chat_model (BYOK) / OpenHumanBackendModel (managed); make_openhuman_backend returns the ChatModel.
3. TurnModelSource/build_turn_models: per-tier crate ChatModels → crate ModelRegistry (routing rework).
4. Delete compatible*.rs/reliable.rs/router.rs; re-express the 207-test suite; retire TurnModelSource::provider().
5. Per-provider wire-parity validation before each deletion.

The provider building blocks now exist for every non-subprocess provider; the wiring (steps 2–3) is the interlocking return-type change, still behavior-changing + parity-gated.

https://claude.ai/code/session_018MnMVgnzxchtcs1DDUxTfF