Cap API provider output tokens and web-search spend per request#491
Merged
Conversation
Bound a single tracked run's cost on the four direct-API providers: - Output-token caps on anthropic-api, openai-api, openrouter, mistral-api. - Web-search budget caps: Anthropic max_uses=1, OpenAI maxToolCalls=2, OpenRouter native-first web plugin with search_context_size + Exa max_results. The onboarding research path keeps a deeper, named cap. Code defaults centralized in providers/config.ts; no schema, no config dependency. Extracted from the DB-config PR so it can ship on its own.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…d-caps # Conflicts: # packages/lib/src/providers/registry/anthropic-api.ts # packages/lib/src/providers/registry/openai-api.ts
Revert the OpenRouter plugin switch: ":online" is exactly equivalent to
plugins: [{ id: "web" }] and already routes to native provider search, so
the explicit config changed nothing but the request shape.
Raise the OpenAI and OpenRouter caps to 16k and Mistral to 8k. The caps
bound worst-case spend, and on OpenAI/OpenRouter they also count reasoning
tokens, so the old 8k left little room on reasoning targets.
A capped response still stores as a normal run, so log the finish reason
rather than failing — a truncated answer would otherwise land as a
real-looking result with fewer brand mentions.
Measured against ~30k tracked ChatGPT answers as a proxy for answer length: p99 is ~3.4k tokens and ~0.4% run past 4k, so the long-standing 4k cap was clipping roughly 1 run in 240. A clipped answer stores as a normal run with fewer brand mentions, so those were scoring as quiet false negatives.
Anthropic was already hard-capped at 4000 in production, so keep it there — this changes nothing for Anthropic. OpenAI, OpenRouter, and Mistral get a uniform 8000 worst-case bound; on OpenAI/OpenRouter that ceiling also covers reasoning tokens, and warnIfOutputCapped logs any run that hits it so the number can be revisited from real traffic.
The /v1/conversations response has no finish_reason field (confirmed against the client-ts schema), so warnIfOutputCapped only applies to the chat path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First of a stack extracted from #483 so the DB-config work can land in small, releasable pieces.
What
Hard per-request spend caps on the four direct-API providers, so no single tracked run can spend unboundedly:
anthropic-api,openai-api,openrouter,mistral-api(centralized inproviders/config.ts).max_uses = 1, OpenAImaxToolCalls = 2, OpenRouter native-first web plugin withsearch_context_size+ Examax_results. The onboarding structured-research path keeps its own deeper, named cap.Why it's low-risk
No schema, no migration, no dependency on the config hierarchy — just provider-layer constants and their use. Supersedes the mechanism half of #452.
Tests: 4 new provider cap suites (10 cases); lib typechecks.