feat(ai): add promptCacheKey stream option to override the prompt cache key - #6654
feat(ai): add promptCacheKey stream option to override the prompt cache key#6654alasano wants to merge 5 commits into
Conversation
…-override # Conflicts: # packages/ai/test/openai-codex-stream.test.ts
…-override # Conflicts: # packages/agent/src/agent.ts
|
Resolved the conflict caused by main's stream compat refactor touching the Agent constructor. |
…-override # Conflicts: # packages/ai/src/api/openai-codex-responses.ts # packages/ai/src/providers/opencode.models.ts # packages/ai/src/providers/openrouter.models.ts # packages/ai/src/providers/vercel-ai-gateway.models.ts
|
Merged main again. The only real conflict was openai-codex-responses, where #6618 made codexSessionId feed both prompt_cache_key and the session-id / x-client-request-id headers. That was fine while they were the same value, but with an override they're not, so I split them apart the way openai-responses already does it. The override only reaches the request body, the affinity headers and the websocket pool key stay on sessionId, otherwise two sessions sharing a key would share a pooled connection and its previous_response_id state. cacheRetention none still wins over the override. This is a pretty small change. Any chance we can get it merged so I don't have to keep fixing conflicts? If there's a reason it needs to wait, just let me know. |
Closes #6627
Summary
Adds an opt-in
promptCacheKey?: stringtoStreamOptions. When set, it replacessessionIdas the input toclampOpenAIPromptCacheKey()in the four providers that sendprompt_cache_key:When unset, it falls back to
sessionIdand behavior is unchanged.Not changed
nonestill omits the key on the responses pathsessionIdThe option is also forwarded where the other cache-related stream options already flow in
packages/agent: theAgentconfig, thestreamProxyserialized options, andAgentHarnessStreamOptions(so it's patchable per request viabefore_provider_request).Why
Without this there's no way to share the prompt cache between sessions that have a large identical prompt prefix, because the key is hardcoded to the per-session
sessionIdand identical prefixes end up routed to different cache shards. More details in #6627.Validation
sessionIdand is clamped to 64 chars, retentionnonestill omits it with the override set, and the codex and azure paths include it in the request bodynpm run checkand./test.shpass