Skip to content

feat(ai): add promptCacheKey stream option to override the prompt cache key - #6654

Open
alasano wants to merge 5 commits into
earendil-works:mainfrom
alasano:feat/prompt-cache-key-override
Open

feat(ai): add promptCacheKey stream option to override the prompt cache key#6654
alasano wants to merge 5 commits into
earendil-works:mainfrom
alasano:feat/prompt-cache-key-override

Conversation

@alasano

@alasano alasano commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Closes #6627

Summary

Adds an opt-in promptCacheKey?: string to StreamOptions. When set, it replaces sessionId as the input to clampOpenAIPromptCacheKey() in the four providers that send prompt_cache_key:

  • openai-responses
  • openai-completions
  • openai-codex-responses (covers both the SSE and websocket transports, they build the same request body)
  • azure-openai-responses

When unset, it falls back to sessionId and behavior is unchanged.

Not changed

  • retention none still omits the key on the responses path
  • the completions baseUrl/retention gating is unchanged, the override only replaces the value
  • the 64-char clamp still applies to the overridden value
  • session-affinity headers stay tied to sessionId

The option is also forwarded where the other cache-related stream options already flow in packages/agent: the Agent config, the streamProxy serialized options, and AgentHarnessStreamOptions (so it's patchable per request via before_provider_request).

Why

Without this there's no way to share the prompt cache between sessions that have a large identical prompt prefix, because the key is hardcoded to the per-session sessionId and identical prefixes end up routed to different cache shards. More details in #6627.

Validation

  • New assertions in the existing request-body test files: the override wins over sessionId and is clamped to 64 chars, retention none still omits it with the override set, and the codex and azure paths include it in the request body
  • npm run check and ./test.sh pass

…-override

# Conflicts:
#	packages/agent/src/agent.ts
@alasano

alasano commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

Resolved the conflict caused by main's stream compat refactor touching the Agent constructor.

…-override

# Conflicts:
#	packages/ai/src/api/openai-codex-responses.ts
#	packages/ai/src/providers/opencode.models.ts
#	packages/ai/src/providers/openrouter.models.ts
#	packages/ai/src/providers/vercel-ai-gateway.models.ts
@alasano

alasano commented Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

Merged main again. The only real conflict was openai-codex-responses, where #6618 made codexSessionId feed both prompt_cache_key and the session-id / x-client-request-id headers. That was fine while they were the same value, but with an override they're not, so I split them apart the way openai-responses already does it. The override only reaches the request body, the affinity headers and the websocket pool key stay on sessionId, otherwise two sessions sharing a key would share a pooled connection and its previous_response_id state. cacheRetention none still wins over the override.

This is a pretty small change. Any chance we can get it merged so I don't have to keep fixing conflicts? If there's a reason it needs to wait, just let me know.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow overriding the OpenAI prompt_cache_key via stream options

1 participant