Conditional useTool() changes can invalidate the provider prompt cache

[zackleman]

Hope this is helpful

The conditional tools guide says that Flue announces tool changes through a resources signal, “keeping the transcript coherent while preserving the provider's prompt cache.”

I don't think that guarantee holds for custom tools mounted with conditional useTool().

At Flue v2.0.0, a rerender does append the resource delta, but it also rebuilds and replaces the native tools array passed to the provider. The appended signal preserves the history of when the tool changed; replacing the provider tool array changes the prompt prefix itself.

This matters because provider-native tool definitions precede the conversation in the cached prompt. Anthropic documents the cache hierarchy as tools → system → messages and says that modifying tool definitions invalidates the entire cache. OpenAI likewise requires an exact prefix match, including identical tools.

This report is specifically about conditional custom tools. Skills and subagents generally use stable dispatcher tools such as activate_skill and task, so they have a different model-facing shape.

Expected Behavior

Either:

1. conditional useTool() changes preserve the provider-visible tool prefix through a cache-safe mechanism; or
2. the documentation explains that resources signals preserve transcript coherence, while native custom-tool changes may still invalidate the prompt cache depending on the provider and model.

Steps to Reproduce

The ReleaseManager pattern from the docs is enough to expose the issue:

'use agent';
import { useModel, usePersistentState, useTool } from '@flue/runtime';
import * as v from 'valibot';

export function ReleaseManager() {
  useModel('anthropic/claude-sonnet-4-6');
  const [approved, setApproved] = usePersistentState('approved', false);

  useTool({
    name: 'record_approval',
    description: 'Record an operator approval code.',
    input: v.object({ code: v.string() }),
    async run() {
      setApproved(true);
      return 'Approval recorded.';
    },
  });

  if (approved) {
    useTool({
      name: 'publish_release',
      description: 'Publish the approved release.',
      input: v.object({}),
      async run() {
        return 'Published.';
      },
    });
  }

  return 'Prepare the release.';
}

1. Capture the serialized provider request before record_approval completes.
2. Complete record_approval, causing the agent to rerender with approved === true.
3. Capture the next provider request.
4. Compare the two native tools arrays.

The second request should include publish_release, so the native tool prefix is no longer identical. Under Anthropic's normal Messages API cache rules, that invalidates the cached tools/system/messages prefix.

Version Inspected

• Flue commit 5485efe2fc6e84fb2c1c0f98e5ccb8068f9394b6 (v2.0.0)
• @earendil-works/pi-ai and @earendil-works/pi-agent-core: 0.83.0

Relevant Runtime Path

In prepareRerenderTurn(), Flue:

1. evaluates the agent again;
2. assigns the newly declared custom tools to this.agentTools;
3. rebuilds the complete tool collection;
4. replaces this.agentLoop.state.tools;
5. returns that array as the next Pi Context.tools value; and
6. separately appends the resource delta narration.

The resource implementation also notes that a changed schema reaches the model through the native request tool array.

The resources signal is useful, but it is appended after the existing history. It cannot make an earlier, changed tool prefix identical again.

Provider references:

• Anthropic: Tool use with prompt caching
• OpenAI: Prompt caching and allowed_tools

Suggested First Step

I would start by qualifying the documentation. The current runtime can keep the transcript coherent, but it cannot make a general cache-preservation guarantee for conditional custom tools while it replaces the native provider tool array.

For a runtime fix, the existing resource reconciliation around prepareRerenderTurn() / narrateResourceDelta() looks like the natural place to persist a chronological tool-availability delta and let each provider adapter choose the strongest mapping it supports.

Acceptance Criteria

• The docs distinguish transcript coherence from provider prompt-cache preservation.
• Conditional tool behavior is documented for additions, removals, schema updates, and unsupported provider/model fallbacks.
• Cache-safe provider paths keep the original native tool prefix stable and express availability changes later in the conversation.
• The runtime rejects calls to tools that are no longer live, even if the model saw them earlier.
• Tool-availability changes survive persistence, recovery, replay, join/resume, and compaction.
• Tests compare raw serialized provider requests across add, remove, and update transitions and assert provider cache metrics where available.

Related: #456 covered dynamic tool discovery. This is narrower: it concerns the documented cache-preservation guarantee for deterministic conditional useTool() changes.

Verification Note

This is based on source inspection of the Flue and Pi paths plus the providers' documented caching contracts. A regression test that records the outgoing payload and provider-reported cache tokens before and after the transition would provide the final end-to-end confirmation.

[zackleman]

Some additional implementation context:

Pi already has a partial additions-only path

Pi 0.83.0 defines AgentToolResult.addedToolNames. The agent loop copies it into the persisted tool-result message, and supported provider adapters can move those definitions out of the initial top-level prefix and load them at the historical tool-result position:

• Contract: types.ts
• Agent-loop propagation: agent-loop.ts
• Deferred classification and fallback: deferred-tools.ts
• Anthropic adapter: anthropic-messages.ts
• OpenAI Responses adapter: openai-responses.ts

Flue currently does not emit addedToolNames, persist it in its canonical tool-outcome records, or reconstruct it during replay, so conditional additions bypass this optimization.

Wiring that field through would help with additions directly caused by a completed tool call, but it would not cover the whole Flue resource lifecycle:

• removal has no marker;
• updating an existing tool's description or schema changes the definition at its earlier load point;
• remove → re-add can reuse an old historical addition marker;
• a tool added because of an external delivery, signal, or initialization change has no causal tool result to annotate; and
• recovery and replay need the transition in durable canonical state, not only in an in-memory Pi message.

Provider mappings

One provider-neutral representation could be a durable chronological delta such as:

type ToolAvailabilityDelta = {
  added: ToolDefinition[];
  removed: string[];
  updated: ToolDefinition[];
  allowed?: string[];
};

Adapters could then choose the strongest available mapping:

Capability	Possible mapping
Anthropic models with mid-conversation tool changes	Stable definition catalog plus tool_addition / tool_removal blocks
Anthropic models with deferred references only	Pi's addedToolNames / defer_loading path for eligible additions; explicit fallback for removals and updates
OpenAI with a finite known catalog	Stable top-level definitions plus per-step allowed_tools
OpenAI with genuinely late-bound additions	Tool-search output or additional_tools, appended at a stable historical point
Provider without a suitable native mechanism	Stable dispatcher/meta-tool, or an explicitly documented cache-busting fallback

Anthropic's current mid-conversation tool changes are the closest native match for conditional workflow tools, but model support is limited. The docs' Sonnet 4.6 example is not among the supported models.

OpenAI explicitly recommends a stable tool list with allowed_tools when the goal is to vary availability while maximizing cache reuse.

For a provider-neutral precedent, OpenCode's active v2 branch keeps one invariant provider-visible execute({ code }) tool and moves the changing registry behind it:

• Stable dispatcher: codemode/tool.ts
• Catalog delta rendering: codemode/instructions.ts
• Per-request tool snapshot: session/context.ts

The important invariant is the same across these approaches: keep the provider-visible prefix stable, append availability changes at their chronological point, and enforce the live registry when a tool is executed.

Extra cases worth testing

• A → B and B → A
• zero tools → one tool and one tool → zero tools
• same-name description/schema updates
• remove → re-add
• changes caused by tool results and by external deliveries/signals
• persistence, crash recovery, replay, join/resume, and compaction
• raw serialized provider payloads plus cache-read/cache-write metrics where the provider exposes them