Skip to content

Deferred-tools listing in the system prompt busts prompt cache on every MCP discovery / tool reveal #4777

Description

@qqqys

Summary

The "Deferred Tools" listing (MCP tools — always shouldDefer=true, reachable only via ToolSearch) is baked into the cached system prompt. Whenever the deferred set changes — MCP progressive discovery completing after startChat(), or the model revealing a tool via ToolSearch — setTools() / startChat() call setSystemInstruction() to rewrite the whole system instruction. That mutates the cached prefix, invalidating the prompt cache for the rest of the conversation.

Impact

  • Every late MCP discovery and every ToolSearch reveal drops the prompt-cache hit → higher latency + cost (cache_read → full input) for the remainder of the session.
  • Affects both Anthropic (explicit cache_control on the system block) and Gemini (implicit prefix caching).
  • Non-interactive --prompt runs are hit hardest (that's where progressive MCP discovery fires).

Root cause

getMainSessionSystemInstruction() embeds the deferred-tools section (getCoreSystemPrompt(..., deferredTools)), and setTools() rewrites the system instruction to refresh it on every registry change.

Proposed fix

Move the deferred-tools listing out of the cached prefix into a per-turn <system-reminder> in the message tail, rebuilt from live registry state each UserQuery/Cron turn. The system prefix then stays byte-stable across MCP discovery / tool reveals; deferred tools still surface, and remain visible across turns because the reminder persists in chat history.

Files involved:

  • packages/core/src/core/client.ts
  • packages/core/src/core/prompts.ts

A fix with tests is ready; happy to open a PR.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions