Better Prompt Caching #8277

Prompt caching affects latency, token cost, and local-model prefill time across long-running Qwen Code sessions. The work is currently spread across provider adapters, prompt construction, tool discovery, local KV-cache reuse, forks, and telemetry.

Goal: keep the reusable prompt prefix stable and observable so unchanged system instructions, tool schemas, and conversation history are not repeatedly reprocessed or rebilled.

This means:

• Semantically identical requests serialize cacheable prefixes deterministically.
• Dynamic state is appended after stable content instead of rewriting the front of the prompt.
• Side queries, deferred-tool discovery, forks, and resumed agents preserve cache reuse where their semantics allow it.
• Provider-specific cache controls, breakpoints, scopes, and retention settings work across supported endpoints.
• Local backends avoid unnecessary KV-cache invalidation and full re-prefill.
• Cache reads, writes, misses, and effective savings are visible in usage diagnostics.

How to Use This Tracker

This issue coordinates independently shippable work; it is not intended to become one large refactor.

• Own: take one scoped issue and deliver a focused PR with tests.
• Measure: include a before/after request-prefix comparison, provider usage counters, or local-backend prefill evidence.
• Triage: link new prompt-cache issues here and distinguish client-side prefix changes from provider-side cache behavior.
• Maintain: move completed work to the historical section and keep active PR status current.

Open Work

Stable prompt and tool prefixes

• Deferred-tools listing in the system prompt busts prompt cache on every MCP discovery / tool reveal #4777 — keep deferred-tool listings from rewriting the cached system prompt during MCP discovery and tool reveal.
• Keep deferred tool discovery from invalidating prompt cache prefixes #6721 — keep the provider-facing function declaration list stable after deferred-tool discovery.
  • Draft implementation for both issues: fix(core): preserve prompt cache across deferred tool discovery #8276.

Local backends and compaction

• feat: use llama.cpp slot state save/restore instead of text-based compression to eliminate re-prefill #5760 — investigate llama.cpp slot save/restore so compaction does not require a full re-prefill.

Forks and cache sharing

• Fork profiles: named tool-restriction presets for cache-sharing forks (also Swarm thinking / Self-reflection) #7625 — add fork profiles without changing inherited tool declarations, preserving cache-sharing semantics.

Adjacent prompt composition

• Split System Prompt & System Reminder to improve context management and avoid hallucination #2653 — separate long-lived system instructions from dynamic reminders; any design should preserve a stable cacheable prefix.
  • Cache-oriented groundwork has landed in perf(core): keep the volatile auto-memory section last in the system prompt #7651 (put the volatile auto-memory layer last), refactor(core): assemble the system prompt through one layered builder #7707 (centralize the explicit stable → context → volatile assembler), and perf(core): add early Anthropic cache breakpoint on the stable system prefix #7912 (use the stable boundary as an early Anthropic cache breakpoint). These partially address Split System Prompt & System Reminder to improve context management and avoid hallucination #2653's prompt-structure concerns; its broader system/reminder semantic separation remains open.

Completed and Historical Work

Prompt stability and tool discovery

• Tool call results with float-formatted integers cause tokenization instability with local backends #2667 — normalize tool-call number serialization for token-stable local-backend prompts.
• tool_search invalidates LLM server KV-cache on every deferred-tool load #6265 — reduce KV-cache invalidation when tool_search reveals deferred tools.
• Stabilize tool schema order to avoid unnecessary prompt cache misses #6338 — make tool schema ordering deterministic.
• tools.visible: selective deferred-tool visibility at startup #6368 — allow selected deferred tools to be visible at startup.
• Preload deferred MCP tools when their schemas fit within a context-window threshold #7920 — preload deferred MCP schemas when they fit a context-window threshold.

Provider cache behavior

• ## [Bug] Implicit cache not working on /apps/anthropic endpoint — $456 charged for 20hrs casual Claude Code usage (March 2026) #2823 — investigate missing implicit cache reuse on the Alibaba Anthropic endpoint.
• Prompt cache miss at user task boundary despite identical prefix #2986 — investigate cache misses at user-task boundaries despite identical prefixes.
• Anthropic provider: avoidable prompt-cache misses inflate cost — side-queries use a different prefix; conversation breakpoint sits on the moving last message #5942 — reuse the main prefix for side queries and improve Anthropic conversation breakpoints.
• Anthropic converter: low cache hit rate (~20%) through proxy providers (e.g. Routify) #6642 — improve Anthropic-compatible proxy cache-control behavior.
• Anthropic: support the 1-hour extended cache_control TTL #8047 — support Anthropic's one-hour cache retention tier.

Local prefill and cache reuse

• 发现qwen-code无法缓存prompt #153 — original report of poor prompt-cache reuse and repeated local-model prefill.
• more full prompt reprocessing in recent update? #5736 — investigate repeated full prompt reprocessing on local backends.
• System prompt fixed overhead reaches ~22k tokens for minimal input (0.2% signal-to-noise ratio) #6097 — reduce fixed system-prompt overhead, which lowers the cost of unavoidable cold writes and misses.

Forks, side queries, and resumed agents

• Title: feat(cli): add /btw slash command for ephemeral side questions #2370 — add cache-reusing ephemeral /btw side queries.
• [P0] Fork Subagent / Fork Subagent #3016 — establish fork-subagent shared-prefix caching.
• feat(cli): make /fork spawn a background fork agent (separate from /branch session copy) #4757 — expose a background fork command with prompt-cache parity.
• Enable the fork subagent by default and let default agents bubble permission prompts #4956 — enable cache-sharing fork agents by default.
• Keep completed background agents resident in-session instead of rebuilding them from the transcript #7410 — keep completed background-agent runtimes resident so continuations retain provider cache state.

Observability

• The cache calculation method does not take effect for valid models #1033 — read cached-token usage across provider response shapes.
• Token Plan未启用缓存 session cache #4444 — show cache information in model statistics.

Related Trackers

• Better Qwen Code #1047 — Better Qwen Code roadmap.
• Better context & tools #1050 — Better context & tools.
• Better Context & Memory #7021 — Better Context & Memory.

Scope Boundary

This tracker covers model prompt/prefix caching, provider prompt-cache controls, and local model KV/prefix reuse. It intentionally excludes unrelated caches such as file-search results, memory-recall storage, build artifacts, authentication state, and UI data caches.