feat(ci): the token work has no recall or task-success gate — teach the existing benchmark to compare two configurations #12333

Part of #12028. This is the acceptance criterion that has no owner: every token change on this umbrella is measured for what it saves, and nothing measures what it costs in tool recall or task success. Until that exists, the largest saving available cannot responsibly be turned on.

What already exists

.github/workflows/dsw-swe-verified-release.yml — DSW Harbor Benchmark Release — is a workflow_dispatch that evaluates a published npm release against SWE-bench Verified (500 instances) and Terminal-Bench 2.0 (89 tasks), with inputs for release_tag / qwen_release_tag, per-suite limits, a single-instance selector, and execution_backend.

So task success on a standard suite is already harnessed. The gap is narrower than "build an eval harness".

What is missing

1. No settings overlay. The workflow installs a published release and runs it with default settings; nothing lets a run carry a settings.json. So the one question this umbrella needs answered — does task success drop when tools.eager withholds most tool schemas? — cannot be asked of it at all. An input that writes a settings file into the run's QWEN_HOME (and an A/B mode that runs the same manifest twice, with and without it) is the whole change.

2. No token or recall profile per run. The suite reports task outcomes. It does not report:

   • idle cost — input tokens of a session that asks one question and calls no tool;
   • total input tokens per task, which is the number that decides whether a saving is real or was merely moved into the conversation (take away grep_search and the model reaches for grep through the shell, whose output is never cached);
   • tool_search calls per session — the routing-miss proxy. The tracking(core): non-conversation context token governance #12028 analysis put the break-even at ≤2 reveals per session under today's reveal semantics; feat(core): preserve prompt cache for deferred tools #10410 changes that arithmetic, but the metric stays the right one;
   • bad-call rate — calls naming an undeclared tool, or failing parameter validation.

   None of this needs new instrumentation. /context reports the breakdown with an explicit residual since fix(cli): make /context categories add up to the provider total #12119, ToolCallEvent.function_name (packages/core/src/telemetry/types.ts) already carries per-call tool names, and tool-call errors are already events. It is aggregation and reporting.

3. No task set for the sessions this actually governs. SWE-bench and Terminal-Bench are coding suites. A deployment whose sessions are mostly data queries and file work will see its recall regressions somewhere those suites do not look. A stratified set of real-shaped tasks (pure Q&A, file work, shell, multi-step) plus a handful of safety cases belongs beside them — stated as a requirement here, not designed here.

Why it blocks, concretely

• The tools.eager allowlist is worth 21,461 → ~5,868 tokens in the tracking(core): non-conversation context token governance #12028 sample, with no code change — more than every merged PR on this umbrella combined. Nobody should enable it in production on the strength of "the prefix got smaller".
• A demoted tool the model never thinks to look for does not fail loudly. It stops being used, and the task quietly gets worse. That is invisible to unit tests by construction, and it is the exact failure mode feat(core): the eager tool surface is a hand-maintained static list — let something choose it, without invalidating the prompt prefix #12326 calls starvation.
• feat(core): preserve prompt cache for deferred tools #10410 makes deferral nearly free on the cost side, which removes the economic argument against deferring everything and leaves recall as the only remaining question — sharpening the need for this rather than softening it.

Acceptance

A single dispatched run answers, for two configurations of the same commit:

and the comparison is legible enough that "we saved 33% of the prefix and lost nothing" is either supported or refuted by it.

Not proposed here

• A new benchmark suite. The existing two plus a stratified real-task set is enough.
• New telemetry fields. If aggregation turns out to need one, that is a separate, small change with its own justification — note that fix(core): context-usage telemetry is dropped entirely when a non-function tool is present, and mixes two token estimators #12048 was closed NOT_PLANNED after per-tool telemetry was validated against real samples and judged not worth it, so do not reopen that route by default.