feat(managed-agent): Implement private Hosted MCP runtime (H1) - #12946

E2E and pre-commit verification

Verified on macOS with Node.js 22.22.2, JDK 21.0.11 and an isolated MariaDB 10.11.19 instance. The locally built Hosted CLI used the Java Broker and provisioned Runtime with real stdio and Streamable HTTP MCP servers; only the model was a deterministic fixture. Credentials and test records were synthetic.

• Three model requests completed with exactly two physical tool calls, one per transport. The SQL results_ready checkpoint was committed before the next model request.
• Binary resource responses retained their exact base64 bytes; the prompt retained both messages. There were exactly five physical MCP operations total: two tools, two resource reads and one prompt get.
• Duplicate operation IDs replayed; conflicting content was rejected without another effect. Dropped configuration and invocation responses recovered through the original IDs without resending the effect.
• After setting Workspace can_read=FALSE in SQL, the existing owner's acquire, MCP release/status and final Broker release all returned 200. The two connections closed on their original owner.
• After close and reload, the Broker proxy returned 503. Replaying the committed prompt result and closing again added zero Broker requests and zero MCP effects.
• All test process groups were confirmed exited after completion.

The first expanded run used a stale Broker dependency embedded in the Spring JAR and failed the revoked-access close case. After reinstalling the Broker dependency and a clean server package, all 81 embedded Broker classes matched the current compiled classes; the complete E2E above passed. Final CLI SHA-256: 604ef0a1f586b0aae9b6a96dc07e650e06093e452ebf3ea91da7797aefa17ca1. Final server JAR SHA-256: 3e967c89df9f2ca9e8a82029a4d62524fb435246f840d932c8d498ba94c75064.

Additional checks passed: root build/typecheck/bundle; 904 CLI + 346 Core + 157 Broker + 43 Java server tests; changed-file ESLint/Prettier; staged diff checks. Independent failure reproductions verified both dispatch/cancel race orderings, durable replay, real 25-second timeout followed by native cancellation and late settlement, discovery invalidation during listing, original-owner recovery and grant/effect identity isolation.

Repeated open-ended and targeted audits found defects that were fixed before commit. The final two consecutive passes found no remaining actionable issues in snapshot c35e3302c8cf685f99945f9a757b4c5dfe4b3a3be9cefe7cc0591a13b13d6341, based on origin/main at bd45b95f826b3873e25ae84d78ae35f1599c141a.

Windows/Linux and the full production restart matrix were not exercised locally. Full SQL E2E covers stdio and Streamable HTTP; SSE, replacement, quotas and cancellation have focused transport/contract coverage. Production profile enablement and SDK reverse clients remain outside this private H1 slice.

Real-stack verification — H1 Hosted MCP runtime @ e3ecba66

Merge reference in one paragraph. On a locally built real stack, everything this PR claims holds on both MySQL 8.4.11 and MariaDB 10.11.18. That includes SSE, which the PR covered only in unit tests, and every recovery claim in the author's E2E note, each re-run independently. The CI red leg I found at a390b5d5 (process-env guard) was fixed by the author during my run. The PR is not ready to land as it stands, for three reasons:

1. It is now CONFLICTING with main. feat(serve): add gated Hosted foreground Shell turns #12848 landed and collides in 8 serve/ files; the 6 production files among them hold 17 hunks.
2. It collides with open feat(managed-agent): Add durable remote Shell result delivery #12894 on Flyway V19. Git does not flag this; the merged server fails to start.
3. Five behaviour findings (F1–F5). Ordinary events leave a Session stuck or its MCP unusable, and through F1 they also lock the whole Workspace. The events are: a user cancel, a tool that runs longer than 25 s, an MCP server outage or restart, and a list_changed notification.

Because the profile is private and opt-in, maintainers could accept F1–F5 as documented H1 limits with follow-up issues. I would not let F3–F5 reach any enablement.

What was run

• Build under test. The a390b5d5 bundle (dist/cli.js) plus the Spring jar with the embedded Broker. During the run the head moved to e3ecba66, which adds only 16 lines to process-env-guard.test.ts. The diff of all other paths is empty, so every runtime result below applies to e3ecba66.
• Stack. Packaged Hosted Harness (qwen serve --profile hosted-harness) → embedded Java Broker (session store on) → provisioned Runtime worker with a deployment manifest in QWEN_MANAGED_MCP_CONFIG → real MCP servers built on the repo's @modelcontextprotocol/sdk 1.30.0:
  • stdio, as a child process of the worker;
  • Streamable HTTP, with a Bearer header;
  • SSE, with a Bearer header.
• Instrumentation. Only the model is a fixture. Broker traffic goes through a counting and fault-injecting proxy. Every physical MCP request is appended to a JSONL ledger. The test-only piece is a small Java main that seeds Workspaces and supplies the catalog actor.
• Environment. MySQL 8.4.11 and MariaDB 10.11.18 (Docker), Node 22.23.2, JDK 25 (release 21).
• Scope. 11 scenarios, 24 mutants (16 TS + 8 Java), the focused TS/Java suites, and a trial merge with feat(managed-agent): Add durable remote Shell result delivery #12894.

✅ Confirmed on the real stack

• One tool round across all three transports. It produced exactly 3 physical effects. The checkpoint phase observed at the second model request was results_ready.
• Resource and prompt operations.
  • A binary resource_read came back byte-exact (300 bytes).
  • prompt_get over SSE kept both messages.
  • Repeating the same operationId returned the identical response.
  • Reusing the ID with a different URI or server was refused, with 0 extra effects.
• Catalogs. Neither the private nor the public catalog contains the token, Bearer, URL, command or env; I checked 10 needles and got 0 hits. MCP records have task_kind NULL, and the public /tasks list stays empty.
• Detach and load. Detach releases all 3 connections and the Broker tool-session. Load creates a new owner mcp:<sid>:<digest>, and replaying a committed prompt result is identical.
• Author's recovery claims, all reproduced (the triage bot listed these as "not verified"):
  • Dropped replies. Dropping both the configure reply and the invoke reply recovers through mcp-status on the original IDs, with exactly 1 physical read.
  • Revoked access. With can_read=FALSE, close returns 204 and the lease is released; new work is refused (Broker acquire 409).
  • Broker down. After a reload, with the Broker returning 503 for everything, the committed replay returns 202, close returns 204, with 0 Broker requests and 0 effects.
  • Stale catalogs. A stale catalog never authorised a call: every refused call in F2 below has 0 effects.
• Replacement during an in-flight call. Replacing r1 → r2 while a 6 s call ran on r1 returned 202. The call finished on r1, and the next turn advertised and used only r2.
• Limits. A 65-tool server gets tools: partial with 33 tools advertised (16 KiB cap). A 70 KB result becomes managed_mcp_output_limit, and the Session stays healthy.

Findings

F1 — An attached MCP Session holds the Workspace lease between turns. HostedWorkspaceToolTurn skips broker.release() for the MCP profile, and HostedMcpSession releases only in close(). I put three Sessions in one Workspace; the outcome is the same on both databases:

• B (MCP profile), while A sits idle but attached: POST /prompt → 503 hosted_prompt_admission_failed (the Broker acquire returned 409).
• C (files profile): turn_error, and nothing is written.
• After A detaches, B works, and then B blocks C in the same way.

So one attached MCP Session makes its Workspace unusable for tools in every other Session. Combined with F3–F5, one stuck Session locks the Workspace indefinitely.

F3/F4 — Cancel, or a call longer than 25 s, leaves the Session and its Workspace stuck. Per the MCP spec, the SDK server drops the response to a request it was told to cancel. I checked the success and error paths in shared/protocol.js. So after notifications/cancelled no settlement ever comes back.

The late reply in the 32 s case does settle the Runtime's own view. But qwen_tool_execution.execution_state stays UNKNOWN and is never re-queried, so the Session never heals. Close runs mcp-release 200, then tool-sessions/mcp:<sid>:release returns 409, and detach returns 503. For comparison, ordinary MCP uses MCP_DEFAULT_TIMEOUT_MSEC = 10 min, configurable per server.

F2 / F5 / F6 / F7 — connection lifecycle after the first turn.

• F2. A list_changed notification for resources or tools bumps the single catalogRevision that every call kind is checked against, and the Harness never dispatches mcp-discover. From then on:

  • the next tool call in the same turn gets catalog_conflict;
  • so does every later turn;
  • so does prompt_get;
  • and the Harness catalog still says revision 1, complete.

  An SSE server restart has the same effect, because onerror retires the connection. The SDK's McpServer emits list_changed whenever a tool is registered, enabled or disabled after connect, so this trigger is not exotic.

• F5. Detach returns 503 within 1 s: 678 ms on MySQL, 228 ms on MariaDB. After the server comes back, detach returns 503 again. The configuration stays releasing and the lease stays held. terminateSession() throws when the DELETE cannot be sent, although that connection had 0 pending operations.

• F6. Each POST /mcp/configurations left the previous stdio process running: 1 → 16 live processes for one server in one Session. The 17th reconfiguration got 409. Only detach reaped them, and that took 6.9 s. This is stage-2 point 4 of the triage bot's review, now measured. Together with F2, the documented recovery path works about 15 times per Session.

• F7. After a refused first install, status reports recoveryBlocked:true while /prompt still admits. Detach stays 503 even after the Workspace frees up. It succeeds only after a later prompt dispatches the leftover configuration intent.

Possible directions (the choice is the maintainers'):

• F1: either stop holding the storage lease between turns for MCP owners, or document "one attached MCP Session per Workspace" as an H1 limit.
• F2: dispatch the existing mcp-discover on catalog_conflict / stale, and reconnect with a new generation after a transport error.
• F3: for resource_read / prompt_get, settle a cancelled operation as a non-blocking cancelled, since its result is data only.
• F4:
  • make the timeout a manifest field;
  • either don't send notifications/cancelled for tool calls, or bound the wait and settle them;
  • re-query the execution status on close or load, so a late success is reconciled.
• F5: with 0 pending operations, treat a failed or 405 DELETE as released.
• F6: close a retired connection once its pending set drains.
• F7: have close() locally settle configuration intents that are still at execution: intent, as not_started_proven, the same way operations are handled.

Gates

• CI.
  • At a390b5d5, Test (ubuntu) was red: process-env-guard.test.ts flagged QWEN_MANAGED_MCP_CONFIG, PATH and SystemRoot.
  • The author fixed it in e3ecba66, and it is now green: 22 pass, 2 pending.
  • Locally the guard passes 4 of 5 runs; the one failure was at a load average near 100.
• Main. The PR is CONFLICTING after feat(serve): add gated Hosted foreground Shell turns #12848 landed: 8 files, 17 hunks in the 6 production files (8 of them in hosted-workspace-tool-turn.ts). That file is where F1 and F4 live, so the post-merge head needs a re-run of the relevant scenarios.
• Merge order with feat(managed-agent): Add durable remote Shell result delivery #12894 (a7c0a391).
  • Both PRs add V19 (V19__managed_mcp_records.sql and V19__managed_tool_publication.sql), and git reports no conflict on those files.
  • The trial-merged jar on MySQL 8.4 fails at startup with Found more than one migration with version 19.
  • The trial merge also has 8 textual conflicts.
  • Whichever PR lands second has to renumber its migration.
• Local suites at a390b5d5.
  • cli focused: 79/79. core focused: 230/230. managed-agent-server: 180/180. runtime-broker: 395/396.
  • The one Broker failure, HttpRuntimeTransportTest#closesTheConnectionOnTheDeadlineAndOnCallerCancel, also fails on main bd45b95f on this host under load, and passes 5/5 alone.
• Mutation. Java: 8/8 killed. TS: 13/16 killed. Three survivors point at untested behaviour:
  • M7: a -32601 from a tools-only server should mean an empty complete list.
  • M11: the prompt route's hasPendingOperations() check. This guard returned the 409 in F3, yet no unit test pins it.
  • M13: configurations owned by two Runtime Sessions.

Smaller notes

• Identity conflicts on /mcp/operations and /mcp/configurations return 503, which looks retryable. A 409 would fit better.
• Model-facing tool names are mcp_<32 hex>. They include the catalog revision and connection generation, so every name changes on each load. After a reload, the transcript refers to tools that are no longer advertised.
• The Runtime's error receipts reach the model as bare codes, e.g. managed_mcp_catalog_conflict.
• stdio servers run with HOME set to the Workspace directory (observed: HOME = cwd = <ws>/child). Anything a server caches under ~ therefore lands in the user's Workspace.

Not verified

• Windows and Linux.
• A real model; like the author, I used a fixture model.
• Whether a restarted Harness can recover a Session stuck by F3–F5.
• The 32 in-flight quota.
• Interaction with feat(serve): implement generic Broker provider controls #12868, which the triage bot flagged.
• The post-merge head, after resolving feat(serve): add gated Hosted foreground Shell turns #12848.

Harness, scenarios, mutants and raw results: wenshao/qwen-code@a5ef4914/pr12946

Merge validation — 7a32435

Merged main at a765229c0a and resolved the overlapping Hosted Shell/MCP changes. The shared tool turn retains both profiles, asynchronous pinned declarations, the Shell input digest and the MCP prompt/owner distinction. Two independent open-ended and targeted audit passes found no remaining issue in the resolved paths.

Local checks passed: full build/typecheck/bundle, formatting/lint, 951 CLI + 104 Core + 160 Broker + 48 Java server tests. All 81 Broker classes embedded in the clean-packaged server match the current compilation.

Both independent E2E paths passed against the final artifacts:

• MCP: real Hosted → Java Broker → Runtime → stdio/Streamable HTTP with isolated MariaDB. Exactly five physical operations and three model requests; committed result ordering, original-ID lost-ACK recovery, revoked-access close and offline durable replay all passed. Offline replay added zero Broker requests or MCP effects.
• Foreground Shell: the upstream packaged-Harness integration test passed with six workspaces, complete 100 MiB output, reload, lost start reply, publication faults, cancellation and at-most-once checks. All six producers exited before a fresh reader verified the retained output. This test uses H2 in MySQL mode.

The model was deterministic in both paths; Runtime/MCP/Shell processes were real. All test process groups were confirmed exited. Local execution was on macOS. GitHub reports the PR as mergeable without conflicts; fresh CI is running.

Real-stack verification, round 2 — 7a324350 (main merged)

Follow-up to round 1. 7a324350 merges main a765229c into e3ecba66, bringing in #12848 (Hosted Shell turns), #12942 and #12918. It contains no other change.

Merge reference in one paragraph. The conflict resolution is faithful to both sides, and GitHub CI is green at this head. I rebuilt the bundle and jar from 7a324350 and re-ran every round-1 scenario on the real stack. Each outcome is identical: the author's claims still hold, and F1–F7 all reproduce unchanged, because the merge touches none of that logic. The V19 clash with open #12894 (a7c0a391) is also unchanged. My round-1 recommendation therefore stands. The conflict blocker is gone. What remains is the #12894 merge order, plus a maintainer decision on F1–F5: fix them here, or record them as H1 limits with follow-ups.

Same scenarios, new head

• Rebuild and scope. Fresh dist/cli.js and Spring jar from 7a324350; MySQL 8.4.11; the same rig, manifest and MCP servers (stdio, Streamable HTTP, SSE) as round 1. I compared all 11 scenarios one by one.
• Timings, from the MCP server ledger.
  • The cancelled resource_read was aborted 1.4 s after it started.
  • The tool was aborted 1.1 s after the prompt cancel.
  • The 32 s tool finished at +32.0 s, 7 s after the Runtime's 25 s timeout.
• New measurement. After the prompt cancel (F4), detach made 190 Broker calls before its 503: 95 acquire, 94 mcp-status and 1 mcp-release. This puts a number on triage stage-2 point 3: lookup() re-acquires the owner on every 100 ms poll.
• The stuck Workspaces stay stuck. The lease snapshot taken minutes later in S2 still shows the three Sessions stuck by F3/F4 holding their Workspaces.

Merge-resolution audit and gates

• Hand-resolved parts (git show --remerge-diff: 9 files, +76/−133):
  • toolProfile accepts files, shell or mcp, one per Session, so Shell and MCP never meet in one turn.
  • Declarations are (shell ? Shell tools : file tools) plus the MCP tools.
  • prepare(runtimeCallId, digest, shellInputDigest, promptId) keeps both sides' arguments.
  • Shell's input, digest and timeout stay on the Shell branch.
  • The worker keeps both the remote Shell publishers and the MCP runtime.
  • if (!this.mcp) await this.broker.release() is unchanged, so the F1 root cause is too.
• GitHub CI at 7a324350: 22 pass. That includes Hosted process fault gates / MySQL 8.4 (HostedWorkspaceToolTurnIT 5/5 in 96.8 s, with the six-Workspace 100 MiB Shell path and FG6a/b/d/e) and the MariaDB job. The author's note says the Shell IT ran on H2 locally; CI runs it on MySQL.
• Local, same command as CI (-Phosted-harness-mysql verify checkstyle:check), at a load average of 50–100:
  • unit 182/182, HostedHarnessMySqlIT 2/2, 0 Checkstyle violations;
  • HostedWorkspaceToolTurnIT 3/5 in 439 s. The failures were FG6B writers:renew → 409 (writer lease lapsed) and a 130 s driver timeout on the six-Workspace path. Both are timing failures in drivers this PR does not change, and the class passes 5/5 on CI in 96.8 s. Re-running just those two alone at a load of 31–54 passed 2/2 in 203.7 s.
• Mutation on the merged tree (6 cli test files, 112 tests):
  • Three new resolution mutants are killed:
    • Shell profile advertises only file tools (15 tests fail);
    • prepare drops promptId (5 fail);
    • prepare drops the Shell input digest (1 fails).
  • The promptId mutant survives on the a390b5d5 tree, so triage stage-2 point 7 is now pinned.
  • M13 (two-owner configurations) is now killed; it survived in round 1.
  • M7 (-32601 → empty complete list) and M11 (the prompt route's hasPendingOperations() guard, which is what returns the 409 in F3) still survive.

Unchanged from round 1

• F1–F7: as described in round 1; every one reproduces with the same status codes.
• Merge order: feat(managed-agent): Add durable remote Shell result delivery #12894 still adds V19__managed_tool_publication.sql next to this PR's V19__managed_mcp_records.sql. Whichever lands second has to renumber.
• Not verified: Windows/Linux, a real model, a restarted Harness recovering an F3–F5 Session, feat(serve): implement generic Broker provider controls #12868 interplay. MariaDB was not re-run this round; CI's MariaDB job is green and the merge changes nothing database-specific in the MCP path.

Evidence: wenshao/qwen-code@9e3c79fe/pr12946/r2 (raw results per scenario, remerge diff, CI and local gate extracts, mutant scripts).

Review follow-up and merge verification — 9b4886793a

Addresses the triage review and both real-stack reports, round 2. Merged main at 693528b004; the D5 Turn list/detail contract and MCP catalog contract are both retained, with OpenAPI version 1.21. The unreleased MCP migration is now V21, avoiding the V19/V20 files on #12894. Deployment ordering must still be checked when either branch lands.

Finding disposition

Two further audit findings were fixed before commit: an idle Broker restart must reacquire and validate the original owner before new work/close; an idle Harness restart must advance durable configuration record revisions before issuing new-writer grants. The Runtime gate continues to reject owner replacement at the same revision. Explicit replacement also works after the first definition fails, without retrying that failed definition first.

Other review points

• Status polling now reuses the validated owner and backs off from 100 ms to one second. A failed status query reacquires only the original owner and retries status, never an effect. New work and close still validate ownership at their boundary.
• Idempotent shared Broker acquire and the additional binding/generation fields are intentional. Existing file/Shell reservation identities and Shell input digest remain covered by explicit assertions and the shared Shell integration regression.
• Record-domain recognition is global; execution admission remains private-profile and scoped-manifest gated. This distinction is now documented.
• Acquire failures preserve uncertainty on lost replies; definite Broker rejections restore the previous local acquired flag. Grants require a successfully validated Runtime binding.
• Raw operation ID conflicts now return HTTP 409 without another effect. M7 (method-not-found means authoritative empty discovery) and M11 (pending raw work blocks model prompt admission) have direct regression assertions; the existing M13 two-owner guard remains covered.
• Deferred explicitly: process-lifetime operation receipts / closed-generation tombstones and the release waiter for permanently unknown work remain unbounded history. The concurrent connection/request quotas do not claim to bound this memory. Durable-acknowledged pruning and bounded waiter cleanup are follow-ups before production enablement, recorded here and in both design languages. Tool names intentionally include immutable catalog/connection identity, and stdio HOME remains the Workspace unless a deployment overrides it.
• feat(serve): implement generic Broker provider controls #12868 is still an open semantic integration dependency. The later merge must preserve original-owner status/cancel/release after revocation and drain-before-storage-release ordering while adopting the generic control mechanism; this PR does not claim that cross-branch combination has been verified.

Verification on the final tree

• Build/static checks: repository build, final CLI rebuild/bundle, repository typecheck, changed-file ESLint/Prettier, and both Java modules’ Checkstyle passed. The pre-commit hook also passed without changing the audited tree.
• Focused tests: 961 CLI (121 directly affected + 840 shared consumers), 59 Core, 160 Broker and 54 Server tests passed: 1,234 tests, zero failures. The new grant test also failed under a deliberately disabled-renewal mutation, then passed after restoring the implementation.
• Real stack: macOS, Node 22.22.2, JDK 21.0.11, isolated MariaDB 10.11.19, packaged Hosted Harness → Java Broker/SQL store → provisioned Runtime → local stdio/Streamable HTTP wire fixtures; only the model is deterministic. All 11 formal lifecycle cases passed, with F1 checked as a retained limit:

• The original five-effect MCP happy/recovery scenario also passed: byte-preserving resource/prompt results, lost configure/invoke ACK recovery by original ID, revoked-access close, and offline durable replay with 0 Broker requests / 0 new effects.
• The upstream six-Workspace, 100 MiB Shell packaged integration passed against H2, preserving the shared Broker/Shell path. This is separate from the MariaDB MCP scenarios; no MySQL run is claimed for this revision.
• Every recorded lifecycle test process group and native child was confirmed gone. One preliminary test-driver assertion incorrectly included an unrelated file-profile control owner; it was narrowed to the original MCP owner and the entire case rerun successfully.

Artifacts exercised: CLI SHA-256 7db44d0d703e3c45f7d04148834ecdbbe6e9f5b529b4249b6e819d4355a92746; packaged Server SHA-256 7586d6c4a502f4fb8ee573ae6f139e4ad17fe6252da71107ef38216bb1b6e188. The 81 embedded Broker classes match the compiled Broker byte-for-byte. Windows/Linux, a real model, full restart matrix, and cross-branch #12868 integration were not verified locally; SSE retains focused transport coverage. New GitHub CI is tracked separately from these local results.

The prior finding-bearing audit rounds reset the clean count. After the last fixes, two consecutive open-ended and targeted passes were clean, including independent CLI and Core/Java review. These are verification results for this private H1 scope, not an approval or production-enablement claim.

中文摘要：已同时修复冲突和评论确认的问题，并补上审计发现的 Broker/Harness 空闲重启及首次失败配置替换问题。F1 存储租约独占、观察窗口外/turn 中途重启恢复、历史回执与永久等待者的回收仍为明确的 H1 后续项；未宣称这些限制已修复，也未启用生产 profile。

Real-stack verification, round 3 — 9b488679 (F1–F7 dispositions)

Follow-up to round 1 and round 2, checking the author's disposition table.

Merge reference in one paragraph. Every disposition in that table reproduces on the real stack on MySQL 8.4.11, including SSE, which the author covered only in unit tests: F2–F7 are fixed, F1 behaves exactly as now documented, identity conflicts return 409, and the round-1 mutant survivors M7/M11 are now pinned. The V21 rename removes the duplicate-version crash with #12894. It is not mergeable yet, because of one blocker: CI's Hosted process fault gates / MySQL 8.4 job is red. I reproduced it locally and traced it, by a one-line A/B test, to this PR's change in RuntimeBrokerHttpServer.observe(). There are also three behaviour notes (N1–N3) that widen the documented H1 limits, a Flyway ordering rule for #12894, and four surviving mutants.

F1–F7, re-run

Fresh bundle and jar from 9b488679; same rig, manifest and MCP servers as before, plus a timeoutMs: 5000 definition.

Still holding from earlier rounds:

• three transports with results_ready ordering, byte-exact blobs and 0 catalog leaks;
• lost configure and invoke replies → exactly 1 read;
• revoke then close → 204;
• Broker down → replay 202, close 204, 0 Broker requests, 0 effects;
• replacement during an in-flight call;
• the 16 KiB / 60 KiB limits.

B1 — the CI red is caused by one Broker line

Failing job. CI Hosted process fault gates / MySQL 8.4 fails in HostedProcessCrashIT.processCrashesNeverReplayHostedToolsOnMySql (FG6C). CI stops at the first failure, so I ran each case alone on the PR tree:

Cause. The PR removes && Integer.valueOf(3).equals(record.getReference().get("runtimeProtocol")) from observe(). As a result, status reads of UNKNOWN v2 (file) and MCP executions now go through reconcileExecution, which has to reach the original worker. That changes the Broker's status contract for every Hosted tool profile, not only MCP.

Options (the maintainers' call). Either:

• keep observe() protocol-3-only and reconcile MCP executions on their own path; or
• update the FG6C expectations for v2, with the reason stated in the PR.

Why the author missed it. The author's note says the Shell/file IT ran on H2 only and no MySQL run was claimed for this revision. HostedProcessCrashIT is the gate that catches this change.

Behaviour notes the fixes leave open

• N1 — a reply that can never arrive still ends in a stuck Session, now after 10.5 minutes.
  • Connection lost: I restarted the SSE server 2 s into a 20 s call. The execution was UNKNOWN at 6.8 s, the turn stayed active until 633 s, then the Session was recoveryBlocked with no terminal event.
  • Tool never answers: on a timeoutMs: 5000 server, the execution was UNKNOWN at 7.8 s and the turn stayed active until 634 s.
  • Both cases end the same way: next prompt 409, detach 503, lease held (so, by F1, the Workspace is locked), qwen_tool_execution UNKNOWN.
  • The configured timeoutMs does not shorten the Hosted wait, and a lost connection is known at once but still waits out the 630 s window. The author documents "results after that window" as follow-up work; these two cases show the window itself is spent even when the outcome is already decided.
• N2 — one pinned server down makes every turn fail. Because the Harness now refreshes before each model request, a single unavailable pinned server (SSE or Streamable HTTP) makes every turn in the Session end in turn_error, with 0 effects. That includes a text-only answer and a turn that only calls the healthy stdio server. Each failed turn records a failed mcp_configuration revision, and the Session recovers once the server returns. One option is to drop that server's tools for the round instead of failing the turn.
• N3 — cancellation is deferred. The prompt cancel above took 9.1 s to end the turn because the tool ran to completion; with a 10-minute tool, the user waits up to 10 minutes and the effect still happens. This is documented by the author, and I'm listing it here only with the measured number.

Migrations, mutation, cost, CI

• Flyway ordering. Tested with a trial merge with feat(managed-agent): Add durable remote Shell result delivery #12894 at 033c74d3, which carries V19 and V20.
  • On a fresh database, the merged jar applies 19 → 20 → 21 and starts.
  • If this PR runs first (database at 21), the merged jar then refuses to start with Detected resolved migration not applied to database: 19 … set -outOfOrder=true. outOfOrder is not enabled.
  • So feat(managed-agent): Add durable remote Shell result delivery #12894 should land first, or whichever lands second should renumber above the other.
• Mutation (6 cli test files, 122 tests):
  • M7 and M11 are now killed, and so are the inverses of the fixes: native cancel, idle sibling close, bounded DELETE, wait-through-unknown, intent cancel on close, unchanged-catalog skip, and grant renewal.
  • Survivors:
    • X7, no refresh before each model request. This is the F2 fix itself; it is proven end to end above but not by a unit test.
    • X3, a busy retired connection is not closed after its last settle.
    • X5, the 600 s default timeout falls back to 25 s.
    • X9, identity conflicts return 503 again.
• Refresh cost. With text-only turns at a load average of about 25–30: 1 mcp-discover per server per model request. Time to the first model request was 103–185 ms with 1 server and 154–232 ms with 6 servers.
• CI at 9b488679: 21 pass and 1 fail (B1). HostedWorkspaceToolTurnIT is 5/5 on MySQL in that same job (115.5 s, including the six-Workspace 100 MiB Shell path). Test, Lint, MariaDB and the Java matrix are green.

Not verified

• Windows/Linux.
• A real model.
• MariaDB this round (CI's MariaDB job is green).
• A Harness restart during an unfinished turn.
• The feat(serve): implement generic Broker provider controls #12868 combination.

Evidence: wenshao/qwen-code@9fd4f16f/pr12946/r3 (per-scenario raw results, S12 timelines, FG6C A/B extracts, Flyway logs, remerge diff, mutants, scripts).

B1 is fixed in 7444dc9. The shared v2 status contract is preserved: default reads no longer contact an UNKNOWN execution's worker. MCP polling explicitly opts into original-execution reconciliation with GET ...?reconcile=true; v3 automatic reconciliation remains unchanged. Start/cancel paths, ownership/generation checks and the original FG6C assertions are unchanged. No dispatch retry or replacement owner was added.

Verification before pushing:

• Reproduced both worker-kill and worker-stop on the previous artifacts. On the rebuilt artifacts, all six original FG6C process scenarios pass against isolated MariaDB 10.11.19: harness-prepare, harness-start, harness-result, spring-kill, worker-kill and worker-stop. Both Java and TypeScript gate-source hashes match the failing baseline.
• The separate real-MCP late-result case passes: the same SQL execution transitions UNKNOWN → SETTLED/success through explicit reconciliation, with one Broker start and one native invocation. The next prompt and detach succeed; all test process groups and injected workers exited.
• 52 focused CLI tests and 161 Broker tests pass, along with build, bundle, typecheck, lint/format and Java Checkstyle. New regression tests were red before the fix.
• Two consecutive open-ended and targeted audit passes completed without new findings, including independent reviewers. The committed tree matches the audited tree.

These local process tests use macOS/MariaDB and a deterministic model fixture. The separate Linux/MySQL 8.4 CI gate now passes on 7444dc9: all six original FG6C success markers are present, Hosted Workspace tool turns pass 5/5, Hosted Harness MySQL checks pass 2/2, and 29 additional Runtime fault-gate tests pass. The Linux unit suite, lint/static checks, Serve A/B, no-AK integration, Java matrix, MariaDB integration, real daemon E2E and TUI checks also pass. The downstream Web Shell browser smoke gate also passes. Final check snapshot: 23 passed, 8 skipped by workflow policy, no failures; only the separate automatic code review remains in progress.

Round-3 follow-ups: N1's full 630-second wait and N2's all-pinned-server availability requirement are now explicit in both design languages and the PR body. N3's deferred cancellation remains documented. Flyway guidance now states that if V21 has already run, pending publication migrations must be renumbered above it, without enabling out-of-order execution. N1–N3 behavior changes and the non-blocking X7/X3/X5/X9 unit-test gaps are deferred to follow-up work; this CI fix does not expand the established H1 scope.

Real-stack verification, round 4 — 7444dc97 (B1 fix)

Follow-up to round 3 and the author's B1 note.

Merge reference in one paragraph. B1 is fixed. The fix leaves the FG6C gate sources untouched (no diff since 9b488679), and the full FG6C class passes locally on MySQL 8.4. The MCP late-result path still works through the new ?reconcile=true opt-in, and all four mutants of the fix are killed. CI is green: 23 pass, only review-pr pending. The round-3 behaviour notes N1–N3 and the Flyway rule are now stated in both design languages; the behaviour itself is unchanged, as the author said. From my side there is no remaining correctness blocker. What is left is a maintainer decision: accept the documented H1 limits (F1, N1–N3), leave the four unit-test gaps (X3/X5/X7/X9) for follow-up, and settle the merge order with #12894 (still open with V19/V20; main is at V18).

• Build. Fresh bundle and Spring jar from 7444dc97. The Broker class embedded in the server jar is byte-identical to the compiled one.
• B1 / FG6C. Full HostedProcessCrashIT on local MySQL 8.4: harness-prepare, harness-start, harness-result, spring-kill, worker-kill and worker-stop give 6 × HOSTED_PROCESS_CRASH_OK, 1/1 in 119.4 s. The two cases that failed in round 3 now pass without any change to their expectations.
• MCP paths after the change.
  • A 32 s tool on a timeoutMs: 5000 server goes UNKNOWN, then the late result is accepted through ?reconcile=true: end_turn at 37.0 s, and the next prompt and detach succeed. This is the path that needs the opt-in.
  • A 32 s tool on the default timeout ends at 37.0 s.
  • A prompt cancel at 5.5 s ends the turn at 15.0 s (deferred cancellation, as documented).
  • S1 regression passes: three transports, results_ready ordering, exact blob, both prompt messages, 409 conflicts, 0 catalog leaks, detach and load.
• New probe — Runtime worker SIGKILLed during an MCP tool call. The reconcile status read answers 409 once, and the turn ends 2 s after the kill: recoveryBlocked, next prompt 409, detach 503, lease held, qwen_tool_execution UNKNOWN. So worker loss fails fast into the documented blocked state, the same outcome FG6C expects for file tools. That contrasts with N1 (a lost MCP server connection), which still waits out the 630 s window.
• Mutants of the fix. All four are killed:
  • Y1, the MCP opt-in is ignored: 2 Broker tests fail.
  • Y2, every UNKNOWN read reconciles again (B1 comes back): 2 fail.
  • Y3, Hosted polls without ?reconcile=true: hosted-workspace-broker.test.ts fails.
  • Y4, the reconcile value is not validated: 1 fails.
• Still open, all documented by the author.
  • F1: one attached MCP owner per storage lease.
  • N1: a lost connection or a server that never replies means a 630 s wait, then a blocked Session.
  • N2: every pinned server must refresh before a model request.
  • N3: cancellation is deferred.
  • Flyway: if V21 has already run, feat(managed-agent): Add durable remote Shell result delivery #12894's migrations must be renumbered above it.
  • Unit-test gaps: X3, X5, X7 (the F2 refresh wiring) and X9.
• Not verified. Windows/Linux, a real model, MariaDB this round (CI's MariaDB job is green), a Harness restart mid-turn, and feat(serve): implement generic Broker provider controls #12868.

Evidence: wenshao/qwen-code@17e44727/pr12946/r4.

@qwen-code /triage

Addressed Finding 1 in commit 2159a735ee, from the sandbox verification.

New Hosted Sessions now reject more than 16 server pins with HTTP 400 invalid_hosted_mcp_servers, before creating the Session or contacting the Broker. Admission and Runtime allocation use the same cap. The original 32-pin format remains accepted when loading an already saved Session, with exact definition matching, so older records can still attach and detach for cleanup.

Runtime connection quota failures now return HTTP 409 managed_mcp_connection_quota from prompt admission, configuration replacement and raw-operation initialization. Other errors retain their existing handling. A failure discovered inside an already accepted prompt remains a turn error; no attempt is made to change an HTTP 202 response after admission.

Verification:

• Before the fix, the real Hosted → Java Broker → Runtime → stdio chain accepted 17 pins and failed both subsequent prompts with the generic 503. After the fix, the same creation returns 400 with zero Broker requests, Runtime Sessions, native processes or model calls.
• The real 16-pin happy path still completes a model-driven native tool call. At full capacity, explicit configuration and resource requests now both return 409 managed_mcp_connection_quota. A warmed prompt still returns 202 and reports the later quota failure within its turn, without another model/native call. Detach returns 204, the original Runtime Session is RELEASED and the storage lease is empty. All test processes exited.
• Eight new regression cases cover 16/17/32 creation, loading and detaching saved 17/32-pin records, all three error entry points and prompt retry after capacity becomes available. The initial regression run failed in the expected five places; all 130 focused tests now pass.
• Repository build, bundle and typecheck, focused ESLint and formatting pass. Two consecutive clean open-ended and targeted audits followed a correction for legacy Session cleanup.

The real-stack checks use macOS, MariaDB 10.11.19, real Java/Runtime/MCP processes and a deterministic model fixture. No new Java implementation or migration was introduced. CI for the new commit is reported separately.

One evidence correction: two prompts against the original 17-pin Session reused the same 16 native PIDs and produced only 16 initialization events total. We reproduced repeated quota failure, but not the report's inferred repeated 16-process spawn cascade. Finding 2's diagnostic-code distinction remains deferred: both malformed pins and wrong-profile input are already rejected, and this patch stays focused on the confirmed capacity defect.

R5 Suggestion disposition: this PR has exceeded five review rounds, so the repository’s late-review scope rule applies. The confirmed correctness fixes above are included; the following 26 Suggestions are explicitly deferred for follow-up, not claimed fixed or silently resolved. Their original threads remain open and retain the detailed acceptance criteria. R5-17 is covered by the approval integration regression, and the previously deferred R4-8 is now fixed together with R5-5.

@qwen-code /triage

Merged latest origin/main 3b18cfe5e4ab7ea72f1a92736186dacf753bf727 in f2618e3, resolving the conflicts introduced by #12894 and retaining #13098's queued approval expiry guard.

The main Shell publication/recovery flow and the existing MCP flow both remain wired through the refactored Hosted turn entry point. MCP retains its shared owner, per-prompt preparation identity, cancellation/reconciliation behavior and post-approval grant renewal; remote Shell uses its publication/v3 path. Both sides' tests are retained, and the three direct MCP constructor fixtures were adapted to the combined signature.

The migration dependency is now in this branch: main includes publication V20–V22, followed by MCP V23. Both design languages and the PR description reflect that #12894 has landed. Migrations remain ordered; this does not authorize applying V20–V22 retroactively to a database where standalone V23 was already deployed.

Validation for the merge:

• Build, typecheck, bundle, lint, formatting and diff checks passed.
• CLI: 1077 tests across 10 files; Core: 85 tests across 3 files; Managed Agent: 56 tests including H2/Flyway migrations through V23 and publication/MCP/Workspace authorization contracts.
• Java Broker: 538 cases, zero failures/errors. The two initially skipped interop cases were rerun after supplying the built artifacts and both passed without skips.
• Independent source verification: 9/9 probes passed, covering refresh cancellation windows, Unicode admission, delayed-approval grant renewal, and two actual Hosted HTTP prompt turns through the refactored execution helper. The two turns retain the same MCP owner, use distinct turn IDs, make no v3 calls, and release the owner only at detach. Model/Broker transport and approval time are controlled; this is not an OSS or deployed Java/SQL E2E claim.
• Two consecutive clean audits, open-ended and directed, completed before commit. The committed tree cea90f9c809f3a53773d282e81501fb80e835a78 matches the audited tree; hooks made no additional changes.

Previously deferred Suggestions and the documented R3-11 H1 limitation remain unchanged. New CI will validate the pushed merge commit.

@qwen-code /triage

Real-stack verification, round 9 — 40b05a74 and f2618e30

Follow-up to round 8 and the author's 40b05a74 update. This round also answers the question in the latest Critical-only review: whether the R5 Criticals hold at this head. The previous head, 67c908ea, is the A/B baseline throughout.

The head moved to f2618e30 while I was writing; it merges main, which now includes #12894. That merge leaves hosted-mcp-session.ts and managed-mcp-runtime.ts unchanged, but the Hosted turn path and the Broker changed. So I rebuilt f2618e30 and reran the key scenarios on it. Every result below is the same on both heads.

Merge reference in one paragraph.

• Fixed on the real stack. Each item below was A/B'd against 67c908ea:

  • the R4-9 trap from my round 8;
  • the refresh-cancellation P1;
  • R5-1 and R5-3;
  • the configuration half of R5-4.

  The approval path merged from main also works when the answer comes after the 300 s MCP grant lease.

• Not fixed: R5-2 for a Session with one pinned server. This is the most common shape. If its first mcp-release never reaches the Runtime, detach stays 503 and the Workspace lease stays held, exactly as on 67c908ea. Only Sessions with two or more servers now recover.

• Not new, and not covered by the R5-2 retry. A release that timed out (managed_mcp_drain_unknown) is answered with the same stored result forever. A Session with two servers never detaches, on either head.

• Flyway. feat(managed-agent): Add durable remote Shell result delivery #12894 has landed. f2618e30 carries V20–V22 and V23, and an empty database migrates to 1..23, so the ordering concern is gone.

I recommend fixing R5-2 for the one-server Session before merge, or explicitly accepting it next to R3-11.

¹ Labels follow the linked inline threads. The Critical-only review numbers some items differently: its "R5-6" is the inline R5-7, and its "R5-1" points at WorkspaceRuntimeTransport.java:145, which is not the inline R5-1 above and which I did not drive.

• Build. Fresh bundles and Spring jars from 40b05a74 and then f2618e30, on MySQL 8.4.11. main's new @a2a-js/sdk dependency needed a pnpm install first. The old arm is my 67c908ea build, with its Broker jar kept aside. The stdio server gained three test hooks:
  • addtool changes its tool list at runtime;
  • rawnotify sends a raw notification;
  • --sticky makes a grandchild keep stdout open for about 45 s after the server exits.
• R4-9 (fixed). A lone surrogate in a resource URI, a prompt name, an argument key or an argument value:
  • all four return 400 invalid_mcp_operation with 0 Broker requests and 0 server reads;
  • the next read settles, detach returns 204 and the lease goes to 0;
  • the well-formed emoji control passes through unchanged.
• Refresh-cancellation P1 (fixed). The model calls addtool, so the next refresh in the same turn starts a real replacement, and the Broker proxy holds that replacement's acquire:
  • 67c908ea: the turn is cancelled at 2.0 s, but the replacement mcp-configure is still sent at 4.9 s. A detach during the hold returns 204, releasing before the acquisition finished.
  • 40b05a74: no configure is sent after the cancel. A detach during the hold returns 503, so nothing is released. The next turn is OK and detach returns 204.
  • If the replacement's mcp-configure is held instead (after dispatch_started), it is sent once and not resent, on both heads.
• R5-1 and R5-4. A turn is running a 3 s tool on remote when an explicit replacement of the sticky server (rev 1 → 2) is posted. Retiring the old sticky connection takes more than 5 s. The table above has both outcomes.
• Approval merged from main. With approvalMode: default, the MCP tool call raises an approval request. I allowed it at 310 s, after the 300 s grant lease: the tool ran once with the original arguments, the turn ended end_turn with no Broker errors, and detach returned 204. A 2 s control behaved the same.

• R5-2, one pinned server (not fixed). The Broker proxy answers the first mcp-release with 503 without forwarding it.

  • Detach 1 returns 503.
  • Detach 2 goes straight to the Broker tool-session release, which returns 409 managed_runtime_identity_conflict ("still owns unfinished work"). After that, mcp-status and acquire return 409 runtime_session_not_ready, so the original release is never re-sent. Every later detach repeats this, and the lease stays held.
  • Cause: when every configuration is releasing, close() tries broker.release() before looking up or re-sending the release. The Runtime still holds the open connection, so that call fails, and afterwards the runtime Session is not ready for the re-send path.
  • With two pinned servers, the second configuration is still active, so this shortcut is skipped. Detach 2 re-sends both releases and returns 204; on 67c908ea it stayed stuck.
  • The new unit test retries an undelivered release with the original immutable identity kills the inverse mutant, yet the one-server shape above still fails on the real stack.

• A release that timed out (not new, not covered). Here the stdio server's stdout outlives the process by about 45 s, so its release exceeds the 5 s drain bound.

  • One server: detach returns 503 while the pipe is open and 204 at 52 s, on both heads.
  • Two servers, or a server replaced on 40b05a74: detach stayed 503 for as long as I watched (98 s and 134 s), and the lease stayed held.
  • On 40b05a74, each retry re-sends mcp-release with the same operation ID and gets back the stored outcome_unknown/managed_mcp_drain_unknown. The retry never drains again, so the later configurations are never released and their server process stays alive. 67c908ea only looks the release up and is equally stuck.
  • For example, a stdio server launched through a wrapper whose child keeps stdout open can take this path.

• Flyway. On f2618e30 the jar carries V20–V22 and V23, and an empty database migrates to 1..23 and starts. On 40b05a74, applying this PR first and feat(managed-agent): Add durable remote Shell result delivery #12894's V20–V22 afterwards failed validation (Detected resolved migration not applied to database: 20). Now that feat(managed-agent): Add durable remote Shell result delivery #12894 is on main, that order can no longer happen.

• Regression. The full round 1–8 set holds on 40b05a74, except that I did not rerun the worker SIGKILL scenario. The key subset (S1, list_changed, prompt cancel, busy owner, R1-2, hung-read status/cancel, lost replies, orphaned Session) reran on f2618e30 with the same results:

  Area	What I saw
  S1, three transports	exact blob, 409 conflicts, 0 leaks, detach and load
  list_changed	tools/list_changed and resources/list_changed recover in the same turn
  Raw-op cancel	202 running
  Prompt cancel	the turn ends cancelled
  Late settlement	33.6 s
  F6	17 reconfigures keep 1 live stdio process
  R1-2 / R1-5	both hold
  Busy owner	B detaches 204 and the holder is unchanged
  Hung-read status/cancel	200 / 202
  Lost replies	1 effect; revoked-access close 204
  SSE restart / HTTP down at close	served / detach 204
  Orphaned Session	once the old writer's store lease expires: load 200, prompt 202, detach 204, lease 0
  N1	unchanged: UNKNOWN from 37.6 s, turn ends at 638.5 s, then blocked

• Mutants. 9 of 10 are killed. Baselines were cli 226/227 and broker 530 with 1 skipped; the one cli failure is the known 404 load flake in serves MCP cancel during dispatch…. K2 survives because it removes the first configuring check in refresh(), and the same check is repeated right after the discover dispatch.

  Mutant	Change it inverts	Killing test
  K1	R4-9 route check	rejects invalid MCP strings before committing or dispatching (4 cases)
  K3	R5-2 re-send	retries an undelivered release with the original immutable identity
  K4	R5-3 Object.hasOwn	ignores inherited notification names without changing the catalog
  K5	R5-4 tolerated sibling drain	keeps a healthy replacement while retaining the undrained predecessor hold
  K6 / K7	refresh signal / close fence	stops a refresh replacement cancelled during intent / acquire / dispatch_started and fences close until it drains
  K8	late-approval grant renewal	honors allow approval for an MCP call while retaining its shared owner
  J14	R5-5 413	HttpRuntimeTransportTest.oversizedMcpRequestHasDefinitiveClassification and one more
  J15	R5-7 eviction	RuntimeBrokerServiceTest.failedAdoptedReleaseLeavesReclamationReachable

• CI. 40b05a74: 24 checks pass. f2618e30: 22 pass, and Test (ubuntu) and review-pr were still running when I posted. web-shell E2E Smoke and Desktop Shell are listed as failures, but both jobs were cancelled rather than failed.

• Not verified. Windows/Linux native runs, a real model, and MariaDB this round. I also did not rerun the worker SIGKILL scenario, and did not drive R5-5 or R5-7 end to end.

Evidence: wenshao/qwen-code@78144486/pr12946/r9. It contains per-scenario results for both arms and for f2618e30, the regression summaries, mutants by test name, the Flyway boots, and the harness, including the three new stdio server hooks.

The round 9 release-recovery findings are addressed in follow-up PR #13109, since this PR has merged. The earlier R5-2 fix was incomplete for a single pinned server: its test double did not preserve the Broker state transition after an unsuccessful owner release.

The follow-up persists connection drain proof before owner release, settles late physical closure on the original release receipt, and retains old-writer lost-ack recovery without treating network errors as closure proof. Final real-stack verification passed all five scenarios, including a baseline Harness writing the legacy records and a new Harness recovering them. Verification report. Unknown invocations still retain their holds.

第九轮报告的两项释放恢复问题已在后续 PR #13109 修复。此前单 server 的 R5-2 判为完全修复不准确：测试替身遗漏了失败 owner 释放造成的 Broker 状态变化。新 PR 已通过五项最终真实链路验证，包含真实旧写者记录的升级接管；未知调用仍保留占用。