feat(serve): implement generic Broker provider controls - #12868

E2E verification on 233d49b5cc plus this feature, using a fresh build/typecheck/bundle and JDK 21:

• Provider contract: 39/39 check groups passed, no skips. The built TypeScript provider calls the production Java Broker/HTTP transport and the bundled boot-v1 worker, with H2 JDBC execution persistence. Checks cover all nine controls, exact seven-field persisted references, no effects before start, one dispatch across retries, approval/cancellation, active-release rejection, unreserved cleanup, identity/fencing, and lost-response UNKNOWN without replay.
• HostedWorkspaceToolTurnIT: 1/1 passed. The packaged Hosted Harness, Spring Broker, boot-v2 Workspace workers and SQL store preserve feat(serve): add gated Hosted Workspace file tool turns #12831's raw payloadJson path: two Workspaces, read/write/edit, history reload, lost start acknowledgement without duplicate dispatch, Shell refusal, and blocking on unknown outcomes.
• Fault gates: 28/28 passed (context 12, lost response 7, process crash 6, concurrency/storage 2, control 1). After correcting the managed test adapter to use real provider release before Workspace deactivation, the affected context group passed again, including the observed control → activation request order.
• Other validation: 966 relevant TypeScript tests; 384 Broker unit tests with zero failures/errors and one existing optional real-worker test skipped; 12 Workspace tests; build/typecheck/bundle, changed-file ESLint/Prettier, and Checkstyle for both affected Java modules.

The provider E2E uses a static pre-attested lease and deterministic Session/binding fixtures; the execution transport, worker, tools and H2 execution repository are real. The Hosted test uses a deterministic local model endpoint. These runs do not claim live-model, MySQL, cross-platform or worker-restart recovery coverage. Worker observation remains current-turn scoped; Broker HTTP observation requires a READY Session, while durable execution evidence survives release.

提交前验证：provider 39/39 组、Hosted 集成 1/1、故障门禁 28/28 均通过；审计修正后受影响的 12 项故障测试再次通过，并确认真实 release 在 Workspace 停用前完成。另有 966 个 TypeScript 测试、384 个 Broker 单测（既有可选跳过 1 个）和 12 个 Workspace 测试通过，构建、类型、格式、lint 及两个 Java 模块的 Checkstyle 均通过。覆盖边界如上，未把确定性模型或本地 H2 测试宣称为线上、MySQL 或重启恢复验证。

Three complete open-ended and reverse audit rounds preceded submission. The first found and corrected the managed fault-gate release adapter; rounds 2 and 3 independently reviewed the full updated diff and found no remaining actionable findings. The final staged tree matched the reviewed snapshot, and the actual E2E runtime artifact hashes remained unchanged. This records manual audits; the earlier model-CLI review timed out and is not counted as a passing check.

提交前完成三轮完整无方向审计与反向审计。第一轮修正 managed 故障测试的 release 适配层，第二、三轮独立全量复核均无剩余可操作问题。最终暂存内容与审计快照完全一致，E2E 运行产物哈希保持匹配。此处记录人工式独立代码审计；此前模型 CLI review 超时，未计入通过项。

Real-stack verification of f67bde6b41

Verdict: ready to merge from the real-stack side. No blocking finding. Every item of the reviewer test plan held on a stack the PR's own evidence did not cover: MySQL 8.4.7 instead of H2, workers provisioned by the production local-process provisioner instead of a static lease, and a live model on the Hosted Workspace loop. Three follow-up observations (F1–F3) and two notes are listed below; none of them changes an outcome the PR claims.

Setup

Everything on the request path is the shipped artifact: the built TypeScript BrokerManagedRuntimeProvider (packages/cli/dist), the Spring managed-agent-server jar with its embedded Runtime Broker and production HttpRuntimeTransport / WorkspaceRuntimeTransport, Flyway-migrated MySQL 8.4.7, and bundled workers (dist/cli.js managed-runtime-worker) started by the local-process provisioner. Sessions are created through the public /v1/agents/sessions API: a plain Session gives boot v1, a Workspace Session gives boot v2 with installed context, activation and storage ownership.

Two things are rig-only: a small auth adapter that names the actor for the public API, and an HTTP forward proxy between Broker and worker (JVM -Dhttp.proxyHost, empty nonProxyHosts). The proxy keeps a ledger of every Broker → worker request and injects the faults. Effects are counted with a shell command that appends a line; records are read from MySQL directly.

1. Before and after

2. The reviewer test plan

3. Fault injection

After the worker is killed, start for the reserved invocation fails closed with no worker request at all, and nothing is rebuilt from the stored reference. That matches the open boundary the design document states. On boot v1 the next Runtime Session gets a replacement worker and runs normally.

4. Live model on the Hosted Workspace loop

The PR changes the release path of raw-only Sessions too: the worker now acknowledges release before the Workspace is deactivated. Two runs cover it.

HostedWorkspaceToolTurnIT pins an in-memory H2 database, so I pointed the PR's own driver, unchanged, at the Spring server on MySQL 8.4.7. It prints HOSTED_WORKSPACE_TOOLS_OK (two Workspaces, two tool rounds, lost start acknowledgement, replay, Shell refusal, unknown blocking), with 7 executions SETTLED/success and both worker release acknowledgements followed by the Workspace deactivation.

Then the same loop with qwen3.8-max:

5. Repository suites on this head

The 5 failing cli files are local to my machine: ssh-workspace, session-pr-backfill, session-pr-refresh and aone-mrs fail with the same 36 tests on the merge-base build, and web-shell-pairing passed on rerun. CI is green on this head (23 checks passed), and the bot review approved this commit.

6. Mutation matrix

33 of 40 single-edit mutants are killed by the repository's unit suites. The 7 survivors are all second guards; the first guard for the same input sits on the other side of the wire.

• J4, J5, J13, J14: the closed field sets of the reserve and start bodies, unknown fields in a control operation, and the file-history owner check in the Broker. I compiled these four into one server jar and reran the probes. B.7, B.9, C.2 and C.3 fail on that jar, so the real stack notices what the unit suites do not. Four small HTTP-level tests would pin them.
• T2, T8, T18: closeSessionAdmission on provider release (the Session is already fenced by providerSessions), the worker's refusal to release with pending controls (the Broker refuses first), and the worker's 413 for an oversized response (the transport has its own bound).

J2 mutates a guard that predates this PR; it is listed because the new mixing rule depends on it. Four failures in hosted-harness-session.test.ts appeared during the full runs under load. They are not kills: with the same mutants applied, that file passes 16/16 on its own.

Points from the bot review, measured

The bot review read the code and named the missing end-to-end run as its one gap. This report is that run. Its other points, against measurements:

Observations (non-blocking)

F1. A definite worker refusal is recorded as UNKNOWN, and the Session can then never be released. When the worker answers execute with 409 and a reason, the Broker stores UNKNOWN, exactly as it does for a lost reply. UNKNOWN counts as active work, :resolve answers 501, and the HTTP face has no reconcile route, so release answers 409 runtime_session_busy from then on. On boot v2 the storage claim stays: the next Runtime Session and a sibling Harness Session of the same Workspace both get 409 workspace_busy.

This is inherited, not introduced. The merge-base build does the same on the raw path (Managed Runtime does not admit this tool. → UNKNOWN → release refused, storage held). What this PR adds is more refusals that land there. I saw three new ones: preflight has not permitted execution, invocation cancelled and Managed Runtime protocol conflicts (a raw call into a provider-owned Session, C.6). So "mixed admission is rejected" holds without effect, but it costs the Session. A possible direction: let the worker mark refusals that happen before dispatch with their own code, and let the transport settle those as not_started, the way WorkspaceRuntimeTransport.execute already does for its own pre-dispatch checks. A plain "409 means not started" rule would be unsafe: by my reading of the route's catch block, an error raised after the tool started is answered with 409 as well.

F2. Invalid tool arguments reach the caller without the reason. For write_file with a relative path the worker answers 409 managed_runtime_identity_conflict with File path must be absolute: relative/path.txt. The Broker forwards the code and replaces the message with Managed Runtime control request failed (HTTP 409).; the TypeScript client keeps only status and code. An unknown tool name gives the same code. The worker used this one code for 10 different reasons in my runs. A Harness that wants to hand a validation error back to the model cannot tell it from an identity conflict. An oversized argument is renamed on the way: the worker says 400 managed_runtime_provider_invalid, the caller sees 400 managed_runtime_attestation_invalid.

F3. The worker keeps every released provider Session. One worker, 1,500 Runtime Sessions acquired, used and released in sequence: RSS 161 → 316 MiB, 278 MiB after 15 s idle, about 80 KiB per released Session. The same count through the raw path on the same build stays flat (152 → 146 MiB). ManagedRuntimeProviderWorker.sessions never drops an entry, and the entry holds the Session's Config, tool set and file history. After release the Broker answers 404 runtime_session_not_found for history, status and cancel, so nothing can read that state any more. With one Runtime Session per turn this grows with conversation length.

Note 1. Approval is the caller's duty, not the worker's. On boot v1 prepare reports defaultPermission: "ask", yet prepare → preflight → start with no confirm writes the file (B.1b). That is core's existing contract ("the private parent calls this only after permission"). I mention it because "boot v1 uses DEFAULT approval" can be read as the worker enforcing it.

Note 2. The argument bound is 256 KiB, not the 1 MiB envelope. A 255 KiB write_file goes through all steps (the largest response is 511 KiB); 256 KiB is refused at prepare by core's MAX_JSON_BYTES. This equals the raw path's payload bound, so the two paths agree.

Merge order. A trial merge with current main (3f5ae3ffeb), #12855 and #12870 is clean. With #12848 it conflicts in managed-context-worker.ts, RuntimeBrokerHttpServer.java and RuntimeBrokerService.java; with the W0e drafts #12839, #12865 and #12869 it conflicts in RuntimeBrokerService.java and WorkspaceRuntimeTest.java. Whichever lands second needs a manual resolution and a rerun of the fault gates.

Not covered

Linux and Windows; MariaDB (CI covers it); Broker restart with live provider Sessions and prepared-state recovery after a worker restart, which the PR lists as out of scope; a product flow that drives the provider path with a model, because no consumer of that path exists yet. The provider probes are therefore at the level of the provider API.

Reproduce

Rig, probes, mutant list and raw logs: pr12868/harness and pr12868/results. Order: spring.sh with the proxy, then s4-ab, s1-contract, s2-faults, s5-raw, s9-author-driver-mysql, s6-hosted-real-model, s3-retention, s8-limits, mutate.

Addressed the code review and real-stack findings in f0f217dfa52fe00dea669b9c9ffdad05c2efb832.

F2 is fixed. Unknown tools and invalid tool construction now return 400 managed_runtime_tool_invalid; unsupported provider profiles return 501 managed_runtime_provider_unsupported. Known provider errors retain their status, code and bounded reason through Java and TypeScript. Java only accepts the matching status/code, closed JSON body and expected response headers; malformed or unrecognized responses keep the generic error. The TypeScript client only displays a bounded reason from the Broker's closed three-field JSON error envelope. These changes preserve execution uncertainty and replay rules.

E2E report: the final rebuilt node dist/cli.js worker, production Java Broker, built TypeScript client and H2 journal passed 39 existing check groups plus 8 targeted groups (including two startup checks), without mocked control/error responses. Relative WriteFile paths and unknown tools reach the caller with the original reason and tool_invalid; oversized arguments retain provider_invalid instead of becoming an attestation error; a real unsupported boot-v2 profile retains 501 provider_unsupported. Correcting the arguments in the same Session then prepares and executes successfully. A deliberate execute without preflight still becomes UNKNOWN: a repeated start is rejected, exactly one execute reaches the worker, and no file is written.

Mutation survivors are covered. Four HTTP regression tests now reject extra reserve/start fields, unknown control fields and foreign history owners before any transport effect or reservation mutation. In isolated source copies, removing each J4/J5/J13/J14 guard makes its corresponding test fail with HTTP 200 instead of 400; each run has one assertion failure and zero errors. The intact HTTP suite passes all 11 tests.

Other comments are recorded explicitly:

• F1 — deferred: HTTP execute rejection conservatively becoming UNKNOWN is inherited behavior. Error text alone cannot prove dispatch never occurred. Safe pre-dispatch evidence and settlement need a separate change; release can still return runtime_session_busy and Workspace ownership can remain held. The new E2E deliberately confirms this limitation rather than treating it as fixed.
• F3 — deferred: retained heavyweight released-Session state is real. Independent reproduction retained about 100.70 MiB of post-GC heap after 1500 Sessions; idle/GC did not remove it. However, the private worker still serves current-turn status/cancel and bound history after release, while Broker HTTP rejects them. Immediate deletion/disposal would remove existing evidence. A follow-up needs lighter retained evidence and an explicit retention policy; this revision does not claim bounded memory.
• Approval and size limits — clarified in both design languages: boot-v1 DEFAULT approval relies on the caller to enforce confirmation; arguments retain the existing 256 KiB canonical-JSON bound independently of the 1 MiB control envelope / 8 MiB history envelope.
• 25 ms polling and a sibling JSON schema — deferred: bounded backoff and schema consistency are useful follow-ups. Current cancellation observation is serial and bounded by the operation lease, and both languages already enforce the closed contract through parsers/shared fixtures. No correctness fix was established for these suggestions.
• JsonCodec persistence — no confirmed regression: current provision seeds use six non-null fields; local-process handles use non-null provider data. The reported MySQL old-database adoption and comparison support that existing path. Static provisioning does not persist a handle here, and this repository has no Kubernetes provisioner implementation. This is not a claim about arbitrary external handle extensions.
• Cancelled Shell / consumers: cancellation remains an explicit cancelled outcome with failure-hook behavior, covered through the private provider runtime. This does not enable the separately gated Hosted product path.

Final validation: full build, typecheck and bundle; 136 core + 892 CLI tests; all 390 Broker tests with zero failures and one existing optional bundle-provisioner test skipped; changed-file ESLint/Prettier, Java Checkstyle and diff checks. The initial legacy interop run collided with the TypeScript rebuild before worker readiness; the post-build full Java suite passed that exact test. Existing feature-stage Hosted integration evidence remains in the earlier E2E report; it was not rerun for this diagnostic-only revision.

Before this commit, three fresh full open-ended and reverse audit rounds completed. Round 1 tightened the TypeScript error envelope and reset the clean count; rounds 2 and 3 independently found no new actionable issues on the unchanged patch. The audits traced the lifecycle/ownership, control/execute separation, error consumers and test oracles; no unresolved Critical was identified. Deferred suggestions above remain visible.

The reviewed and E2E-verified patch against 1b179d35a3 has SHA-256 f065e042080ba0e13884fbac30c02304b9839781ce547268540340e89e91fcef. No deferred item above is silently counted as fixed.

Real-stack verification, round 2: f0f217dfa5

Verdict: ready to merge. No blocking finding. The fix commit does what its description says: F2 is fixed on the real stack, the four Java mutation survivors are pinned by tests, and nothing else moved. One residual of the F2 fix is worth a small follow-up (R1 below); a candidate patch with a test is attached. It does not block.

Round 1 was at f67bde6b41. The head now adds a merge of main and f0f217dfa5. Same rig as round 1: built TypeScript provider → Spring-embedded Broker on MySQL 8.4.7 → bundled workers, with the JVM HTTP proxy between Broker and worker.

What the head contains

The merge commit 1b179d35a3 has the same tree as a mechanical merge of f67bde6b41 with 0a136f89d7 (b8f0272e), so nothing was changed by hand in it. It brings in #12863 and #12864, both test-only. f0f217dfa5 touches 12 files, 447 lines added.

1. F2 is fixed

After a refusal the Session stays usable. Two refused preparations, then corrected arguments under the same call identity: prepare, reserve, approve, preflight and start succeed, one row, one effect, release works (Q.1–Q.3 on both boots).

2. R1: the code is lost again when the reason is long

The Broker forwards a provider error only when the reason has at most 4,096 characters and no NUL. That bound is right. The gap is on the sending side: validation messages echo the caller's input, and the worker sends them at any length. A model-written shell script of 5,969 characters that trips the sleep guard gives a reason of 6,242 characters. The Broker then drops the code together with the text, and the caller sees 400 managed_runtime_attestation_invalid with no reason, which is the round-1 behaviour. A path with a NUL does the same.

Candidate: bound the reason in the worker and keep the code. 33 changed lines in managed-runtime-provider-worker.ts plus one test; Java is unchanged. On the real stack it keeps managed_runtime_tool_invalid in all 11 cases. The worker test file passes 11/11 with it, and ESLint, Prettier and the cli typecheck are clean.

+const PROVIDER_REASON_LIMIT = 4096;
+const PROVIDER_REASON_OMISSION = ' … [reason truncated]';
+
+/**
+ * The Broker forwards a reason only when it fits its bound and holds no NUL;
+ * otherwise it drops the code together with the text. A reason that echoes
+ * caller input can exceed the bound, so bound it here and keep the code.
+ */
+function providerReason(error: unknown): string {
+  const text =
+    error instanceof Error && error.message
+      ? error.message
+      : 'Managed Runtime provider operation failed.';
+  const clean = text.replaceAll('\0', '�');
+  if (clean.length <= PROVIDER_REASON_LIMIT) return clean;
+  let end = PROVIDER_REASON_LIMIT - PROVIDER_REASON_OMISSION.length;
+  const last = clean.charCodeAt(end - 1);
+  // Never split a surrogate pair.
+  if (last >= 0xd800 && last <= 0xdbff) end--;
+  return clean.slice(0, end) + PROVIDER_REASON_OMISSION;
+}

3. The Broker trusts only a closed, matching error

This is the description's claim "Java only accepts the matching status/code, closed JSON body and expected response headers", checked on the production transport with a real worker behind it. The last block matters most: a well-formed refusal in place of the answer of an execute that really ran settles nothing.

4. Regression and new probes

Two things here are new and answer the review by qqqys, which asked for the parts it could not reach:

The author did not rerun the Hosted integration for this revision, so both Hosted runs in the figure are on this head.

5. Mutation matrix

56 of 61 mutants are killed by the repository's unit suites. Round 1 was 33 of 40.

• The four Java survivors of round 1 (J4, J5, J13, J14) are killed by the four new HTTP tests.
• 21 new mutants on the lines the fix adds: 19 killed. The two survivors, N17 and N18, are both about which code a 409 carries. Removing the ManagedToolConflictError branch, or turning the catch-all back into managed_runtime_identity_conflict, fails no test. No test asserts the code of a 409 refusal, which is the same area as the label note below.
• T2, T8 and T18 survive as in round 1. They are second guards behind a first guard on the other side of the wire.
• The other 33 round-1 mutants were rerun on this head and are all killed.

Failures in hosted-harness-session.test.ts appeared in 15 of the full TypeScript runs while the host was under load. They are not counted as kills: with T2, T14 and N19 applied, that file passes 16/16 on its own.

Status of the round-1 items

Two small notes on the new codes. Neither blocks.

• Labels are partly swapped. A tampered argsDigest, a changed decision and changed arguments are real identity mismatches, and they now carry the catch-all managed_runtime_provider_operation_failed. A background shell and a media context on the wrong tool are input problems, and they still carry managed_runtime_identity_conflict. The reason text makes each one readable.
• The reason is forwarded verbatim, including newlines and ESC sequences (R5). A consumer that prints or logs it should treat it as untrusted text.

Merge order

A trial merge with current main (c767078ff7) is clean, and on that merged tree the Broker unit suite (390), the fault gates (28), the server suites and the Hosted ITs pass, including the Hosted reply-loss gate that #12873 added. #12855 and #12870 merge cleanly on top. #12848 conflicts in 4 files; #12839, #12865 and #12869 conflict in 3 files each (broker-managed-runtime-provider.ts, RuntimeBrokerService.java, WorkspaceRuntimeTest.java).

Not covered

Linux and Windows; MariaDB; Broker restart with live provider Sessions; a worker that never settles a prepared cancellation; the unsupported boot-v2 profile on the real stack, which the Spring server cannot produce (its 501 was checked on the transport by substitution, and the worker branch by the unit test).

Reproduce

Round-2 rig, probes, candidate and logs: pr12868/r2. New probes are s11-error-envelope and s12-lost-control; proxy.mjs gained a rewrite fault.

@qwen-code /triage

Resolved the three conflicts with main d66fdadd2794 in merge commit ac0b6cf966.

The merge preserves provider/raw protocol separation, bounded error reasons, main’s terminal/runtime-loss receipts, and Workspace loss fences. Repeated cancellation of a prepared provider invocation still waits for worker cleanup rather than acknowledging only the Broker’s settled record. Both English and Chinese designs and the PR description now distinguish terminal receipts from live Session/history controls.

Validation on the exact committed tree:

• Build, typecheck, bundle, ESLint/Prettier and Java Checkstyle passed.
• CLI: 115 tests; Broker: 402 tests with no skips; Workspace/schema: 17 tests; packaged Hosted Workspace tool loop: 1 integration test, all passed.
• Real built TS provider → production Java Broker → bundled worker: 40 provider groups and 8 error/recovery groups passed (four readiness groups included). The added released-receipt probe verifies unchanged Session state/version, persisted execution and file effects, with zero additional worker requests. It uses H2 execution storage with static placement and in-memory Session metadata; the added real-chain probe covers SETTLED, while service/recovery tests cover ABANDONED.
• Two consecutive full conflict-resolution audits, each open-ended and reverse-checked, found no new Critical issues. The committed tree exactly matches the audited/tested tree; source and build fingerprints were unchanged during E2E.

Per the requested Critical-only policy after round 5, the non-blocking R1 oversized/NUL-reason suggestion in the latest real-stack review, along with its error-label, terminal-escaping and remaining mutation-coverage suggestions, is deferred to follow-up work. Earlier F1/F3 limitations remain documented and are not claimed fixed.

已解决 3 处冲突并合入 main；连续两轮无方向审计与反向审计未发现新增 Critical。上述本地构建、测试和真实链路验证全部通过；非阻塞建议按第 5 轮后仅修复 Critical 的要求保留为后续工作。

@qwen-code /triage

Real-stack verification, round 3: ac0b6cf966

Verdict: ready to merge. No blocking finding in this PR. The head adds one commit, the merge of main after #12839, and that merge was resolved by hand. Both seams of the resolution behave as the merge note says, and nothing from rounds 1 and 2 moved.

Two things to know before merging:

• Sequencing, not caused by this PR. Current main holds two Flyway V16 migrations (feat(managed-agent): Add W0e terminal recovery fences #12839 and feat(managed-agent): Make Session close, archive and delete durable operations (Stage D4) #12881), so no Spring context and no server jar starts on it. This head predates feat(managed-agent): Make Session close, archive and delete durable operations (Stage D4) #12881 and is unaffected. fix(managed-agent): Move the Session operation migration to V17 #12900 renumbers the later migration; a trial merge of main + fix(managed-agent): Move the Session operation migration to V17 #12900 + this head is clean and green (section 6).
• One guard written during the merge has no test (mutant G4, section 2). It is correct on this head. Non-blocking.

Same rig as before: built TypeScript provider → Spring-embedded Broker on MySQL 8.4.7 → bundled workers, with the JVM HTTP proxy between Broker and worker.

What the head contains (figure 1)

The round-2 merge had the same tree as a mechanical merge. This one does not, so I read it with git show --remerge-diff, which prints only what the recorded merge changes against a mechanical one. The production decisions are all in RuntimeBrokerService.java:

• Reservations (createExecution, prepareExecution) now go through main's receipt lookup, with this PR's validated reference.
• cancelExecution answers a terminal record from storage, except a prepared provider call whose Session is not yet released, which still goes to the worker.
• That exception is decided by a new helper, isPreparedProviderCancellation(): settled, cancel requested, dispatch generation 0, provider reference.

1. Seam 1: terminal receipts reach the provider path (figure 2)

2. The guard written during the merge (figure 6)

isPreparedProviderCancellation() has four conditions. Two mutants each remove one. Removing the provider-reference test (G5) is killed by the repository's tests. Removing getDispatchGeneration() == 0 (G4) is not: all 402 Broker tests pass with it.

So I compiled G4 into a server jar and ran two probes on both jars:

• M4, a call that was dispatched and then cancelled. This head answers a repeated cancellation from the record. The mutant asks the worker again each time. The answer is the same, 200 cancelled.
• M5, a call that finished before the worker heard the cancellation (the cancel request is delayed 5 s on the wire; the command takes 2 s). The record is SETTLED/success with cancel requested. This head answers a later cancellation 200 success. The mutant answers 503 runtime_execution_cancel_failed, every time, because the worker reports success and the prepared-cancellation path accepts only cancelled or not started.

The guard is right. A service test for the M5 case (settled, success, cancel requested, generation 1, cancel again → stored record, transport not called) would pin it.

3. Seam 2: worker death under main's Runtime-loss fences (figure 3)

The worker is killed with SIGKILL while one invocation is reserved. On every build nothing executes and nothing is replayed.

The change is inherited from #12839. The merge neither weakened nor extended it. For the rig this meant one fresh database and Broker per worker-death scenario, because a lost placement refuses later placements.

4. Regression (figure 4)

The cli src/serve failures in the figure are local. CI on this head is green: 21 checks pass, 4 are skipped, none fail. That includes Test (ubuntu-latest, Node 22.x).

5. Mutation matrix (figure 5)

Nine new mutants (G1–G9) edit the lines written or relied on while resolving the merge: 8 are killed, G4 survives (section 2). The 61 mutants of rounds 1 and 2 were rerun on the merged code: 56 killed, and the same five survive (T2, T8, T18, N17, N18). No verdict changed, so the merge lost none of the behaviour the tests pin.

Kills are attributed per test from the JSON reports. Failures of hosted-harness-session.test.ts in full runs under load are not counted: N17 had only such failures, and with N17 applied that file passes 16/16 when run alone.

6. Current main carries two V16 migrations (figure 7)

Found while trial-merging this head with current main. #12839 merged at 05:14Z with V16__runtime_loss_evidence.sql. This head's merge commit was made at 05:32Z. #12881 merged at 05:37Z with V16__managed_session_operation.sql. git sees two different file names and reports nothing; Flyway refuses to start.

• main's own CI shows it: job Runtime Broker and Managed Agent MariaDB / Java 21 fails on 42d7a2b833 with Found more than one migration with version 16.
• The server jar built from main + this head exits at startup on a fresh MySQL database with the same message.
• This PR's CI is green because its Java jobs started at 05:32:45Z, five minutes before feat(managed-agent): Make Session close, archive and delete durable operations (Stage D4) #12881 landed.

#12900 moves the later migration to V17. A trial merge of main (fc4e01b9fc) + #12900 + this head has no conflict. On that tree the Broker unit suite (407), the server suite (154 and 11 ITs), the Hosted suite (154 and 5 ITs) and core (190) pass. Its server jar starts on a fresh MySQL database and applies versions 15, 16 and 17 in order. On that jar the reviewer test plan passes 76/76 and the merge probes 22/22.

Status of earlier items

One small interaction between the two sides of the merge. It does not block. The client attaches a reason only when the error body has exactly three keys. main added an optional fourth key, details. An error with details therefore keeps its code and never its reason (N.2, stand-in peer). Today the only such answer is the ABANDONED one, which the client translates to unknown / terminal, so no caller sees a difference yet.

Merge order

Not covered

Linux and Windows; MariaDB; an ABANDONED record on the real stack, which needs the recovery flow of #12839; Broker restart with live provider Sessions; a worker that never settles a prepared cancellation; F3 memory growth was not remeasured.

Reproduce

Round-3 rig, probes, mutants and logs: pr12868/r3. New probes are s13-merge.mjs (M1–M5), kill-scenario.sh and s14-details-envelope.mjs. remerge-r3.diff is the output of git show --remerge-diff.

Review comments addressed on cc06ee0e4e.

R1-30 and R1-32 are fixed with focused regression tests; the mixed R1-2 comment has a separate evidence-based reply. The remaining 27 inline Suggestions were checked against the current code and are recorded below under the requested policy to land only Critical fixes after round 5.

Additional verification comments: the lone-surrogate TS/Java divergence, synthetic cancelled/error combination coverage, G4 mutation coverage and previously disclosed F1/F3/long-reason limitations remain follow-ups. The proposed correction from “402 / zero skips” to one skip does not apply to our recorded run: we passed -Dqwen.runtime.worker.bundle=/Users/wenshao/git/qwen-code-k1/dist/cli.js, and the saved Maven result is Tests run: 402, Failures: 0, Errors: 0, Skipped: 0. The reviewer’s run without that property legitimately skipped its optional worker test; these are different runs. We have not rerun that whole Java suite for this TS-only revision.

Validation of this committed tree: build/typecheck/bundle and ESLint/Prettier passed; 136 core tests and 859 CLI tests passed. Three isolated source mutations were each caught by the target regression assertions. Real bundled-worker verification passed (three corrected-behavior groups plus two unchanged diagnostics), followed by 40/40 provider and 8/8 error/recovery groups through the production Java Broker, H2 execution storage and real worker. The 40/8 totals include four readiness groups. All recorded source, scripts and built artifacts were unchanged during the final runs. The first real-worker attempt exposed a fixture collision from reusing raw callId across Sessions; the fixture was corrected, evidence retained, and the final runs passed without another product change.

Three complete open-ended and reverse audit rounds preceded this commit. Round 1 found a stale test expectation; after correcting it and rerunning validation, rounds 2 and 3 were clean. The committed tree exactly matches the audited/tested tree. These checks do not claim live-model, new-platform, scale, or runtime-loss recovery coverage.

本轮修复 R1-30、R1-32；R1-2 拆分核验后按非阻塞后续项记录，其他 27 条建议逐项保留上表结论。没有扩大已声明的权限、容量或兼容性契约。

@qwen-code /triage

Real-stack verification, round 7: ebea694e4e, merged as 1f1209bc70

The PR was merged while this round ran. Two findings of the round are on main now and need a follow-up: R7-1, and the smaller R7-2. 9cb9dc86e8 introduced both, the head had them, and main a63157304a has them. R7-1: a start the worker refuses never returns to the caller. Nothing in production calls the provider yet, so nothing that runs today is affected. One condition in the Broker removes it (candidate B, below); with it every probe of all seven rounds passes, and the patch applies to main. Everything else the round measured is in order.

The squash commit carries the files of the head byte for byte, except config.ts, which main had changed as well. On main a63157304a, one commit after the merge, the start of R7-1 has no answer after 20 s while the worker is asked 335 times; with candidate B it is refused in 0.1 s, Y1 to Y8 pass 21 of 21, and the Broker suite passes with 514 tests.

The head is the round-6 head plus seven commits. The branch moved five times while this round ran. Each state was built and measured, and every number below is from the head unless it names another build. 2 reviewers have approved this head; all 96 review threads are resolved; the review decision on the PR reads APPROVED. None of the reviews names R7-1. Earlier rounds: 1, 2, 3, 4, 5, 6.

R7-1. A start the worker refuses never returns

A call is prepared and reserved, and the caller starts it before preflight has permitted it. The worker refuses the execute with 409, as it should.

Since 9cb9dc86e8 the Broker asks the worker about a provider execution whose record is UNKNOWN, and answers 200 with the state the worker names. For a lost answer that is right: the worker says executing or settled. For a refused start the worker says prepared. Nothing will start that call, because the Broker never dispatches a second time, so the provider client, which polls every 50 ms until the state is settled, polls without an end. Only disposing the provider stops it.

The same request pattern is what a Hosted turn would send if it started a call it had not yet been allowed to run. The provider has no production caller today, so nothing that runs is affected. But before this commit the mistake failed closed in a tenth of a second, and now it hangs the turn and loads the worker. It also shows in earlier probes: E.2 of round 1 gets no answer, and checks of groups B and C that count requests to the worker find the polling of an unrelated call (section 3).

ebea694e4e softens one consequence and leaves the cause: a caller that cancels now reaches the worker, the record settles as not_started, and the Session is released. The start itself still does not return.

Candidate B is one condition in RuntimeBrokerHttpServer.observedExecutionEnvelope: a provider execution the worker still holds as prepared was never started, so the Broker keeps the answer it gave before, 409 runtime_broker_execution_unknown. With it on this head: the start is refused after 0.1 s with 409, and the worker is asked about the call once, not 17 times a second; Y1 to Y8 pass 21 of 21, every probe of rounds 1 to 6 passes, the Broker suite passes with 514 tests, and Checkstyle is clean. Its test fails without the condition. Lost answers still settle (Y6), and the cancellation of ebea694e4e still works.

diff --git a/packages/sdk-java/runtime-broker/src/main/java/com/alibaba/qwen/code/runtimebroker/RuntimeBrokerHttpServer.java b/packages/sdk-java/runtime-broker/src/main/java/com/alibaba/qwen/code/runtimebroker/RuntimeBrokerHttpServer.java
index 1ddaf1d258..772e1655c9 100644
--- a/packages/sdk-java/runtime-broker/src/main/java/com/alibaba/qwen/code/runtimebroker/RuntimeBrokerHttpServer.java
+++ b/packages/sdk-java/runtime-broker/src/main/java/com/alibaba/qwen/code/runtimebroker/RuntimeBrokerHttpServer.java
@@ -332,7 +332,12 @@ public final class RuntimeBrokerHttpServer implements AutoCloseable {
             String runtimeSessionId, ExecutionReconciliation observation) {
         ToolExecutionRecord record = observation.getRecord();
         String state = observation.getRuntimeState();
-        if (record.getState() == ToolExecutionRecord.State.UNKNOWN
+        // A provider call the worker still holds as prepared was never
+        // started, and nothing dispatches it a second time: it stays UNKNOWN
+        // for the caller, who would otherwise wait for it for ever.
+        boolean neverStarted = "prepared".equals(state)
+                && ProviderRuntimeProtocol.isReference(record.getReference());
+        if (record.getState() == ToolExecutionRecord.State.UNKNOWN && !neverStarted
                 && ("prepared".equals(state) || "executing".equals(state)
                     || "cancel_requested".equals(state))) {
             Map<String, Object> response = envelope(harnessSessionId, runtimeSessionId,
diff --git a/packages/sdk-java/runtime-broker/src/test/java/com/alibaba/qwen/code/runtimebroker/RuntimeBrokerHttpServerTest.java b/packages/sdk-java/runtime-broker/src/test/java/com/alibaba/qwen/code/runtimebroker/RuntimeBrokerHttpServerTest.java
index 2be20af7c3..53464b377b 100644
--- a/packages/sdk-java/runtime-broker/src/test/java/com/alibaba/qwen/code/runtimebroker/RuntimeBrokerHttpServerTest.java
+++ b/packages/sdk-java/runtime-broker/src/test/java/com/alibaba/qwen/code/runtimebroker/RuntimeBrokerHttpServerTest.java
@@ -226,6 +226,34 @@ class RuntimeBrokerHttpServerTest {
         }
     }
 
+    @Test
+    void providerStartTheWorkerNeverBeganStaysUnknownForTheCaller() throws Exception {
+        String runtime = "550e8400-e29b-41d4-a716-446655440303";
+        try (Fixture fixture = new Fixture()) {
+            fixture.service.acquire("harness", runtime, "bootstrap").toCompletableFuture().join();
+            Map<String, Object> reference = Map.of("sessionId", runtime, "promptId", "turn",
+                    "callId", "worker-call", "capabilityDigest", "a".repeat(64),
+                    "policyRevision", "policy", "invocationId", "invocation", "argsDigest", "b".repeat(64));
+            String id = fixture.service.prepareExecution("harness", runtime, "provider", reference)
+                    .toCompletableFuture().join().getExecutionCallId();
+            Map<String, Object> start = Map.of("protocolVersion", 1, "requestId", "start",
+                    "harnessSessionId", "harness", "runtimeSessionId", runtime);
+            // The worker refused the execute: it still holds the call as prepared.
+            fixture.transport.runtimeStatus = Map.of("state", "prepared");
+            for (int attempt = 0; attempt < 2; attempt++) {
+                HttpResponse<String> started = fixture.post("/executions/" + id + ":start", start);
+                assertEquals(409, started.statusCode(), started.body());
+                assertEquals("runtime_broker_execution_unknown",
+                        JSON.parseObject(started.body()).getString("code"));
+            }
+            HttpRequest read = HttpRequest.newBuilder(fixture.uri("/executions/" + id
+                    + "?requestId=read&harnessSessionId=harness&runtimeSessionId=" + runtime))
+                    .header("Authorization", "Bearer secret").GET().build();
+            assertEquals(409, fixture.client.send(read, HttpResponse.BodyHandlers.ofString()).statusCode());
+            assertEquals(1, fixture.transport.executions.get());
+        }
+    }
+
     @Test
     void providerObservesTheOriginalExecutionAfterResponseLoss() throws Exception {
         // Provider references carry a UUID Runtime Session id.

R7-2. After a worker is lost, an inspection throws where it answered "unknown"

Smaller, from the same commit, and for the same follow-up. A worker is killed while a command runs. The record is UNKNOWN, as before, nothing is started twice, and the storage stays held. What the caller is told changed with 9cb9dc86e8:

The Broker now tries to ask the worker, finds it gone, and answers with the error of that attempt. The provider turns only runtime_broker_execution_unknown into {outcome: 'unknown'}, so a caller that recovers by inspecting gets an exception where it got an answer. The state on the stack is as safe as it was; what moved is what that caller must handle. There is no candidate for this one: whether the Broker's answer or the client's reading should move is a decision of the design.

The rest of the round

What remains besides R7-1 and R7-2. None of it is urgent, and none of it came with this PR:

1. The raw path keeps about 325 KiB per released Session, on main too. It is the path the Hosted harness uses today, and this PR does not touch it: the fix bounds what the provider route keeps, the executor's journal is outside the diff. It deserves an issue of its own (section 1).
2. 16 of 200 mutants survive, 7 of them new. The new ones are edges of the retirement, of the response buffer, of the cursor and of the directory check; none is a rule the real stack was seen to break (section 6).
3. One fault gate that came with main fails on a loaded host, on main as well. DurableLocalRuntimeFaultGateTest, run by itself on a host under load, passes 2 of 3 times on the head, 1 of 3 on main; two builds at once fail every time. The whole profile passed 44 of 44 on its first run on the head (section 3).

Same rig as before: built TypeScript provider → Spring-embedded Broker on MySQL 8.4.7 → bundled workers, with the JVM HTTP proxy between Broker and worker; a trial merge with main be1ebc74d7, made before the merge, as a second arm; main a63157304a after the merge as a third; candidate B on the head and on main; a Linux container for the durable provisioner.

What the head contains (figure 1)

1. What a worker keeps after a release (figure 2)

One worker, Sessions one after another, each used for one write_file of 200 KiB and released, as a Hosted turn does. The memory is the resident set of the worker process, at the first release and 15 s after the last.

The raw rows are the same on this head and on main. That retention is older than this PR and lives in the executor, which the fix does not touch. It matters more than the provider rows did, because the Hosted harness runs on the raw path today and the provider has no production caller yet.

2. What a released Session still answers (figure 3)

The Broker sends none of these to a released Session; the rig did, with the Broker's headers. Through the Broker nothing changes: the execution of a retired Session reads 200 settled/success from the Broker's journal, and a repeated release answers 200.

Retirement shuts the Config of the retired Session down. It does not reach the others on the same worker:

• A Session acquired first, with a call prepared and approved, runs that call after twelve others were released and four retired, and then runs another.
• A Shell command that runs while those twelve come and go finishes with all its output.
• 32 Sessions, six at a time, all run and release; no worker answer is a refusal.

Boot v2 admits one tool turn per Workspace at a time (409 workspace_busy), so the last two ran on boot v1.

warm now refuses a Harness Session id outside the allow-list with 400 runtime_broker_invalid_request, before a worker is started. The previous head answered a retryable 503 runtime_scope_resolution_failed.

3. Regression, the restart and the suites (figure 4)

Regression. On this head, on the trial merge and on main after the merge the probes of rounds 1 to 6 pass except where R7-1 reaches them. E.2 starts a call the worker refused and gets no answer in 30 s, on boot v1 and on boot v2. Checks of round 1 that count the requests which reach the worker in a window find a status request of the call that B.1 started before and that is still polling: C.5 in these runs, on the head, on the trial merge and on main; which check it hits differs from run to run. With candidate B all of them pass, on the head and on main.

Three checks state a rule that 9cb9dc86e8 changed by design, and were rewritten to the new rule: F.1 and F.2 of round 1 and S.exec of round 2 said that the record of a call whose answer was lost stays UNKNOWN. It now settles from the result the worker kept. The logs of the earlier builds in this round carry the old wording.

The restart. As in round 6. With the default provisioner the repeated cancellation answers a retryable 503, acquire answers 503 after 120 s, and that repeats. On Linux, with the durable provisioner, which the head now has from main, acquire adopts the worker in 0.1 to 0.2 s and the cancellation is confirmed: 4 of 4 scenarios, 5/5 each.

A worker killed under a running command. On both rigs the record becomes UNKNOWN, the call the caller waited on answers 409, start sent again is refused, and release answers 503 until the Broker knows the old writer stopped; the storage stays held. The worker is gone, so there is nobody to ask: the new reconciliation changes the code of the answers (R7-2), not the state. The client's message carries the Broker's reason. The answer has no details on either rig, so the terminal ABANDONED answer was not produced on the real stack, and R4-3 rests on its unit test and its mutant (section 6). On Linux the command's own process outlives the worker and finishes its work once; the Broker's refusal to give the storage away fits that.

Suites and CI. The repository suites pass on the head and on the trial merge (figure 4). CI on the head: 22 checks pass, none fails; assign was still queued when this was written. No suite of the repository starts a call the worker refuses and waits for it, which is why R7-1 shows in none of them.

One fault gate depends on the load of the host. DurableLocalRuntimeFaultGateTest came with #12865; this PR changes none of it. While other runs of this round loaded the host, the fault-gate profile failed in that class on two earlier states of the branch (1 and 3 of 44); on the head and on the trial merge it passed 44 of 44 on its first run. Run by itself and repeated, with the load average between 13 and 54 on ten cores, the class passes 2 of 3 times on the head, 1 of 3 on main 12793013c4, and 1 of 4 on the main before #12975; with two builds running it at once it fails every time, on the branch as on main. The failures are the same two assertions throughout: absentWorkerAfterRestartLeavesItsDomainPinned gets runtime_provision_fenced where it expects runtime_broker_runtime_lost, and adoptedWorkerCanCancelItsOriginalActiveCall sees no cancellation reach the worker. One run names the cause, Runtime recovery claim expired.

That answers a question the review bot left open. In CI the job Hosted process fault gates failed once on this head, in this class and with this error (closingAnObserverCannotKillTheSharedWorker, 503 runtime_provision_fenced, Runtime recovery claim expired), and passed when it was run again. The bot would not call it a flake, because 760174073b and ebea694e4e had landed since the last green run. In the runs above the same error, runtime_provision_fenced, fails other tests of the same class on main without either commit, and on the main before #12975. It is a lease that expires on a busy host; neither commit is needed for it. The test CI named did not fail here.

4. The merge with main, and the rules main had meanwhile (figure 5)

main had moved by 38 commits since this branch last merged it. Three files conflicted: two design documents and RuntimeBrokerService.java. There, #12975 had changed the deferred start, which this PR had moved into a branch of its own; the merge carried the two checks of #12975 into that branch by hand. No migration version exists twice.

The part written by hand is right. A deferred start whose payload holds an unpaired surrogate, in a string of the input, in a key of it or in the tool name, answers 400 runtime_payload_invalid on the head as on main, and nothing is sent to the worker. Before the merge all three were sent, with ? in place of the surrogate. The reservation of a refused start stands until the caller cancels it; then the Session is released and its storage is free. The three mutants of those lines (M1 to M3) are killed by the tests #12975 brought.

What merging could not bring. The provider path is this branch's own; rules main gave the raw path do not reach it by merging. 50fb28301e names three and adds them. Each was measured on the merge commit 174f974e3e and on the head:

A surrogate pair passes both paths unchanged and is written as the four bytes of its code point. A Shell call with a directory inside the workspace runs there. A cursor written as 4 is accepted. On the raw path the call with a directory outside the workspace does not run, before and after.

The cursor forms are not something a worker of this repository writes; the proxy of the rig rewrote the worker's answer to produce them.

When the trial merge was made, main was six commits further than the head held (#12987, #12990, #13025, #13021, #13022, #12945). None touches the Broker, the provider or its worker. The trial merge was clean, and on it the probes of all rounds give the results of the head.

5. Lost answers, moved links, a refused start, artifacts (figure 6)

A lost answer settles (Y6). The proxy drops the worker's answer to execute. The command ran once. The start answers 200 executing; a read asks the worker and answers 200 settled/success; start sent again and cancel answer the settled result with nothing sent to the worker; release succeeds and the storage is free. One execute on the wire. Before 9cb9dc86e8 the read, the repeated start and the cancel answered 409, release answered 409 too, and the Session and its storage stayed held.

A link moved after prepare (Y7). The directory of a Shell call is a link to a directory inside the workspace. After prepare, reserve and preflight the link is pointed at the directory of another storage. On 50fb28301e the command ran there. On the head the call settles as an error and does not run.

A refused start (Y8) is R7-1, above. The figure shows it on the commit before, on 9cb9dc86e8, on the head, on main after the merge, and on main with candidate B.

Cancellation reaches the worker. On 9cb9dc86e8 the caller of a refused start had no way out: cancel answered 200 prepared and did nothing, the record stayed UNKNOWN, release answered 409, the storage stayed held. On the head cancel answers 200 settled/not_started, the record is settled, release answers 200. That closes F1 of round 1 on the provider path. On the raw path a tool v2 record keeps its UNKNOWN, as before.

Artifacts. The fitter is a function, so this probe calls it as built, without a stack. With artifacts of 1.5 MiB beside 1,000 characters for the model, 9cb9dc86e8 returned a result of 1,536 KiB that did not fit the limit of 1 MiB, with the model's text replaced by the stub and the hook result dropped. The head drops the artifacts and keeps the text and the hook result whole. Artifacts that fit are kept; a structured display goes before them; a result that already fits is returned unchanged. The fuzz of round 6 passes, 8/8.

6. Mutation matrix (figure 7)

200 mutants ran on this head: 184 killed, 16 survive.

57 are new: L1 to L35 edit lines e94f781523 adds, M1 to M3 the lines the merge wrote by hand, P1 to P9 the lines of 50fb28301e, Q1 to Q6 those of 9cb9dc86e8, R1 and R2 those of 760174073b, S1 and S2 those of ebea694e4e. 50 are killed. What the later commits add is pinned where it matters: a provider operation that is not checked, or checked for prepare only, or for the input only (P1 to P3); a cursor read the old way (P6); a directory outside the workspace, at prepare and before the run, and links not followed (P7, Q2, Q3, Q5); a lost answer that is not reconciled and a cancellation that does not reach the worker (Q1, S1, S2); artifacts that are not dropped (R1). 7 survive:

• L11, a retirement that failed reports nothing. No test makes a retirement fail and looks at what is reported.
• L20, a release retires at most one Session. No test has two released Sessions past the bound at one release.
• L34, the response buffer grows to the needed size only. The bytes are the same; only the number of copies differs.
• L35, the ownership check of a terminal record no longer compares the generation. No test reads a terminal record of another generation through this check.
• P5, a status cursor beyond 2^53 - 1 is accepted. No test sends a cursor beyond the largest safe integer.
• Q4, a relative directory is admitted. Equivalent: core refuses a relative directory before this line is reached.
• Q6, a directory that cannot be resolved is admitted. No test names a directory that does not exist.

No mutant stands for R7-1: there the line is as the commit meant it, and the tests of the commit pin exactly that (state: executing). The case the tests do not have is the worker that answers prepared; the test of candidate B is that case.

Of the 150 earlier mutants, seven edit lines that no longer exist (I9, I23, I24, I25, I26, I28, I2). The other 143 were rerun: 134 killed, 9 survive (H3, H8, I6, K36, K39, N17, T18, T2, T8), the same nine as on e94f781523. Three verdicts changed against round 6, all to killed: K7 and K41, the rule admits letters outside ASCII, and G4, the guard of the round-3 merge.

TypeScript mutants of the provider protocol and its worker ran against the eleven test files that exercise them; every survivor was then confirmed against all twenty. Failures of hosted-harness-session.test.ts are not counted as kills. The TypeScript mutants ran on 760174073b, whose TypeScript sources and tests are the ones of the head; the Java mutants ran on the head. The host was under load while the matrix ran, so every kill was compared with the tests that killed the same mutant on e94f781523.

Status of earlier items

The merge

Merged as 1f1209bc70 on 29 September, 16:37 UTC, by squash onto main 8c914ebe03. The trial merge of this round was made with main be1ebc74d7, two commits earlier; it was clean and ran as a second arm. main is at a63157304a, one commit after the merge (#12955); Y1 to Y8 and the probes of rounds 1 to 6 were run on it as well.

Not covered

Windows; MariaDB; a live-model Hosted Shell turn (the ITs cover the Shell turn with a fake model); the 501 branch of a refused acquire; an ABANDONED record on the real stack; more than eight runtimes alive while retirements are in flight (the author records it); a retirement that hangs; an unpaired surrogate in modification, in the payload of confirm or in bind-history on the real stack (the unit test of 50fb28301e covers them); artifacts from a real tool on the real stack (the probe calls the fitter as built).

Reproduce

Round-7 rig, probes, mutants and logs: pr12868/r7. New probes are s21-r7.mjs (Z1 to Z4), s22-loss-in-flight.mjs, s23-merge-r7.mjs (Y1 to Y8) and fit-artifacts.mjs; s3-retention.mjs and s3b-retention-raw.mjs measure the memory; candidate-B.patch is the candidate.