Skip to content

fix(runtime-broker): Close the deferred W0c-2 findings from #12761 - #12975

Merged
wenshao merged 4 commits into
mainfrom
fix/runtime-broker-w0c2-followups
Sep 29, 2026
Merged

wenshao merged 4 commits into
mainfrom
fix/runtime-broker-w0c2-followups

Conversation

@wenshao

@wenshao wenshao commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator

What this PR does

Closes the seven findings deferred from #12730 (W0c-2 managed-context Broker) and tracked in #12761.

  • Context installation checks the Session record's state. The Broker now refuses, before sending anything, to install a context for a Session record that is releasing, released or failed, or on a binding that has a drain request. It cannot simply require a READY record: the Workspace acquisition path installs the context while the record is still ACQUIRING, so ACQUIRING and READY are both accepted. The interface documentation and the design document now say exactly this instead of "the binding the Session was acquired on".
  • A non-retryable startup failure keeps its answer when the deadline fires. For a managed-context startup, the failure handler now publishes a non-retryable failure before anything else, and the deadline reads it when it fires. If the failure came first, the deadline answers with it, also when it finds the block the handler already recorded (it used to answer 409 runtime_broker_recovery_blocked there). A failure that arrives after the deadline fired, and legacy startup, keep exactly the old answers.
  • The deadline no longer drops a failed block write. When the deadline cannot record or read back the block, the repository failure is kept as a suppressed exception on the answer it gives, as the failure handler already did.
  • Tool names and tool input with an unpaired surrogate are refused. The JSON writer and the JDBC codec turn an unpaired surrogate into ?, so the Worker used to run a different call than the one requested, and ? is a wildcard in a shell command. The Broker now refuses such a tool name, or any key or string anywhere in the tool input: 400 runtime_reference_invalid when an execution is created (before it is admitted), 400 runtime_payload_invalid when a deferred execution starts (for a raw surrogate and for one written as a JSON escape, which passes a check on the text), and an argument error before sending in the HTTP transport, whose check also fails closed on values that are not JSON. Well-formed Unicode, surrogate pairs and nulls are unchanged.
  • Test coverage. Tests now pin the suppressed repository failure on both the failure-handler and deadline paths, lone low surrogates in the reference identity fields and in Runtime Session IDs, and the deadline races above.
  • The MySQL integration test re-runs on the same database. It now uses a per-run prefix, as the shared repository contract already asks.

Why it's needed

These were left open when #12730 merged after three audit rounds. Two of them were decisions: for the block race I kept the original non-retryable error rather than narrowing the documented guarantee, and for tool names and input I chose to refuse unpaired surrogates, since silently executing rm file? instead of the requested command is worse than a refusal.

Reviewer Test Plan

How to verify

  • Run the runtime-broker suite and Checkstyle; all 426 tests should pass.
  • The new tests fail on main: the installation refusals, the deadline keeping a non-retryable answer, the suppressed failure on the deadline path, and every tool name or input refusal.
  • Run the MySQL integration test twice against the same database: on main the second run fails with expected: <[1]> but was: <[2]>; with this change both runs pass.
  • Run the Stage F fault gates and the Hosted MySQL integration tests: the real Broker, Worker and Workspace acquisition path still install contexts and run tool turns.

Evidence (Before & After)

N/A (no user-visible UI). Local results:

  • runtime-broker: 426 tests, 0 failures; Checkstyle clean; Prettier clean on both design documents.
  • Stage F fault gates with the bundled CLI worker: 39/39, including 12 context-installation gates.
  • managed-agent-server against this Broker build: 180 unit tests, plus the Hosted MySQL integration tests (8/8) on MySQL 8.4. Replacing the new state check with "READY only" makes the Hosted Workspace tool-turn test fail at acquisition (503 runtime_session_acquire_failed), which shows that production installs while the record is ACQUIRING.
  • MySQL integration test: two consecutive runs on the same database pass on MySQL 8.4 and MariaDB 10.11 (the CI version); on main the second run fails.
  • A/B on main: seven of the new assertions fail there. A mutation sweep of the new checks (16 mutants, including the two named in the issue: dropping addSuppressed and narrowing the check to high surrogates) was killed in full.

Tested on

OS Status
🍏 macOS ⚠️
🪟 Windows ⚠️
🐧 Linux ✅

Environment (optional)

JDK 25 (module release 21), Maven 3, Node 22 for the fault gates and Hosted integration tests, Docker MySQL 8.4 and MariaDB 10.11.18.

Risk & Scope

  • Main risk or tradeoff: a tool call whose arguments hold an unpaired surrogate is now refused. On the deferred path the TypeScript client treats any 400 from start as a lost reply, as it already does for other start refusals such as an oversized payload, so it polls until its window ends and the Hosted session becomes recovery-blocked. Before this change the call ran with ? in place of the surrogate. Handling that refusal on the client is left to a follow-up, listed in the audit comment.
  • Not validated / out of scope: macOS and Windows (Java-only change, no platform-specific code). Findings from the final audit round and pre-existing issues found along the way are listed in a separate comment.
  • Breaking changes / migration notes: a raw unpaired surrogate in a deferred payload now answers 400 runtime_payload_invalid instead of 400 runtime_broker_invalid_request. Nothing else changes for valid requests.

Design document updated in both languages: English · 中文

Linked Issues

Closes #12761

中文说明

这个 PR 做了什么

处理 #12730(W0c-2 managed-context Broker)推迟、并由 #12761 跟踪的七条发现。

  • 安装 context 时检查 Session 记录的状态。 对于处于 releasing、released 或 failed 的 Session 记录,或者已请求 drain 的 binding,Broker 现在在发送任何内容之前就拒绝安装。不能简单地要求记录为 READY:Workspace 的 acquire 路径是在记录仍为 ACQUIRING 时安装 context 的,所以 ACQUIRING 和 READY 都被接受。接口文档和设计文档也改为准确描述这一点,不再写“Session 被 acquire 的那个 binding”。
  • 期限触发时,不可重试的启动失败保留自己的应答。 对 managed-context 启动,失败处理现在第一步就发布不可重试的失败,期限在触发时读取它。如果失败先到,期限就用这个失败作答,包括它发现失败处理已经记录了阻塞的情况(以前在这种情况下回答 409 runtime_broker_recovery_blocked)。期限触发之后才到达的失败,以及 legacy 启动,应答与以前完全相同。
  • 期限不再丢弃写入阻塞失败的异常。 期限无法记录或回读阻塞时,仓库异常作为 suppressed 异常保留在它给出的应答上,与失败处理原有的做法一致。
  • 拒绝带有未配对代理项的工具名和工具输入。 JSON 写入器和 JDBC codec 会把未配对代理项变成 ?,因此 Worker 执行的调用与请求的不同,而 ? 在 shell 命令里是通配符。Broker 现在拒绝这样的工具名,以及工具输入中任意位置的键或字符串:创建 execution 时返回 400 runtime_reference_invalid(在准入之前);启动 deferred execution 时返回 400 runtime_payload_invalid(包括原始代理项,以及写成 JSON 转义、能通过文本检查的代理项);HTTP transport 在发送前抛出参数错误,其检查对非 JSON 的值也一律拒绝。格式正确的 Unicode、代理对和 null 不受影响。
  • 测试覆盖。 测试现在固定了失败处理路径和期限路径上的 suppressed 仓库异常、引用身份字段和 Runtime Session ID 中的单独低位代理项,以及上述期限竞态。
  • MySQL 集成测试可以在同一个库上重跑。 它现在按共享仓库契约的要求,每次运行使用独立的前缀。

为什么需要

这些问题在 #12730 经过三轮审计合入时被留下。其中两条需要决策:对于阻塞竞态,我选择保留原来的不可重试错误,而不是收窄文档中的承诺;对于工具名和输入,我选择拒绝未配对代理项,因为静默执行 rm file? 而不是所请求的命令,比拒绝更糟。

评审测试计划

如何验证

  • 运行 runtime-broker 测试套件和 Checkstyle,426 个测试应全部通过。
  • 新测试在 main 上会失败:安装拒绝、期限保留不可重试应答、期限路径上的 suppressed 异常,以及每一种工具名或输入拒绝。
  • 对同一个库连续运行两次 MySQL 集成测试:在 main 上第二次运行失败,报 expected: <[1]> but was: <[2]>;本改动下两次都通过。
  • 运行 Stage F fault gates 和 Hosted MySQL 集成测试:真实的 Broker、Worker 和 Workspace acquire 路径仍能安装 context 并执行工具回合。

证据(前后对比)

不适用(无用户可见 UI)。本地结果:

  • runtime-broker:426 个测试,0 失败;Checkstyle 通过;两份设计文档 Prettier 通过。
  • 使用打包 CLI worker 的 Stage F fault gates:39/39,其中包括 12 个 context 安装门。
  • 基于本次 Broker 构建的 managed-agent-server:180 个单元测试,以及 MySQL 8.4 上的 Hosted MySQL 集成测试(8/8)。把新的状态检查换成“只接受 READY”后,Hosted Workspace 工具回合测试在 acquire 处失败(503 runtime_session_acquire_failed),说明生产环境确实在记录为 ACQUIRING 时安装。
  • MySQL 集成测试:在 MySQL 8.4 和 MariaDB 10.11(CI 使用的版本)上,同一个库连续运行两次都通过;在 main 上第二次失败。
  • 在 main 上做 A/B:新断言中有七个在那里失败。对新增检查的变异扫描(16 个变异体,包括 issue 点名的两个:删除 addSuppressed、把检查收窄到高位代理项)全部被杀死。

测试平台

OS 状态
🍏 macOS ⚠️
🪟 Windows ⚠️
🐧 Linux ✅

环境(可选)

JDK 25(模块 release 21)、Maven 3;fault gates 和 Hosted 集成测试使用 Node 22;Docker 中的 MySQL 8.4 和 MariaDB 10.11.18。

风险与范围

  • 主要风险或取舍:参数中含未配对代理项的工具调用现在会被拒绝。在 deferred 路径上,TypeScript 客户端把 start 返回的任何 400 当作丢失的应答(对超大 payload 等其他 start 拒绝也是如此),于是会轮询到窗口结束,Hosted 会话随后进入 recovery-blocked。改动之前,这样的调用会以 ? 代替代理项被执行。在客户端处理这个拒绝留作后续工作,已列在审计评论中。
  • 未验证 / 不在范围内:macOS 和 Windows(仅 Java 改动,没有平台相关代码)。最后一轮审计的发现和过程中发现的既有问题列在单独的评论中。
  • 破坏性变更 / 迁移说明:deferred payload 中的原始未配对代理项现在返回 400 runtime_payload_invalid,而不是 400 runtime_broker_invalid_request。有效请求的行为没有变化。

设计文档已同步更新两种语言:English · 中文

关联 Issue

Closes #12761

- Context installation refuses a Session record that is RELEASING,
  RELEASED or FAILED, and a binding with a drain request, before
  sending. Acquisition installs while the record is still ACQUIRING,
  so the check accepts ACQUIRING and READY rather than READY alone.
- When a managed-context startup fails with a non-retryable error and
  the deadline fires before the call answers, the deadline answers
  that error, including when it finds the block the failure handler
  recorded. It used to answer 409 runtime_broker_recovery_blocked. A
  failure published after the deadline fired, and legacy startup, keep
  the deadline's own answer.
- A block write or read that fails on the deadline path is kept as a
  suppressed exception on the deadline's answer instead of dropped.
- A tool name, or any key or string in the tool input, holding an
  unpaired surrogate is refused instead of reaching the Worker as '?':
  runtime_reference_invalid on create, runtime_payload_invalid on a
  deferred start (raw or escaped), and before sending in the HTTP
  transport, whose check fails closed on values that are not JSON.
- Tests pin addSuppressed on both paths, lone low surrogates in the
  identity fields and Runtime Session IDs, and the deadline races.
- JdbcRuntimeBrokerMySqlIT uses a per-run prefix, so it re-runs on the
  same database.
@github-actions github-actions Bot added the review/self-reported The linked issue was opened by the PR author (self-reported) label Sep 29, 2026
@wenshao

wenshao commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator Author

Pre-commit audit summary

Three rounds, each with two independent auditors (one undirected, one adversarial against the stated claims), on frozen snapshots. Rounds 1 and 2 fixed every real finding; round 3 acted on Critical findings only and found none, so the commit is the round-3 snapshot byte for byte (tree ec0bc8ac5f).

  • Round 1 (snapshot aa1a2c92f5): 1 Critical — the reflowed design-document table failed the CI Prettier check; fixed. Also fixed: the deadline read the published failure when it finished rather than when it fired, so a failure that arrived after the deadline changed its answer (for legacy durable startup it answered a non-retryable error while a retry would re-provision). The deadline now reads it when it fires, and legacy startup keeps its old answer. The well-formedness check now fails closed on values that are not JSON (arrays and sets reached the writer through the public transport), the handler publishes the failure before taking the claim, a raw unpaired surrogate in a deferred payload uses the coded refusal, and the service test pins which check refuses identity fields.
  • Round 2 (snapshot 46388f585b): no Critical. Fixed: a design-document sentence about where the suppressed repository failure lands, and the legacy guard, which no test pinned.
  • Round 3 (snapshot f60b44b880): no Critical; all claims held. Deferred below.

Deferred

  1. The TypeScript client treats a 400 from start as a lost reply. When a model's tool arguments hold an unpaired surrogate, the client writes it as a JSON escape, the Broker refuses the deferred start with 400 runtime_payload_invalid, and the client polls until its window ends (120 s, or the shell timeout plus 60 s), cancels, and the Hosted session becomes recovery-blocked. The execution stays PREPARED meanwhile, so the Runtime Session cannot be released. Every existing start refusal (for example an oversized payload) takes the same path. Before this change the call ran with ? in place of the surrogate. Options: refuse such arguments on the client before prepare through the existing validation error, or treat a 400 runtime_payload_invalid from start as a definite not-started refusal.
  2. The handler's publish-before-claim ordering has no deterministic test. Moving the publish after taking the claim still passes; it matters only when a renewal holds the claim across a database round trip while the deadline fires.
  3. Diagnosability of the suppressed failure. The HTTP layer serializes only the code and message, and the module has no logging, so an operator sees the kept repository failure only if the embedding service logs the exception. The issue suggested logging; this change keeps the failure on the answer, as the failure-handler path already did.
  4. The well-formedness check accepts any Number, including NaN and custom subclasses the writer serializes as objects. Only reachable by calling the public transport directly; service callers produce standard JSON scalars.
  5. A drain refusal during installation happens after Workspace storage ownership is claimed. Unreachable today (nothing sets a drain request in production); revisit when drain lands.
  6. Design-document wording: section 3 still says installation is not invoked by acquire, which the Workspace acquisition path (W0c-3) now does.
  7. Test hygiene: a static exception instance is shared across the race tests; the non-Broker-exception branch of the suppressed-failure wording has no test.

Pre-existing, not introduced here

中文说明

提交前审计汇总

共三轮,每轮由两名独立审计员进行(一名无方向,一名针对所述声明做反向审计),在冻结的快照上完成。第 1、2 轮修复了所有真实问题;第 3 轮只处理 Critical,且没有发现 Critical,因此提交内容与第 3 轮快照逐字节一致(tree ec0bc8ac5f)。

  • 第 1 轮(快照 aa1a2c92f5):1 条 Critical——重新排版后的设计文档表格未通过 CI 的 Prettier 检查;已修复。另外已修复:期限在结束时而不是触发时读取已发布的失败,导致期限之后才到达的失败也会改变它的应答(legacy durable 启动在重试会重新 provision 的情况下回答了不可重试错误)。现在期限在触发时读取,legacy 启动保持原有应答。格式检查现在对非 JSON 值一律拒绝(数组和 Set 曾经可以经公开 transport 到达写入器);失败处理在获取 claim 之前发布失败;deferred payload 中的原始未配对代理项改用带错误码的拒绝;服务层测试固定了由哪一道检查拒绝身份字段。
  • 第 2 轮(快照 46388f585b):无 Critical。已修复:设计文档中关于 suppressed 仓库异常挂在何处的一句话,以及没有测试固定的 legacy 守卫。
  • 第 3 轮(快照 f60b44b880):无 Critical,所有声明均成立。推迟项如下。

推迟项

  1. TypeScript 客户端把 start 返回的 400 当作丢失的应答。 当模型的工具参数含未配对代理项时,客户端将其写成 JSON 转义,Broker 以 400 runtime_payload_invalid 拒绝 deferred start,客户端轮询到窗口结束(120 秒,shell 为超时加 60 秒)后取消,Hosted 会话进入 recovery-blocked。期间 execution 停在 PREPARED,Runtime Session 无法释放。现有的所有 start 拒绝(例如超大 payload)都走同一条路径。改动之前,这样的调用会以 ? 代替代理项被执行。可选方案:客户端在 prepare 之前通过已有的校验错误拒绝这类参数,或者把 start 返回的 400 runtime_payload_invalid 当作确定的“未启动”拒绝。
  2. 失败处理“在获取 claim 之前发布”的顺序没有确定性测试。 把发布挪到获取 claim 之后,测试仍然通过;只有当续约在一次数据库往返期间持有 claim、同时期限触发时,这个顺序才有影响。
  3. suppressed 异常的可诊断性。 HTTP 层只序列化错误码和消息,模块本身没有日志,因此只有嵌入方记录该异常时,运维才能看到保留下来的仓库异常。issue 建议记日志;本改动选择把异常保留在应答上,与失败处理路径原有的做法一致。
  4. 格式检查接受任意 Number,包括 NaN 以及会被写入器序列化成对象的自定义子类。只有直接调用公开 transport 才能触发;服务层调用方只会产生标准 JSON 标量。
  5. 安装时的 drain 拒绝发生在 Workspace 存储所有权被认领之后。 目前不可达(生产代码中没有任何地方设置 drain 请求);实现 drain 时需要重新审视。
  6. 设计文档措辞: 第 3 节仍写着安装不会由 acquire 调用,而 Workspace acquire 路径(W0c-3)现在正是这样做的。
  7. 测试卫生: 竞态测试共享一个静态异常实例;suppressed 措辞中“非 Broker 异常”这一分支没有测试。

既有问题(不是本改动引入的)

@wenshao

wenshao commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator Author

Real-environment verification — a2658844

Verdict: ready to merge. Every claim in the description reproduces on real processes (real Worker, MySQL 8.4 / MariaDB 10.11, the Spring server with its embedded Broker, and the packaged Hosted Harness), and the new tests fail on the merge base. One item needs a maintainer decision, but it does not block this Java change: what the unpaired-surrogate refusal does to a Hosted turn. I measured that below and tested a small client-side candidate for audit item 1.

Setup

  • Arms: main = origin/main 1b69629; PR = a local merge of a2658844 with that main. That merge is 4 commits newer than CI's merge ref 8c125e6, which was built on b1eb94da and so does not include fix(managed-agent): Close the post-merge review of event replay #12968. The test A/B uses the merge base 99adce25. The PR is Java-only, so both arms run the same dist/cli.js for the Harness and the Worker. Only the Broker jar and classes differ.
  • Hosted real stack: the Spring server jar (Session Store and embedded Runtime Broker), MySQL 8.4.11, the packaged Hosted Harness, real local-process Workers, and a fake OpenAI server that emits the tool call. I captured the Broker→Worker hop with a forward proxy (JVM -Dhttp.proxyHost, empty -Dhttp.nonProxyHosts).
  • Broker-level probe: Rig12975.java runs in the Broker package against each arm's own target/classes, with a real managed-context Worker and MySQL. I injected faults with MySQL triggers on the RECOVERY_BLOCKED write.
  • Environment: macOS arm64, JDK 21 for Spring and the probe, Maven on JDK 26, Node 22.23.2. MySQL 8.4.11 and MariaDB 10.11.18 ran in Docker. Unrelated jobs kept the machine load at 15–45.

Results

Claim What I ran Result
Context installation checks the Session record's state installContext for each record state on a READY managed-context binding, counting the Worker's /v3/context requests main installs for RELEASING, RELEASED, FAILED and for a drain-requested binding (the Worker receives each one). PR refuses those 4 before sending (+0 requests) and still installs ACQUIRING and READY. A real Hosted turn and HostedWorkspaceToolTurnIT 6/6 still acquire on the PR build. ✅
A non-retryable failure keeps its answer when the deadline fires A real Worker booted with another storage ID (non-retryable runtime_provision_failed), a 1 s deadline, and the block write stalled 2 s by a trigger. 6 trials per arm PR returns the original non-retryable failure 6/6 and records the block. main did not return 409 in 6/6 trials: the failure handler answers one DB round trip before the deadline, so this race is hard to hit on a real stack. The new unit test forces the race and fails on main. ✅ (pinned at unit level)
The deadline keeps a failed block write A hung Worker, a 4 s deadline, and a trigger that makes the RECOVERY_BLOCKED write fail with SIGNAL main: 503 provision_timeout with suppressed=[]. PR: the same answer with suppressed=[IllegalStateException(SQLException: rig: recovery block write refused)]. ✅
Tool names and input with an unpaired surrogate are refused The Hosted stack and direct Broker API calls (figures 1 and 2) Confirmed. /executions answers 400 runtime_reference_invalid with 0 rows admitted. Deferred :start answers 400 runtime_payload_invalid for both the escaped and the raw form. A raw surrogate on main answered runtime_broker_invalid_request, which is the documented code change. Controls with emoji and CJK are unchanged. ✅
The new tests fail on main The PR's test files on 99adce25 6 fail, plus 1 new BrokerValues helper test that does not compile there. That matches the "seven" in the description. ✅
The MySQL IT re-runs JdbcRuntimeBrokerMySqlIT twice on one database main: the second run fails expected: <[1]> but was: <[2]> on MySQL 8.4.11 and on MariaDB 10.11.18. PR: 4/4 runs pass. ✅
Suites runtime-broker; managed-agent-server on the merge runtime-broker: 426 run, 0 failures, Checkstyle 0. On the merge: 182 unit tests, ManagedAgentMySqlIT 14/14 (MariaDB), Hosted IT 9/9 (MySQL 8.4), Checkstyle 0. ✅
Stage F fault gates CI, plus a local A/B of the failing class CI (Linux): 39/39. Locally: 36/39 under load. DurableLocalRuntimeFaultGateTest failed the same way on main (runtime_provision_fenced from a claim expiry; main failed 1 of 5 runs, PR 2 of 6), so it is a load flake, not caused by this PR.
Mutation 20 mutants of the production changes, run against the 5 affected test classes 19 killed. The survivor, M17 (publish the failure after taking the claim), is the gap you already listed as audit item 2.

Hosted tool call with an unpaired surrogate, real stack A/B

Decision item: the surrogate refusal on a Hosted turn (not blocking)

On the real Hosted stack, the refusal changes different things on the two tool paths:

  • Hosted Shell (v3). On main, the Broker did send rm victim-?.txt to the Worker. The Worker refused it because the recomputed inputDigest no longer matched (409 managed_runtime_identity_conflict), so nothing ran. The session was recovery-blocked within 0.9 s. With the PR the Worker receives nothing, and the turn ends after the client's polling window: 65.8 s with timeout: 5000, 180 s with the default timeout. The end state is the same. On this path the PR does not prevent a wrong execution that used to happen. It moves the refusal into the Broker (so the execution settles as not_started, not UNKNOWN), and the user waits longer for the failure.
  • Hosted file tools (v2, no digest check). Here main really did run a different call. Given edit with old_string: "a\ud800b", it replaced a?b in note.txt, a string the model never mentioned. Given write_file, it wrote half emoji: ? end. The PR stops both. The cost: a turn that finished in 0.9 s now hangs for 120.7 s (1277–1638 status polls). The client then cancels, the execution settles as cancelled, and the Hosted session stays recovery-blocked with its Workspace lease held. The lease table has no expiry column, and those leases were still held an hour later (the same class as the open Hosted Shell: recover a Workspace lease after incomplete pipe capture #12904). On main only the Shell case left a lease held.
  • This makes audit item 1, the TS client, the user-visible half of this change. Triage noted it is untracked. A small Harness-side check before prepare removes the whole cost: cand-client-refusal.diff (+18 lines in hosted-workspace-tool-turn.ts, using the existing validationError path). I rebuilt the bundle with it and ran it against this PR's Broker. All three calls end in 0.6–0.8 s with turn_complete, the model receives a correctable error, and no prepare reaches the Broker. There is no recovery block and no held lease, and the emoji control still succeeds.
  • Suggestion: merge this, then open the client follow-up (or land the candidate) before Hosted file tools get real traffic. One small correction to the earlier review: the execution stays PREPARED only during the polling window. What persists is the recovery-blocked session and the held Workspace lease.

Broker HTTP API and context installation A/B

Other notes (non-blocking)

  • The Hosted deferred client (hosted-workspace-broker.ts) is the only production client that sends a tool name and input to the Broker. Nothing in production calls /executions with a tool name and input, so the create-time check guards the public Broker API. On main that API admitted the calls and the JDBC codec stored the tool name as read_file?.
  • Removing if (cause != null) in blockRecoveryQuietly is safe. All three call sites pass a non-null cause: the deadline passes answer, and the handler passes unwrap(error) inside error != null.

Tests, real MySQL faults and mutation sweep

Evidence, probe scripts and logs (publisher tokens redacted) are at wenshao/qwen-code@5e890ad0/pr12975.

中文版

真实环境验证 —— a2658844

结论:可以合并。 PR 描述中的每一条主张都在真实进程上复现成立:真实 Worker、MySQL 8.4 / MariaDB 10.11、内嵌 Broker 的 Spring 服务,以及打包后的 Hosted Harness。新测试在合并基点上确实失败。有一项需要维护者决策,但不阻塞这个 Java 改动:拒绝未配对代理项后,Hosted 轮次会受到什么影响。下文给出了实测数据,并针对审计推迟项 1 实测了一个很小的客户端候选补丁。

环境

  • 对照臂: main = origin/main 1b69629;PR = a2658844 与该 main 的本地合并。这个合并比 CI 的合并引用 8c125e6 新 4 个提交;8c125e6 建在 b1eb94da 上,不含 fix(managed-agent): Close the post-merge review of event replay #12968。测试 A/B 用合并基点 99adce25。PR 只改 Java,所以两臂的 Harness 和 Worker 都用同一份 dist/cli.js,只有 Broker 的 jar 和 classes 不同。
  • Hosted 真实栈: Spring 服务 jar(Session Store 和内嵌 Runtime Broker)、MySQL 8.4.11、打包的 Hosted Harness、真实本地 Worker 进程,以及发出工具调用的假 OpenAI 服务。Broker→Worker 这一跳用正向代理抓包(JVM -Dhttp.proxyHost,-Dhttp.nonProxyHosts 置空)。
  • Broker 级探针: Rig12975.java 放在 Broker 包内,用各臂自己的 target/classes,对接真实 managed-context Worker 和 MySQL。故障由 RECOVERY_BLOCKED 写入上的 MySQL 触发器注入。
  • 运行环境: macOS arm64,Spring 和探针用 JDK 21,Maven 用 JDK 26,Node 22.23.2。MySQL 8.4.11 和 MariaDB 10.11.18 跑在 Docker 中。机器同时被无关任务压着,负载 15–45。

结果

主张 做法 结果
安装 context 时检查 Session 记录状态 在 READY 的 managed-context binding 上,对每种记录状态调用 installContext,并统计 Worker 收到的 /v3/context 请求数 main 对 RELEASING、RELEASED、FAILED 以及已请求 drain 的 binding 都会安装(Worker 都收到了请求)。PR 在发送前拒绝这 4 种(请求 +0),ACQUIRING 和 READY 照常安装。在 PR 构建上,真实 Hosted 轮次和 HostedWorkspaceToolTurnIT 6/6 仍能正常 acquire。✅
期限触发时,不可重试的失败保留原应答 让真实 Worker 以另一个 storage ID 启动(得到不可重试的 runtime_provision_failed),期限 1 s,用触发器让阻塞写入卡 2 s。每臂 6 次 PR 6/6 返回原始的不可重试错误,并记录了阻塞。main 6 次里没有一次返回 409:失败处理比期限少一次数据库往返,先作答,所以这个竞态在真实栈上很难撞中。新单测能强制出这个竞态,并且在 main 上失败。✅(由单测固定)
期限保留写入失败的阻塞异常 Worker 挂起,期限 4 s,用触发器让 RECOVERY_BLOCKED 写入以 SIGNAL 失败 main:503 provision_timeout,suppressed=[]。PR:同样的应答,带 suppressed=[IllegalStateException(SQLException: rig: recovery block write refused)]。✅
拒绝含未配对代理项的工具名和输入 Hosted 栈和直接调用 Broker API(图 1、图 2) 成立。/executions 返回 400 runtime_reference_invalid,0 行落库。deferred :start 对转义和原始两种形态都返回 400 runtime_payload_invalid。原始代理项在 main 上返回 runtime_broker_invalid_request,这正是文档中说明的错误码变化。含 emoji 和中文的对照组不变。✅
新测试在 main 上失败 把 PR 的测试文件放到 99adce25 上运行 6 个失败,另有 1 个新增的 BrokerValues 辅助方法测试在该版本上无法编译,与描述中的"七条"一致。✅
MySQL IT 可以在同一个库上重跑 JdbcRuntimeBrokerMySqlIT 在同一个库上连跑两次 main:第二次在 MySQL 8.4.11 和 MariaDB 10.11.18 上都失败,expected: <[1]> but was: <[2]>。PR:4/4 次全部通过。✅
测试套件 runtime-broker;合并树上的 managed-agent-server runtime-broker:426 个测试,0 失败,Checkstyle 0。合并树上:182 个单测、ManagedAgentMySqlIT 14/14(MariaDB)、Hosted IT 9/9(MySQL 8.4)、Checkstyle 0。✅
Stage F fault gates CI,加上对失败类的本地 A/B CI(Linux)39/39。本地高负载下 36/39。DurableLocalRuntimeFaultGateTest 在 main 上以同样的方式失败(runtime_provision_fenced,claim 过期;main 5 次中失败 1 次,PR 6 次中失败 2 次),属于负载导致的抖动,不是本 PR 引起的。
变异测试 对生产代码改动造 20 个变异体,跑受影响的 5 个测试类 杀死 19 个。唯一存活的 M17(先占 claim 再发布失败),正是你在审计评论中列出的推迟项 2。

需要决策:代理项拒绝对 Hosted 轮次的影响(不阻塞)

在真实 Hosted 栈上,这个拒绝对两条工具路径的影响不同:

  • Hosted Shell(v3)。 在 main 上,Broker 确实把 rm victim-?.txt 发给了 Worker。但 Worker 重算的 inputDigest 对不上,以 409 managed_runtime_identity_conflict 拒绝,所以什么都没执行,会话在 0.9 s 内进入 recovery-blocked。在 PR 上 Worker 什么都收不到,轮次要等客户端的轮询窗口结束才结束:timeout: 5000 时 65.8 s,默认 timeout 时 180 s。最终状态相同。所以在这条路径上,PR 并没有阻止一个以前真的会发生的错误执行。它把拒绝提前到 Broker(执行结算为 not_started,而不是 UNKNOWN),用户则要等更久才看到失败。
  • Hosted 文件工具(v2,没有 digest 校验)。 这里 main 确实执行了一个不同的调用。edit 的 old_string 是 "a\ud800b",结果它把 note.txt 里的 a?b 替换掉了,而模型从未提到过这个字符串。write_file 写出了 half emoji: ? end。PR 阻止了这两种情况。代价是:原本 0.9 s 就结束的轮次,现在要挂 120.7 s(期间状态轮询 1277–1638 次)。之后客户端取消,执行结算为 cancelled,但 Hosted 会话一直处于 recovery-blocked,并且占着 Workspace 租约。租约表没有过期列,这些租约一小时后仍被占用(与仍开放的 Hosted Shell: recover a Workspace lease after incomplete pipe capture #12904 同类)。在 main 上只有 Shell 那个用例会留下被占的租约。
  • 因此,审计推迟项 1(TS 客户端)是这项改动中用户能直接感知的一半。triage 已指出目前没有跟踪项。在 prepare 之前加一个很小的 Harness 端检查,就能消除全部代价:cand-client-refusal.diff(hosted-workspace-tool-turn.ts 中 +18 行,复用已有的 validationError 路径)。我用它重新打包,并对接本 PR 的 Broker 实测:三种调用都在 0.6–0.8 s 内以 turn_complete 结束,模型收到可以纠正的错误,没有 prepare 到达 Broker。没有 recovery-block,也没有被占的租约,emoji 对照组照常成功。
  • 建议: 合并本 PR,然后开一个客户端跟进项(或直接落地这个候选补丁),在 Hosted 文件工具有真实流量之前完成。对之前评审的一处小更正:执行只在轮询窗口内停在 PREPARED;持续存在的是 recovery-blocked 的会话和被占的 Workspace 租约。

其他(不阻塞)

  • 生产代码中,只有 Hosted deferred 客户端(hosted-workspace-broker.ts)会把工具名和输入发给 Broker。生产代码里没有哪里带着工具名和输入调用 /executions,所以创建时的检查保护的是 Broker 的公开 API。在 main 上,这个 API 会准入这些调用,JDBC codec 还把工具名存成了 read_file?。
  • 去掉 blockRecoveryQuietly 中的 if (cause != null) 是安全的。三个调用点传入的 cause 都不为空:期限传的是 answer,失败处理在 error != null 分支内传 unwrap(error)。

证据、探针脚本和日志(publisher token 已脱敏)见 wenshao/qwen-code@5e890ad0/pr12975。

@wenshao

wenshao commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator Author

@qwen-code /takeover

@qwen-code-dev-bot qwen-code-dev-bot added the autofix/takeover Summon the autofix loop to manage this PR (remove to release; needs triage+) label Sep 29, 2026
@qwen-code-dev-bot

Copy link
Copy Markdown
Collaborator

🤝 Takeover engaged: the autofix loop now manages this PR — it will address new review feedback and resolve base conflicts until the label is removed or the round cap is reached. Remove the autofix/takeover label (or comment @qwen-code /takeover stop) to release.

中文说明

🤝 已接管:autofix 循环现在管理此 PR —— 将持续处理新的评审反馈与 base 冲突,直到移除标签或达到轮次上限。移除 autofix/takeover 标签(或评论 @qwen-code /takeover stop)即可释放。

@qwen-code-dev-bot

qwen-code-dev-bot commented Sep 29, 2026 •

Copy link
Copy Markdown
Collaborator

✅ AutoFix round 1 finished — view run. See this round's report below.

中文说明

✅ AutoFix 第 1 轮已完成 —— 查看运行。本轮报告见下方。

… from one unwrapped cause (#12975)

The provisioning failure handler computed unwrap(error) and the
instanceof/isRetryable classification twice — once for the
non-retryable publish and again for the retryable flag. Hoisting the
unwrap keeps a single statement of the rule so a future change cannot
make the two diverge undetected. unwrap(null) returns null, so the
success path is unchanged.

Co-authored-by: Qwen-Coder <[email protected]>
@qwen-code-dev-bot

Copy link
Copy Markdown
Collaborator

🤖 Addressed the latest review feedback (round 1/100). What changed, and what I pushed back on: · 已处理最新评审反馈(第 1/100 轮)。改动内容与我反驳保留之处如下:

Address-review round — PR #12975

What this round changed

One inline suggestion was in scope this round, and it is implemented in a single commit (3fd42b4d87):

  • [rc:4130196860] R1-4 — duplicated retryability classification (Suggestion): addressed. The publish added by this PR re-derived the retryability classification that the same handler body computes seven lines below: a second unwrap(error) at the publish guard plus the same instanceof RuntimeBrokerException / isRetryable() test in the opposite polarity. The two expressions are equivalent today, but nothing would detect a future divergence (the reviewer's mutant of only the handler-side derivation passed all 114 tests of the two owning classes). The fix hoists Throwable cause = unwrap(error); to the top of the .handle(...) body so the publish guard and the handler's retryable flag derive from one statement of the rule. unwrap(null) returns null (verified in the source of unwrap), so the success path is unchanged, and the change is +2/−2 lines in packages/sdk-java/runtime-broker/src/main/java/com/alibaba/qwen/code/runtimebroker/RuntimeBrokerService.java. After the hoist the divergence hazard is eliminated structurally — there is only one derivation left — so no new test is needed to pin it, and no new guard or branch was added (no mutation-probe witness applies to a pure deduplication).

Other feedback, and why it required no code change

  • [rv:5348392738] review body (COMMENTED): the review disclosed its own coverage gaps and repeated no new actionable findings beyond R1-4 above. The two Suggestion-level findings it confirmed (R1-3 well-formedness Number acceptance; R1-6 publish-before-claim ordering has no deterministic test) are the author's own already-recorded deferrals (audit items 4 and 2 in [ic:5883139520]) and are not repeated here.
  • [ic:5883139520] pre-commit audit summary (maintainer's own deferral record): items 2–7 remain as the author deliberately deferred them; they are in-footprint design/test-hygiene choices or already-reported items, and this round does not override the maintainer's own deferral decisions on their PR. Item 1 is real, verified, and out of this Java-only PR's footprint — see the deferral below.
  • [ic:5884214191] real-environment verification: verdict "ready to merge"; its one decision item (what the unpaired-surrogate refusal costs a Hosted turn) is explicitly non-blocking, with the recommendation to merge this PR and then open the client follow-up. No code action in this PR.
  • Deferred to follow-up (recorded in deferred-findings.json): audit item 1 / the verification comment's decision item — the TypeScript Hosted client treats the Broker's 400 runtime_payload_invalid refusal of a deferred start as a lost reply, polling up to 180 s and leaving the session recovery-blocked with its Workspace lease held. The fix belongs in packages/cli/src/serve/hosted-workspace-tool-turn.ts (outside this PR's footprint), and the maintainer asked for it as a follow-up after merge; a tested +18-line candidate using the existing validationError path is linked from the verification comment.
  • Pre-existing items listed in the audit comment (reconciliation-deadline race class, unbounded-surrogate idempotency keys, the test(runtime-broker): Add Stage F fault gates for W0c context installation #12804 Worker-refusal note) are not introduced by this PR and are unchanged by this round.

Conflict notes

--conflict false; no merge was performed and none was needed.

Verification

  • npm run build — passed (exit 0; run with COREPACK_HOME=/tmp/corepack because the runner's default corepack cache under $HOME/.cache is not writable — environmental only).
  • npm run typecheck — passed (exit 0, all workspaces plus integration tsconfig).
  • npm run lint — passed (exit 0, eslint over .ts/.tsx).
  • Focused Vitest — not applicable: the change touches only a .java file; no npm workspace package was modified (the npm toolchain does not parse Java sources, confirmed from the lint/format/build configuration).
  • mvn --batch-mode --no-transfer-progress clean test and mvn checkstyle:check for packages/sdk-java/runtime-broker (the checks sdk-java.yml runs for this module) — not runnable on this runner: no JDK or Maven is installed (command -v java/mvn empty; no JVM under the usual install roots). This is an exact CI check unavailable on the current runner, not a failed runnable check. The edit was instead verified by close inspection: it is a semantics-preserving hoist of an existing statement — both expressions now read the same local cause, and unwrap(null) == null keeps the success path identical. Brace/paren balance and single-occurrence replacement were checked programmatically, and the final git diff (+2/−2, one hunk) was re-read. The sdk-java CI workflow remains the final verification gate for the Java module.
  • Mutation probe — not applicable: the round adds no new guard, branch, or behavior; it removes a duplicate computation (the reviewer demonstrated the equivalence algebraically and by a mutant).
中文说明

评审处理轮次 —— PR #12975

本轮改动

本轮只有一条行内建议在处理范围内,已在单个提交(3fd42b4d87)中实现:

  • [rc:4130196860] R1-4 —— 可重试性分类被重复推导(Suggestion):已处理。 本 PR 新增的发布逻辑重新推导了同一处理体在下方七行处已经算出的可重试性分类:发布判断处第二次调用 unwrap(error),加上同一个 instanceof RuntimeBrokerException / isRetryable() 判断(极性相反)。两个表达式今天等价,但一旦未来发生分歧没有任何测试能发现(评审者实测:只改动处理体一侧推导的变异体,在拥有该逻辑的两个测试类的全部 114 个测试下依然全绿)。修复方式是把 Throwable cause = unwrap(error); 提升到 .handle(...) 处理体开头,让发布判断与处理体的 retryable 标志都从同一条规则推导。已核对 unwrap 源码:unwrap(null) 返回 null,因此成功路径行为不变;改动为 packages/sdk-java/runtime-broker/src/main/java/com/alibaba/qwen/code/runtimebroker/RuntimeBrokerService.java 内 +2/−2 行。提升之后,分歧隐患在结构上被消除——只剩一份推导——因此无需新增测试来固定它,也没有新增任何守卫或分支(纯去重不适用变异探针要求)。

其他反馈及无需改动代码的原因

  • [rv:5348392738] 评审正文(COMMENTED): 评审自行披露了覆盖缺口,除上述 R1-4 外没有新的可执行发现。它确认的两条 Suggestion 级发现(R1-3 格式检查接受任意 Number;R1-6 “先发布后取 claim”的顺序没有确定性测试)正是作者在 [ic:5883139520] 中已记录的推迟项(审计第 4、2 条),此处不再重复。
  • [ic:5883139520] 提交前审计汇总(维护者自己的推迟记录): 第 2–7 条维持作者有意推迟的状态;它们属于本 PR 范围内的设计/测试卫生取舍或已被报告的条目,本轮不越权推翻维护者对自己 PR 的推迟决定。第 1 条真实存在、已被验证,且修复点在本 Java PR 的范围之外——见下方推迟项。
  • [ic:5884214191] 真实环境验证: 结论“可以合并”;其中唯一需要决策的事项(未配对代理项拒绝对 Hosted 轮次的影响)明确不阻塞合并,建议是合并本 PR 后再做客户端后续修复。本 PR 内不做代码改动。
  • 推迟到后续跟进(已记录到 deferred-findings.json): 审计第 1 条 / 验证评论中的决策事项 —— TypeScript Hosted 客户端把 Broker 对 deferred start 的 400 runtime_payload_invalid 拒绝当作丢失的应答,最长轮询 180 秒后取消,并使会话保持 recovery-blocked、Workspace 租约被占用。修复点在 packages/cli/src/serve/hosted-workspace-tool-turn.ts(在本 PR 范围之外),维护者明确要求合并后作为后续跟进处理;验证评论中附有一个已实测的 +18 行候选补丁(复用现有 validationError 路径)。
  • 既有问题(审计评论中列出的 reconciliation 期限竞态同类、幂等键未校验未配对代理项、test(runtime-broker): Add Stage F fault gates for W0c context installation #12804 的 Worker 拒绝记录问题)不是本 PR 引入的,本轮不作改动。

冲突说明

--conflict false;未执行合并,也不需要合并。

验证

  • npm run build —— 通过(退出码 0;因运行器默认的 corepack 缓存目录 $HOME/.cache 不可写,使用 COREPACK_HOME=/tmp/corepack 运行——纯属环境处理)。
  • npm run typecheck —— 通过(退出码 0,覆盖全部 workspace 及 integration 的 tsconfig)。
  • npm run lint —— 通过(退出码 0,eslint 检查 .ts/.tsx)。
  • 针对性 Vitest —— 不适用:本次改动只涉及一个 .java 文件,未修改任何 npm workspace 包(已核对 lint/format/build 配置,npm 工具链不解析 Java 源码)。
  • mvn --batch-mode --no-transfer-progress clean test 与 mvn checkstyle:check(即 sdk-java.yml 对 packages/sdk-java/runtime-broker 模块运行的检查)—— 本运行器上无法执行:环境中没有安装 JDK 和 Maven(command -v java/mvn 均为空,常见安装目录下也没有 JVM)。这属于当前运行器上不可用的 CI 检查,而非本地可运行但失败的检查。作为替代,本改动通过仔细审查验证:这是对既有语句的语义保持提升——两个表达式现在都读取同一个局部变量 cause,且 unwrap(null) == null 保证成功路径完全一致;花括号/圆括号配平与替换唯一性已用脚本检查,最终 git diff(单 hunk,+2/−2)已重新通读。Java 模块的最终验证门槛仍是 sdk-java CI 工作流。
  • 变异探针 —— 不适用:本轮没有新增任何守卫、分支或行为,只是删除了一份重复计算(评审者已用代数推导和变异体验证了等价性)。

Base-conflict check · 基分支冲突检查: no conflict with main. · 与 main 无冲突。

🧵 Resolved all 1 selected review thread(s). · 已关闭全部选中的 1 条评审线程。

Re-review when you have a moment. After round 100 this bot stops and leaves the PR for a human. · 有空请复审;第 100 轮后本 bot 停止并将 PR 交给人工。


🧠 Handled by Qwen Code · model/模型 kimi-k3 · CLI 0.24.6

@wenshao wenshao removed the autofix/takeover Summon the autofix loop to manage this PR (remove to release; needs triage+) label Sep 29, 2026
@wenshao

wenshao commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator Author

@qwen-code /triage

@qwen-code-dev-bot

Copy link
Copy Markdown
Collaborator

👋 Takeover released: the autofix loop will no longer engage this PR (an in-flight round, if any, completes its bounded work). Re-apply autofix/takeover (or comment @qwen-code /takeover) to re-engage.

中文说明

👋 已释放:autofix 循环不再介入此 PR(在飞的一轮如有,将完成其有界工作)。重新打上 autofix/takeover 标签(或评论 @qwen-code /takeover)即可再次接管。

wenshao added a commit to wenshao/qwen-code that referenced this pull request Sep 29, 2026
@wenshao

wenshao commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator Author

Real-environment verification, round 2 — 0b529019

Follow-up to round 1 at a2658844.

Verdict: still ready to merge. Everything from round 1 re-runs with the same results on the new head, and also on the new head merged with the current main a8ba9b50. One non-blocking note on the R1-4 fix: it shares cause, but the retryable classification is still written twice. The reviewer's witness mutant still survives; a candidate that computes the flag once is below.

What changed since round 1

Re-run on the new head

Check Result
runtime-broker unit tests and Checkstyle 443 tests, 0 failures (17 more than round 1, from #12964), 2 skipped; 0 violations ✅
PR tests on the new merge base 5ef79837 The same 6 tests fail as in round 1 ✅
JdbcRuntimeBrokerMySqlIT twice on one database main: the second run fails expected: <[1]> but was: <[2]> on MySQL 8.4.11 and MariaDB 10.11.18. PR: 4/4 runs pass ✅
Hosted real stack (figure 1) The same as round 1. main runs the wrong call for the v2 file tools (a?b gets edited; ? gets written); on the Shell path the Worker's digest check refuses it and the session is recovery-blocked in 0.76 s. PR: nothing reaches the Worker, but the turns take 65.8 s (Shell) and 120.7 / 120.8 s (edit / write) before they fail, leaving the session recovery-blocked and the Workspace lease held ✅
Direct calls to the Broker API PR: 400 runtime_reference_invalid with 0 rows written for a bad tool name, input key or input value; deferred :start answers 400 runtime_payload_invalid for both the escaped and the raw form; controls unchanged ✅
installContext for each Session state (real Worker) main installs for RELEASING, RELEASED, FAILED and a draining binding. PR refuses all 4 before sending anything (0 Worker requests) and still installs ACQUIRING and READY ✅
Deadline with a failing block write (MySQL trigger SIGNAL) main: suppressed=[]. PR: suppressed=[IllegalStateException(SQLException: rig: recovery block write refused)] ✅
Non-retryable failure, block write stalled 2 s, deadline 1 s PR returns the original non-retryable runtime_provision_failed in 3/3 runs, so the hoist changed nothing here. main also gives 3/3; as in round 1, this race can only be forced in the unit test ✅
managed-agent-server, PR merged with main a8ba9b50 187 unit tests; ManagedAgentMySqlIT 15/15 (MariaDB); Hosted IT 10/10 (MySQL 8.4, one more than CI because of #12869); Checkstyle 0 ✅
Mutation sweep 19/21 killed. M01–M20 behave as in round 1; M17 is audit item 2; M21 is below

Round 2 Hosted surrogate A/B

R1-4: the hoist shares cause, but the classification is still written twice (non-blocking)

The R1-4 thread is resolved, but the handler still states the rule twice, with opposite polarity:

Throwable cause = unwrap(error);
if (cause instanceof RuntimeBrokerException failure && !failure.isRetryable()) { // publish guard
    nonRetryable.set(failure);
}
...
boolean retryable = !(cause instanceof RuntimeBrokerException brokerFailure)     // second statement
        || brokerFailure.isRetryable();

The reviewer's witness (M21: change only the retryable line) still survives on 0b529019: 185 tests run, 0 failures. The commit message says the hoist "keeps a single statement of the rule"; that holds for unwrap, not for the classification.

A candidate computes retryable once at the top and publishes when !retryable && cause instanceof RuntimeBrokerException failure: cand-r1-4-single-classification.diff. With it, runtime-broker passes 443 tests with 0 failures and Checkstyle 0. The publish and the block path then read the same flag. Take it here or leave it for later; nothing is broken today.

Client follow-up

It is now tracked in #12993, which covers every tool path. #13010 (from the #12894 review) covers only Shell arguments. main actually executed the wrong call on the file tools, so the fix should cover edit and write too. The +18-line Harness candidate still applies unchanged on this tree: all three calls end in 0.35–0.72 s with turn_complete, no prepare is sent, and the emoji control succeeds.

Sandboxed verification

The triage sandbox reached the same A/B result on this head. I did not re-test its Minor finding (the new recursive isWellFormedJson has no depth limit and throws StackOverflowError at about 7000 levels). All my runs sent HTTP bodies, where fastjson2 rejects nesting deeper than 2049 before that check is reached.

Round 2 checks, R1-4 witness and mutation sweep

Evidence, rig and logs (publisher tokens redacted): wenshao/qwen-code@f1d6bcdc/pr12975/r2.

中文版

真实环境验证第二轮 —— 0b529019

接续 a2658844 上的第一轮。

结论:仍然可以合并。 第一轮的所有验证在新 head 上重跑,结果相同;在新 head 与当前 main a8ba9b50 合并后的树上也是如此。R1-4 的修复有一点不阻塞的补充:它共享了 cause,但"是否可重试"的分类仍然写了两遍,评审者的见证变异体依然存活。下文给出一个只计算一次该标志的候选写法。

与第一轮相比的变化

新 head 上的复验

检查 结果
runtime-broker 单测和 Checkstyle 443 个测试,0 失败(比第一轮多 17 个,来自 #12964),2 个跳过;0 违规 ✅
在新合并基点 5ef79837 上跑 PR 的测试 与第一轮相同,6 个失败 ✅
JdbcRuntimeBrokerMySqlIT 在同一个库上连跑两次 main:第二次在 MySQL 8.4.11 和 MariaDB 10.11.18 上都失败,expected: <[1]> but was: <[2]>。PR:4/4 次通过 ✅
Hosted 真实栈(图 1) 与第一轮相同。main 在 v2 文件工具上执行了错误的调用(a?b 被改掉、写入了 ?);Shell 路径上 Worker 的 digest 校验拒绝了它,会话在 0.76 s 内进入 recovery-blocked。PR:没有任何请求到达 Worker,但轮次分别要 65.8 s(Shell)和 120.7 / 120.8 s(edit / write)才失败,之后会话处于 recovery-blocked,Workspace 租约被占住 ✅
直接调用 Broker API PR:工具名、输入键或输入值有问题时返回 400 runtime_reference_invalid,写入 0 行;deferred :start 对转义和原始两种形态都返回 400 runtime_payload_invalid;对照组不变 ✅
各 Session 状态下的 installContext(真实 Worker) main 对 RELEASING、RELEASED、FAILED 和 drain 中的 binding 都会安装。PR 在发送前拒绝这 4 种(Worker 请求为 0),ACQUIRING 和 READY 仍照常安装 ✅
阻塞写入失败时的期限路径(MySQL 触发器 SIGNAL) main:suppressed=[]。PR:suppressed=[IllegalStateException(SQLException: rig: recovery block write refused)] ✅
不可重试失败,阻塞写入卡 2 s,期限 1 s PR 3/3 次返回原始的不可重试 runtime_provision_failed,说明重构没有改变这里的行为。main 也是 3/3;和第一轮一样,这个竞态只能在单测里强制触发 ✅
managed-agent-server(PR 合并 main a8ba9b50) 187 个单测;ManagedAgentMySqlIT 15/15(MariaDB);Hosted IT 10/10(MySQL 8.4,因 #12869 比 CI 多 1 个);Checkstyle 0 ✅
变异测试 21 个杀死 19 个。M01–M20 与第一轮一致,M17 即审计推迟项 2,M21 见下文

R1-4:重构共享了 cause,但分类规则仍写了两遍(不阻塞)

R1-4 的 thread 已被标为 resolved,但处理器里仍然以相反的极性把同一条规则写了两遍(代码见英文部分)。评审者的见证变异体 M21(只改 retryable 那一行)在 0b529019 上依然存活:185 个测试,0 失败。提交说明称这次提升"只保留一处规则表述",这对 unwrap 成立,对分类规则并不成立。

候选写法:在开头只计算一次 retryable,当 !retryable && cause instanceof RuntimeBrokerException failure 时发布:cand-r1-4-single-classification.diff。应用后 runtime-broker 443 个测试 0 失败,Checkstyle 0。这样发布判断和阻塞路径读取的是同一个标志。可以放在本 PR 里改,也可以以后再做;目前没有实际错误。

客户端跟进项

现已由 #12993 跟踪,覆盖所有工具路径。#13010(来自 #12894 的评审)只覆盖 Shell 参数。main 上真正执行了错误调用的是文件工具,所以修复应同时覆盖 edit 和 write。+18 行 Harness 候选补丁在这棵树上无需改动即可应用:三种调用都在 0.35–0.72 s 内以 turn_complete 结束,没有发出 prepare,emoji 对照组照常成功。

沙箱验证

triage 沙箱在这个 head 上得到了相同的 A/B 结论。它报告的 Minor 发现(新的递归 isWellFormedJson 没有深度上限,约 7000 层时抛出 StackOverflowError)我没有复测。我的所有实测都走 HTTP 请求体,fastjson2 会在 2049 层时先拒绝,到不了这个检查。

证据、装置和日志(publisher token 已脱敏)见 wenshao/qwen-code@f1d6bcdc/pr12975/r2。

@wenshao
wenshao dismissed a stale review via d2d4d0a September 29, 2026 11:04
@wenshao

wenshao commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator Author

@qwen-code /triage

@qqqys qqqys left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Critical-only scan — APPROVE

Head reviewed: d2d4d0af5d09aff4dcd786356c187d837c85b4c9

I read the complete production surface: four Java files, +102/−18. No Critical, and no blocking issue has ever been filed here.

Historical blocking issues — none exist

There is no CHANGES_REQUESTED on this PR. Both DISMISSED reviews are approvals that a later push invalidated — their bodies read "LGTM, looks ready to ship" — not withdrawn blockers. The one COMMENTED round, at a2658844, posted a single sev:S finding about re-deriving the retryability classification, and names two further Suggestions already on record. There is no [Critical] inline comment anywhere in this PR's history.

The theme of every production change is fail-closed tightening

BrokerValues. requireWellFormed is refactored onto a private isWellFormed(String) with the identical surrogate-range test, so its behaviour is unchanged. The new isWellFormedJson(Object) accepts only recognized JSON shapes — a well-formed String, a Map whose keys are all well-formed Strings and whose values recurse, a List whose items recurse, and null/Number/Boolean — and answers false for anything else, including an array or a set a writer would still serialize. Rejecting the unrecognized rather than assuming it is the right default for a validator whose job is to stop corrupted text reaching an executor. The recursion adds no new depth hazard: the parsed Map/List tree it walks was already built by a parser that recursed to the same depth.

HttpRuntimeTransport. Three tightenings. installContext now requires sessionRecord.isAcquirable(), so a Session being released installs no context, and adds runtime.isDrainRequested() to the existing binding checks, so a draining binding admits no new Session. execute and executeV3 both route the tool name through the new referenceToolName, applying requireWellFormed, and referenceInput now requires isWellFormedJson. Each is a rejection that previously did not happen.

RuntimeTransport. Javadoc only — the interface's installContext contract is updated to state the ACQUIRING-or-READY session and the no-drain binding preconditions the implementation now enforces. No signature changes, so no implementor is affected.

RuntimeBrokerService. startExecution replaces the raw-text requireWellFormed(payloadJson, …) with a null check plus isWellFormedJson, answering 400 runtime_payload_invalid instead of letting an IllegalArgumentException escape, and then — importantly — re-checks isWellFormedJson(payload) on the parsed map. That second check is the real fix: an escaped unpaired surrogate such as "\ud800" is perfectly well-formed JSON text, so the old raw-text check passed it, and the UTF-8 encoder then delivered '?' to the Worker. createExecution gains the same check on safeReference before admitting, under its own runtime_reference_invalid code.

The defect being closed is worth stating plainly, because it is the reason the strictness is justified rather than merely defensive: a tool name or input containing an unpaired surrogate was silently rewritten to '?' and the Worker would then execute a different tool name or different arguments than the caller sent. Turning that into an explicit 400 is a correctness fix, and no legitimate caller produces unpaired surrogates, so nothing valid is newly refused.

The provisioning-deadline change preserves the real failure, and its ordering holds

provisionDurableBinding adds an AtomicReference<RuntimeBrokerException> nonRetryable. I checked the publication order the comment claims rather than accepting it: the .handle failure handler sets nonRetryable before calling renewal.stopAndGet(), and the deadline task reads it before its own stopAndGet(). Since both contend for the same renewal claim, which can be held across a database round trip, a deadline that runs after the failure handler cannot miss the value, and the AtomicReference supplies the visibility. The answer selection is blocked && failure == null ? conflict("runtime_broker_recovery_blocked") : answer, so a known non-retryable cause is no longer masked by the generic recovery-blocked conflict, while a genuinely timeout-only block still reports as before. blockRecoveryQuietly(timedOut, answer) now records that reason instead of null. Legacy startup is untouched: request.isManagedContext() ? nonRetryable.get() : null keeps its timeout answer.

CI

Green at this head: the full Java matrix (ubuntu-latest / Java 11, 17, 21, windows-latest / Java 21, macos-latest / Java 21), Runtime Broker and Managed Agent MariaDB / Java 21, Hosted process fault gates / MySQL 8.4 / Java 21, Real daemon E2E / Java 11, Test (ubuntu-latest, Node 22.x), Lint & Static, web-shell E2E Smoke and both Desktop Shell lanes all pass. review-pr and Integration Tests (no-AK, No Sandbox) were still pending; neither is a gating check and I did not wait on them. No failure is attributable to this PR.

Coverage note

I read all four production files in full. I did not read the 323 lines of new and changed tests beyond confirming the Java matrix runs them green, nor the two design-doc pairs beyond noting they are +9/−9 symmetric edits in both languages.

@yiliang114 yiliang114 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Verified at head d2d4d0a against the seven deferred findings in #12761 — all seven are addressed, and CI is green at this head (full Java matrix, Hosted process fault gates / MySQL 8.4, Runtime Broker MariaDB, Lint & Static, web-shell E2E).

What I checked beyond the diff:

  • Session-state check on install: RuntimeSessionRecord.isAcquirable() is exactly ACQUIRING || READY, matching the new refusal and the updated docs; the drain-requested refusal rides the existing binding check, and the ACQUIRING acceptance is pinned by a positive test, not just the absence of refusal.
  • Deadline/failure-handler race: the publish uses an AtomicReference set before the handler takes the claim (safe across the DB round trip), the deadline reads it only for managed-context requests, and legacy startup still answers the plain timeout — all three orderings are pinned by the new race tests.
  • Suppressed-failure path: removing the null guard in blockRecoveryQuietly is safe — all three call sites pass a non-null cause (deadline passes answer, which is always non-null; the handler passes cause inside error != null, and unwrap never returns null for non-null input). Both paths now have getSuppressed() assertions.
  • Surrogate refusals: the reference-identity check fires before the whole-reference check (message pinned in the test), the deferred payload check catches JSON-escaped surrogates post-parse, and the fail-closed isWellFormedJson recursion is bounded in practice by the 256 KiB payload limit and upstream JSON parser depth limits. The well-formed-Unicode round-trip test covers the no-false-positive direction, including null values.
  • The prior bot suggestion (R1-4, duplicated retryability classification) is partially addressed at this head — cause is now computed once — and what remains is style-level, already on record.

@wenshao
wenshao added this pull request to the merge queue Sep 29, 2026
Merged via the queue into main with commit 1279301 Sep 29, 2026
88 of 90 checks passed
wenshao added a commit that referenced this pull request Sep 29, 2026
After merging main:
- The Broker refuses a provider control operation with an unpaired
  surrogate in any key or string (400 runtime_control_operation_invalid),
  as #12975 does for references and deferred payloads. The JSON writer
  would otherwise send it as '?', a shell wildcard.
- The provider worker refuses a run_shell_command whose directory lies
  outside the Session's workspace, as the tool executor does since
  #12927. Core's shell tool now asks there, and a preapproved Session
  never asks.
- Provider status cursors are read as exact integers, as #12972 reads
  the others.
pull Bot pushed a commit to a1wjun/qwen-code that referenced this pull request Sep 29, 2026
* feat(serve): implement generic Broker provider controls

* fix(runtime): preserve broker provider validation errors

* fix(runtime): restore admission and unknown observation semantics

* fix(serve): keep oversized provider results observable and tighten the envelope

Fit execute/status/cancel results into the per-kind wire budget instead of
failing validation: evict oldest progress events, then cut bulk text fields
head-and-tail with an inline notice (shell displays set `truncated`), so a
legitimately large tool result keeps a terminal observation and release stays
answerable instead of stranding the execution UNKNOWN behind a 400.

Also address the round-2 review:

- Require lowercase UUID envelope Session ids, closing the raw-id path to
  core's session id and the file-history directory name.
- Emit policy refusals as managed_runtime_provider_operation_failed instead
  of the identity-fencing code; drop the unreachable turnStarted guard; skip
  the redundant operation-level digest on the composed request path; enforce
  the declared response bound and tie it to the per-kind limits.
- Broker: answer an unencodable control with a definitive 413, keep a
  null-valued deferred reference on the stage channel as 400, and report the
  prepare envelope violation as runtime_broker_invalid_request. Hoist the
  protocol's per-call pattern compiles.
- Pin the changes with real-worker, protocol, transport and service tests,
  extend the shared corpus to every accepted confirm outcome, and refresh
  the affected design docs in both languages.

The immediate-route payload remainder is tracked by QwenLM#12936 and the
release-ordering strand by QwenLM#12937.

* fix(cli): narrow the status binding before progress eviction

The fitting path read status fields inside a guard that only proved
progress is an array, which tsc --build rejects under strictNullChecks.
Narrow the binding explicitly. No runtime behavior change.

* fix(runtime): keep terminal cancellation receipts answerable without a READY Session

A settled cancel of a prepared provider invocation was re-driven through
requireReadySession on every retry, so once the Session was lost, releasing
or absent the duplicate cancel failed 404/409 forever even though the
terminal answer was already durable. Return the stored receipt unless the
owner Session is READY and can actually refresh the worker evidence, and
tolerate a missing binding instead of throwing a 500.

Round-4 review follow-ups: the Broker client's control closure now rejects
asynchronously instead of throwing synchronously; new pins cover the exact
4096-byte reason boundary, the provider rejection codes, a non-string
payloadJson, the in-flight checkpoint guard, in-flight snapshot and
preparation release fences, provider reconciliation on terminal evidence,
rejected cancellation evidence (unknown / settled-success), the acquire
prelude guard, the confirm outcome and modality refusals, negative status
shapes, and corpus cases for content modification, media context and the
continuation turn kind.

* fix(runtime): fail a prepared-provider cancel non-retryably when it can never confirm

observePreparedCancellation mapped the worker's permanent answers — an
unknown invocation (no longer retained) or a settled result whose status is
not cancelled/not_started — to a retryable 503, so the cancel retry looped
the identical round trip forever against a record already settled locally.
Both now fail with a non-retryable 409 runtime_execution_cancel_unconfirmed.
unknown is never turned into not_started evidence; the local receipt is
unchanged.

* fix(serve): admit path-safe opaque Session ids on the provider envelope

The lowercase-UUID envelope gate broke the Broker's own fault gates, which
drive the provider route with opaque ids (harness-1 / runtime-session-1) and
represent the wire contract's real compatibility surface: non-identity
operations (acquire/release/manifest) never needed the UUID form. The gate
now refuses only what is unsafe to interpolate into file names — separators,
dot segments, control characters, unpaired surrogates — while identity-bearing
operations keep requiring core's UUID form through the identity check.

* fix(serve): address the round-3 review of the Broker provider controls

Result fitting (R3-9):
- Measure the cut in JSON-encoded UTF-8 bytes and remove whole code
  points, so CJK text is no longer over-cut, surrogate pairs are never
  split and the notice counts the omitted code points exactly.
- Cut every field once, to one common size that accounts for each
  field's notice, so no field is emptied while another keeps its text
  and the budget is met in one pass; a field shorter than its notice is
  never grown.
- When even fully cut text cannot fit, stub a structured display (an
  edit's file diff) and then drop hook results before cutting text, so
  the model keeps its own content; only unreachable content (inline
  media) turns llmContent into a stub.

Broker (R2-1 and the provider-envelope id rule):
- A repeated cancellation of a settled prepared-provider call answers
  from its receipt once the Session is not READY or its binding can no
  longer answer, and asks for adoption (503
  runtime_reconciliation_required) while this process holds no live
  Session for it, instead of 404 runtime_session_not_found.
- Refuse at acquire the Session ids the worker envelope refuses, so no
  Session can be acquired that cannot be released.

Worker:
- Envelope Session ids are an ASCII allow-list (1-512 of A-Za-z0-9._-,
  no '.'/'..' segment), pinned character by character in TS and Java.
- Release and shutdown drop core's per-session project-dir and model
  registry entries for the Runtime Session.
- Refuse prepare with content modification explicitly (400
  managed_runtime_tool_invalid): this profile has no notebook_edit.

Tests pin the legacy-admission re-checks after each await, the
composed envelope bound, the eviction direction, the corpus marker, the
provider decoders' rejection arms, the control dispatch and a
result-bearing control over HTTP. Design docs are updated in both
languages.

* fix(serve): address the round-4 review of the Broker provider controls

Worker (R4-1):
- Keep the full state of up to eight released Sessions. When another is
  released, retire the oldest one not being observed: dispose its
  runtime, drain its file history and shut its Config down without
  telemetry or the session writer, then drop the references, leaving a
  tombstone whose status and cancel answer unknown. Each step runs even
  if an earlier one fails, calls keep reaching the runtime until the
  end, and close() waits for retirements in flight. A failed retirement
  is logged and never fails the release that triggered it.
- A release that came before any acquire leaves the same tombstone.

Protocol and client:
- confirmation results require each variant's fields as core emits
  them (R4-4).
- The Broker client keeps the reason of an error that carries details,
  such as a terminal answer (R4-3).
- The docs say acquire and release results are exactly true (R4-2).

Broker:
- warm applies the Session-id allow-list to the Harness id.
- The response buffer grows with the body up to the cap instead of
  reserving the cap (R4-5).
- The terminal cancel branch reuses the binding and Session rows its
  ownership check loaded, after that check (R4-6).

Tests pin a repeated cancellation after dispatch (R4-7), a lost
immediate execute response in the fault gates (R4-8), request headers
asserted on the test thread (R4-9), ids with letters and digits outside
ASCII, and the retirement rules, including close() racing a retirement
and an acquire. Design docs are updated in both languages.

* docs(runtime-broker): count every fault gate the profile runs

After merging main, mvn -Pfault-gates also runs the local-reboot and
durable-worker gates: 44 in about 4 minutes, measured on this tree.

* fix(serve): keep main's input rules on the provider path

After merging main:
- The Broker refuses a provider control operation with an unpaired
  surrogate in any key or string (400 runtime_control_operation_invalid),
  as QwenLM#12975 does for references and deferred payloads. The JSON writer
  would otherwise send it as '?', a shell wildcard.
- The provider worker refuses a run_shell_command whose directory lies
  outside the Session's workspace, as the tool executor does since
  QwenLM#12927. Core's shell tool now asks there, and a preapproved Session
  never asks.
- Provider status cursors are read as exact integers, as QwenLM#12972 reads
  the others.

* fix(serve): reconcile lost provider answers over Broker HTTP

- Broker HTTP start, read and cancel reconcile a provider execution
  left UNKNOWN by a lost execute answer, as they do for tool v3, so the
  worker's retained result settles it without a second dispatch instead
  of answering 409 runtime_broker_execution_unknown.
- The provider worker's shell tool refuses a directory outside the
  workspace when the call is prepared and again just before it runs,
  resolving the path afresh the way the kernel follows it. A link
  retargeted in between settles the call as an error.

* fix(serve): fit provider artifacts and pin the retention bound

- The result fitter drops artifacts, which only feed a client surface,
  after stubbing a structured display and before dropping hook results,
  when even fully cut text cannot fit beside them. A result whose bulk
  sits in an artifact now fits without stubbing the model content.
- Tests pin that the Session just released is retired when every
  retained one is being observed, so no release leaves more than eight,
  and that a zero response limit completes truncated.

* fix(runtime-broker): cancel a lost provider dispatch at its worker

A provider execution whose execute answer was lost is UNKNOWN with no
invocation running in this process. Cancelling it skipped the worker,
because only tool v3 references counted as answerable after a loss, so
the Broker reported cancel_requested while the tool kept running.
ToolExecutionRecord.observableAfterLoss() now states that rule once,
for tool v3 and provider references alike, and both the cancel path and
Broker HTTP observation use it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/self-reported The linked issue was opened by the PR author (self-reported)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Follow-up: deferred findings from #12730 (W0c-2 managed-context Broker)

4 participants