Repository navigation
fix(runtime-broker): Close the deferred W0c-2 findings from #12761 - #12975
Conversation
- Context installation refuses a Session record that is RELEASING, RELEASED or FAILED, and a binding with a drain request, before sending. Acquisition installs while the record is still ACQUIRING, so the check accepts ACQUIRING and READY rather than READY alone. - When a managed-context startup fails with a non-retryable error and the deadline fires before the call answers, the deadline answers that error, including when it finds the block the failure handler recorded. It used to answer 409 runtime_broker_recovery_blocked. A failure published after the deadline fired, and legacy startup, keep the deadline's own answer. - A block write or read that fails on the deadline path is kept as a suppressed exception on the deadline's answer instead of dropped. - A tool name, or any key or string in the tool input, holding an unpaired surrogate is refused instead of reaching the Worker as '?': runtime_reference_invalid on create, runtime_payload_invalid on a deferred start (raw or escaped), and before sending in the HTTP transport, whose check fails closed on values that are not JSON. - Tests pin addSuppressed on both paths, lone low surrogates in the identity fields and Runtime Session IDs, and the deadline races. - JdbcRuntimeBrokerMySqlIT uses a per-run prefix, so it re-runs on the same database.
Pre-commit audit summaryThree rounds, each with two independent auditors (one undirected, one adversarial against the stated claims), on frozen snapshots. Rounds 1 and 2 fixed every real finding; round 3 acted on Critical findings only and found none, so the commit is the round-3 snapshot byte for byte (tree
Deferred
Pre-existing, not introduced here
中文说明提交前审计汇总共三轮,每轮由两名独立审计员进行(一名无方向,一名针对所述声明做反向审计),在冻结的快照上完成。第 1、2 轮修复了所有真实问题;第 3 轮只处理 Critical,且没有发现 Critical,因此提交内容与第 3 轮快照逐字节一致(tree
推迟项
既有问题(不是本改动引入的)
|
Real-environment verification —
|
| Claim | What I ran | Result |
|---|---|---|
| Context installation checks the Session record's state | installContext for each record state on a READY managed-context binding, counting the Worker's /v3/context requests |
main installs for RELEASING, RELEASED, FAILED and for a drain-requested binding (the Worker receives each one). PR refuses those 4 before sending (+0 requests) and still installs ACQUIRING and READY. A real Hosted turn and HostedWorkspaceToolTurnIT 6/6 still acquire on the PR build. ✅ |
| A non-retryable failure keeps its answer when the deadline fires | A real Worker booted with another storage ID (non-retryable runtime_provision_failed), a 1 s deadline, and the block write stalled 2 s by a trigger. 6 trials per arm |
PR returns the original non-retryable failure 6/6 and records the block. main did not return 409 in 6/6 trials: the failure handler answers one DB round trip before the deadline, so this race is hard to hit on a real stack. The new unit test forces the race and fails on main. ✅ (pinned at unit level) |
| The deadline keeps a failed block write | A hung Worker, a 4 s deadline, and a trigger that makes the RECOVERY_BLOCKED write fail with SIGNAL |
main: 503 provision_timeout with suppressed=[]. PR: the same answer with suppressed=[IllegalStateException(SQLException: rig: recovery block write refused)]. ✅ |
| Tool names and input with an unpaired surrogate are refused | The Hosted stack and direct Broker API calls (figures 1 and 2) | Confirmed. /executions answers 400 runtime_reference_invalid with 0 rows admitted. Deferred :start answers 400 runtime_payload_invalid for both the escaped and the raw form. A raw surrogate on main answered runtime_broker_invalid_request, which is the documented code change. Controls with emoji and CJK are unchanged. ✅ |
The new tests fail on main |
The PR's test files on 99adce25 |
6 fail, plus 1 new BrokerValues helper test that does not compile there. That matches the "seven" in the description. ✅ |
| The MySQL IT re-runs | JdbcRuntimeBrokerMySqlIT twice on one database |
main: the second run fails expected: <[1]> but was: <[2]> on MySQL 8.4.11 and on MariaDB 10.11.18. PR: 4/4 runs pass. ✅ |
| Suites | runtime-broker; managed-agent-server on the merge | runtime-broker: 426 run, 0 failures, Checkstyle 0. On the merge: 182 unit tests, ManagedAgentMySqlIT 14/14 (MariaDB), Hosted IT 9/9 (MySQL 8.4), Checkstyle 0. ✅ |
| Stage F fault gates | CI, plus a local A/B of the failing class | CI (Linux): 39/39. Locally: 36/39 under load. DurableLocalRuntimeFaultGateTest failed the same way on main (runtime_provision_fenced from a claim expiry; main failed 1 of 5 runs, PR 2 of 6), so it is a load flake, not caused by this PR. |
| Mutation | 20 mutants of the production changes, run against the 5 affected test classes | 19 killed. The survivor, M17 (publish the failure after taking the claim), is the gap you already listed as audit item 2. |
Decision item: the surrogate refusal on a Hosted turn (not blocking)
On the real Hosted stack, the refusal changes different things on the two tool paths:
- Hosted Shell (v3). On
main, the Broker did sendrm victim-?.txtto the Worker. The Worker refused it because the recomputedinputDigestno longer matched (409managed_runtime_identity_conflict), so nothing ran. The session was recovery-blocked within 0.9 s. With the PR the Worker receives nothing, and the turn ends after the client's polling window: 65.8 s withtimeout: 5000, 180 s with the default timeout. The end state is the same. On this path the PR does not prevent a wrong execution that used to happen. It moves the refusal into the Broker (so the execution settles asnot_started, notUNKNOWN), and the user waits longer for the failure. - Hosted file tools (v2, no digest check). Here
mainreally did run a different call. Giveneditwithold_string: "a\ud800b", it replaceda?binnote.txt, a string the model never mentioned. Givenwrite_file, it wrotehalf emoji: ? end. The PR stops both. The cost: a turn that finished in 0.9 s now hangs for 120.7 s (1277–1638 status polls). The client then cancels, the execution settles ascancelled, and the Hosted session stays recovery-blocked with its Workspace lease held. The lease table has no expiry column, and those leases were still held an hour later (the same class as the open Hosted Shell: recover a Workspace lease after incomplete pipe capture #12904). Onmainonly the Shell case left a lease held. - This makes audit item 1, the TS client, the user-visible half of this change. Triage noted it is untracked. A small Harness-side check before
prepareremoves the whole cost:cand-client-refusal.diff(+18 lines inhosted-workspace-tool-turn.ts, using the existingvalidationErrorpath). I rebuilt the bundle with it and ran it against this PR's Broker. All three calls end in 0.6–0.8 s withturn_complete, the model receives a correctable error, and nopreparereaches the Broker. There is no recovery block and no held lease, and the emoji control still succeeds. - Suggestion: merge this, then open the client follow-up (or land the candidate) before Hosted file tools get real traffic. One small correction to the earlier review: the execution stays PREPARED only during the polling window. What persists is the recovery-blocked session and the held Workspace lease.
Other notes (non-blocking)
- The Hosted deferred client (
hosted-workspace-broker.ts) is the only production client that sends a tool name and input to the Broker. Nothing in production calls/executionswith a tool name and input, so the create-time check guards the public Broker API. Onmainthat API admitted the calls and the JDBC codec stored the tool name asread_file?. - Removing
if (cause != null)inblockRecoveryQuietlyis safe. All three call sites pass a non-null cause: the deadline passesanswer, and the handler passesunwrap(error)insideerror != null.
Evidence, probe scripts and logs (publisher tokens redacted) are at wenshao/qwen-code@5e890ad0/pr12975.
中文版
真实环境验证 —— a2658844
结论:可以合并。 PR 描述中的每一条主张都在真实进程上复现成立:真实 Worker、MySQL 8.4 / MariaDB 10.11、内嵌 Broker 的 Spring 服务,以及打包后的 Hosted Harness。新测试在合并基点上确实失败。有一项需要维护者决策,但不阻塞这个 Java 改动:拒绝未配对代理项后,Hosted 轮次会受到什么影响。下文给出了实测数据,并针对审计推迟项 1 实测了一个很小的客户端候选补丁。
环境
- 对照臂:
main= origin/main1b69629;PR=a2658844与该 main 的本地合并。这个合并比 CI 的合并引用8c125e6新 4 个提交;8c125e6建在b1eb94da上,不含 fix(managed-agent): Close the post-merge review of event replay #12968。测试 A/B 用合并基点99adce25。PR 只改 Java,所以两臂的 Harness 和 Worker 都用同一份dist/cli.js,只有 Broker 的 jar 和 classes 不同。 - Hosted 真实栈: Spring 服务 jar(Session Store 和内嵌 Runtime Broker)、MySQL 8.4.11、打包的 Hosted Harness、真实本地 Worker 进程,以及发出工具调用的假 OpenAI 服务。Broker→Worker 这一跳用正向代理抓包(JVM
-Dhttp.proxyHost,-Dhttp.nonProxyHosts置空)。 - Broker 级探针:
Rig12975.java放在 Broker 包内,用各臂自己的target/classes,对接真实 managed-context Worker 和 MySQL。故障由RECOVERY_BLOCKED写入上的 MySQL 触发器注入。 - 运行环境: macOS arm64,Spring 和探针用 JDK 21,Maven 用 JDK 26,Node 22.23.2。MySQL 8.4.11 和 MariaDB 10.11.18 跑在 Docker 中。机器同时被无关任务压着,负载 15–45。
结果
| 主张 | 做法 | 结果 |
|---|---|---|
| 安装 context 时检查 Session 记录状态 | 在 READY 的 managed-context binding 上,对每种记录状态调用 installContext,并统计 Worker 收到的 /v3/context 请求数 |
main 对 RELEASING、RELEASED、FAILED 以及已请求 drain 的 binding 都会安装(Worker 都收到了请求)。PR 在发送前拒绝这 4 种(请求 +0),ACQUIRING 和 READY 照常安装。在 PR 构建上,真实 Hosted 轮次和 HostedWorkspaceToolTurnIT 6/6 仍能正常 acquire。✅ |
| 期限触发时,不可重试的失败保留原应答 | 让真实 Worker 以另一个 storage ID 启动(得到不可重试的 runtime_provision_failed),期限 1 s,用触发器让阻塞写入卡 2 s。每臂 6 次 |
PR 6/6 返回原始的不可重试错误,并记录了阻塞。main 6 次里没有一次返回 409:失败处理比期限少一次数据库往返,先作答,所以这个竞态在真实栈上很难撞中。新单测能强制出这个竞态,并且在 main 上失败。✅(由单测固定) |
| 期限保留写入失败的阻塞异常 | Worker 挂起,期限 4 s,用触发器让 RECOVERY_BLOCKED 写入以 SIGNAL 失败 |
main:503 provision_timeout,suppressed=[]。PR:同样的应答,带 suppressed=[IllegalStateException(SQLException: rig: recovery block write refused)]。✅ |
| 拒绝含未配对代理项的工具名和输入 | Hosted 栈和直接调用 Broker API(图 1、图 2) | 成立。/executions 返回 400 runtime_reference_invalid,0 行落库。deferred :start 对转义和原始两种形态都返回 400 runtime_payload_invalid。原始代理项在 main 上返回 runtime_broker_invalid_request,这正是文档中说明的错误码变化。含 emoji 和中文的对照组不变。✅ |
新测试在 main 上失败 |
把 PR 的测试文件放到 99adce25 上运行 |
6 个失败,另有 1 个新增的 BrokerValues 辅助方法测试在该版本上无法编译,与描述中的"七条"一致。✅ |
| MySQL IT 可以在同一个库上重跑 | JdbcRuntimeBrokerMySqlIT 在同一个库上连跑两次 |
main:第二次在 MySQL 8.4.11 和 MariaDB 10.11.18 上都失败,expected: <[1]> but was: <[2]>。PR:4/4 次全部通过。✅ |
| 测试套件 | runtime-broker;合并树上的 managed-agent-server | runtime-broker:426 个测试,0 失败,Checkstyle 0。合并树上:182 个单测、ManagedAgentMySqlIT 14/14(MariaDB)、Hosted IT 9/9(MySQL 8.4)、Checkstyle 0。✅ |
| Stage F fault gates | CI,加上对失败类的本地 A/B | CI(Linux)39/39。本地高负载下 36/39。DurableLocalRuntimeFaultGateTest 在 main 上以同样的方式失败(runtime_provision_fenced,claim 过期;main 5 次中失败 1 次,PR 6 次中失败 2 次),属于负载导致的抖动,不是本 PR 引起的。 |
| 变异测试 | 对生产代码改动造 20 个变异体,跑受影响的 5 个测试类 | 杀死 19 个。唯一存活的 M17(先占 claim 再发布失败),正是你在审计评论中列出的推迟项 2。 |
需要决策:代理项拒绝对 Hosted 轮次的影响(不阻塞)
在真实 Hosted 栈上,这个拒绝对两条工具路径的影响不同:
- Hosted Shell(v3)。 在
main上,Broker 确实把rm victim-?.txt发给了 Worker。但 Worker 重算的inputDigest对不上,以 409managed_runtime_identity_conflict拒绝,所以什么都没执行,会话在 0.9 s 内进入 recovery-blocked。在 PR 上 Worker 什么都收不到,轮次要等客户端的轮询窗口结束才结束:timeout: 5000时 65.8 s,默认 timeout 时 180 s。最终状态相同。所以在这条路径上,PR 并没有阻止一个以前真的会发生的错误执行。它把拒绝提前到 Broker(执行结算为not_started,而不是UNKNOWN),用户则要等更久才看到失败。 - Hosted 文件工具(v2,没有 digest 校验)。 这里
main确实执行了一个不同的调用。edit的old_string是"a\ud800b",结果它把note.txt里的a?b替换掉了,而模型从未提到过这个字符串。write_file写出了half emoji: ? end。PR 阻止了这两种情况。代价是:原本 0.9 s 就结束的轮次,现在要挂 120.7 s(期间状态轮询 1277–1638 次)。之后客户端取消,执行结算为cancelled,但 Hosted 会话一直处于 recovery-blocked,并且占着 Workspace 租约。租约表没有过期列,这些租约一小时后仍被占用(与仍开放的 Hosted Shell: recover a Workspace lease after incomplete pipe capture #12904 同类)。在main上只有 Shell 那个用例会留下被占的租约。 - 因此,审计推迟项 1(TS 客户端)是这项改动中用户能直接感知的一半。triage 已指出目前没有跟踪项。在
prepare之前加一个很小的 Harness 端检查,就能消除全部代价:cand-client-refusal.diff(hosted-workspace-tool-turn.ts中 +18 行,复用已有的validationError路径)。我用它重新打包,并对接本 PR 的 Broker 实测:三种调用都在 0.6–0.8 s 内以turn_complete结束,模型收到可以纠正的错误,没有prepare到达 Broker。没有 recovery-block,也没有被占的租约,emoji 对照组照常成功。 - 建议: 合并本 PR,然后开一个客户端跟进项(或直接落地这个候选补丁),在 Hosted 文件工具有真实流量之前完成。对之前评审的一处小更正:执行只在轮询窗口内停在 PREPARED;持续存在的是 recovery-blocked 的会话和被占的 Workspace 租约。
其他(不阻塞)
- 生产代码中,只有 Hosted deferred 客户端(
hosted-workspace-broker.ts)会把工具名和输入发给 Broker。生产代码里没有哪里带着工具名和输入调用/executions,所以创建时的检查保护的是 Broker 的公开 API。在main上,这个 API 会准入这些调用,JDBC codec 还把工具名存成了read_file?。 - 去掉
blockRecoveryQuietly中的if (cause != null)是安全的。三个调用点传入的 cause 都不为空:期限传的是answer,失败处理在error != null分支内传unwrap(error)。
证据、探针脚本和日志(publisher token 已脱敏)见 wenshao/qwen-code@5e890ad0/pr12975。
|
@qwen-code /takeover |
|
🤝 Takeover engaged: the autofix loop now manages this PR — it will address new review feedback and resolve base conflicts until the label is removed or the round cap is reached. Remove the 中文说明🤝 已接管:autofix 循环现在管理此 PR —— 将持续处理新的评审反馈与 base 冲突,直到移除标签或达到轮次上限。移除 |
… from one unwrapped cause (#12975) The provisioning failure handler computed unwrap(error) and the instanceof/isRetryable classification twice — once for the non-retryable publish and again for the retryable flag. Hoisting the unwrap keeps a single statement of the rule so a future change cannot make the two diverge undetected. unwrap(null) returns null, so the success path is unchanged. Co-authored-by: Qwen-Coder <[email protected]>
|
🤖 Addressed the latest review feedback (round 1/100). What changed, and what I pushed back on: · 已处理最新评审反馈(第 1/100 轮)。改动内容与我反驳保留之处如下: Address-review round — PR #12975What this round changedOne inline suggestion was in scope this round, and it is implemented in a single commit (
Other feedback, and why it required no code change
Conflict notes
Verification
中文说明评审处理轮次 —— PR #12975本轮改动本轮只有一条行内建议在处理范围内,已在单个提交(
其他反馈及无需改动代码的原因
冲突说明
验证
Base-conflict check · 基分支冲突检查: no conflict with main. · 与 main 无冲突。 🧵 Resolved all 1 selected review thread(s). · 已关闭全部选中的 1 条评审线程。 Re-review when you have a moment. After round 100 this bot stops and leaves the PR for a human. · 有空请复审;第 100 轮后本 bot 停止并将 PR 交给人工。 🧠 Handled by Qwen Code · model/模型 |
|
@qwen-code /triage |
|
👋 Takeover released: the autofix loop will no longer engage this PR (an in-flight round, if any, completes its bounded work). Re-apply 中文说明👋 已释放:autofix 循环不再介入此 PR(在飞的一轮如有,将完成其有界工作)。重新打上 |
Real-environment verification, round 2 —
|
| Check | Result |
|---|---|
| runtime-broker unit tests and Checkstyle | 443 tests, 0 failures (17 more than round 1, from #12964), 2 skipped; 0 violations ✅ |
PR tests on the new merge base 5ef79837 |
The same 6 tests fail as in round 1 ✅ |
JdbcRuntimeBrokerMySqlIT twice on one database |
main: the second run fails expected: <[1]> but was: <[2]> on MySQL 8.4.11 and MariaDB 10.11.18. PR: 4/4 runs pass ✅ |
| Hosted real stack (figure 1) | The same as round 1. main runs the wrong call for the v2 file tools (a?b gets edited; ? gets written); on the Shell path the Worker's digest check refuses it and the session is recovery-blocked in 0.76 s. PR: nothing reaches the Worker, but the turns take 65.8 s (Shell) and 120.7 / 120.8 s (edit / write) before they fail, leaving the session recovery-blocked and the Workspace lease held ✅ |
| Direct calls to the Broker API | PR: 400 runtime_reference_invalid with 0 rows written for a bad tool name, input key or input value; deferred :start answers 400 runtime_payload_invalid for both the escaped and the raw form; controls unchanged ✅ |
installContext for each Session state (real Worker) |
main installs for RELEASING, RELEASED, FAILED and a draining binding. PR refuses all 4 before sending anything (0 Worker requests) and still installs ACQUIRING and READY ✅ |
Deadline with a failing block write (MySQL trigger SIGNAL) |
main: suppressed=[]. PR: suppressed=[IllegalStateException(SQLException: rig: recovery block write refused)] ✅ |
| Non-retryable failure, block write stalled 2 s, deadline 1 s | PR returns the original non-retryable runtime_provision_failed in 3/3 runs, so the hoist changed nothing here. main also gives 3/3; as in round 1, this race can only be forced in the unit test ✅ |
managed-agent-server, PR merged with main a8ba9b50 |
187 unit tests; ManagedAgentMySqlIT 15/15 (MariaDB); Hosted IT 10/10 (MySQL 8.4, one more than CI because of #12869); Checkstyle 0 ✅ |
| Mutation sweep | 19/21 killed. M01–M20 behave as in round 1; M17 is audit item 2; M21 is below |
R1-4: the hoist shares cause, but the classification is still written twice (non-blocking)
The R1-4 thread is resolved, but the handler still states the rule twice, with opposite polarity:
Throwable cause = unwrap(error);
if (cause instanceof RuntimeBrokerException failure && !failure.isRetryable()) { // publish guard
nonRetryable.set(failure);
}
...
boolean retryable = !(cause instanceof RuntimeBrokerException brokerFailure) // second statement
|| brokerFailure.isRetryable();The reviewer's witness (M21: change only the retryable line) still survives on 0b529019: 185 tests run, 0 failures. The commit message says the hoist "keeps a single statement of the rule"; that holds for unwrap, not for the classification.
A candidate computes retryable once at the top and publishes when !retryable && cause instanceof RuntimeBrokerException failure: cand-r1-4-single-classification.diff. With it, runtime-broker passes 443 tests with 0 failures and Checkstyle 0. The publish and the block path then read the same flag. Take it here or leave it for later; nothing is broken today.
Client follow-up
It is now tracked in #12993, which covers every tool path. #13010 (from the #12894 review) covers only Shell arguments. main actually executed the wrong call on the file tools, so the fix should cover edit and write too. The +18-line Harness candidate still applies unchanged on this tree: all three calls end in 0.35–0.72 s with turn_complete, no prepare is sent, and the emoji control succeeds.
Sandboxed verification
The triage sandbox reached the same A/B result on this head. I did not re-test its Minor finding (the new recursive isWellFormedJson has no depth limit and throws StackOverflowError at about 7000 levels). All my runs sent HTTP bodies, where fastjson2 rejects nesting deeper than 2049 before that check is reached.
Evidence, rig and logs (publisher tokens redacted): wenshao/qwen-code@f1d6bcdc/pr12975/r2.
中文版
真实环境验证第二轮 —— 0b529019
接续 a2658844 上的第一轮。
结论:仍然可以合并。 第一轮的所有验证在新 head 上重跑,结果相同;在新 head 与当前 main a8ba9b50 合并后的树上也是如此。R1-4 的修复有一点不阻塞的补充:它共享了 cause,但"是否可重试"的分类仍然写了两遍,评审者的见证变异体依然存活。下文给出一个只计算一次该标志的候选写法。
与第一轮相比的变化
3fd42b4d把Throwable cause = unwrap(error);提到失败处理的开头(+2/−2)。0b529019合入了 main5ef79837,树与git merge-tree的结果一致,没有手工改动。它带进了 feat(runtime-broker): reconcile executions on session takeover #12964(takeover 时对账执行)和 test(runtime-broker): Accept the takeover scan's settlement in the durable-local fault gate #13000。takeover 扫描通过状态查询对账 EXECUTING、CANCEL_REQUESTED 和 UNKNOWN 的执行,这个查询不经过本 PR 新增的工具名和输入检查。被拒绝的:start会让执行停在 PREPARED,而扫描不处理 PREPARED,所以没有发现交互问题。- CI 对这个 head 的合并引用
a04301b建在5ef79837上。我另外在 PR 与当前 maina8ba9b50的合并上跑了一遍,它多了 feat(managed-agent): Recover Workspace holders after trusted local reboot (W0e-3) #12869。
新 head 上的复验
| 检查 | 结果 |
|---|---|
| runtime-broker 单测和 Checkstyle | 443 个测试,0 失败(比第一轮多 17 个,来自 #12964),2 个跳过;0 违规 ✅ |
在新合并基点 5ef79837 上跑 PR 的测试 |
与第一轮相同,6 个失败 ✅ |
JdbcRuntimeBrokerMySqlIT 在同一个库上连跑两次 |
main:第二次在 MySQL 8.4.11 和 MariaDB 10.11.18 上都失败,expected: <[1]> but was: <[2]>。PR:4/4 次通过 ✅ |
| Hosted 真实栈(图 1) | 与第一轮相同。main 在 v2 文件工具上执行了错误的调用(a?b 被改掉、写入了 ?);Shell 路径上 Worker 的 digest 校验拒绝了它,会话在 0.76 s 内进入 recovery-blocked。PR:没有任何请求到达 Worker,但轮次分别要 65.8 s(Shell)和 120.7 / 120.8 s(edit / write)才失败,之后会话处于 recovery-blocked,Workspace 租约被占住 ✅ |
| 直接调用 Broker API | PR:工具名、输入键或输入值有问题时返回 400 runtime_reference_invalid,写入 0 行;deferred :start 对转义和原始两种形态都返回 400 runtime_payload_invalid;对照组不变 ✅ |
各 Session 状态下的 installContext(真实 Worker) |
main 对 RELEASING、RELEASED、FAILED 和 drain 中的 binding 都会安装。PR 在发送前拒绝这 4 种(Worker 请求为 0),ACQUIRING 和 READY 仍照常安装 ✅ |
阻塞写入失败时的期限路径(MySQL 触发器 SIGNAL) |
main:suppressed=[]。PR:suppressed=[IllegalStateException(SQLException: rig: recovery block write refused)] ✅ |
| 不可重试失败,阻塞写入卡 2 s,期限 1 s | PR 3/3 次返回原始的不可重试 runtime_provision_failed,说明重构没有改变这里的行为。main 也是 3/3;和第一轮一样,这个竞态只能在单测里强制触发 ✅ |
managed-agent-server(PR 合并 main a8ba9b50) |
187 个单测;ManagedAgentMySqlIT 15/15(MariaDB);Hosted IT 10/10(MySQL 8.4,因 #12869 比 CI 多 1 个);Checkstyle 0 ✅ |
| 变异测试 | 21 个杀死 19 个。M01–M20 与第一轮一致,M17 即审计推迟项 2,M21 见下文 |
R1-4:重构共享了 cause,但分类规则仍写了两遍(不阻塞)
R1-4 的 thread 已被标为 resolved,但处理器里仍然以相反的极性把同一条规则写了两遍(代码见英文部分)。评审者的见证变异体 M21(只改 retryable 那一行)在 0b529019 上依然存活:185 个测试,0 失败。提交说明称这次提升"只保留一处规则表述",这对 unwrap 成立,对分类规则并不成立。
候选写法:在开头只计算一次 retryable,当 !retryable && cause instanceof RuntimeBrokerException failure 时发布:cand-r1-4-single-classification.diff。应用后 runtime-broker 443 个测试 0 失败,Checkstyle 0。这样发布判断和阻塞路径读取的是同一个标志。可以放在本 PR 里改,也可以以后再做;目前没有实际错误。
客户端跟进项
现已由 #12993 跟踪,覆盖所有工具路径。#13010(来自 #12894 的评审)只覆盖 Shell 参数。main 上真正执行了错误调用的是文件工具,所以修复应同时覆盖 edit 和 write。+18 行 Harness 候选补丁在这棵树上无需改动即可应用:三种调用都在 0.35–0.72 s 内以 turn_complete 结束,没有发出 prepare,emoji 对照组照常成功。
沙箱验证
triage 沙箱在这个 head 上得到了相同的 A/B 结论。它报告的 Minor 发现(新的递归 isWellFormedJson 没有深度上限,约 7000 层时抛出 StackOverflowError)我没有复测。我的所有实测都走 HTTP 请求体,fastjson2 会在 2049 层时先拒绝,到不了这个检查。
证据、装置和日志(publisher token 已脱敏)见 wenshao/qwen-code@f1d6bcdc/pr12975/r2。
|
@qwen-code /triage |
qqqys
left a comment
There was a problem hiding this comment.
Critical-only scan — APPROVE
Head reviewed: d2d4d0af5d09aff4dcd786356c187d837c85b4c9
I read the complete production surface: four Java files, +102/−18. No Critical, and no blocking issue has ever been filed here.
Historical blocking issues — none exist
There is no CHANGES_REQUESTED on this PR. Both DISMISSED reviews are approvals that a later push invalidated — their bodies read "LGTM, looks ready to ship" — not withdrawn blockers. The one COMMENTED round, at a2658844, posted a single sev:S finding about re-deriving the retryability classification, and names two further Suggestions already on record. There is no [Critical] inline comment anywhere in this PR's history.
The theme of every production change is fail-closed tightening
BrokerValues. requireWellFormed is refactored onto a private isWellFormed(String) with the identical surrogate-range test, so its behaviour is unchanged. The new isWellFormedJson(Object) accepts only recognized JSON shapes — a well-formed String, a Map whose keys are all well-formed Strings and whose values recurse, a List whose items recurse, and null/Number/Boolean — and answers false for anything else, including an array or a set a writer would still serialize. Rejecting the unrecognized rather than assuming it is the right default for a validator whose job is to stop corrupted text reaching an executor. The recursion adds no new depth hazard: the parsed Map/List tree it walks was already built by a parser that recursed to the same depth.
HttpRuntimeTransport. Three tightenings. installContext now requires sessionRecord.isAcquirable(), so a Session being released installs no context, and adds runtime.isDrainRequested() to the existing binding checks, so a draining binding admits no new Session. execute and executeV3 both route the tool name through the new referenceToolName, applying requireWellFormed, and referenceInput now requires isWellFormedJson. Each is a rejection that previously did not happen.
RuntimeTransport. Javadoc only — the interface's installContext contract is updated to state the ACQUIRING-or-READY session and the no-drain binding preconditions the implementation now enforces. No signature changes, so no implementor is affected.
RuntimeBrokerService. startExecution replaces the raw-text requireWellFormed(payloadJson, …) with a null check plus isWellFormedJson, answering 400 runtime_payload_invalid instead of letting an IllegalArgumentException escape, and then — importantly — re-checks isWellFormedJson(payload) on the parsed map. That second check is the real fix: an escaped unpaired surrogate such as "\ud800" is perfectly well-formed JSON text, so the old raw-text check passed it, and the UTF-8 encoder then delivered '?' to the Worker. createExecution gains the same check on safeReference before admitting, under its own runtime_reference_invalid code.
The defect being closed is worth stating plainly, because it is the reason the strictness is justified rather than merely defensive: a tool name or input containing an unpaired surrogate was silently rewritten to '?' and the Worker would then execute a different tool name or different arguments than the caller sent. Turning that into an explicit 400 is a correctness fix, and no legitimate caller produces unpaired surrogates, so nothing valid is newly refused.
The provisioning-deadline change preserves the real failure, and its ordering holds
provisionDurableBinding adds an AtomicReference<RuntimeBrokerException> nonRetryable. I checked the publication order the comment claims rather than accepting it: the .handle failure handler sets nonRetryable before calling renewal.stopAndGet(), and the deadline task reads it before its own stopAndGet(). Since both contend for the same renewal claim, which can be held across a database round trip, a deadline that runs after the failure handler cannot miss the value, and the AtomicReference supplies the visibility. The answer selection is blocked && failure == null ? conflict("runtime_broker_recovery_blocked") : answer, so a known non-retryable cause is no longer masked by the generic recovery-blocked conflict, while a genuinely timeout-only block still reports as before. blockRecoveryQuietly(timedOut, answer) now records that reason instead of null. Legacy startup is untouched: request.isManagedContext() ? nonRetryable.get() : null keeps its timeout answer.
CI
Green at this head: the full Java matrix (ubuntu-latest / Java 11, 17, 21, windows-latest / Java 21, macos-latest / Java 21), Runtime Broker and Managed Agent MariaDB / Java 21, Hosted process fault gates / MySQL 8.4 / Java 21, Real daemon E2E / Java 11, Test (ubuntu-latest, Node 22.x), Lint & Static, web-shell E2E Smoke and both Desktop Shell lanes all pass. review-pr and Integration Tests (no-AK, No Sandbox) were still pending; neither is a gating check and I did not wait on them. No failure is attributable to this PR.
Coverage note
I read all four production files in full. I did not read the 323 lines of new and changed tests beyond confirming the Java matrix runs them green, nor the two design-doc pairs beyond noting they are +9/−9 symmetric edits in both languages.
yiliang114
left a comment
There was a problem hiding this comment.
LGTM. Verified at head d2d4d0a against the seven deferred findings in #12761 — all seven are addressed, and CI is green at this head (full Java matrix, Hosted process fault gates / MySQL 8.4, Runtime Broker MariaDB, Lint & Static, web-shell E2E).
What I checked beyond the diff:
- Session-state check on install:
RuntimeSessionRecord.isAcquirable()is exactlyACQUIRING || READY, matching the new refusal and the updated docs; the drain-requested refusal rides the existing binding check, and the ACQUIRING acceptance is pinned by a positive test, not just the absence of refusal. - Deadline/failure-handler race: the publish uses an
AtomicReferenceset before the handler takes the claim (safe across the DB round trip), the deadline reads it only for managed-context requests, and legacy startup still answers the plain timeout — all three orderings are pinned by the new race tests. - Suppressed-failure path: removing the null guard in
blockRecoveryQuietlyis safe — all three call sites pass a non-null cause (deadline passesanswer, which is always non-null; the handler passescauseinsideerror != null, andunwrapnever returns null for non-null input). Both paths now havegetSuppressed()assertions. - Surrogate refusals: the reference-identity check fires before the whole-reference check (message pinned in the test), the deferred payload check catches JSON-escaped surrogates post-parse, and the fail-closed
isWellFormedJsonrecursion is bounded in practice by the 256 KiB payload limit and upstream JSON parser depth limits. The well-formed-Unicode round-trip test covers the no-false-positive direction, including null values. - The prior bot suggestion (R1-4, duplicated retryability classification) is partially addressed at this head —
causeis now computed once — and what remains is style-level, already on record.
After merging main: - The Broker refuses a provider control operation with an unpaired surrogate in any key or string (400 runtime_control_operation_invalid), as #12975 does for references and deferred payloads. The JSON writer would otherwise send it as '?', a shell wildcard. - The provider worker refuses a run_shell_command whose directory lies outside the Session's workspace, as the tool executor does since #12927. Core's shell tool now asks there, and a preapproved Session never asks. - Provider status cursors are read as exact integers, as #12972 reads the others.
* feat(serve): implement generic Broker provider controls * fix(runtime): preserve broker provider validation errors * fix(runtime): restore admission and unknown observation semantics * fix(serve): keep oversized provider results observable and tighten the envelope Fit execute/status/cancel results into the per-kind wire budget instead of failing validation: evict oldest progress events, then cut bulk text fields head-and-tail with an inline notice (shell displays set `truncated`), so a legitimately large tool result keeps a terminal observation and release stays answerable instead of stranding the execution UNKNOWN behind a 400. Also address the round-2 review: - Require lowercase UUID envelope Session ids, closing the raw-id path to core's session id and the file-history directory name. - Emit policy refusals as managed_runtime_provider_operation_failed instead of the identity-fencing code; drop the unreachable turnStarted guard; skip the redundant operation-level digest on the composed request path; enforce the declared response bound and tie it to the per-kind limits. - Broker: answer an unencodable control with a definitive 413, keep a null-valued deferred reference on the stage channel as 400, and report the prepare envelope violation as runtime_broker_invalid_request. Hoist the protocol's per-call pattern compiles. - Pin the changes with real-worker, protocol, transport and service tests, extend the shared corpus to every accepted confirm outcome, and refresh the affected design docs in both languages. The immediate-route payload remainder is tracked by QwenLM#12936 and the release-ordering strand by QwenLM#12937. * fix(cli): narrow the status binding before progress eviction The fitting path read status fields inside a guard that only proved progress is an array, which tsc --build rejects under strictNullChecks. Narrow the binding explicitly. No runtime behavior change. * fix(runtime): keep terminal cancellation receipts answerable without a READY Session A settled cancel of a prepared provider invocation was re-driven through requireReadySession on every retry, so once the Session was lost, releasing or absent the duplicate cancel failed 404/409 forever even though the terminal answer was already durable. Return the stored receipt unless the owner Session is READY and can actually refresh the worker evidence, and tolerate a missing binding instead of throwing a 500. Round-4 review follow-ups: the Broker client's control closure now rejects asynchronously instead of throwing synchronously; new pins cover the exact 4096-byte reason boundary, the provider rejection codes, a non-string payloadJson, the in-flight checkpoint guard, in-flight snapshot and preparation release fences, provider reconciliation on terminal evidence, rejected cancellation evidence (unknown / settled-success), the acquire prelude guard, the confirm outcome and modality refusals, negative status shapes, and corpus cases for content modification, media context and the continuation turn kind. * fix(runtime): fail a prepared-provider cancel non-retryably when it can never confirm observePreparedCancellation mapped the worker's permanent answers — an unknown invocation (no longer retained) or a settled result whose status is not cancelled/not_started — to a retryable 503, so the cancel retry looped the identical round trip forever against a record already settled locally. Both now fail with a non-retryable 409 runtime_execution_cancel_unconfirmed. unknown is never turned into not_started evidence; the local receipt is unchanged. * fix(serve): admit path-safe opaque Session ids on the provider envelope The lowercase-UUID envelope gate broke the Broker's own fault gates, which drive the provider route with opaque ids (harness-1 / runtime-session-1) and represent the wire contract's real compatibility surface: non-identity operations (acquire/release/manifest) never needed the UUID form. The gate now refuses only what is unsafe to interpolate into file names — separators, dot segments, control characters, unpaired surrogates — while identity-bearing operations keep requiring core's UUID form through the identity check. * fix(serve): address the round-3 review of the Broker provider controls Result fitting (R3-9): - Measure the cut in JSON-encoded UTF-8 bytes and remove whole code points, so CJK text is no longer over-cut, surrogate pairs are never split and the notice counts the omitted code points exactly. - Cut every field once, to one common size that accounts for each field's notice, so no field is emptied while another keeps its text and the budget is met in one pass; a field shorter than its notice is never grown. - When even fully cut text cannot fit, stub a structured display (an edit's file diff) and then drop hook results before cutting text, so the model keeps its own content; only unreachable content (inline media) turns llmContent into a stub. Broker (R2-1 and the provider-envelope id rule): - A repeated cancellation of a settled prepared-provider call answers from its receipt once the Session is not READY or its binding can no longer answer, and asks for adoption (503 runtime_reconciliation_required) while this process holds no live Session for it, instead of 404 runtime_session_not_found. - Refuse at acquire the Session ids the worker envelope refuses, so no Session can be acquired that cannot be released. Worker: - Envelope Session ids are an ASCII allow-list (1-512 of A-Za-z0-9._-, no '.'/'..' segment), pinned character by character in TS and Java. - Release and shutdown drop core's per-session project-dir and model registry entries for the Runtime Session. - Refuse prepare with content modification explicitly (400 managed_runtime_tool_invalid): this profile has no notebook_edit. Tests pin the legacy-admission re-checks after each await, the composed envelope bound, the eviction direction, the corpus marker, the provider decoders' rejection arms, the control dispatch and a result-bearing control over HTTP. Design docs are updated in both languages. * fix(serve): address the round-4 review of the Broker provider controls Worker (R4-1): - Keep the full state of up to eight released Sessions. When another is released, retire the oldest one not being observed: dispose its runtime, drain its file history and shut its Config down without telemetry or the session writer, then drop the references, leaving a tombstone whose status and cancel answer unknown. Each step runs even if an earlier one fails, calls keep reaching the runtime until the end, and close() waits for retirements in flight. A failed retirement is logged and never fails the release that triggered it. - A release that came before any acquire leaves the same tombstone. Protocol and client: - confirmation results require each variant's fields as core emits them (R4-4). - The Broker client keeps the reason of an error that carries details, such as a terminal answer (R4-3). - The docs say acquire and release results are exactly true (R4-2). Broker: - warm applies the Session-id allow-list to the Harness id. - The response buffer grows with the body up to the cap instead of reserving the cap (R4-5). - The terminal cancel branch reuses the binding and Session rows its ownership check loaded, after that check (R4-6). Tests pin a repeated cancellation after dispatch (R4-7), a lost immediate execute response in the fault gates (R4-8), request headers asserted on the test thread (R4-9), ids with letters and digits outside ASCII, and the retirement rules, including close() racing a retirement and an acquire. Design docs are updated in both languages. * docs(runtime-broker): count every fault gate the profile runs After merging main, mvn -Pfault-gates also runs the local-reboot and durable-worker gates: 44 in about 4 minutes, measured on this tree. * fix(serve): keep main's input rules on the provider path After merging main: - The Broker refuses a provider control operation with an unpaired surrogate in any key or string (400 runtime_control_operation_invalid), as QwenLM#12975 does for references and deferred payloads. The JSON writer would otherwise send it as '?', a shell wildcard. - The provider worker refuses a run_shell_command whose directory lies outside the Session's workspace, as the tool executor does since QwenLM#12927. Core's shell tool now asks there, and a preapproved Session never asks. - Provider status cursors are read as exact integers, as QwenLM#12972 reads the others. * fix(serve): reconcile lost provider answers over Broker HTTP - Broker HTTP start, read and cancel reconcile a provider execution left UNKNOWN by a lost execute answer, as they do for tool v3, so the worker's retained result settles it without a second dispatch instead of answering 409 runtime_broker_execution_unknown. - The provider worker's shell tool refuses a directory outside the workspace when the call is prepared and again just before it runs, resolving the path afresh the way the kernel follows it. A link retargeted in between settles the call as an error. * fix(serve): fit provider artifacts and pin the retention bound - The result fitter drops artifacts, which only feed a client surface, after stubbing a structured display and before dropping hook results, when even fully cut text cannot fit beside them. A result whose bulk sits in an artifact now fits without stubbing the model content. - Tests pin that the Session just released is retired when every retained one is being observed, so no release leaves more than eight, and that a zero response limit completes truncated. * fix(runtime-broker): cancel a lost provider dispatch at its worker A provider execution whose execute answer was lost is UNKNOWN with no invocation running in this process. Cancelling it skipped the worker, because only tool v3 references counted as answerable after a loss, so the Broker reported cancel_requested while the tool kept running. ToolExecutionRecord.observableAfterLoss() now states that rule once, for tool v3 and provider references alike, and both the cancel path and Broker HTTP observation use it.





What this PR does
Closes the seven findings deferred from #12730 (W0c-2 managed-context Broker) and tracked in #12761.
runtime_broker_recovery_blockedthere). A failure that arrives after the deadline fired, and legacy startup, keep exactly the old answers.?, so the Worker used to run a different call than the one requested, and?is a wildcard in a shell command. The Broker now refuses such a tool name, or any key or string anywhere in the tool input: 400runtime_reference_invalidwhen an execution is created (before it is admitted), 400runtime_payload_invalidwhen a deferred execution starts (for a raw surrogate and for one written as a JSON escape, which passes a check on the text), and an argument error before sending in the HTTP transport, whose check also fails closed on values that are not JSON. Well-formed Unicode, surrogate pairs and nulls are unchanged.Why it's needed
These were left open when #12730 merged after three audit rounds. Two of them were decisions: for the block race I kept the original non-retryable error rather than narrowing the documented guarantee, and for tool names and input I chose to refuse unpaired surrogates, since silently executing
rm file?instead of the requested command is worse than a refusal.Reviewer Test Plan
How to verify
main: the installation refusals, the deadline keeping a non-retryable answer, the suppressed failure on the deadline path, and every tool name or input refusal.mainthe second run fails withexpected: <[1]> but was: <[2]>; with this change both runs pass.Evidence (Before & After)
N/A (no user-visible UI). Local results:
runtime_session_acquire_failed), which shows that production installs while the record is ACQUIRING.mainthe second run fails.main: seven of the new assertions fail there. A mutation sweep of the new checks (16 mutants, including the two named in the issue: droppingaddSuppressedand narrowing the check to high surrogates) was killed in full.Tested on
Environment (optional)
JDK 25 (module release 21), Maven 3, Node 22 for the fault gates and Hosted integration tests, Docker MySQL 8.4 and MariaDB 10.11.18.
Risk & Scope
?in place of the surrogate. Handling that refusal on the client is left to a follow-up, listed in the audit comment.runtime_payload_invalidinstead of 400runtime_broker_invalid_request. Nothing else changes for valid requests.Design document updated in both languages: English · 中文
Linked Issues
Closes #12761
中文说明
这个 PR 做了什么
处理 #12730(W0c-2 managed-context Broker)推迟、并由 #12761 跟踪的七条发现。
runtime_broker_recovery_blocked)。期限触发之后才到达的失败,以及 legacy 启动,应答与以前完全相同。?,因此 Worker 执行的调用与请求的不同,而?在 shell 命令里是通配符。Broker 现在拒绝这样的工具名,以及工具输入中任意位置的键或字符串:创建 execution 时返回 400runtime_reference_invalid(在准入之前);启动 deferred execution 时返回 400runtime_payload_invalid(包括原始代理项,以及写成 JSON 转义、能通过文本检查的代理项);HTTP transport 在发送前抛出参数错误,其检查对非 JSON 的值也一律拒绝。格式正确的 Unicode、代理对和 null 不受影响。为什么需要
这些问题在 #12730 经过三轮审计合入时被留下。其中两条需要决策:对于阻塞竞态,我选择保留原来的不可重试错误,而不是收窄文档中的承诺;对于工具名和输入,我选择拒绝未配对代理项,因为静默执行
rm file?而不是所请求的命令,比拒绝更糟。评审测试计划
如何验证
main上会失败:安装拒绝、期限保留不可重试应答、期限路径上的 suppressed 异常,以及每一种工具名或输入拒绝。main上第二次运行失败,报expected: <[1]> but was: <[2]>;本改动下两次都通过。证据(前后对比)
不适用(无用户可见 UI)。本地结果:
runtime_session_acquire_failed),说明生产环境确实在记录为 ACQUIRING 时安装。main上第二次失败。main上做 A/B:新断言中有七个在那里失败。对新增检查的变异扫描(16 个变异体,包括 issue 点名的两个:删除addSuppressed、把检查收窄到高位代理项)全部被杀死。测试平台
环境(可选)
JDK 25(模块 release 21)、Maven 3;fault gates 和 Hosted 集成测试使用 Node 22;Docker 中的 MySQL 8.4 和 MariaDB 10.11.18。
风险与范围
?代替代理项被执行。在客户端处理这个拒绝留作后续工作,已列在审计评论中。runtime_payload_invalid,而不是 400runtime_broker_invalid_request。有效请求的行为没有变化。设计文档已同步更新两种语言:English · 中文
关联 Issue
Closes #12761