Repository navigation
feat(sdk-java): Adopt a Managed Runtime only after attestation - #12552
Conversation
The Broker starts the merged worker over stdin and stores the lease as ready only after the process attests. A later use attests again, and tool calls against the attestation-only worker fail closed.
qqqys
left a comment
There was a problem hiding this comment.
Blocking: this diff turns Lint & Static red — 9 no-undef errors in the new .mjs test resource
Where
packages/sdk-java/runtime-broker/src/test/resources/fake-attestation-worker.mjs — the file this PR adds. It does not exist at the base 94104d5565b1; I confirmed that arm on its own route as an HTTP 404, so it is a proven absence rather than an unchanged file.
What CI measured at this head (a20ea2910d09f6b599bdee5c71fe7f56aa0ebe2e)
Lint & Static (ubuntu-latest, Node 22.x) → completed / failure, 12:00:18Z → 12:07:07Z (409 s), check-run id 107170403311.
The failing step is Run ESLint (step 22). Step 8 Check lint gate freshness succeeded and step 14 Install dependencies succeeded, so this is neither the sub-minute freshness gate nor a branch-staleness abort — ESLint really ran, and it reported:
packages/sdk-java/runtime-broker/src/test/resources/fake-attestation-worker.mjs
5:3 error 'process' is not defined no-undef
6:3 error 'process' is not defined no-undef
6:41 error 'Buffer' is not defined no-undef
7:3 error 'process' is not defined no-undef
14:21 error 'URL' is not defined no-undef
48:3 error 'process' is not defined no-undef
59:39 error 'process' is not defined no-undef
60:1 error 'process' is not defined no-undef
61:1 error 'process' is not defined no-undef
✖ 9 problems (9 errors, 0 warnings)
##[error]Process completed with exit code 1.
Run ESLint is node scripts/lint.js --eslint → npm run lint:ci → eslint . --ext .ts,.tsx --max-warnings 0, i.e. zero tolerance.
I re-read all nine positions against the byte-faithful blob at this head (sha c4439a93127b86f6b99c891f45ce72f02c612832, 1,975 B, 62 lines; recomputed sha1('blob ' + len + '\0' + bytes) equals the API's own sha, NUL_bytes=0), and every line and column matches exactly: :5/:6/:7 are the process.stdin.on(…) triple, 6:41 is Buffer.concat, 14:21 is new URL(request.url, 'http://127.0.0.1'), :48 is process.stdout.write, 59:39 is process.exit(0), and :60/:61 are the two process.once(…) handlers. The file carries no eslint-disable comment and no globals declaration.
Why this is the PR and not main
A sibling arm on the same base: #12549 (base 94104d5565b1, head committed 11:41:50Z) ran the identical lane green — Lint & Static (ubuntu-latest, Node 22.x) completed / success, 11:42:48Z → 11:54:36Z — roughly 18 minutes before this PR's run failed. Same base, same runner family, same hour, opposite conclusion, so the cause is inside this diff.
Mechanism
In eslint.config.js (blob-identical at this head and at my checkout: 9d3322575016424f349eac56281efe41430a801e, 21,191 B) the block that injects Node globals is scoped to TypeScript only:
files: [
'packages/**/src/**/*.{ts,tsx}',
'integrations/**/src/**/*.{ts,tsx}',
],
languageOptions: { globals: { ...globals.node, ...globals.es2021 } },The new file sits under packages/**/src/**, so it is linted, but its .mjs extension matches no languageOptions.globals block — so no-undef fires on every Node global. The config's global ignores list does not exclude packages/sdk-java/** or **/test/resources/**.
I did not establish why other .mjs files under packages/ that also use a bare process (for example packages/node-repl/src/runtime/kernel.mjs) are not reported. That is unmeasured, and nothing in this finding rests on it — the finding rests on the executed lane and the byte-faithful blob.
Impact
Two things, and the second is the one I'd weigh more heavily:
- The lane is deterministically red at this head — ESLint emitted a complete enumerated error set and the step exited 1, so this is not a flake and a re-run will not help. AGENTS.md requires
npm run lintto pass before a change is called done. - Because step 22 failed, steps 23–35 were all
skipped:actionlint,shellcheck,yamllint, Prettier, the sensitive-keyword linter, the i18n check, settings-schema generation and its up-to-date drift check, the VS Code companion notice check, the serve fast-path bundle closure, core subpath exports resolution, and the.github/scriptshelper tests. So none of those has been evaluated against this diff at all — the red lane is currently masking the whole static surface behind it.
To be precise about what I am not claiming: main's ruleset has no required_status_checks rule, so branch protection would not mechanically refuse the merge. I am filing this as blocking because the red is caused by this diff and because it silently skipped thirteen downstream checks, not because the platform enforces it.
Fix direction
Any one of these clears it, and the choice is yours:
- Give
.mjsunderpackages/**/src/**Node globals — extend the existing block'sfiles:(or add a sibling block) with something like'packages/**/src/**/*.mjs'andlanguageOptions: { globals: { ...globals.node } }. Smallest change, and consistent with how.ts/.tsxalready get them. - Treat the file as what it is — a test resource, not source — and add its path to the global
ignoresineslint.config.js. - Declare the globals in the file (
/* global process, Buffer, URL */). Most contained, least consistent with the rest of the repo.
Options 1 and 2 also settle the policy for the next .mjs fixture, which is probably worth deciding deliberately rather than per file.
What I am not blocking on
I read the rest of the production delta (all four Java files) and found nothing else I'd call merge-blocking. The two things I checked hardest both hold up:
- The
confirmdefault really is a no-op, and that is what the PR body and the design doc say is intended ("Provisioners that do not overrideconfirmkeep the previous ready path"; "it does not add a server"). There is also no productionRuntimeProvisionerimplementation onmainfor it to regress — only the test fake atRuntimeBrokerServiceTest.java:1026— so this is not a dead switch, it is an unwired slice that says so. - Every failure path I traced in
LocalProcessRuntimeProvisionerdestroys the process before the exception escapes, andowned.put(…)happens only afterattestsucceeds.
qwen-code-ci-bot's stage-2 review is the reason this is worth filing separately. Its prose was written while the lane was still in flight and still says so — "Four lanes were still running at review time (the Node-side unit, lint, and integration jobs plus the Java 11 daemon E2E). I am reporting them as pending rather than guessing an outcome." Its <!-- qwen-triage-ci --> status table was subsequently edited (updated_at 2026-09-23T12:37:36Z, body grown 13,791 → 14,094 chars) and now does show Lint & Static (ubuntu-latest, Node 22.x) | ❌ failure, so the red is visible there. But no part of that review diagnoses it: in its full 14,094-char body the strings no-undef, ESLint and 9 problems each occur 0 times, and it filed no review row at all (reviews rows=0, so reviewDecision is still REVIEW_REQUIRED). What is missing is the part that decides what to do: that all nine errors are no-undef on Node globals, that they are all in the file this PR adds, that a same-base sibling was green 18 minutes earlier, and that thirteen downstream static checks were skipped as a result.
Its erasure point about HttpRuntimeTransport.execute versus RuntimeTransport.execute is factually correct (RuntimeTransport.java:23 declares CompletionStage<Map<String, Object>>; the new HttpRuntimeTransport.java:120 declares CompletionStage<Void>), but it is latent rather than blocking — HttpRuntimeTransport.java:25 is public final class HttpRuntimeTransport { with no implements clause, so nothing conflicts until a later slice wires that interface.
Reviewed at a20ea2910d09f6b599bdee5c71fe7f56aa0ebe2e. Evidence: check-run 107170403311 job log (236,926 B, read as bytes with NUL_bytes=0, ANSI-stripped before matching); byte-faithful blobs for fake-attestation-worker.mjs and eslint.config.js, each with its recomputed blob sha equal to the API's; the live check-run census for #12549 as the same-base sibling arm; and ci-bot's comment 5794738330 re-read from the owning route at post time.
ESLint treats the test script as project JavaScript, so process, Buffer, and URL cannot be used as undeclared globals.
|
Fixed the Lint & Static failure in f57765b.
|
|
Two CI results since f57765b:
|
Local verification against the real worker — head
|
| # | Scenario | Result |
|---|---|---|
| S1 | warm() spawns the worker, writes the boot doc, reads ready, attests |
READY in ~180 ms. Listens on 127.0.0.1 only. |
| S2 | 20× warm() on a READY binding |
Each one re-attests over HTTP: p50 2.0 ms vs 0.005 ms without confirm. This runs per warm/new-session acquire, not per tool call. |
| S3 | HttpRuntimeTransport.execute against the real worker |
404 managed_runtime_incompatible, retryable=false, fails closed as described |
| S4a/b | stop(lease) → next warm() |
Worker exits on SIGTERM. warm → 503 runtime_provision_failed |
| S5 | external kill -9 of the worker → next warm() |
503, fails closed |
| S6 | provisioner.close() |
Ends all owned workers. service.close() alone does not, so the owner has to close both. |
| S7 | 8 workspaces warmed in parallel | 8 distinct attested workers in 127 ms, none left after close |
| B1 | Worker that never prints ready | Refused at 30.1 s, process gone, and the next warm re-provisions |
| B2 | Real worker behind a ready record with a foreign leaseId |
ready record is invalid, wrapper + worker gone |
The same checks on JDK 21 give identical results. The Java↔TS boot/ready contract matches field for field against the real binary, which settles the "double could drift" gap from triage.
Findings
F1 — A dead worker bricks its binding for the Broker's lifetime, while the error says retryable (fix before wiring; not a regression).
confirm failure leaves the record READY and liveBindings populated. ensureBinding only provisions from PROVISIONING, so nothing ever replaces the dead worker. Three arms, same scenario:
- Arm A:
mainbehaviour (head with theRuntimeBrokerServicehunk reverted). After the worker is gone,warm()returns the READY lease 5/5 times, pointing at a dead port. This is the bug the PR fixes. - Arm B: head. It is refused correctly, but 5/5 retries give
503 runtime_provision_failed retryable=trueand 0 workers are spawned. Until a restart or reconcile, the caller is told to retry something that cannot succeed. - Arm C: head + patch. The first retry provisions a new generation, and the next 5 warms are OK.
The patch has two parts:
- On
confirmfailure the service claims the operation, CASes READY→FAILED and dropsliveBindings. - The provisioner stops the process it failed to confirm.
The extended PR test is RED on head (Managed Runtime process is not alive. on the recovery step) and GREEN with the patch. With the patch plus the follow-up tests the full suite is 83/83 and checkstyle is clean. Two caveats:
- Recovery was exercised with the in-memory repository only.
- A transient attest failure now retires a healthy worker instead of sticking. I think that trade is right for a fail-closed gate, but it is the owner's call.
Patch (production part, +51/−7)
--- a/.../RuntimeBrokerService.java
+++ b/.../RuntimeBrokerService.java
@@ ensureBinding
BindingContext live = requireLiveBinding(record);
return safeStage(() -> provisioner.confirm(request, live.lease()))
- .thenApply(ignored -> live);
+ .handle((ignored, error) -> {
+ if (error == null) {
+ return live;
+ }
+ retireUnconfirmedBinding(record);
+ throw new CompletionException(unwrap(error));
+ });
@@
+ /**
+ * A READY binding whose Runtime no longer attests cannot recover in
+ * place. Retire it so the next ensureBinding provisions a new generation.
+ */
+ private void retireUnconfirmedBinding(RuntimeBindingRecord record) {
+ RuntimeBindingRecord claimed = bindingRepository.claimOperation(
+ record.getBindingId(), brokerOwnerId, operationLeaseDuration);
+ if (claimed == null
+ || claimed.getState() != RuntimeBindingRecord.State.READY
+ || claimed.getGeneration() != record.getGeneration()) {
+ return;
+ }
+ if (bindingRepository.compareAndSet(claimed, claimed.withState(
+ RuntimeBindingRecord.State.FAILED, null, clock.instant()))
+ != null) {
+ liveBindings.remove(record.getBindingId());
+ }
+ }
--- a/.../LocalProcessRuntimeProvisioner.java
+++ b/.../LocalProcessRuntimeProvisioner.java
+ private static final Duration STOP_GRACE = Duration.ofSeconds(5);
@@ stop()
- process.process.destroy();
+ terminate(process.process);
+ private static void terminate(Process process) {
+ process.destroy();
+ process.onExit().orTimeout(STOP_GRACE.toMillis(),
+ TimeUnit.MILLISECONDS).exceptionally(ignored -> {
+ process.destroyForcibly();
+ return null;
+ });
+ }
@@ start()
+ URI endpoint = URI.create(String.valueOf(ready.get("url")));
+ if (!"127.0.0.1".equals(endpoint.getHost())) {
+ throw failed("Managed Runtime ready record is invalid.");
+ }
RuntimeLease lease = new RuntimeLease(runtimeInstanceId,
- URI.create(String.valueOf(ready.get("url"))), token,
- leaseId, 1);
+ endpoint, token, leaseId, 1);
@@ both catch blocks
- ownedProcess.process.destroy();
+ terminate(ownedProcess.process);
@@ attestOwned()
if (process == null || !process.process.isAlive()) {
+ stop(lease);
throw failed("Managed Runtime process is not alive.");
}
- attest(request, process.seed, lease);
+ try {
+ attest(request, process.seed, lease);
+ } catch (RuntimeException exception) {
+ stop(lease);
+ throw exception;
+ }Full patch: patches/fix-ac63adc4.patch. Combined with the tests: fix-plus-tests-ac63adc4.patch.
F2 — Failure paths only send SIGTERM (hardening). A worker that refuses attestation and ignores SIGTERM is still alive 6.5 s after the failure. Each retry adds another one: 2 alive after the second warm. The merged worker does exit on SIGTERM, so this only bites a hung worker. The patch above adds a 5 s grace and then destroyForcibly() (B3 → 0 alive).
F3 — A non-loopback url in the ready record receives the bearer token and identity (hardening; confirms triage's question). A double that reports http://172.16.1.234:<port> got Authorization: Bearer … plus tenant/workspace/cwd/capability digest. The patch pins the host to 127.0.0.1, which is the only address the worker binds, and the credentials are then never sent (B4).
F4 — A Broker crash orphans the worker (follow-up, worker side). I kill -9ed a JVM that had provisioned a real worker. 5 s later the worker had been re-parented to PID 1 and was still listening with a live token. stdin is closed right after the boot doc, so the worker has no lifeline, and Java cannot clean up after its own crash. That needs a worker-side parent watch or a held pipe. It was already visible in #12506 and now has an owner that can die.
F5 — The execute half: I confirm triage's facts. Keeping it here or not is the maintainers' scope call.
- The erasure clash is real. Making
HttpRuntimeTransport implements RuntimeTransportfailsjavacwithreturn type CompletionStage<Void> is not compatible with CompletionStage<Map<String,Object>>. - The real-worker 404 is reported as
Managed Runtime attestation endpoint is incompatible., which is misleading for a tool call. - Mutant M14, which sends execute to a different unmounted path, survives every test, because nothing pins the execute wire shape.
I'd drop it from this slice, or rename it, and land it with the tool-execution contract.
Test strength (mutation)
The PR's tests kill 2/14. Only the re-attest path is pinned. "No attestation before READY" (M03), every ready-record identity check (M04–M08), and all teardown paths (M09–M11) survive.
The follow-up tests (followup-tests-ac63adc4.patch, +87/−6) add these to the fake worker:
- a
modeargument (wrong-type|wrong-instance|wrong-incarnation|wrong-lease|wrong-epoch|no-attest) - a pid file, so each refusal also asserts the process exited, and so do
stop()andclose()
That brings kills to 11/14. M12 and M13 are equivalent, and M14 is the unpinned execute route from F5. The fixture stays eslint/Prettier clean.
CI
The two red lanes are infra, not this diff:
Integration Tests (no-AK)failed inVerify checkout includes expected head commitbefore any test ran.web-shell E2E Smokewas cancelled at 20 min with the web server refusing on :4170.
All Java lanes (11/17/21, macOS, Windows, MariaDB) and Lint & Static are green at ac63adc4.
Evidence (harness sources, raw logs, patches, mutant driver): wenshao/qwen-code@asserts/pr-12552
中文版
基于真实 worker 的本地验证 —— head ac63adc4
结论:作为库切片可以合入。但在任何生产代码构造 LocalProcessRuntimeProvisioner 之前,有一个问题应先修。
PR 的核心主张在真实的 qwen managed-runtime-worker 二进制上成立,不只是在 Node 替身上成立:
- worker 能被拉起、完成证明、进入 READY;
- 复用内存 lease 时会重新证明;
- 被 stop 或被
kill -9的 worker 会被拒绝。
同一场景在 main 上每次都会把死掉的 lease 交回去。
缺口在恢复。worker 死后,它的 binding 在 Broker 整个生命周期内一直是 READY。每次重试都返回 503 retryable=true,而且永远不会再拉起新 worker。我准备了修复补丁(+61/−8,已验证),以及测试补丁(+87/−6,变异杀伤从 2/14 提到 11/14)。
qqqys 的阻塞项已修复:head 的 .mjs 通过 eslint --max-warnings 0 和 Prettier;本地用 a20ea29 仍能复现那 9 个 no-undef 错误。那条 CHANGES_REQUESTED 需要重新评审或 dismiss。
环境
- 真实 worker 取自已构建的 bundle(
node dist/cli.js managed-runtime-worker)。worker、契约和cli.ts源码与本 head 逐字节一致(git diff --quiet已确认)。 - 用 Java harness 直接驱动 PR 自己的
LocalProcessRuntimeProvisioner+RuntimeBrokerService+HttpRuntimeTransport,没有用 mock。 - JDK 25(宿主)和 JDK 21.0.12(temurin 容器,即模块的
release),Node 22。 - 模块测试两个 JDK 下都是 76/76、0 跳过,checkstyle 干净。
成立的部分(真实 worker)
| # | 场景 | 结果 |
|---|---|---|
| S1 | warm 拉起 worker、写 boot、读 ready、完成证明 | 约 180 ms 进入 READY,只监听 127.0.0.1 |
| S2 | 对 READY binding 连续 warm 20 次 | 每次都走 HTTP 重新证明:p50 2.0 ms,去掉 confirm 时为 0.005 ms。开销按 warm / 新会话计,不按工具调用计 |
| S3 | 对真实 worker 调 execute | 404 managed_runtime_incompatible、不可重试,与描述一致 |
| S4/S5 | stop 或外部 kill -9 后再 warm |
503,失败关闭;worker 收到 SIGTERM 会退出 |
| S6 | provisioner.close() |
结束全部 worker。只调 service.close() 不会结束,属主需要两个都关 |
| S7 | 8 个工作区并行 warm | 127 ms 内得到 8 个独立且已证明的 worker,关闭后无残留 |
| B1 | worker 始终不输出 ready | 30.1 s 被拒并被杀,下一次 warm 会重新 provision |
| B2 | 真实 worker 套上带假 leaseId 的 ready 记录 |
被拒,包装进程和 worker 都已退出 |
JDK 21 下结果相同。Java↔TS 的 boot/ready 契约在真实二进制上逐字段对上,triage 担心的"替身可能漂移"这一点得到了验证。
发现
F1:worker 死后 binding 在 Broker 生命周期内永久不可用,报错却标为可重试。 应在接线前修复;不是回归。
原因是 confirm 失败后记录仍为 READY,liveBindings 也还在,而 ensureBinding 只会从 PROVISIONING 状态发起 provision,所以死掉的 worker 永远没有东西来替换。三臂对照:
- A:
main行为(回退 service 那段改动):worker 死后 warm 5/5 次都返回指向死端口的 READY lease。这正是本 PR 修掉的问题。 - B:head:能正确拒绝,但 5/5 次重试都是
503 retryable=true,新拉起的 worker 为 0。 - C:head + 补丁:第一次重试就拉起新一代,之后 5 次 warm 全部成功。
补丁分两部分:
confirm失败时,service 先 claim operation,再用 CAS 把 READY 改为 FAILED,并移除liveBindings;- provisioner 停掉证明失败的那个进程。
扩展后的测试在 head 上是红的(恢复那一步报 Managed Runtime process is not alive.),打上补丁后是绿的。修复加测试后全量 83/83,checkstyle 干净。两点说明:
- 恢复路径只在内存 repository 上验证过;
- 一次瞬时证明失败现在会让健康的 worker 退役,而不是卡住。对失败关闭的闸门来说我认为这个取舍合理,但由负责人决定。
F2:失败路径只发 SIGTERM(加固)。 一个拒绝证明、同时无视 SIGTERM 的 worker,在失败 6.5 s 后仍然存活,而且每次重试再多一个(第二次 warm 后存活 2 个)。已合入的 worker 会响应 SIGTERM,所以只有挂死的 worker 才会触发。补丁改为给 5 s 宽限,之后 destroyForcibly(),B3 存活数变为 0。
F3:ready 记录里的非 loopback url 会收到 bearer token 和身份信息(加固,印证 triage 的疑问)。 上报 http://172.16.1.234:<port> 的替身收到了 Authorization: Bearer …,以及租户、工作区、cwd 和 capability digest。补丁把 host 固定为 127.0.0.1(worker 唯一会绑定的地址),之后凭据不会再发出(B4)。
F4:Broker 崩溃会让 worker 成为孤儿(后续项,worker 侧)。 kill -9 掉已经拉起真实 worker 的 JVM,5 s 后 worker 已被 PID 1 收养,仍在监听,token 仍然有效。boot 写完后 stdin 就关闭了,worker 没有生命线;Java 也无法在自己崩溃后做清理。这需要 worker 侧监视父进程,或保持一根管道。该问题在 #12506 就已存在,现在它有了一个可能会死掉的属主。
F5:execute 那一半。 triage 说的事实我都复核成立,是否留在本切片由维护者定:
- 擦除冲突属实:让
HttpRuntimeTransport implements RuntimeTransport,javac 报返回类型不兼容; - 真实 worker 返回的 404 被报成 "attestation endpoint is incompatible",对工具调用来说有误导;
- 变异 M14(把 execute 发到另一个未挂载路径)在所有测试下都存活,因为 execute 的线上格式没有被任何东西约束。
我倾向于从本切片移除或改名,随工具执行契约一起落地。
测试强度(变异)
PR 自带测试只杀死 2/14,只钉住了重新证明那条路径。以下变异全部存活:
- M03:READY 之前不做证明;
- M04–M08:ready 记录的全部身份校验;
- M09–M11:所有进程清理路径。
跟进测试(+87/−6)给 fake worker 增加了:
mode参数(wrong-type|wrong-instance|wrong-incarnation|wrong-lease|wrong-epoch|no-attest);- pid 文件,用来断言每种拒绝之后进程都已退出,
stop()和close()之后也一样。
杀伤率提到 11/14。M12、M13 是等价变异,M14 就是 F5 里没被钉住的 execute 路由。fixture 仍然通过 eslint 和 Prettier。
CI
两条红 lane 都是基础设施问题,与本 diff 无关:
Integration Tests (no-AK)在任何测试开始前就失败于 checkout 未包含 head;web-shell E2E Smoke在 20 分钟时被取消,web server 在 :4170 拒绝连接。
ac63adc4 上全部 Java lane(11/17/21、macOS、Windows、MariaDB)和 Lint & Static 均为绿。
证据(harness 源码、原始日志、补丁、变异驱动):wenshao/qwen-code@asserts/pr-12552
🤖 Generated with Claude Code — Claude Opus 5.5 (1M context)
wenshao
left a comment
There was a problem hiding this comment.
4 Suggestion-level finding(s) this review confirmed are already reported on this PR and are not repeated:
- worker stderr discarded via Redirect.DISCARD, so no boot failure carries the worker's own reason - already reported (comment 4085208800)
- ready record read with an unbounded readLine() - already reported (comment 4085208805)
- attest flattens the transport's non-retryable classification into a retryable 503 - already reported (comment 4085208855)
- adoption test does not observe the attestation it claims to prove - already reported (comment 4085208875)
[Critical] Carried forward from the earlier review round on this PR (its id there: R1-1), and still standing at ac63adc: provision adopts a live OS process and hands back a lease the caller is free to discard, while the only per-lease teardown (stop) is package-private with no production caller and RuntimeProvisioner exposes no release verb. Independently re-confirmed here: after service.close() the discarded worker was still answering on its loopback port, and with a candidate provisioner-close fix the same probe found it dead. Required: a per-lease release verb on the interface, called from the two runtime_provision_fenced paths in RuntimeBrokerService.provisionBinding, plus a closed fence before owned.put(...). Acceptance criterion: a test that provisions, drives a fenced path (or closes the service with a READY binding) and asserts the child process is gone - red today. Constraint the fix must not violate: RuntimeProvisioner.java:8 - "Retries for the same request must converge on one live resource".
[Critical] Carried forward from the earlier review round on this PR (its id there: R1-25), and still standing: start()'s cleanup lives in two catch clauses covering only RuntimeException and IOException, with no finally, so an Error thrown between the spawn and owned.put(...) leaks a token-bearing process that no teardown path can reach. The two commits since that round touch only fake-attestation-worker.mjs, so this code is byte-identical to the round it was filed against. Witness: not run - a leak reachable only through an Error between the spawn and owned.put(...) has no deterministic trigger to probe; the claim is the reading that start() has no finally.
中文说明
本轮确认的 4 条建议级发现已在 PR 上报告过,不再重复发布(列表见上方英文部分)。
[Critical] Carried forward from the earlier review round on this PR (its id there: R1-1), and still standing at ac63adc: provision adopts a live OS process and hands back a lease the caller is free to discard, while the only per-lease teardown (stop) is package-private with no production caller and RuntimeProvisioner exposes no release verb. Independently re-confirmed here: after service.close() the discarded worker was still answering on its loopback port, and with a candidate provisioner-close fix the same probe found it dead. Required: a per-lease release verb on the interface, called from the two runtime_provision_fenced paths in RuntimeBrokerService.provisionBinding, plus a closed fence before owned.put(...). Acceptance criterion: a test that provisions, drives a fenced path (or closes the service with a READY binding) and asserts the child process is gone - red today. Constraint the fix must not violate: RuntimeProvisioner.java:8 - "Retries for the same request must converge on one live resource".
[Critical] Carried forward from the earlier review round on this PR (its id there: R1-25), and still standing: start()'s cleanup lives in two catch clauses covering only RuntimeException and IOException, with no finally, so an Error thrown between the spawn and owned.put(...) leaks a token-bearing process that no teardown path can reach. The two commits since that round touch only fake-attestation-worker.mjs, so this code is byte-identical to the round it was filed against. Witness: not run - a leak reachable only through an Error between the spawn and owned.put(...) has no deterministic trigger to probe; the claim is the reading that start() has no finally.
— DeepSeek/deepseek-v4.1-flash@2d473174 via Qwen Code /review (v0.24.4)
Address the review findings on the process-adoption slice: - Release the adopted worker when provisioning is fenced after adoption, via a new RuntimeProvisioner.release default, so a fenced attempt never orphans a running process. - Reap the child in a finally keyed on adoption, so an Error between spawn and adoption can no longer leak a token-bearing process. - Retire the binding when re-attestation or the cheap liveness check fails, so the next call re-provisions instead of returning a retryable 503 forever; dispatch/control/cancel/release now consult RuntimeProvisioner.isUsable before using a lease. - Bound the ready-record read (32 KiB), catch Throwable in the reader thread, keep the worker's stdout open and drained, and stop swallowing the closed-before-ready diagnostic. - Validate capabilityDigest shape before spawning; stop flattening transport 4xx classifications into a retryable 503. - Close the provisioner when the service closes; drop the unread OwnedProcess.request component. - Update the module README boundary statement and scope the design doc's re-attestation claim; provision Node for the hosted runtime-broker test job so the process tests cannot silently skip.
|
Review round addressed in 8157a21 ( Fixed (14 threads, resolved):
Deferred (7 threads, resolved with reasons): R1-4 (env scrub needs a deliberate allowlist; prerequisite for the tools slice), R1-5 / R1-6 / R1-7 (unwired execute path, rework lands with the tools slice), R1-11 (bounded redacted stderr capture), R1-16 (fixture contract conformance), R1-18 (confirm coalescing / bounded executor), R1-23 (SIGTERM→SIGKILL escalation semantics). Partially addressed (left open): R1-13 — lifecycle/diagnostic behaviors are now pinned; the execute-route request assertions (M3/M4/M7b) and the mismatched-identity fixture (M9/M5) remain for the tools slice. CI note: the earlier |
Round 2 local verification against the real worker — head
|
| # | Finding | Severity |
|---|---|---|
| N1 | A binding retired after a failed re-attest leaves its worker alive and listening | should-fix (R1-3 is only half done) |
| N2 | A session acquired before the worker died can never be released. release returns 503 retryable=true forever |
should-fix / follow-up |
| F3 | A non-loopback url in the ready record still receives the bearer token (round 1) |
hardening, not addressed |
| F4 | A Broker crash still orphans the worker (round 1). The stdout drain does not help | worker-side follow-up |
| E2 | stop/close still only send SIGTERM |
deferred as R1-23, confirmed |
| T | Mutation: the PR's tests kill 8/16 mutants of this fix | test gap |
Environment
- The real worker is the bundle's
node dist/cli.js managed-runtime-worker. The worker and contract sources are byte-identical to this head (git diff --quiet 04a146c9a7 8157a218 -- managed-runtime-attestation-{worker,contract}.ts). - A Java harness drives the PR's own
LocalProcessRuntimeProvisioner+RuntimeBrokerService+HttpRuntimeTransportwith in-memory repositories. Session verbs use an acceptingRuntimeTransport, because the worker still exposes only attest. - macOS, JDK 26, Node 24. Module suite 84/84, 0 skipped, and
checkstyle:checkis clean. CI is green at this head on Java 11/17/21, macOS, Windows, MariaDB and Lint.
N1: a retired worker is never released
invalidateBinding removes the lease from liveBindings and fails the record. It never calls provisioner.release. The process stays in owned and keeps its port and token until provisioner.close().
- C1: a worker that attests at adoption and then answers
409 managed_runtime_identity_conflicton re-attest. Across 8 warms the binding alternatesOK gen=N/409. Live workers go1,1,2,2,3,3,4,4, and all 4 retired endpoints are stillLISTEN. This is the case R1-3 called security-relevant: the thing answering on the lease is not the adopted identity, yet the process stays up. - D2: the real worker is
SIGSTOPped. The re-attest times out and a new generation comes up. AfterSIGCONTthe old pid is alive and still listening on127.0.0.1. It only dies atservice.close(). - Minor: a
409on re-attest is returned asretryable=false, yet the very next warm succeeds, because the binding was already retired.
Fix: invalidateBinding calls releaseQuietly(record.getRequest(), record.getLease()). release is keyed by runtimeInstanceId, so it cannot touch the new generation. With the patch, C1 goes 1,0,1,0,… with every retired endpoint closed, and D2 has the old pid gone.
N2: a session on a dead worker cannot be released
releaseSession starts with requireUsableLease. The session context keeps the old lease forever, so once its worker dies:
release(rt-1)returns503 runtime_provision_failed retryable=trueon every call (B4). The binding has already recovered at gen 2, and the session still never releases.acquire(rt-1)returnsREADY, because the existing context is reused, and the nextcontrolgets 503 again.- Only a new
runtimeSessionIdworks (B5).
So the session record and the sessions entry stay until the Broker restarts, and the caller is told to retry something that cannot succeed. That is the same shape as round-1 F1, one level down. Fix: when the lease is not usable, release completes locally. It retires the binding, keeps the runtime_session_busy guard, moves the record to RELEASED and removes the context. With the patch, 3× release → OK true. Re-acquiring the released id is a 409 runtime_session_conflict, as for any released id.
A session holding the UNKNOWN execution from B2 then answers 409 runtime_session_busy. That is the existing "UNKNOWN is not settled" rule and waits for reconciliation; I left it alone.
F3 / F4 / E2 (round 1, still open)
- F3: a double whose ready
urlishttp://<en0 LAN ip>:<port>still receivedAuthorization: Bearer …plus tenant, workspace, cwd and capability digest (F1 in the figure). It was not in a review thread, so it may simply have been missed. The patch adds a 4-linehttp+127.0.0.1pin, the only address the worker binds. - F4: I
kill -9ed a Broker JVM that had provisioned a real worker. 5 s later the worker had PPID 1 and was still listening. The new drain keeps the stdout pipe open, but the real worker writes nothing after ready, so it never sees the pipe close. It still needs a parent watch on the worker side.
- E2: an adopted worker that ignores SIGTERM is still alive 6.5 s after
service.close(). This confirms deferred R1-23 and is not a new ask.
Candidate patch (+27/−3): 84/84 and checkstyle clean
The same harness on head + patch gives 12 pass / 1 fail. The one failure is E2, deferred as R1-23.
Patch
--- a/.../LocalProcessRuntimeProvisioner.java
+++ b/.../LocalProcessRuntimeProvisioner.java
@@ start()
+ URI endpoint = URI.create(String.valueOf(ready.get("url")));
+ if (!"http".equals(endpoint.getScheme())
+ || !"127.0.0.1".equals(endpoint.getHost())) {
+ throw failed("Managed Runtime ready record is invalid.");
+ }
RuntimeLease lease = new RuntimeLease(runtimeInstanceId,
- URI.create(String.valueOf(ready.get("url"))), token,
- leaseId, 1);
+ endpoint, token, leaseId, 1);
--- a/.../RuntimeBrokerService.java
+++ b/.../RuntimeBrokerService.java
@@ releaseSession
- requireUsableLease(context);
+ if (!provisioner.isUsable(context.lease())) {
+ invalidateBinding(context.binding());
+ synchronized (context) {
+ if (context.hasActiveControl()
+ || executionRepository.hasActiveByRuntimeSession(
+ context.session().getRuntimeSessionId())) {
+ throw conflict("runtime_session_busy",
+ "Runtime Session has an active operation");
+ }
+ RuntimeSessionRecord gone =
+ transitionSessionToReleasing(context);
+ if (gone.getState() != RuntimeSessionRecord.State.RELEASED) {
+ finishSessionRelease(gone);
+ }
+ sessions.remove(context.session().getRuntimeSessionId());
+ return CompletableFuture.completedFuture(true);
+ }
+ }
@@ invalidateBinding
liveBindings.remove(record.getBindingId());
+ if (record.getLease() != null) {
+ releaseQuietly(record.getRequest(), record.getLease());
+ }File: candidate-fix-8157a218.patch. It comes without tests. The real-worker scenarios above are its witness.
Test strength (mutation of the fix)
The reply says "each fix is pinned by a test that goes red when the fix is reverted". The full 84-test suite kills 8/16. The survivors:
- N03/N04/N05, the
control/cancel/releaseliveness checks, which are the R2-1 fix. Only the dispatch check (N06) is pinned. - N15,
isUsableignoringisAlive. - N08, the second fenced exit.
- N10, the R1-25
finally(a regression to "never kill" stays green). - N14, the R1-3 "don't flatten 409 into 503" half.
- N12, the drain. This one is near-equivalent: keeping the pipe open without draining only bites after 64 KiB of chatter.
N03's first-run kill came from HttpRuntimeTransportTest under parallel load. In isolation N03 survives, so that class may be timing-sensitive.
Housekeeping
qqqys's CHANGES_REQUESTED (the 9 no-undef errors) was fixed at f57765b and Lint is green here. The review decision is still CHANGES_REQUESTED, so it needs a re-review or a dismissal before merge.
Evidence (harness, worker doubles, raw logs, mutant driver, patch): wenshao/qwen-code@assets-pr12552/pr-12552-r2
中文版
第 2 轮:基于真实 worker 的本地验证 —— head 8157a218
结论:第 1 轮的阻塞项已修复。同一处修复里还剩两个较小的缺口,一个 27 行的补丁可以同时补上(已验证)。 建议合入前采纳这个补丁或等价改法。其余都属于加固或后续项。
在真实 qwen managed-runtime-worker 上验证已修复的部分:
- 第 1 轮 F1(worker 死后 binding 永久不可用)已修复。
SIGTERM或kill -9之后,第一次 warm 返回503,下一次 warm 拉起新一代(A3/A4)。挂死的情况也成立:SIGSTOP之后重新证明在 30 s 超时,下一次 warm 重新 provision(D1)。 service.close()现在会结束 worker(A5)。control在调用 transport 之前就拒绝死 lease(B1)。在死 lease 上的 dispatch 会进入UNKNOWN,不会调用 execute(B2)。- 接管失败时,会杀掉无视 SIGTERM 的 worker(E1,新的
finally里走destroyForcibly)。
仍未解决:
| # | 发现 | 级别 |
|---|---|---|
| N1 | 重新证明失败而退役的 binding,其 worker 仍然存活并在监听 | 建议修复(R1-3 只修了一半) |
| N2 | worker 死之前 acquire 的 session 永远无法 release,release 一直返回 503 retryable=true |
建议修复 / 后续 |
| F3 | ready 记录里的非 loopback url 仍会收到 bearer token(第 1 轮) |
加固,未处理 |
| F4 | Broker 崩溃仍让 worker 成为孤儿(第 1 轮),stdout drain 对此无效 | worker 侧后续项 |
| E2 | stop/close 仍只发 SIGTERM |
已按 R1-23 推迟,复核属实 |
| T | 变异:PR 自带测试对本次修复的变异只杀死 8/16 | 测试缺口 |
环境
- 真实 worker 用的是 bundle 里的
node dist/cli.js managed-runtime-worker。worker 和契约的源码与本 head 逐字节一致(已用git diff --quiet确认)。 - 用 Java harness 驱动 PR 自己的
LocalProcessRuntimeProvisioner+RuntimeBrokerService+HttpRuntimeTransport,repository 用内存实现。worker 目前只暴露 attest,所以 session 动词用一个全部接受的RuntimeTransport。 - macOS、JDK 26、Node 24。模块测试 84/84、0 跳过,
checkstyle:check干净。本 head 的 CI 在 Java 11/17/21、macOS、Windows、MariaDB 和 Lint 上全绿。
N1:退役的 worker 从未被释放
invalidateBinding 把 lease 移出 liveBindings,并把记录置为 FAILED,但从不调用 provisioner.release。进程仍留在 owned 里,端口和 token 一直有效,直到 provisioner.close()。
- C1: worker 在接管时证明成功,之后重新证明都回
409 managed_runtime_identity_conflict。8 次 warm 里 binding 在OK gen=N和409之间交替。存活 worker 数为1,1,2,2,3,3,4,4,4 个退役端点全部仍在LISTEN。这正是 R1-3 说的安全相关情形:lease 背后应答的已经不是被接管的身份,进程却仍在运行。 - D2: 对真实 worker 发
SIGSTOP。重新证明超时,新一代起来。SIGCONT之后旧 pid 仍然存活,仍在127.0.0.1上监听,直到service.close()才退出。 - 小问题:重新证明遇到的
409以retryable=false返回,但 binding 其实已经退役,紧接着的下一次 warm 就会成功。
修法:invalidateBinding 调用 releaseQuietly(record.getRequest(), record.getLease())。release 按 runtimeInstanceId 定位,不会误伤新一代。打上补丁后,C1 的存活数变为 1,0,1,0,…,退役端点全部关闭;D2 的旧 pid 已退出。
N2:死 worker 上的 session 无法 release
releaseSession 一开头就是 requireUsableLease,而 session context 永远持有旧 lease。所以 worker 一死:
release(rt-1)每次都返回503 runtime_provision_failed retryable=true(B4)。此时 binding 已经恢复到 gen 2,session 仍然释放不了。acquire(rt-1)复用已有 context,返回READY,下一次control又是 503。- 只有换一个新的
runtimeSessionId才能用(B5)。
于是 session 记录和 sessions 条目会一直留到 Broker 重启,调用方还被告知可以重试一件不可能成功的事。这和第 1 轮 F1 是同一种形态,只是下沉到了 session 层。修法:lease 不可用时就在本地完成 release:退役 binding,保留 runtime_session_busy 守卫,把记录置为 RELEASED,并移除 context。打上补丁后,3 次 release 都是 OK true。重新 acquire 已释放的 id 返回 409 runtime_session_conflict,和其它已释放的 id 一样。
持有 B2 中 UNKNOWN 执行的 session 则返回 409 runtime_session_busy。这是现有的"UNKNOWN 不算已结算"规则,要等 reconciliation 处理,这里没有改动。
F3 / F4 / E2(第 1 轮,仍未解决)
- F3: ready
url为http://<局域网 IP>:<port>的替身,仍然收到了Authorization: Bearer …以及租户、工作区、cwd 和 capability digest。这一项当时不在评审 thread 里,可能只是被漏看了。补丁加了 4 行,把地址固定为http+127.0.0.1,即 worker 唯一会绑定的地址。 - F4:
kill -9掉已经拉起真实 worker 的 Broker JVM,5 s 后 worker 的 PPID 为 1,仍在监听。新加的 drain 让 stdout 管道保持打开,但真实 worker 在 ready 之后什么都不写,所以永远察觉不到管道已关闭。仍然需要在 worker 侧监视父进程。 - E2: 一个无视 SIGTERM 的已接管 worker,在
service.close()6.5 s 后仍然存活。这只是复核已推迟的 R1-23,不是新诉求。
候选补丁(+27/−3):84/84,checkstyle 干净
同一套 harness 在 head + 补丁上是 12 通过 / 1 失败,唯一的失败是已按 R1-23 推迟的 E2。补丁不含测试,上面这些真实 worker 场景就是它的验证。
测试强度(对本次修复做变异)
作者回复说"每处修复都由回退即变红的测试钉住"。实测完整的 84 个测试只杀死 8/16。存活的变异:
- N03/N04/N05:
control/cancel/release的存活检查,即 R2-1 的修复。只有 dispatch 的检查(N06)被钉住。 - N15:
isUsable忽略isAlive。 - N08:第二个 fenced 出口。
- N10:R1-25 的
finally(退化成"从不杀进程"测试仍然全绿)。 - N14:R1-3 中"不再把 409 压成 503"的那一半。
- N12:drain。这个接近等价变异:保持管道打开但不读,只有输出超过 64 KiB 才会出问题。
N03 在第一轮被杀,是 HttpRuntimeTransportTest 在并行负载下失败导致的;单独运行时 N03 存活,说明这个测试类可能对时序敏感。
其它
qqqys 的 CHANGES_REQUESTED(9 个 no-undef)已在 f57765b 修复,本 head 的 Lint 为绿。但 review decision 仍是 CHANGES_REQUESTED,合入前需要重新评审或 dismiss。
证据(harness、worker 替身、原始日志、变异驱动、补丁):wenshao/qwen-code@assets-pr12552/pr-12552-r2
A failed re-attestation now releases the worker it retires, and a session on a dead lease finishes release locally. A ready URL that is not http on 127.0.0.1 is rejected before the bearer token is sent. Co-authored-by: jinye <[email protected]>
Head branch was pushed to by a user without write access
Round 2采纳 N1、N2、F3,在
这一次证明失败返回的 不采纳
EnglishAdopted N1, N2, and F3 in Not adopted: F4 needs a worker-side parent watch after a Broker crash, which this Java slice does not own. E2 is the already-deferred R1-23 SIGTERM escalation. The remaining mutation survivors are outside this round; the new tests pin the three adopted fixes. A |
Round 3 local verification against the real worker — head
|
| # | Scenario | 8157a218 |
9d08c65a |
|---|---|---|---|
| N1 | C1: the worker answers 409 on re-attest, 8 warms |
live workers 1,1,2,2,3,3,4,4, all retired ports LISTEN |
live 1,0,1,0,…, all 4 retired ports closed |
| N1 | D2: SIGSTOP, re-attest times out, then SIGCONT |
old pid alive and listening | old pid gone |
| N2 | B4: release of a session whose worker died |
503 retryable=true ×3 |
OK true ×3. Re-acquiring that id returns 409 runtime_session_conflict |
| F3 | F1: ready url = LAN IP |
bearer token and scope sent | refused, 0 bytes captured |
| new | G1: after recovery, release the stale gen-1 session | 503 |
OK true. Gen 2 keeps its bindingId, port and pid. Worker count stays 1→1 |
I added G1 to check whether the retirement that release now triggers can hit the recovered binding. It can't: every re-provision mints a new bindingId, so invalidateBinding(context.binding()) only ever touches the dead generation.
Head totals are 13 pass / 1 fail. The one failure is E2: an adopted worker that ignores SIGTERM outlives close(). That is the deferred R1-23, as agreed. F4 (worker-side parent watch) also stays a follow-up.
macOS CI: fencedProvisioningReapsTheWorker depends on timing
Run 35959440392 failed with Expected ExecutionException ... nothing was thrown in 0.082 s. The same test passed on macOS at 8157a218 and passes on ubuntu, Windows and MariaDB at this head.
FencingBindingRepository fences only renewOperation. The first renewal runs operationLease / 3 = 33 ms after the claim. If the worker boots and attests before that tick, stopAndGet() still returns the claim, the READY CAS succeeds and nothing is fenced. The whole local class took 60–90 ms to provision, so the margin is small. Changing only the lease value makes this deterministic:
Failing runs take about 0.09 s, which matches the CI failure: the tick lost the race. Fix in the test only: the double also fences the READY compareAndSet. The outcome then no longer depends on scheduling. With this patch the test passes at every lease value (16/16). It still fails 3/3 against a mutant that drops both fenced-exit releaseQuietly calls. The suite stays 87/87 and checkstyle stays clean.
Test patch (+6/−1)
--- a/packages/sdk-java/runtime-broker/src/test/java/com/alibaba/qwen/code/runtimebroker/LocalProcessRuntimeProvisionerTest.java
+++ b/packages/sdk-java/runtime-broker/src/test/java/com/alibaba/qwen/code/runtimebroker/LocalProcessRuntimeProvisionerTest.java
- /** Fences every renewal so provisioning loses its claim mid-boot. */
+ /** Fences renewal and the READY write, so provisioning always loses. */
@@ FencingBindingRepository
public RuntimeBindingRecord compareAndSet(
RuntimeBindingRecord expected,
RuntimeBindingRecord replacement) {
+ // Also fence the READY write, so the outcome does not depend on
+ // whether the first renewal tick beats the worker's boot.
+ if (replacement.getState() == RuntimeBindingRecord.State.READY) {
+ return null;
+ }
return delegate.compareAndSet(expected, replacement);
}Full file: test-fence-9d08c65a.patch
Mutation on this round's fixes: 7/9 killed
The new tests pin the release-on-retire, local-release and loopback-host changes. Two mutants survive. Neither blocks the merge:
- M5: dropping the "join an in-flight release" branch in
releaseUnusableSession. No test drives a release whose transport call is still pending when the worker dies. - M9: dropping the
httpscheme half of the ready-URL check, sohttps://127.0.0.1:…is accepted. A one-line--https-urlvariant of the existing test would pin it.
Before merge
- Take the test patch above, or an equivalent (the macOS red).
- Merge
origin/maininto the branch (the lint freshness red). Please don't rebase: the inline threads would lose their anchors. - Earlier
CHANGES_REQUESTEDreviews still block the PR. Their items are addressed at this head, so they need a re-review or a dismissal.
Harness, logs and mutation driver: pr-12552-r3/
中文说明
第三轮本地真实 worker 验证 —— head 9d08c65a
结论:第二轮采纳的三项修复(N1、N2、F3)在真实 qwen managed-runtime-worker 上成立,没有发现新的正确性问题。合入前还有两个 CI 项要处理,都不在生产代码里:
- macOS / Java 21 红,原因是
fencedProvisioningReapsTheWorker依赖时序。 只有第一次 33 ms 的续约 tick 抢在 Node 替身启动完成之前,fence 才会生效。给测试替身改 6 行就能让结果确定(下文已验证)。 - Lint & Static 只红在
Check lint gate freshness。 main 在a5a5eb55(ci(web-shell): give E2E Smoke 30 minutes #12593)改了.github/workflows/ci.yml,把origin/mainmerge 进分支即可。改动的.mjs在这个 head 上过 ESLint--max-warnings 0和 Prettier。
环境
- 真实 worker 用现有 bundle 的
node dist/cli.js managed-runtime-worker。managed-runtime-attestation-{worker,contract}.ts与这个 head、与当前origin/main逐字节相同。 - 沿用第二轮的 Java harness,新增场景 G。它驱动 PR 的
LocalProcessRuntimeProvisioner、RuntimeBrokerService、HttpRuntimeTransport,仓库用内存实现。两个 head 用同一套 harness、同一份 bundle。 - macOS,JDK 21,Node 24。模块测试在 JDK 21、25、26 上都是 87/87,0 跳过,
checkstyle:check通过。
第二轮修复:上一个 head 对比这个 head
| # | 场景 | 8157a218 |
9d08c65a |
|---|---|---|---|
| N1 | C1:re-attest 返回 409,连续 8 次 warm |
存活 worker 1,1,2,2,3,3,4,4,退役端口全部还在 LISTEN |
1,0,1,0,…,4 个退役端口全部关闭 |
| N1 | D2:SIGSTOP,re-attest 超时,再 SIGCONT |
旧 pid 存活且还在监听 | 旧 pid 已结束 |
| N2 | B4:worker 已死的 session 做 release |
503 retryable=true ×3 |
OK true ×3;再 acquire 同一个 id 返回 409 runtime_session_conflict |
| F3 | F1:ready url 为局域网 IP |
bearer token 和 scope 被发出 | 被拒,抓到 0 字节 |
| 新 | G1:恢复之后 release 旧的 gen-1 session | 503 |
OK true;gen 2 的 bindingId、端口、pid 都不变,worker 数保持 1→1 |
G1 是我新加的,用来检查 release 现在会触发的退役会不会误伤已经恢复的 binding。不会:每次重新 provision 都会生成新的 bindingId,所以 invalidateBinding(context.binding()) 只会碰到已经死掉的那一代。
head 合计 13 通过 / 1 失败。唯一的失败是 E2:无视 SIGTERM 的已接管 worker 在 close() 之后仍然存活,这就是已约定推迟的 R1-23。F4(worker 侧监视父进程)同样留作后续。
macOS CI:fencedProvisioningReapsTheWorker 依赖时序
运行 35959440392 报 Expected ExecutionException ... nothing was thrown,耗时 0.082 s。同一个测试在 8157a218 的 macOS 上通过,在这个 head 的 ubuntu、Windows、MariaDB 上也通过。
FencingBindingRepository 只 fence 了 renewOperation。第一次续约在 claim 之后 operationLease / 3 = 33 ms 才跑。如果 worker 在这次 tick 之前就启动并完成证明,stopAndGet() 仍会返回 claim,READY 的 CAS 成功,fence 不会发生。本地整个类的 provision 耗时 60–90 ms,余量很小。只改 lease 取值就能稳定复现:lease 100 ms 时 4/4 通过;300 ms、600 ms 时 4/4 失败;60 s 单独运行 3/3 失败。失败的运行约 0.09 s,与 CI 那次失败的耗时一致:tick 没抢过启动。
只改测试的修法:替身同时 fence READY 的 compareAndSet,结果不再依赖调度。打上补丁后,各个 lease 取值下全部通过(16/16)。对于同时去掉两个 fenced 出口 releaseQuietly 的变异体,测试仍然 3/3 失败。模块测试保持 87/87,checkstyle 通过。补丁见上方英文部分。
本轮修复的变异测试:9 个杀掉 7 个
新测试钉住了退役时 release、本地 release 和 loopback host 检查。两个存活体都不阻塞合入:
- M5: 去掉
releaseUnusableSession里「join 进行中的 release」分支。没有测试覆盖 transport release 还没完成、worker 就已经死掉的情况。 - M9: 去掉 ready URL 检查里
httpscheme 那一半,https://127.0.0.1:…会被接受。给现有测试加一个--https-url变体,一行就能钉住。
合入前
- 采纳上面的测试补丁或等价修改(macOS 的红)。
- 把
origin/mainmerge 进分支(lint freshness 的红)。请不要 rebase,否则行内评论会失去锚点。 - 早先的
CHANGES_REQUESTED评审仍在阻塞 PR。它们的问题在这个 head 上已经处理,需要重新评审或 dismiss。
harness、日志、变异脚本:pr-12552-r3/
Pick up the lint gate change so CI revalidates this head. Co-authored-by: jinye <[email protected]>
Refuse the READY publish instead of waiting for the renewal timer, so a fast machine cannot finish warm before the claim is lost. Co-authored-by: jinye <[email protected]>
Head branch was pushed to by a user without write access
Round 4 local verification against the real worker — head
|
|
@qwen-code /triage |
…s-p0-p8 Resolve conflicts with main and carry main's runtime-broker changes into this branch's broker architecture: - Keep the branch's RuntimeBrokerService, HttpRuntimeTransport and LocalProcessRuntimeProvisioner, which supersede main's narrower versions (QwenLM#12552). - Port reconcileExecution (QwenLM#12655): an UNKNOWN execution is settled only on the original binding generation's terminal status answer, through a health-checked lease, one bounded in-flight lookup per execution. Worker status responses carry progress fields, so only state and result are validated. - Port the lapsed-claim cancel fix (QwenLM#12583): a cancel still reaches an invocation this process is running after its dispatch claim lapsed or was fenced to UNKNOWN, without writing the Runtime's answer. - Send Cache-Control: no-store on Managed Runtime v2 requests. - Restore main's tool-execution repository contract (typed round trip, takeover fences, forgeries) in JdbcRepositoryContract. - Keep main's Settings deep-link when initialising the active panel in the web shell, alongside the Managed panel selection.














What this PR does
Starts the merged Managed Runtime worker from the Broker, attests it, and only then records the lease as ready. Reusing a ready binding (
warm/acquire) attests again throughconfirm; the session-scoped verbs (dispatch/control/cancel/release) check the adopted process is alive before using the lease, and a dead process retires the binding so the next call re-provisions. A provisioning attempt fenced after adoption releases its worker. Tool HTTP is on the Java client; the current worker still exposes only attestation, so an execute call fails closed with a non-retryable 404.Why it's needed
Attestation alone does not start a process or stop the Broker from treating a dead endpoint as ready. This is the next slice after the attestation client: own the worker process, prove it, then allow the lease to be used.
Reviewer Test Plan
How to verify
From
packages/sdk-java/runtime-broker, runmvn test -Dtest=LocalProcessRuntimeProvisionerTest,RuntimeBrokerServiceTest,HttpRuntimeTransportTest. Withnodeon PATH, the process test should reach READY, reject execute with HTTP 404, and fail the next warm after the worker is stopped. Existing broker tests should stay green becauseconfirmis a no-op unless a provisioner overrides it.Evidence (Before & After)
N/A
Tested on
Environment (optional)
JDK 21 and Node.
mvn testandmvn checkstyle:checkinpackages/sdk-java/runtime-broker.Risk & Scope
confirmkeep the previous ready path.Linked Issues
Related to #12380.
中文说明
这个 PR 做什么
Broker 启动已经合入的 Managed Runtime worker,证明通过之后才把 lease 记为 ready。复用 ready binding(
warm/acquire)会经由confirm重新证明;session 级动词(dispatch/control/cancel/release)在使用 lease 前做本地存活检查;进程已死则让 binding 失效,下一次调用重新 provision。被 fence 的 provision 尝试会释放它接管的 worker。Java 客户端带上了工具 HTTP;当前 worker 仍然只暴露 attestation,所以 execute 会以不可重试的 404 失败关闭。为什么需要
只有 attestation 客户端还不能启动进程,也不能阻止 Broker 把一个已经死掉的 endpoint 当成 ready。这一笔接在 attestation 客户端之后:先拥有 worker 进程,再证明它,然后才允许使用这条 lease。
评审验证
在
packages/sdk-java/runtime-broker运行mvn test -Dtest=LocalProcessRuntimeProvisionerTest,RuntimeBrokerServiceTest,HttpRuntimeTransportTest。PATH 上有node时,进程测试应进入 READY,execute 返回 HTTP 404,停掉 worker 后下一次 warm 失败。现有 Broker 测试应保持通过,因为没有覆写confirm的 provisioner 行为不变。风险与范围
Spring 和 Flyway 留在还不在 main 上的 Java 控制面模块。不包含 Kubernetes。worker 上还没有真正的工具处理,execute 在那一笔之前保持失败关闭。进程测试使用的是 stdin boot / stdout ready 契约的 Node 替身,不是 TypeScript worker 二进制。没有覆写
confirm的 provisioner 保持原来的 ready 路径。关联 #12380。