Skip to content

feat(sdk-java): Adopt a Managed Runtime only after attestation - #12552

Merged
doudouOUC merged 7 commits into
QwenLM:mainfrom
doudouOUC:feat/managed-runtime-process-adoption
Sep 24, 2026
Merged

doudouOUC merged 7 commits into
QwenLM:mainfrom
doudouOUC:feat/managed-runtime-process-adoption

Conversation

@doudouOUC

@doudouOUC doudouOUC commented Sep 23, 2026 •

Copy link
Copy Markdown
Collaborator

What this PR does

Starts the merged Managed Runtime worker from the Broker, attests it, and only then records the lease as ready. Reusing a ready binding (warm/acquire) attests again through confirm; the session-scoped verbs (dispatch/control/cancel/release) check the adopted process is alive before using the lease, and a dead process retires the binding so the next call re-provisions. A provisioning attempt fenced after adoption releases its worker. Tool HTTP is on the Java client; the current worker still exposes only attestation, so an execute call fails closed with a non-retryable 404.

Why it's needed

Attestation alone does not start a process or stop the Broker from treating a dead endpoint as ready. This is the next slice after the attestation client: own the worker process, prove it, then allow the lease to be used.

Reviewer Test Plan

How to verify

From packages/sdk-java/runtime-broker, run mvn test -Dtest=LocalProcessRuntimeProvisionerTest,RuntimeBrokerServiceTest,HttpRuntimeTransportTest. With node on PATH, the process test should reach READY, reject execute with HTTP 404, and fail the next warm after the worker is stopped. Existing broker tests should stay green because confirm is a no-op unless a provisioner overrides it.

Evidence (Before & After)

N/A

Tested on

OS Status
🍏 macOS ✅
🪟 Windows ⚠️
🐧 Linux ⚠️

Environment (optional)

JDK 21 and Node. mvn test and mvn checkstyle:check in packages/sdk-java/runtime-broker.

Risk & Scope

  • Main risk or tradeoff: the binding entrance re-attests on reuse while session verbs use a cheap local liveness check; a dead process retires the binding and the next call re-provisions instead of reusing the old endpoint.
  • Not validated / out of scope: Spring and Flyway stay with the Java control-plane module, which is not on main. Kubernetes provisioning is not included. Real tool handlers are not mounted on the worker; execute is fail-closed until that slice. The process test drives a Node double of the stdin boot and stdout ready contract, not the TypeScript worker binary.
  • Breaking changes / migration notes: none. Provisioners that do not override confirm keep the previous ready path.

Linked Issues

Related to #12380.

中文说明

这个 PR 做什么

Broker 启动已经合入的 Managed Runtime worker,证明通过之后才把 lease 记为 ready。复用 ready binding(warm/acquire)会经由 confirm 重新证明;session 级动词(dispatch/control/cancel/release)在使用 lease 前做本地存活检查;进程已死则让 binding 失效,下一次调用重新 provision。被 fence 的 provision 尝试会释放它接管的 worker。Java 客户端带上了工具 HTTP;当前 worker 仍然只暴露 attestation,所以 execute 会以不可重试的 404 失败关闭。

为什么需要

只有 attestation 客户端还不能启动进程,也不能阻止 Broker 把一个已经死掉的 endpoint 当成 ready。这一笔接在 attestation 客户端之后:先拥有 worker 进程,再证明它,然后才允许使用这条 lease。

评审验证

在 packages/sdk-java/runtime-broker 运行 mvn test -Dtest=LocalProcessRuntimeProvisionerTest,RuntimeBrokerServiceTest,HttpRuntimeTransportTest。PATH 上有 node 时,进程测试应进入 READY,execute 返回 HTTP 404,停掉 worker 后下一次 warm 失败。现有 Broker 测试应保持通过,因为没有覆写 confirm 的 provisioner 行为不变。

风险与范围

Spring 和 Flyway 留在还不在 main 上的 Java 控制面模块。不包含 Kubernetes。worker 上还没有真正的工具处理,execute 在那一笔之前保持失败关闭。进程测试使用的是 stdin boot / stdout ready 契约的 Node 替身,不是 TypeScript worker 二进制。没有覆写 confirm 的 provisioner 保持原来的 ready 路径。

关联 #12380。

The Broker starts the merged worker over stdin and stores the lease as ready only after the process attests. A later use attests again, and tool calls against the attestation-only worker fail closed.
@doudouOUC doudouOUC self-assigned this Sep 23, 2026
qqqys
qqqys previously requested changes Sep 23, 2026

@qqqys qqqys left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: this diff turns Lint & Static red — 9 no-undef errors in the new .mjs test resource

Where

packages/sdk-java/runtime-broker/src/test/resources/fake-attestation-worker.mjs — the file this PR adds. It does not exist at the base 94104d5565b1; I confirmed that arm on its own route as an HTTP 404, so it is a proven absence rather than an unchanged file.

What CI measured at this head (a20ea2910d09f6b599bdee5c71fe7f56aa0ebe2e)

Lint & Static (ubuntu-latest, Node 22.x) → completed / failure, 12:00:18Z → 12:07:07Z (409 s), check-run id 107170403311.

The failing step is Run ESLint (step 22). Step 8 Check lint gate freshness succeeded and step 14 Install dependencies succeeded, so this is neither the sub-minute freshness gate nor a branch-staleness abort — ESLint really ran, and it reported:

packages/sdk-java/runtime-broker/src/test/resources/fake-attestation-worker.mjs
   5:3   error  'process' is not defined  no-undef
   6:3   error  'process' is not defined  no-undef
  6:41   error  'Buffer' is not defined   no-undef
   7:3   error  'process' is not defined  no-undef
 14:21  error  'URL' is not defined      no-undef
  48:3   error  'process' is not defined  no-undef
 59:39   error  'process' is not defined  no-undef
  60:1   error  'process' is not defined  no-undef
  61:1   error  'process' is not defined  no-undef

✖ 9 problems (9 errors, 0 warnings)
##[error]Process completed with exit code 1.

Run ESLint is node scripts/lint.js --eslint → npm run lint:ci → eslint . --ext .ts,.tsx --max-warnings 0, i.e. zero tolerance.

I re-read all nine positions against the byte-faithful blob at this head (sha c4439a93127b86f6b99c891f45ce72f02c612832, 1,975 B, 62 lines; recomputed sha1('blob ' + len + '\0' + bytes) equals the API's own sha, NUL_bytes=0), and every line and column matches exactly: :5/:6/:7 are the process.stdin.on(…) triple, 6:41 is Buffer.concat, 14:21 is new URL(request.url, 'http://127.0.0.1'), :48 is process.stdout.write, 59:39 is process.exit(0), and :60/:61 are the two process.once(…) handlers. The file carries no eslint-disable comment and no globals declaration.

Why this is the PR and not main

A sibling arm on the same base: #12549 (base 94104d5565b1, head committed 11:41:50Z) ran the identical lane green — Lint & Static (ubuntu-latest, Node 22.x) completed / success, 11:42:48Z → 11:54:36Z — roughly 18 minutes before this PR's run failed. Same base, same runner family, same hour, opposite conclusion, so the cause is inside this diff.

Mechanism

In eslint.config.js (blob-identical at this head and at my checkout: 9d3322575016424f349eac56281efe41430a801e, 21,191 B) the block that injects Node globals is scoped to TypeScript only:

files: [
  'packages/**/src/**/*.{ts,tsx}',
  'integrations/**/src/**/*.{ts,tsx}',
],
languageOptions: { globals: { ...globals.node, ...globals.es2021 } },

The new file sits under packages/**/src/**, so it is linted, but its .mjs extension matches no languageOptions.globals block — so no-undef fires on every Node global. The config's global ignores list does not exclude packages/sdk-java/** or **/test/resources/**.

I did not establish why other .mjs files under packages/ that also use a bare process (for example packages/node-repl/src/runtime/kernel.mjs) are not reported. That is unmeasured, and nothing in this finding rests on it — the finding rests on the executed lane and the byte-faithful blob.

Impact

Two things, and the second is the one I'd weigh more heavily:

  1. The lane is deterministically red at this head — ESLint emitted a complete enumerated error set and the step exited 1, so this is not a flake and a re-run will not help. AGENTS.md requires npm run lint to pass before a change is called done.
  2. Because step 22 failed, steps 23–35 were all skipped: actionlint, shellcheck, yamllint, Prettier, the sensitive-keyword linter, the i18n check, settings-schema generation and its up-to-date drift check, the VS Code companion notice check, the serve fast-path bundle closure, core subpath exports resolution, and the .github/scripts helper tests. So none of those has been evaluated against this diff at all — the red lane is currently masking the whole static surface behind it.

To be precise about what I am not claiming: main's ruleset has no required_status_checks rule, so branch protection would not mechanically refuse the merge. I am filing this as blocking because the red is caused by this diff and because it silently skipped thirteen downstream checks, not because the platform enforces it.

Fix direction

Any one of these clears it, and the choice is yours:

  1. Give .mjs under packages/**/src/** Node globals — extend the existing block's files: (or add a sibling block) with something like 'packages/**/src/**/*.mjs' and languageOptions: { globals: { ...globals.node } }. Smallest change, and consistent with how .ts/.tsx already get them.
  2. Treat the file as what it is — a test resource, not source — and add its path to the global ignores in eslint.config.js.
  3. Declare the globals in the file (/* global process, Buffer, URL */). Most contained, least consistent with the rest of the repo.

Options 1 and 2 also settle the policy for the next .mjs fixture, which is probably worth deciding deliberately rather than per file.

What I am not blocking on

I read the rest of the production delta (all four Java files) and found nothing else I'd call merge-blocking. The two things I checked hardest both hold up:

  • The confirm default really is a no-op, and that is what the PR body and the design doc say is intended ("Provisioners that do not override confirm keep the previous ready path"; "it does not add a server"). There is also no production RuntimeProvisioner implementation on main for it to regress — only the test fake at RuntimeBrokerServiceTest.java:1026 — so this is not a dead switch, it is an unwired slice that says so.
  • Every failure path I traced in LocalProcessRuntimeProvisioner destroys the process before the exception escapes, and owned.put(…) happens only after attest succeeds.

qwen-code-ci-bot's stage-2 review is the reason this is worth filing separately. Its prose was written while the lane was still in flight and still says so — "Four lanes were still running at review time (the Node-side unit, lint, and integration jobs plus the Java 11 daemon E2E). I am reporting them as pending rather than guessing an outcome." Its <!-- qwen-triage-ci --> status table was subsequently edited (updated_at 2026-09-23T12:37:36Z, body grown 13,791 → 14,094 chars) and now does show Lint & Static (ubuntu-latest, Node 22.x) | ❌ failure, so the red is visible there. But no part of that review diagnoses it: in its full 14,094-char body the strings no-undef, ESLint and 9 problems each occur 0 times, and it filed no review row at all (reviews rows=0, so reviewDecision is still REVIEW_REQUIRED). What is missing is the part that decides what to do: that all nine errors are no-undef on Node globals, that they are all in the file this PR adds, that a same-base sibling was green 18 minutes earlier, and that thirteen downstream static checks were skipped as a result.

Its erasure point about HttpRuntimeTransport.execute versus RuntimeTransport.execute is factually correct (RuntimeTransport.java:23 declares CompletionStage<Map<String, Object>>; the new HttpRuntimeTransport.java:120 declares CompletionStage<Void>), but it is latent rather than blocking — HttpRuntimeTransport.java:25 is public final class HttpRuntimeTransport { with no implements clause, so nothing conflicts until a later slice wires that interface.


Reviewed at a20ea2910d09f6b599bdee5c71fe7f56aa0ebe2e. Evidence: check-run 107170403311 job log (236,926 B, read as bytes with NUL_bytes=0, ANSI-stripped before matching); byte-faithful blobs for fake-attestation-worker.mjs and eslint.config.js, each with its recomputed blob sha equal to the API's; the live check-run census for #12549 as the same-base sibling arm; and ci-bot's comment 5794738330 re-read from the owning route at post time.

ESLint treats the test script as project JavaScript, so process, Buffer, and URL cannot be used as undeclared globals.
@doudouOUC

Copy link
Copy Markdown
Collaborator Author

Fixed the Lint & Static failure in f57765b.

fake-attestation-worker.mjs now imports process and Buffer from node:process and node:buffer, and compares the request path directly instead of new URL. That removes the 9 no-undef errors. LocalProcessRuntimeProvisionerTest still passes locally.

@doudouOUC

Copy link
Copy Markdown
Collaborator Author

Two CI results since f57765b:

  • Lint & Static failed on Prettier for fake-attestation-worker.mjs. Formatted in ac63adc.
  • ubuntu-latest / Java 21 failed in Verify checkout includes expected head commit before tests started. The checked-out ref did not contain f57765b256. That is a runner checkout mismatch, not this diff. Reran the failed job.

@wenshao

wenshao commented Sep 23, 2026

Copy link
Copy Markdown
Collaborator

Local verification against the real worker — head ac63adc4

Verdict: mergeable as a library slice, with one issue to fix before anything constructs LocalProcessRuntimeProvisioner in production. The headline claim holds against the real qwen managed-runtime-worker binary, not only against the Node double. The worker spawns, attests and reaches READY. Reusing the in-memory lease re-attests. A stopped or kill -9ed worker is refused. On main the same scenario hands back the dead lease every time.

The gap is recovery. Once a worker dies, its binding stays READY for the rest of the Broker's life. Every retry gets 503 retryable=true and no new worker is ever started. I have a patch for that (+61/−8, verified) and a test patch (+87/−6) that raises mutation kills from 2/14 to 11/14.

qqqys's blocking item is fixed. The .mjs passes eslint --max-warnings 0 and Prettier at head, and a20ea29 still reproduces the 9 no-undef errors locally. That CHANGES_REQUESTED review needs a re-review or a dismissal.

Environment

  • The real worker came from an already-built bundle (node dist/cli.js managed-runtime-worker). Its sources are byte-identical to this head: git diff --quiet 075de399 ac63adc4 -- managed-runtime-attestation-worker.ts managed-runtime-attestation-contract.ts cli.ts.
  • The PR's own LocalProcessRuntimeProvisioner + RuntimeBrokerService + HttpRuntimeTransport were driven by a Java harness, not mocks.
  • JDK 25 (host) and JDK 21.0.12 (temurin container, the module's release), Node 22.
  • Module suite: 76/76, 0 skipped on both JDKs. checkstyle:check is clean.

What holds up (real worker)

real worker E2E

# Scenario Result
S1 warm() spawns the worker, writes the boot doc, reads ready, attests READY in ~180 ms. Listens on 127.0.0.1 only.
S2 20× warm() on a READY binding Each one re-attests over HTTP: p50 2.0 ms vs 0.005 ms without confirm. This runs per warm/new-session acquire, not per tool call.
S3 HttpRuntimeTransport.execute against the real worker 404 managed_runtime_incompatible, retryable=false, fails closed as described
S4a/b stop(lease) → next warm() Worker exits on SIGTERM. warm → 503 runtime_provision_failed
S5 external kill -9 of the worker → next warm() 503, fails closed
S6 provisioner.close() Ends all owned workers. service.close() alone does not, so the owner has to close both.
S7 8 workspaces warmed in parallel 8 distinct attested workers in 127 ms, none left after close
B1 Worker that never prints ready Refused at 30.1 s, process gone, and the next warm re-provisions
B2 Real worker behind a ready record with a foreign leaseId ready record is invalid, wrapper + worker gone

The same checks on JDK 21 give identical results. The Java↔TS boot/ready contract matches field for field against the real binary, which settles the "double could drift" gap from triage.

Findings

F1 — A dead worker bricks its binding for the Broker's lifetime, while the error says retryable (fix before wiring; not a regression).
confirm failure leaves the record READY and liveBindings populated. ensureBinding only provisions from PROVISIONING, so nothing ever replaces the dead worker. Three arms, same scenario:

dead worker A/B/C

  • Arm A: main behaviour (head with the RuntimeBrokerService hunk reverted). After the worker is gone, warm() returns the READY lease 5/5 times, pointing at a dead port. This is the bug the PR fixes.
  • Arm B: head. It is refused correctly, but 5/5 retries give 503 runtime_provision_failed retryable=true and 0 workers are spawned. Until a restart or reconcile, the caller is told to retry something that cannot succeed.
  • Arm C: head + patch. The first retry provisions a new generation, and the next 5 warms are OK.

The patch has two parts:

  • On confirm failure the service claims the operation, CASes READY→FAILED and drops liveBindings.
  • The provisioner stops the process it failed to confirm.

The extended PR test is RED on head (Managed Runtime process is not alive. on the recovery step) and GREEN with the patch. With the patch plus the follow-up tests the full suite is 83/83 and checkstyle is clean. Two caveats:

  • Recovery was exercised with the in-memory repository only.
  • A transient attest failure now retires a healthy worker instead of sticking. I think that trade is right for a fail-closed gate, but it is the owner's call.
Patch (production part, +51/−7)
--- a/.../RuntimeBrokerService.java
+++ b/.../RuntimeBrokerService.java
@@ ensureBinding
             BindingContext live = requireLiveBinding(record);
             return safeStage(() -> provisioner.confirm(request, live.lease()))
-                    .thenApply(ignored -> live);
+                    .handle((ignored, error) -> {
+                        if (error == null) {
+                            return live;
+                        }
+                        retireUnconfirmedBinding(record);
+                        throw new CompletionException(unwrap(error));
+                    });
@@
+    /**
+     * A READY binding whose Runtime no longer attests cannot recover in
+     * place. Retire it so the next ensureBinding provisions a new generation.
+     */
+    private void retireUnconfirmedBinding(RuntimeBindingRecord record) {
+        RuntimeBindingRecord claimed = bindingRepository.claimOperation(
+                record.getBindingId(), brokerOwnerId, operationLeaseDuration);
+        if (claimed == null
+                || claimed.getState() != RuntimeBindingRecord.State.READY
+                || claimed.getGeneration() != record.getGeneration()) {
+            return;
+        }
+        if (bindingRepository.compareAndSet(claimed, claimed.withState(
+                RuntimeBindingRecord.State.FAILED, null, clock.instant()))
+                != null) {
+            liveBindings.remove(record.getBindingId());
+        }
+    }
--- a/.../LocalProcessRuntimeProvisioner.java
+++ b/.../LocalProcessRuntimeProvisioner.java
+    private static final Duration STOP_GRACE = Duration.ofSeconds(5);
@@ stop()
-            process.process.destroy();
+            terminate(process.process);
+    private static void terminate(Process process) {
+        process.destroy();
+        process.onExit().orTimeout(STOP_GRACE.toMillis(),
+                TimeUnit.MILLISECONDS).exceptionally(ignored -> {
+                    process.destroyForcibly();
+                    return null;
+                });
+    }
@@ start()
+            URI endpoint = URI.create(String.valueOf(ready.get("url")));
+            if (!"127.0.0.1".equals(endpoint.getHost())) {
+                throw failed("Managed Runtime ready record is invalid.");
+            }
             RuntimeLease lease = new RuntimeLease(runtimeInstanceId,
-                    URI.create(String.valueOf(ready.get("url"))), token,
-                    leaseId, 1);
+                    endpoint, token, leaseId, 1);
@@ both catch blocks
-                ownedProcess.process.destroy();
+                terminate(ownedProcess.process);
@@ attestOwned()
         if (process == null || !process.process.isAlive()) {
+            stop(lease);
             throw failed("Managed Runtime process is not alive.");
         }
-        attest(request, process.seed, lease);
+        try {
+            attest(request, process.seed, lease);
+        } catch (RuntimeException exception) {
+            stop(lease);
+            throw exception;
+        }

Full patch: patches/fix-ac63adc4.patch. Combined with the tests: fix-plus-tests-ac63adc4.patch.

F2 — Failure paths only send SIGTERM (hardening). A worker that refuses attestation and ignores SIGTERM is still alive 6.5 s after the failure. Each retry adds another one: 2 alive after the second warm. The merged worker does exit on SIGTERM, so this only bites a hung worker. The patch above adds a 5 s grace and then destroyForcibly() (B3 → 0 alive).

F3 — A non-loopback url in the ready record receives the bearer token and identity (hardening; confirms triage's question). A double that reports http://172.16.1.234:<port> got Authorization: Bearer … plus tenant/workspace/cwd/capability digest. The patch pins the host to 127.0.0.1, which is the only address the worker binds, and the credentials are then never sent (B4).

misbehaving workers

F4 — A Broker crash orphans the worker (follow-up, worker side). I kill -9ed a JVM that had provisioned a real worker. 5 s later the worker had been re-parented to PID 1 and was still listening with a live token. stdin is closed right after the boot doc, so the worker has no lifeline, and Java cannot clean up after its own crash. That needs a worker-side parent watch or a held pipe. It was already visible in #12506 and now has an owner that can die.

orphan

F5 — The execute half: I confirm triage's facts. Keeping it here or not is the maintainers' scope call.

  • The erasure clash is real. Making HttpRuntimeTransport implements RuntimeTransport fails javac with return type CompletionStage<Void> is not compatible with CompletionStage<Map<String,Object>>.
  • The real-worker 404 is reported as Managed Runtime attestation endpoint is incompatible., which is misleading for a tool call.
  • Mutant M14, which sends execute to a different unmounted path, survives every test, because nothing pins the execute wire shape.

I'd drop it from this slice, or rename it, and land it with the tool-execution contract.

Test strength (mutation)

mutation table

The PR's tests kill 2/14. Only the re-attest path is pinned. "No attestation before READY" (M03), every ready-record identity check (M04–M08), and all teardown paths (M09–M11) survive.

The follow-up tests (followup-tests-ac63adc4.patch, +87/−6) add these to the fake worker:

  • a mode argument (wrong-type|wrong-instance|wrong-incarnation|wrong-lease|wrong-epoch|no-attest)
  • a pid file, so each refusal also asserts the process exited, and so do stop() and close()

That brings kills to 11/14. M12 and M13 are equivalent, and M14 is the unpinned execute route from F5. The fixture stays eslint/Prettier clean.

CI

The two red lanes are infra, not this diff:

  • Integration Tests (no-AK) failed in Verify checkout includes expected head commit before any test ran.
  • web-shell E2E Smoke was cancelled at 20 min with the web server refusing on :4170.

All Java lanes (11/17/21, macOS, Windows, MariaDB) and Lint & Static are green at ac63adc4.

Evidence (harness sources, raw logs, patches, mutant driver): wenshao/qwen-code@asserts/pr-12552

中文版

基于真实 worker 的本地验证 —— head ac63adc4

结论:作为库切片可以合入。但在任何生产代码构造 LocalProcessRuntimeProvisioner 之前,有一个问题应先修。

PR 的核心主张在真实的 qwen managed-runtime-worker 二进制上成立,不只是在 Node 替身上成立:

  • worker 能被拉起、完成证明、进入 READY;
  • 复用内存 lease 时会重新证明;
  • 被 stop 或被 kill -9 的 worker 会被拒绝。

同一场景在 main 上每次都会把死掉的 lease 交回去。

缺口在恢复。worker 死后,它的 binding 在 Broker 整个生命周期内一直是 READY。每次重试都返回 503 retryable=true,而且永远不会再拉起新 worker。我准备了修复补丁(+61/−8,已验证),以及测试补丁(+87/−6,变异杀伤从 2/14 提到 11/14)。

qqqys 的阻塞项已修复:head 的 .mjs 通过 eslint --max-warnings 0 和 Prettier;本地用 a20ea29 仍能复现那 9 个 no-undef 错误。那条 CHANGES_REQUESTED 需要重新评审或 dismiss。

环境

  • 真实 worker 取自已构建的 bundle(node dist/cli.js managed-runtime-worker)。worker、契约和 cli.ts 源码与本 head 逐字节一致(git diff --quiet 已确认)。
  • 用 Java harness 直接驱动 PR 自己的 LocalProcessRuntimeProvisioner + RuntimeBrokerService + HttpRuntimeTransport,没有用 mock。
  • JDK 25(宿主)和 JDK 21.0.12(temurin 容器,即模块的 release),Node 22。
  • 模块测试两个 JDK 下都是 76/76、0 跳过,checkstyle 干净。

成立的部分(真实 worker)

# 场景 结果
S1 warm 拉起 worker、写 boot、读 ready、完成证明 约 180 ms 进入 READY,只监听 127.0.0.1
S2 对 READY binding 连续 warm 20 次 每次都走 HTTP 重新证明:p50 2.0 ms,去掉 confirm 时为 0.005 ms。开销按 warm / 新会话计,不按工具调用计
S3 对真实 worker 调 execute 404 managed_runtime_incompatible、不可重试,与描述一致
S4/S5 stop 或外部 kill -9 后再 warm 503,失败关闭;worker 收到 SIGTERM 会退出
S6 provisioner.close() 结束全部 worker。只调 service.close() 不会结束,属主需要两个都关
S7 8 个工作区并行 warm 127 ms 内得到 8 个独立且已证明的 worker,关闭后无残留
B1 worker 始终不输出 ready 30.1 s 被拒并被杀,下一次 warm 会重新 provision
B2 真实 worker 套上带假 leaseId 的 ready 记录 被拒,包装进程和 worker 都已退出

JDK 21 下结果相同。Java↔TS 的 boot/ready 契约在真实二进制上逐字段对上,triage 担心的"替身可能漂移"这一点得到了验证。

发现

F1:worker 死后 binding 在 Broker 生命周期内永久不可用,报错却标为可重试。 应在接线前修复;不是回归。

原因是 confirm 失败后记录仍为 READY,liveBindings 也还在,而 ensureBinding 只会从 PROVISIONING 状态发起 provision,所以死掉的 worker 永远没有东西来替换。三臂对照:

  • A:main 行为(回退 service 那段改动):worker 死后 warm 5/5 次都返回指向死端口的 READY lease。这正是本 PR 修掉的问题。
  • B:head:能正确拒绝,但 5/5 次重试都是 503 retryable=true,新拉起的 worker 为 0。
  • C:head + 补丁:第一次重试就拉起新一代,之后 5 次 warm 全部成功。

补丁分两部分:

  • confirm 失败时,service 先 claim operation,再用 CAS 把 READY 改为 FAILED,并移除 liveBindings;
  • provisioner 停掉证明失败的那个进程。

扩展后的测试在 head 上是红的(恢复那一步报 Managed Runtime process is not alive.),打上补丁后是绿的。修复加测试后全量 83/83,checkstyle 干净。两点说明:

  • 恢复路径只在内存 repository 上验证过;
  • 一次瞬时证明失败现在会让健康的 worker 退役,而不是卡住。对失败关闭的闸门来说我认为这个取舍合理,但由负责人决定。

F2:失败路径只发 SIGTERM(加固)。 一个拒绝证明、同时无视 SIGTERM 的 worker,在失败 6.5 s 后仍然存活,而且每次重试再多一个(第二次 warm 后存活 2 个)。已合入的 worker 会响应 SIGTERM,所以只有挂死的 worker 才会触发。补丁改为给 5 s 宽限,之后 destroyForcibly(),B3 存活数变为 0。

F3:ready 记录里的非 loopback url 会收到 bearer token 和身份信息(加固,印证 triage 的疑问)。 上报 http://172.16.1.234:<port> 的替身收到了 Authorization: Bearer …,以及租户、工作区、cwd 和 capability digest。补丁把 host 固定为 127.0.0.1(worker 唯一会绑定的地址),之后凭据不会再发出(B4)。

F4:Broker 崩溃会让 worker 成为孤儿(后续项,worker 侧)。 kill -9 掉已经拉起真实 worker 的 JVM,5 s 后 worker 已被 PID 1 收养,仍在监听,token 仍然有效。boot 写完后 stdin 就关闭了,worker 没有生命线;Java 也无法在自己崩溃后做清理。这需要 worker 侧监视父进程,或保持一根管道。该问题在 #12506 就已存在,现在它有了一个可能会死掉的属主。

F5:execute 那一半。 triage 说的事实我都复核成立,是否留在本切片由维护者定:

  • 擦除冲突属实:让 HttpRuntimeTransport implements RuntimeTransport,javac 报返回类型不兼容;
  • 真实 worker 返回的 404 被报成 "attestation endpoint is incompatible",对工具调用来说有误导;
  • 变异 M14(把 execute 发到另一个未挂载路径)在所有测试下都存活,因为 execute 的线上格式没有被任何东西约束。

我倾向于从本切片移除或改名,随工具执行契约一起落地。

测试强度(变异)

PR 自带测试只杀死 2/14,只钉住了重新证明那条路径。以下变异全部存活:

  • M03:READY 之前不做证明;
  • M04–M08:ready 记录的全部身份校验;
  • M09–M11:所有进程清理路径。

跟进测试(+87/−6)给 fake worker 增加了:

  • mode 参数(wrong-type|wrong-instance|wrong-incarnation|wrong-lease|wrong-epoch|no-attest);
  • pid 文件,用来断言每种拒绝之后进程都已退出,stop() 和 close() 之后也一样。

杀伤率提到 11/14。M12、M13 是等价变异,M14 就是 F5 里没被钉住的 execute 路由。fixture 仍然通过 eslint 和 Prettier。

CI

两条红 lane 都是基础设施问题,与本 diff 无关:

  • Integration Tests (no-AK) 在任何测试开始前就失败于 checkout 未包含 head;
  • web-shell E2E Smoke 在 20 分钟时被取消,web server 在 :4170 拒绝连接。

ac63adc4 上全部 Java lane(11/17/21、macOS、Windows、MariaDB)和 Lint & Static 均为绿。

证据(harness 源码、原始日志、补丁、变异驱动):wenshao/qwen-code@asserts/pr-12552


🤖 Generated with Claude Code — Claude Opus 5.5 (1M context)

@wenshao wenshao left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

4 Suggestion-level finding(s) this review confirmed are already reported on this PR and are not repeated:

  • worker stderr discarded via Redirect.DISCARD, so no boot failure carries the worker's own reason - already reported (comment 4085208800)
  • ready record read with an unbounded readLine() - already reported (comment 4085208805)
  • attest flattens the transport's non-retryable classification into a retryable 503 - already reported (comment 4085208855)
  • adoption test does not observe the attestation it claims to prove - already reported (comment 4085208875)

[Critical] Carried forward from the earlier review round on this PR (its id there: R1-1), and still standing at ac63adc: provision adopts a live OS process and hands back a lease the caller is free to discard, while the only per-lease teardown (stop) is package-private with no production caller and RuntimeProvisioner exposes no release verb. Independently re-confirmed here: after service.close() the discarded worker was still answering on its loopback port, and with a candidate provisioner-close fix the same probe found it dead. Required: a per-lease release verb on the interface, called from the two runtime_provision_fenced paths in RuntimeBrokerService.provisionBinding, plus a closed fence before owned.put(...). Acceptance criterion: a test that provisions, drives a fenced path (or closes the service with a READY binding) and asserts the child process is gone - red today. Constraint the fix must not violate: RuntimeProvisioner.java:8 - "Retries for the same request must converge on one live resource".

[Critical] Carried forward from the earlier review round on this PR (its id there: R1-25), and still standing: start()'s cleanup lives in two catch clauses covering only RuntimeException and IOException, with no finally, so an Error thrown between the spawn and owned.put(...) leaks a token-bearing process that no teardown path can reach. The two commits since that round touch only fake-attestation-worker.mjs, so this code is byte-identical to the round it was filed against. Witness: not run - a leak reachable only through an Error between the spawn and owned.put(...) has no deterministic trigger to probe; the claim is the reading that start() has no finally.

中文说明

本轮确认的 4 条建议级发现已在 PR 上报告过,不再重复发布(列表见上方英文部分)。

[Critical] Carried forward from the earlier review round on this PR (its id there: R1-1), and still standing at ac63adc: provision adopts a live OS process and hands back a lease the caller is free to discard, while the only per-lease teardown (stop) is package-private with no production caller and RuntimeProvisioner exposes no release verb. Independently re-confirmed here: after service.close() the discarded worker was still answering on its loopback port, and with a candidate provisioner-close fix the same probe found it dead. Required: a per-lease release verb on the interface, called from the two runtime_provision_fenced paths in RuntimeBrokerService.provisionBinding, plus a closed fence before owned.put(...). Acceptance criterion: a test that provisions, drives a fenced path (or closes the service with a READY binding) and asserts the child process is gone - red today. Constraint the fix must not violate: RuntimeProvisioner.java:8 - "Retries for the same request must converge on one live resource".

[Critical] Carried forward from the earlier review round on this PR (its id there: R1-25), and still standing: start()'s cleanup lives in two catch clauses covering only RuntimeException and IOException, with no finally, so an Error thrown between the spawn and owned.put(...) leaks a token-bearing process that no teardown path can reach. The two commits since that round touch only fake-attestation-worker.mjs, so this code is byte-identical to the round it was filed against. Witness: not run - a leak reachable only through an Error between the spawn and owned.put(...) has no deterministic trigger to probe; the claim is the reading that start() has no finally.

— DeepSeek/deepseek-v4.1-flash@2d473174 via Qwen Code /review (v0.24.4)

@doudouOUC
doudouOUC requested review from qqqys and wenshao September 24, 2026 00:38
@doudouOUC
doudouOUC enabled auto-merge September 24, 2026 00:38
Address the review findings on the process-adoption slice:

- Release the adopted worker when provisioning is fenced after
  adoption, via a new RuntimeProvisioner.release default, so a fenced
  attempt never orphans a running process.
- Reap the child in a finally keyed on adoption, so an Error between
  spawn and adoption can no longer leak a token-bearing process.
- Retire the binding when re-attestation or the cheap liveness check
  fails, so the next call re-provisions instead of returning a
  retryable 503 forever; dispatch/control/cancel/release now consult
  RuntimeProvisioner.isUsable before using a lease.
- Bound the ready-record read (32 KiB), catch Throwable in the reader
  thread, keep the worker's stdout open and drained, and stop
  swallowing the closed-before-ready diagnostic.
- Validate capabilityDigest shape before spawning; stop flattening
  transport 4xx classifications into a retryable 503.
- Close the provisioner when the service closes; drop the unread
  OwnedProcess.request component.
- Update the module README boundary statement and scope the design
  doc's re-attestation claim; provision Node for the hosted
  runtime-broker test job so the process tests cannot silently skip.
@doudouOUC

Copy link
Copy Markdown
Collaborator Author

Review round addressed in 8157a21 (fix(sdk-java): Reap discarded workers and retire dead bindings). All 84 module tests pass locally (mvn clean test), checkstyle is clean, and each fix is pinned by a test that goes red when the fix is reverted (verified by mutation).

Fixed (14 threads, resolved):

  • R1-1: RuntimeProvisioner.release default; called at both runtime_provision_fenced exits — a fenced attempt no longer orphans a live worker.
  • R1-25: start() cleanup is a finally keyed on adoption, so an Error between spawn and adoption cannot leak the process.
  • R1-3: confirm failure evicts the lease and retires the record (claim → fail), so the next call re-provisions instead of returning a retryable 503 forever; attest no longer flattens transport 4xx (e.g. 409 identity conflict) into 503.
  • R2-1: dispatch/control/cancel/releaseSession consult provisioner.isUsable(lease) before using the lease; a dead process retires the binding. No per-call HTTP attestation on the hot path.
  • R1-10 + R1-31 + R1-30: bounded (32 KiB) ready-record read, reader catches Throwable, and the "closed before ready" diagnostic is no longer swallowed.
  • R1-26: capabilityDigest shape validated before spawn (non-retryable 400).
  • R1-33: RuntimeProvisioner extends AutoCloseable; RuntimeBrokerService.close() closes the provisioner.
  • R1-20: dropped the unread OwnedProcess.request component.
  • R1-12: hosted CI provisions Node 22 for the runtime-broker tests; the test fails loudly on hosted runners without node.
  • R1-24: design doc (EN + zh-CN) and PR description now scope re-attestation to the binding entrance.
  • wenshao R1-1: README boundary statement updated.
  • wenshao R1-2: worker stdout stays open and drained for the worker's lifetime.

Deferred (7 threads, resolved with reasons): R1-4 (env scrub needs a deliberate allowlist; prerequisite for the tools slice), R1-5 / R1-6 / R1-7 (unwired execute path, rework lands with the tools slice), R1-11 (bounded redacted stderr capture), R1-16 (fixture contract conformance), R1-18 (confirm coalescing / bounded executor), R1-23 (SIGTERM→SIGKILL escalation semantics).

Partially addressed (left open): R1-13 — lifecycle/diagnostic behaviors are now pinned; the execute-route request assertions (M3/M4/M7b) and the mismatched-identity fixture (M9/M5) remain for the tools slice.

CI note: the earlier Integration Tests (no-AK) failure was a self-hosted runner environment issue (dubious ownership at checkout-verify), unrelated to this diff; rerun requested. The 4 Qwen Autofix cancellations were a trigger storm cancel-in-progress; the latest was rerun.

@wenshao

wenshao commented Sep 24, 2026

Copy link
Copy Markdown
Collaborator

Round 2 local verification against the real worker — head 8157a218

Verdict: the round-1 blocker is fixed. Two smaller gaps sit in the same fix, and a 27-line patch closes both (verified). I'd take that patch, or an equivalent, before merging. Everything else here is hardening or a follow-up.

Fixed and verified against the real qwen managed-runtime-worker:

  • Round-1 F1 (a dead worker bricks its binding) is fixed. After SIGTERM or kill -9, the first warm gets 503 and the next warm provisions a new generation (A3/A4). The hung-worker case works too: after SIGSTOP, the re-attest times out at 30 s and the next warm re-provisions (D1).
  • service.close() now ends the workers (A5).
  • control refuses a dead lease before it reaches the transport (B1). A dispatch on a dead lease goes to UNKNOWN without calling execute (B2).
  • A failed adoption kills a worker that ignores SIGTERM (E1, via destroyForcibly in the new finally).

Still open:

# Finding Severity
N1 A binding retired after a failed re-attest leaves its worker alive and listening should-fix (R1-3 is only half done)
N2 A session acquired before the worker died can never be released. release returns 503 retryable=true forever should-fix / follow-up
F3 A non-loopback url in the ready record still receives the bearer token (round 1) hardening, not addressed
F4 A Broker crash still orphans the worker (round 1). The stdout drain does not help worker-side follow-up
E2 stop/close still only send SIGTERM deferred as R1-23, confirmed
T Mutation: the PR's tests kill 8/16 mutants of this fix test gap

Environment

  • The real worker is the bundle's node dist/cli.js managed-runtime-worker. The worker and contract sources are byte-identical to this head (git diff --quiet 04a146c9a7 8157a218 -- managed-runtime-attestation-{worker,contract}.ts).
  • A Java harness drives the PR's own LocalProcessRuntimeProvisioner + RuntimeBrokerService + HttpRuntimeTransport with in-memory repositories. Session verbs use an accepting RuntimeTransport, because the worker still exposes only attest.
  • macOS, JDK 26, Node 24. Module suite 84/84, 0 skipped, and checkstyle:check is clean. CI is green at this head on Java 11/17/21, macOS, Windows, MariaDB and Lint.

head

N1: a retired worker is never released

invalidateBinding removes the lease from liveBindings and fails the record. It never calls provisioner.release. The process stays in owned and keeps its port and token until provisioner.close().

  • C1: a worker that attests at adoption and then answers 409 managed_runtime_identity_conflict on re-attest. Across 8 warms the binding alternates OK gen=N / 409. Live workers go 1,1,2,2,3,3,4,4, and all 4 retired endpoints are still LISTEN. This is the case R1-3 called security-relevant: the thing answering on the lease is not the adopted identity, yet the process stays up.
  • D2: the real worker is SIGSTOPped. The re-attest times out and a new generation comes up. After SIGCONT the old pid is alive and still listening on 127.0.0.1. It only dies at service.close().
  • Minor: a 409 on re-attest is returned as retryable=false, yet the very next warm succeeds, because the binding was already retired.

Fix: invalidateBinding calls releaseQuietly(record.getRequest(), record.getLease()). release is keyed by runtimeInstanceId, so it cannot touch the new generation. With the patch, C1 goes 1,0,1,0,… with every retired endpoint closed, and D2 has the old pid gone.

N2: a session on a dead worker cannot be released

releaseSession starts with requireUsableLease. The session context keeps the old lease forever, so once its worker dies:

  • release(rt-1) returns 503 runtime_provision_failed retryable=true on every call (B4). The binding has already recovered at gen 2, and the session still never releases.
  • acquire(rt-1) returns READY, because the existing context is reused, and the next control gets 503 again.
  • Only a new runtimeSessionId works (B5).

So the session record and the sessions entry stay until the Broker restarts, and the caller is told to retry something that cannot succeed. That is the same shape as round-1 F1, one level down. Fix: when the lease is not usable, release completes locally. It retires the binding, keeps the runtime_session_busy guard, moves the record to RELEASED and removes the context. With the patch, 3× release → OK true. Re-acquiring the released id is a 409 runtime_session_conflict, as for any released id.

A session holding the UNKNOWN execution from B2 then answers 409 runtime_session_busy. That is the existing "UNKNOWN is not settled" rule and waits for reconciliation; I left it alone.

F3 / F4 / E2 (round 1, still open)

  • F3: a double whose ready url is http://<en0 LAN ip>:<port> still received Authorization: Bearer … plus tenant, workspace, cwd and capability digest (F1 in the figure). It was not in a review thread, so it may simply have been missed. The patch adds a 4-line http + 127.0.0.1 pin, the only address the worker binds.
  • F4: I kill -9ed a Broker JVM that had provisioned a real worker. 5 s later the worker had PPID 1 and was still listening. The new drain keeps the stdout pipe open, but the real worker writes nothing after ready, so it never sees the pipe close. It still needs a parent watch on the worker side.

orphan

  • E2: an adopted worker that ignores SIGTERM is still alive 6.5 s after service.close(). This confirms deferred R1-23 and is not a new ask.

Candidate patch (+27/−3): 84/84 and checkstyle clean

fix

The same harness on head + patch gives 12 pass / 1 fail. The one failure is E2, deferred as R1-23.

Patch
--- a/.../LocalProcessRuntimeProvisioner.java
+++ b/.../LocalProcessRuntimeProvisioner.java
@@ start()
+            URI endpoint = URI.create(String.valueOf(ready.get("url")));
+            if (!"http".equals(endpoint.getScheme())
+                    || !"127.0.0.1".equals(endpoint.getHost())) {
+                throw failed("Managed Runtime ready record is invalid.");
+            }
             RuntimeLease lease = new RuntimeLease(runtimeInstanceId,
-                    URI.create(String.valueOf(ready.get("url"))), token,
-                    leaseId, 1);
+                    endpoint, token, leaseId, 1);
--- a/.../RuntimeBrokerService.java
+++ b/.../RuntimeBrokerService.java
@@ releaseSession
-        requireUsableLease(context);
+        if (!provisioner.isUsable(context.lease())) {
+            invalidateBinding(context.binding());
+            synchronized (context) {
+                if (context.hasActiveControl()
+                        || executionRepository.hasActiveByRuntimeSession(
+                                context.session().getRuntimeSessionId())) {
+                    throw conflict("runtime_session_busy",
+                            "Runtime Session has an active operation");
+                }
+                RuntimeSessionRecord gone =
+                        transitionSessionToReleasing(context);
+                if (gone.getState() != RuntimeSessionRecord.State.RELEASED) {
+                    finishSessionRelease(gone);
+                }
+                sessions.remove(context.session().getRuntimeSessionId());
+                return CompletableFuture.completedFuture(true);
+            }
+        }
@@ invalidateBinding
         liveBindings.remove(record.getBindingId());
+        if (record.getLease() != null) {
+            releaseQuietly(record.getRequest(), record.getLease());
+        }

File: candidate-fix-8157a218.patch. It comes without tests. The real-worker scenarios above are its witness.

Test strength (mutation of the fix)

mutation

The reply says "each fix is pinned by a test that goes red when the fix is reverted". The full 84-test suite kills 8/16. The survivors:

  • N03/N04/N05, the control/cancel/release liveness checks, which are the R2-1 fix. Only the dispatch check (N06) is pinned.
  • N15, isUsable ignoring isAlive.
  • N08, the second fenced exit.
  • N10, the R1-25 finally (a regression to "never kill" stays green).
  • N14, the R1-3 "don't flatten 409 into 503" half.
  • N12, the drain. This one is near-equivalent: keeping the pipe open without draining only bites after 64 KiB of chatter.

N03's first-run kill came from HttpRuntimeTransportTest under parallel load. In isolation N03 survives, so that class may be timing-sensitive.

Housekeeping

qqqys's CHANGES_REQUESTED (the 9 no-undef errors) was fixed at f57765b and Lint is green here. The review decision is still CHANGES_REQUESTED, so it needs a re-review or a dismissal before merge.

Evidence (harness, worker doubles, raw logs, mutant driver, patch): wenshao/qwen-code@assets-pr12552/pr-12552-r2

中文版

第 2 轮:基于真实 worker 的本地验证 —— head 8157a218

结论:第 1 轮的阻塞项已修复。同一处修复里还剩两个较小的缺口,一个 27 行的补丁可以同时补上(已验证)。 建议合入前采纳这个补丁或等价改法。其余都属于加固或后续项。

在真实 qwen managed-runtime-worker 上验证已修复的部分:

  • 第 1 轮 F1(worker 死后 binding 永久不可用)已修复。 SIGTERM 或 kill -9 之后,第一次 warm 返回 503,下一次 warm 拉起新一代(A3/A4)。挂死的情况也成立:SIGSTOP 之后重新证明在 30 s 超时,下一次 warm 重新 provision(D1)。
  • service.close() 现在会结束 worker(A5)。
  • control 在调用 transport 之前就拒绝死 lease(B1)。在死 lease 上的 dispatch 会进入 UNKNOWN,不会调用 execute(B2)。
  • 接管失败时,会杀掉无视 SIGTERM 的 worker(E1,新的 finally 里走 destroyForcibly)。

仍未解决:

# 发现 级别
N1 重新证明失败而退役的 binding,其 worker 仍然存活并在监听 建议修复(R1-3 只修了一半)
N2 worker 死之前 acquire 的 session 永远无法 release,release 一直返回 503 retryable=true 建议修复 / 后续
F3 ready 记录里的非 loopback url 仍会收到 bearer token(第 1 轮) 加固,未处理
F4 Broker 崩溃仍让 worker 成为孤儿(第 1 轮),stdout drain 对此无效 worker 侧后续项
E2 stop/close 仍只发 SIGTERM 已按 R1-23 推迟,复核属实
T 变异:PR 自带测试对本次修复的变异只杀死 8/16 测试缺口

环境

  • 真实 worker 用的是 bundle 里的 node dist/cli.js managed-runtime-worker。worker 和契约的源码与本 head 逐字节一致(已用 git diff --quiet 确认)。
  • 用 Java harness 驱动 PR 自己的 LocalProcessRuntimeProvisioner + RuntimeBrokerService + HttpRuntimeTransport,repository 用内存实现。worker 目前只暴露 attest,所以 session 动词用一个全部接受的 RuntimeTransport。
  • macOS、JDK 26、Node 24。模块测试 84/84、0 跳过,checkstyle:check 干净。本 head 的 CI 在 Java 11/17/21、macOS、Windows、MariaDB 和 Lint 上全绿。

N1:退役的 worker 从未被释放

invalidateBinding 把 lease 移出 liveBindings,并把记录置为 FAILED,但从不调用 provisioner.release。进程仍留在 owned 里,端口和 token 一直有效,直到 provisioner.close()。

  • C1: worker 在接管时证明成功,之后重新证明都回 409 managed_runtime_identity_conflict。8 次 warm 里 binding 在 OK gen=N 和 409 之间交替。存活 worker 数为 1,1,2,2,3,3,4,4,4 个退役端点全部仍在 LISTEN。这正是 R1-3 说的安全相关情形:lease 背后应答的已经不是被接管的身份,进程却仍在运行。
  • D2: 对真实 worker 发 SIGSTOP。重新证明超时,新一代起来。SIGCONT 之后旧 pid 仍然存活,仍在 127.0.0.1 上监听,直到 service.close() 才退出。
  • 小问题:重新证明遇到的 409 以 retryable=false 返回,但 binding 其实已经退役,紧接着的下一次 warm 就会成功。

修法:invalidateBinding 调用 releaseQuietly(record.getRequest(), record.getLease())。release 按 runtimeInstanceId 定位,不会误伤新一代。打上补丁后,C1 的存活数变为 1,0,1,0,…,退役端点全部关闭;D2 的旧 pid 已退出。

N2:死 worker 上的 session 无法 release

releaseSession 一开头就是 requireUsableLease,而 session context 永远持有旧 lease。所以 worker 一死:

  • release(rt-1) 每次都返回 503 runtime_provision_failed retryable=true(B4)。此时 binding 已经恢复到 gen 2,session 仍然释放不了。
  • acquire(rt-1) 复用已有 context,返回 READY,下一次 control 又是 503。
  • 只有换一个新的 runtimeSessionId 才能用(B5)。

于是 session 记录和 sessions 条目会一直留到 Broker 重启,调用方还被告知可以重试一件不可能成功的事。这和第 1 轮 F1 是同一种形态,只是下沉到了 session 层。修法:lease 不可用时就在本地完成 release:退役 binding,保留 runtime_session_busy 守卫,把记录置为 RELEASED,并移除 context。打上补丁后,3 次 release 都是 OK true。重新 acquire 已释放的 id 返回 409 runtime_session_conflict,和其它已释放的 id 一样。

持有 B2 中 UNKNOWN 执行的 session 则返回 409 runtime_session_busy。这是现有的"UNKNOWN 不算已结算"规则,要等 reconciliation 处理,这里没有改动。

F3 / F4 / E2(第 1 轮,仍未解决)

  • F3: ready url 为 http://<局域网 IP>:<port> 的替身,仍然收到了 Authorization: Bearer … 以及租户、工作区、cwd 和 capability digest。这一项当时不在评审 thread 里,可能只是被漏看了。补丁加了 4 行,把地址固定为 http + 127.0.0.1,即 worker 唯一会绑定的地址。
  • F4: kill -9 掉已经拉起真实 worker 的 Broker JVM,5 s 后 worker 的 PPID 为 1,仍在监听。新加的 drain 让 stdout 管道保持打开,但真实 worker 在 ready 之后什么都不写,所以永远察觉不到管道已关闭。仍然需要在 worker 侧监视父进程。
  • E2: 一个无视 SIGTERM 的已接管 worker,在 service.close() 6.5 s 后仍然存活。这只是复核已推迟的 R1-23,不是新诉求。

候选补丁(+27/−3):84/84,checkstyle 干净

同一套 harness 在 head + 补丁上是 12 通过 / 1 失败,唯一的失败是已按 R1-23 推迟的 E2。补丁不含测试,上面这些真实 worker 场景就是它的验证。

测试强度(对本次修复做变异)

作者回复说"每处修复都由回退即变红的测试钉住"。实测完整的 84 个测试只杀死 8/16。存活的变异:

  • N03/N04/N05:control/cancel/release 的存活检查,即 R2-1 的修复。只有 dispatch 的检查(N06)被钉住。
  • N15:isUsable 忽略 isAlive。
  • N08:第二个 fenced 出口。
  • N10:R1-25 的 finally(退化成"从不杀进程"测试仍然全绿)。
  • N14:R1-3 中"不再把 409 压成 503"的那一半。
  • N12:drain。这个接近等价变异:保持管道打开但不读,只有输出超过 64 KiB 才会出问题。

N03 在第一轮被杀,是 HttpRuntimeTransportTest 在并行负载下失败导致的;单独运行时 N03 存活,说明这个测试类可能对时序敏感。

其它

qqqys 的 CHANGES_REQUESTED(9 个 no-undef)已在 f57765b 修复,本 head 的 Lint 为绿。但 review decision 仍是 CHANGES_REQUESTED,合入前需要重新评审或 dismiss。

证据(harness、worker 替身、原始日志、变异驱动、补丁):wenshao/qwen-code@assets-pr12552/pr-12552-r2

A failed re-attestation now releases the worker it retires, and a session
on a dead lease finishes release locally. A ready URL that is not http on
127.0.0.1 is rejected before the bearer token is sent.

Co-authored-by: jinye <[email protected]>
auto-merge was automatically disabled September 24, 2026 05:19

Head branch was pushed to by a user without write access

@doudouOUC

Copy link
Copy Markdown
Collaborator Author

Round 2

采纳 N1、N2、F3,在 9d08c65aa(fix(sdk-java): Release retired workers and pin the ready origin)。RuntimeBrokerServiceTest + LocalProcessRuntimeProvisionerTest + HttpRuntimeTransportTest 为 57/57,checkstyle:check 通过。

  • N1: invalidateBinding 在退役 binding 时调用 releaseQuietly。release 按 runtimeInstanceId 定位,不会动到新一代。重新证明失败后退役的 worker 会被停掉,而不是一直占着端口和 token。
  • N2: lease 已经不可用时,release 在本地完成:退役 binding,未结算执行仍然返回 runtime_session_busy,否则把 session 置为 RELEASED 并移出 context,不再调 transport。已经在进行中的 release 会被 join,避免和正在走的 transport release 抢写。重新 acquire 已释放的 id 仍然是 409 runtime_session_conflict。
  • F3: ready url 必须是 http 且 host 为 127.0.0.1,否则在发出 bearer token 之前按无效 ready 记录失败。进程测试用 --foreign-url 报告 172.16.1.234,断言没有 attest 请求落到替身上。

这一次证明失败返回的 409 仍然是 retryable=false。那是这一次调用的结果;binding 已经退役,下一次 warm 是新的 provision,不是同一次失败调用的重试。

不采纳

  • F4(Broker 崩溃后 worker 成为孤儿): Java 进程被 kill -9 之后没法再收割子进程。stdout drain 只是保持管道打开;真实 worker 在 ready 之后不再写 stdout,看不到管道关闭。这需要 worker 侧监视父进程,不在这一笔 Java broker 里改。
  • E2 / R1-23(stop/close 只发 SIGTERM): 与上一轮已经推迟的项相同,不是新问题。已合入的 worker 会在 SIGTERM 时退出;对无视 SIGTERM 的进程做 SIGKILL 升级会改变 teardown 时序,仍留在生命周期加固。
  • 变异测试里其余存活项: 这一轮用测试钉住的是退役时 release、死 lease 的本地 release,以及非 loopback ready URL。control/cancel 的存活检查、isUsable 忽略 isAlive、第二个 fenced 出口、finally、409 不压成 503、以及 drain,不作为这一轮的合入条件。drain 保持管道打开但不读,只有输出超过 64 KiB 才会分岔,接近等价变异。
English

Adopted N1, N2, and F3 in 9d08c65aa. Retiring a binding releases that lease. release on an unusable lease finishes locally (runtime_session_busy still wins while an execution is unsettled) and joins a release that is already in flight. A ready URL other than http on 127.0.0.1 is rejected before the bearer token is sent.

Not adopted: F4 needs a worker-side parent watch after a Broker crash, which this Java slice does not own. E2 is the already-deferred R1-23 SIGTERM escalation. The remaining mutation survivors are outside this round; the new tests pin the three adopted fixes. A 409 from re-attestation stays retryable=false because that call failed; the next warm is a new provision.

@wenshao

wenshao commented Sep 24, 2026

Copy link
Copy Markdown
Collaborator

Round 3 local verification against the real worker — head 9d08c65a

Verdict: the three fixes adopted from round 2 (N1, N2, F3) hold against the real qwen managed-runtime-worker. I found no new correctness problem. Before merging, two CI items need fixing, and neither is in the production code:

  1. macOS / Java 21 is red because fencedProvisioningReapsTheWorker depends on timing. The fence fires only if the first 33 ms renewal tick beats the Node double's boot. A 6-line change to the test double makes the result deterministic (verified below).
  2. Lint & Static is red only at Check lint gate freshness. .github/workflows/ci.yml changed on main in a5a5eb55 (ci(web-shell): give E2E Smoke 30 minutes #12593). Merging origin/main into the branch clears it. ESLint --max-warnings 0 and Prettier both pass on the changed .mjs at this head.

Environment

  • The real worker is node dist/cli.js managed-runtime-worker from an existing bundle. Its managed-runtime-attestation-{worker,contract}.ts are byte-identical to this head and to current origin/main.
  • The round-2 Java harness is extended with a new scenario G. It drives the PR's LocalProcessRuntimeProvisioner, RuntimeBrokerService and HttpRuntimeTransport with in-memory repositories. Both heads ran on the same harness and the same bundle.
  • macOS, JDK 21, Node 24. The module suite is 87/87, 0 skipped on JDK 21, 25 and 26. checkstyle:check is clean.

A/B

The round-2 fixes: previous head vs this head

# Scenario 8157a218 9d08c65a
N1 C1: the worker answers 409 on re-attest, 8 warms live workers 1,1,2,2,3,3,4,4, all retired ports LISTEN live 1,0,1,0,…, all 4 retired ports closed
N1 D2: SIGSTOP, re-attest times out, then SIGCONT old pid alive and listening old pid gone
N2 B4: release of a session whose worker died 503 retryable=true ×3 OK true ×3. Re-acquiring that id returns 409 runtime_session_conflict
F3 F1: ready url = LAN IP bearer token and scope sent refused, 0 bytes captured
new G1: after recovery, release the stale gen-1 session 503 OK true. Gen 2 keeps its bindingId, port and pid. Worker count stays 1→1

I added G1 to check whether the retirement that release now triggers can hit the recovered binding. It can't: every re-provision mints a new bindingId, so invalidateBinding(context.binding()) only ever touches the dead generation.

Head totals are 13 pass / 1 fail. The one failure is E2: an adopted worker that ignores SIGTERM outlives close(). That is the deferred R1-23, as agreed. F4 (worker-side parent watch) also stays a follow-up.

macOS CI: fencedProvisioningReapsTheWorker depends on timing

Run 35959440392 failed with Expected ExecutionException ... nothing was thrown in 0.082 s. The same test passed on macOS at 8157a218 and passes on ubuntu, Windows and MariaDB at this head.

FencingBindingRepository fences only renewOperation. The first renewal runs operationLease / 3 = 33 ms after the claim. If the worker boots and attests before that tick, stopAndGet() still returns the claim, the READY CAS succeeds and nothing is fenced. The whole local class took 60–90 ms to provision, so the margin is small. Changing only the lease value makes this deterministic:

fence race

Failing runs take about 0.09 s, which matches the CI failure: the tick lost the race. Fix in the test only: the double also fences the READY compareAndSet. The outcome then no longer depends on scheduling. With this patch the test passes at every lease value (16/16). It still fails 3/3 against a mutant that drops both fenced-exit releaseQuietly calls. The suite stays 87/87 and checkstyle stays clean.

Test patch (+6/−1)
--- a/packages/sdk-java/runtime-broker/src/test/java/com/alibaba/qwen/code/runtimebroker/LocalProcessRuntimeProvisionerTest.java
+++ b/packages/sdk-java/runtime-broker/src/test/java/com/alibaba/qwen/code/runtimebroker/LocalProcessRuntimeProvisionerTest.java
-    /** Fences every renewal so provisioning loses its claim mid-boot. */
+    /** Fences renewal and the READY write, so provisioning always loses. */
@@ FencingBindingRepository
         public RuntimeBindingRecord compareAndSet(
                 RuntimeBindingRecord expected,
                 RuntimeBindingRecord replacement) {
+            // Also fence the READY write, so the outcome does not depend on
+            // whether the first renewal tick beats the worker's boot.
+            if (replacement.getState() == RuntimeBindingRecord.State.READY) {
+                return null;
+            }
             return delegate.compareAndSet(expected, replacement);
         }

Full file: test-fence-9d08c65a.patch

Mutation on this round's fixes: 7/9 killed

mutation

The new tests pin the release-on-retire, local-release and loopback-host changes. Two mutants survive. Neither blocks the merge:

  • M5: dropping the "join an in-flight release" branch in releaseUnusableSession. No test drives a release whose transport call is still pending when the worker dies.
  • M9: dropping the http scheme half of the ready-URL check, so https://127.0.0.1:… is accepted. A one-line --https-url variant of the existing test would pin it.

Before merge

  • Take the test patch above, or an equivalent (the macOS red).
  • Merge origin/main into the branch (the lint freshness red). Please don't rebase: the inline threads would lose their anchors.
  • Earlier CHANGES_REQUESTED reviews still block the PR. Their items are addressed at this head, so they need a re-review or a dismissal.

Harness, logs and mutation driver: pr-12552-r3/

中文说明

第三轮本地真实 worker 验证 —— head 9d08c65a

结论:第二轮采纳的三项修复(N1、N2、F3)在真实 qwen managed-runtime-worker 上成立,没有发现新的正确性问题。合入前还有两个 CI 项要处理,都不在生产代码里:

  1. macOS / Java 21 红,原因是 fencedProvisioningReapsTheWorker 依赖时序。 只有第一次 33 ms 的续约 tick 抢在 Node 替身启动完成之前,fence 才会生效。给测试替身改 6 行就能让结果确定(下文已验证)。
  2. Lint & Static 只红在 Check lint gate freshness。 main 在 a5a5eb55(ci(web-shell): give E2E Smoke 30 minutes #12593)改了 .github/workflows/ci.yml,把 origin/main merge 进分支即可。改动的 .mjs 在这个 head 上过 ESLint --max-warnings 0 和 Prettier。

环境

  • 真实 worker 用现有 bundle 的 node dist/cli.js managed-runtime-worker。managed-runtime-attestation-{worker,contract}.ts 与这个 head、与当前 origin/main 逐字节相同。
  • 沿用第二轮的 Java harness,新增场景 G。它驱动 PR 的 LocalProcessRuntimeProvisioner、RuntimeBrokerService、HttpRuntimeTransport,仓库用内存实现。两个 head 用同一套 harness、同一份 bundle。
  • macOS,JDK 21,Node 24。模块测试在 JDK 21、25、26 上都是 87/87,0 跳过,checkstyle:check 通过。

第二轮修复:上一个 head 对比这个 head

# 场景 8157a218 9d08c65a
N1 C1:re-attest 返回 409,连续 8 次 warm 存活 worker 1,1,2,2,3,3,4,4,退役端口全部还在 LISTEN 1,0,1,0,…,4 个退役端口全部关闭
N1 D2:SIGSTOP,re-attest 超时,再 SIGCONT 旧 pid 存活且还在监听 旧 pid 已结束
N2 B4:worker 已死的 session 做 release 503 retryable=true ×3 OK true ×3;再 acquire 同一个 id 返回 409 runtime_session_conflict
F3 F1:ready url 为局域网 IP bearer token 和 scope 被发出 被拒,抓到 0 字节
新 G1:恢复之后 release 旧的 gen-1 session 503 OK true;gen 2 的 bindingId、端口、pid 都不变,worker 数保持 1→1

G1 是我新加的,用来检查 release 现在会触发的退役会不会误伤已经恢复的 binding。不会:每次重新 provision 都会生成新的 bindingId,所以 invalidateBinding(context.binding()) 只会碰到已经死掉的那一代。

head 合计 13 通过 / 1 失败。唯一的失败是 E2:无视 SIGTERM 的已接管 worker 在 close() 之后仍然存活,这就是已约定推迟的 R1-23。F4(worker 侧监视父进程)同样留作后续。

macOS CI:fencedProvisioningReapsTheWorker 依赖时序

运行 35959440392 报 Expected ExecutionException ... nothing was thrown,耗时 0.082 s。同一个测试在 8157a218 的 macOS 上通过,在这个 head 的 ubuntu、Windows、MariaDB 上也通过。

FencingBindingRepository 只 fence 了 renewOperation。第一次续约在 claim 之后 operationLease / 3 = 33 ms 才跑。如果 worker 在这次 tick 之前就启动并完成证明,stopAndGet() 仍会返回 claim,READY 的 CAS 成功,fence 不会发生。本地整个类的 provision 耗时 60–90 ms,余量很小。只改 lease 取值就能稳定复现:lease 100 ms 时 4/4 通过;300 ms、600 ms 时 4/4 失败;60 s 单独运行 3/3 失败。失败的运行约 0.09 s,与 CI 那次失败的耗时一致:tick 没抢过启动。

只改测试的修法:替身同时 fence READY 的 compareAndSet,结果不再依赖调度。打上补丁后,各个 lease 取值下全部通过(16/16)。对于同时去掉两个 fenced 出口 releaseQuietly 的变异体,测试仍然 3/3 失败。模块测试保持 87/87,checkstyle 通过。补丁见上方英文部分。

本轮修复的变异测试:9 个杀掉 7 个

新测试钉住了退役时 release、本地 release 和 loopback host 检查。两个存活体都不阻塞合入:

  • M5: 去掉 releaseUnusableSession 里「join 进行中的 release」分支。没有测试覆盖 transport release 还没完成、worker 就已经死掉的情况。
  • M9: 去掉 ready URL 检查里 http scheme 那一半,https://127.0.0.1:… 会被接受。给现有测试加一个 --https-url 变体,一行就能钉住。

合入前

  • 采纳上面的测试补丁或等价修改(macOS 的红)。
  • 把 origin/main merge 进分支(lint freshness 的红)。请不要 rebase,否则行内评论会失去锚点。
  • 早先的 CHANGES_REQUESTED 评审仍在阻塞 PR。它们的问题在这个 head 上已经处理,需要重新评审或 dismiss。

harness、日志、变异脚本:pr-12552-r3/

Pick up the lint gate change so CI revalidates this head.

Co-authored-by: jinye <[email protected]>
@doudouOUC
doudouOUC enabled auto-merge September 24, 2026 06:17
Refuse the READY publish instead of waiting for the renewal timer, so a
fast machine cannot finish warm before the claim is lost.

Co-authored-by: jinye <[email protected]>
auto-merge was automatically disabled September 24, 2026 06:34

Head branch was pushed to by a user without write access

@doudouOUC
doudouOUC enabled auto-merge September 24, 2026 06:44
@wenshao

wenshao commented Sep 24, 2026

Copy link
Copy Markdown
Collaborator

Round 4 local verification against the real worker — head d1689c16

Verdict: mergeable. Both CI items from round 3 are fixed, and CI is fully green at this head. The merge of main (#12583's cancel changes) composes correctly with the liveness check, verified on the real worker. The real-worker results match round 3. What's left is one optional test gap: the fence-test change made the test deterministic, but the first fenced exit is now covered by no test at all. A 56-line companion test covers it (verified). It's fine as a follow-up.

What changed since 9d08c65a

  • 74f62d94 merges main. For this module that brings in fix(java): Deliver broker cancellation after a dispatch lease lapses #12583 (invocations set, cancel of an UNKNOWN call that is still in flight). The only textual crossing with this PR is in cancelExecution: the new UNKNOWN-still-running branch now reaches requireUsableLease.
  • d1689c16 makes the fencing double refuse the READY compareAndSet and lets renewal delegate, with a 1-minute lease. That is the round-3 suggestion, and the renewal half goes one step further.

Results

head

  • CI: macOS / Java 21 and Lint & Static, both red last round, now pass. So do MariaDB, Windows, ubuntu Java 11/17/21, Real daemon E2E and the Node lanes.

  • Module suite: 94/94, 0 skipped on JDK 21, 25 and 26. checkstyle:check is clean.

  • Fence test: 20/20 pass with the lease swept over 100 ms, 300 ms, 600 ms and 1 min. At 9d08c65a it failed 4/4 at 300 ms and 600 ms.

  • Real worker: the bundle's qwen managed-runtime-worker; its worker and contract sources are identical to this head and to origin/main. Scenarios A–G give the same results as round 3. N1, N2 and F3 stay fixed. The only failure is E2, the deferred R1-23.

  • New I1, for the merge crossing: start a call whose execute stays pending, then cancel it.

    • Worker alive: transport.cancel is called once.
    • After kill -9 of the worker, cancel returns 503 runtime_provision_failed and transport is never reached. When execute then fails, the call becomes UNKNOWN. A later cancel returns UNKNOWN. release returns 409 runtime_session_busy, the existing "UNKNOWN is not settled" rule. The next warm provisions gen 2.

    So fix(java): Deliver broker cancellation after a dispatch lease lapses #12583's physical cancel works on a live worker and fails closed on a dead one. Total: 14 pass / 1 fail (E2).

Optional: the first fenced exit has lost its test

provisionBinding has two runtime_provision_fenced exits, each with its own releaseQuietly:

  1. The renewal lost the claim (currentClaim == null).
  2. The READY CAS was refused (ready == null).

The PR's test now reaches only exit 2. Before, it reached exit 1 whenever the timer won, which was flaky but real coverage.

exits

Deleting exit 1's release survives the PR's tests (3/3 pass). The companion test is deterministic by construction. The double fences renewal and counts down a latch. A gated provisioner starts the boot only after that renewal has already lost the claim.

  • It passes 10/10.
  • It kills X1 (3/3). X2 and X3 are still caught.
  • The suite is 95/95 and checkstyle is clean.
Companion test patch (+56/−3)
@@ LocalProcessRuntimeProvisionerTest
     @Test
     void fencedProvisioningReapsTheWorker() throws Exception {
+        assertFencedProvisioningReaps(false);
+    }
+
+    @Test
+    void provisioningFencedByRenewalReapsTheWorker() throws Exception {
+        assertFencedProvisioningReaps(true);
+    }
+
+    private void assertFencedProvisioningReaps(boolean atRenewal)
+            throws Exception {
         requireNode();
@@
         Set<Long> before = childPids();
+        FencingBindingRepository bindings =
+                new FencingBindingRepository(atRenewal);
         try (LocalProcessRuntimeProvisioner provisioner = ...) {
+            // In renewal mode, provisioning starts only after the first
+            // renewal has already lost the claim.
+            RuntimeProvisioner gated = request -> {
+                try {
+                    assertTrue(bindings.renewed.await(10, TimeUnit.SECONDS));
+                } catch (InterruptedException exception) {
+                    throw new IllegalStateException(exception);
+                }
+                return provisioner.provision(request);
+            };
             RuntimeBrokerService service = new RuntimeBrokerService(
                     ...,
-                    provisioner, new AcceptingTransport(),
-                    new FencingBindingRepository(),
+                    atRenewal ? delegating(gated, provisioner) : provisioner,
+                    new AcceptingTransport(), bindings,
                     ...,
-                    "broker", Duration.ofMinutes(1), Duration.ofMinutes(1));
+                    "broker",
+                    atRenewal ? Duration.ofMillis(300) : Duration.ofMinutes(1),
+                    Duration.ofMinutes(1));
@@
+    /** Gates provision; forwards release so the service can reap. */
+    private static RuntimeProvisioner delegating(RuntimeProvisioner gate,
+            LocalProcessRuntimeProvisioner target) {
+        return new RuntimeProvisioner() {
+            @Override
+            public java.util.concurrent.CompletionStage<RuntimeLease>
+                    provision(RuntimeProvisionRequest request) {
+                return gate.provision(request);
+            }
+
+            @Override
+            public java.util.concurrent.CompletionStage<Void> release(
+                    RuntimeProvisionRequest request, RuntimeLease lease) {
+                return target.release(request, lease);
+            }
+        };
+    }
@@ FencingBindingRepository
+        private final boolean fenceRenewal;
+        final java.util.concurrent.CountDownLatch renewed =
+                new java.util.concurrent.CountDownLatch(1);
+
+        FencingBindingRepository(boolean fenceRenewal) {
+            this.fenceRenewal = fenceRenewal;
+        }
@@ renewOperation
+            if (fenceRenewal) {
+                renewed.countDown();
+                return null;
+            }
             return delegate.renewOperation(bindingId, owner,
                     operationGeneration, leaseDuration);

Full file: test-renewal-fence-d1689c16.patch

Round-3 mutation on this head: the same 7/9 are killed. The survivors are unchanged: M5 (in-flight release join) and M9 (http scheme half of the ready-URL check). Both are small, and neither blocks.

Before merge

  • The standing CHANGES_REQUESTED reviews need a re-review or a dismissal. Their items are addressed at this head.
  • Optional: the companion test above, now or as a follow-up.
  • Follow-ups, as already agreed: E2/R1-23 (SIGKILL escalation) and F4 (worker-side parent watch).

Harness, logs and mutation driver: pr-12552-r4/

中文说明

第四轮本地真实 worker 验证 —— head d1689c16

结论:可以合入。 第三轮的两个 CI 问题都已修好,这个 head 上 CI 全绿。merge 进来的 main(#12583 的 cancel 改动)和存活检查组合正确,已在真实 worker 上验证。真实 worker 的结果与第三轮一致。剩下一个可选的测试缺口:fence 测试改成确定性之后,第一个 fenced 出口已经没有任何测试覆盖。我写了一个 56 行的配套测试补上(已验证),作为后续处理也可以。

相对 9d08c65a 的变化

  • 74f62d94 merge 了 main。对这个模块来说是带进了 fix(java): Deliver broker cancellation after a dispatch lease lapses #12583(invocations 集合、对仍在执行的 UNKNOWN 调用做 cancel)。与这个 PR 唯一的文本交叉在 cancelExecution:新增的「UNKNOWN 但仍在运行」分支现在会走到 requireUsableLease。
  • d1689c16 让 fencing 替身拒绝 READY 的 compareAndSet,续约改为委托给真实仓库,lease 设为 1 分钟。这是第三轮的建议,续约那一半还多走了一步。

结果

  • CI: 上轮红的 macOS / Java 21 和 Lint & Static 现在都通过;MariaDB、Windows、ubuntu Java 11/17/21、Real daemon E2E、Node 各条也都通过。

  • 模块测试: JDK 21、25、26 上都是 94/94,0 跳过。checkstyle:check 通过。

  • fence 测试: lease 取 100 ms、300 ms、600 ms、1 分钟,20/20 全部通过。在 9d08c65a 上 300 ms 和 600 ms 时是 4/4 失败。

  • 真实 worker: 用 bundle 里的 qwen managed-runtime-worker,worker 和 contract 源码与这个 head、与 origin/main 相同。A–G 场景与第三轮结果一致,N1、N2、F3 保持修复。唯一的失败是 E2,即推迟的 R1-23。

  • 新增 I1(merge 交叉点): 发起一个 execute 一直挂起的调用,然后 cancel。

    • worker 存活时:transport.cancel 被调用 1 次。
    • kill -9 掉 worker 之后:cancel 返回 503 runtime_provision_failed,完全不碰 transport。随后 execute 失败,调用变为 UNKNOWN;再 cancel 返回 UNKNOWN;release 返回 409 runtime_session_busy,这是现有的「UNKNOWN 不算已结算」规则。下一次 warm 起了 gen 2。

    也就是说,fix(java): Deliver broker cancellation after a dispatch lease lapses #12583 的物理 cancel 在 worker 存活时生效,在 worker 已死时失败关闭。合计 14 通过 / 1 失败(E2)。

可选:第一个 fenced 出口失去了测试

provisionBinding 有两个 runtime_provision_fenced 出口,各自调用 releaseQuietly:

  1. 续约丢失了 claim(currentClaim == null)。
  2. READY 的 CAS 被拒(ready == null)。

PR 现在的测试只能走到出口 2。之前在计时器抢赢时会走到出口 1,那是不稳定但真实的覆盖。

删掉出口 1 的 release,PR 的测试检测不出来(3/3 通过)。配套测试在构造上就是确定的:替身让续约 fence 并对 latch 计数,受控的 provisioner 等到续约已经丢失 claim 之后才启动 worker。

  • 通过 10/10。
  • 能杀掉 X1(3/3);X2、X3 仍然能检测到。
  • 模块测试 95/95,checkstyle 通过。

补丁见上方英文部分。

本 head 上重跑第三轮的变异:同样杀掉 7/9。存活的仍是 M5(join 进行中的 release)和 M9(ready URL 检查里 http scheme 那一半),都很小,都不阻塞合入。

合入前

  • 仍在生效的 CHANGES_REQUESTED 评审需要重新评审或 dismiss;它们提的问题在这个 head 上已经处理。
  • 可选:上面的配套测试,现在加或后续加都可以。
  • 已约定的后续项:E2/R1-23(SIGKILL 升级)和 F4(worker 侧监视父进程)。

harness、日志、变异脚本:pr-12552-r4/

@wenshao

wenshao commented Sep 24, 2026

Copy link
Copy Markdown
Collaborator

@qwen-code /triage

@doudouOUC
doudouOUC added this pull request to the merge queue Sep 24, 2026
Merged via the queue into QwenLM:main with commit 3c6b614 Sep 24, 2026
59 checks passed
wenshao added a commit to doudouOUC/qwen-code that referenced this pull request Sep 25, 2026
…s-p0-p8

Resolve conflicts with main and carry main's runtime-broker changes into
this branch's broker architecture:

- Keep the branch's RuntimeBrokerService, HttpRuntimeTransport and
  LocalProcessRuntimeProvisioner, which supersede main's narrower
  versions (QwenLM#12552).
- Port reconcileExecution (QwenLM#12655): an UNKNOWN execution is settled only
  on the original binding generation's terminal status answer, through a
  health-checked lease, one bounded in-flight lookup per execution.
  Worker status responses carry progress fields, so only state and
  result are validated.
- Port the lapsed-claim cancel fix (QwenLM#12583): a cancel still reaches an
  invocation this process is running after its dispatch claim lapsed or
  was fenced to UNKNOWN, without writing the Runtime's answer.
- Send Cache-Control: no-store on Managed Runtime v2 requests.
- Restore main's tool-execution repository contract (typed round trip,
  takeover fences, forgeries) in JdbcRepositoryContract.
- Keep main's Settings deep-link when initialising the active panel in
  the web shell, alongside the Managed panel selection.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants