Skip to content

fix(serve): stop renewing settled Hosted Shell grants - #13199

Draft
yc2bgr8 wants to merge 2 commits into
QwenLM:mainfrom
yc2bgr8:fix/hosted-shell-settled-grants-12957
Draft

yc2bgr8 wants to merge 2 commits into
QwenLM:mainfrom
yc2bgr8:fix/hosted-shell-settled-grants-12957

Conversation

@yc2bgr8

@yc2bgr8 yc2bgr8 commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

What this PR does

Stops renewing a Hosted Shell publication once the original execution has authoritatively settled, while retaining its binding until receipt acceptance so failure cleanup still works. Each renewal pass attempts all live bindings before reporting failures with their publication IDs. A renewal already in flight when settlement is observed can finish without producing a misleading warning; genuine renewal failures still block dispatch.

Why it's needed

A slow Session receipt leaves the settled execution in the batch's bookkeeping. The publication service correctly rejects renewal of settled executions, but the client previously kept renewing them and logged a misleading warning. The first rejection also prevented queued publications later in the same pass from being renewed.

Reviewer Test Plan

How to verify

  1. Hold the first Shell execution open in a two-call batch and advance the renewal timer. Both its running publication and the queued second publication should renew.
  2. Settle the first execution, hold its receipt commit across two timer ticks, and check that only the queued publication renews. Release the receipt: both calls should complete, ownership should release normally, and no publication-renewal warning should appear. Repeat for original-finished-publication recovery and a proven not-started execution.
  3. Let an already-running renewal reject after settlement is observed. It should not warn or prevent renewal of the queued publication.
  4. Reject an unsettled timer renewal with HTTP 503. The same tick should still renew the queued publication and log exactly one warning naming the failed publication. The next successful tick should retry both live publications. HTTP 400 or 503 during pre-dispatch renewal must still block dispatch after attempting all bindings.
  5. Return cancelled-before-dispatch with no capture, then reject the finished-publication lookup. Recovery must cancel the original execution and send close_not_started with the original publication ID and token, without releasing uncertain ownership.

Evidence (Before & After)

N/A for visual evidence: internal Hosted lifecycle behavior, no TUI layout change. With the final regression tests on unchanged base 8eaaa13, seven cases fail and the cleanup-preservation case passes. Settled publications are renewed, delayed receipts produce spurious warnings and skip queued grants, and genuine per-binding errors stop the renewal pass. The exact same eight cases all pass with the fix. After the fix, all 581 Hosted CLI tests (15 files) and 1,927 managed-runtime core tests (34 files) pass, 2,508 in total. Full build, full typecheck (4 GiB Node heap), bundle, repository-wide lint:ci, changed-file Prettier, diff whitespace checks, and the serve-fast-path bundle closure check also pass.

Tested on

OS Status
🍏 macOS ⚠️ not tested
🪟 Windows ⚠️ not tested
🐧 Linux ✅ deterministic component tests

Environment (optional)

Debian 13, Node.js 24.19.0, repository-pinned pnpm 11.24.0. Usage statistics and OTel are disabled with QWEN_USAGE_STATISTICS_ENABLED=false and QWEN_TELEMETRY_ENABLED=false. The component tests exercise the actual Hosted turn coordinator and local Session/harness persistence with controlled Broker and publication-service fixtures. Only interval timers are advanced. The global qwen CLI is unavailable in this environment.

Risk & Scope

  • Main risk or tradeoff: renewal eligibility must not erase the original binding needed by failure cleanup; the regression suite pins both paths. Review remains important for this Hosted lifecycle code.
  • Not validated / out of scope: full npm run preflight did not complete. Its default 2 GiB integration-typecheck heap exhausted memory; a separate full typecheck passes with NODE_OPTIONS=--max-old-space-size=4096. The repository-wide unit-test run then encounters EPERM when browser-use tests bind Unix-domain sockets in this environment. The scoped Hosted suites reported above pass; no full-suite pass or real Java Broker, Session Store, database/OSS E2E is claimed. Tenant-wide reservation-expiry races remain outside this change.
  • Breaking changes / migration notes: none; no protocol, server, ownership, or public-interface changes.
  • AI assistance: implementation and tests were produced with OpenAI Codex. Submitted as a draft; human review is pending.

Linked Issues

Fixes #12957. Follow-up to merged #12894.

中文说明

本 PR 的改动

在原始执行被权威地确认 settled 后,停止续期对应的 Hosted Shell publication,同时保留其 binding 直到回执接纳完成,确保失败清理仍能工作。每轮续期都会尝试所有仍有效的 binding,然后携带 publication ID 报告失败。在观察到 settled 时已经发出的续期请求,即使随后失败也不会产生误导性告警;真实的续期失败仍会阻止派发。

为什么需要

Session 回执提交缓慢时,已 settled 的执行仍保留在批次的记录中。publication 服务按设计拒绝对已 settled 执行续期,但客户端此前仍持续请求续期并输出误导性告警。第一次拒绝还会阻断本轮后续排队 publication 的续期。

Reviewer 测试计划

如何验证

  1. 在包含两个 Shell 调用的批次中,让第一个执行保持运行并推进续期定时器。预期正在运行的第一个 publication 与排队中的第二个 publication 都得到续期。
  2. 让第一个执行 settled,将回执提交阻塞两个定时器周期,并确认仅排队中的 publication 继续续期。释放回执后,两个调用都应完成、正常释放归属,且不出现 publication 续期告警。对通过原始 finished publication 恢复以及已证明未启动的执行重复验证。
  3. 让一个已发出的续期请求在客户端观察到 settled 后才拒绝。预期不会告警,也不会阻止排队 publication 的续期。
  4. 让尚未 settled 的 publication 在定时续期时返回 HTTP 503。该轮仍应续期排队的 publication,且只输出一条包含失败 publication ID 的告警。下一轮成功续期应再次尝试两个仍有效的 publication。派发前续期遇到 HTTP 400 或 503 时,仍必须在尝试所有 binding 后阻止派发。
  5. 返回无 capture、派发前取消的执行结果,再拒绝 finished publication 查询。恢复路径必须取消原始执行,并使用原始 publication ID 和 token 发送 close_not_started,且不能释放尚不确定的归属。

证据(修改前与修改后)

不适用可视化证据:这是内部 Hosted 生命周期行为,不涉及 TUI 布局变化。将最终回归测试运行在未修改的基线 8eaaa13 上,七项失败、清理行为保留测试通过。已 settled 的 publication 仍被续期,延迟回执产生误报告警并跳过排队 grant,而单个 binding 的真实错误也会终止本轮续期。相同的八项测试在应用修复后全部通过。修复后,581 项 Hosted CLI 测试(15 个文件)和 1,927 项 managed-runtime core 测试(34 个文件)全部通过,合计 2,508 项。完整 build、完整 typecheck(4 GiB Node 堆)、bundle、全仓 lint:ci、变更文件 Prettier 检查、diff 空白检查以及 serve-fast-path bundle 闭包检查均通过。

已测试的平台

操作系统 状态
🍏 macOS ⚠️ 未测试
🪟 Windows ⚠️ 未测试
🐧 Linux ✅ 确定性组件测试

环境(可选)

Debian 13、Node.js 24.19.0、仓库固定版本 pnpm 11.24.0。测试通过 QWEN_USAGE_STATISTICS_ENABLED=false 和 QWEN_TELEMETRY_ENABLED=false 关闭使用统计与 OTel。组件测试使用真实的 Hosted turn 协调器和本地 Session/harness 持久化,Broker 和 publication 服务由可控 fixture 提供。只推进 interval 定时器。此环境没有全局 qwen CLI。

风险与范围

  • 主要风险或取舍:续期资格变化不能删除失败清理仍需使用的原始 binding;回归测试同时约束这两条路径。此 Hosted 生命周期代码仍需仔细评审。
  • 未验证或范围外:完整 npm run preflight 未完成。集成类型检查使用默认 2 GiB 堆时耗尽内存;以 NODE_OPTIONS=--max-old-space-size=4096 单独重跑完整 typecheck 后通过。随后全仓单元测试中的 browser-use 测试在此环境绑定 Unix-domain socket 时遇到 EPERM。下述限定范围的 Hosted 测试均通过;未声称全量测试或真实 Java Broker、Session Store、数据库/OSS E2E 通过。租户级 reservation 过期竞争仍不在本次变更范围内。
  • 破坏性变更或迁移说明:无;协议、服务端、归属和公共接口均无变更。
  • AI 辅助说明:实现与测试由 OpenAI Codex 辅助完成。以草稿提交,尚待人工评审。

关联 Issue

Fixes #12957。作为已合入 #12894 的后续修复。

@yc2bgr8

yc2bgr8 commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

CI follow-up for a48c37a:

  • The Linux unit-test job failed only on workspace-agents.test.ts > closes an event stream before a replacement runtime can publish to it: ENOTEMPTY while removing its temporary agent-host directory, including both built-in retries. The CLI suite had 38,049 passing tests; the affected Hosted Shell suite passed all 143 tests.
  • This appears to be an existing teardown race. Route registration starts asynchronous recovery, but stopping recovery does not wait for an in-flight recovery before the test deletes its directory. The test never executes the changed Hosted Shell coordinator. Its route and test are unchanged on current main 5fca3be7.
  • An isolated rerun passed (1 passed, 5 skipped), while logging the corresponding ENOENT during an atomic write to agent-host/workspace.json after teardown. Lint/static, Serve A/B, no-AK integration, and the Hosted process fault gates passed.

Could a maintainer rerun the failed Linux unit-test job? Keeping this draft focused on #12957; no unrelated cleanup changes or extra push have been made.

中文说明

a48c37a 的 CI 跟进:

  • Linux 单元测试任务 仅在 workspace-agents.test.ts > closes an event stream before a replacement runtime can publish to it 失败:删除临时 agent-host 目录时出现 ENOTEMPTY,内置的两次重试也遇到同样问题。CLI 测试有 38,049 项通过,本次涉及的 Hosted Shell 测试全部 143 项通过。
  • 证据指向已有的 teardown 竞争:路由注册会启动异步恢复,但停止恢复时不会等待已在执行的恢复结束,测试随后便删除临时目录。该测试 不执行本次修改的 Hosted Shell 协调器。当前 main 5fca3be7 上的相关路由和测试内容均未变化。
  • 单独重跑该测试通过(1 项通过、5 项跳过),但记录了对应的 teardown 后原子写入 agent-host/workspace.json 时的 ENOENT。Lint/static、Serve A/B、no-AK 集成和 Hosted 进程故障检查均已通过。

请维护者协助重跑失败的 Linux 单元测试任务。此草稿继续聚焦 #12957,没有加入无关清理改动,也没有额外 push。

…led-grants-12957-update

# Conflicts:
#	packages/cli/src/serve/hosted-workspace-tool-turn.ts
@qwen-code-review-bot

Copy link
Copy Markdown
Collaborator

🩺 serve daemon A/B

Built the PR base vs this PR head b6ee9a1, drove a fixed endpoint set against each, and diffed the JSON responses. Only fields that changed are shown.

health-deep-with-session

field PR base (before) this PR (after)
activeWorkStaleMs 5 4

— Qwen Code · serve A/B

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(serve): Stop renewing settled Hosted Shell publication grants

2 participants