Skip to content

feat(serve): Expose active work state - #8588

Merged
doudouOUC merged 9 commits into
QwenLM:mainfrom
doudouOUC:agent/serve-active-work-state
Aug 8, 2026
Merged

doudouOUC merged 9 commits into
QwenLM:mainfrom
doudouOUC:agent/serve-active-work-state

Conversation

@doudouOUC

@doudouOUC doudouOUC commented Aug 5, 2026 •

Copy link
Copy Markdown
Collaborator

What this PR does

Adds three additive fields to GET /health?deep=1 — activeWork, activeWorkReporting, and activeWorkStaleMs — and the reporting machinery behind them.

activeWork is true while any managed workspace has an accepted-but-unsettled prompt, a running background Agent, or an Agent terminal notification that is queued, awaiting acceptance, or being processed by its parent continuation. It does not cover background shells, Monitors, workflows, or cron; that exclusion is deliberate and documented, because a controller that reads activeWork: false as "nothing at all is running" will be wrong about those.

activeWorkReporting (full / partial / none) says how much of that boolean is actually vouched for, and activeWorkStaleMs is the age of the oldest snapshot it rests on (0 when nothing is covered). Without the grade, activeWork: false cannot be told apart from "no child told me anything", which is the one case where acting on it is unsafe.

Reporting. The daemon and ACP child negotiate a private versioned capability through initialization _meta; the child answers with the cadence it will use and the categories it covers, and each side clamps the other's value into an agreed range. A supported child then publishes channel-wide full snapshots of named holds:

{ "v": 1, "seq": 12, "sessions": [ { "sessionId": "…", "holds": [ { "category": "agent", "id": "a1b2" } ] } ] }

Holds are derived on every report from the owners of the work — the background-task registry's unfinalized set, the notification queue, the in-flight acceptance and continuation state. There is no acquire/release ledger, because a ledger can miss a release and a leaked hold would pin its Session forever while every snapshot faithfully republished the leak. Full snapshots make a dropped report self-correcting in both directions, and because a report is complete, a Session absent from a fresh snapshot holds nothing on the child side. Absence and reported-with-no-holds are therefore the same fact and take the same path — one that ends in asking the child, never in assuming.

Prompts are deliberately absent from the child's report: the daemon accepts, queues, dispatches, and settles them, so its own count is authoritative and strictly wider (it covers prompts still waiting in the FIFO, which the child cannot see). A snapshot is flushed ahead of the prompt response on the same stream, so a hold the prompt left behind is on the wire before the daemon drops that count.

Cleanup. Automatic cleanup no longer destroys a Session on the strength of a cached snapshot. It asks the child to close only if unheld, and the child answers under its own close gate — with the gate held no prompt is admitted and no automatic turn starts, so a hold cannot appear between the check and the teardown. A refusal hands back the current holds and the daemon adopts them. An unanswered request is neither retried nor assumed: the Session stays, and the next snapshot settles it. The child's gate makes the check atomic on the child side only; the daemon marks the Session in-flight across the whole confirm-then-teardown span, and attach, prompt, and rewind refuse it there exactly as they refuse one already closing. Detach, attach rollback, prompt settle, notification settle, a child reporting itself idle, and the idle reaper's TTL all funnel through one decision point — the reaper included, so a TTL that says the client stopped caring no longer destroys work the child is still running. Explicit close, kill, shutdown, and channel exit keep their force semantics.

Per Session the daemon tracks three states: unsupported (channel never negotiated — contributes nothing, pre-existing cleanup behavior unchanged), unknown (negotiated, not heard from recently enough — reads as retained, and prompts the daemon to ask), and known. Never-reported and gone-quiet are the same state deliberately: a snapshot older than three report intervals is not a report that the Session is idle, so it stops counting as evidence. Reclaiming a channel that has genuinely stopped answering belongs to transport liveness, not here.

Why it's needed

activePrompts reaches zero when a main prompt finishes, even if background Agents started by that prompt are still running. A restart controller that reads zero active prompts as idle can therefore restart the daemon before those Agents finish and before their terminal notifications reach the parent session. activeWork supplies the missing fact without embedding restart policy in the daemon.

Controllers should use:

const busy =
  health.activePrompts > 0 ||
  health.activeWork ||
  health.activeWorkReporting !== 'full';

Dropping the third term makes activeWork === false indistinguishable from an unreported channel.

What this deliberately does not do

There is no heartbeat watchdog and no channel kill driven by work state. Inferring "this channel is dead" from "one Session stopped reporting" kills every Session on that process, and a host suspend, a long event-loop stall, or a single dropped notification all look identical to a stalled child. Transport/process liveness (channel ping-pong) and stalled-Agent detection (progress-based watchdog) are separate mechanisms tracked as follow-ups under the umbrella issue.

These fields are an observation cache, not a restart lease. Even a fresh, fully-graded, empty answer describes the moment it was sampled; work can start immediately afterwards. The rule above lowers the risk of a wrong restart substantially but does not eliminate it — strict safety needs a prepare-restart fence that stops new work admission, confirms the drain, and only then shuts down. That is out of scope here and stated as such in the docs.

Reviewer Test Plan

  1. Start a prompt that launches a background Agent and lets the main prompt finish first. Confirm deep health reports activePrompts: 0 with activeWork: true until the Agent terminal notification and its parent continuation settle.
  2. Cancel a running background Agent in a detached session. Confirm the Session survives the cancel() → finalizeCancelled() window (up to the 5s grace timer) and is not reaped with the terminal notification still owed. This is the concrete bug the hasUnfinalizedTasks() predicate fixes.
  3. Detach the last client while an Agent is active. Confirm the Session is preserved, and that when it does go idle the daemon issues a conditional close rather than closing on the cached snapshot.
  4. Make the child refuse a conditional close (start work between the snapshot and the request). Confirm the Session stays and the daemon adopts the returned hold set.
  5. Stall the child's close response. Confirm the daemon neither retries in place nor assumes closure, that a prompt or rewind attempted during the stalled round trip is refused rather than accepted and lost, and that the next snapshot settles it by asking the child once more.
  6. Connect a child that does not acknowledge the capability. Confirm cleanup behaves exactly as before, activeWorkReporting is none, and the shallow health response remains exactly { "status": "ok" }.
  7. Confirm a true value from one managed or draining workspace makes daemon-wide activeWork true, and that an exception from a later workspace getter still returns 503 aggregation_failed.

Evidence (Before & After)

N/A — daemon protocol, lifecycle, and health-state change with no TUI presentation change. The Serve A/B job's diff table is the check on the public response shape: it should now show three changed fields rather than one.

Tested on

OS Status
🍏 macOS ⚠️ not tested
🪟 Windows ⚠️ not tested
🐧 Linux ✅ unit suites only

Testing status — please read

Unit suites run locally and green: ACP bridge 489/489, ACP agent 383/383, Session 534/534, core background-tasks 123/123, and the three serve suites 1187/1188. That one failure is a pre-existing cross-file flake in the Live Appshot integration tests — it reproduces on the unmodified tree and fails a different test each run.

End-to-end has not been run. None of the seven items above were executed manually; they are for the reviewer (and for CI's Real daemon E2E). Typecheck is clean for core and acp-bridge; cli has 22 residual errors, all from unbuilt workspace packages in the local sandbox and none in any file this PR touches.

Risk & Scope

  • Main risk: this is a cross-package private protocol addition. The failure mode is now bounded in the safe direction — an unreported or unconfirmed Session is retained, never destroyed — so the realistic cost of a bug is a Session lingering until the idle reaper, not lost work or a killed channel. The previous revision of this PR could recycle a healthy ACP channel after missed heartbeats; that mechanism is gone.
  • Not validated / out of scope: macOS and Windows untested; no end-to-end run. Channel liveness, stalled-Agent detection, runtime-generation draining, and unresponsive-Agent escalation are deferred to follow-up PRs under the umbrella issue.
  • Breaking changes / migration notes: the deep-health JSON response gains three additive fields. AcpSessionBridge gains three required readonly members (activeWork, activeWorkReporting, activeWorkOldestReportAt), which is breaking for any external implementer of that interface; all in-repo implementations are updated. The shallow health response and persisted formats are unchanged. Existing restart controllers may ignore the new fields; activePrompts keeps its exact previous meaning as an independent compatibility signal.

Linked Issues

Refs #8586

中文说明

本 PR 做了什么

为 GET /health?deep=1 增加三个向后兼容字段 —— activeWork、activeWorkReporting、activeWorkStaleMs —— 以及支撑它们的上报机制。

只要任一受管 workspace 存在已接受但未 settle 的 Prompt、运行中的后台 Agent,或正在排队/等待接收/由父 continuation 处理的 Agent 终态通知,activeWork 即为 true。它不包含后台 shell、Monitor、workflow 和 cron;这是有意为之并写进了文档,因为把 activeWork: false 理解成"什么都没在跑"对这几类就是错的。

activeWorkReporting(full / partial / none)说明这个布尔量有多少是真正被担保的,activeWorkStaleMs 是它所依赖的最旧快照的年龄(无覆盖时为 0)。没有这个分级,activeWork: false 就无法与"没有任何子进程告诉过我"区分开,而后者恰恰是唯一不能据此行动的情形。

上报机制。 daemon 与 ACP 子进程通过初始化 _meta 协商一个私有、带版本的能力;子进程回复它实际采用的上报周期和覆盖的类别,两侧都会把对方给的值钳制到约定区间内。支持该能力的子进程随后发布 channel 级全量快照(见英文段的 JSON 示例)。

Hold 每次上报时现算,来源是工作的真正持有者:后台任务注册表的 unfinalized 集合、通知队列、在途的 acceptance 与 continuation 状态。没有 acquire/release 账本 —— 账本可能漏掉一次 release,而泄漏的 hold 会永久钉住 Session,并被每一份快照忠实地重复上报。全量快照让丢失的报文在两个方向上都能自愈,且某个 Session 未出现在新快照中,就是子进程已释放它的正面证据。

Prompt 有意不由子进程上报:daemon 自己负责接受、排队、下发和 settle,它的计数既权威又严格更宽(覆盖了子进程看不到的 FIFO 等待)。快照会在 prompt 响应之前 flush 到同一条流上,确保该 prompt 留下的 hold 先于 daemon 清零计数抵达。

清理路径。 自动清理不再凭缓存快照销毁 Session,而是请求子进程"仅在无 hold 时关闭",由子进程在自己的 close gate 下作答 —— gate 持有期间不接受新 Prompt、不启动新的自动 turn,因此 hold 不可能在检查与拆除之间出现。被拒绝时返回当前 hold 集合,daemon 予以采纳。请求无应答时既不重试也不假设:Session 保留,由下一份快照裁决。detach、attach 回滚、prompt settle、通知 settle、子进程自报空闲,现在全部汇入同一个决策点,取代原先四处近似重复的逻辑。显式 close、kill、shutdown 和 channel 退出保持强制语义。

daemon 按 Session 维护三态:unsupported(通道从未协商 —— 不贡献任何值,既有清理行为不变)、unknown(已协商但尚未收到上报 —— 视为保留,并促使 daemon 主动询问)、known。

为什么需要它

主 Prompt 结束后 activePrompts 即归零,即使它拉起的后台 Agent 仍在运行。把"零活跃 Prompt"直接当作空闲的重启控制器,就可能在这些 Agent 完成、其终态通知抵达父 session 之前重启 daemon。activeWork 补上这个缺失的事实,同时不把重启策略嵌进 daemon。控制器判据见英文段的代码块;去掉第三项会让 activeWork === false 与"未上报的通道"无法区分。

明确不做的事

没有心跳看门狗,也没有由工作状态驱动的 channel kill。 从"某个 Session 停止上报"推断"整条通道已死"会连带杀死该进程上的所有 Session,而主机休眠、长时间 event loop 阻塞、单次报文丢失,在观测上与子进程卡死完全一样。传输/进程存活(通道 ping-pong)与 Agent 停滞检测(基于进度的 watchdog)是独立机制,作为后续 PR 由 umbrella issue 跟踪。

这些字段是观测缓存,不是重启租约。即使是新鲜、分级完整、且为空的回答,描述的也只是采样那一刻;工作可能紧随其后开始。上面的判据能显著降低误重启风险,但不能消除 —— 严格安全需要一个 prepare-restart 栅栏:先停止新工作准入,确认 drain,然后才停机。这不在本 PR 范围内,文档中已如实写明。

Reviewer 测试计划

见英文段的 7 条。其中第 2 条(在 detached session 中 cancel 一个后台 Agent,确认它能挺过 cancel() → finalizeCancelled() 窗口)针对的是本次修复的一个具体缺陷。

测试状态 —— 请务必阅读

本地单测全绿:ACP bridge 489/489、ACP agent 383/383、Session 534/534、core background-tasks 123/123、serve 三套 1187/1188。那 1 条失败是 Live Appshot 集成测试里既有的跨文件 flake —— 在未修改的代码树上同样复现,且每次失败的是不同的 test。

端到端没有跑过。 上述 7 条没有任何一条被手动执行,它们留给评审者(以及 CI 的 Real daemon E2E)。typecheck 方面 core 与 acp-bridge 干净;cli 残留 22 条,全部源自本地沙箱中未构建的 workspace 包,无一落在本 PR 改动的文件上。

风险与范围

  • 主要风险: 这是一个跨 package 的私有协议增量。故障方向现在被限制在安全侧 —— 未上报或未确认的 Session 一律保留,绝不销毁 —— 所以出 bug 的现实代价是 Session 滞留到 idle reaper 回收,而不是丢失工作或杀掉通道。本 PR 的上一版会在心跳缺失后回收健康的 ACP 通道,该机制已被移除。
  • 未验证 / 范围外: 未测试 macOS 与 Windows;未跑端到端。通道存活、Agent 停滞检测、runtime 代际 draining、不响应 Agent 的升级处置,均留给 umbrella issue 跟踪的后续 PR。
  • 破坏性变更 / 迁移说明: 深度健康 JSON 响应新增三个字段。AcpSessionBridge 新增三个必需只读成员(activeWork、activeWorkReporting、activeWorkOldestReportAt),对该接口的外部实现者构成破坏性变更;仓库内所有实现均已更新。浅层健康响应与持久化格式不变。现有重启控制器可以忽略新字段;activePrompts 保持原有语义,作为独立兼容信号。

@doudouOUC

Copy link
Copy Markdown
Collaborator Author

E2E test report

Automated verification completed on macOS:

  • npm run build && npm run typecheck — passed.
  • ACP bridge unit suite — 485/485 passed.
  • Session unit suite — 533/533 passed.
  • ACP agent unit suite — 383/383 passed.
  • Focused deep-health aggregation tests — 13/13 passed.
  • Focused daemon startup deep-health test — 1/1 passed.
  • Pre-commit Prettier and ESLint checks — passed.

The full serve server test file completed with 865/867 passing. The two failures are pre-existing workspace tool-auth baseline failures unrelated to this diff: passes client identity into the bridge expected 200 and received 401, and 400 invalid_client_id... expected 400 and received 404.

The manual daemon restart scenario has not been executed in this local environment. The prepared E2E plan covers the released-build baseline, activePrompts: 0 with activeWork: true while a background Agent remains active, terminal-notification continuation, FIFO hand-off, managed/draining workspace aggregation, per-Session heartbeat loss, legacy-child compatibility, and the unchanged shallow health response.

doudouOUC and others added 3 commits August 6, 2026 08:03
Reworks the active-work signal after review. Three changes of substance.

Drops the 45s heartbeat watchdog entirely. It inferred "this channel is
dead" from "one Session stopped reporting" and killed the whole channel,
taking every Session on that process with it — including on a suspend,
a long event-loop stall, or a single dropped notification. Channel
liveness is a transport concern and gets its own mechanism.

Replaces the per-Session boolean with a channel-wide snapshot of named
holds, derived on every report from the owners of the work (the
registry's unfinalized set, the notification queue) rather than from a
ledger kept alongside them. Full snapshots make a dropped report
self-correcting in both directions, and a Session's absence from one is
positive evidence the child released it. Agent holds now use
hasUnfinalizedTasks()'s predicate, closing the cancel to
finalizeCancelled() window where a cancelled agent looked idle and its
terminal notification could be stranded.

Leaves prompts out of the child's report: the daemon accepts, queues,
dispatches, and settles them, so its own count is authoritative and
covers the FIFO wait the child cannot see. A snapshot is flushed ahead
of the prompt response so a hold the prompt left behind is on the wire
before the daemon drops that count.

Co-Authored-By: Claude Opus 5 <[email protected]>
Completes the active-work rework with the two facts a restart controller
was still missing and the one guarantee automatic cleanup was missing.

Automatic cleanup no longer destroys a Session on the strength of a
cached snapshot. It asks the child to close only if unheld, and the
child answers under its own close gate — with the gate held no prompt is
admitted and no automatic turn starts, so a hold cannot appear between
the check and the teardown. A refusal hands back the current holds and
the daemon adopts them. An unanswered request is neither retried nor
assumed: the Session stays, and the next snapshot settles it, because a
Session absent from one has provably been released. Every automatic path
— detach, attach rollback, prompt settle, notification settle, a child
reporting itself idle — now funnels through one decision point instead
of four near-copies.

Health gains activeWorkReporting and activeWorkStaleMs. Without them
activeWork:false cannot be told apart from "no child told me anything",
which is the one case where acting on it is unsafe. Freshness is graded
by the daemon rather than the controller, since the cadence is negotiated
per channel; a stale snapshot or a child omitting a category degrades the
grade instead of silently narrowing what the boolean covers.

Tests: acp-bridge 489/489, acpAgent 383/383, Session 534/534,
serve suites 1188 with one pre-existing cross-file flake in the Live
Appshot integration tests (reproduces on the unmodified tree, failing a
different test each run).

Co-Authored-By: Claude Opus 5 <[email protected]>
@doudouOUC
doudouOUC force-pushed the agent/serve-active-work-state branch from 5d145ae to 612bcb7 Compare August 6, 2026 04:42
@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Please do not rebase or force-push to an active PR as it invalidates existing review comments. Note for future reference, the bots always squash all changes into a single commit automatically as part of the integration.

中文

请勿对活跃的 PR 执行 rebase 或 force-push,因为这会使已有的评审评论失效。另外,供日后参考:作为集成流程的一部分,机器人始终会自动将所有改动压缩(squash)为单个提交。

@doudouOUC

Copy link
Copy Markdown
Collaborator Author

Force-pushed: the design changed, not just the code

Rebased onto current main (this PR was conflicting) and reworked in response to review. If you looked at the previous revision, the mechanism is different now — please re-read rather than diffing against memory. The PR body has been rewritten to match.

Removed: the per-Session boolean heartbeat and the 45s channel recycle. Inferring channel death from one Session's silence kills every Session on that process, and a host suspend, a long event-loop stall, or a single dropped notification are all indistinguishable from a wedged child. Transport liveness is now its own layer (new PR 2 in #8586).

Replaced with: channel-wide full snapshots of named holds, derived on every report from the owners of the work rather than kept in a parallel ledger; a three-state daemon cache that distinguishes "never negotiated" from "not yet heard from"; and a conditional close where the child confirms under its own close gate before anything is destroyed.

Added: activeWorkReporting and activeWorkStaleMs, so activeWork: false can be told apart from "no child reported". The busy rule for controllers is three terms now, not two.

Concrete bug fixed along the way: agent holds key on hasUnfinalizedTasks(), not hasRunningTasks(). A cancelled Agent still owes its terminal notification, and the previous predicate made a detached Session look idle for that whole window — reaping it and stranding the notification.

Full reasoning: #8586 (comment)

Please note the testing status section in the body: unit suites are green, but end-to-end has not been run and none of the seven reviewer test items were executed manually.

@doudouOUC doudouOUC left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the current force-pushed implementation at 612bcb7. The full-snapshot hold model and fail-closed reporting direction look sound, but the inline findings below still leave one unsafe automatic-close path, a health aggregation edge case, and a red test suite. Current failing CI: https://github.com/QwenLM/qwen-code/actions/runs/31071986719/job/92521662522

Comment thread packages/acp-bridge/src/bridge.ts Outdated
Comment thread packages/cli/src/acp-integration/session/Session.ts Outdated
Comment thread packages/cli/src/acp-integration/active-work-reporter.ts Outdated
Comment thread packages/cli/src/serve/routes/health-demo.ts Outdated
doudouOUC and others added 2 commits August 6, 2026 15:27
…ion mocks

CI caught two things the local runs missed.

The reporter's snapshot construction was unguarded. Only the send was
wrapped, so a throw while collecting a Session's holds escaped through
setInterval and queueMicrotask as an uncaught exception — capable of
taking down the ACP child — and through flush() into the prompt path,
turning a reporting problem into a failed prompt. Collection is now
wrapped and a failed snapshot is abandoned whole rather than sent
partially: a Session missing from a report reads as released, and one
reported with no holds reads as safe to close, so publishing a partial
snapshot would actively invite the daemon to destroy live work. Sending
nothing lets the daemon's copy age instead, which its freshness grading
already treats as untrustworthy and retains. flush() no longer rejects.

Session.review-lease and Session.worktree mock the background-task
registry without setStatusChangeCallback, so constructing a Session threw.
That break arrived with the original commit, which verified only
Session.test.ts; the sibling Session.*.test.ts files were never run. Both
mocks now carry the methods the constructor and the hold collector need.

Co-Authored-By: Claude Opus 5 <[email protected]>
…lures

The previous commit added the guard but could not have demonstrated it:
the same commit also gave the acpAgent Session mock a
collectActiveWorkHolds, removing the very condition that triggered the
throw. The unhandled error disappearing was therefore explained by the
mock alone, and active-work-reporter.ts had no tests at all.

These cover the escape routes that matter — the interval timer, the
coalescing microtask, and flush() on the prompt path — plus the choice to
abandon a whole snapshot rather than send a partial one, since a session
omitted from a report reads as released and one reported with no holds
reads as safe to close.

Verified by removing the guard: five of the nine fail with the collection
error escaping, and pass again once it is restored.

Co-Authored-By: Claude Opus 5 <[email protected]>
@doudouOUC

doudouOUC commented Aug 7, 2026 •

Copy link
Copy Markdown
Collaborator Author

Self-audit: three things to fix before this leaves draft

I re-audited this adversarially. The read side — the part the title is about — holds up. The write side does not, and the three problems below share one root cause, so they should be fixed as one change rather than three patches.

Since this is a draft I am not calling these merge blockers, but they are design defects rather than unfinished work: the functions involved are all written, and it is the guard model itself that does not hold.

What is actually fine

The exposed surface is a whitelist projection of three scalars on GET /health?deep=1 — no object serialization, no absolute paths, no $HOME, no prompt text, no tool arguments, no tokens. Hold payloads ({category, id}) stay in the daemon's in-memory SessionEntry.childHolds and never reach an HTTP response. Auth reuses the existing /health tiering, and the CORS deny-wall plus host allowlist are registered ahead of the pre-auth /health route, so page JS cannot read it. Scope matches the existing sessions / activePrompts fields — process-global aggregation, not a new scope. collectActiveWorkHolds() is derived per call rather than a ledger, so it does not grow with session lifetime. Cleanup is sound: closeSessionImpl deletes the entry and all three getters fall to zero.

The root cause

This PR promotes a cached child-reported snapshot from a hint into the authority that permits destroying a session. Three destruction paths now consult it, their guards are inconsistent with each other, and every one of them is weaker than what main had.

1. Snapshot absence tears down sessions a user is actively watching. The absence loop checks only activeWorkCloseInFlight and entryHasLocalWork. It does not check entry.events.subscriberCount or entry.clientIds.size — and maybeCloseIdleSession treats both as hard guards. So a session with a live SSE subscriber and a registered client is destroyed because one snapshot omitted it.

This also contradicts the PR description directly. I wrote that an unreported or unconfirmed Session is retained, never destroyed. bridge.test.ts:406-428 encodes the opposite as expected behaviour: the session is never detached, its spawn-owner clientId is still registered, and the test asserts sessionCount goes to 0 after an empty snapshot. The description claims a guarantee the test proves absent. That wording has to change or the behaviour does — currently they disagree, and the test would keep the wrong one honest.

2. A ten-second TOCTOU window that main does not have. confirmChildUnheld is a withTimeout(..., ACTIVE_WORK_CLOSE_TIMEOUT_MS) of 10 seconds, and it never sets entry.closing. So between the last guard and closeSessionImpl setting closing = true there is a window up to ten seconds wide, and attach, sendPrompt and rewindSession all gate on closing alone. On main the detach path was a synchronous guard sequence followed directly by closeSessionImpl, whose first await comes after closing is set — the transition was atomic. I split it. A client that attaches inside the window gets its session destroyed out from under it, with the prompt lost.

3. Arbitrarily stale cache authorizes reaping. entryHasActiveWork never looks at childHoldsAt. Staleness is implemented — activeWorkReporting compares against intervalMs * ACTIVE_WORK_STALE_INTERVALS — but only to report health, never to gate destruction. So a child that goes silent after emitting one empty snapshot leaves a cache that permits reaping indefinitely, even if it has since restarted a background agent.

Fix direction

One shared helper for "may this session be destroyed", with a single in-flight flag and a single freshness gate, used by all three paths. Concretely: add the in-flight gate before confirmChildUnheld and have attach/sendPrompt/rewindSession check it; give the absence loop the same subscriber/client guards maybeCloseIdleSession uses, and recover a lost close response by re-sending sessionClose{onlyIfUnheld} for that one session instead of treating absence as consent; make entryHasActiveWork fall back to the conservative childHolds === null semantics once the snapshot is stale.

Also worth noting: confirmChildUnheld's comment argues that a hold cannot appear between check and teardown while the gate is held. That is true on the child side and I verified it, but it says nothing about the daemon-side window in item 2, and a reader will take it as covering the whole path. It should say what it does not cover.

Smaller items, not part of the above

  • A runtime with zero sessions reports 'full', which sets reportingAny and makes a mixed deployment aggregate to 'partial' when real coverage is zero. No test can currently produce an aggregate 'none'.
  • activeWorkOldestReportAt does not use the covered predicate that activeWorkReporting uses, so with sessions present but all stale it returns a positive activeWorkStaleMs while activeWorkReporting says 'none'. Three places document the opposite ("0 when nothing is covered").
  • setStatusChangeCallback is a single slot, and Session now takes it unconditionally and clears it to undefined on dispose even if someone else owned it. Harmless under ACP today because the TUI is absent, but it should be an add/remove listener pair.
  • parseActiveWorkSnapshot validates every field type and discards the whole packet on malformation (correct), but caps neither sessions[] nor holds[] length. Zero-cost to add.
  • activeWork only walks byId, so it reports false while hasNoChannelWork is simultaneously refusing to kill the channel for in-flight spawns, pending restores, or MCP discovery. A field whose stated purpose is "can this daemon be reclaimed" should not disagree with the daemon's own answer.

@doudouOUC doudouOUC self-assigned this Aug 7, 2026
Self-review of the previous revision found that this PR had promoted a
cached child report from a hint into the authority that permits destroying
a Session. Four teardown paths consulted it, their guards disagreed with
each other, and each was weaker than what main had. The four are one
defect with four exits, so they are fixed as one change.

Absence from a snapshot no longer authorizes teardown. Because reports
are complete, a Session the child omits holds nothing on the child side —
so absence and reported-with-no-holds are the same fact and now take the
same path. The separate absence loop is gone; it lacked the subscriber
and client guards `maybeCloseIdleSession` applies, so one snapshot could
destroy a Session with a live SSE subscriber and a registered client.
That contradicted this PR's own claim that an unreported Session is
retained, and the old test asserted the destruction. Both are corrected.

A conditional close is now marked in flight across the whole confirm-then-
teardown span, and attach, prompt, and rewind refuse a Session in that
state exactly as they refuse one already closing. `closeSessionImpl` sets
`closing` synchronously, but the round trip in front of it is an await of
up to ten seconds; on main the guard sequence ran straight into teardown,
so splitting it is what opened the window.

A snapshot older than the freshness window stops counting as evidence.
Staleness was already computed, but only to grade health, never to gate
destruction — so a child that went quiet after one empty report left a
cache that permitted reaping indefinitely. Never-reported and gone-quiet
now land in the same retained bucket. Reclaiming a channel that has truly
stopped answering belongs to transport liveness, not here.

The idle reaper asks the child too. Its TTL says the client stopped
caring, which is not the same as the child having nothing left to run.

Health coverage is exposed as counts and graded once daemon-wide, because
grades do not compose: a runtime with zero Sessions is vacuously `full`,
and folding that in let an empty workspace vouch for another workspace's
unreported Sessions. `activeWorkStaleMs` now measures only covered
Sessions, so it can no longer report positive staleness beside a grade
saying nothing is covered.

Also: bound snapshot `sessions[]` and `holds[]` so a buggy child cannot
make the daemon walk an unbounded structure per report, and retract the
background-task status callback by identity rather than blanking a
single-slot setter the TUI also uses.

Tests: the absence test now asserts retention under a registered client
and under a live subscriber; new regressions cover the recovered lost
close response, the stale-snapshot gate, admission refusal during a
conditional close, the reaper's confirmation, the oversized-snapshot
discard, and the mixed empty/uncovered health aggregate.
@doudouOUC
doudouOUC marked this pull request as ready for review August 7, 2026 15:38
@doudouOUC
doudouOUC enabled auto-merge August 7, 2026 15:38
Triage review found that the design doc, the PR description, and the
comment on `entryHasActiveWork` all promised the daemon *asks* the child
about a Session it has not heard about, while no code path ever did:
`entryHasActiveWork` returns true when the child's side is unknown, and
the cleanup path returned early on exactly that. The finding predates the
guard rework and survived it unchanged.

Skipping on unknown looks like the safe direction and is in fact the worse
failure. Nothing resolves it — a Session on a channel that went quiet is
retained forever, and the idle reaper skips it too, so there is no path
out at all. Asking resolves it definitively: the child answers under its
own close gate whether or not its snapshots are arriving, the round trip
is bounded, and every non-answer still retains.

So the predicate is split by what it actually knows. `childReportsHeldWork`
is positive knowledge only; `childWorkIsUnknown` is the absence of a
gradeable report. The health surface ORs both, because a controller must
never read "nobody told me" as "nothing is running". Automatic cleanup
blocks only on known work and lets unknown through to `confirmChildUnheld`.

Also moves `parseActiveWorkSnapshot` out from between two import blocks
(pure relocation, no logic change) and aligns the doc wording, including
the shared-guard table, with what the code now does.
@doudouOUC

Copy link
Copy Markdown
Collaborator Author

Thanks — both findings taken, and one correction to the record: this review is against fb68eeb53, but that commit is not the current state of the work. An adversarial self-audit after it turned up three defects in the write side that the review's "safety posture is consistent throughout" reading does not cover, so the fixes below are on top of a guard rework, not a patch to the tree you read.

Finding 1 — the "unknown" state promises an ask the code never made. Correct, and it survived the rework unchanged. Fixed by splitting the predicate on what it actually knows rather than by softening the wording: childReportsHeldWork is positive knowledge only, childWorkIsUnknown is the absence of a gradeable report. The health surface ORs both (a controller must never read "nobody told me" as "nothing is running"); automatic cleanup blocks only on known work and lets unknown through to confirmChildUnheld.

Worth stating why the wording was not the thing to change. Skipping on unknown looks like the safe direction and is in fact the worse failure — nothing resolves it, the reaper skips it too, and a Session on a channel that went quiet is retained forever with no path out. Asking resolves it definitively: the child answers under its own close gate whether or not its snapshots are arriving, the round trip is bounded, and every non-answer still retains. Two regression tests cover it: the ask happens for a never-reported Session, and a refusal resolves the unknown toward retention with the reason attached.

Finding 2 — parseActiveWorkSnapshot between two import blocks. Fixed, pure relocation.

What the review could not have seen. The self-audit found that this PR had promoted a cached child report from a hint into the authority that permits destroying a Session, across four teardown paths whose guards disagreed with each other:

  1. Absence from a snapshot tore down a Session with a live SSE subscriber and a registered client — guards maybeCloseIdleSession treats as hard. That contradicted this PR's own "unreported or unconfirmed is retained, never destroyed" claim, and bridge.test.ts asserted the destruction as expected behaviour.
  2. confirmChildUnheld is a 10s round trip that never set closing, so attach/prompt/rewind could be admitted into a Session already authorized for teardown. On main that sequence was synchronous; splitting it opened the window.
  3. Staleness was computed but only ever used to grade health, never to gate destruction — so a child that went quiet after one empty report left a cache that permitted reaping indefinitely.
  4. The idle reaper did not call confirmChildUnheld at all.

All four are the same defect with four exits, so they are fixed as one change: one shared candidacy predicate, one in-flight flag held across confirm-then-teardown, one freshness gate, and the reaper routed through the same ask. The absence loop is gone — because reports are complete, absence and reported-with-no-holds are the same fact and now take the same path.

On the two deferred items: agreed that /verify is the right instrument, since the central claims are live-lifecycle behaviours that the unit suite pins only through mocked channels — the earlier "E2E test report" comment on this PR was automated verification, not a live lifecycle exercise, and I have not run one. macOS/Windows remain untested. Neither the docs/comment mismatch nor the four items above would have been caught by CI, which is the honest argument for the sandboxed lane before merge.

Re-review will need to happen at the new head rather than fb68eeb53.

@doudouOUC

Copy link
Copy Markdown
Collaborator Author

Self-audit + triage findings addressed — head is now 41853db8

Two commits on top of fb68eeb53 (appended, not force-pushed, so existing review comments stay anchored). Everything reviewed at fb68eeb53 predates them.

The root cause worth stating plainly

This PR had promoted a cached child report from a hint into the authority that permits destroying a Session. Four teardown paths consulted it, their guards disagreed with each other, and every one was weaker than what main had. They are one defect with four exits, so they are fixed as one change rather than four patches.

# Defect Fix
1 Absence from a snapshot tore down a Session with a live SSE subscriber and a registered client — guards maybeCloseIdleSession treats as hard Separate absence loop deleted. Because reports are complete, absence and reported-with-no-holds are the same fact and take the same path, through every shared guard
2 confirmChildUnheld is a 10s round trip that never set closing, so attach / prompt / rewind could be admitted into a Session already authorized for teardown One in-flight flag held across the whole confirm-then-teardown span; all three admission paths refuse it exactly as they refuse closing
3 Staleness was computed but only ever graded health, never gated destruction — a child that went quiet after one empty report left a cache permitting reaping indefinitely Freshness gate moved into the retention predicate; never-reported and gone-quiet are now the same state
4 The idle reaper never called confirmChildUnheld at all Routed through the same ask, keeping its own TTL and crash-path policy

Defect 1 also contradicted this PR's own description — it claimed an unreported or unconfirmed Session is retained, never destroyed, while bridge.test.ts asserted the destruction as expected behaviour. Both are corrected; the description is updated, and the test now asserts retention under a registered client and under a live subscriber.

Triage findings

  • "unknown promises an ask the code never made" — correct, and it survived the rework. Fixed by splitting the predicate on what it actually knows rather than by softening the wording: childReportsHeldWork (positive knowledge) vs childWorkIsUnknown (absence of a gradeable report). Health ORs both; cleanup blocks only on known work and lets unknown through to the ask. Skipping on unknown looks safe and is the worse failure — nothing resolves it, so the Session is retained forever with no path out.
  • parseActiveWorkSnapshot between two import blocks — fixed, pure relocation.

Also in scope

Snapshot sessions[] / holds[] are now bounded; health coverage is exposed as counts and graded once daemon-wide (an empty runtime is vacuously full and could vouch for another workspace's unreported Sessions); activeWorkStaleMs counts only covered Sessions; the background-task status callback is retracted by identity instead of blanking a single-slot setter the TUI shares; and activeWork's Session-scoped boundary vs channel-level work is documented rather than widened.

Verification

Check Result
acp-bridge suite 1085/1085
active-work subset 17/17, incl. 8 new regressions
Session.test.ts 534/534
core background-tasks 123/123
serve suites 1188/1189 — the one failure is a pre-existing cross-file flake (passes in isolation, fails a different test each run)
deep-health subset 14/14
tsc core / acp-bridge clean
tsc cli clean in every touched file
eslint + prettier clean

New regressions: retention under a registered client and under a live subscriber; recovery of a lost close response; the stale-snapshot gate; admission refusal during a conditional close; the reaper's confirmation; the oversized-snapshot discard; the mixed empty/uncovered health aggregate; and the ask-on-unknown path in both directions.

Still not done

No live end-to-end run. Agreed that @qwen-code /verify is the right instrument — the central claims are lifecycle behaviours the unit suite pins only through mocked channels, and note that none of the four defects above would have been caught by CI, which is the honest argument for the sandboxed lane before merge. macOS and Windows remain untested.

Syncs 46 commits of base drift. CI's 'Check voice guard mirror sync' step
runs from main and invokes `npm run check:voice-guard-sync`, a script
added alongside that step in 732f4d8 (QwenLM#8350) and absent from this
branch — so the check failed on missing-script, not on anything in this
diff. Merged rather than rebased so existing review comments stay
anchored.
@doudouOUC

Copy link
Copy Markdown
Collaborator Author

CI: Check voice guard mirror sync was base drift, not this diff — synced in 80019aca

Test (ubuntu-latest, Node 22.x) failed on 41853db8 at step 19:

npm error Missing script: "check:voice-guard-sync"

Not a flake and not in this diff. The step and the script it calls were added together in 732f4d8 (#8350); the workflow runs from main, this branch was 46 commits behind, so the check invoked a script absent from the checkout. A rerun would have failed identically.

Merged main in rather than rebasing, so the existing review comments stay anchored — the repo's own force-push reminder is the reason. Merge was conflict-free. npm run check:voice-guard-sync now passes locally.

Re-verified on the merged tree, because main had touched many of the same files (acpAgent.ts, bridgeTypes.ts, Session.ts, health-demo.ts, background-tasks.ts, plus run-qwen-serve.ts and server.ts):

Check Result
acp-bridge suite 1110/1110 (25 files)
Session + acpAgent + reporter 929/929
serve suites 1216/1218
check:voice-guard-sync passes

The two serve failures are Live conversation runtime lifecycle — a pre-existing cross-file flake in this file, unrelated to active work. All 6 tests in that describe block pass in isolation, and the same file fails a different test on different runs.

Two interactions worth naming explicitly, since both are the class of thing a clean textual merge hides:

  • main changed AcpSessionBridge in bridgeTypes.ts too — getChildResourceSnapshot gained an optional ageMs. Independent of activeWorkCoverage; no interaction.
  • main added 184 lines to acpAgent.test.ts, the same file whose Session mock previously caused an uncaught exception in the reporter. Checked directly: none of the new tests construct a Session, and the SessionMock.prototype.collectActiveWorkHolds patch survived the merge.

Also picked up cb3dc107f (#8604), which deflakes the GlobTool external-path test that had to be reran on an earlier revision of this PR.

The three route checks reported cancelled on separate workflow runs — CI runner routing pre-empted by the newer push, no action needed; the merge push re-triggers them.

@wenshao

wenshao commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

@qwen-code /triage

@wenshao

wenshao commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Runtime verification (maintainer review)

I built a real end-to-end environment for this PR and executed all seven reviewer test-plan items, plus a genuine mixed-version run. All seven pass. Two non-blocking notes below.

Setup. Both arms compiled from source into runnable dist artifacts and driven as real processes — no mocks of the bridge, the child, or the health route:

AFTER PR head 80019acab7
BEFORE merge-base 20b9504276
Daemon node packages/cli/dist/index.js serve --workspace … --token …
Child real qwen --acp process
Model mock OpenAI: returns an agent tool-call with run_in_background: true, then parks the subagent's own completion so the background Agent stays running on demand
Wire capture QWEN_CLI_ENTRY stdio tee — every JSON-RPC frame in both directions, plus fault injection (drop snapshots / swallow the close reply / point at the old child)

verification report

Reviewer test plan

# Item Result
1 Background Agent outlives the main prompt ✅ activePrompts: 0 with activeWork: true for the Agent's whole life; clears 500 ms after the terminal notification settles
2 Cancel a background Agent in a detached Session ✅ hold hands off agent → notification with no gap on the wire (seq 5 → 6 → 8), then conditional close
3 Detach the last client while an Agent is active ✅ Session preserved; closed only after the child confirms unheld
4 Child refuses the conditional close ✅ {"closed":false,"holds":[{"category":"agent",…}]} on the wire; daemon adopts the set (grade flips partial → full)
5 Stall the child's close response ✅ no retry, no assumption, Session survives — ⚠️ see Note A on the prompt path
6 Child that does not acknowledge the capability ✅ ran a genuinely pre-PR qwen --acp build behind the new daemon: activeWorkReporting: "none", shallow health exactly {"status":"ok"}, cleanup identical to pre-PR
7 Daemon-wide aggregation + 503 aggregation_failed ✅ 2 workspaces / 3 Sessions, one workspace's Agent drives daemon-wide activeWork: true; a throwing getter still yields 503 {"reason":"aggregation_failed"} with the workspace attributed on stderr

The decisive A/B (item 3)

Same script, same mock, two builds:

step AFTER (PR head) BEFORE (merge-base)
prompt settled, Agent running sessions=1 activePrompts=0 activeWork=true reporting=full sessions=1 activePrompts=0
POST /detach → t+2s Session alive Session destroyed
t+10s Session alive destroyed
after the Agent + notification settle closed cleanly, sessions=0 (already gone, Agent killed with it)

raw evidence

Extra checks beyond the plan

  • Capability negotiation captured verbatim: daemon sends {"v":1,"intervalMs":15000}, child answers {"v":1,"intervalMs":15000,"categories":["agent","notification"]}, then publishes channel-wide full snapshots on qwen/notify/channel/active-work.
  • unknown grading is honest. With every snapshot dropped by the tee, an adopted hold set ages past 3 × 15 s and the daemon degrades to activeWorkReporting: "partial", activeWorkStaleMs: 0, activeWork: true — retained, and correctly flagged as not-vouched-for.
  • Mutation teeth on the hasUnfinalizedTasks() predicate. Driving the real compiled BackgroundTaskRegistry: inside the cancel window hasRunningTasks() is false, hasUnfinalizedTasks() is true, and listUnfinalizedBackgroundAgentIds() still returns the id. Patching the compiled dist to the running-only predicate makes it return [] — the choice is load-bearing, exactly as documented.
  • Unit suites at PR head, all green: acp-bridge 498, cli acp-integration 929, cli serve + Session 1230, core background-tasks 123 — 2780 tests, no failures (I did not hit the Live Appshot flake).

Note A — a prompt inside the confirm window is accepted (202) and lost

rewindSession and spawnOrAttach throw synchronously, so REST answers 404 "The session is closing" — verified. sendPrompt instead returns a rejected promise, and the route's try/catch around it only catches synchronous throws, so it has already replied 202 {promptId}. Measured inside a stalled 10 s confirm window:

POST /session/:id/rewind -> 404  "The session is closing"
POST /session/:id/prompt -> 202  {promptId}
    daemon log: [SessionNotFoundError] ... The session is closing; retry after close completes
    events published for that promptId: 0

The caller gets no error and nothing on the event stream. This shape predates the PR (if (entry.closing) return Promise.reject(...)), so it is not a regression — but the PR widens the window it applies to from a fast synchronous teardown to a bounded 10 s round trip, and test-plan item 5 states the prompt is "refused rather than accepted and lost", which is not what a REST caller observes. Worth either softening that sentence or making sendPrompt's guard throw synchronously like its two siblings.

Note B — the conditional close does not carry the daemon's drain budget

confirmChildUnheld sends only {sessionId, onlyIfUnheld} (confirmed on the wire), so the child falls back to SESSION_DRAIN_TIMEOUT_MS = 30_000, while the real close sends drainTimeoutMs = initTimeoutMs * 0.8 = 8000 and the daemon itself only waits ACTIVE_WORK_CLOSE_TIMEOUT_MS = 10_000. A Session whose drain legitimately takes 10–30 s therefore times out the confirm and is retained for a later round. Self-correcting and in the safe direction, but it is an extra cycle the 8 s budget exists to avoid — consider forwarding drainTimeoutMs on the conditional call. (The missing parameter is runtime-confirmed; the consequence is a code read.)

Verdict

The mechanism does what the description says, the failure direction is genuinely the safe one (unreported or unconfirmed ⇒ retained, never destroyed), and the mixed-version path leaves old children behaving exactly as before. The two notes are documentation/polish, not correctness blockers.

LGTM — recommend merge. Note A's wording is worth a one-line fix before or after merge.

中文版

运行时验证(维护者评审)

我为本 PR 搭建了真实的端到端环境,执行了 Reviewer 测试计划的全部 7 条,另加一次真实的跨版本(新 daemon + 旧子进程)验证。7 条全部通过,另有 2 条不阻塞的说明。

环境。 两侧均从源码编译成可运行的 dist,以真实进程驱动 —— bridge、子进程、health 路由都没有被 mock:

AFTER PR head 80019acab7
BEFORE merge-base 20b9504276
Daemon node packages/cli/dist/index.js serve --workspace … --token …
子进程 真实 qwen --acp 进程
模型 mock OpenAI:先返回 run_in_background: true 的 agent tool-call,再挂起子 agent 自己的补全请求,从而按需让后台 Agent 保持 running
抓包 QWEN_CLI_ENTRY stdio tee —— 双向抓取每一条 JSON-RPC 帧,并支持故障注入(丢弃快照 / 吞掉 close 应答 / 指向旧版子进程)

Reviewer 测试计划

# 条目 结果
1 后台 Agent 存活时间超过主 Prompt ✅ 整个 Agent 生命周期内 activePrompts: 0 而 activeWork: true;终态通知 settle 后 500 ms 归零
2 在 detached session 中 cancel 后台 Agent ✅ hold 在线上从 agent → notification 无空档交接(seq 5 → 6 → 8),随后条件关闭
3 Agent 运行中断开最后一个客户端 ✅ Session 被保留;只有在子进程确认无 hold 后才关闭
4 子进程拒绝条件关闭 ✅ 线上出现 {"closed":false,"holds":[{"category":"agent",…}]};daemon 采纳该 hold 集(分级由 partial 翻为 full)
5 子进程 close 应答卡住 ✅ 不重试、不假设、Session 存活 —— ⚠️ prompt 路径见说明 A
6 未确认能力的子进程 ✅ 用真正的 PR 前 qwen --acp 构建挂在新 daemon 后面:activeWorkReporting: "none",浅层 health 严格为 {"status":"ok"},清理行为与 PR 前完全一致
7 daemon 级聚合 + 503 aggregation_failed ✅ 2 个 workspace / 3 个 Session,单个 workspace 的 Agent 即可让全局 activeWork: true;让 getter 抛异常仍返回 503 {"reason":"aggregation_failed"},并在 stderr 标注是哪个 workspace

决定性 A/B(第 3 条)

同一脚本、同一 mock、两个构建:

步骤 AFTER(PR head) BEFORE(merge-base)
prompt settle,Agent running sessions=1 activePrompts=0 activeWork=true reporting=full sessions=1 activePrompts=0
POST /detach → t+2s Session 存活 Session 已销毁
t+10s Session 存活 已销毁
Agent 与通知 settle 之后 干净关闭,sessions=0 (早已消失,Agent 一并被杀)

计划之外的补充验证

  • 能力协商逐字抓取:daemon 发 {"v":1,"intervalMs":15000},子进程回 {"v":1,"intervalMs":15000,"categories":["agent","notification"]},随后在 qwen/notify/channel/active-work 上发布 channel 级全量快照。
  • unknown 分级是诚实的。 用 tee 丢弃全部快照后,被采纳的 hold 集老化超过 3 × 15 s,daemon 降级为 activeWorkReporting: "partial"、activeWorkStaleMs: 0、activeWork: true —— 保留 Session,并如实标注"这个值没人担保"。
  • 对 hasUnfinalizedTasks() 判据做了变异测试。 直接驱动真实编译后的 BackgroundTaskRegistry:在 cancel 窗口内 hasRunningTasks() 为 false、hasUnfinalizedTasks() 为 true,而 listUnfinalizedBackgroundAgentIds() 仍返回该 id。把编译产物改成 running-only 判据后返回 [] —— 判据选择确实承重,与文档描述一致。
  • PR head 单测全绿: acp-bridge 498、cli acp-integration 929、cli serve + Session 1230、core background-tasks 123,合计 2780 条,无失败(我没有遇到 Live Appshot 那条 flake)。

说明 A —— confirm 窗口内的 prompt 会被接受(202)并丢失

rewindSession 与 spawnOrAttach 是同步 throw,所以 REST 返回 404 "The session is closing" —— 已验证。而 sendPrompt 是返回一个 rejected promise,路由外层的 try/catch 只能捕获同步抛出,此时它已经回过 202 {promptId} 了。在被卡住的 10 秒 confirm 窗口内实测:

POST /session/:id/rewind -> 404  "The session is closing"
POST /session/:id/prompt -> 202  {promptId}
    daemon 日志: [SessionNotFoundError] ... The session is closing; retry after close completes
    该 promptId 产生的事件数: 0

调用方既拿不到错误,事件流上也什么都没有。这个形状早于本 PR(if (entry.closing) return Promise.reject(...)),因此不是回归;但本 PR 把它适用的窗口从"一次快速的同步拆除"扩大到"有界的 10 秒往返",而测试计划第 5 条写的是 prompt 会"被拒绝而不是被接受后丢失",这与 REST 调用方观察到的不符。建议要么调整这句措辞,要么让 sendPrompt 的守卫像另外两处一样同步抛出。

说明 B —— 条件关闭没有携带 daemon 的 drain 预算

confirmChildUnheld 只发送 {sessionId, onlyIfUnheld}(已在线上确认),因此子进程回落到 SESSION_DRAIN_TIMEOUT_MS = 30_000,而真正的 close 发送的是 drainTimeoutMs = initTimeoutMs * 0.8 = 8000,daemon 自身也只等 ACTIVE_WORK_CLOSE_TIMEOUT_MS = 10_000。于是一个 drain 合理耗时 10–30 秒的 Session 会让 confirm 超时,被保留到下一轮。方向是安全的且能自愈,但多出了一个 8 秒预算本想避免的循环 —— 建议在条件调用上一并转发 drainTimeoutMs。(缺少该参数是运行时确认的;后果部分是代码阅读推论。)

结论

机制与描述相符,失败方向确实落在安全侧(未上报或未确认 ⇒ 保留,绝不销毁),跨版本路径也让旧子进程保持原样。两条说明属于文档与打磨,不构成正确性阻塞。

LGTM —— 建议合并。 说明 A 的措辞值得在合并前后顺手改一行。

wenshao
wenshao previously approved these changes Aug 7, 2026
Review found three ways the conditional close can still destroy a live
Session. All three share a cause: the round trip turned a synchronous
guard-then-teardown into an awaited span, and three things that were
previously impossible to observe mid-teardown now are.

**A restore in flight looks exactly like an abandoned Session.**
`session/load` registers the entry before awaiting `artifacts.restore()`
and `seedSessionUpdates()`, and registers its first client only after —
so for that whole window there are no clients, no subscribers, nothing
held, and the child answers the conditional close truthfully. The
snapshot trigger this PR added fires inside it. Excluded in
`entryIsAutoCloseCandidate` rather than at the snapshot trigger, so the
reaper's TTL elapsing inside a slow restore is covered too.
`pendingRestoreIds` already existed but was read only by
`hasNoChannelWork`, never by the close funnel.

**Teardown re-resolved the target by id without re-checking identity.**
`closeSessionImpl` does a fresh `byId.get`, and the id can be
re-registered to a different entry during the round trip: an explicit
kill removes this one (kill ignores the in-flight flag by design, keeping
its force semantics) and a `session/load` for the same persisted id
registers a fresh one. The stale continuation then tore down the newly
restored Session under its just-attached client. One identity re-check
after the await.

**The restore path was not upgraded to the new admission predicate.**
`sendPrompt`, `rewindSession`, and single-scope attach check
`isClosingOrAuthorizingClose`; `restoreSession` still checked bare
`closing` at both its guards, so a client could attach inside the window
and lose the session under it. That directly contradicted the
`closeIfChildUnheld` comment claiming every admission path checks the
flag. Its `racedEntry` branch had no closing guard at all — a narrower
pre-existing hole, same defect, same predicate.

Regression test covers the restore-path admission refusal. The other two
need a mid-restore snapshot and a kill-then-reload interleave that the
mocked-channel harness cannot stage honestly; both are pinned by reading
the code paths, which is weaker and worth saying.
@doudouOUC

Copy link
Copy Markdown
Collaborator Author

Review round addressed — 3 Critical fixed in 12adf869, 9 Suggestions deferred

All three [Critical] findings were real. I verified each against the code rather than taking the tag, and one had an inaccurate provenance claim that I've corrected in-thread while accepting the underlying bug.

Fixed

Finding Verified how Fix
Snapshot auto-close can tear down a session/load mid-restore Entry is registered before await artifacts.restore() / seedSessionUpdates(), first client only after; pendingRestoreIds was read solely by hasNoChannelWork, never by the close funnel Excluded in entryIsAutoCloseCandidate — not at the snapshot trigger as the inline suggestion had it, because the reaper's TTL can elapse inside a slow restore too
Teardown re-resolves by id after the await without an identity re-check closeSessionImpl does a fresh byId.get; kill deliberately ignores the in-flight flag, so the id can be re-registered mid-round-trip One identity re-check after confirmChildUnheld
restoreSession guards on bare closing, not the new predicate Three admission paths upgraded, this one missed — contradicting this PR's own "every admission path checks this flag" comment Both guards upgraded, plus the racedEntry branch which had no closing guard at all

All three share one cause: turning a synchronous guard-then-teardown into an awaited span made three previously unobservable mid-teardown states reachable. That is the same root cause as the guard rework earlier in this PR, which is the honest reading — the confirm window is genuinely the risky part of this design, and this is the third round of finding things inside it.

Verification: acp-bridge 1111/1111, active-work subset 18/18 including a new regression for the restore-path admission refusal. The other two fixes are pinned only by reading the code paths — a mid-restore snapshot and a kill-then-reload interleave are not something the mocked-channel harness can stage honestly, and I'd rather say so than imply test coverage I don't have.

Deferred — 9 Suggestions, replied and resolved individually

Not disputed, and none silently dropped. This is round 6+ of automated review on a PR that already carries a maintainer approval, so autonomous changes are held to critical-level findings rather than accumulating commits on an approved diff.

Five are flagged to the author as worth promoting, because they are the same class of problem this PR has already had to correct twice — a claim broader than the implementation:

  • The refusal-response hold set is adopted unbounded, while parseActiveWorkSnapshot caps the identical payload. Capping only one of two ingestion paths is an inconsistency introduced here.
  • A single huge-but-valid seq permanently latches the channel high-water mark, costing the self-healing property the design doc advertises (fails closed, so no destruction).
  • The reporter's eager constructor publish appears to be always discarded, which would make the comment justifying it false.
  • Design doc drift: the close-response success shape, and the child proposes, the daemon clamps is inverted.
  • Protocol doc drift: the activeWork enumeration omits the fail-closed unknown ⇒ active behavior a restart controller depends on.

The remaining four are bounded churn or scope (enqueueBackgroundNotification, setSessionApprovalMode / setSessionModel, a dead readonly, and the test-coverage gap — partly exercised by the maintainer's live run, though not by a committed test, so the mutation described would still pass CI).

Threads: 12/12 replied and resolved. 0 unresolved.

@wenshao

wenshao commented Aug 8, 2026

Copy link
Copy Markdown
Collaborator

Runtime re-verification at 12adf869 — the three critical fixes

My earlier report tested 80019acab7. Three [Critical] fixes landed since, and the author states two of them are "pinned only by reading the code paths" — a mid-restore snapshot and a kill-then-reload interleave. I rebuilt the environment at the new head and went after exactly those.

Setup. Three arms, all compiled from source into runnable dist and driven as real processes — no mocked bridge, no mocked child, no mocked health route:

AFTER PR head 12adf869
BEFORE (fix-level) previous head 80019acab7 — isolates the three fixes
BEFORE (PR-level) merge-base 20b9504276
Daemon node packages/cli/dist/index.js serve --workspace … --token …, fresh daemon + fresh workspace per scenario
Child real qwen --acp process
Fault injection QWEN_CLI_ENTRY stdio tee: logs every JSON-RPC frame both ways, and can delay the response to qwen/control/session/close {onlyIfUnheld:true}, delay any close, drop snapshots, or replay the child's own snapshot at a chosen instant
Model scripted OpenAI-compatible mock that launches a background Agent and then parks the sub-agent's completion on demand

The three fixes

# Fix Verdict
1 restoreSession guards on the new predicate, not bare closing ✅ confirmed live — decisive A/B below
2 identity re-check after confirmChildUnheld ⚠️ not stageable over REST — see below
3 pendingRestoreIds excluded from entryIsAutoCloseCandidate ⚠️ attempted hard, inconclusive

Fix 1 — restore admission (confirmed)

Same driver, same stalled child (conditional-close response held 7 s), two daemons. POST /session/:id/load issued 1.2 s into the confirm window:

On 80019ac the caller is told 200 {"attached":true, clientId:…} and the teardown it raced then destroys the Session under it — no error ever reaches the client. On 12adf869 it is refused up front with 404 "The session is closing; retry after close completes". This is a real defect, really fixed.

The full admission matrix inside an 8 s window at the new head is in the screenshot: load 404, rewind 404, attach-by-id 409, prompt 202 then lost, DELETE 409.

Fix 2 — identity re-check (could not be staged, and that is informative)

To reach it I need the id removed and re-registered during the confirm round trip. Over REST that is not reachable at this head:

  • there is no kill route (killSession is only called from sub-session/archive/rollback paths), and
  • DELETE /session/:id inside the window returns 409 session_archiving, Retry-After: 5 — because POST /session/:id/detach runs under withOwnerMutableSession, which holds a shared archive lock for the whole handler, and the handler now awaits the conditional close.

So the lock that blocks my repro is also what narrows this race's real-world exposure — the re-registration has to come from a non-REST path. The guard costs one map lookup and fails safe; I'd keep it, but it stays code-read-only.

Fix 3 — mid-restore exclusion (inconclusive, and I tried hard)

I could not reach the state it prevents:

  • The window is ~1 ms. Polling /health?deep=1 at ~5 samples/ms across a 501 ms session/load of a 1200-record transcript, exactly 1 of 2453 samples saw the entry registered while the load was still in flight.
  • Snapshot path: replaying the child's own qwen/notify/channel/active-work snapshot 700 times at 1 ms resolution across the whole restore did not tear the Session down on 80019ac.
  • Reaper path: with --session-reap-interval-ms 1 --session-idle-timeout-ms 1, both builds behave identically — the conditional close for the restoring id goes out 5–6 ms after the child's load response on 80019ac and on 12adf869. I cannot separate "decided inside the guarded window" from "decided in the unguarded tail between the restore promise settling and the HTTP response being written", so the test does not discriminate.

Not evidence the fix is wrong — evidence that the live path is too narrow to pin. If you want it pinned, a unit test with a controllable await inside the restore is the only honest way.

What still holds at the new head

  • Headline claim, re-proven against the merge base. Main prompt settled with a background Agent still running: activePrompts: 0, activeWork: true, reporting: "full". Detach the last client → merge-base destroys the Session (404, channel dead, Agent killed with it); PR head retains it (200). Agent + terminal notification settle → clean close, sessions: 0, activeWork: false.
  • Negotiation and holds captured verbatim. Daemon sends initialize._meta {"v":1,"intervalMs":15000}; child answers {"v":1,"intervalMs":15000,"categories":["agent","notification"]}; holds appear as {"category":"agent","id":"general-purpose-call_3"} while activePrompts is already 0.
  • Fail-closed is honest. A Session on a negotiated channel that has not been named by a snapshot yet reads activeWork: true, reporting: "partial" until its first report — the documented "unknown ⇒ retained" state, observed live.
  • Unit suites at 12adf869: acp-bridge 499/499, cli acp-integration 929/929, cli serve 1218/1218, core background-tasks 123/123 — 2769 tests, no failures. CI on this head is green on every job.

Notes (non-blocking)

A. A prompt inside the confirm window is still accepted (202) and lost. Unchanged from my last report and re-measured here: POST /session/:id/prompt → 202 {promptId}, and after the window GET /session/:id/pending-prompts → 404. load and rewind both throw synchronously and answer 404; sendPrompt returns a rejected promise the route's try/catch cannot see. Pre-existing shape, but the PR widens the window it applies to, and test-plan item 5 claims the prompt is "refused rather than accepted and lost", which is not what a REST caller observes.

B. The conditional close still omits the drain budget. On the wire: {"sessionId":…,"onlyIfUnheld":true} versus the real close's {"sessionId":…,"drainTimeoutMs":8000}. The child falls back to SESSION_DRAIN_TIMEOUT_MS = 30_000 while the daemon waits only 10 s. Self-correcting, safe direction, one wasted cycle.

C. Detach costs one extra bounded round trip. I checked whether this is new before reporting it: with every session/close response stalled 8 s, POST /session/:id/detach takes 8019 ms on the merge base, 16017 ms on 80019ac, 16016 ms on 12adf869. So blocking detach is pre-existing; the PR adds one bounded confirm in front of it (worst case 10 s confirm + 8 s drain). With a healthy child it is 15 ms. Worth knowing because the route holds its shared archive lock for that whole span, so DELETE/archive on that Session are refused with 409 meanwhile — "explicit close keeps force semantics" is true of the bridge, but not of what a REST caller sees.

Verdict

The one fix that is observable from outside the daemon is genuinely fixed, and I reproduced the bug it fixes on the previous head. The other two are in the safe direction and cheap; I could not stage either, which matches the author's own statement rather than contradicting it. Everything I verified last round still holds at 12adf869, and the failure direction remains "unreported or unconfirmed ⇒ retained, never destroyed".

LGTM — recommend merge, my previous approval stands at this head. Note A is worth one line of code or one line of prose before or after merge.

中文版

在 12adf869 上的运行时复验 —— 针对三个 Critical 修复

我上一份报告测的是 80019acab7。此后合入了三个 [Critical] 修复,作者明确说明其中两个只靠读代码确认,没有测试覆盖(restore 期间的快照、kill 后重新加载的交错)。我在新 head 上重建了环境,专门去打这两个点。

环境。 三个 arm,全部从源码编译成可运行的 dist,以真实进程驱动 —— bridge、子进程、health 路由都没有 mock:

AFTER PR head 12adf869
BEFORE(修复级) 上一个 head 80019acab7 —— 用于隔离这三个修复
BEFORE(PR 级) merge-base 20b9504276
Daemon node packages/cli/dist/index.js serve --workspace … --token …,每个场景都用全新 daemon + 全新 workspace
子进程 真实 qwen --acp 进程
故障注入 QWEN_CLI_ENTRY stdio tee:双向记录每一条 JSON-RPC 帧,并可延迟 qwen/control/session/close {onlyIfUnheld:true} 的应答、延迟任意 close、丢弃快照,或在指定时刻重放子进程自己的快照
模型 脚本化的 OpenAI 兼容 mock:先拉起后台 Agent,再按需挂起子 agent 的补全

三个修复

# 修复 结论
1 restoreSession 改用新判据而非裸 closing ✅ 实测确认
2 confirmChildUnheld 之后补身份重校验 ⚠️ REST 层无法构造
3 entryIsAutoCloseCandidate 排除 pendingRestoreIds ⚠️ 尽力尝试,无法判定

修复 1(已确认)。 同一脚本、同一被卡住的子进程(条件关闭应答延迟 7 秒),两个 daemon;在 confirm 窗口内 1.2 秒处发 POST /session/:id/load:80019ac 返回 200 {"attached":true, clientId:…},随后它所竞争的拆除把这个 Session 在客户端脚下销毁 —— 调用方拿不到任何错误;12adf869 直接 404 "The session is closing; retry after close completes"。真实缺陷,真实修好。窗口内完整的准入矩阵见截图:load 404、rewind 404、按 id attach 409、prompt 202 然后丢失、DELETE 409。

修复 2(无法构造,但这件事本身有信息量)。 要触发它,必须在 confirm 往返期间把同一个 id 移除再重新注册。当前 head 上 REST 做不到:没有 kill 路由(killSession 只在子会话/归档/回滚路径里被调用);而窗口内的 DELETE /session/:id 会返回 409 session_archiving(Retry-After: 5)—— 因为 POST /session/:id/detach 跑在 withOwnerMutableSession 里,整个 handler 期间持有 archive 的共享锁,而该 handler 现在要 await 条件关闭。也就是说,挡住我复现的那把锁,同时也压缩了这个竞态的现实暴露面 —— 重新注册只能来自非 REST 路径。这个守卫只值一次 map 查找且失败方向安全,我建议保留,但它仍然只是"读代码确认"。

修复 3(无法判定,且我确实尽力了)。

  • 窗口只有约 1 毫秒:以约 5 次/毫秒的频率轮询 /health?deep=1,覆盖一次 1200 条记录、耗时 501 ms 的 session/load,2453 个采样里只有 1 个看到"条目已注册但 load 仍在进行"。
  • 快照路径:把子进程自己的 qwen/notify/channel/active-work 快照以 1 ms 粒度重放 700 次贯穿整个 restore,80019ac 上 Session 依然没有被拆除。
  • reaper 路径:用 --session-reap-interval-ms 1 --session-idle-timeout-ms 1,两个构建表现完全一致 —— 针对正在 restore 的 id 的条件关闭,都在子进程 load 应答之后 5–6 ms 发出。我无法区分"在被守卫的窗口内做的决定"和"在 restore promise settle 之后、HTTP 响应写出之前那段无守卫尾巴里做的决定",因此该测试不具备区分力。

这不是说修复错了,而是说这条活路径太窄、钉不住。若要钉死,只能写一个能控制 restore 内部 await 时机的单测。

新 head 上依然成立的部分。 主 Prompt 结束、后台 Agent 仍在跑时:activePrompts: 0、activeWork: true、reporting: "full";断开最后一个客户端后,merge-base 销毁 Session(404,通道死亡,Agent 一并被杀),PR head 保留(200);Agent 与终态通知 settle 后干净关闭,sessions: 0、activeWork: false。能力协商逐字抓取:daemon 发 {"v":1,"intervalMs":15000},子进程回 {"v":1,"intervalMs":15000,"categories":["agent","notification"]},hold 形如 {"category":"agent","id":"general-purpose-call_3"}。尚未被快照点名的 Session 读作 activeWork: true, reporting: "partial",即文档中的"unknown ⇒ 保留",实测如此。单测:acp-bridge 499/499、cli acp-integration 929/929、cli serve 1218/1218、core background-tasks 123/123,合计 2769 条全绿;该 head 的 CI 全部 job 通过。

说明(均不阻塞)。

  • A. confirm 窗口内的 prompt 仍然被 202 接受然后丢失。 与上次一致并重新实测:POST /session/:id/prompt → 202 {promptId},窗口结束后 GET /session/:id/pending-prompts → 404。load 与 rewind 都是同步抛出并返回 404;sendPrompt 返回的是 rejected promise,路由的 try/catch 看不见。形状早于本 PR,但本 PR 扩大了其适用窗口,而测试计划第 5 条写的是 prompt 会"被拒绝而非被接受后丢失",与 REST 调用方观察到的不符。
  • B. 条件关闭仍未携带 drain 预算。 线上为 {"sessionId":…,"onlyIfUnheld":true},而真正的 close 是 {"sessionId":…,"drainTimeoutMs":8000}。子进程回落到 SESSION_DRAIN_TIMEOUT_MS = 30_000,daemon 却只等 10 秒。可自愈、方向安全,只是多一个循环。
  • C. detach 多出一次有界往返。 我在报告前先做了对照:把所有 session/close 应答都延迟 8 秒,POST /session/:id/detach 在 merge-base 上耗时 8019 ms、80019ac 上 16017 ms、12adf869 上 16016 ms。所以"detach 会阻塞"并非新增,本 PR 只是在它前面加了一次有界 confirm(最坏 10 秒 confirm + 8 秒 drain);子进程健康时是 15 ms。之所以值得知道,是因为该路由在整段时间里持有 archive 共享锁,期间对该 Session 的 DELETE/归档会被 409 拒绝 —— "显式 close 保持强制语义"对 bridge 成立,但对 REST 调用方看到的结果并不成立。

结论

唯一能从 daemon 外部观测到的那个修复确实修好了,我也在上一个 head 上复现了它所修的缺陷。另外两个方向安全、代价极小;我两个都没能构造出来,这与作者自己的说法一致,而非相反。上一轮验证过的内容在 12adf869 上全部依然成立,失败方向仍然落在安全侧(未上报或未确认 ⇒ 保留,绝不销毁)。

LGTM —— 建议合并,我此前的 approve 在本 head 上继续有效。说明 A 值得在合并前后顺手改一行代码或一行措辞。

@wenshao

wenshao commented Aug 8, 2026

Copy link
Copy Markdown
Collaborator

@qwen-code /triage

@wenshao

wenshao commented Aug 8, 2026

Copy link
Copy Markdown
Collaborator

@qwen-code /review

@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor
_Qwen Code review request accepted. Review is queued in [workflow run](https://github.com/QwenLM/qwen-code/actions/runs/31243191061)._

@yiliang114 yiliang114 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The three round-1 Criticals are fixed at this head — verified by reading the current code, not just the author notes: snapshot auto-close now excludes pendingRestoreIds, the stale continuation is re-checked by identity immediately before closeSessionImpl, and all three admission paths (including the previously unguarded raced-entry branch) use isClosingOrAuthorizingClose.

Counter pairing on the reporting side holds on all paths (inc-before-try / dec-in-finally), holds are derived per report from the real work owners so nothing can outlive the work it names, unknown or stale state fails closed to busy with a downgraded reporting grade, and the new health fields are purely additive. CI is green on this head.

The remaining items are the ones already deferred with maintainer approval and tracked in the automated review thread — non-blocking: uncapped refusal-hold adoption, the seq high-water latch, the eager constructor publish that is deterministically discarded, the three admission paths still missing the close guard, and the doc drift. Nothing here blocks merge.

@doudouOUC
doudouOUC dismissed a stale review August 8, 2026 06:34

Already have 2 appoved,3ks

@doudouOUC
doudouOUC added this pull request to the merge queue Aug 8, 2026
Merged via the queue into QwenLM:main with commit 59b750f Aug 8, 2026
161 of 171 checks passed
doudouOUC added a commit to doudouOUC/qwen-code that referenced this pull request Aug 8, 2026
Merging main's active-work close protocol (QwenLM#8588) into this PR's abandoned
restore bound produced a deadlock that neither side has on its own, and the
conflict resolution was committed without running tests.

`maybeCloseIdleSession` now routes through `confirmChildUnheld`, which asks
the child whether it still holds work before closing a session nobody is
attached to. That is right in general and wrong for a channel this PR has
already condemned. `restoreSettlementOverdue` and quarantine exist precisely
because the child stopped being answerable, and their whole premise is that
visible work drains so the channel can be reaped — closing the transport is
the only thing that can release a restore we cannot cancel. Making that
drain depend on a round trip to the wedged child inverts it: a child stuck
in a non-cancellable restore is exactly the one that cannot reply inside
`ACTIVE_WORK_CLOSE_TIMEOUT_MS`, so the sessions never close, the channel
never drains, the reap never fires, and the bound never takes effect.

A channel condemned by the restore lifecycle now skips the round trip and
proceeds to local teardown. Nothing is attached to the session by then —
`maybeCloseIdleSession` gates on that — and the sibling-safety invariant is
untouched: this closes sessions whose clients have already left, it does not
force-kill a channel that still has live ones.

The regression test drives an overdue channel whose child never answers the
close-if-unheld probe and asserts the detach still reaps it. Reverting the
guard reproduces the deadlock as a test timeout.

Co-Authored-By: Claude Opus 5 <[email protected]>
pull Bot pushed a commit to mcx/qwen-code that referenced this pull request Aug 9, 2026
…#8691)

* fix(serve): make session restore timeouts safe

Co-authored-by: Qwen-Coder <[email protected]>

* fix(cli): restore missing core mock exports in the ACP worktree suite

The restore-tracing change added `extractDaemonTraceContext` and
`withDaemonSpan` to `acpAgent.ts`, but `acpAgent.worktree.test.ts`
replaces `@qwen-code/qwen-code-core` with a full mock factory that never
listed them. `loadSession` then failed on an undefined export, taking all
three cases down and producing teardown rejections from the half-built
agent. The sibling suite was updated; this one was missed.

Co-Authored-By: Claude Opus 5 <[email protected]>

* fix(serve): bound and disambiguate the abandoned restore lifecycle

Four follow-ups from review of the restore timeout work.

A startup budget may now raise the restore budget but never lower it.
Taking an explicitly configured `initializeTimeoutMs` as the restore
fallback meant a deployment that tightened its child-initialize check
still inherited a sub-default restore deadline — exactly the failure this
change exists to remove. An explicit `sessionRestoreTimeoutMs` still wins
outright, including below the default, for deployments that want restore
to fail fast. Validation now names the field actually at fault.

A restore fenced behind a timed-out predecessor is no longer reported as
an ordinary in-flight restore. It carries `reason:
awaiting_abandoned_cleanup` and a retry hint of one restore budget
(capped at 120s) instead of the ordinary 5 seconds, because the fence
cannot clear until the non-cancellable ACP request settles and a 5-second
cadence just spins the caller against a 409 it cannot resolve.

Whether a channel is condemned is now derived rather than sticky. A
timeout recorded `emptyReapPending` permanently, so any channel that had
ever seen one was guaranteed to be reaped once its remaining work
drained, forcing a cold respawn even when the late restore had landed and
closed cleanly. The reap condition is now computed from an outstanding
`unsettledAbandonedRestores` set, quarantine, or an ordinary pending
empty reap; real settlement clears the entry and hands the channel back
to the configured idle policy.

Abandonment no longer retains ownership without bound. One further
restore budget after the deadline, a still-unsettled restore marks the
channel `restoreSettlementOverdue`: existing sessions and workspace
control keep working, but fresh session work is refused so the channel
can drain, since closing the transport is the only lever that releases a
permanently hung request. Releasing capacity while hidden work runs would
allow unbounded oversubscription, and force-killing a channel with live
siblings would reintroduce the failure this work removes, so neither is
done. Fresh-admission blocking is now scanned across alive channels
rather than tracked in a single reference, so a second condemned channel
cannot silently displace the first.

Co-Authored-By: Claude Opus 5 <[email protected]>

* fix(serve): keep the abandoned restore lifecycle off ids it no longer owns

Two correctness gaps in the abandoned-restore machinery introduced by this
PR, both reported by automated review and both confirmed by mutation
testing (each new test fails when its fix is reverted).

A caller-supplied `sessionId` is used verbatim by the agent, but
`spawnOrAttach` never consulted `inFlightRestores`. A fresh spawn could
therefore take an id that a restore still owns, in either lifecycle phase.
The consequences were silent: `abandonedRestoreIds` suppresses session
updates, guardrail events, and child notifications, so the new session
would have registered successfully and then emitted nothing; and a late
`settleAbandonedRestore` would have closed and tombstoned it out from
under its owner. Such a spawn is now rejected with the same
`RestoreInProgressError` and reason the restore path uses, so the caller
gets the correct retry hint for whichever phase is holding the id.

The cleanup path is guarded independently, because the request-level check
only covers the id the caller asked for and a session registers under the
id the child returns. An abandoned restore never reaches
`createSessionEntry` — the deadline rejects before registration — so any
live entry under that id belongs to someone else. Cleanup now detects that
and returns without closing or tombstoning, releasing its own bookkeeping
instead.

The notification fence has no TTL and was only cleared by
`markRestoreInFlight`, which covers a subsequent restore and nothing else.
`createSessionEntry` now clears it for every registration route, so a
legitimate owner of the id is never handed a session that silently drops
everything the child sends it.

Also tightens two tests that could not observe the values they pin. The
SDK default restore timeout admitted any value in (30s, 70s]; it is now
split at the exact boundary, so collapsing the default onto the 60s server
budget — which would make the client abort race the daemon's own deadline
and cost the caller its structured 504 — fails. And the advertised-budget
propagation from capabilities through to the SDK call had no live-path
assertion; dropping the capabilities argument at the real call site left
every existing test green. The `as never` casts are replaced with typed
`DaemonCapabilities` values so a field rename fails typecheck.

Co-Authored-By: Claude Opus 5 <[email protected]>

* fix(serve): let a condemned channel drain without its wedged child

Merging main's active-work close protocol (QwenLM#8588) into this PR's abandoned
restore bound produced a deadlock that neither side has on its own, and the
conflict resolution was committed without running tests.

`maybeCloseIdleSession` now routes through `confirmChildUnheld`, which asks
the child whether it still holds work before closing a session nobody is
attached to. That is right in general and wrong for a channel this PR has
already condemned. `restoreSettlementOverdue` and quarantine exist precisely
because the child stopped being answerable, and their whole premise is that
visible work drains so the channel can be reaped — closing the transport is
the only thing that can release a restore we cannot cancel. Making that
drain depend on a round trip to the wedged child inverts it: a child stuck
in a non-cancellable restore is exactly the one that cannot reply inside
`ACTIVE_WORK_CLOSE_TIMEOUT_MS`, so the sessions never close, the channel
never drains, the reap never fires, and the bound never takes effect.

A channel condemned by the restore lifecycle now skips the round trip and
proceeds to local teardown. Nothing is attached to the session by then —
`maybeCloseIdleSession` gates on that — and the sibling-safety invariant is
untouched: this closes sessions whose clients have already left, it does not
force-kill a channel that still has live ones.

The regression test drives an overdue channel whose child never answers the
close-if-unheld probe and asserts the detach still reaps it. Reverting the
guard reproduces the deadlock as a test timeout.

Co-Authored-By: Claude Opus 5 <[email protected]>

* test(serve): pin the restore-timeout contract the review found unasserted

Automated review identified eleven places where the restore-timeout work's
behavior was correct but unpinned — each with a mutation that ships green.
Every fix below was verified the same way: apply the mutation, watch the new
assertion fail, revert, watch it pass.

The timeout path's telemetry had no coverage at all, which is the sharpest
gap given that observability is what this work exists to deliver. A shared
recorder now asserts the public timeout result and its kill_empty-vs-
fence_shared signal, the late arrival, and the cleanup outcome for both the
closed and quarantined cases.

The deadline timer's cancellation on a successful restore was likewise
unpinned: deleting both `clearTimeout` calls kept the whole suite green,
while in production the stale timer fires one budget after a successful
restore and abandons a live session — fencing its frames, closing its event
bus, and emitting a spurious timeout. A success-path test now advances past
the deadline and asserts no second public result.

Three more bridge assertions proved less than they claimed: the concurrent-
restore case never checked that the abandoned restore settles, the
workspace-control case never checked that the deferred reap eventually
fires, and the resolver never pinned the accepting side of the MAX boundary
(a `>` to `>=` mutation rejects the largest legal delay at boot). The
workspace-control case also needed a positive channel idle budget, since
with the default zero the idle-timer kill substitutes for the reap junction
under test; its assertions are rewritten around the derived reap semantics
rather than the sticky flag they predate.

Outside the bridge: the scheduled-task timeout wiring had no test, so
deleting the arguments silently fell back to the helpers' own defaults; the
cold restore path never asserted that `live_restore_ms` is absent; the SDK's
per-request validation and its over-ceiling clamp were untested; the WebUI
watchdog test jumped straight to its own value, staying green for any
watchdog at or below it, including the 30s attach value that would recreate
the original symptom in the browser; and the two new known error types were
unexercised, so dropping either would relabel every restore-timeout and
quarantine error as unknown.

Two review items are deliberately not taken here and are recorded in the
design doc's non-goals instead: transcript materialization is still not
separately attributable from `config_setup`, which needs instrumentation
inside the core session loader that P1/P2 restructures anyway, and sibling
event-loop latency during a large restore remains unmeasured.

Co-Authored-By: Claude Opus 5 <[email protected]>

* fix(serve): bound the condemned-channel close and complete the fence contract

Second automated review round, on the code the first round produced. One
Critical and twelve suggestions; all verified by mutation before and after.

**The Critical is a regression I introduced.** Letting a condemned channel
skip the bounded hold probe routed it into `closeSessionImpl`, whose agent
close is unbounded when it throws on failure — so the fix traded a bounded
wait on a wedged child for an unbounded one. A settlement-overdue channel
with an unresponsive child would hang `detachClient` forever, strand the
session in `closing`, never drain, never reap, and 503 every new session
until restart: strictly worse than before. `CloseSessionOpts` now carries an
`agentCloseTimeoutMs` that the condemned path sets, so a hang lands in the
existing unknown-outcome recovery, which kills the channel — the teardown
the drain was waiting for. The earlier test missed this because its fake
child still answered the plain close; it now answers nothing at all, and
asserts the detach itself returns.

**The fence was invisible on the transports clients actually use.**
`toRpcError` had no `RestoreInProgressError` case, so over acp-http and
acp-ws — which SDK negotiation prefers over REST — the fence degraded to an
opaque internal 500 with no code, reason, or hint, and the backoff contract
this work documents was impossible to honor.

**Two retry hints still advertised five seconds for states that outlive a
budget.** The restore 504 creates the fence, and quarantine lasts until the
channel drains; a fresh-id caller never reaches the 409 that carries the
real hint, so its header was the only signal it got. Both now derive from
the budget through one shared clamp helper, which also replaces the formula
that was inlined in the bridge and gives the documented 5-120s bounds a
test.

**A spawn collision reported an operation the caller never issued**, naming
the restore owner's action as both the active and the requested one and
telling the caller to retry an endpoint it never called.

The rest: five places still described the initialize-timeout fallback as a
plain chain rather than raise-only, contradicting sibling docs shipped in
this same PR; the design doc omitted the retry-hint clamp; the protocol
reference omitted the new spawn emission site; the error taxonomy omitted
`restore_settlement_overdue`, which matters because its audience is
monitoring. Test-only gaps: the dynamic 409 had no HTTP-layer coverage, the
120-second cap was unpinned, and the SDK's precedence of an explicit global
timeout over the advertised budget was pinned only branch-by-branch.

Co-Authored-By: Claude Opus 5 <[email protected]>

* fix(serve): preserve restore session ownership handoff

Co-authored-by: Qwen-Coder <[email protected]>

---------

Co-authored-by: Qwen-Coder <[email protected]>
Co-authored-by: Claude Opus 5 <[email protected]>
Co-authored-by: qwen-code-dev-bot <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants