Skip to content

fix(core): add SSE stream idle watchdog to abort hung streams (Fixes #4177) - #5330

Closed
aspnmy wants to merge 3 commits into
QwenLM:mainfrom
aspnmy:fix/stream-idle-watchdog-4177
Closed

aspnmy wants to merge 3 commits into
QwenLM:mainfrom
aspnmy:fix/stream-idle-watchdog-4177

Conversation

@aspnmy

@aspnmy aspnmy commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

What this PR does

Adds a configurable SSE stream idle watchdog that aborts hung streams in weak-network scenarios. When an SSE connection silently drops between events, the stream would previously hang indefinitely — the user saw a frozen spinner and the only recovery was Ctrl+C → restart.

This PR adds a setTimeout-based idle watchdog inside processStreamResponse that:

  • Resets on every received chunk
  • Fires after QWEN_CODE_STREAM_IDLE_TIMEOUT_MS (default 90s) of silence
  • Emits a one-shot warning at half the timeout interval
  • Aborts via linked AbortController, which triggers the existing InvalidStreamError retry budget

Why it's needed

Fixes #4177. The PR #4176 (merged) fixed the partial-history side of weak-network failures. This PR addresses the remaining case where the SSE stream goes dead-silent, producing no chunk to trigger our existing timeout logic.

Design based on Claude Code's CLAUDE_ENABLE_STREAM_WATCHDOG subsystem (Layer 1 of the Streaming Resilience proposal), adapted for qwen-code's codebase conventions.

Reviewer Test Plan

How to verify

  1. Set QWEN_CODE_STREAM_IDLE_TIMEOUT_MS=5000 for a short test timeout
  2. Start a session and trigger a streaming API call
  3. Kill the network mid-stream (iptables DROP, airplane mode, etc.)
  4. Observe: ~2.5s warning log, ~5s hard abort with InvalidStreamError('STREAM_IDLE_TIMEOUT')
  5. The stream retries via existing transient-retry budget and recovers

Evidence (Before & After)

N/A (non-UI, stream transport layer change).

Tested on

OS Status
🍏 macOS ⚠️
🪟 Windows ⚠️
🐧 Linux ✅
  • Linux: container typecheck ✅ (zero errors), prettier ✅
  • Windows/macOS: not tested (no environment available)

Risk & Scope

  • Main risk: AbortController.abort() may surface differently across content generators. Tested with the default Gemini provider path.
  • Not validated: non-Gemini providers, sandbox environments.
  • Breaking changes: None. Watchdog defaults to active but can be disabled via QWEN_CODE_DISABLE_STREAM_WATCHDOG=1.
  • This is a clean implementation of PR fix(core): add stream idle watchdog for silent responses #4256's reviewed-and-approved design, rebuilt atop current upstream/main with correct UTF-8 encoding.

Linked Issues

Fixes #4177

Credit

Design and original implementation: @Alexxigang in PR #4256
Rebased and encoding-fixed: this PR

中文说明

本 PR 做了什么

添加了一个可配置的 SSE 流空闲看门狗,在弱网环境下中止挂起的流。当 SSE 连接在事件之间静默断开时,流会无限挂起——用户看到冻结的 spinner,只能 Ctrl+C 重启。

实现方式:在 processStreamResponse 内添加基于 setTimeout 的空闲看门狗,每收到 chunk 重置计时器,QWEN_CODE_STREAM_IDLE_TIMEOUT_MS(默认 90 秒)无数据后触发中止。

为什么需要

修复 #4177。PR #4176(已合并)修复了弱网故障中"部分历史丢失"的问题。本 PR 处理剩余情况:SSE 流完全静默断开。

审查者测试计划

  1. 设置 QWEN_CODE_STREAM_IDLE_TIMEOUT_MS=5000 缩短测试超时
  2. 触发流式 API 调用后切断网络
  3. 观察 ~2.5 秒警告日志,~5 秒硬中止
  4. 流通过已有的瞬态重试预算恢复

风险与范围

  • 主要风险:AbortController.abort() 在不同内容生成器上表现可能不同
  • 未验证:非 Gemini 提供商、沙箱环境
  • 破坏性变更:无。可通过 QWEN_CODE_DISABLE_STREAM_WATCHDOG=1 禁用

关联 Issue

Fixes #4177

@qqqys qqqys left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Critical: the watchdog never observes the stream iterator. processStreamResponse now accepts _streamWatchdog/_cleanupAbortLink, but the generator still uses for-await over streamResponse directly and never calls streamWatchdog.next(...); the only cleanup path added is around synchronous generator creation. If the provider hangs while awaiting the next SSE chunk, that for-await remains stuck forever, the timeout never rejects, and the abort controller is never triggered. This leaves the core session hang from #4177 unfixed after merge. Please route each streamResponse.next() through the watchdog and clean it up when iteration finishes or throws.

@wenshao

wenshao commented Jun 18, 2026

Copy link
Copy Markdown
Collaborator

@qwen-code /triage

1 similar comment
@wenshao

wenshao commented Jun 18, 2026

Copy link
Copy Markdown
Collaborator

@qwen-code /triage

});

return this.processStreamResponse(model, streamResponse);
try {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Critical] Dead try/catch around an async generator — cleanup is unreachable

processStreamResponse is declared async *, so calling it returns an AsyncGenerator object synchronously without executing any function body. Errors during stream iteration happen in the caller's for await loop, not here. The catch block (which contains the only calls to streamWatchdog.cleanup() and cleanupAbortLink()) can never fire.

This means cleanup of both the watchdog timers and the abort-signal listener never runs on any exit path — not on normal completion, not on error, not on caller abort.

Fix: Remove this try/catch and move cleanup into a try/finally inside processStreamResponse's generator body (wrapping the for await loop). Generator finally blocks run on .return(), .throw(), and normal completion.

— qwen3.7-max via Qwen Code /review

prompt_id: string,
): Promise<AsyncGenerator<GenerateContentResponse>> {
const streamAbortController = new AbortController();
const cleanupAbortLink = linkAbortSignal(params.config?.abortSignal, streamAbortController);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Critical] Abort event listener leak — cleanupAbortLink is never called

linkAbortSignal adds an 'abort' listener to the caller's AbortSignal. The returned cleanup function is stored as cleanupAbortLink but is only called in the dead catch block at line 2671 (see other comment). On the normal completion path, the listener is never removed.

Each stream call in a long-lived session leaks one listener on the parent signal. With enough turns, this triggers MaxListenersExceededWarning and unbounded listener growth.

Fix: Call _cleanupAbortLink?.() from a finally block inside processStreamResponse's generator body, ensuring it runs on all exit paths.

— qwen3.7-max via Qwen Code /review

model,
contents: requestContents,
config: { ...this.generationConfig, ...params.config },
config: { ...this.generationConfig, ...params.config, abortSignal: streamAbortController.signal },

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Critical] Existing test failure — config: {} assertion no longer matches

This line injects abortSignal: streamAbortController.signal into the config passed to generateContentStream. The existing test at geminiChat.test.ts:1476 asserts config: {}, which now fails because the actual config is { abortSignal: AbortSignal{...} }.

CI confirms: Test fails on all 3 platforms (macOS, Windows, Linux).

Suggested change
config: { ...this.generationConfig, ...params.config, abortSignal: streamAbortController.signal },
config: { ...this.generationConfig, ...params.config, abortSignal: streamAbortController.signal },

Update the test to use an asymmetric matcher:

config: { abortSignal: expect.any(AbortSignal) },

— qwen3.7-max via Qwen Code /review

));
}, timeoutMs);
});
return Promise.race([nextPromise, timeoutPromise]).finally(clearTimers);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Critical] Unhandled promise rejection when timeout wins the race

When the watchdog timeout fires before nextPromise settles, Promise.race rejects with the timeout error. However, nextPromise (the underlying stream's .next() call) is still pending. The timeout handler calls abortController.abort(), which causes the stream to eventually reject. Since Promise.race already settled, the nextPromise rejection has no handler attached — triggering an unhandled promise rejection warning (or crash in strict mode).

Suggested change
return Promise.race([nextPromise, timeoutPromise]).finally(clearTimers);
return Promise.race([nextPromise, timeoutPromise]).finally(clearTimers);
},

Should be:

    next<T>(nextPromise: Promise<IteratorResult<T>>): Promise<IteratorResult<T>> {
      nextPromise.catch(() => {}); // prevent unhandled rejection if timeout wins
      const timeoutPromise = new Promise<never>((_, reject) => {
        // ...
      });
      return Promise.race([nextPromise, timeoutPromise]).finally(clearTimers);
    },

The .catch(() => {}) doesn't prevent Promise.race from seeing the original rejection — it just silences the unhandled rejection when the losing promise later rejects.

— qwen3.7-max via Qwen Code /review

warningId = setTimeout(() => {
debugLogger.warn('Stream idle for ' + warningMs + 'ms from ' + model);
}, warningMs);
timeoutId = setTimeout(() => {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] abortController.abort() called without a reason argument

When the watchdog timeout fires, abort() is called with no reason. Downstream consumers that inspect signal.reason (retry predicates, error classifiers, telemetry) see undefined instead of the InvalidStreamError that is simultaneously thrown via reject().

Suggested change
timeoutId = setTimeout(() => {
const err = new InvalidStreamError(
'Stream idle timeout after ' + timeoutMs + 'ms',
'STREAM_IDLE_TIMEOUT',
);
abortController.abort(err);
reject(err);

This ensures signal.reason matches the thrown error and propagates correctly through linkAbortSignal's controller.abort(signal.reason) chain.

— qwen3.7-max via Qwen Code /review

@@ -0,0 +1,79 @@
import { createDebugLogger } from '../utils/debugLogger.js';

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] No test file for this new module

This module exports 4 functions with non-trivial logic: env var parsing, timer management via Promise.race, abort signal linking (with pre-aborted signal handling), and cleanup. No streamIdleWatchdog.test.ts exists.

Key untested scenarios:

  • isStreamWatchdogDisabled: env var '1', 'true', absent, other values
  • getStreamIdleTimeoutMs: disabled path, invalid values ('abc', '-1', '0')
  • linkAbortSignal: undefined signal, already-aborted signal, cleanup removing listener
  • createStreamIdleWatchdog.next(): timeout fires, promise resolves first, cleanup clears timers

— qwen3.7-max via Qwen Code /review

@wenshao

wenshao commented Jun 19, 2026

Copy link
Copy Markdown
Collaborator

Maintainer verification — local merge build (🔴 not mergeable as-is)

Verified the 3-way merge result (current origin/main e234cf6 + this PR head 0ae2a54) on macOS (Darwin arm64, Node v22.22.2). The PR is up to date with main (diff: a new streamIdleWatchdog.ts + geminiChat.ts integration, +95/-4). Two blocking problems: CI's Test jobs are red, and the watchdog is never actually invoked — so the #4177 fix does not run.

1. 🔴 The watchdog is plumbed but never invoked — the fix doesn't function

createStreamIdleWatchdog() returns { next, cleanup }, where next(streamStep) is the mechanism: it races each stream step against the idle timeout and aborts on stall. But in geminiChat.ts the watchdog object is only created (2611), passed to processStreamResponse (2668), and cleanup-attempted (2670) — there is no streamWatchdog.next(...) call anywhere. processStreamResponse receives it as _streamWatchdog (underscore = intentionally unused) and iterates the stream directly (for await (const chunk of streamResponse)), bypassing the watchdog entirely.

I confirmed the watchdog unit itself is correct (fake-timer test): next() on a stalled promise rejects with STREAM_IDLE_TIMEOUT and aborts after the timeout; next() on a resolved step clears both timers. So the building block works — it's just never wired into the stream loop. As written, a hung SSE stream is not detected or aborted by this watchdog.

(This compiles cleanly precisely because the unused params are _-prefixed, so no-unused-vars stays quiet — which is why it passed lint/typecheck while doing nothing.)

2. 🔴 CI Test jobs fail — geminiChat.test.ts not updated for the new abortSignal

The integration now passes abortSignal: streamAbortController.signal into the generateContentStream config. The existing assertion should call generateContentStream with the correct parameters checks the exact config and was not updated, so it fails on the extra field.

geminiChat.test.ts
origin/main (no PR) ✅ 192 / 192
merge result (with PR) ❌ 191 / 192

This matches the PR's CI (Test macOS + ubuntu: fail).

3. ⚠️ Cleanup sits in a catch that can never fire

try {
  return this.processStreamResponse(model, streamResponse, streamWatchdog, cleanupAbortLink);
} catch (error) {
  streamWatchdog?.cleanup(); cleanupAbortLink?.(); throw error;
}

processStreamResponse is an async generator — calling it returns the generator synchronously without running the body, so this catch only catches synchronous construction errors (there are none). Stream/iteration errors happen later, during consumption, and never reach here. Net effect: cleanupAbortLink() (which removes the abort listener added by linkAbortSignal) is never called on the normal path or on iteration errors → a per-stream abort-listener leak. Cleanup needs to live inside the generator (e.g. a finally) or wrap the iteration.

4. ⚠️ Format + tests

  • prettier --check packages/core/src/core/streamIdleWatchdog.ts reports style issues (the new file isn't formatted). eslint passes.
  • No tests are added for the watchdog. A hung-stream test (fake timers, a stalled next()) would have caught that the watchdog never fires.

What it needs before merge

  1. Wire streamWatchdog.next(...) into the stream loop in processStreamResponse (replace the bare for await with a manual loop that races each streamResponse.next() through the watchdog, or wrap streamResponse in a watchdog generator). Drop the _ once it's actually used.
  2. Move cleanup into a finally inside the generator so cleanupAbortLink() / cleanup() always run (the current try/catch around the generator construction is dead).
  3. Update geminiChat.test.ts for the new abortSignal config field (turn the failing assertion green).
  4. prettier --write the new file; add a watchdog unit test (the timeout-fires / timers-cleared cases).

Verdict

The watchdog primitive is well-designed and verified to work in isolation, but as integrated it is inert (never invoked) and the PR is red on CI (a test wasn't updated for the new abortSignal), plus a dead-catch cleanup leak and an unformatted file. 🔴 Changes requested — wire next() into the loop, fix the cleanup placement, and update the test; then re-verify that a stalled stream actually aborts.

Verified by maintainer @wenshao: 3-way merge build (main e234cf6 + PR 0ae2a54) + whole-file usage grep (watchdog next() never called) + a fake-timer unit test of the watchdog (passes in isolation) + geminiChat.test.ts on main (192/192) vs merge (191/192). Env: Darwin arm64, Node v22.22.2.

中文版(点击展开)

维护者验证 —— 本地合并构建(🔴 当前不可合并)

在 macOS(Darwin arm64,Node v22.22.2)上验证了三方合并结果(当前 origin/main e234cf6 + PR head 0ae2a54)。PR 与 main 同步(diff:新增 streamIdleWatchdog.ts + geminiChat.ts 接入,+95/-4)。两处阻塞:CI 的 Test job 红了,而且这个 watchdog 根本没被调用 —— 所以 #4177 的修复并没有生效。

1. 🔴 watchdog 接线了却从未被调用 —— 修复不生效

createStreamIdleWatchdog() 返回 { next, cleanup },其中 next(streamStep) 才是机制本身:它把每个流步骤与空闲超时做竞速,卡住就 abort。但在 geminiChat.ts 里,这个 watchdog 只是被创建(2611)、传给 processStreamResponse(2668)、尝试 cleanup(2670) —— 任何地方都没有 streamWatchdog.next(...) 调用。processStreamResponse 以 _streamWatchdog(下划线=有意未用)接收它,并直接迭代流(for await (const chunk of streamResponse)),完全绕过了 watchdog。

我确认了 watchdog 单元本身是对的(fake-timer 测试):next() 对卡住的 promise 会在超时后以 STREAM_IDLE_TIMEOUT reject 并 abort;对已 resolve 的步骤会清掉两个定时器。所以这块积木是好的 —— 只是从没接进流循环。按现状,卡死的 SSE 流不会被这个 watchdog 检测或中止。

(它能干净编译,恰恰是因为未用参数加了 _ 前缀,no-unused-vars 不报 —— 这也是它在"什么都不做"的同时还能过 lint/typecheck 的原因。)

2. 🔴 CI Test 失败 —— geminiChat.test.ts 没为新增的 abortSignal 更新

接入后会把 abortSignal: streamAbortController.signal 传进 generateContentStream 的 config。既有断言 should call generateContentStream with the correct parameters 校验的是精确 config,没有更新,于是因为多出的字段而失败。

geminiChat.test.ts
origin/main(无 PR) ✅ 192 / 192
合并结果(含 PR) ❌ 191 / 192

这与 PR 的 CI 一致(Test macOS + ubuntu: fail)。

3. ⚠️ cleanup 放在了永远不会触发的 catch 里

try {
  return this.processStreamResponse(model, streamResponse, streamWatchdog, cleanupAbortLink);
} catch (error) {
  streamWatchdog?.cleanup(); cleanupAbortLink?.(); throw error;
}

processStreamResponse 是异步生成器 —— 调用它会同步返回生成器、并不执行函数体,所以这个 catch 只能捕获同步构造期的错误(没有)。流/迭代错误发生在之后的消费阶段,根本到不了这里。净效果:cleanupAbortLink()(用于移除 linkAbortSignal 添加的 abort 监听器)在正常路径和迭代错误路径上都不会被调用 → 每条流泄漏一个 abort 监听器。cleanup 需要放进生成器内部(如 finally)或包住迭代。

4. ⚠️ 格式 + 测试

  • prettier --check packages/core/src/core/streamIdleWatchdog.ts 报告格式问题(新文件未格式化)。eslint 通过。
  • 没有为 watchdog 加测试。一个卡死流的测试(fake timers + 卡住的 next())本可以发现 watchdog 从未触发。

合并前需要

  1. 把 streamWatchdog.next(...) 接进 processStreamResponse 的流循环(把裸 for await 换成手动循环、让每次 streamResponse.next() 经过 watchdog 竞速,或把 streamResponse 包成 watchdog 生成器)。真正用上后去掉 _。
  2. 把 cleanup 移到生成器内部的 finally,让 cleanupAbortLink() / cleanup() 总会执行(当前包在生成器构造外的 try/catch 是死的)。
  3. 更新 geminiChat.test.ts 以适配新增的 abortSignal config 字段(把失败的断言改绿)。
  4. 对新文件 prettier --write;补一个 watchdog 单测(超时触发 / 定时器清理两种情形)。

结论

watchdog 这个原语设计不错、单独验证也能工作,但接入后是惰性的(从未被调用),且 PR 在 CI 上是红的(有个测试没为新的 abortSignal 更新),外加一处 dead-catch 的 cleanup 泄漏和一个未格式化的文件。🔴 请求修改 —— 把 next() 接进循环、修正 cleanup 位置、更新测试;然后再验证一次"卡住的流确实会被中止"。

维护者 @wenshao 验证:三方合并构建(main e234cf6 + PR 0ae2a54)+ 全文件用法 grep(watchdog next() 从未被调用)+ watchdog 的 fake-timer 单测(单独能工作)+ geminiChat.test.ts 在 main(192/192)对比合并(191/192)。环境:Darwin arm64,Node v22.22.2。

@aspnmy aspnmy closed this Jun 19, 2026
@aspnmy
aspnmy deleted the fix/stream-idle-watchdog-4177 branch June 19, 2026 08:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add SSE stream idle watchdog to abort hung streams (weak-network hang)

4 participants