Skip to content

fix(opencode): chunkTimeout ignores SSE comment heartbeats that keep stalled streams warm - #51879

Open
afonsoft wants to merge 2 commits into
anomalyco:devfrom
afonsoft:sse-keepalive-stall
Open

afonsoft wants to merge 2 commits into
anomalyco:devfrom
afonsoft:sse-keepalive-stall

Conversation

@afonsoft

Copy link
Copy Markdown

Issue for this PR

Closes #43519. Related: #51857 (client-side SSE watchdog, PR #51871) — this is the provider-side counterpart; the same stall class exists on both SSE boundaries.

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

wrapSSE (provider fetch wrapper for chunkTimeout) re-armed its read timeout on any chunk. SSE comment lines (: keepalive) carry no data: event — the SSE spec explicitly allows them for keep-alive — so a provider or LB that holds a stalled generation open with comment heartbeats keeps the timer resetting forever. Result, per #43519: opencode run hangs indefinitely — no stdout event, no log line, no error, until an external watchdog kills it.

The fix replaces the per-read timer with a stall deadline that only re-arms on SSE field progress:

  • A tiny sseFieldScanner inspects each chunk's line heads: a line starting with : (comment), \n/\r (blank) is not progress; any other line start (data:, event:, id:, retry:, any field) is.
  • Line-head state is tracked across chunk boundaries, so a data: marker split between two reads still counts, and a comment split the same way still doesn't.
  • The deadline fires the same typed, retryable error as before — ProviderError.ResponseStreamError("SSE read timed out") (v1) / Error("SSE read timed out") (core v2) — so SessionRetry classification and provider retry behavior are unchanged.
  • Byte-silence is still covered: no bytes → no field lines → deadline fires, same as before.

Applied identically to both copies of wrapSSE: packages/opencode/src/provider/provider.ts (v1 runtime) and packages/core/src/aisdk.ts (v2 SDK path).

How did you verify your code works?

New regression test in packages/opencode/test/provider/header-timeout.test.ts: a server sends one partial SSE event, then streams : keepalive\n\n every 10ms forever. With chunkTimeout: 100, result.fullStream now surfaces ProviderError.ResponseStreamError ("SSE read timed out") — previously it hung forever.

(pass) chunkTimeout fires while a stalled stream is kept warm by comment heartbeats [902.91ms]
(pass) configured chunkTimeout raises a retryable response stream error when SSE body stalls
(pass) chunkTimeout can be disabled with false
10 pass / 3 fail — the 3 failures are pre-existing environment flakes
(delayed-body timing + cloudflare-ai-gateway URL capture) that fail
identically on a clean baseline of origin/dev.

bun typecheck (tsgo) clean; oxlint on touched files: 0 errors, no new warnings.

Checklist

  • I have read the contributing docs
  • Regression test added using the file's own live-server fixture
  • Same error type/message — retryability and error reporting unchanged
  • Both duplicated wrapSSE call sites updated (v1 + v2 runtime paths)

wrapSSE re-armed the read timeout on any chunk, including SSE comment
lines (`: keepalive`). Providers and load balancers that hold a stalled
generation open with comment heartbeats kept the connection byte-warm
forever: no event ever arrived, the timeout never fired, and the
session hung with no error (anomalyco#43519).

Replace the per-read timer with a stall deadline that only re-arms when
a chunk contains the start of a real SSE field line (data:, event:,
id:, retry:). A tiny line-head scanner tracks progress across chunk
boundaries without parsing the stream. The deadline still fires the
same typed, retryable timeout error (ResponseStreamError /
"SSE read timed out") and still covers fully-silent connections.

Applied identically to the v1 provider wrapper (packages/opencode) and
the v2 SDK wrapper (packages/core/aisdk.ts).

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

The following comment was made by an LLM, it may be inaccurate:

@afonsoft

afonsoft commented Sep 28, 2026 •

Copy link
Copy Markdown
Author

cc @thdxr @adamdotdevin — requesting review when you get a chance.

Fix details:

The bug is subtle: wrapSSE re-armed its timeout inside each reader.read() promise, so any bytes — including SSE comment lines (: keepalive) that the spec explicitly reserves for connection keep-alive — reset the clock. A provider or LB that holds a stalled generation open with comment heartbeats produces an infinite warm-but-dead stream: no event, no error, no log line (#43519 has a deterministic repro).

The fix swaps the per-read timer for a single data-stall deadline that only re-arms when a chunk contains the start of a non-comment SSE line:

  • sseFieldScanner() checks each line's head byte — :/\n/\r are not progress, anything else is. Because it tracks line-head state across chunk boundaries, a data: marker split across two reads still counts (and a comment split the same way still doesn't).
  • Byte-level silence is still covered: no bytes → no field lines → deadline fires.
  • Error type, message and retryability are unchanged: ResponseStreamError("SSE read timed out") (v1) / Error("SSE read timed out") (v2 aisdk.ts).

One semantic difference worth noting in review: while consumer backpressure pauses pull(), the deadline keeps running — if it expires mid-backpressure the abort lands via ctl.abort + reader.cancel and surfaces on the next read. Given the default chunkTimeout is 300s, a consumer stalled that long was already a candidate for the timeout.

New regression test reproduces the keepalive-warm stall end-to-end (data event → : keepalive every 10ms, chunkTimeout: 100): now errors in ~900ms instead of hanging forever. The 3 remaining failures in that file are pre-existing env flakes (identical on baseline origin/dev).

Both wrapSSE copies updated — packages/opencode/src/provider/provider.ts (v1) and packages/core/src/aisdk.ts (v2).

@afonsoft

afonsoft commented Oct 3, 2026

Copy link
Copy Markdown
Author

Reviewed — this is the right fix for #43519 and the sseFieldScanner approach is nicely minimal: line-head inspection catches the comment-heartbeat case without parsing full events, and tracking atLineHead across chunk boundaries handles split markers correctly. Findings:

Consider — continuation bytes of a field line don't count as progress. The scanner only credits a line's head. A single data: line longer than one read — a large payload streamed slowly — emits no further progress until its \n, so a generation streaming one huge line over >chunkTimeout will abort mid-line even though bytes are flowing. With the 300s default this is near-impossible; with aggressive chunkTimeout configs it could bite. Cheap hardening if you want it: keep a lineIsField flag and count any byte consumed while inside a non-comment line as progress (comment lines still don't count, so the keepalive bypass stays closed).

FYI — event-level heartbeats still keep the deadline warm. Providers/LBs that heartbeat with real SSE events — Anthropic-style event: ping + data: {"type": "ping"} — re-arm the deadline forever, so a generation stalled-but-pinging still hangs. Closing that requires parsing the event type, not just line heads — legitimately out of scope here, but worth a follow-up issue so the remaining gap is tracked.

Verified the backpressure trade-off the author flagged: the deadline keeps running while the consumer pauses pull(), so a >300s stall downstream now kills the stream rather than waiting as before. Agree that's acceptable — a consumer stalled that long was already pathological, and ctl.abort + reader.cancel surface the error on the next read rather than losing it.

Verified both copies are identical (provider.ts v1 / aisdk.ts v2) — same scanner, same error type and message, so retry classification is unchanged on both paths.

Verdict: approve — closes the deterministic infinite-hang repro with minimal machinery and no contract change.

cc @Hona @Brendonovich — could you review and approve? Provider-side counterpart to the client watchdog in #51871; together they close the silent-stall class on both SSE boundaries.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

chunkTimeout resets on SSE comment heartbeats — keepalive-warm stalled streams never time out

1 participant