fix(app): recover stale event streams — stall watchdog, foreground resync, reconnect backoff - #51871

The following comment was made by an LLM, it may be inaccurate:

cc @thdxr @adamdotdevin @kitlangton — requesting review when you have a moment.

Why this PR: the web UI's SSE client had no liveness detection after the v2 refactor removed the stall watchdog that #47571 introduced — a half-open connection leaves the app stale until a hard refresh (daily pain behind reverse proxies; tracked in #39030, #47258, #45860, #51857). This restores that watchdog in server-sdk.tsx, plus foreground resync (visibilitychange/online), an always-restart on pageshow, and reconnect backoff (#48014).

Scope is deliberately narrow: one file's recovery loop + its test file; no server changes, no contract changes — it consumes the existing server.heartbeat.

Verified locally: 14/14 unit tests pass, tsgo typecheck clean, no new lint warnings.

Happy to split the backoff into a separate PR if you'd rather land #48015 first — the two changes touch the same loop, so whichever merges second will need a small rebase either way.

Why this fix matters

This is currently the single largest source of "dead" web UI reports.

The failure class this PR closes — an SSE stream that dies silently and never recovers — accounts for a long chain of user-visible reports, and it keeps recurring because every previous fix addressed only one symptom instead of stream liveness itself:

• Mobile browser tab does not reconnect SSE stream after returning from another app #39030, Event stream (SSE) doesn't resume after tab is backgrounded and restored — requires manual page refresh #47258, Web UI (mobile): session status goes stale after tab suspension — spinner stays 'thinking' until full page reload #45860 — tab suspended / app switched / screen locked → UI frozen, only Ctrl+F5 recovers. This is a daily defect for anyone running opencode web/serve remotely or using it on mobile.
• [BUG] SSE connection becomes stale after laptop sleep/wake — no detection or recovery #21716, SSE /global/event and /event deliver only server.connected + heartbeats — no message events reach any subscriber (1.18.25) #46733, SSE stream closes mid-turn (not persistent as documented) #38458 — same invisible-failure family across releases.
• fix(client): detect stalled event streams and resync on foreground #47571 already proved the fix shape (stall watchdog + foreground resync) — it was merged, worked, and was silently lost when packages/client/src/solid/connection.ts was deleted in the v2 refactor. This is the second time this bug class has been fixed; landing it in the current code path with a heartbeat invariant test is what stops it from regressing a third time.

Impact it removes:

1. Silent data staleness — users see a UI that looks live but is not. Messages arrive server-side, the page just never shows them. That is the worst kind of bug: it erodes trust in correctness, not just responsiveness.
2. Reconnect storms — the fixed 250ms retry loop meant each reconnect re-bootstraps every open project directory, and a flapping connection aborts each bootstrap with the next reconnect (self-inflicted HTTP 499s). Exponential backoff converts a sustained storm into a handful of attempts.
3. Lost multi-session reliability — behind reverse proxies/tunnels (nginx, Cloudflare Tunnel), idle connections are dropped without TCP RST routinely. Every such deployment is disproportionately affected.

Cost/risk of the change is minimal: ~75 lines in one file, no server or API contract changes, driven entirely by events already on the wire. The unit tests cover the new invariants (reconnectDelay growth/cap/jitter bounds, stall timeout > 3 heartbeats). In exchange it retires the most-reported web reliability complaint in the tracker.

If it helps review: the watchdog + backoff + resync are separable — happy to split if that gets parts merged faster.

cc @thdxr @adamdotdevin — adding implementation detail for reviewers.

Exact behavior changes in server-sdk.tsx:

1. armStallWatchdog() is created per attempt and re-armed inside the for await loop on every event — heartbeats included. After STREAM_STALL_TIMEOUT_MS (45s = >3× the 10s server heartbeat) with zero events, it sets stalled and aborts the attempt's AbortController. The catch branch logs event stream stalled once per incident (same once-flag pattern as the existing streamErrorLogged) and the loop proceeds to a fresh connect — this is the watchdog half of what fix(client): detect stalled event streams and resync on foreground #47571 did in packages/client/src/solid/connection.ts.
2. resync(reason) aborts the in-flight attempt (no-op if none is in flight — e.g. between reconnects, the loop is already on its way). Triggered by visibilitychange→visible and window.online — the two browser signals that fire exactly when a half-open SSE fetch is most likely.
3. pageshow now calls start() unconditionally. The previous persisted-only check was effectively dead code for this app: an open streaming fetch makes the page ineligible for bfcache, so persisted is almost always false when pagehide had killed the stream. start() is idempotent (if (started) return run), so double-fires are safe.
4. Reconnect delay: reconnectDelay(consecutiveFastDrops) — 250ms base doubling per fast drop (<3s connected), capped at 10s, plus jitter bounded by min(backoff, 250ms). A connection that survives ≥ STABLE_CONNECTION_MS resets the counter, so a single hiccup on a healthy link still reconnects fast.

Why heartbeat-based detection is safe: the server emits server.heartbeat unconditionally every 10s on /event — see handlers/event.ts. 45s of silence therefore means the connection is dead with very high confidence; abort+reconnect is strictly better than the previous behavior (wait forever).

14/14 unit tests pass (new: delay growth 250→500→1000→2000, 10s cap, jitter bounds, stall>3 heartbeats); typecheck and lint clean.

Reviewed the diff in detail — the recovery semantics are right and the scope discipline (one file, no contract changes, reuses the existing 10s server.heartbeat) is exactly what makes this landable. A few findings:

Consider — visibilitychange/online resync is unconditional. Every tab foreground now aborts the attempt → reconnect → server.connected → full re-bootstrap of every open project directory. A user who alt-tabs frequently generates the same self-inflicted bootstrap churn the backoff is meant to dampen. Suggest gating on staleness so a healthy stream isn't torn down on every focus:

let lastEventAt = 0
// inside the for-await loop, next to armStallWatchdog():
lastEventAt = Date.now()

makeEventListener(document, "visibilitychange", () => {
  if (document.visibilityState !== "visible") return
  if (Date.now() - lastEventAt < STREAM_STALL_TIMEOUT_MS) return
  resync("Page returned to the foreground")
})

online can keep the same gate — window.online fires on any network flap, not just recovery from a dead connection, so a healthy stream shouldn't pay a re-bootstrap for it either. The watchdog remains the backstop for the genuinely half-open case; this just avoids paying a full re-bootstrap on every foreground when the stream is fine.

Nit — only the pure reconnectDelay is tested. The interesting invariants (watchdog aborts after STREAM_STALL_TIMEOUT_MS of silence, resync on visible, pageshow restart) are untested — understandable since they live inside the closure, but a fake-timer test around the loop would catch a future regression where someone accidentally drops the re-arm inside for await. Non-blocking.

FYI — stalled reset ordering is correct. stalled = false at attempt start plus clearTimeout(stallTimer) in finally means a stale watchdog can't leak across attempts; single-threaded timers can't race the synchronous sections. Just confirming I traced it.

Verdict: approve modulo the resync gating — that's the only behavioral surprise I'd want addressed or explicitly justified.

cc @Hona @Brendonovich — could you review and approve when you get a chance? This closes the largest remaining "dead UI" failure class (SSE half-open after tab suspend / proxy drop), restoring the watchdog semantics #47571 had before the v2 refactor.