Skip to content

fix(sidecar): a sidecar cannot outlive a parent that died before it started - #113

Merged
androidand merged 4 commits into
devfrom
sidecar-orphan-cleanup
Oct 3, 2026
Merged

androidand merged 4 commits into
devfrom
sidecar-orphan-cleanup

Conversation

@androidand

Copy link
Copy Markdown
Owner

Problem

44 sidecar processes accumulated on one machine over ~16 hours and had to be reclaimed by explicit PID. They were all bun.exe run <deleted-worktree>/… debug claude-sidecar-entry with ppid 1, sleeping, holding no session database — and unregistered, so resolveOpencodeSender could not resolve them and no peer could address them at all. Pure overhead.

Two independent mechanisms miss them, and neither is accidental:

  1. The sweep cannot, by design. sweepStaleSidecars (sidecar-registry.ts:98-126) removes registration files whose process has already exited. It never signals a live process, so it cannot reclaim an orphan.
  2. The self-termination guard has a startup blind spot. sidecar-entry.ts:93 compares process.ppid against a value captured at startup, so it only sees a parent that dies afterwards.

Root cause

The blind window is the duration of startSidecar, not an abstract read of process.ppid:

  • sidecar-entry.ts:37 — await startSidecar(...), async, writes the registration from inside sidecar-server.ts:156
  • sidecar-entry.ts:93 — originalPpid = process.ppid
  • sidecar-entry.ts:94 — the poll interval starts

A parent dying anywhere inside that window leaves originalPpid reading as 1, the comparison 1 !== 1 false forever. Two sub-cases, distinguished by whether the registration reached disk:

  • dies before the registration write → alive, unregistered: invisible to the sweep, unroutable
  • dies after it, before line 93 → alive, registered, still skipped by the sweep

Reproduced before fixing (Bun 1.3.14/macOS): SIGKILL the spawner before the child's first ppid read and the child records ppid 1 and never notices. Two peer sessions reproduced it independently; one narrowed the window to the above, which corrected my original description.

Fix

Exit on stdin EOF. The parent's pipe closes when the parent dies, at any point in the sidecar's life, so this covers both sub-windows — including the one where originalPpid is captured already-reparented. The ppid guard stays as a second net.

The discriminator is isSocket(), not isFIFO(). Measured on Bun 1.3.14: a child spawned with stdio: ["pipe"] reports isSocket()=true, isFIFO()=false (mode 140000), while stdio: "ignore" reports isCharacterDevice()=true (/dev/null). My first attempt checked isFIFO(), which reads as correct, matches nothing in production, and leaves the guard dead — caught by the failing test, not by review. The check exists at all because every test spawns with stdio: "ignore", where stdin reads EOF immediately and an unguarded listener would kill a healthy sidecar on startup.

Bound stop(). shutdown() set shuttingDown before awaiting sidecar.stop(), so a stop that never settled left the process alive with every later SIGTERM swallowed — killable only by SIGKILL. The exit now happens on a timer rather than on stop() settling.

Verification

  • New test observed red with the guard disabled, green with it restored.
  • The race test retries rather than asserting once: hitting the window depends on bun's startup time, so an attempt where the sidecar never registered proves nothing and is discarded. It only passes having observed a registration appear and then disappear, so it cannot pass vacuously.
  • test/peer 211 pass / 0 fail. Typecheck clean. openspec validate passes. fork:verify 251/251 owned, 117/117 patched — and it correctly failed on the new test file until it was committed.

Not in this PR

Reclaiming sidecars already leaked elsewhere, and the operator command to do it — separate tasks in the change. Prevention alone does not reclaim existing leaks; both peers agreed on that split, and the boot sweep stays non-signalling.

One path is mitigated but not confirmed: shutdown()'s stop() never settling was reported by reading, not reproduced. Treat the timer as mitigation rather than a proven fix.

🤖 Generated with Claude Code

44 sidecars accumulated on one machine and had to be killed by explicit PID. Two
independent mechanisms miss them, and neither is accidental.

sweepStaleSidecars removes registration FILES whose process has already exited. It
never signals a running process, so it cannot reclaim an orphan by construction.

The self-termination guard tests process.ppid !== originalPpid every 2s, which
catches a parent that dies after startup. If the parent dies BEFORE the child reads
process.ppid, the child is already reparented to 1, originalPpid captures that, and
the comparison is false forever. The sidecar then has no parent to signal it and no
sweep that can see it.

Worse in combination: sidecar-manager.ts:121 spawns before writeSidecarRegistration,
so a parent exiting in that window leaves a process that is both unregistered and
unreclaimable. All 44 were in exactly that state — none held a session database, and
none were routable, since resolveOpencodeSender resolves only through the registry.
They could not receive anything and could be addressed by nothing.

The existing test at sidecar-e2e.test.ts:222 waits 1500ms for registration before
killing the parent, so it covers the case the guard handles. Neither startup race
is tested.

Phase 0 is confirmation, not implementation: reproduce the race, establish how a
sidecar loses its registration, census for other leak shapes, and decide whether
the fix must also reclaim sidecars already leaked elsewhere. A fix that only
prevents future leaks leaves existing ones running.

The spec keeps the boot sweep's safety property explicit: files for dead processes
only, never a signal to a running one. Reclaiming a live sidecar is an explicit,
confirming operator action that must refuse any candidate whose session is live.

No code change.
…tarted

The ppid guard compares process.ppid against a value captured at startup, so it
can only see a parent that dies afterwards. If the parent is already gone by the
time the sidecar reads its own ppid, that value is the reparented one, the
comparison is false forever, and the sidecar survives indefinitely — which is
how 44 of them accumulated here and had to be killed by explicit PID. Mechanism
reproduced before fixing (Bun 1.3.14/macOS): SIGKILL the spawner before the
child's first ppid read and the child records ppid 1 and never notices.

The parent's stdin pipe closes when the parent dies, at any point in the sidecar's
life, so EOF cannot be missed by a startup race. Exit on it.

The discriminator is isSocket(), not isFIFO(). Measured: a child spawned with
stdio ["pipe"] reports isSocket()=true, isFIFO()=false (mode 140000), while
stdio "ignore" reports isCharacterDevice()=true (/dev/null). My first attempt
checked isFIFO(), which silently disabled this guard in production and matched
nothing — the failing test caught it, not review. The check exists because every
test spawns with stdio "ignore", where stdin is /dev/null and reads EOF
immediately; an unguarded listener would kill a healthy sidecar on startup.

Also bound the stop. shutdown() set shuttingDown before awaiting sidecar.stop(),
so a stop() that never settled left the process alive and every later SIGTERM
swallowed — killable only by SIGKILL. The exit now happens on a timer rather than
on stop() settling.

The new test retries rather than asserting once: hitting the race depends on bun's
startup time, so an attempt where the sidecar never registered proves nothing and
is discarded. It only passes having observed a registration appear and then
disappear, so it cannot pass vacuously. Verified by disabling the guard — the test
fails, and passes with it restored. test/peer 211 pass / 0 fail.

Not addressed here: reclaiming sidecars already leaked elsewhere, and the operator
command. Those are separate tasks in the change.
Two peer sessions reproduced the race independently. The second narrowed the
window in a way that corrects this document: it is the duration of
startSidecar, not an abstract read of process.ppid. Registration is written from
inside startSidecar (sidecar-server.ts:156) before originalPpid is captured
(sidecar-entry.ts:93), which splits the window in two — alive-and-unregistered
before the write, alive-and-registered after it. Both are real; the sweep skips
both. All 44 observed were the first.

Also records that the discriminator is isSocket() and not isFIFO(), because an
isFIFO check reads as correct, matches nothing in production, and would have left
the guard dead. That was caught by the failing test rather than by review, which
is the only reason it is written down here.

Phase 1 is done in 7efff55; Phase 0's reproduction and registration questions
are answered. Reclamation of already-leaked sidecars is untouched.
@github-actions

github-actions Bot commented Oct 3, 2026

Copy link
Copy Markdown

Thanks for your contribution!

This PR doesn't have a linked issue. All PRs must reference an existing issue.

Please:

  1. Open an issue describing the bug/feature (if one doesn't exist)
  2. Add Fixes #<number> or Closes #<number> to this PR description

See CONTRIBUTING.md for details.

@github-actions

github-actions Bot commented Oct 3, 2026

Copy link
Copy Markdown

This PR doesn't fully meet our contributing guidelines and PR template.

What needs to be fixed:

  • PR description is missing required template sections. Please use the PR template.

Please edit this PR description to address the above within 2 hours, or it will be automatically closed.

If you believe this was flagged incorrectly, please let a maintainer know.

… exits

Reviewer was right, and the criticism was of the same kind I have been making
of other checks: the test proved the sidecar terminates, not that the EOF guard
terminates it. If the parent dies after the child reads process.ppid, the
pre-existing ppid guard kills the sidecar anyway, so removing the EOF guard left
the test green. It was a real mutation check on paper and not one in fact.

Replaced with a deterministic test. The parent is this test process and stays
alive for the whole test, so process.ppid never changes and the ppid guard cannot
fire. Closing only the stdin pipe gives the child EOF and nothing else, so if the
sidecar exits and unregisters, the EOF guard is the only possible cause.
Verified: with the guard disabled this test fails, and only it fails.

Removed the timing-based test. With the fix in place the sidecar now dies so fast
after its parent exits that the test could no longer reliably observe the
registration it was asserting on, and the 4-attempt retry was passing for the
wrong reason or failing for a timing reason. Hitting that race depends on bun's
startup time and cannot be made deterministic; the new test isolates the
mechanism without depending on it at all. Suite went from 15s and flaky to 1.7s
and stable.

test/peer 211 pass / 0 fail. Typecheck clean, openspec validate passes,
fork:verify 251/251 owned, 117/117 patched.
@androidand
androidand merged commit 224bcc4 into dev Oct 3, 2026
2 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant