Repository navigation
fix(sidecar): a sidecar cannot outlive a parent that died before it started - #113
Merged
Merged
Conversation
44 sidecars accumulated on one machine and had to be killed by explicit PID. Two independent mechanisms miss them, and neither is accidental. sweepStaleSidecars removes registration FILES whose process has already exited. It never signals a running process, so it cannot reclaim an orphan by construction. The self-termination guard tests process.ppid !== originalPpid every 2s, which catches a parent that dies after startup. If the parent dies BEFORE the child reads process.ppid, the child is already reparented to 1, originalPpid captures that, and the comparison is false forever. The sidecar then has no parent to signal it and no sweep that can see it. Worse in combination: sidecar-manager.ts:121 spawns before writeSidecarRegistration, so a parent exiting in that window leaves a process that is both unregistered and unreclaimable. All 44 were in exactly that state — none held a session database, and none were routable, since resolveOpencodeSender resolves only through the registry. They could not receive anything and could be addressed by nothing. The existing test at sidecar-e2e.test.ts:222 waits 1500ms for registration before killing the parent, so it covers the case the guard handles. Neither startup race is tested. Phase 0 is confirmation, not implementation: reproduce the race, establish how a sidecar loses its registration, census for other leak shapes, and decide whether the fix must also reclaim sidecars already leaked elsewhere. A fix that only prevents future leaks leaves existing ones running. The spec keeps the boot sweep's safety property explicit: files for dead processes only, never a signal to a running one. Reclaiming a live sidecar is an explicit, confirming operator action that must refuse any candidate whose session is live. No code change.
…tarted The ppid guard compares process.ppid against a value captured at startup, so it can only see a parent that dies afterwards. If the parent is already gone by the time the sidecar reads its own ppid, that value is the reparented one, the comparison is false forever, and the sidecar survives indefinitely — which is how 44 of them accumulated here and had to be killed by explicit PID. Mechanism reproduced before fixing (Bun 1.3.14/macOS): SIGKILL the spawner before the child's first ppid read and the child records ppid 1 and never notices. The parent's stdin pipe closes when the parent dies, at any point in the sidecar's life, so EOF cannot be missed by a startup race. Exit on it. The discriminator is isSocket(), not isFIFO(). Measured: a child spawned with stdio ["pipe"] reports isSocket()=true, isFIFO()=false (mode 140000), while stdio "ignore" reports isCharacterDevice()=true (/dev/null). My first attempt checked isFIFO(), which silently disabled this guard in production and matched nothing — the failing test caught it, not review. The check exists because every test spawns with stdio "ignore", where stdin is /dev/null and reads EOF immediately; an unguarded listener would kill a healthy sidecar on startup. Also bound the stop. shutdown() set shuttingDown before awaiting sidecar.stop(), so a stop() that never settled left the process alive and every later SIGTERM swallowed — killable only by SIGKILL. The exit now happens on a timer rather than on stop() settling. The new test retries rather than asserting once: hitting the race depends on bun's startup time, so an attempt where the sidecar never registered proves nothing and is discarded. It only passes having observed a registration appear and then disappear, so it cannot pass vacuously. Verified by disabling the guard — the test fails, and passes with it restored. test/peer 211 pass / 0 fail. Not addressed here: reclaiming sidecars already leaked elsewhere, and the operator command. Those are separate tasks in the change.
Two peer sessions reproduced the race independently. The second narrowed the window in a way that corrects this document: it is the duration of startSidecar, not an abstract read of process.ppid. Registration is written from inside startSidecar (sidecar-server.ts:156) before originalPpid is captured (sidecar-entry.ts:93), which splits the window in two — alive-and-unregistered before the write, alive-and-registered after it. Both are real; the sweep skips both. All 44 observed were the first. Also records that the discriminator is isSocket() and not isFIFO(), because an isFIFO check reads as correct, matches nothing in production, and would have left the guard dead. That was caught by the failing test rather than by review, which is the only reason it is written down here. Phase 1 is done in 7efff55; Phase 0's reproduction and registration questions are answered. Reclamation of already-leaked sidecars is untouched.
|
Thanks for your contribution! This PR doesn't have a linked issue. All PRs must reference an existing issue. Please:
See CONTRIBUTING.md for details. |
|
This PR doesn't fully meet our contributing guidelines and PR template. What needs to be fixed:
Please edit this PR description to address the above within 2 hours, or it will be automatically closed. If you believe this was flagged incorrectly, please let a maintainer know. |
… exits Reviewer was right, and the criticism was of the same kind I have been making of other checks: the test proved the sidecar terminates, not that the EOF guard terminates it. If the parent dies after the child reads process.ppid, the pre-existing ppid guard kills the sidecar anyway, so removing the EOF guard left the test green. It was a real mutation check on paper and not one in fact. Replaced with a deterministic test. The parent is this test process and stays alive for the whole test, so process.ppid never changes and the ppid guard cannot fire. Closing only the stdin pipe gives the child EOF and nothing else, so if the sidecar exits and unregisters, the EOF guard is the only possible cause. Verified: with the guard disabled this test fails, and only it fails. Removed the timing-based test. With the fix in place the sidecar now dies so fast after its parent exits that the test could no longer reliably observe the registration it was asserting on, and the 4-attempt retry was passing for the wrong reason or failing for a timing reason. Hitting that race depends on bun's startup time and cannot be made deterministic; the new test isolates the mechanism without depending on it at all. Suite went from 15s and flaky to 1.7s and stable. test/peer 211 pass / 0 fail. Typecheck clean, openspec validate passes, fork:verify 251/251 owned, 117/117 patched.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
44 sidecar processes accumulated on one machine over ~16 hours and had to be reclaimed by explicit PID. They were all
bun.exe run <deleted-worktree>/… debug claude-sidecar-entrywithppid 1, sleeping, holding no session database — and unregistered, soresolveOpencodeSendercould not resolve them and no peer could address them at all. Pure overhead.Two independent mechanisms miss them, and neither is accidental:
sweepStaleSidecars(sidecar-registry.ts:98-126) removes registration files whose process has already exited. It never signals a live process, so it cannot reclaim an orphan.sidecar-entry.ts:93comparesprocess.ppidagainst a value captured at startup, so it only sees a parent that dies afterwards.Root cause
The blind window is the duration of
startSidecar, not an abstract read ofprocess.ppid:sidecar-entry.ts:37—await startSidecar(...), async, writes the registration from insidesidecar-server.ts:156sidecar-entry.ts:93—originalPpid = process.ppidsidecar-entry.ts:94— the poll interval startsA parent dying anywhere inside that window leaves
originalPpidreading as 1, the comparison1 !== 1false forever. Two sub-cases, distinguished by whether the registration reached disk:Reproduced before fixing (Bun 1.3.14/macOS): SIGKILL the spawner before the child's first ppid read and the child records ppid 1 and never notices. Two peer sessions reproduced it independently; one narrowed the window to the above, which corrected my original description.
Fix
Exit on stdin EOF. The parent's pipe closes when the parent dies, at any point in the sidecar's life, so this covers both sub-windows — including the one where
originalPpidis captured already-reparented. The ppid guard stays as a second net.The discriminator is
isSocket(), notisFIFO(). Measured on Bun 1.3.14: a child spawned withstdio: ["pipe"]reportsisSocket()=true, isFIFO()=false(mode 140000), whilestdio: "ignore"reportsisCharacterDevice()=true(/dev/null). My first attempt checkedisFIFO(), which reads as correct, matches nothing in production, and leaves the guard dead — caught by the failing test, not by review. The check exists at all because every test spawns withstdio: "ignore", where stdin reads EOF immediately and an unguarded listener would kill a healthy sidecar on startup.Bound
stop().shutdown()setshuttingDownbefore awaitingsidecar.stop(), so a stop that never settled left the process alive with every later SIGTERM swallowed — killable only by SIGKILL. The exit now happens on a timer rather than onstop()settling.Verification
test/peer211 pass / 0 fail. Typecheck clean.openspec validatepasses.fork:verify251/251 owned, 117/117 patched — and it correctly failed on the new test file until it was committed.Not in this PR
Reclaiming sidecars already leaked elsewhere, and the operator command to do it — separate tasks in the change. Prevention alone does not reclaim existing leaks; both peers agreed on that split, and the boot sweep stays non-signalling.
One path is mitigated but not confirmed:
shutdown()'sstop()never settling was reported by reading, not reproduced. Treat the timer as mitigation rather than a proven fix.🤖 Generated with Claude Code