feat(managed-agent): Hosted Turn takeover and G1 failover E2E - #13083

Please do not rebase or force-push to an active PR as it invalidates existing review comments. Note for future reference, the bots always squash all changes into a single commit automatically as part of the integration.

Real-environment verification of 13cbd974

Maintainer-side verification, meant as input for the merge decision. I built the branch and ran the packaged stack locally: the Spring fat jar, the dist/cli.js Hosted Harness, the Runtime Broker with durable local workers and a real MySQL, on Linux (Ubuntu 24.04 container in a dedicated VM) and on macOS. The model is a scripted local OpenAI-compatible server, so every scenario is repeatable.

Result: the takeover works, I could not make it run a tool twice, and the regression on ordinary Turns that the last review found is fixed on this head. Two defects remain on the takeover path itself. Neither makes anything worse than main, but both are small to fix and I would fix them before merging.

A candidate patch for F2 and F3 (4 files, +78 −1) is linked at the end. I ran it through the same scenarios.

What was built

• The head moved four times while I was testing: eb06f7f9 → 1606fe07 (rebase onto feat(managed-agent): Add durable remote Shell result delivery #12894) → 5e81d704 → fcd2dc2c (rebase onto feat(managed-agent): Implement private Hosted MCP runtime (H1) #12946) → 13cbd974 (review fixes). I built eb06f7f9, 1606fe07, fcd2dc2c and 13cbd974.
• Everything below is from 13cbd974 unless a head is named. The mutants and the quiet-host timings come from earlier heads and are marked.
• A/B arms: base = main at 3b18cfe5, the previous head fcd2dc2c, the current head, the current head plus the candidate. I did not rebuild the base arm at the current merge base 7827a3ff.
• Linux: Ubuntu 24.04.5, kernel 6.8, arm64, JDK 21.0.12, Node 24.18.1, private mysqld 8.0.46. macOS: arm64, JDK 26, Node 24.18.1, Homebrew mysqld.
• The host was busy with unrelated builds for most of the run (load average 40 to 220 on 10 cores). Where that affected a result I say so.

1. The central claim holds

• --inflight-failover, --continuation-failover and --session-failover pass on Linux: 5/5, 5/5 and 2/2 on 13cbd974. On earlier heads they passed 10/10, 10/10 and 3/3, once on a quiet host and once under load. One earlier pass at load 60 to 170 had 7/10 and 2/3; each of those runs stopped on The Managed Session writer grant is stale, the runner's 1 s writer lease expiring on a starved host, and --session-failover, which this PR does not change, failed the same way. CI on 13cbd974 ran both new modes green.
• I re-implemented both scenarios in a separate driver with its own oracles, because the runner prints promptReplayed: false and physicalToolExecutions: 1 as literals and infers them from row counts and file bytes. An inotify watcher on the Workspace mount, a ledger of every Harness→Broker call and the file's inode and timestamps agree with the runner. In-flight executes once, on the replacement, under the original executionCallId. Continuation executes nothing (no Broker call, file untouched), blanks the dead owner's chunk and publishes only the replacement's answer.
• Removing pieces makes the modes fail, so they are not passing by accident (section 6).
• On the base commit both modes exit at once with the stale "not yet enabled" error, as the description says.
• F6. The runner generates the Broker token as base64url and passes it to qwen serve as a separate argv word. 1.5% of such tokens start with -; qwen serve then prints its usage and exits 0, and the mode fails before the scenario starts. That is about 1.5% of runs of each new mode and about 3% of CI jobs. --managed-runtime-broker-token=<value> or a hex token avoids it. My own driver had copied the pattern and hit it once.

2. F1 is fixed

No crash is involved. One Session, one Spring, one Harness.

Linux with MySQL 8.0 agrees: Turn 2 failed after 32.1 s on the earlier rebase 1606fe07 and completes in 0.5 s on 13cbd974, and the cut Turn completes 1.3 s after the cut. This is the defect doudouOUC raised inline on fcd2dc2c from a route-level reproduction of the first row; the second row is the stream-reconnect half that the same comment anticipated. QwenHostedHarnessConnector.recoverManagedRuntime now reuses the cached attachment, and both rows are back to base behaviour.

3. Faults on the replacement owner

In every case the execution row stayed single and SETTLED at generation 1, the file was written once or not at all, and the Prompt was never sent to the model a second time. What differs is whether the Turn finishes.

• Takeover load lost before it reaches the Harness: completes on the retry.
• First :start lost before it reaches the Broker: completes, 124 s later. The load waits out the 120 s observation window (2,137 status polls), the coordinator's request times out, and the next load starts the execution under the same id. The release that this head adds on that failure exit is refused by the Broker (409 runtime_session_busy, the execution is still pending), so the lease is kept until the retry finishes the Turn. The default request-timeout is 30 s; I did not run that combination.
• F3. A taken-over Turn that is re-entered does not finish. Two triggers, same cause.
  • Reply to continue lost. The Harness ran the continuation exactly once. The coordinator retried, got the cached attachment together with its old recovery snapshot, and sent continue again. The Turn had already settled, so settledReplay (hosted-harness-session.ts:1388) answered with the current sequence, 30, where the admission had been at 18. The coordinator stored 30 as its cursor and opened the stream after the Turn's own events. It never saw the answer or the terminal event: the Turn stays RUNNING and the public transcript has no text.
  • Event stream cut while the replacement is answering. The coordinator re-entered the Turn and retracted the output of the current epoch, which by then is the replacement's own first chunk. It sent continue again and got the watermark 31, where the admission had been at 15. The public transcript ends with two blank text events and no terminal event.
  • Two things combine. The connector hands out the recovery snapshot of the takeover load on every later call (QwenHostedHarnessConnector.java:260), so a Turn whose continuation is already admitted is retracted and continued again. And the continue route does not remember its admission the way the prompt route does, so a replay returns the wrong watermark. The candidate fixes both and both scenarios complete.
• F4. Reply to the takeover load lost: never finishes, on this head and with the candidate. The Harness has settled the execution and attached the Session, and refuses every later load with 409 hosted_session_already_attached. Nothing is executed twice, but the recovery snapshot cannot be fetched again. This needs an idempotent takeover load on the Harness side; I did not write that. Note that the load can legitimately take up to 120 s while the default request-timeout is 30 s.
• Owner dies in the first model round, text only: the Turn stays RUNNING (409 hosted_turn_recovery_required on every retry, no bound) and the dead owner's partial text stays public. This answers the question in the first review. The description scopes it out, and before this PR the Turn was stuck as well, but without visible partial text.
• A public agent.session.cancel on a parked Workspace Turn returns 409 workspace_unavailable, so the managed-runtime/cancel route cannot be reached through the public API yet. It is covered by unit tests only. I did not test the cancellation changes made on this head.

4. F2: the Workspace stays locked after a continuation takeover

After the taken-over Turn ended I created a second Session on the same Workspace and asked for one write_file.

When the tool had already settled before the crash, recoverHostedRuntimeTurn has nothing to drive, never calls acquire, and so runtimeLeaseHeld is false and the terminal route releases nothing. The lease row has no expiry. --continuation-failover does not look at the lease, so it passes. This is the continue-path sibling of the cancel-path lease finding that this head addresses. Releasing without re-attaching does not work on this path (the Broker answers 503 runtime_reconciliation_required; I tried that on the first head); the candidate re-attaches first.

5. Trusted-actor header, rollback, streaming cost

• Trusted-actor header. With the property unset every request behaves as on the base build: the header is ignored and a bound create is 401 actor_required. With it set, a granted actor gets 202, an actor without a grant or from another tenant gets 404 workspace_not_found, a blank value gets 401, and unbound Sessions are unaffected. It replaces a missing principal and does not bypass the grant check. The matrix is identical before and after this head's change that lets an existing principal win; the packaged server has no other source of a principal, so I could not exercise that branch.
• F5, rollback. A Workspace-bound Session whose journal contains message.delta cannot be opened by a Harness of the base build: 503 managed_session_open_failed. The control (an unbound Session, which writes no deltas) opens on the base build, and the bound Session opens on a PR-build Harness. The description says "Breaking changes / migration notes: none". That is true for readers of this build, not for a rollback or a mixed fleet during a rolling deploy.
• Cost of durable deltas. One journal transaction and about 1.9 KB of journal per model chunk, regardless of chunk size. On a quiet host (first head) a Workspace-bound Turn took 2.4 s, 5.2 s and 15.7 s for 200, 1,000 and 3,000 chunks, about 5 ms per chunk; the same answers on an unbound Session took 0.3 to 0.4 s. On the loaded host the bound figures were two to three times higher. First visible text is as fast as before. This matches the tradeoff named in the description; with a database that is not on localhost the per-chunk round trip sets the ceiling on streaming speed.

6. What the tests pin

I compiled seven mutants into the bundle (one into the jar) on 1606fe07 and ran the five vitest files this PR changes, and both modes, against each. Under load a whole-file vitest run had one or two unrelated failures that pass alone, so a mutant counts as caught only when a test also fails alone.

• With the takeover switched off in the Harness, both modes fail. This is the A/B the first review asked for.
• The text de-duplication and the durable deltas of a live Turn are pinned only by --continuation-failover, which runs on Linux only. Re-checked on 13cbd974: its 127 unit tests still pass with either one removed.
• Removing the coordinator's retraction is caught by HarnessCoordinatorTest and by the continuation mode, where the public text becomes CONTINUATION_PARTIALCONTINUATION_TURN_RECOVERED.
• Never releasing the recovered Runtime Session is caught by one unit test, for the in-flight case, and by neither mode. No test covers the continuation case, which is F2.

7. Earlier review points

• Six red fault gates, and the two modes not running in CI: resolved. CI on 13cbd974: Hosted ITs 13/13, Broker fault gates 44/44, both modes ran.
• Second Turn on a healthy Session: fixed, section 2.
• Runtime lease not released when recovery bails out after acquire: this head releases on those exits. In the lost-:start run the Broker refused that release while the execution was pending, and the next attempt finished the Turn and released the lease.
• Cancellation settling without confirmed stop, and cancellation not releasing the Runtime Session: changed on this head; not tested here, and not reachable from the public API (section 3).
• The projection scan, the empty messageComplete(), and the actor filter overriding a principal: changed on this head. The Turn time grew linearly up to 3,000 chunks on the earlier heads.

Candidate patch

candidate.patch applies to 13cbd974:

1. QwenHostedHarnessConnector.recoverManagedRuntime hands out the cached recovery snapshot only until a continue or cancel has been admitted for it. One unit test; it fails on the unpatched head. (F3)
2. The continue route records its admission and replays it, as the prompt route does. (F3)
3. recoverHostedRuntimeTurn re-attaches to the Runtime Session when nothing is left to drive, so the terminal route can release it. (F2)

With it, on 13cbd974: both F3 scenarios complete with one continuation and the whole answer public; the lease is released after a continuation takeover and the second Session runs; the scenarios without faults are unchanged; the three runner modes pass 3/3, 3/3 and 1/1; QwenHostedHarnessConnectorTest plus HarnessCoordinatorTest pass 22/22; the five changed vitest files pass (127 tests, 3 of them only when run alone on the loaded host). F4 is unchanged. I did not run the Hosted IT job against the candidate, so treat it as a starting point.

Not covered

• No real model. The scripted model is what makes the crash points exact.
• The Hosted IT job and the Broker fault gates were not replayed locally; I relied on CI for those.
• Multi-Runtime takeover, the Shell profile and Windows are out of scope of the PR and of this run.
• Locally, under load, hosted-harness-session.test.ts failed one test per whole-file run, a different one each time, and each passes alone. CI is green. I count that as host noise.

Driver scripts, raw results for all four heads and the patch are under pr13083/.

Thanks for the thorough round — all five Criticals were real and are fixed at the rebased head, along with most Suggestions. Summary below; the commit message and design doc carry the details.

Criticals — all fixed:

• R1-1 (message.delta rejected newlines): new rawText payload field kind (non-empty + maxTextBytes, no control-character rejection) now types the delta's text; boundedString and the id kinds are untouched. Tests stream 'line one\nline two' plus a multi-byte chunk straddling the 3072-byte split and assert the committed payloads rejoin byte-exact.
• R1-2 (Runtime lease leaked on several exits): the lease flag now stores the recovered promptId and clears only after the release is confirmed, so a failed handback stays owed and the next terminal route retries it. close() (detach/DELETE) releases the lease, the load route's post-takeover refusals release it inline (the Session is never registered there), and the continue/cancel refusal exits release it once no drive is in flight. New witness: load with driveRuntimeRecovery then POST /session/:id/detach releases with the recovered promptId.
• R1-3 (cancel path journaled no tool_result): settleParkedTurnCancelled now writes the tool_result ChatRecord per settled item before resolving — same daemonPromptId, parentUuid: item.modelMessageId, dedup against already-journaled functionCallIds — so the next turn's history never carries an unanswered functionCall. The cancel route test asserts the transcript holds exactly one tool_result with the parked call's id.
• R1-4 (status() rethrew the Broker's terminal unknown): 409 runtime_broker_execution_unknown (UNKNOWN/ABANDONED) now resolves to undefined, which both callers already map correctly (outcome: 'unknown' / already-stopped). 404 and every other status behave as before; a new case pins the mapping.
• R1-5 (cancel settled a cancellation it never performed): the cancel route checks harnessRunAuthorization() before driving and answers 409 hosted_turn_recovery_required when it is not runnable; both helpers keep their existing behavior behind that gate. The unconfirmable-stop 503 case is unchanged.

Suggestions — addressed: R1-6 (behavior change documented in the design doc, both languages), R1-7 (dead AbortSignal params removed), R1-8 (cancel settle reuses outcomeBytes), R1-9 (post-cancel terminal-state wait pinned by call order + still-in_progress checkpoint assertion), R1-10 (re-attach branch test: settled parked turn still acquires once and reports acquiredRuntime), R1-11 (per-Content-chunk delta witness in the model-boundary suite), R1-13 (driveRuntimeRecovery/passiveManagedRuntimeRecovery wire flags asserted in both directions), R1-14 (job ceiling raised to 60 with the measured ~20 min noted, and the two failover modes split into separate steps so a failure names the mode), R1-16 (replay while the turn is still running gets the admission watermark), R1-17 (a runnable checkpoint parked under another turn now reaches the turnId guard), R1-18 (two-execution fixture; per-execution id/argument mapping asserted), R1-19 (10 s wait budgets), R1-20 (the 2026-09-26 design doc's CI scope note corrected in both languages), R1-21 (two-round assertion scoped to tool_result records), R1-22 (missing-intent fixture declines and releases), R1-23 (status() drops the unread result field), R1-24 (blocked continuation's replay now meets 409; regression test included), R1-25 (oversized fallback carries the live path's read_file hint), R1-26 (continuationAdmitted dropped from the report, the TS interface, the Java holder/parser, and its stub), R1-27 (binary-search split, ~150x fewer measurements on multi-byte text, byte-identical boundaries) plus the multi-byte fixture, R1-28 (cancel identity-mismatch refusal pinned), R1-29 (idempotency test now asserts the report reaches results_ready), R1-30 (all three passive cases assert zero journaled tool_results), R1-31 (acquire spy captured and asserted silent for bare/passive loads; release asserted absent after the takeover load and called exactly once after the settle).

Deferred, with reasons:

• R1-12 (route-level delta-projection test): every route-level suite mocks the whole model boundary (vi.mock('./hosted-harness-model.js')), so the streamedDeltaIds suppression cannot execute there without a dedicated unmocked harness. The contract is now pinned one layer down — per-chunk commit at the model boundary (R1-11's test) and byte-exact rejoin in the delta-stream suite — and end-to-end by the Linux/MySQL E2E. Happy to add the route-level case as a follow-up if you want the extra fixture.
• R1-15 (latency ordering bound): the suggested check (first visible text before the round's finish by more than the 250 ms poll granularity) does not hold on the current fixture — the fake model emits instantly, so the SSE poll observes the first visible text ~185 ms after finishedMs (baseline: 518.9 vs 334.0). Any ordering bound that only holds while streaming would flake on this fixture. The durable pin that survives poll granularity is a journal-level assertion (a message.delta precedes the turn's message.committed), which belongs in the latency driver; left as a follow-up rather than weakening the gate blindly.

Verification at this head: hosted-harness-session 91/91, hosted-runtime-recovery 13/13, broker/deltas/model suites green, core schema 80/80, managed-agent-server 353/353 + checkstyle, and the remote Linux FG6 profile (unit + MySQL ITs) plus all three failover E2E modes.

Real-environment verification, round 2: 912c3b57

Follow-up to round 1 on 13cbd974. Same rig: packaged Spring fat jar, dist/cli.js Hosted Harness, Runtime Broker with durable local workers, real MySQL, scripted local model, on Linux (Ubuntu 24.04 container) and macOS. The host was quiet this time (load average 3 to 10).

Result: everything round 1 asked to fix before merging is fixed and holds on the real stack. I found no new defect. Two things stand between this head and a merge: it conflicts with main again since #13110, and after the rebase the takeover scenarios should be re-run once, because #13110 touches the same session routes.

What I ran on 912c3b57

• Takeover with faults (Linux). No fault, stream cut during the replacement's answer, lost continue reply, lost takeover load, lost :start, lost takeover-load reply, death in the first model round, public cancel. Every run kept one execution row at generation 1, one physical write or none, and no second Prompt. Every Turn that completed released the Workspace lease, and a second Session on the same Workspace then ran its tool. The lost :start still takes 124 s (the 120 s observation window); the release this head adds on that failure exit is refused by the Broker while the execution is pending (409 runtime_session_busy), and the next attempt finishes and releases.
• The PR's runner (Linux, quiet host). --inflight-failover 10/10, --continuation-failover 9/10, --session-failover 3/3. CI on 912c3b57 ran both new modes green. The one failure stopped before the scripted crash: owner A's writer grant went stale ("The Managed Session writer grant is stale or unavailable") right after the first prompt was admitted. That is the same signature as the loaded-host failures in round 1, here at load 5. The runner uses a 1 s writer lease (QWEN_MANAGED_AGENT_SESSION_STORE_WRITER_LEASE_DURATION). I would lengthen it if CI shows the same thing; I did not change it to test that.
• Streamed text (macOS and Linux). See the image below. On 13cbd974 any answer with a newline failed the Workspace Turn with "payload.text must not contain control characters", so R1-1 was real. My round-1 scenarios used one-line answers and missed it.
• Mid-stream provider drop (macOS). The provider cut the stream after the first chunk. Both a Workspace-bound and an unbound Session retried as a continuation and completed. I could not reach the non-continuation retry or model-fallback path that the design doc now lists as failing the Turn, so that limit is untested.
• Unchanged. The trusted-actor header matrix is identical to round 1; the rollback load is still 503 on a base-build Harness.

What the new tests pin

On 912c3b57 I compiled the fixes away one at a time and ran the five changed vitest files (151 tests; a test counts as catching only if it also fails when run alone) and the connector tests:

• F2 re-attach removed: caught by re-attaches the Runtime Session when nothing is left to drive.
• F3 admission replay removed: caught by two tests, including replays a lost continue admission at its original watermark.
• F3 connector snapshot handed out after admission: caught by takeoverSnapshotIsReportedUntilItsContinuationIsAdmitted.
• R1-1 rawText reverted to text: caught by commits model text with newlines and tabs verbatim.
• Still not caught by any unit test: the de-duplication of a message whose text already streamed, and the live Turn's durable deltas. Only the Linux-only --continuation-failover catches them. This is R1-12, which the author deferred with a reason.

Before merging

1. Rebase onto main. 912c3b57 conflicts with e083d6a6 (feat(managed-agent): Add hosted file history and undo #13110 "hosted file history and undo") in hosted-harness-session.ts and hosted-workspace-broker.test.ts.
2. After the rebase, re-run the takeover scenarios once (the in-flight and continuation modes, and ideally the lost-continue-reply case), since feat(managed-agent): Add hosted file history and undo #13110 changes the same session routes. The rig is ready for that.
3. Optional: change the "Breaking changes / migration notes" line of the description to match the design doc's rollback note.

Not covered

• No real model. The Hosted IT job and the Broker fault gates were not replayed locally; CI covered them.
• The cancellation fixes (R1-3, R1-4, R1-5) are not reachable from the public API at this head (a public cancel of a parked Workspace Turn is 409 workspace_unavailable), so I did not test them beyond the unit suites.
• In one whole-file vitest run at load 7, replays a lost continue admission at its original watermark failed once and passed alone twice.

Evidence, driver scripts and raw results: pr13083/r2/ (round 1 files are still in pr13083/).

@qwen-code /triage

Thanks for the re-review at b4e9d71b. The new head addresses all of it:

The still-open R1-2 window is closed. recoverHostedRuntimeTurn now wraps the whole post-acquire span in one release-on-failure guarantee: the drive loop's broker.acquire() moved inside the existing try, the re-attach branch (pending.length === 0 && items.length > 0) releases on a failed acquire, and a rejection from the final harnessRunAuthorization() releases before rethrowing. Every failure exit between acquire and the returned report now hands the lease back.

Finding 1 (Important) — fixed. The continue route now proves the checkpoint is continuable before answering: harnessRunAuthorization() + continuation.phase === 'results_ready' run ahead of the 200 admission, and a non-continuable checkpoint gets 409 hosted_turn_recovery_required (with the lease handed back) instead of a bare turn.settled that wedges the Session. This mirrors the R1-5 fix on cancel.

Finding 2 (Minor) — fixed. The cancel route now records the admission (recovery:<checkpointId>:<activationId>) before driving, and serves a replay with the admission watermark while the confirmation is still in flight (or after it settles) instead of 409 hosted_turn_active. The first answer also moved to the admission watermark, so the cancelled turn_result streams in from there on both the original and the replayed call. A failed cancellation drops the admission so a retry re-drives. New test: a replay during the open broker confirmation gets 200 with the same watermark.

Finding 3 (Minor) — fixed. The continue route now hands session.mcp to the tool turn (the tenth constructor argument), so an MCP-profile continuation drives the MCP session's own broker instead of building a separate promptId-keyed one.

Finding 4 (Minor) — fixed. The continuation failover gate now pins continuationRequests.length !== 2 (exactly), and the evidence block derives promptReplayed (initialModelRequests.length !== 1) and physicalToolExecutions (executionCount) from the run instead of printing literals.

Finding 5 — confirmed pre-existing, not this slice. The bindRecoveredHarness-false path in HarnessCoordinator predates this PR (that file is untouched here); the new passive snapshot only makes the path reachable. Left as-is in this PR — it wants its own fix, since it sits on the cancellation path this slice adds.

The P1 regression is fixed with the regression test you asked for. The continue route now calls toolTurn.resumeCommittedResults() before inference, so a recovered Write/Edit's durable pendingTurn is reconciled (snapshot identity compared against the original Runtime, then cleared) even when the continuation answers with text only. The new regression test parks a Write, continues it to a text-only answer, then asserts pendingTurn === null and that a subsequent cold load succeeds.

Also fixed since the last pass: the hosted-harness-mysql profile's failsafe fork was killed at its 600 s ceiling once the file-history control calls stretched the ToolTurn driver — raised to 900 s with the reason recorded in the POM.

Verification at this head: the takeover/recovery suites pass, npm run build + npm run typecheck clean, and the remote Linux FG6 profile (MySQL, all Hosted ITs) plus the three failover E2E modes pass.

One environment note, for completeness: shellOutputFailuresNeverReplayEffectsOnMySql's receipt-failure case fails deterministically on my aarch64 test rig (the probe's finished gate never answers within its 15 s window) on main's tree too — same stack, same point. So it is an environment-specific issue on that box, not from this PR; CI is green for it.

Thanks for the round-3 real-stack run — and for the candidate patch. Status at the new head (7ae1fa05):

F7 (file-history marker left by an in-flight takeover) — fixed, using the resume-reconciliation approach doudouOUC suggested (which you preferred over candidate A): the continue route now runs toolTurn.resumeCommittedResults() before inference, so the parked batch's pendingTurn is reconciled against the original Runtime's snapshot and cleared even when the continuation answers with text only. The cancel path clears the marker too when the cancelled Turn never continues. Regression test: park a Write, continue to a text-only answer, assert pendingTurn === null and a subsequent cold load + file tool succeed. I skipped candidate A itself because clearing at drive time would leave a window between the takeover load and the continue/cancel where undo still 409s; reconciling at continue (with the snapshot-identity check) keeps that failure fail-closed.

F8 (taken-over Turn cannot run another tool batch) — fixed with your candidate B, unchanged: consumeRuntimeResults now commits the consumed checkpoint under the current activation when the previous checkpoint's activation differs and every item is settled. Your test lets the activation that consumed a taken-over batch commit the next one is in verbatim; it goes red without the fix and green with it, and the sibling refuses an approval in an unfinished turn from another activation still passes. core managed-harness-factory 32/32.

The earlier review round's items are also in this head (the continue route now proves the checkpoint is continuable before answering 200, cancel records its admission and replays at its watermark, the recovery's whole post-acquire span releases on failure, and continue hands session.mcp to the tool turn).

Verification at 7ae1fa05: the PR's runner passes --inflight-failover, --continuation-failover and --session-failover on the same Linux rig; the FG6 profile (353 unit + 16 IT) is green except receipt-failure, which fails identically on main's tree on that box (environmental — CI is green for it). CI on this head is running.

@qwen-code /triage

Real-environment verification, round 4: 7ae1fa05 (post-merge)

This PR was merged at 16:35 UTC (squash 728c13de) while this round was running, so this is a post-merge confirmation rather than a merge gate. The merged code is the code I tested: git diff 7ae1fa05 728c13de restricted to the PR's 36 files is empty.

This follows round 3 on b4e9d71b. What changed since then:

• The takeover commit was amended on the same base (e083d6a6, feat(managed-agent): Add hosted file history and undo #13110).
• Candidate B is now a separate commit.
• There are no Java changes, so the server jar is the same build as in round 3.

Same rig as before: Linux in an Ubuntu 24.04 container, plus macOS. Host load was 20 to 35.

Result: F7 and F8 are fixed on the real stack, and nothing from rounds 1 to 3 regressed. The review fixes in this head also hold where the rig can reach them.

One new finding, F9: the cancellation takeover cannot finish against a replacement Broker. The public API cannot reach that path in this slice: the store refuses cancel for Workspace Sessions, and the design says cancellation takeover "has no E2E mode". It is a follow-up for main: either fix it (candidate C below, about 30 lines; it applies cleanly to main at 728c13de) or write it down as not working yet.

F7 and F8 at 7ae1fa05

F7 (resume reconciliation in continue):

• Every in-flight scenario ends with the marker cleared, undo returning 200 and removing the file, and a fresh Harness loading the Session with 200. The scenarios are: no fault, lost continue reply, lost takeover load, and lost :start (124 s, as before).
• Continuation takeovers are unaffected: the saved history binds to the reclaimed Runtime and undo still works.

F8 (candidate B as committed): two-tool takeovers complete in every scenario:

• in-flight: 2.5 s;
• continuation: 2.6 s;
• continuation with a stream cut: 4.7 s;
• in-flight with a lost continue reply: 3.9 s;
• in-flight with a lost takeover load: 4.7 s.

In every one of them, both executions stay at generation 1, undo restores both files, and a second Session on the same Workspace then runs its tool.

F9: the cancellation takeover cannot finish against a replacement Broker

How I reached it. The public API refuses cancel for Workspace Sessions: insertCancelCommand answers 409 workspace_unavailable. So after owner A dies, the rig applies the same transition the store would (status = 'CANCELLING') and then starts owner B. Owner B's coordinator then does exactly what it does for a real cancel: a passive takeover load, then managed-runtime/cancel.

In-flight (the execution is parked and not settled):

1. The passive load reads the execution status from Broker B without adopting the Runtime Session.
2. Broker B answers 404 runtime_session_not_found: for an unsettled execution, RuntimeBrokerService.getExecution calls requireReadySession, which needs the Runtime Session to be active in this Broker process.
3. The load answers 409 hosted_turn_recovery_required.
4. The coordinator retried 6 times in 90 s, and the Turn stays CANCELLING.

Continuation (the execution is settled):

1. The passive load succeeds, because a terminal execution is answered from the durable record.
2. The cancel route's explicit broker.release() gets 503 runtime_reconciliation_required ("Runtime Session is not active in this Broker process"). This is the same release-without-acquire refusal that was behind round 1's F2.
3. The cancel route answers 503 managed_runtime_cancel_failed on every retry.

b4e9d71b behaves the same way. The unit tests stub status and release so that they succeed without an acquire. reports a parked execution passively and cancels the turn also asserts expect(acquireSpy).not.toHaveBeenCalled(). So the tests encode a Broker contract that the real Broker does not offer after an owner change.

Candidate C (about 30 lines in hosted-runtime-recovery.ts): when a passive load has Runtime items, it acquires the Runtime Session first. Acquiring dispatches nothing. If the acquire or a status read fails, it hands the lease back. In the tests, the candidate flips that one assertion and adds an acquire stub to the three passive recovery tests. With it:

• Every case ends turn.cancelled, in 1.6 to 4.0 s.
• In-flight, the tool never runs. There are no Workspace file events, and undo has nothing to restore.
• The Workspace is freed. The lease is released, and a second Session on the same Workspace then runs.
• A lost cancel reply is replayed at the same watermark (13 → 13 and 18 → 18). This is the new cancel admission replay working end to end.
• The drive path is unchanged. Two-tool in-flight and continuation takeovers complete, and the runner passes 3/3 in both modes.
• The changed CLI test files pass (173 tests; the 4 that failed in the loaded full run pass when run alone, as on the unmodified head).

Patch: candidate-r4.patch. Adopting at managed-runtime/cancel instead would also work, but the status read happens during the load, so the load is where the adoption has to happen.

Everything else at 7ae1fa05

• Takeover with faults (Linux). Every scenario has the same outcome as on b4e9d71b:
  • each run keeps one execution row at generation 1, with one physical write or none, and no second Prompt;
  • every completed Turn releases the Workspace lease, and a second Session runs.
  • F4 and the out-of-scope first-round crash are still stuck, as documented.
• The PR's runner. --inflight-failover 10/10, --continuation-failover 10/10 (it now pins exactly 2 continuation requests), --session-failover 3/3.
• Unchanged.
  • F1: Turn 2 on a live owner takes 0.3 s, and a dropped SSE recovers in 1.1 s.
  • R1-1: streamed text stays byte-exact on macOS and Linux.
  • A provider drop after the first chunk is retried as a continuation.
  • The trusted-actor header matrix is identical to earlier heads.
  • Rollback load on a base-build Harness is still 503 (F5, documented).
• Mutants (the changed CLI test files; a test counts only if it also fails alone):
  • Caught: removing the F7 reconciliation (2 tests), the F8 adoption (1 core test, managed-harness-factory), the cancel admission replay and its watermark (1 test each), and the round-2 fixes R01 to R03.
  • Not caught by any unit test:
    • the continue check before the 200 (yiliang114, item 1);
    • all three new release-on-failure exits in recoverHostedRuntimeTurn (R1-2);
    • handing session.mcp to the continuation;
    • dropping the admission after a failed cancel.
  • My rig cannot reach any of those either, so I checked them by reading only.
  • T02 and T03 are unchanged since round 3.

Follow-ups on main

1. F9: take candidate C, or add it to the design doc's known follow-ups: the cancellation takeover does not work against a replacement Broker yet, and the unit assertion that a passive load never acquires does not match the real Broker.
2. Carried over, optional: the description's "Breaking changes / migration notes" line, and R1-12 (T02/T03 have no unit coverage).

Not covered

• No real model. CI covers the Hosted IT job and the Broker fault gates.
• The rig set the cancel state through the store transition, because the public API refuses cancel for Workspace Sessions.
• Restarting only the Harness behind a live Spring owner.

Evidence, driver scripts and raw results: pr13083/r4/ (earlier rounds are in pr13083/, pr13083/r2/ and pr13083/r3/).