bug(hosted): a transient Managed Session Store outage permanently stops Session log writes, wedging the running Turn #13413

Summary

If the Managed Session Store (hosted by Spring) is unreachable while the Hosted Harness is writing a running Turn, the Harness stops writing that Session for good. The Turn can then never complete and never be cancelled. A transient outage becomes a permanent wedge.

Observed by @wenshao during the Round 4 real-stack re-verification of #13163 (comment), who noted it "is a different failure from #13054 and deserves its own issue", and that it reproduces on main — #13163 neither introduces nor fixes it.

Observed behaviour

wenshao's rig, identical on both arms (the PR head and its merge base with main).

Repro: a bound Turn whose fake model streams a text delta every 200 ms for 40 s. Spring is restarted mid-stream. No grant changes.

• The Harness logs session log writes stopped after an earlier failure: Managed Session Store request failed: fetch failed, then could not settle.
• 35 s after the stream ended, the Turn is still RUNNING.
• The creator's cancel gets 202. 60 s later the Turn is only CANCELLING.
• The next submit gets 409 turn_active.

Mechanism

packages/core/src/managed-runtime/managed-session-authority.ts. Read statically from source — I did not execute this.

The write path latches on the first failure (line 2152-2160):

    try {
      await this.journal.appendTransaction(records);
      this.lastRecordUuid = final.uuid;
    } catch (cause) {
      // Records may already be on disk, so the sequences this transaction
      // claimed are spent whether or not the marker landed.
      this.writeFailure =
        cause instanceof Error ? cause : new Error(String(cause));
      throw cause;
    }

Every later write is then refused (line 1980-1982):

    if (this.writeFailure !== undefined) {
      ...
        `session log writes stopped after an earlier failure: ${this.writeFailure.message}`,

writeFailure is declared at line 378, read at 463 / 1980 / 1982, and assigned at exactly one place, line 2157. Nothing ever resets it to undefined. The latch is one-way for the lifetime of the Session authority instance.

The in-code comment shows the latch is deliberate: after a failed appendTransaction the sequences that transaction claimed are spent whether or not the marker landed, so reusing them could corrupt the journal. The gap is that the same permanent latch is applied to a transient cause — a store that is briefly unreachable during a restart — with no retry, no re-probe, and no path back once the store returns.

Impact

• The Turn never reaches a terminal state, so the Session stays RUNNING and every later submit is refused 409 turn_active.
• Cancellation cannot rescue it: the cancel is accepted (202) but the settlement write is refused by the same latch, so the Turn parks in CANCELLING.
• One Spring restart during an active Turn is enough. wenshao hit it by chance in a WebShell run where Spring began shutting down about 0.6 s after the Turn started.

What a fix has to reconcile

Recovery must not reuse the sequences the failed transaction already claimed. Directions worth weighing:

• re-probe the store and resume from a freshly claimed sequence range;
• persist or derive the spent-sequence high-water mark so the latch can be lifted safely;
• or escalate the Session to a terminal failed state instead of leaving it indefinitely RUNNING, so a later submit is not refused turn_active forever.

Scope

• Reproduces on main and on fix(managed-agent): stop a bound Turn under refused authorization #13163 at 30f092d0. Not a regression from that PR.
• Hosted Harness + Managed Session Store (Spring) deployment.
• Distinct from feat(managed-agent): define recovery visibility for post-submission Workspace refusal #13054 (inherited recovery-blocked Turn).