fix(runtime-broker): close cross-process release race and harden scheduling - #13214

Verification report

Reproduce-first, then fix, then verify — same Maven harness throughout (JDK 21, Maven 3.9.16, H2 in MODE=MySQL; the raced constructs — FOR UPDATE row locks, version-guarded CAS, separate transactions — behave identically to MySQL there, and the module ships a mysql-integration failsafe profile for the real engine).

• The three high-risk findings were each reproduced deterministically against the base commit before any fix (two repository stacks over one database standing in for two Broker processes; a parked renewal JDBC call; real HTTP socket binds), and the same scenarios pass with flipped assertions on this branch.
• packages/sdk-java/runtime-broker: mvn clean test — 578 tests, 0 failures/errors. New regression coverage: Issue13183RegressionTest (11) and Issue13183AdversarialTest (4, including a 200-round cross-process admit/release stress and a forked-JVM exit-hook proof).
• mvn checkstyle:check clean. packages/sdk-java/managed-agent-server fix-adjacent suites green (16 tests).
• The fix went through /review plus three adversarial audit rounds after the initial verification; they caught two real issues the first pass missed (workers spawned-but-unregistered being invisible to the exit hook/close during the ready handshake; a spawn↔register orphan window under an exit-hook-only shutdown), both fixed and covered by tests. The last two audit rounds were clean.
• Pre-existing, fix-independent flakes on this machine (confirmed failing on the base commit under load, passing in isolation): DurableRuntimeRecoveryTest.lostBranchReclaimsBeyondTheReconcileDeadline, two ToolPublicationStoreTest cases.

Round-2 review dispositions (commit 34b3f07)

All 21 findings were addressed; each new guard now carries a mutation-proven test (mutant red, fix green).

Fixed:

• R1-1 (Critical): the cache-hit path replays a settled/abandoned cached reconciliation whole; only a still-UNKNOWN cached record is paired with the caller's fresher one (cancel flag stays fresh). Pinned by cooledObservationNeverPairsASettledAnswerWithAStaleRecord (red when the guard is reverted).
• R1-2: v3ResultWindow floors at MIN_V3_RESULT_WINDOW (1 s) in the service constructor and in EmbeddedRuntimeBroker before anything is built; a suffix-less 30 now fails startup instead of degrading v3 executions. Pinned by v3ResultWindowBelowTheFirstPollTickIsRefused + the broker test.
• R1-4: :start/:cancel mutation responses bypass the cooldown (coolable=false); only plain GET polling is cooled. The v3 HTTP test catches a revert (cancel answered executing from cache → red).
• R1-5: the drain loop measures its baseline (stall check fires on pass 1) and stops after 16 bounded passes per call, answering runtime_broker_runtime_lost so a later reclaim resumes; pinned by reclaimBeyondThePassBudgetAnswersLost (4801 sessions). Declined the orTimeout: the drain is synchronous JDBC on the caller's thread, so a timeout would free the caller while the loop keeps holding the operation claim — the pass budget is the honest bound.
• R1-6: pinning tests added per guard — service-level cross-process release race (red when the wiring is reverted), verifyBeginSessionRelease contract on both backends (busy/not-ready/passthrough/stale), v3ResultWindowBoundsThePolling (red when the window is hardcoded), opt-in wildcard bind positive case, provisionDuringCloseNeverOrphansAWorker (red without the terminated guard), dispatchRenewalsRunOnTheRenewalPool (red if dispatch renewals move pools).
• R1-7: cooldownIsScopedPerExecution — a second UNKNOWN execution in the same session is asked on its own first observation (red if the key widens to the session).
• R1-8: the redundant pre-flight hasActiveByRuntimeSession is dropped; beginSessionRelease's in-transaction check answers the identical 409.
• R1-9: dead lookupOnce overload deleted.
• R1-10: new design doc docs/design/2026-10-02-runtime-broker-hardening.md (+ zh-CN); the stale statements in managed-runtime-broker-service-core and managed-runtime-broker-jdbc updated in both languages.
• R1-11: shared fixtures deduplicated; ProcessTrees.childPids() reused.
• R1-12: the inert control test deleted (the guard is pinned by the race test and the contract).
• R1-13: the mixed-drain test asserts exactly 150 active executions (this caught a real fixture issue — PREPARED is outside findUnsettled, so the executions now run EXECUTING).
• R1-14: gap assertions are now two-sided (lower + upper bounds).
• R1-15: stalledRenewalTickDoesNotDeadlockClose brackets close() alone with a 5 s bound.
• R1-16: bounded waits everywhere plus @Timeout on both new classes.
• R1-17: FakeAttestationWorkerTest.ignoreTermSurvivesSigterm pins the wedge itself (red when reverted).
• R1-18/19: the exit-hook test renamed to the post-fix behavior, harness output kept to a log file with the tail in failure messages.
• R1-20: cooldown eviction consults the service clock, so the frozen-clock tests can't be evicted early by wall time.
• R1-21: UNKNOWN_LOOKUP_COOLDOWN is package-visible and the helper reads it.

Declined with rationale:

• R1-3: the 5 s cap is the load/latency trade-off the issue asked for; a result landing in the final poll gap flips UNKNOWN and is still recoverable through the existing reconcile path, same as the pre-PR 100 ms behavior. Kept.

Verification: mvn clean test in runtime-broker — 587 tests, the only failure being the known pre-existing lostBranchReclaimsBeyondTheReconcileDeadline flake (fails identically on the base commit under load, passes in isolation); mvn checkstyle:check clean; managed-agent-server fix-adjacent 17/17 green. A follow-up adversarial audit of this commit's delta came back clean on all five attack surfaces.

Round-3 review dispositions + merge (commit 3710ef7, merged with main at a011f66)

The branch merged main (the conflict was RuntimeBindingRepository's adjacent additions — the harness-drain methods and beginSessionRelease coexist; the merged tree's full suite passes, see below). Round-3 findings handled:

Fixed:

• R2-1 (Critical): FakeAttestationWorkerTest.ignoreTermSurvivesSigterm now carries @DisabledOnOs(WINDOWS) — Windows CI has no SIGTERM to swallow; destroy() there is TerminateProcess. Verified against sdk-java.yml: the windows-latest / java 21 leg does run this module. (Not re-runnable locally on macOS; the guard is structural.)
• R2-2: the cache-hit replay now serves the fresher of the caller's snapshot and the lookup's own re-read while both are UNKNOWN (a cancel landing between them stays visible). Pinned by cooledObservationServesTheFresherUnknownRecord — red when reverted to caller-wins.
• R2-4: the orphaned Collectors import dropped.
• R2-5: startResponsesNeverServeTheCooldownCache pins the :start half — a cooled "executing" stamp, then a :start retry inside the window answers "settled" from the worker. Red when the site's coolable flips.
• R2-6: FakeAttestationWorkerTest.defaultWorkerExitsOnSigterm pins the mirror arm — a default worker dies on destroy() within 2 s.
• R2-7: reclaimBeyondThePassBudgetAnswersLost now drives a second warm after the 503 and asserts the reclaim resumes: the generation drains and a new generation is READY.
• R2-8: provisionDuringCloseNeverOrphansAWorker matches the guard's exact message ("Managed Runtime provisioner is closed." — no longer confused with "closed before ready"), catches the RejectedExecutionException a post-shutdown submission throws, and stops polling once the closer exits.

Declined with rationale:

• R2-3: the cooldown window is deliberately measured in the service clock (the same clock that owns every other deadline in the service); a backward wall-clock step stretching the window is consistent with that. The eviction task is hygiene only — the read-side deadline check is the authority, so a never-firing eviction cannot serve an expired answer. Entries are bounded by distinct execution ids (re-stamps overwrite the same key).

Deferred items from the round-3 body (v3-window forwarding witness, renewalScheduler.shutdownNow witness, harness constructor cleanup) were recorded by the review as not-requested this round; left as-is.

Verification on the merged tree (34b3f07f + merge + this round): mvn clean test in runtime-broker — 620 tests, the only failure the known pre-existing lostBranchReclaimsBeyondTheReconcileDeadline flake (fails identically on base under load, passes in isolation); managed-agent-server fix-adjacent 17/17 green; the R2-2 and R2-5 pins were each proven red under their mutations before landing green.

Self-audit round 1 (undirected + adversarial) — dispositions

Ran an undirected pass and an adversarial pass over the branch head after re-merging main (merge 51c1841). Result: 1 Critical + 10 Suggestions. All are fixed in af6691c except three declined below with rationale. Each behavioural fix is pinned by a test that was proven red against a mutant.

Fixed

C1 — FakeAttestationWorkerTest.defaultWorkerExitsOnSigterm proved nothing. It called destroy() without first reading the worker's ready line. The fake installs its SIGTERM handler in top-level script flow and only writes the ready line from the listen callback, i.e. on a later event-loop turn, so a destroy() issued straight after spawn lands before any handler exists and the worker dies through SIGTERM's default disposition — the test passed whether or not the default handler worked. It now waits for the ready line before destroying.

Mutation proof: with the fake's default arm wedged (process.on('SIGTERM', () => {}) unconditionally), the pre-fix body passed 3/3 runs (~0.28 s each); the fixed body fails with a default worker must exit on SIGTERM ==> expected: <true> but was: <false>.

S3 — v3 result-poll cap 5 s → 2 s. The cap only exists to bound how late a finished result is picked up, and 5 s is coarser than the polling it replaces. v3ResultPollingBacksOff now waits for seven polls and pins the first capped gap into [1.9 s, 2.9 s) plus a ≥ 1.5 s fifth gap, so both the doubling and the cap are bounded; 2.9 s stays under the 3.2 s an uncapped doubling would schedule, with enough slack for scheduler jitter.

Mutation proof: restoring Math.min(5_000, …) at that site fails with the backoff must stop at the 2s cap: 3321145416.

S4 + Adv2 — the pass-budget test raced its own deadline. reclaimBeyondThePassBudgetAnswersLost built both reclaiming services with a 3 s lease, so the measured operation deadline (4 × lease = 12 s) sat well inside the test's 60 s get(). On a slow runner the reclaim could answer on the deadline rather than on the 16-pass budget and the assertion would still see runtime_broker_runtime_lost — the budget would be untested. Both services now take a 10 s lease (40 s deadline) through a new service(...) overload.

The comment also miscounted the drain: it claimed three drain calls release 16 bounded passes each. Traced behaviour is that the first site returns before releasing anything (the loss evidence is not written yet), sites two and three drain 16 × 100 rows each, and the ~1.6k rows that remain stop the reclaim with LOST. The comment now says that. Side effect: the test went from ~23 s to ~2.2 s, because it no longer waits for a deadline to lapse.

S7 — the deployment README documents the two new knobs (QWEN_MANAGED_AGENT_RUNTIME_BROKER_ALLOW_NON_LOOPBACK, QWEN_MANAGED_AGENT_RUNTIME_BROKER_V3_RESULT_WINDOW) with their defaults and what each changes, next to the existing Broker environment list.

S2 + Adv3 — design doc corrected, both languages. The renewal-pool paragraph claimed "one stalled renewal blocks at most one pool thread"; with a two-thread pool, two parked ticks saturate it. What the split actually buys is that coordination work no longer queues behind a parked tick and that close() interrupts both pools without awaiting one — the text now says exactly that. The same paragraph still described a 5 s poll cap, and described the cooldown as replaying a cached lookup whose record "already settled", where the code replays a cached record that is no longer UNKNOWN and otherwise serves whichever of the two records carries the higher version.

Declined

S5 — "the service outlives the HTTP server": false positive. RuntimeBrokerHttpServer.close() stops the server, shuts its executor down and then calls service.close(), so the reported path cannot leave the service running.

S6 — relaxing the in-memory release gate: declined after trying it. InMemoryRuntimeBindingRepository.beginSessionRelease refuses a session/execution repository pair that is not the matching in-memory type. I removed the gate and pinned a pass-through wrapper in RuntimeRecoveryContract, and the JDBC half of the contract failed immediately with Release requires the same DataSource — JdbcRuntimeBindingRepository.beginSessionRelease enforces the same precondition at both of its entry points, because the single-transaction guarantee only holds when both repositories reach the same store. The in-memory check is the sibling of that one, not drift from it, and accepting a mismatched pairing silently would weaken exactly the release decision this fix rests on. Per-backend alignment is already tracked in #13204. Both edits were reverted; the contract is unchanged.

Adv-S1 — injecting a clock into the HTTP fixture: declined. The cooldown assertions hold with roughly an order of magnitude of wall-clock margin, and that fixture already sleeps for the same purpose elsewhere; a clock seam there costs more than it buys.

Verification

• packages/sdk-java/runtime-broker: mvn clean test → 627 tests, 0 failures, 0 errors, 2 skipped; mvn checkstyle:check and mvn spotbugs:check clean.
• packages/sdk-java/managed-agent-server (fix-adjacent): EmbeddedRuntimeBrokerTest + ManagedAgentPropertiesTest → 19/19 green against the freshly installed broker jar.
• Mutation proofs above were run against the same tree and reverted from cp backups; git diff confirms no mutant remains.

Real-stack verification — head af6691ce1f

Verdict: the fixes hold on a real stack. Fix one thing before merging: bot R3-1 (still open) reproduces end to end. A release retry over a Session row that is already RELEASING finishes the release over a live execution. Base refuses that retry. The 9-line fix the bot suggested, plus a regression test, closes it without breaking anything else I measured (details below). Everything else is non-blocking.

Head was unchanged for the whole run. main moved to 1a4de7486a (#13206, web-shell only, no sdk-java change). A trial merge is clean. CI is green apart from review-pr, which is still running.

Setup

• Arms: head af6691ce1f, base 5130c1a734 (the merge-base), and cand (head + the R3-1 fix). Each arm was built in its own Maven repo, and javap confirmed each arm's bytecode.
• Database: MySQL 8.4.7 (InnoDB, REPEATABLE-READ, UTC). No H2 anywhere.
• Broker processes: each Broker is its own JVM running the production RuntimeBrokerService, JDBC repositories and HTTP face. Workers are the bundled node dist/cli.js managed-runtime-worker; this PR changes no TS, so dist was built from the merge-base.
• Spring: the real managed-agent-server jar, configured only through the documented env names.
• TS client: the shipped Hosted Harness client (hosted-workspace-broker.ts) is driven directly.

Test suites:

• runtime-broker mvn verify: 627 tests, 0 failures (2 skipped); SpotBugs 0; Checkstyle 0.
• mysql-integration failsafe profile (not run in the PR, run here against MySQL 8.4.7): 7/7, including the new verifyBeginSessionRelease contract.
• managed-agent-server EmbeddedRuntimeBrokerTest + ManagedAgentPropertiesTest: 19/19.

F1 — bot R3-1 confirmed: a release retry over a RELEASING row ignores a live execution (fix before merge)

Setup: a real worker over the HTTP face, a Session row seeded as RELEASING with a PREPARED execution on the same binding and generation. That is what a pre-fix peer leaves behind when its unguarded check-then-act lets an admission in (card 1) and its worker release then fails, e.g. during a mixed-version rolling deploy.

Head's transitionSessionToReleasing short-circuits on RELEASING without asking the execution repository. The candidate adds that check inside the short-circuit; it is the bot's suggestion (candidate-r31.patch, 9 lines plus one test). With the candidate:

• A RELEASING row with nothing active still releases (200), so the idempotent retry is kept.
• The new test releaseRetryOverAPersistedReleasingRowRefusesALiveExecution fails without the 9 lines (Expected CompletionException … nothing was thrown) and passes with them.
• The full module suite runs 628 tests with 0 failures; SpotBugs 0, Checkstyle 0.
• The 300-round race still has 0 contradictions.

F2 — the cooldown does not reach the shipped poller (non-blocking, bot R3-3)

The only shipped path that polls an UNKNOWN execution repeatedly is HostedWorkspaceBroker.execute(…, waitForUnknown = mcp !== undefined). It sends GET …?reconcile=true every 250 ms, and reconcile=true bypasses the cooldown by design.

I made an execution UNKNOWN for real (the worker's /execute response is dropped while the shell keeps running) and drove the real client for 20 s:

At ≈1.4 status calls/s over the 630 s MCP window, that is still about 900 status calls per execution on both arms — the #13183 symptom. The v3 and provider plain-GET path the cooldown does cover is pinned only by unit tests here; the shipped v3 client stops at its first UNKNOWN. Options: apply a (shorter) cooldown to reconcile=true as well, or have the client back off. Otherwise, narrow the PR text so it doesn't claim the #13183 fan-out is fixed.

F3 — the two-thread renewal pool absorbs exactly one parked renewal (non-blocking, as the design doc says)

With two parked renewals on two bindings, head's healthy execution also ended UNKNOWN, the same as base. The pool helps against one stuck row; it does not help against a database-wide stall, where every renewal's JDBC call parks together.

Rig deviation: the worker request timeout was 120 s (production 30 s), so a v2 call can outlive one lease. It stands in for long v3 executions, whose dispatch claims are renewed the same way.

Notes (no action needed)

• N1: the shutdown escalation and the exit hook apply only to non-durable provisioners. Since feat(managed-agent): default durable local-process and trusted reboot recovery on #13211, Spring defaults to durable local-process (Linux), where workers are adopted on restart, so these paths are inert in the default deployment. A SIGKILLed Broker cannot run any hook on either arm, and the worker survives.
• N2: the drain loop needs stop evidence. After a plain worker crash under the non-durable provisioner, the generation stays LOST with JOURNAL_LOST on both arms; that is the pre-existing writer-domain rule.
• N3: V3_RESULT_WINDOW=45m is accepted, but the TS client's own v3 deadline is fixed at 30 min (bot R3-4).
• N4: HOST=127.0.0.2 fails on macOS with a BindException on both arms because lo0 has no such alias. Head's loopback check accepts it.

Evidence: assets-pr13214@e8f18ecf. It holds the harness (RigBroker/RigAdmitter JVM mains over the module's test-jar, the drivers, the Spring matrix and lifecycle scripts), the raw results, and the candidate patch.

Self-audit round 2 (adversarial + undirected) — dispositions

Two independent passes over af6691c: an adversarial one attacking the tests and the concurrency claims, and an undirected one over the whole diff. Neither found a Critical. Together they returned 15 Suggestions: 13 fixed in 20d8f48, 2 declined with measurements below. Every new or strengthened assertion was proven red against a mutant.

The headline: the core race test was one-sided

concurrentCrossProcessAdmitAndReleaseNeverContradict counted contradictions but never counted which side won. Instrumenting it showed admission won 0 of 200 rounds: the release reaches the session row first every time, because its path goes straight to the row lock while admission first takes the placement-domain lock and does two reads. So the stress never exercised the runtime_session_busy direction at all — only runtime_admission_closed, 200 times.

Fix: half the rounds now hold the release until the admission has committed (a second latch), half stay a tight race, and the test asserts both counters are non-zero, so a one-sided schedule fails loudly instead of passing quietly.

Mutation proof, and the reason this mattered: changing beginSessionRelease's selectSession(…, true) to false — dropping the FOR UPDATE that is the whole fix — now fails with admission and release contradicted each other ==> expected: <0> but was: <2>. admitExecution never updates the session row, so the version CAS cannot catch a racing admission; the row lock is the only cross-process guard, and it is now pinned from both directions.

Fixed — tests

• Cooldown replay, both arms pinned. cooledObservationNeverPairsASettledAnswerWithAStaleRecord was satisfied by the version comparison alone — deleting the "replay a settled cache whole" branch left it green. The stale snapshot is now pinned to the settled record's own version, so only that branch can keep a terminal answer off a stale record. Proof: deleting the branch fails with a settled answer must never ride a stale UNKNOWN record ==> expected: <SETTLED> but was: <UNKNOWN>.
• The fresher-record rule's other arm had no test. Every existing case had the cache holding the newer record, so freshest = cached.getRecord() survived the whole suite. New cooledObservationServesACancelThatLandsAfterTheCachedLookup cancels after the cached lookup and asserts the caller's re-read wins while the cached worker answer is still the answer. Proof: that mutant now fails the new test (a cancel after the cached lookup must stay visible) and still passes the old one — exactly the reported gap.
• close() had an unpinned pool shutdown. Deleting renewalScheduler.shutdownNow() left the entire suite green: a renewal tick cancels itself once the service is closed, but a pool's threads exit only on shutdown, so every rebuilt service leaked two daemon threads. New closeStopsTheRenewalPool records the renewal threads that appear while a dispatch is in flight and asserts those exact thread IDs are gone after close(). Proof: deleting the shutdown fails it. The coordination pool's shutdown is pre-existing code and no flow in these fixtures starts that thread, so it stays unpinned.
• The v3 poll cap is pinned deterministically. The wall-clock assertion could not tell a 2 s cap from a 2.5 s one and needed a 900 ms scheduler-jitter budget on a loaded runner. The computation is extracted to v3PollDelayMillis(attempt) and asserted exactly for attempts 0–7 (100, 200, 400, 800, 1600, 2000, 2000, 2000); the wall-clock test goes back to four polls and keeps only bounds a scheduler cannot fire early. Proof: caps of 5 s and 2.5 s both fail in 0.2 s.
• Windows-vacuous tests are skipped there. destroy() terminates outright on Windows, so --ignore-term cannot wedge a worker: the two escalation tests and round 1's SIGTERM mirror test passed without exercising anything, while the mirror's premise test was already @DisabledOnOs(WINDOWS). All three now carry the same annotation, with the reason in the javadoc.
• The exit-hook test's observation window is test-controlled. The forked harness used to sleep(2000) and exit, so the window in which the worker is visible belonged to the harness: a polling thread preempted for 2 s reported "the harness never spawned a worker". The harness now stays in the handshake (bounded at 60 s so an orphaned harness cannot linger) and the test ends the JVM with destroy() once it has seen the worker — SIGTERM runs the same shutdown hooks. The javadoc also records that this path covers the hook's destroy(), not the forcible fallback, because the forked worker obeys SIGTERM.
• provisionDuringCloseNeverOrphansAWorker's claim narrowed to what it proves. Its leverage is the 5 s forcible-destroy grace, which widens the terminated-guard window, not the nanosecond-wide spawn-and-register window; moving starting.add(…) outside the lifecycle lock survives it. The javadoc now says the test pins the guard and the absence of lingering children, and that spawn-and-register atomicity is structural — both happen inside the lock terminateAll() also holds — rather than observable from outside it.
• A refused release must leave the path usable. serviceReleaseRejectsARacingCrossProcessAdmission retries the release and asserts the same 409, so a half-set release slot cannot hide behind the first failure.
• The embedded-broker window validation covers its whole branch. refusesAV3ResultWindowBelowThePollFloor (renamed — "before anything is built" was never asserted) walks null, ZERO, -1ms and 999ms; the null arm and the realistic suffix-less-milliseconds case were untested.

Fixed — documentation that described behaviour the code does not have

• The README described v3-result-window as a retention period. It said "how long a finished v3 result stays retrievable". The value has exactly one read site: it is the deadline passed to the post-dispatch result poll, and when it lapses the execution is marked UNKNOWN (which is what v3ResultWindowBoundsThePolling pins). An operator reading "retention" and setting 5s would silently degrade every tool call longer than 5 s to UNKNOWN. The entry now says it is a polling deadline and names that consequence; the design doc, in both languages, says the same and states the 30 m default.
• The new repository primitive's contract omitted one of its four outcomes. beginSessionRelease's javadoc listed null / already-releasing / runtime_session_busy, but both implementations also throw runtime_session_not_ready for any other state — the case the contract test's "a terminal session is refused" branch covers. A caller reading the contract would expect a FAILED session to come back as a null CAS retry.
• MIN_V3_RESULT_WINDOW's stated reason was wrong. Both the constant's javadoc ("the smallest window that can complete a second poll") and the constructor comment ("below the first poll tick the window cannot complete even one retry") justify 1 s by the poll schedule, but the first retry is scheduled 100 ms out, so that reasoning would support a ~100 ms floor. The real reason — a suffix-less duration binds as milliseconds, so "30" meant as 30 minutes arrives as 30 ms — is now the only reason given.
• The HTTP layer claimed a guarantee it does not make. The comment said a mutation's own response "must describe the post-mutation truth". :start/:cancel do bypass the cooldown cache, but they can still join an in-flight lookup that a polling GET started before the mutation (the pre-existing putIfAbsent merge), so the response can describe the pre-mutation state. The comment now claims only what the code does: a mutation response never serves the cooldown cache.
• The design doc's Problem section said "three high-risk defects" while Decisions listed five entries. Both languages now name the two medium findings the change also fixes (bounded-force worker shutdown, and the LOST reclaim's pass budget), so the two sections agree.

Declined

• A longer lease for the pass-budget test. The suggestion inferred from round 1's comment that the drain needs more than 12 s, leaving the 40 s deadline under 3.3× margin. Measured, the budget path answers in 2.2 s; the ~23 s seen before round 1 was the reclaim waiting for a 12 s deadline to lapse, twice — which is exactly why the lease was raised. 40 s against 2.2 s is an 18× margin, and a longer lease would not change what the test asserts: its oracle is 0 < remaining < 4801 plus the runtime_broker_runtime_lost code, neither of which depends on the lease. The comment now states the measured numbers instead of implying a 12 s workload.
• Re-scheduling the cooldown eviction task. The task is one-shot and gated on the service clock, so a clock that jumps backwards leaves that entry in the map: until the clock passes the entry's deadline again the read side treats it as fresh, and after that the entry is simply retained — one small record per affected execution — because no second task ever runs. Both effects are bounded by the size of the jump and confined to the automatic-observation path: an explicit reconcile=true never reads the cache and a mutation response never serves it. The alternatives are worse — re-scheduling on a miss becomes a perpetual 1 Hz timer per entry under the injected clocks the cooldown tests use, and clearing the map in close() only frees memory the service's own death already reclaims.

Verification

• packages/sdk-java/runtime-broker: mvn clean test → 630 tests, 0 failures, 0 errors, 2 skipped; mvn checkstyle:check and mvn spotbugs:check clean.
• packages/sdk-java/managed-agent-server (fix-adjacent): EmbeddedRuntimeBrokerTest + ManagedAgentPropertiesTest → 19/19 green against the reinstalled jar.
• Five mutation proofs (row lock, settled-cache branch, freshest-record ternary, renewal-pool shutdown, poll cap ×2) were run against this tree and reverted from cp backups; the production diff is the v3PollDelayMillis extraction plus comment and javadoc corrections.

Self-audit round 3 (adversarial + undirected) — dispositions

Two passes over 20d8f48: the adversarial one reported 1 Critical + 3 Suggestions, the undirected one 1 Critical + 7. The undirected Critical is real and CI confirms it — round 2 broke the Windows leg. The adversarial Critical's main claim does not survive measurement, but two of its sub-points are correct. Everything below is fixed in 56c2096 except three declined items.

Round 2 broke windows-latest / Java 21 — fixed without waiving the platform

gh pr checks on 20d8f48: 612 tests, 2 failures, both on Windows only (ubuntu Java 11/17/21 and macOS Java 21 green).

exitHookReclaimsAWorkerStillInStartup — a regression round 2 introduced. Round 2 moved the decision of when the forked harness exits from a sleep(2000) inside it to harness.destroy() in the test, on the reasoning that SIGTERM runs the same shutdown hooks. That is POSIX-only: on Windows Process.destroy() is TerminateProcess, which runs no hooks, so the worker legitimately outlived the harness and the assertion failed (worker 6864 survived the broker JVM exit mid-handshake) — the exact rationale round 2 wrote into three @DisabledOnOs(WINDOWS) annotations elsewhere in the same commit. Fixed by keeping both properties: the harness now polls a sentinel file and exits itself with System.exit(0) once the test has observed the worker, so the window is still test-controlled and the hooks still run on every platform. No Windows waiver.

provisionDuringCloseNeverOrphansAWorker — a pre-existing Windows flake. a provision racing the close must hit the closed guard ==> expected: not <null>. The test widens the race window with a --ignore-term worker, which cannot wedge anything on Windows, so close() finishes in milliseconds and the racing loop can miss the guard entirely. The code is identical to af6691c, whose Windows leg was green, so this is timing, not a new defect. Fixed by asserting the guard only where destroy() is a signal (DESTROY_IS_SIGTERM), keeping the platform-independent half — no lingering child processes — asserted everywhere.

A real orphan window in the release path (undirected S5) — fixed in production code

stop(lease) removed the worker from owned and then handed the forcible escalation to a daemon thread, so for the whole 5 s grace the process was in neither owned nor starting. close() was covered (its executor.shutdownNow() interrupts the escalation, whose handler destroys forcibly), but a bare JVM exit was not: the daemon dies with the JVM and the exit hook's snapshot could not see the worker. A released worker that ignores SIGTERM, plus a SIGTERM to the Broker inside 5 s, stranded it — the same class of gap the starting set was added to close for the ready handshake.

stop() now moves the worker from owned to starting under the same lifecycle lock terminateAll() snapshots with, and leaves it when the escalation finishes (or immediately in the rejected-execution fallback). Doing the move under that lock also removes the microsecond gap between the two set operations, during which the worker was in neither set — the same shape as the spawn-and-register window the lock already closed. New forked test exitHookReclaimsAWorkerInsideItsReleaseGraceWindow releases a wedged worker and exits the harness JVM immediately, inside the grace window. Proof: without the tracking, it fails with worker 20917 survived a JVM exit inside its release grace window (re-verified after the locking change: worker 25506 survived…).

The stress test's reported Critical — main claim refuted, two sub-points accepted

The adversarial pass predicted that dropping the FOR UPDATE from beginSessionRelease would leave the whole module green, because round 2's even rounds commit the admission before the release starts and its odd rounds are won by the release almost every time — making the new "both interleavings occurred" assertion satisfiable by the test's own latch ordering.

Measured on this tree the prediction does not hold: with selectSession(…, true) → false, concurrentCrossProcessAdmitAndReleaseNeverContradict failed 3 runs out of 3, with 5, 3 and 3 contradictions. The raced arm reaches the dangerous interleaving in roughly 1% of rounds. Two sub-points are correct and fixed:

• The counters assertion proved less than its wording claimed. It shows both arms ran; it cannot show a contradictory interleaving was reached, since the even rounds guarantee admitWins > 0 by construction. The javadoc, latch comment and assertion message now say what the counters prove and name the raced arm as what detects a missing row lock.
• Round 2 halved the raced rounds. The odd/even split cut the tight race from 200 rounds to 100. Rounds are now 600 — 300 raced, 300 sequential — above the original count, still ~1.9 s.

Declined: the suggested package-private hook inside the release transaction to make the pin deterministic. That is a test seam in core repository code for a property the raced arm already detects at ~97% per run, and the transition's in-transaction re-check is separately pinned deterministically by serviceReleaseRejectsARacingCrossProcessAdmission.

Fixed — smaller items

• The backoff's inner shift clamp was unpinned (adversarial S1). v3PollDelayMillis is Math.min(2_000, 100L << Math.min(attempt, 6)); the round-2 test asserted attempts 0–7, where the outer cap hides the clamp, and a 30-minute window reaches attempt ~900. The test now sweeps 0–1000 and requires every delay inside [100 ms, 2 s]. Proof: 100L << attempt fails at backoff left its bounds at attempt 57: -4035225266123964416, with the other two v3 tests green.
• A comment described a synthetic input as a production threat (adversarial S2). No repository can produce the pair the settled-cache pin injects — both backends refuse a CAS whose expectation is terminal and always bump the version — so the version comparison already prefers the cache for every reachable input and the branch is defence-in-depth. The comment now says that, and says the stamped version is synthetic; the branch and the pin stay.
• A failed assertion could orphan the harness's worker on the runner (adversarial S3): the harness-exit assertion ran before the worker cleanup and the finally destroyed only the harness. The cleanup moved into the finally.
• The design doc misdescribed the pre-fix LOST reclaim (undirected S2). Both languages said the old code "repeated its bounded 100-row passes with no budget", which is what an intermediate state of this branch did, not what main does: verified at the merge base, cleanupLost called recoverLost once per phase with no loop, so a generation larger than one pass stayed LOST and answered runtime_broker_runtime_lost on every later attempt. Corrected in English and Chinese.
• A test still carried the rationale round 2 retracted in production (undirected S3): v3ResultWindowBelowTheFirstPollTickIsRefused is now v3ResultWindowBelowTheFloorIsRefused, with its javadoc restated in terms of the suffix-binds-as-milliseconds reason.
• An assertion message repeated a guarantee round 2 removed from the production comment (undirected S4): RuntimeBrokerHttpServerTest now says "a mutation response must not serve the cooldown cache" rather than "must describe post-mutation truth".

Declined

• The production seam for a deterministic row-lock pin — see above.
• Undirected S6 (low confidence, explicitly asking for the experiment): same subject as the adversarial Critical; the experiment is reported above.
• Undirected S7 (annotation style): the fully-qualified @org.junit.jupiter.api.condition.DisabledOnOs(…) in FakeAttestationWorkerTest matches the pre-existing annotation on its sibling ignoreTermSurvivesSigterm; rewriting both is unrelated churn.

Verification

• packages/sdk-java/runtime-broker: mvn clean test → 631 tests, 0 failures, 0 errors, 2 skipped; mvn checkstyle:check and mvn spotbugs:check clean.
• packages/sdk-java/managed-agent-server (fix-adjacent): 19/19 against the reinstalled jar.
• Three mutation proofs this round (FOR UPDATE ×3 runs, shift clamp, release-grace tracking), all reverted from cp backups.
• Windows cannot be exercised locally here (macOS). The two Windows fixes remove the platform dependency rather than waive it, so the windows-latest / Java 21 leg on this push is the real check; I will report its result.

Self-audit round 4 (adversarial + undirected) — dispositions

Two passes over 56c2096, including a line-by-line review of the only production change in that commit (LocalProcessRuntimeProvisioner.stop()). No Critical from either pass. Eleven Suggestions between them, all fixed in 4e91dc5. The findings are now at comment/doc/robustness level rather than behaviour level, which is the convergence signal this loop was looking for.

Both passes cleared the new production change

stop() moving the worker from owned to starting under lifecycle was traced through every interleaving the reports could construct: stop() against close(), against the exit hook, against start() for the same ownership key, and against a second stop() for the same lease. lifecycle is a leaf lock on all three paths that take it (start() spawns inside it, terminateAll() only snapshots inside it, stop() only touches two concurrent maps), the destroy and the 5 s wait both happen outside it, so neither a release nor JVM exit can stall on the other. The terminated check is correct in both directions: if stop() wins the lock the later snapshot sees the worker in starting; if terminateAll() wins, its snapshot already took the worker from owned; after close() returned, owned is empty so stop() finds nothing. No interleaving leaves a worker undestroyed, and OwnedProcess identity means a removal cannot evict a different worker.

Fixed — tests

• Two comments described a mechanism that is not what happens. The release-grace harness said its exit kills the daemon escalation thread, "so the hook is the only thing left that can reclaim" the worker. System.exit runs shutdown hooks before halting, so during the hook's own 5 s grace the daemon is alive and fires destroyForcibly() within ~0.1 ms of the hook's. What the snapshot actually buys is keeping the JVM alive until the reclaim completes and issuing the destroy itself — which is still exactly what the fix provides, and the mutant confirms it (untracked, the JVM halts at ~T+0.2 ms, the daemon dies mid-wait, the worker survives). The test-side comment said its 20 s poll "only has to outlast" the hook's grace window; the poll starts after harness.waitFor returns, i.e. after the JVM halted, so it waits out pid reaping. Both reworded.
• A platform that cannot reach an assertion reported a pass. provisionDuringCloseNeverOrphansAWorker's guard assertion was the only check on "Managed Runtime provisioner is closed." anywhere in either module, and making it conditional on DESTROY_IS_SIGTERM left the Windows leg green for a guard it never exercised (measured on macOS: deleting the guard turns both the new guard test and the orphan test red — 2 failures; on Windows the race cannot reach the guard at all, which is precisely why that half is now an assumption rather than a conditional assertion). It is now its own test, provisionAfterCloseHitsTheTerminatedGuard, using assumeTrue, so Windows reports skipped; the shared race fixture moved into a helper and the platform-independent orphan check still runs everywhere. (The os.name derivation itself was verified correct across Linux, macOS, Windows, WSL and Cygwin.)
• Both forked tests killed a bare pid in their cleanup. A recycled pid would turn that into SIGKILLing an unrelated process on a shared host, and the same absence of an identity check made the liveness poll a false-red risk. Both now require the pid to still name a node process. The two sides deliberately differ when the command line cannot be read: the poll treats it as still alive (so an unreadable command line cannot produce a false green), the cleanup treats it as not ours (so it never kills a stranger).
• The cooldown tests' clock had a data race. MutableClock.now was a plain field written by the test thread and read by the HTTP handler and the coordination scheduler when judging cooldown expiry — no happens-before edge, and the only nondeterminism in those two assertions. Now volatile.
• The backoff sweep's range was tied to the default window, which has a floor but no ceiling, so a larger configured window reaches attempts the sweep did not cover. It now runs 0–100 000 (still microseconds) and the javadoc no longer justifies the range with the default.

Fixed — production, small

• close() left an asymmetric cleanup: it cleared owned and shut the executor down, but a queued escalation task then never runs, and its finally was the only thing removing that worker from starting. No wrong behaviour (the process is already destroyed by terminateAll, and terminated blocks new entries), just a retained reference to a dead Process. close() now clears the set too.
• The lifecycle block in stop() cannot be pinned from outside the lock, and the report confirmed the mutant (drop the monitor, keep both statements) leaves all 631 tests green because the harness releases before it exits. Its comment now says the guarantee is structural, mirroring the spawn-and-register javadoc, so nobody assumes it is test-pinned. It also records that the !terminated half is an equivalent mutant.
• The starting field's invariant comment described only one of its two membership windows (pre-issuance handshake); it now names the post-release grace window as well, since that is where a future "tidy-up" would go wrong.

Fixed — documentation

• The design docs still said "a 200-round cross-process stress" in their Validation section while the code runs 600 (300 raced). The same commit that raised the count edited these files and missed it. Both languages now state the split, and the release primitive's coverage is enumerated rather than called "four outcomes".
• The README's opt-in guidance for a non-loopback bind was weaker than the code's own refusal message. The server says pass allowNonLoopback only behind a TLS-terminating layer that authorizes callers; the README said "that interface is already restricted to trusted peers". On a face that serves one global bearer token over plaintext HTTP with no per-tenant authorization, a network restriction alone still puts that token on the wire. The README now asks for the TLS-terminating, authorizing layer and says what a token leak costs.
• The release transaction's statement order is load-bearing and was undocumented. Under InnoDB REPEATABLE READ the guarantee additionally requires that the locking session read stays the transaction's first statement and the plain execution read stays its first consistent read, so the read view is built after the row lock is held. Moving any read ahead of the lock would silently reopen the original race, and H2 (MODE=MySQL, RC semantics) would not show it. There is now a comment saying exactly that.

Also noted, no change

• MutableClock's remaining uses and the 600-round stress were checked for deadlock and leaked rounds: the 2-thread pool gives each round two distinct threads by construction, admitDone.countDown() sits in a finally, and an admission that throws still trips the exact-error-code check. Measured cost at 600 rounds: 0.82 s.
• The win-counter comment was tightened once more: admitWins > 0 is guaranteed by the even rounds' latch, so what the assertion really guards is that the raced half still produces committed releases.

Verification

• packages/sdk-java/runtime-broker: mvn clean test → 632 tests, 0 failures, 0 errors, 2 skipped; mvn checkstyle:check and mvn spotbugs:check clean.
• packages/sdk-java/managed-agent-server (fix-adjacent): 19/19 against the reinstalled jar.
• One mutation proof: deleting the terminated guard in start() turns provisionAfterCloseHitsTheTerminatedGuard and provisionDuringCloseNeverOrphansAWorker red, so the split did not weaken the pin. Everything else this round is comments, documentation, test hygiene, or close() clearing a set no test can observe; the stop() tracking fix from the previous commit remains pinned by exitHookReclaimsAWorkerInsideItsReleaseGraceWindow.
• windows-latest / Java 21 passed on 56c2096, confirming the two Windows fixes from the previous round; the new skipped-guard test will show as a skip there rather than a pass.

Review-thread dispositions — all 18 open threads answered

Every open thread now has a reply with its disposition and, where a fix landed, the commit and the mutation proof. Threads are left unresolved so the maintainer can close them in one pass.

Fixed — Critical

• R3-1 (a release completing over a live execution on the RELEASING re-entry) → 445964f46e, with releaseRetryOverAPersistedReleasingRowRefusesALiveExecution. This was the one merge-blocking finding on the PR: it reproduced end to end in the three-arm rig, and it is a regression this branch introduced when the unconditional pre-check moved into the transition that the re-entry path never reaches.

Fixed — production

• R3-5 (the closed-provisioner refusal retyped to a non-retryable 503 "recovery is blocked" for managed context) → 6460cd9387, with provisionRacingCloseStaysRetryableForManagedContext; removing the rethrow fails it.
• R3-2 (the listener failure message hid the real refusal, e.g. an over-long token) → 6460cd9387.
• R3-4 (v3-result-window above 30 m buys nothing because the shipped client stops observing at its own fixed 30-minute deadline) → 6460cd9387, recorded in the deployment README.
• R3-3 (the cooldown cannot reach the shipped reconcile=true poller, so the fix(runtime-broker): cross-process admit/release race, shared single-thread renewals, single-token HTTP #13183 fan-out on that path is unchanged) → documented in both design-doc languages at 445964f46e; the fix needs a decision, deferred to runtime-broker: remaining test-oracle gaps and the reconcile=true fan-out decision (deferred from #13214) #13275 item 4 with both options written down.

Fixed — tests

• R2-6 and its fix-induced follow-up: the missing SIGTERM mirror arm was added, and then fixed for real — it destroyed the worker before reading the ready line, so SIGTERM's default disposition could kill it before any handler existed. It now waits for readiness and carries @DisabledOnOs(WINDOWS). Wedging the fake's default arm leaves the old body green 3/3 and turns the new one red.
• R3-7 (POSIX-only tests with no OS gate): three @DisabledOnOs(WINDOWS) annotations, and the provision-race guard became its own assumeTrue-gated test so the Windows leg reports skipped instead of passing over a guard it never reached.
• R3-9 (the only concurrent witness never asserted the reported ordering occurred): instrumenting it showed admission had won 0 of 200 rounds. Now 600 rounds, half of them holding the release until the admission commits, both outcomes asserted; dropping FOR UPDATE from beginSessionRelease fails 3 runs out of 3 with 3–5 contradictions.
• R3-13 (contract javadoc claimed atomicity its single-threaded legs cannot show) → narrowed at 6460cd9387.
• R3-14 (the busy leg could not distinguish a session-scoped from a binding-scoped predicate) → sibling-session leg added on both backends at 6460cd9387.
• R3-8 → renamed to drop the claim nothing asserted, and extended to null / ZERO / -1ms / 999ms; the placement half stays unpinned and is declined in the thread.

Declined, with evidence in each thread

• R1-16 — both classes carry a class-level @Timeout(180), so an unbounded join() fails the test with a report rather than hanging the job.
• R2-3 — the one-shot cooldown eviction under a backwards clock is bounded by the step and confined to the automatic-observe path; re-arming becomes a perpetual per-entry timer under the injected clocks the cooldown tests use.
• R3-6 — the in-memory type gate: relaxing it was tried, and the JDBC half of the shared contract fails with Release requires the same DataSource, so both gates are the same precondition. Tracked in test(runtime-broker): InMemory repositories drift from JDBC semantics, hiding JDBC-only races from service tests #13204.

Deferred to #13275 under this repo's ~5-review-round rule (three bot rounds plus five self-audit rounds; the last waves land Critical fixes only)

• R3-10, R3-11, R3-12 — test oracles that cannot currently fail for the property they are named after. Each has its concrete fix written down in the issue, and none is a production defect.

Verification at 6460cd9387: runtime-broker 634 tests / 0 failures / 2 skipped with checkstyle:check and spotbugs:check clean; managed-agent-server fix-adjacent 19/19 against the reinstalled jar. windows-latest / Java 21 is green again after the two Windows fixes in 56c209634b.

Self-audit round 5 (undirected + adversarial) — dispositions and convergence

Two passes over 4e91dc5. The undirected one independently re-derived the RELEASING re-entry regression — the same defect as bot thread R3-1, fixed in 445964f while it was running — and found one new Suggestion. The adversarial one found no Critical and three Suggestions. Everything is fixed across 608f6d6 and 51a526d.

The Critical both passes converged on

The undirected pass traced the same regression from the other direction: at 4e91dc5 releaseSession kept only the process-local hasActiveControl() test, and transitionSessionToReleasing returned an already-RELEASING row before reaching beginSessionRelease, so a release retry over such a row told the worker to release and marked the row RELEASED over a live execution. It also verified what the merge base did — hasActiveControl() || executionRepository.hasActiveByRuntimeSession(...) on every attempt — i.e. this branch had narrowed an existing safety check, which is exactly how R3-1 and my three-arm rig characterised it. Fixed in 445964f; deleting the re-check fails releaseRetryOverAPersistedReleasingRowRefusesALiveExecution.

That makes three independent sources converging on one defect (bot review, the verification rig, this pass) and no other Critical in five rounds.

Fixed

• An Error was cached as an UNKNOWN observation (undirected) → 608f6d6. The cooldown stamp ran for any throwable, so an Error escaping the lookup was served to every cooled observer arriving within the next second as a successful UNRESOLVED reconciliation — HTTP 409 runtime_broker_execution_unknown, retryable=false, with no trace of the internal failure — while both neighbouring paths (the takeover handle, the HTTP observation) explicitly refuse to downgrade an Error. The stamp now skips them; an ordinary failed lookup still cools, which failedUnknownLookupCoolsDownLikeACompletedOne pins. Proof: removing the filter fails the new test (expected: <2> but was: <1>) and leaves the failed-lookup test green, so the filter excludes Error and nothing else.
• A fix was described as doing something it does not (adversarial S1) → 51a526d. The previous commit's comment claimed the managed-context refusal fix keeps a rolling restart from "permanently failing the turn that raced it". RuntimeBrokerService blocks recovery for any managed-context provision failure, retryable or not, so that claim was false. What the fix actually preserves is the refusal's own message and retryable flag for callers that provision directly, and it stops reporting "recovery is blocked" when the provisioner was merely closed. The production comment and the test's javadoc now say that.
• Both forked tests counted a Linux zombie as alive (adversarial S2) → 51a526d. ProcessHandle.isAlive() reports a zombie as alive, and on Linux a zombie's /proc/<pid>/exe readlink fails, so the identity filter's unreadable-command fallback (orElse(true)) would have held alive for the full 20 s poll — a false red on a runner whose PID 1 does not reap. The module already owns the fix for this in ProcessTrees.running(); both sites use it now. The identity filter itself was verified safe on all three platforms (Windows QueryFullProcessImageNameW, Linux /proc readlink with a cmdline fallback, and macOS empirically through the discovery filter, which uses orElse(false) and would have failed the run that reported green).
• The backoff's tail was pinned by bounds instead of by value (adversarial S3) → 51a526d. An in-bounds schedule dipping back to 100 ms on every other attempt — which restores the 10/s polling the backoff exists to remove — passed the bounds version and the 0–7 list. The tail is now asserted equal to 2 s for attempts 8–100 000. Proof: that exact mutant fails at the backoff left the cap at attempt 9 ==> expected: <2000> but was: <100> while the wall-clock test stays green, and an unclamped shift still fails at attempt 57.
• The race helper could leak a wedged worker, and could not distinguish a wedged close() from a lingering child (adversarial, reported under "no finding") → 51a526d. It now closes the provisioner in a finally and asserts the closer thread finished after the join.
• The zh-CN design doc was missing the EN paragraph's closing clause about the un-fixed fan-out (undirected) — fixed in 445964f. That clause is the documents' only record of a known limitation left open, so a Chinese-only reader would have concluded finding 3 was fully fixed.

Convergence

Round 5 is the last full round under the stated rule (Suggestions in scope through round 5, Criticals only afterwards). Its yield: one Critical that three sources converged on and that is fixed, plus five Suggestions of which four were claim/oracle corrections and one a convention fix — no new production race, deadlock, security or data-loss defect. Both passes reported the areas they attacked and found solid, including the four blind spots round 5 was aimed at:

• The seam with the harness-drain feature that arrived from main mid-branch. The drain funnels into this PR's primitive (drainClaimedBinding → drainSessions → releaseSavedSession → releaseSession → beginSessionRelease) with the binding already DRAINING, which RuntimeAdmission.requireRelease accepts and RuntimeHarnessDrainTest drives against JDBC/H2. A busy session under drain is refused before the primitive is reached (the drain's own hasActiveByBinding pre-check plus requireReady's drain refusal), so beginSessionRelease's busy branch is unreachable there by construction. The LOST drain loop and a live drain claim are serialised by the binding's operation claim, so their interleaving is a claim conflict, not state corruption.
• The durable defaults flipped on by feat(managed-agent): default durable local-process and trusted reboot recovery on #13211. With durable-local-process=true, store != null, so the exit hook is not registered and close() skips terminateAll(): the shutdown-escalation half of this PR is inert in the shipped default configuration, and the release-grace tracking is non-durable-only because release() calls stop() only when store == null. The design doc's "for non-durable provisioners" wording already says this.
• The SQL and lock ordering. Placement domain → binding → session → execution everywhere, with the new primitive taking a same-order subset (session FOR UPDATE, then a non-locking execution read, then an update of the row it already holds), so no cycle with admitExecution, completeSessionRelease, recoverLost or the renewal ticks. Both "active" predicates match their table's state enum, the new per-pass reads are index-covered, and the 409s thrown inside the transaction propagate unretyped, so code, message and retryable are byte-identical to the conflict(...) they replaced.
• The HTTP face. Constant-time token comparison, the loopback refusal before HttpServer.create on the same InetSocketAddress (no re-resolution seam), all three observe(...) call sites updated, and no response path that can carry the token.

Remaining Suggestions are recorded in #13275: three test oracles that cannot currently fail for the property they are named after, and the reconcile=true fan-out decision. I am treating the audit as converged.

Verification

• packages/sdk-java/runtime-broker: mvn clean test → 635 tests, 0 failures, 0 errors, 2 skipped; mvn checkstyle:check and mvn spotbugs:check clean.
• packages/sdk-java/managed-agent-server (fix-adjacent): 19/19 against the reinstalled jar.
• Two mutation proofs this round (the Error-cache filter, and the in-bounds tail mutant the old bounds assertion could not see), both reverted from cp backups.

The red Hosted process fault gates / MySQL 8.4 / Java 21 check on 51a526d is #13255, not this PR

The job failed in HostedWorkspaceToolTurnIT.packagedHarnessUsesSavedWorkspacesThroughRealBrokerWorkerAndSqlStore:

qwen serve: Hosted Harness turn 095f5aac-… failed: Error: Hosted Workspace profile refused a tool call.
qwen serve: Hosted Harness turn de176687-… failed: Error: Hosted Workspace profile refused a tool call.
AssertionError [ERR_ASSERTION]: {"error":"hosted_turn_recovery_required","code":"hosted_turn_recovery_required"}
409 !== 200
    at javaLoad (integration-tests/helpers/hosted-workspace-tool-turn-driver.ts:513)

Three things place it outside this PR:

• The same failure, with an identical signature, is on unrelated branches right now — including a docs-only branch, which cannot have caused it: docs/h3-shell-monitor-design @ 8173837 (8 tests, 1 failure, 18.69 s) and fix/managed-agent-quality-hardening @ eb6c54c (8 tests, 1 failure, 20.85 s), against this branch's 18.84 s. Same method, same two refused a tool call turns, same hosted_turn_recovery_required 409.
• The refusal is raised client-side in the driver's tool-call validation (packages/cli/src/serve/hosted-workspace-tool-turn.ts:726), which fires on an undeclared tool name, a duplicate callId, wasOutputTruncated or hadIncompleteArguments — none of which this PR touches. The managed-agent-server logged no WARN or ERROR in the failure window, so no broker 409/503 preceded it. This PR changes only packages/sdk-java/** and documentation.
• The same job passed on this branch at af6691c and 20d8f48, and HostedWorkspaceToolTurnIT was 8/8 green in the latter's failsafe report.

It is tracked as #13255, which has an in-flight fix branch. Re-running the failed jobs (attempt 2) to confirm.

Re-ran: Hosted process fault gates / MySQL 8.4 / Java 21 is green on 51a526d (attempt 2, job 111201300466), confirming the earlier red was #13255 rather than this PR.

Worth recording, because the first attempt hid more than one check: it failed at step 9, which skipped steps 10-15 — including step 12 Run Runtime Broker fault gates and both owner-failover E2Es. Those had therefore never executed on any commit of this branch after af6691c (the intermediate runs were superseded by newer pushes). Attempt 2 ran all of them to success:

9   success  Verify Hosted Java, Spring and MySQL processes
10  success  Check that every Hosted integration test class ran
11  success  Check Hosted latency measurements
12  success  Run Runtime Broker fault gates
13  success  Install MySQL binaries for the failover E2E
14  success  Run in-flight owner failover E2E
15  success  Run continuation owner failover E2E

So the concurrency and fault-injection gates most relevant to this PR — the storage fault gates against the release transition, and the in-flight and continuation owner-failover E2Es — have now run green on the final head, alongside the Java matrix (ubuntu 11/17/21, macOS 21, Windows 21), MariaDB, Real daemon E2E and the Node lanes. Check tally: 21 pass / 0 fail / 8 skipped, with only the review-pr automation still pending.

Bot review round 4 (R4-1 … R4-3) plus two round-1 threads that a pagination window had hidden — all answered

Fixed in b81f21d. The Critical was in this branch's own hardening record rather than in its code.

R4-1 (Critical) — the design doc certified a renewal isolation the code does not provide. Verified against the code, not just the report: BindingRenewal.stopAndGet()/close() and DispatchRenewal.close() are synchronized on the renewal instance, the same monitor renew() holds while parked inside its JDBC call, and three of the callers are tasks on the single-thread coordination scheduler (the provisioning fence, the reclaim fence, the reconcile re-entry). No renewal JDBC call sets a statement or socket timeout. So a stall long enough to park a tick can still block the coordination thread behind that monitor — and with it every other execution's polls, fences, retries and cooldown evictions. The claim I wrote in round 1 ("it no longer delays coordination work") and managed-runtime-broker-service-core.md's stronger one ("a stalled storage call cannot queue coordination work") were both false in general. Both documents, in both languages, now state what the split buys (a parked tick is no longer itself the coordination thread, which at the merge base it was) and what it does not, and name the two changes that would close the rest as tracked in #13275 item 5 rather than done here. No code change: at the merge base the parked tick was the coordination thread, so the split strictly narrowed the harm — the defect was the certification.

R4-2 — the 2 s poll cap was described as a bound on pick-up lateness in the design doc (both languages) and in the v3PollDelayMillis javadoc this branch added. A single-thread executor guarantees a task starts no earlier than its delay, never within it, and every execution's rounds share that thread, so lateness under concurrency is the cap plus the queue wait. All three corrected.

R4-3 — the 2026-09-23 process-adoption page still promised stop/close never escalate past SIGTERM, which this branch's bounded-force shutdown falsifies for non-durable workers. Corrected in both languages with a cross-link to the hardening page; the neighbouring "a Broker crash can still orphan the worker" sentence stays, since a SIGKILLed broker runs no hook. I did not adopt the suggested "durable workers are never force-killed here" phrasing, because the drained path does escalate.

R1-2 and R1-3 were never answered: my earlier thread sweep used a last:30 window on a PR that has 47 threads, so the two oldest fell outside it. Both are now answered in-thread. R1-2's unit trap is guarded (1 s floor, validated before the provisioner exists, four shapes tested) and documented; the missing piece — nothing logged the resolved window — is fixed by logging it at startup, so a suffix-less 30 shows up as PT0.03S. Its 24 h ceiling and ≥60 s floor are declined as config policy, and R1-3's latency half is already fixed by the 2 s cap (now pinned deterministically for attempts 0–100 000); its deadline-boundary case is recorded as #13275 item 6.

Verification: runtime-broker 635 tests / 0 failures / 2 skipped with checkstyle and spotbugs clean; managed-agent-server fix-adjacent 19/19. (The first run of the consumer suite failed to compile against a stale qwencode jar in the local Maven repository — this branch's own qwencode was installed and it went green; no source issue.)

Real-stack verification, round 2 — head 5cd1c832f4

Verdict: from the real-stack side this is ready to merge. Round 1's F1 (bot R3-1) is fixed with exactly the check I proposed, and verified end to end. The new grace-window fix also holds on a real worker. Every round-1 scenario was re-run on the new head, and nothing regressed.

One non-blocking suggestion is new (G1): the InnoDB statement order that the release fix depends on is pinned by no test that runs in CI. A mutant that breaks it reopens the race on MySQL and passes all 635 unit tests plus the existing MySQL IT. A 68-line IT that kills it is attached.

The head did not change during the run; the PR adds 10 commits on top of af6691ce1f with no main merge. A trial merge with today's main (2c591ecc08, which only touches managed-agent-server/README.md inside sdk-java) is clean. CI is 21 pass / 0 fail; review-pr is still running. reviewDecision still shows the bot's older CHANGES_REQUESTED.

Re-run on the same rig

Rig: real MySQL 8.4.7; separate Broker JVMs; the bundled worker; the real managed-agent-server jar. The arms are base 5130c1a734, round 1 af6691ce1f, and round 2 5cd1c832f4, each identified from its own bytecode.

Spring startup matrix (14 cases × 3 arms): same posture as round 1. The startup log now prints the resolved window (listening at … (v3 result window PT30M)). PORT=70000 fails with Runtime Broker listener could not start: port out of range:70000; round 1 named no cause (R3-2).

Test suites:

• runtime-broker: 635 tests, 0 failures (2 skipped); SpotBugs 0; Checkstyle 0.
• MySQL IT profile: 7/7 on a fresh database.
• managed-agent-server fix-adjacent: 19/19.

New fix verified: the release grace window (56c209634b)

Setup:

• The worker: the real bundled worker, launched with a preload that drops every SIGTERM listener, modelling a wedged graceful shutdown.
• The release: the proxy drops the service's attestation reply, so provisioning fails and releaseQuietly → provisioner.stop() runs.

G1 — the load-bearing InnoDB statement order is pinned by no test (suggestion)

The new comment in JdbcRuntimeBindingRepository.beginSessionRelease is right: under REPEATABLE READ the execution read must be the transaction's first consistent read, taken after the FOR UPDATE. I checked it with mutant M1, which adds one consistent read (hasActiveByRuntimeSession, result ignored) ahead of the locking read (mutant-m1.patch).

Candidate: candidate-innodb-order-it.patch, JdbcRuntimeBrokerMySqlIT.releaseTransitionSeesAnAdmissionThatCommittedWhileItWaited, 68 lines, Checkstyle clean. How it works:

1. A holder connection takes the Session row FOR UPDATE.
2. fixture.prepare(...) queues on that lock first, and beginSessionRelease queues behind it. Both waits are confirmed in information_schema.innodb_trx, scoped to the IT's schema.
3. The holder commits.
4. Expected: the admission is PREPARED, the release gets 409 runtime_session_busy, and the Session stays READY. M1 fails with Expected ExecutionException … nothing was thrown: the release went to RELEASING over the admitted execution.

It needs nothing new in CI; it runs in the existing MariaDB mysql-integration job.

Notes

• N1 (pre-existing, unrelated): JdbcRuntimeBrokerMySqlIT.managedRecoveryKeepsPinsUntilTheOriginalHolderCleanupCompletes fails with Runtime credential key is unavailable when the profile is re-run against a database it already ran on. Base fails the same way on its second run. The CI job uses a fresh service container, so this does not affect CI.
• Not re-exercised on the real stack this round, so pinned only by the PR's unit tests: R3-5 (the closed-provisioner refusal stays retryable) and the Error-not-cached change in lookupOnce.

Evidence: assets-pr13214@31717330. It holds the round-2 harness additions (grace.mjs, the ignore-term preload, the extended Spring matrix, the IT runners) and the raw results and logs. Round-1 material stays under pr13214/.

@qwen-code /review

Qwen Code review request accepted. Review is queued for an available runner; follow the workflow run for progress. A command-triggered review is not listed under the checks of this PR; the result is posted here as a review when it finishes.

Real-stack verification, round 3 — bot round-6 findings re-checked (head still 5cd1c832f4)

Verdict: unchanged from round 2 — ready to merge from the real-stack side. The head has not moved since round 2, so this round checks the four [probe] findings the bot recorded in round 6 (none Critical, all deferred) against the real stack, and re-checks the PR against today's main.

One of those findings, D6-8, appeared to contradict my round-2 G1. Measured, both are right about different mutants, and the suggested fix is the same either way.

CI: 23 pass, 0 fail.

D6-8 vs round-2 G1: which read reordering does H2 catch?

The H2 data source in the test's URL shape runs at READ_COMMITTED (I measured: H2 2.4.240 isolation=2; nothing in runtime-broker overrides it). At that isolation every statement sees the latest commit:

• M2 (hoisted read): the read happens before the lock, so it misses an admission committed in between. It loses the race on H2 too, and the PR's stress test catches it. D6-8 is right.
• M1 (added read): this only matters under InnoDB REPEATABLE READ, where the first consistent read freezes the snapshot. On H2 the later read still sees the admission, so all 30 stress runs stay green. G1 is right.

So the comment "H2 would not show it" is wrong for a hoist and right for an added read. The rule it states is correct.

Suggested fix (one change addresses both findings): word the comment as "H2 runs READ COMMITTED: it catches a hoisted read, but not an extra read ahead of the lock, which only breaks under InnoDB REPEATABLE READ", and add the round-2 candidate IT, which kills both mutants deterministically. Patches:

• mutant-m2.patch
• mutant-m1.patch
• candidate-innodb-order-it.patch

The other round-6 probe findings

• R1-2 / D6-4 — confirmed on the real Spring server. QWEN_MANAGED_AGENT_RUNTIME_BROKER_V3_RESULT_WINDOW=1800, meant as 30 minutes, starts and logs (v3 result window PT1.8S); 3600 → PT3.6S; 500 is still refused. The same holds on the trial merge with main. The new startup log line makes the mistake visible, but nothing stops it. The bot's @DurationUnit suggestion would close it at the binding.
• D6-7 — confirmed by rounds 1 and 2. With one renewal parked in a real InnoDB lock wait, the healthy execution on another binding settles. With two parked, it ends UNKNOWN. The pool absorbs one stall, so the javadoc's "two simultaneous stalls" is off by one.
• D6-3 (the dispatch settlement blocks the coordination thread) and D6-6 (the managed-context drain exit is untested) — not reproduced here. They need a v3 publication or a managed-context placement, which this rig doesn't drive. They rest on the bot's unit-level probes.

Against today's main (6136786c0c)

main moved by 7 commits that touch sdk-java: the Druid pool swap (#13363), a V35 migration, and harness changes. Three of the files they change are also changed by this PR: ManagedAgentProperties, application.yml and EmbeddedRuntimeBrokerTest. Results on the merged tree:

• The trial merge is textually clean.
• qwencode, runtime-broker and managed-agent-server all build. The fix-adjacent EmbeddedRuntimeBrokerTest + ManagedAgentPropertiesTest pass 19/19.
• On the merged Druid jar, the Spring startup subset behaves exactly as on head: non-loopback refusal, ALLOW_NON_LOOPBACK opt-in, the 1 s floor, PT1.8S for 1800, and the port out of range message.

Evidence: assets-pr13214@b06d858c. It contains the StressRepeat driver (runs the PR's stress test N times per jar), the H2 isolation probe, the M2 patch, the merge build script and every run log. Rounds 1 and 2 stay under pr13214/ and pr13214/r2/.

@qwen-code /triage