feat(managed-agent): Recover Workspace holders after trusted local reboot (W0e-3) - #12869

W0e-3 E2E and regression report

Environment: macOS arm64, JDK 21, Node.js, bundled Runtime worker, isolated MySQL Community 8.4.11. No external model calls. The dedicated MySQL instance was shut down after validation; test processes were checked for leaks.

Passing evidence

All final targeted runs had zero failures, errors or skips. Counts refer to distinct cases; targeted reruns are not added twice. This is evidence across the complete runs and affected-case reruns, not a claim that an initially failing invocation was green.

What the new tests establish

• A same-host changed boot emits matching journal-loss and writer-stop evidence only after durable retirement of the original registration. Interrupted startup does not need an invented lease.
• More than 100 executions become terminal in bounded transactions. Unknown outcomes remain ABANDONED without replay or fabricated results.
• Cleanup failures and expired claims retain remaining pins. A committed holder clear can be retried, and an old cleanup cannot erase a different current holder.
• Actual MySQL and H2 row locking rejects a late acquire after the loss fence. Cleanup still uses original ownership after grants are revoked, the product Session is deleted, and Registry/mount data changes.
• The real escaped child is alive and its file grows after the restarted Broker has confirmed the original Runtime is lost. No replacement worker is launched.
• The independent scanner has a rotating cursor and does not overlap batches. Maintenance never consults the current Session resolver or provisions a replacement.

Failures found and corrected during validation

An initial MySQL run reused a fixed-prefix database from earlier verification; its generation assertion correctly failed. A fresh isolated database passed. The new MySQL fixture initially bypassed Spring's transaction proxy; adding its explicit creation transaction made the real SQL path pass. The initial escaped-writer test checked the test root instead of its actual Workspace; the directory was corrected and the growth baseline moved after confirmed loss. The two affected gates then passed. The first Stage F run passed all 38 unchanged gates; only these two fixture cases required rerunning.

Pending physical acceptance

A real Linux host reboot has not been run. macOS uses test-only host identity, the Spring worker chain uses a synthetic boot change, and MySQL tests use synthetic stop evidence. Native Linux identity and a real preserved-disk reboot with an escaped writer remain a separate acceptance gate. This PR remains draft until that gate can be performed. Hosted Turn continuation, public Workspace message enablement and hostile-tool containment are outside this slice.

中文

本地完成 Broker 407 项、Spring 145 项常规测试，14 项 MySQL 集成用例，以及 38 项既有真实进程故障门禁。新逃逸写入者的 2 项门禁修复夹具目录后通过；另有 1 项 Spring＋H2＋真实 worker 恢复链路通过。更新后的维护断言和事务夹具做了定向补验，未重复计入数量。build、typecheck、bundle、Checkstyle 和改动文档格式检查均通过，最终补验无失败或跳过。

验证期间发现并修复了旧测试库复用、原始 MySQL store 夹具缺少事务、逃逸写入者测试目录错误，并将增长测量起点移至 Broker 确认失联之后。真实 SQL 验证了分批终态化、清理重试、迟到 acquire 屏障和保留新 holder；真实子进程验证了 worker 退出后继续写入时不能复用原位置。

未执行真实 Linux 整机重启。 模拟 boot、MySQL 测试证据和真实 worker 链路分别报告，不能合称真实重启验收。PR 保持 draft，待专用 Linux 环境验证后再推进。

Please do not rebase or force-push to an active PR as it invalidates existing review comments. Note for future reference, the bots always squash all changes into a single commit automatically as part of the integration.

Rebased this draft PR onto W0e-2 commit 7236322a4, which now includes the W0e-1 transient attestation fix. New head: f8bf5d746. git range-diff shows the W0e-3 commit unchanged; the only rebase conflict in the stack was resolved in the paired W0e-2 design documents.

Validation of this exact stacked head: Runtime Broker 410 tests with 0 failures/errors and 1 conditional skip; managed agent server 145 tests with 0 failures/errors; 12 real-process reboot and durable-runtime fault gates passed; Checkstyle reported 0 violations. The earlier MySQL and broader fault-gate results remain recorded above; this rebase added only the W0e-1 Java fix and paired documentation, with no TypeScript changes. This PR stays draft until the physical Linux reboot acceptance gate and the documented pre-upgrade handling of historical seeded FAILED bindings are complete.

Real-host verification: the pending Linux reboot acceptance, run for real

Verified on a dedicated Linux host at f8bf5d74 (current head). I started at 66bde476; the stack was rebased while I was testing. The W0e-3 commit has the same git patch-id on both heads and the TypeScript tree is unchanged, so I kept the first results and re-ran everything that matters on the new head.

Summary

• The reboot recovery works on a real host. Four real reboots of the same machine (two systemctl reboot, two power cuts): every saved generation was reclaimed by the scheduled scan alone, 21 s after the host went down in the hands-off runs. The escaped writers stopped with the old boot, receipts kept their meaning, only the original holders were cleared, and no worker was started by the recovery.
• The controls stay blocked: option off, a foreign machine-id, worker-only death, systemd soft-reboot, and missing / damaged / foreign registration records.
• One thing to fix before anyone enables the option with Hosted turns (F1): the 5-second scan also observes healthy bindings, and a request that arrives during that observation gets 503 runtime_reconciliation_required. The Hosted Harness turns a failed acquire into a recovery-blocked Session. 6 of 871 healthy turns hit it. A 13-line candidate and a unit test are attached.
• With the option off (the default) nothing changes: 0 of 847 turns refused, and the saved generations stay blocked after a reboot exactly as before.

For the merge decision: from the real-host side the physical acceptance gate is met. I would fix F1 before merging (the change is small), add the two gap tests from section 5, and add the soft-reboot sentence. The pre-upgrade handling of historical FAILED bindings that the author lists as still open belongs to #12839 and was not part of this run. The merge order stays #12839 → #12865 → this PR.

All times are UTC, 2026-09-27.

1. Physical reboot acceptance

The JVM ran in Asia/Shanghai and MySQL in UTC on purpose. The lease checks in releaseLost were not affected.

2. Controls

One more control at f8bf5d74 (run R8, records changed while the server was stopped, then a real reboot):

Enabling the option without durable mode stops the server at startup with Trusted reboot recovery requires durable local-process provisioning.

3. After the recovery

4. F1: the scan refuses healthy requests, and a Hosted Session that hits it is blocked

What happens:

1. findRecoveryCandidates includes READY bindings (JdbcRuntimeBindingRepository.java:340), so each scan calls recoverBinding for every healthy durable binding.
2. recoverBinding registers a reservation in bindingOperations while it claims, observes, re-attests and rewrites the binding (RuntimeBrokerService.java:1549). It always completes that reservation with 503 runtime_reconciliation_required, also when the observation found the binding healthy (:1606).
3. ensureBinding hands the reservation to any request that arrives meanwhile (:825). That covers warm and the first acquire of a Runtime Session.
4. The Hosted Harness treats a failed acquire as possibly effective (hosted-workspace-tool-turn.ts:183-197) and raises HostedToolRecoveryRequiredError. The Session becomes recovery-blocked and every later prompt gets 409 hosted_turn_recovery_required.

The window is about 20 ms in every 5 s per binding, so a Session with a few hundred tool turns is likely to hit it. Nothing is wrong with the Runtime when it happens.

Candidate (diff, test): when the observation ends with a healthy adoption, complete the reservation with that result instead of the 503. Waiters have already resolved current authorization before they reach ensureBinding, so this is the same hand-over reconcileBinding already does. With it: 0 of 837 turns refused, the Hosted chain completes under the same delivery, all suites pass, and a real reboot still recovers (R6).

An alternative is to skip bindings that are live in this Broker process. That would also remove three row writes per healthy binding every 5 s, but the scan would no longer notice the death of an idle worker. Your call.

5. Suites on Linux and mutation results

26 single-site mutants of the PR's production changes at f8bf5d74, each run against the unit suites, then the fault gates, the real-worker IT and the MySQL ITs:

• 15 killed. 13 by the unit suites. Two only by the real-process tests: "the same boot counts as a reboot" (both LocalRebootFaultGateTest cases and WorkspaceRecoveryWorkerIT) and "cleanup reported without clearing the SQL holder" (WorkspaceRecoveryWorkerIT).

• 5 survive because a second guard covers the same path. The stale-version and other-claim checks in releaseLost mask each other; removing both is caught by WorkspaceRecoveryTest and ManagedAgentMySqlIT.

• 6 survive everything. Two of them guard safety properties:

  • rebooted() without the host check: another machine with another boot id then produces stop evidence.
  • the default recoverResources allowing managed cleanup, which the description says must fail closed.

  The PR code is right in both cases (see the controls), but no test would notice a regression. RebootRecoveryGapTest pins both: 2 of 2 pass at the head, and each fails under its mutant.
  The other four are smaller: claim ignoring the Runtime Session row, recoverBinding ignoring the expected generation, overlapping scan batches, and evidence matching any lease.

Full list and verdicts: mutants, pass 1, pass 2.

This PR has no Java CI: sdk-java.yml only runs for PRs whose base is main, and this one is stacked. main has moved three commits since the merge base, none of them under packages/sdk-java; git merge-tree is clean and the merged tree has one migration per version (V1 to V16).

6. Smaller notes

• Soft-reboot (docs). systemctl soft-reboot (systemd 254 and later, so Ubuntu 24.04) kills every writer but keeps the kernel boot id. The bindings become LOST without stop evidence and stay pinned until a full reboot. That is the safe side. One sentence in the README would save an operator some confusion.
• Flaky tests seen under load, none touched by this PR. ManagedAgentServerIntegrationTest failed in 2 of 11 runs with a different test each time. ConcurrencyStorageFaultGateTest.aDatabaseOutageDuringTheResultCommitNeverSettles failed in 1 of 8 runs.
• Context for the receipts. A foreground call sent through POST /executions that runs longer than 30 s is reported UNKNOWN even while a client polls it. This happens with the option off too, so it is not caused by this PR. It is why the in-flight calls in R1 and R2 were already UNKNOWN before the host went down.

What this run does not cover

Bare metal and x86_64 (the host is an aarch64 VM, see below), MariaDB, more than one Broker on the host, hostile same-UID tools, remote writers, restored or cloned disks, Kubernetes. I did not interrupt the server between the holder clear and the final retirement on the real host. That path is covered by the unit tests and the mutation results only.

Real-host verification, round 2: head 9a1de09e

Follows round 1 (head f8bf5d74). Same dedicated Linux host, same method. All times are UTC, 2026-09-28.

What changed since round 1. This push is a rebase. The W0e-3 commit and the W0e-2 commit below it have the same git patch-id as before. The base gained two #12839 commits and two merges of main: 4 Java files and 47 other files, among them the Hosted Harness Broker client. Because the TypeScript changed this time, I rebuilt the worker and Harness bundle from this head and re-ran everything on it.

Summary

• Round 1 reproduces at the new head. Power cut and systemctl reboot: every saved generation is reclaimed by the scan alone. Both controls (option off, foreign machine-id) stay blocked.
• New coverage: a crash in the middle of the cleanup. The server was killed twice while its holder clear was already committed and its final retirement was still in flight. After the restarts everything converged, and all 134 receipts kept their identity.
• F1 is still open. 5 of 470 healthy turns were refused with the option on, 0 of 411 with it off. The Hosted Session is blocked as before. The round-1 candidate applies to this head without change and removes the problem (0 of 433).
• To land on main the stack needs a rebase, because feat(managed-agent): Add W0e terminal recovery fences #12839 went in as a squash. Replaying the two commits onto main gives one conflict in a test file. The result passes all suites and recovers after a real reboot.

For the merge decision: unchanged from round 1. The physical acceptance gate is met at this head too. F1 is the one item I would fix before merging. The two gap tests and the soft-reboot sentence from round 1 are still open suggestions.

1. Reboots, controls and a crash inside the cleanup

How the crash was placed: a MySQL trigger delays the UPDATE that frees the placement slot, which only the final retirement issues. The product is unchanged. The server was killed with SIGKILL only while MySQL showed that statement as held and the holder row was already cleared.

2. F1 at the new head

The base brought a retry for prepare in the Harness (hosted-workspace-broker.ts), for transport errors only. acquire is not retried, and a 503 answer is not a transport error, so the chain in round 1 is unchanged: JdbcRuntimeBindingRepository.java:340 → RuntimeBrokerService.java:1549 / :1606 / :825 → hosted-workspace-tool-turn.ts:185-197.

Candidate for this head: diff, test.

3. What lands on main

• A plain merge of this head into main (6017f11d) conflicts in 9 files. That is the usual effect of a squash-merged base, not a problem in this PR.
• Replaying W0e-2 and W0e-3 onto main is clean except for ManagedAgentMySqlIT.java: main (feat(managed-agent): Make Session close, archive and delete durable operations (Stage D4) #12881) and this PR each add a test at @Order(8). Keeping both and renumbering one resolves it.
• The result has one migration per version, V1 to V17. This PR adds none.
• With that jar and main's bundle, a real reboot recovered 8 of 8 generations 22.0 s after the reboot command (R15).

Mutation re-check. The six mutants that survived every stage in round 1 still do at this head (unit suites of both modules, fault gates, real-worker IT, both MySQL ITs). The control mutant "the same boot counts as a reboot" is still killed by both LocalRebootFaultGateTest cases and WorkspaceRecoveryWorkerIT. RebootRecoveryGapTest passes 2 of 2 at the head and fails under each of its two mutants.

4. Notes

• A SETTLED receipt does not mean the file is on disk after a power cut. In R9 a tool call wrote a file 15 s before the power cut and its receipt is SETTLED / success. After the reboot the file was empty (ext4, the tool does not fsync). This is ordinary page-cache loss and not caused by this PR. It may deserve a sentence in the design, because recovery treats SETTLED as a known outcome.
• New base behaviour, checked on the real host. After a worker died before READY, a second Session of the same Workspace now starts on the same boot (the feat(managed-agent): Add W0e terminal recovery fences #12839 change to blocksPlacement). Both bindings were recovered after the reboot.
• Flaky under load. My first run on the rebased tree had 7 gate failures and one unit error while the rig server and its workers were running on the same VM and the machine under it was at load 50 to 60. A rerun on the quiet VM was green. In a same-conditions A/B the head had 0 failures in 4 gate runs and the rebased tree 1 in 4 (adoptedWorkerCanCancelItsOriginalActiveCall); the Broker code of both trees is identical except for BrokerValues.

What this run does not cover

Same as round 1, without the interruption item: bare metal and x86_64, MariaDB, more than one Broker on the host, hostile same-UID tools, remote writers, restored or cloned disks, Kubernetes.

Real-host verification, round 3: head 8c2b626c

Follows round 2 (head 9a1de09e) and round 1. Fresh dedicated aarch64 KVM VM (Debian 12), same method; everything re-measured at this head — nothing carried over from earlier rounds.

Verdict: findings — 62 pass / 1 fail (the fail is an adjudicated load flake, re-run 5/5 green). Verified head: 8c2b626c74f88fd39ffffd3e168099cab488041d (W0e-3), base 430a76eac9 (W0e-2 stack on main a765229c0a). All times UTC, 2026-09-28.

Previous-finding status (rounds 1–2 → this head)

Central claim — reboot recovery on a real host, re-measured at this head

Same dedicated-VM method as rounds 1–2, rebuilt here from scratch (KVM/qemu, Debian 12 arm64, kernel 6.1.0-53-cloud, JDK 21, MySQL 8.4, worker bundle built from this head; systemd unit, non-root, KillMode=process). Three runs:

F1 at this head (unchanged mechanism, re-measured)

The scan still observes every healthy durable binding every 5 s and still completes the reservation with 503 runtime_reconciliation_required; the Hosted chain still turns that into a recovery-blocked Session. Numbers: 2 of 417 turns refused (option on) vs 0 of 418 (off) vs 0 of 406 (candidate). The aligned-delivery probe shows the exact window: acquire delivered when the scan holds the binding → 503 at head, 200 with the candidate.

Suites and mutation re-check

• Head: broker unit 428, server unit 178, WorkspaceRecoveryWorkerIT 1, broker MySQL IT 3, server MySQL IT 14 — all green incl. checkstyle. Candidate: same, plus the F1 test and both gap tests pass 3/3 (at head the F1 test errors as designed).
• Mutation matrix (7 mutants from the r1/r2 survivor set + control, six stages each): M01 and M25 are now killed by the base's strengthened durable-startup fault gate — two of the six round-1 survivors now have in-tree pins. M11 (claim ignores the Runtime Session row), M16 (fail-open default), M19 (expected generation ignored), M21 (overlapping scan batches) still survive everything; adjudication unchanged from round 2 (M16 is the one guarding a safety property).
• One load flake: ProcessCrashFaultGateTest.aLostJournalEndsPollingWithoutReleasingTheWriterDomain failed once while the host ran the VM plus two Maven containers (runtime_provision_fenced surfaced before runtime_broker_runtime_lost); 5/5 green isolated at head. The mutation runs show the same two codes are order-sensitive at that boundary — worth a look from whoever owns the startup fence, not a blocker.

What changed since round 2 (scope of this round)

git range-diff r2-head..head: the W0e-3 commit touches only the IT fork timeout (180→600 s) and one test's @Order (8→11, base added tests). The base gained 56ada7e510 and 430a76eac9 (durable startup reconciliation termination; legacy startup fence limited to its slot) plus docs, and main advanced ~460 files. Everything above was re-run, not carried over.

Not covered

Bare metal and x86_64 (host is an aarch64 VM), MariaDB, more than one Broker per host, hostile same-UID tools, remote writers, restored/cloned disks, Kubernetes, claim-expiry retry as a dedicated real-host scenario (unit + mutation coverage only). Round 1's SETTLED-vs-fsync note stands unaddressed in the docs.

Methodology

Head worktree at 8c2b626c (pnpm install, npm run build, npm run bundle → worker bundle; Maven in a JDK 21 container → qwen-managed-agent-server jar, sha256 831f89e9…, candidate jar 8109c4aa… = head + the r2 F1 candidate re-applied with offset 58). VM: qemu/KVM arm64, Debian 12 cloud image, MySQL 8.4 in docker, server jar under systemd with the rig auth filter. Power cut = SIGKILL on the qemu process; reboot = systemctl reboot inside the guest. Harness scripts from the round-1/2 evidence bundle, unchanged except a guard in the crash watcher (skip kill while systemd reports MainPID 0 — a rig bug that ate round-3's first crash attempt; product untouched). Scripted assertions: assert.mjs re-validates the saved raw logs (38 checks), plus 8 scripted mutation verdicts, 5 isolated gate re-runs, 12 suite-stage exit checks. One scripted check failed once (the loaded gate run) and passed 5/5 isolated — disclosed above, counted as the flake it is.

Raw logs, harness and assertion script: in the round-3 bundle (assert.mjs re-validates the saved logs; results/ holds the raw per-run output).

For the merge decision: unchanged from rounds 1–2 — the physical acceptance gate is met again at this head, on a fresh independent rig. F1 remains the one item to fix before the option is used with Hosted turns; the attached candidate is now verified against three consecutive heads. The stack merges into current main (a77d80d19f) conflict-free; #12839 is already in, so the order left is #12865 → this PR.

Follow-up to the round 3 real-host review: F1 is fixed in 55ded4c31. A deterministic test first reproduced the healthy warm receiving 503 runtime_reconciliation_required while maintenance observed its binding. A successful, attested READY observation now completes that binding's waiting authorized requests with the adopted context; uncertain, lost, conflicting and failed observations remain blocked. The test now passes for concurrent warm and acquire, and a separate regression pins the fail-closed behavior when a managed provisioner has no holder-cleanup callback.

Because #12865 was squash-merged, I replayed the unchanged W0e-3 commit onto current main (the stable patch-id matches the original), then pushed the two-commit branch with an exact-SHA force lease and retargeted this PR to main. Its diff is now W0e-3 plus this fix, without the merged W0e-2 changes. The bilingual design and server README now record the independent Linux reboot acceptance at 8c2b626c, distinguish that evidence from current-head tests, and clarify soft reboot and file durability limits.

Current-head verification: Broker unit/H2 430 tests, 0 failures/errors and 1 conditional skip; managed-agent server unit/H2 184 passed; real-process LocalReboot and DurableLocalRuntime fault gates 12/12 passed; Spring/H2/real-worker recovery integration 1/1 passed; Java Checkstyle, repository build, typecheck, bundle and changed Markdown formatting passed. The reviewer physically reboot-tested the earlier W0e-3 head; this new SHA has not had another physical reboot run. There are 0 unresolved inline threads. I am moving the PR to ready for review so exact-head CI and review can run.

[codex] Thanks — agreed on the CI blocker. Fixed in 62584d315.

The targeted hosted IT passed (1/1), the guard's hosted and non-hosted family checks passed in an isolated verification, and the repository build and typecheck passed. The new push should trigger the PR CI that did not run on the draft head. Resolved 0/0 inline threads.

Real-host verification, round 4: head 62584d31

Follows round 3 (head 8c2b626c). Same dedicated Linux host as rounds 1 and 2. All times are UTC, 2026-09-29.

What changed since round 3. The branch now sits on main and has three commits:

• the W0e-3 commit, with the same git patch-id as at 8c2b626c;
• 55ded4c3, the F1 fix with two new tests and the doc updates;
• 62584d31, which renames one test class and removes two pom lines.

I tested 55ded4c3 first, and 62584d31 was pushed while I was working. The server jars of both heads have the same 537 class files, byte for byte, so the results carry over. I also ran a reboot and both Java CI jobs on the new head.

Summary

• F1 is fixed. With the option on, 0 of 1383 healthy turns were refused (round 1: 6 of 871, round 2: 5 of 470). The Hosted chain that ended in a blocked Session in rounds 1 to 3 now completes. Requests that must not be served are still refused.
• Physical reboots pass with this code. A power cut with both controls, four hands-off reboots and a reboot with two crashes inside the cleanup all converge. One of the hands-off reboots used the jar built from 62584d31, which closes the gap the author noted for the new SHA.
• The CI blocker found by the triage bot was real, and 62584d31 fixes it. At 55ded4c3 every Maven step passes and the report guard fails in both Java jobs. At 62584d31 both guards pass, in my replays and in the first real CI run of this PR.
• The merge with today's main (1b696297) is clean. The Hosted job passes on the merge result, and a real reboot with that jar and main's bundle recovers 8 of 8 generations.
• One correction to round 3: M01 and M25 are not killed by the tests in the tree. Details are in section 5.

For the merge decision: I found nothing that blocks the merge. The bot's CHANGES_REQUESTED review is on 55ded4c3, and its one blocker is fixed. Three suggestions remain, none of them blocking: a test for the wrong-host case (attached), log lines for the scan, and the items the author deferred.

1. F1 is fixed

• The fix is the candidate from round 1. recoverBinding completes its reservation with the adopted context when the observation ends healthy. Every other outcome still ends with 503 runtime_reconciliation_required.
• Hosted chain. I used the same delivery as in rounds 1 to 3: the unchanged acquire request reaches the Broker while the scan holds its claim on that binding. All three turns complete and the Session is not recovery-blocked.
• The other side. After its worker was killed, 733 back-to-back warm and acquire requests were all refused, the binding became LOST without stop evidence, and no worker was started. With the access revoked, 977 requests were all refused with 409 workspace_unavailable and the binding was left untouched.
• Tests for the fix. Reverting the fix is caught by the new MaintenanceProbeWaiterTest. Two more mutants of the fix survive, and neither points to a missing test: one is equivalent, the other is refused a second time inside adoptObservation.

2. Physical reboots

3. The CI guard, before and after the rename

• At 55ded4c3 the guard fails in both jobs with the two messages the bot predicted. In the Hosted job the guard step comes before the fault gates step, so CI would not have run the gates either.
• At 62584d31 both guards pass. Before the push I had tested the bot's suggested fix as an A/B arm, and 62584d31 is file for file that tree (10019 files, 0 differences).
• First CI run. SDK Java (8 of 8 jobs) and Qwen Code CI are green. The test counts in the CI logs equal the counts of my replays.
• Why CI was missing before. 55ded4c3 was pushed while the PR was a draft on a stacked base. Changing the base and marking the PR ready do not start these workflows.

4. Merge with today's main

main has moved eight commits since the PR's base (99adce25 → 1b696297). Under packages/sdk-java they change Hosted tests, SessionEventHub, the V15 migration and the OpenAPI file.

The same checks passed an hour earlier on main b1eb94da (R21, 18.2 s). main moved again while I was testing, so I repeated them.

5. Mutation results and a second look at RuntimeRecoveryEvidence.matches

Correction to round 3. Round 3 listed M01 (rebooted() ignores the host id) and M25 (evidence matches any lease) as killed by DurableLocalRuntimeFaultGateTest.absentWorkerAfterRestartLeavesItsDomainPinned. That was wrong:

• The gate builds its Brokers with trusted reboot recovery off, so it never calls rebooted().
• In this round both mutants pass all six stages. The runs were sequential on the quiet host, and a stage counts as a kill only if it fails twice.
• The failures in round 3 came from the load-sensitive gate that round 3 itself listed as a flake.

What is pinned now.

M01 is the one survivor that guards a safety property. On the real host the foreign machine-id control stays blocked (R16), so the code is right and only the test is missing. RebootRecoveryGapTest from round 1 covers it: it passes 2 of 2 at 62584d31, fails under M01, and is Checkstyle clean.

RuntimeRecoveryEvidence.matches. The bot asked for a second look at the relaxed lease check.

• With a lease, nothing is relaxed. matches has one caller, RuntimeBindingRecord.requireEvidence. The constructor rejects a lease that does not match the seed before it reaches that call (RuntimeBindingRecord.java:118). So the new clause can never be false there, which is why M25 survives: it is an equivalent mutant, not a test gap.
• Without a lease, the evidence is still tied to the seed (request id, runtime instance id, incarnation, lease id, epoch) and to the resource handle.
• On the real host this case is the worker that died before READY. In R16, binding 1ce9de00 had no lease and was RECOVERY_BLOCKED before the power cut. After the reboot it was released with both evidence records (table). Before this PR, matches returned false for every binding without a lease.

6. Notes, none blocking

• The scan leaves no trace in the log. In R19 the server reclaimed four generations and was killed twice, and its log has no line from the scan or from the recovery (log). An operator can follow a recovery only in the database. The author deferred logging to a follow-up. This is how that item looks on a real host.
• The scan still observes every healthy binding every 5 s and rewrites its row. After the fix this no longer refuses requests. The author deferred the optimization, and I agree that it can wait.
• SPI. RuntimeBindingRepository gains two methods without a default (finishLostRecovery, findRecoveryCandidates), so an implementation outside this repository would stop compiling. Both implementations in the tree are updated, and RuntimeProvisioner.recoverResources has a fail-closed default. The package is 0.1.0-alpha. Whether that needs a release note is a maintainer call.
• The repository's own script tests do not see a guard mismatch. check-failsafe-reports.test.js and hosted-process-ci.test.js pass 13 of 13 at 55ded4c3, where both guards fail.

What this run does not cover

Bare metal and x86_64, more than one Broker on the host, hostile same-UID tools, remote writers, restored or cloned disks, Kubernetes. On MariaDB only the two integration test classes of the CI job ran. The real-host runs use MySQL 8.4.

[codex] Thanks for round 4 and for correcting the M01/M25 mutation result. I updated the PR description in both languages to cite the current-head physical reboot, power-cut, F1 load and CI evidence, and removed the obsolete retest caveat.

I checked that LocalRuntimeStore.rebooted requires the saved host ID to match before a changed boot ID counts as a reboot. The attached M01 test would pin that safety guard; the R16 foreign-host control confirms the current behavior. Since this is a late review round and the gap is test-only, I will carry that regression test into a follow-up issue or PR, alongside the already deferred scan logging and scale work, without widening this PR. The bot's sole CI blocker was fixed in 62584d31; its re-review is still running.

@qwen-code /triage