feat(managed-agent): Restore safe retired tool output collection - #13225

[codex] Restored O4-2 at bea89fcdefc840314738965634a76f827adbcbe5 in replacement #13225, based on main 9478f28731f45ff6df66062e7e05d29e92906f19. GitHub rejected reopening #13087 and changing its closed base, so #13225 supersedes it. O4-3 #13090 remains stacked on the same repaired branch. No merge or GC enablement was performed.

Final committed-head gate: build, typecheck, bundle, migrations, source-matching Java dependency installation, complete ordinary MySQL profiles and source-derived report checks passed. Broker unit reports: 593 total, 592 passed and one optional real-worker bundle test skipped; Broker MySQL: 7/7 passed. Server: 487/487 unit and 35/35 MySQL integration passed. Checkstyle 0. Two clean independent code-review passes and self-audit completed; fixture, GET and controlled performance execution snapshots are bound to the final Git objects.

These local macOS/MySQL and controlled HTTPS results do not establish Linux identity guards, full Hosted producer/Broker/Runtime acceptance or current-revision credentialed cloud OSS. The historical maintainer OSS 36/37 run still lacks the delete-denied identity. Public Workspace DELETE remains unavailable, and #13192 remains an independent open PR. Those are deployment/readiness boundaries, not green checks inferred from admin-only CI. Current #13225 product CI is separately pending; physical GC is disabled and grace remains 24 hours.

中文：O4-2 已以 bea89fcd 在当前 main 上恢复为替代草稿 #13225，接替 GitHub 拒绝重开的 #13087；O4-3 #13090 继续使用同一已修复分支。已修复迁移撞号、带入候选推进/inline/过期 claim 修复、下载 SQL 放大和 GET 重试 hook 回退，并修正真实领取租约的 Recovery fixture。完整本地最终门禁通过：Broker 592 个单元通过、1 个可选 bundle 用例跳过，7 个 MySQL 集成通过；Server 单元 487/487、MySQL 集成 35/35，Checkstyle 为零。58 个 SDK 故障 fixture、20 个 Recovery 用例及受控 HTTPS 成本/权限/退役/真实租约暂停证据均绑定最终源码。

上表中的容量、诊断和 mutation 建议逐项明确延期到 #12380，部分采纳不算全部修复。永久阻塞轮询饱和、已排期宽限期缩短不会即时生效、C3 和 handoff 剩余 mutation 缺口继续保留。Linux/完整 Hosted、真实 OSS 删除拒绝身份、公开 Workspace DELETE 及独立开放的 #13192 不被本地绿灯替代。没有自动合并；物理 GC 关闭，部署宽限保持 24 小时。

Merged current origin/main (c15dd26) into this branch to clear the conflict, as merge commit 0c64229.

The conflict was limited to close-slice files that main has since reworked: the two workspace-session-reliable-close design docs and WorkspaceRecoveryStoreTest.java (operation fixture moved to the RUNNING/LEASED + lease_until schema). Resolution takes main's versions, so this PR now carries only the collection slice.

Numbering re-verified on the merged tree: V31 recovery, V32 session close (SQL bytes identical to main's), V33 tool output collection; Java migrations remain V15/V29 only — no Flyway version collision.

Verified on the merged tree before pushing: sdk-java managed-agent-server full mvn test — 487 tests, 0 failures, including the Flyway-backed WorkspaceRecoveryStoreTest and the new collection tests (matches your recorded 487/487 gate).

Second merge of origin/main (1abccdb) as 88d4ca9. Main took V33 for managed_agent_definitions (#13142), so the collection migration is renumbered V33 -> V34 with SQL bytes unchanged; the retention docs census now reads V31 recovery / V32 close / V33 definitions / V34 collection, with provisional V28/V33 collection histories requiring explicit reconciliation per the existing policy. Verified on the merged tree with a clean build: managed-agent-server mvn test, 514 tests 0 failures (Flyway-backed suites exercise V31 through V34). The earlier note stands: the Hosted MySQL gate failure signature predates this merge and appears on unrelated branches (e.g. codex/managed-writer-epoch run 37029292498, same test, same signature).

Local real-environment verification of #13225 at 88d4ca9466

Verdict: no blocker for landing with GC disabled (the default). The Flyway V33 collision that existed at 0c64229ae6 is fixed by 88d4ca9466. Its packages/sdk-java tree is byte-identical to the tree I verified. The collector did what the PR describes on the real public close → DELETE path on Linux, against both the OSS double and a real Aliyun OSS bucket. Check three things before merging or enabling GC:

1. Migration order with feat(managed-agent): broker authentication and broker-provisioned writer credentials #13210. The open feat(managed-agent): broker authentication and broker-provisioned writer credentials #13210 also adds V34__managed_session_creator.sql. Whichever PR lands second has to be renumbered.
2. F2 claim. The store-side SQL amplification is fixed (711 → 205 statements per MiB). The body's "100 MiB completes in ~108 s at 5 ms SQL latency" did not reproduce here. 100 MiB was cut at 85.8 MiB when the 120 s read lease ran out. A two-line follow-up (below) brings the cost down to 39 statements per MiB, and 128 MiB then completes in 31 s.
3. Before enabling GC: a single transient PUT failure makes a publication permanently uncollectable. This follows the design doc ("unknown … permanently blocks automatic collection until externally resolved"). No resolution path exists yet, so the publication's storage and tenant quota are never released.

Setup

• Gates and boot matrix: macOS arm64, JDK 21, MySQL 8.4.7.
• Every retirement scenario ran on Linux. On macOS the Workspace close path is unavailable: the durable worker identity reads /etc/machine-id and /proc, so session_close=false. The Linux stack was:
  • a colima VM (aarch64, 4 CPU / 8 GB) running a JDK 21 container and a MySQL 8.4.11 container;
  • the Spring fat jar, the packaged Hosted Harness and a durable local-process Broker worker;
  • a fake model, and an OSS double over TLS driven by the real aliyun-sdk-oss 3.18.4.
• Real Aliyun OSS: a temporary private, unversioned bucket in cn-hangzhou, deleted afterwards.
• Seams, stated plainly:
  • A tap rewrites the Harness tool profile hosted-workspace-files/1 → hosted-workspace-shell/1, because public Shell admission is not on main yet.
  • The legacy-write-evidence, recovery-status and stale-read-lease cases are injected by SQL.
  • Everything else goes through the public API: create, Turn, download, POST /close, DELETE.
• The Harness/worker bundle was built from main 623cfc67. The 1abccdb26a merge only adds a packages/core change (fix(core): stop misdiagnosing malformed tool-call args as max_tokens truncation #12982) that is unrelated to this path.

1. Landing and gates

• At 0c64229ae6, the trial merge into current main fails the repo's own check-flyway-migrations.js: two migrations claim version 33. Spring also refuses to boot on it.
• The renumbered tree boots on an empty database and on a database that main has already migrated (V33 definitions → V34 collection).
• A database migrated by the PR head alone cannot be opened by main (checksum mismatch at V33). That is the provisional-history case the PR body already calls out.
• Unit: 514/514, Checkstyle 0.
• -Pmysql-integration: 51/51, including WorkspaceSessionRetentionMySqlIT 16/16; the failsafe report checker exits 0.

2. Public lifecycle on Linux, and against real OSS

• Public calls: POST /close 202 → completed (554 ms); DELETE 202 → completed (292 ms).
• Collection: the row waited for grace_period until exactly retired_at + grace (10 s). It was claimed at +10.5 s and COLLECTED at +10.9 s.
• OSS deletes: 15 DELETEs, one per catalog key, only the deleted Session's keys; no prefix scans.
• Cleanup and quota: inline payloads cleared; held 13,641,895 → 0 and released_held_bytes = 13,641,895.
• Sibling Session: untouched and still readable with the right sha256.
• On main: the same lifecycle leaves the output RETIRING with all objects and quota held.
• Real OSS: 8/8 objects of the deleted Session were removed (bucket 16 → 8 objects); the sibling's 8 were kept.
• Real bucket with versioning enabled: the collector failed closed for two attempts with Tool publication bucket cannot enforce immutable objects and deleted nothing (4/4 objects kept).

3. Blockers and siblings

Quarantine: I flipped one bit of a stored object. A public download detected it (503 artifact_unavailable) and production code quarantined that publication.

After DELETE:

• The quarantined publication kept 7/7 objects and its quota.
• The other publication in the same Session was collected.

The legacy_write_evidence_missing and recovery_protected rows were kept. A stale read lease held the row as reader_active until the next 60 s poll after it expired (collected at +71.2 s). Tenant held bytes dropped by exactly the two collected publications.

Unknown PUT (finding 3). I injected one PUT 500 on the wire. The next attempt returned, and all 11 objects are VERIFIED. The ledger still has UNKNOWN=1, so after DELETE the publication stays RETIRING / put_unresolved with 6,296,450 bytes held. Nothing in the code moves an attempt out of UNKNOWN.

4. Storage faults

One request per failed DELETE, whole-page retry: a DELETE 500 or connection reset on the 5th key produced exactly one request for that key; the SDK did not replay it. The whole page was deferred by 60 s, and generation 2 re-issued the page; the keys already gone answered 204.

Versioning, crash and multi-page:

• With versioning Suspended, the collector issued 0 DELETEs.
• kill -9 of Spring while a DELETE was held: the restarted process waited out the old claim (59.6 s), then finished as generation 2.
• A 113-key publication was collected in two pages, one tick apart.
• Two live Spring instances with GC on, sharing one database: both collected (two owners seen), but every publication had one owner and generation 1, and each of the 93 keys was deleted once.

In every case the quota was released exactly once.

5. F2: download cost and the 120 s lease

A TCP relay in front of MySQL adds 5 ms to every client→server packet. On main, a 40 MiB download is cut at 24.2 MiB when the lease ends; on the PR it completes in 55.5 s. A 100 MiB download still gets cut at 85.8 MiB on the PR, about 0.73 MiB/s.

What remains is about 11 statements per 64 KiB response chunk:

• ManagedArtifactService's 64 KiB write loop checks the lease and access each time;
• VerifiedStream.read caps every read at 64 KiB and runs the full guard before and after.

The candidate raises those two constants to 1 MiB, the granularity the PR already uses in the store path. Results:

• Cost and speed: 38.8 statements/MiB; 40 MiB in 10.3 s and 128 MiB in 31.3 s at +5 ms.
• Revocation unchanged: revoke or DELETE mid-download still aborts the response, after at most 0.88 MiB.

It touches main's code rather than the PR's diff. It can land here or as a follow-up; otherwise the body's 100 MiB claim should be softened.

6. OSS GET retry and mutation testing

GET retry. I injected faults on the third object of a public download through the real SDK.

• Retried only on the PR: a 500 with an unrecognised error code, and a 502 with a non-XML body. On main these abort the response after one request; the PR retries once and the download completes with the right digest.
• Same on both arms: 500 InternalError, 503, a connection reset, six 500s in a row (4 requests, then abort) and 403 (no retry).
• PUT: a PUT 500 results in two separate attempts, each with one wire request; the SDK replays nothing.

Mutation testing: 28 mutants of the new code, each run against the full unit suite (514 tests). 19 were killed. Of the 9 survivors:

• Not real gaps:
  • M04, M06 and M21 are fences that cannot differ while one process owns a claim (runOnce is synchronized; the owner is per process). The two-collector run above exercised the cross-process case.
  • M07 (the per-page probe) is redundant: the per-key probe makes the same check and is pinned by a test (M08); removing both (M29) is killed.
• Real gaps:
  • M24–M27: the collector tests only exercise the grace_period and legacy blockers. Collecting a quarantined, recovery-protected, put_unresolved or reader_active publication passes the suite; only the real-stack runs in section 3 show those are retained.
  • M17: reverting the guarded wrapper to raw read() also passes. It changes nothing on the public download path: the real stack measured 205.6 statements/MiB with the mutant, the same as the PR. But it guards Session-store and verification reads, which no bounded-query test measures.

Suggestions (none blocking)

1. Coordinate V34 with feat(managed-agent): broker authentication and broker-provisioned writer credentials #13210.
2. Take the 2-line read-granularity patch, or reword the 100 MiB claim.
3. Before turning on gc-enabled, add a way to resolve UNKNOWN PUT attempts. For example, a guarded HEAD of the exact key with a digest comparison, or an operator command. Without it, every transient PUT error permanently leaks that publication's storage and tenant quota. It also adds to the already-deferred 60 s polling of permanent blockers.
4. Duplicate versioning probes: each deleted key costs two OSS requests (115 versioning probes for 113 DELETEs), because deleteIfPresent probes again after the per-page probe. Only the per-key probe is pinned by a test (M08); the page probe is redundant today (M07).
5. Add a parameterised collector test over every blocker kind (quarantined, recovery-protected, reader_active, put_unresolved, …), so that M24–M27 fail.

Evidence (figures, harness, logs, candidate patch): assets-pr13225.

@qwen-code /triage