Skip to content

fix(managed-agent): stop database amplification on session hot paths - #13217

Merged
wenshao merged 22 commits into
mainfrom
fix/13181-managed-agent-query-amplification
Oct 4, 2026
Merged

wenshao merged 22 commits into
mainfrom
fix/13181-managed-agent-query-amplification

Conversation

@wenshao

@wenshao wenshao commented Oct 2, 2026 •

Copy link
Copy Markdown
Collaborator

What this PR does

Fixes the four database-amplification hot paths reported in the issue, plus the two related ones. Snapshot materialization no longer rewrites the whole items_json document under the session row lock after every ≤200-event batch — it rewrites at creation, on a terminal event, every 1000 covered events, or on catch-up once the previous snapshot is 5s old, and a deferred rewrite is recorded as a snapshot_stale_since marker on the consumer-progress row (migration V38) so the session is re-selected until the snapshot converges. SSE streams no longer run a read-grant SQL check before nearly every delivered event per subscriber — each stream re-verifies at most once per configurable recheck window (default 5s). The session list endpoints no longer run 2–4 extra queries per row — a page is assembled from a constant number of grouped batch queries that project only the summary columns, and the single-session views share that assembly. Tool-publication authorization no longer rescans the session journal backwards (one locked read per revision) on every publish/seal/prefix/finish — a new migration carries the current activation state on the journal head, maintained transactionally by commit inside the parse pass every commit already runs, with a scan-and-backfill fallback for pre-migration journals; reserve/renew read the tool.intent at its own revision instead of walking the journal, and a fenced activation is answered from the locked head before any journal read. Related: artifact downloads now re-validate content access at most once per revalidation window instead of per 64 KiB chunk; and two further migrations add an event-type index for latest-turn lookups and a (tenant_id, session_id, last_sequence) index for the intent range read.

Because a rolling fleet can briefly run a binary that commits without maintaining the new head columns, trusting them is gated behind qwen.managed-agent.tool-publication.journal-head-authorization (default false = the pre-existing journal scan); an operator flips it once every writer runs the V36 schema's code. The head columns carry an activation_head_revision stamp naming the journal revision they reflect, so a premature flip is still safe: a head last written by an old binary fails the stamp check, gets rescanned and re-stamped — the rolling window self-heals per session instead of certifying stale state. Activation-expiry parsing pre-checks the digit width before materializing any big integer, so a hostile exponent-form expiresAt costs nothing on any of the three read paths.

Why it's needed

A code audit showed the broker's database work scaling with accumulated history rather than with change: snapshot writes grew quadratically with session length while blocking event ingestion on the same row lock, N subscribers × M events produced N×M permission queries on an unbounded executor, a default 20-row session page cost 41–81 queries, and a long turn delayed each 16 MB publication segment by hundreds to thousands of locked journal reads. The issue asks for heat to scale with change; this PR does that with pinned per-endpoint query budgets so none of it can silently regress.

Reviewer Test Plan

How to verify

Run mvn -o test in packages/sdk-java/managed-agent-server (install the sibling modules first: mvn -o install -DskipTests -Dgpg.skip=true in packages/sdk-java/qwencode and packages/sdk-java/runtime-broker). The new Issue13181QueryBudgetTest drives the production stores/services over a statement-recording H2 (MySQL mode) DataSource and pins the measured before→after counts: snapshot rewrites per 9-batch burst materialization go from [1,1,1,1,1,1,1,1] to [1,0,0,0,0,0,0,0,1], and a 12-tick trickle records [1,0,…,0]; a 21-event workspace stream's managed_workspace_access queries go from 43 to 2; a 20-row page costs 3/4/3/3 queries where it cost 41/61/61/81; verifyDispatch with the head 30 revisions past the last activation goes from 31 locked journal reads to 0; renew reads a constant 4 journal statements regardless of filler depth (the revision range read, the chain-contiguity count, and the verified page), and a fenced renew pays zero journal statements; a pre-migration head gets exactly one rescan that backfills the head. The bounded-staleness semantics are pinned both ways: zero-window restores per-event/per-chunk checks (existing revocation tests), and new tests prove revocation lands at the window's end, that a successful recheck re-anchors the window, and that the shipped 5s default is what the suite exercises. Note two pre-existing timing-sensitive ToolPublicationStoreTest cases (renewsTheOriginalClaimWhileScanningSlowObjectBytes, streamsLargeOutputAndReadsItsTailAfterStoreReplacement) fail on loaded machines on the base too — verified identical on an unmodified worktree.

Evidence (Before & After)

N/A (non-UI; measured query counts are in the issue comment I'll post and in Issue13181QueryBudgetTest's recorded output)

Tested on

OS Status
🍏 macOS ✅
🪟 Windows ⚠️
🐧 Linux ⚠️

Environment (optional)

OpenJDK 21 + Maven (offline), H2 in MySQL mode running the real Flyway migrations (V34–V39 apply cleanly; RuntimeBrokerFlywaySchemaTest passes). MySQL *IT integration suites need Docker and were not run locally — left to CI.

Risk & Scope

  • Main risk or tradeoff: two deliberate, issue-sanctioned bounded-staleness relaxations — read-grant revocation on a live stream and artifact-download re-validation take effect within a 5s-default window (both operator-configurable durations with no enforced upper bound; PT0S restores the old per-event/per-chunk behavior); the snapshot may lag the projection by up to 1000 covered events during bursts, rewrites unconditionally on a terminal event, and a deferred trickle snapshot converges within 5s via aged-out reselection. The artifact window also defers the stage-1 DELETING lifecycle observation to the window's end (stage-2 retirement still aborts per chunk through the read lease).
  • Not validated / out of scope: MySQL integration tests locally (no Docker); moving the seal/finish whole-stream byte rehash off the request thread (needs an asynchronous seal contract — follow-up Move the managed-agent seal/finish stream rehash off the request thread #13242); the journal-head flag flip is follow-up Enable journal-head authorization after the fleet runs the V35 schema #13295.
  • Breaking changes / migration notes: four Flyway migrations (V36 five nullable head columns including the revision stamp, V37 event-type index, V38 snapshot deferral marker column, V39 journal sequence index); pre-migration journals are served by a scan-and-backfill fallback; the head columns are only trusted once journal-head-authorization is enabled, which an operator should do after the fleet fully runs the V36 code — and the stamp makes even a premature flip self-healing (rolling-deploy rationale in the design doc).
  • Design docs: English · 简体中文 — complete and synchronized.

Linked Issues

Fixes #13181

中文说明

这个 PR 做了什么

修复 issue 报告的四条数据库放大热路径及两个相关项。快照物化不再在每批 ≤200 事件后、于会话行锁内全量重写 items_json —— 改为首次创建时、terminal 事件批次、每覆盖 1000 事件时,或追平且距上次快照写入满 5 秒时重写;被推迟的重写以 snapshot_stale_since 标记记录在 consumer-progress 行上(迁移 V38),会话因此被重新选中直至快照收敛。SSE 流不再于几乎每个投递事件前、按订阅者执行读授权 SQL —— 每条流在可配置的复检窗口(默认 5 秒)内最多复检一次。会话列表接口不再每行多跑 2–4 条查询 —— 页面由固定数量的分组批量查询装配且只投影摘要列,单会话视图复用同一装配。工具发布授权不再于每次 publish/seal/prefix/finish 倒扫会话 journal(每个 revision 一次持锁读)—— 新迁移把当前 activation 状态承载到 journal head,由 commit 在每次提交本就要做的解析遍历中顺带维护,迁移前的 journal 由"扫描一次 + 回填"回退覆盖;reserve/renew 按 intent 所在 revision 直接读取而非逐 revision 回扫,被围栏的 activation 在任何 journal 读取之前就由已加锁的 head 作答。相关项:artifact 下载的内容访问复检从每 64 KiB 分片一次改为每个复检窗口最多一次;另两个迁移分别新增事件类型索引(最新 turn 查询读索引范围)与 (tenant_id, session_id, last_sequence) 索引(intent 范围读)。

由于滚动部署期间集群可能短暂运行不维护新 head 列的旧二进制,信任这些列由 qwen.managed-agent.tool-publication.journal-head-authorization(默认 false,即沿用旧的 journal 扫描)门控;运维在全部写入方运行 V36 代码后打开。head 列带有 activation_head_revision 时间戳,标明它们反映到哪个 journal revision,因此提前打开开关同样安全:旧二进制最后写入的 head 无法通过时间戳检查,会被重扫并重新打戳 —— 滚动窗口按会话自愈,而不是把陈旧状态认证为有效。activation 过期时间的解析在任何大整数物化之前先做宽度预检,恶意的指数形态 expiresAt 在三条读取路径上都零开销。

为什么需要

代码审计显示 broker 的数据库工作量随累积历史而非变化量增长:快照写入随会话长度平方增长并在同一行锁上阻塞事件摄入;N 订阅者 × M 事件在无界执行器上产生 N×M 条权限查询;默认 20 行会话页成本 41–81 条查询;长 turn 让每 16MB 发布分片拖后数百到数千次持锁 journal 读。issue 要求热度随变化量伸缩;本 PR 做到了,并为每个端点钉住查询预算,防止静默回归。

评审者测试计划

如何验证

在 packages/sdk-java/managed-agent-server 运行 mvn -o test(先安装兄弟模块:在 packages/sdk-java/qwencode 与 packages/sdk-java/runtime-broker 执行 mvn -o install -DskipTests -Dgpg.skip=true)。新增的 Issue13181QueryBudgetTest 在记录语句的 H2(MySQL 模式)DataSource 上驱动生产 store/service,钉住实测的前后对比:9 批突发物化的快照重写从 [1,1,1,1,1,1,1,1] 变为 [1,0,0,0,0,0,0,0,1],12 周期涓流记录为 [1,0,…,0];21 事件工作区流的 managed_workspace_access 查询从 43 降为 2;20 行页的查询数为 3/4/3/3(原 41/61/61/81);head 领先 activation 30 个 revision 时 verifyDispatch 的持锁 journal 读从 31 降为 0;renew 无论堆积多少 revision 恒定 4 条 journal 语句(revision 范围读 + 链式计数 + 校验页),被围栏的 renew 零 journal 语句;迁移前的 head 首次回扫一次并回填。有界陈旧语义双向钉住:零窗口恢复逐事件/逐分片检查(既有撤销测试),新测试证明撤销在窗口末端生效、成功复检会重新装配窗口、以及发布的 5 秒默认值正是套件所覆盖的。注意 ToolPublicationStoreTest 有两个既有的时序敏感用例(renewsTheOriginalClaimWhileScanningSlowObjectBytes、streamsLargeOutputAndReadsItsTailAfterStoreReplacement)在负载高的机器上于基线同样失败 —— 已在未修改的工作树上验证一致。

证据(前后对比)

N/A(非 UI;实测查询计数见我将发布到 issue 的评论与 Issue13181QueryBudgetTest 的记录输出)

已验证平台

OS 状态
🍏 macOS ✅
🪟 Windows ⚠️
🐧 Linux ⚠️

环境(可选)

OpenJDK 21 + Maven(离线),H2 MySQL 模式跑真实 Flyway 迁移(V34–V39 干净应用;RuntimeBrokerFlywaySchemaTest 通过)。MySQL *IT 集成套件需要 Docker,本地未跑 —— 留给 CI。

风险与范围

  • 主要风险或取舍:两处有意的、issue 认可的有界陈旧放宽 —— 存活流的读授权撤销与 artifact 下载复检在默认 5 秒窗口内生效(两者均为运维可配置的 Duration,不设强制上限;PT0S 恢复旧的逐事件/逐分片行为);突发期间快照最多落后投影 1000 条已覆盖事件,terminal 事件批次必重写,被推迟的涓流快照经老化重选在 5 秒内收敛。artifact 窗口同样把 stage-1 DELETING 生命周期观察推迟到窗口末端(stage-2 retirement 仍经读取租约逐分片中止)。
  • 未验证 / 范围外:本地未跑 MySQL 集成测试(无 Docker);seal/finish 全流字节重哈希移出请求线程(需要异步 seal 契约 —— 后续 Move the managed-agent seal/finish stream rehash off the request thread #13242);journal-head 开关切换为后续 Enable journal-head authorization after the fleet runs the V35 schema #13295。
  • 破坏性变更 / 迁移说明:四个 Flyway 迁移(V36 五个可空 head 列含 revision 时间戳、V37 事件类型索引、V38 快照推迟标记列、V39 journal sequence 索引);迁移前的 journal 由"扫描 + 回填"回退服务;head 列仅在 journal-head-authorization 打开后才被信任,运维应在集群全部运行 V36 代码后打开 —— 且时间戳使提前打开也能自愈(滚动部署依据见设计文档)。
  • 设计文档:English · 简体中文 —— 完整且同步。

关联 Issue

Fixes #13181

@github-actions github-actions Bot added the review/self-reported The linked issue was opened by the PR author (self-reported) label Oct 2, 2026
@wenshao

wenshao commented Oct 2, 2026

Copy link
Copy Markdown
Collaborator Author

E2E / verification report

Method: the change lives in the standalone Spring control plane packages/sdk-java/managed-agent-server, which the qwen CLI never exercises, so the verification vehicle is the pinned query-count suite the issue itself recommends. The new Issue13181QueryBudgetTest drives the production stores/services over a statement-recording H2 (MySQL mode) DataSource migrated by the real Flyway scripts.

Measured before → after (same instrumented harness on both runs):

Path Before After
Snapshot rewrite (80 events / 9 batches) rewrite every batch, 352 item rows re-read, 145,280 cumulative bytes insert + catch-up only ([1,0,…,0,1]); threshold path separately pinned ([0,1,1] over 1500 events)
SSE read grants (21-event workspace stream) 43 managed_workspace_access queries 2 (admission + first in-loop check; the 5s window covers the rest)
Session list pages (20 rows) 41 / 61 / 61 / 81 queries 3 / 4 / 3 / 3 (single-session detail: 3)
Publication authorization (activation 30 revisions back) 31 locked journal reads per verifyDispatch, 62 per publishSegment 0; a pre-migration head gets exactly one rescan that backfills the head, then O(1)

Regression: full mvn -o test suite (426 tests after the rebase onto current main) — 0 failures; the only error is the known timing flake streamsLargeOutputAndReadsItsTailAfterStoreReplacement (and its sibling), both reproduced on the unmodified baseline. All 32 Flyway migrations apply cleanly; RuntimeBrokerFlywaySchemaTest passes; the new check-flyway-migrations CI gate passes locally (it caught a real V28/V29 collision with upstream during review — renumbered to V31/V32).

Revocation semantics pinned both ways: a zero window restores per-event/per-chunk checks (existing tests), and new tests prove a revocation within the window takes effect at the window's end.

MySQL *IT integration suites need Docker; not run locally, left to CI.

中文

方法:本变更位于独立的 Spring 控制面 packages/sdk-java/managed-agent-server,qwen CLI 无法触达,因此验证载体是钉住查询数的测试套件(issue 本身也建议钉预算测试)。新增 Issue13181QueryBudgetTest 在记录每条 SQL 的代理 DataSource(H2 MySQL 模式 + 真实 Flyway 迁移)上驱动生产 store/service。

实测前后对比(同一插桩框架两次运行):

路径 修复前 修复后
快照重写(80 事件 / 9 批物化) 每批重写 + 重读 352 行,累计写 145,280 字节 仅首批插入 + 追平批重写([1,0,…,0,1]);阈值路径另测(1500 事件)[0,1,1]
SSE 读授权(21 事件工作区流) 43 次 managed_workspace_access 查询 2 次(准入 + 首次循环检查,5 秒窗口覆盖其余)
列表页(20 行) 41 / 61 / 61 / 81 条查询 3 / 4 / 3 / 3 条(单会话详情 3 条)
发布授权(activation 落后 30 revision) verifyDispatch 31 次持锁读、publishSegment 62 次 均 0 次;迁移前头部回退扫描一次并回填,此后 O(1)

回归:mvn -o test 全套件(rebase 至当前 main 后共 426 个测试)0 失败,唯一的 error 是既有时序 flake(streamsLargeOutputAndReadsItsTailAfterStoreReplacement 及其同类,在未修改基线上同样失败)。Flyway 全部 32 个迁移干净应用;RuntimeBrokerFlywaySchemaTest 通过;新增 CI 门禁 check-flyway-migrations 本地通过(评审中它抓住本分支 V28/V29 与上游同号的真实碰撞,已重编号为 V31/V32 修复)。

撤销语义双向钉住:零窗口恢复逐事件/逐分片复检(既有测试),窗口内撤销至多延迟一个窗口并在窗口末端生效(新增测试)。

MySQL *IT 集成套件需要 Docker,本地未跑,留给 CI。

@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Please do not rebase or force-push to an active PR as it invalidates existing review comments. Note for future reference, the bots always squash all changes into a single commit automatically as part of the integration.

中文

请勿对活跃的 PR 执行 rebase 或 force-push,因为这会使已有的评审评论失效。另外,供日后参考:作为集成流程的一部分,机器人始终会自动将所有改动压缩(squash)为单个提交。

@wenshao

wenshao commented Oct 2, 2026

Copy link
Copy Markdown
Collaborator Author

Review round R1 addressed (2 Criticals + 23 Suggestions, all accepted)

All 25 findings are fixed in the follow-up commit. Summary by area:

Criticals

  • R1-1 (snapshot gate, trickle case): the gate now skips the rewrite only when a snapshot exists, lag < 1000, no terminal event in the batch, and NOT (caught-up AND ≥5s since the last write). Convergence is preserved: findMaterializationTargets re-selects a session whose snapshot lags its drained progress once the snapshot ages past 5s, and the re-selected empty tick rewrites it (pinned by deferredSnapshotConvergesOnTheAgedOutReselection + the existing integration test, which caught the stale-forever shape before this fix).
  • R1-2 (rolling-deploy fail-open): head columns are trusted only under qwen.managed-agent.tool-publication.journal-head-authorization (default false, wired through properties → configuration → store, mirrored in application.yml). With the gate off, authorization always scans the journal; preV33CommitSkewStillFencesWhileTheHeadGateIsOff reproduces the exact skew (journal row + revision bump, columns untouched) and asserts fencing, and disabledJournalHeadAuthorizationKeepsScanningTheJournal proves the scan still runs and repairs the head.

Store/service changes

  • R1-9: the activation extraction now rides the extension-record parse pass (ApplyResult); the third parse is gone. The four acceptance tests stay green.
  • R1-6: all three expiresAt readers share ManagedExtensionRecords.millisLenient (integral number or integral numeric string, else absent); stringExpiresAtReadsConsistentlyAcrossTheScanAndTheHead pins commit-write, scan, backfill, and head-read agreement.
  • R1-7: with the head authoritative, requireEvidence resolves the intent's revision via one first_sequence/last_sequence range read plus one verified page (chain check preserved); renewReadsTheIntentAtItsOwnRevision pins a constant 3 journal statements at filler depth 30 (was 5/revision).
  • R1-16: approval_mode rides the page's SELECT * onto SessionRecord; ManagedActionStore.approvalModes is deleted; page budgets tightened to 3/3/3/2 with a zero-count pin on the removed query. The value still comes from the column (lowercase-normalized at insert), never from the property.
  • R1-17: both batch turn reads project TURN_SUMMARY_COLUMNS (qualified variant for the join) into TurnSummary; projection pinned by SQL-text assertions.
  • R1-5: findActiveTurn/findLatestTurn/findLatestEnvironmentEvent delegate to the batch twins (the session_not_found guard kept); the dead findSnapshotCoveredSequence is gone; LATEST_TURN_ID removed.
  • R1-8: column widths live once on ManagedSessionStoreModels.MAX_ACTIVATION_ID_CHARS/MAX_ACTIVATION_PHASE_CHARS, read by both the commit fits-guard and the backfill.

Test pins (mutant-verified locally)

  • R1-4: batchTurnReadsFollowAdmissionOrderNotCreatedAt drives two out-of-order Turns + inverted environment sequences through the page; both measured mutants (MAX→MIN, pair-filter narrowing) go red.
  • R1-10/R1-11: revocation window widened to 500ms/1000ms keeping grantChecks == 3; both window tests now assert failed == false.
  • R1-20: a default-window case with its own workspace-bound session asserts the window covers the second event (grantChecks == 2); ManagedAgentPropertiesTest pins the 5s defaults and the false flag.
  • R1-23: readGrantRecheckReanchorsAfterEachSuccess (500ms/1000ms, sent == 4, grantChecks == 3); the anchor-once mutant goes red.
  • R1-21: retimed to 400ms reads / 1000ms window asserting exactly "ab"; the per-chunk re-verify mutant goes red. (Constants widened beyond the proposed 200/500 after observing >100ms scheduler overshoot on a loaded machine; margins are now 200ms each way.)
  • R1-22: oneCommitKeepsTheLastActivationChange now blanks the four columns and re-asserts the fence through both scans (renew → "Activation is not active", verifyDispatch → "Original activation is fenced"); both forward-walk mutants go red.
  • R1-3/R1-19: already covered in the reworked gate tests (replay floor clamped to a lagging snapshot; exact per-batch statement/row counts).

Docs

  • R1-12: follow-up issue Move the managed-agent seal/finish stream rehash off the request thread #13242 filed and cited in §2/§9 and this PR body.
  • R1-13/R1-14: supersede notes with cross-links added to the 2026-09-26 and 2026-09-29 contracts (both languages; the per-chunk lease checks noted as unchanged).
  • R1-15: README gained the read-revalidation-interval row, qualified the two immediacy sentences, and links the new design doc pair.
  • R1-18: §2/§9 reworded — the relaxations default to 5s, are operator-configurable, and have no enforced upper bound.
  • R1-24: §7 now documents that the stage-1 DELETING observation is also window-deferred mid-download (stage-2 retirement still aborts per chunk via the read lease), pinned by midstreamDeletionIsDeferredToTheRevalidationWindow.

Full module suite: 471 tests, 0 failures, 2 errors — the two pre-existing ToolPublicationStoreTest timing flakes that fail identically on unmodified origin/main under load (A/B verified on a clean worktree). Flyway uniqueness gate: 34 migrations, all unique.

Post-commit audit rounds (undirected + adversarial pairs over the committed diff, with mutation probes):

  • Round 1: no Criticals; findings fixed in the third commit — renew-path gate pin in the skew test, inline catch-up + debounce pins, missing/overflowing expiresAt clean-refusal pin, artifact test margins widened (500ms/1500ms after a 4/9 load-flake reproduction), dead import and stale comments cleaned.
  • Round 2: no Criticals; fixed in the fourth commit — expired-activation fence pinned on both grant paths (both freshness-removal mutants go red), the two wall-clock-sensitive budget tests moved to a frozen clock, and the commit-side text extraction aligned with the scans' leniency. Three surviving conjunct mutants were declined as non-exploitable (a non-conforming writer's self-harm; the legacy scan keeps identical checks).
  • Round 3 (confirmation, both audit types on the final state): CLEAN — the undirected pass found nothing actionable, and the adversarial pass closed all 16 attacks with the round-2 pin claims independently reproduced (mutants red).

Four hot paths in the Managed Agent Runtime Broker amplified database
work far beyond request volume, two of them while holding row locks:

- materializeNextBatch rewrote the whole items_json snapshot after every
  <=200-event batch under the session row lock. The snapshot is now
  rewritten at creation, on a terminal event, every
  SNAPSHOT_REFRESH_EVENTS (1000) covered events, or on catch-up once the
  previous snapshot is SNAPSHOT_REFRESH_MILLIS (5s) old; a session whose
  snapshot lags its drained progress is re-selected by
  findMaterializationTargets once the snapshot ages out, so an idle
  session still converges instead of staying stale forever.
- SSE fan-out ran a read-grant SQL check before nearly every event per
  subscriber. Streams now recheck the grant at most once per
  events.read-grant-recheck-interval (default 5s; PT0S restores
  per-event checks); session.deleted still terminates immediately.
- listPublicSessions/listWebShellSessions ran 2-4 extra queries per row.
  Pages are now assembled from a constant number of grouped batch
  queries that project only the turn summary columns, with the approval
  mode riding the page's own SELECT *, plus one workspace-close batch
  read when the page holds bound sessions; single-session views share
  the same assembly, and the singular store reads delegate to the batch
  twins so each selection rule has one spelling.
- Publication authorization rescanned the session journal back to the
  last activation.changed per revision under the publication lock on
  every publish/seal/prefix/finish/verifyDispatch. Migration V34 carries
  the current activation on the journal head, maintained by commit()
  inside the parse pass every commit already runs; authorization reads
  the head row it already locks, reserve/renew read the tool.intent at
  its own revision, and pre-migration heads get one legacy scan plus a
  backfill. Because a rolling fleet can briefly run a binary that does
  not maintain the columns, trusting them is gated behind
  tool-publication.journal-head-authorization (default off) until the
  fleet drains.

Related: artifact downloads re-verified content access per 64 KiB chunk;
the re-verification is now throttled to
artifacts.read-revalidation-interval (default 5s), which also defers the
stage-1 DELETING observation to the window's end (stage-2 retirement
still aborts per chunk through the read lease). Migration V35 adds an
event-type index so latest-turn lookups read index ranges instead of a
session's event history.

Query budgets per endpoint are pinned by Issue13181QueryBudgetTest (with
the artifact revalidation window pinned in
ManagedArtifactReadIntegrationTest), the windowed/frozen-clock semantics
are pinned by ManagedEventStreamServiceTest, and the fencing behavior
across the head columns, the legacy scan, and the rolling-deploy gate is
pinned by ToolPublicationStoreTest.

Fixes #13181
@wenshao
wenshao force-pushed the fix/13181-managed-agent-query-amplification branch from 0ff2939 to 1b2496d Compare October 2, 2026 19:59
@wenshao

wenshao commented Oct 2, 2026

Copy link
Copy Markdown
Collaborator Author

Rebase note: the branch was rebased onto current main (which landed #13194's workspace-session archive/delete) and squashed to a single commit 1b2496d10a; the earlier commits fd18586686/0676c68fdb/fdf2f3be07/0ff29396ac are its pre-squash history.

Two integration points came out of the rebase:

  • Retention capability flags are now batched too. feat(managed-agent): archive and delete closed workspace sessions #13194 added a per-row hasCompletedWorkspaceClose read feeding the new session_archive/session_unarchive/session_delete capability flags; on this PR's batched page assembly that would have been a per-row query again. The page assemblers now fetch the close state in one IN query (completedWorkspaceCloses) only when the page holds workspace-bound sessions, so the pins moved from 3/2 to 4/3 for bound pages (unbound pages stay 3/3), with the close-state query count pinned at 1.
  • Migrations renumbered again: upstream added V33__managed_agent_definitions.sql, so this PR's two migrations are now V34 (journal-head activation columns) and V35 (event-type index). The Flyway-uniqueness gate passes (35 migrations, all unique).

Suite on the squashed tree: 498 tests (including upstream's new WorkspaceSessionRetentionTest), 0 failures, 2 pre-existing timing flakes (renewsTheOriginalClaimWhileScanningSlowObjectBytes, streamsLargeOutputAndReadsItsTailAfterStoreReplacement, both reproduced on unmodified origin/main under load).

wenshao added 3 commits October 3, 2026 04:03
Round-3 adversarial follow-ups (no Criticals):

- expiredActivationFencesThroughTheHead now proves the fences came from
  the head columns: renew reads exactly the intent's revision (3
  statements), and verifyDispatch performs zero locked journal reads.
- renewReadsTheIntentAtItsOwnRevision additionally pins zero locked
  journal reads, so a lock-mode upgrade on the intent read cannot slip
  past the statement-count pin.
- Drop the unreachable null-check after queryForObject in
  requireLegacyActivation (the query throws on a missing row, and
  record_bytes is NOT NULL).
Round-4 adversarial follow-up (no Criticals): the 20-row bound-page loops
asserted titles and actions but never the sessionArchive/Unarchive/Delete
flags, so a regression computing retention once per page would have passed.
One bound session now carries a completed CLOSE row and both surfaces
assert sessionDelete per row.
CI (Runtime Broker and Managed Agent MariaDB) caught
preservesTextOrderAcrossToolsAndReasoningInSnapshots racing the 10ms
materializer: a scheduler tick can create the snapshot mid-sequence, and a
non-terminal catch-up drain inside the 5s floor then legally leaves it
behind. The trailing event is now terminal so the explicit drain always
rewrites; the content-ordering assertion is unchanged.
@wenshao

wenshao commented Oct 2, 2026

Copy link
Copy Markdown
Collaborator Author

Final audit trail and CI note:

  • Rounds 3–5 of the pre-push audits (undirected + adversarial, mutation-probed): round 3 found two test-rigor gaps and one dead branch, fixed in f75955cb55 (the expired-activation test now proves the fence came from the head columns — renew reads exactly the intent's revision, verifyDispatch performs zero locked journal reads — and the renew budget additionally pins zero locked reads so a lock-mode upgrade cannot slip past); round 4 covered the rebase surface (retention batching, squash fidelity, renumber) and found one pin gap, fixed in 7de58603bc (per-row sessionDelete assertions on both bound pages with one completed CLOSE row among 20); round 5 re-audited both follow-up commits: CLEAN.
  • CI: the MariaDB job caught one real integration between the snapshot gate and the 10ms test materializer — a scheduler tick could create the snapshot mid-sequence so a non-terminal explicit drain legally left it lagging. Fixed in b0d357ae9c by making the text-order test's trailing event terminal (the drain then always rewrites; the content assertion is unchanged). No production code involved.

…eterministic

Same scheduler race as the text-order test fixed in b0d357a, caught by
the MariaDB CI job on the following run: a 10ms-tick snapshot created
mid-sequence plus a non-terminal catch-up inside the 5s floor legally lags.
The trailing reasoning delta is now terminal so the explicit drain always
rewrites. The contentPartId assertions are unchanged.
wenshao added 4 commits October 3, 2026 11:42
Two correctness findings and the accepted suggestions from the second
review round of the query-amplification fix:

- ManagedExtensionRecords.millisLenient rejects a JSON number or
  numeric string wider than a long before materializing the BigInteger,
  so an exponent-form expiresAt no longer costs a giant allocation.
- The journal head's activation columns carry activation_head_revision,
  the journal revision they reflect; every commit re-stamps it, both
  authorization gates require stamp equality, and the backfill
  re-stamps, so a rolling fleet still running pre-V34 writers detects
  the skew and rescans instead of certifying stale columns.
- requireEvidence answers a fenced activation from the locked head
  before touching the journal: a fenced renew now pays zero journal
  statements, pinned in the budget test.
- Deferred snapshot rewrites are recorded with a snapshot_stale_since
  marker (V36) that findMaterializationTargets re-selects once aged, so
  a drained trickle converges instead of idling stale forever.
- transcript tails past the snapshot are bounded by the caller's limit,
  keeping the newest events and reporting the truncation for paging.
- The latest-environment-event batch reads one row per session via a
  MAX(sequence_id) derived table over the (session, turn) pairs.
- Workspace close state has a single spelling (the singular predicate
  delegates to the new batch query), and the approval mode rides the
  page's SELECT * onto SessionRecord instead of a per-row probe.
- The publication journal fixture is now shared between
  ToolPublicationStoreTest and Issue13181QueryBudgetTest as
  PublicationJournalFixture, ending the drift between the two copies.
Undirected plus adversarial audits of the R2 commit found one Critical
and three Minor issues:

- A commit carrying no activation change re-stamped the carried-forward
  head columns with the new journal revision, laundering a skew left by
  a pre-V34 writer into a fresh-looking head. The stamp now advances
  only when the preserved columns were current at the previous
  revision, keeping the skew detectable until the rescan heals it;
  pinned by noChangeCommitDoesNotRestampASkewedHead, and the
  always-advance mutant goes red.
- The millisLenient width pre-check overflowed int on extreme
  exponent-form strings; the comparison now widens to long, pinned by
  the 1e2147483647 arm.
- sameEpochReleasePreventsReserveAndRenew is parameterized over the
  journal-head-authorization flag, pinning the release fence on the
  shipped legacy scan and the head fast path alike; the suite default
  stays false, matching production.
- The application.yml mirror test also asserts the flattened keys
  exist, so a renamed or dropped key cannot false-pass on the Java
  defaults.
- The design doc (both languages) records that a transcript backward
  walk can repeat the first page's inlined control events at the page
  seam and that clients deduplicate on the event sequence.
The round-2 undirected and adversarial audits both found the same Major:
the millisLenient pre-check guarded only the negative-scale direction,
so a 12-byte "1e-100000000" expiresAt still forced toBigIntegerExact to
expand 10^100000000 before dividing (measured ~56s of CPU). The scale
is now bounded too (at most 19 decimal places, which a representable
long never needs), and a test arm pins the input class.

Two test-coverage Minors from the same round:

- transcriptTailIsBoundedByTheCallerLimit now pins the load-bearing
  "newest kept" property with exact sequences (the tail ends at the
  session's last event; olderCursor names the oldest returned one) —
  the oldest-kept mutant goes red.
- deferredSnapshotConvergesOnTheAgedOutReselection now ages only the
  deferral marker for the selection arm and pins that the tick's scan
  references snapshot_stale_since without touching
  managed_agent_snapshot, so a regression to per-row snapshot probing
  fails twice over.
@wenshao

wenshao commented Oct 3, 2026

Copy link
Copy Markdown
Collaborator Author

Review round R2 addressed (2 Criticals + 38 Suggestions — 39 fixed, 1 declined)

All findings are in the follow-up commit. Summary by area:

Criticals

  • R2-1 (exponent-form expiresAt DoS): millisLenient now pre-checks precision() - scale() > 19 before any BigInteger materialization, so an exponent-form string is rejected from its 11 bytes (the reviewer's exact shape), on the commit path and both scan paths alike. The 1e100000000 arm in unrepresentableExpiresAtFencesTheHeadCleanly pins the clean refusal with a zero-journal-statement budget.
  • R2-2 (stale head certified as live): the head carries a fifth column, activation_head_revision, written on every commit with the journal revision the columns reflect. Both gates require stamp equality with the head's journal_revision; a pre-V34 binary's commit bumps journal_revision without the columns, so the head fails the stamp check, authorization rescans, and the backfill re-stamps — the rolling window self-heals per session. staleHeadStampRescansInsteadOfTrustingTheColumns (gate ON) and preV34CommitSkewStillFencesWhileTheHeadGateIsOff (gate OFF) reproduce the skew by restoring the pre-commit columns and stamp after a real release commit; staleHeadStampRebackfillsAndRejoinsTheHeadPath pins the heal.

Publication authorization

  • R2-40/R2-39: requireEvidence answers the activation term from the locked head before reading the intent — a fenced renew pays zero journal statements (pinned in expiredActivationFencesThroughTheHead and the unrepresentable-expiry test, which now also carries ledger assertions).
  • R2-23: migration V37 adds qwen_managed_session_journal_tx (tenant_id, session_id, last_sequence), so the intent revision lookup is an index range read.
  • R2-24: the fast path restores the chain-contiguity proof with one indexed COUNT over the revision range from the intent up to the locked head.
  • R2-25: the verified page is parsed by byte slice (ToolPublicationContract.readJson(bytes, offset, length)) — no record-wide String decode/split/re-encode.
  • R2-20: the commit-side activation capture now requires the record's index to sit inside the declared event range ("Activation change has an invalid journal position").
  • R2-21: the activation columns fold into the single head UPDATE every commit already runs — one write to the locked row per commit.
  • R2-22: backfillActivation skips its write when the columns already hold exactly those values (all four columns + stamp compared, keeping disabledJournalHeadAuthorizationKeepsScanningTheJournal's restore assertion green).
  • R2-32: the two-arg newStore test helper now defaults journalHeadAuthorization to false, the shipped default; the head-path tests opt in explicitly, and sameEpochReleasePreventsReserveAndRenew is parameterized over the flag so the release fence is pinned on both the legacy scan and the head fast path.
  • R2-34: the budget suite now pins the flag-off costs too — the skew test asserts the legacy branch's locked-read count and a zero head-write count, so a regression on the shipped-default path goes red.
  • R2-38: the publish-side zero pin gained its positive control: the two authorizations' four head FOR UPDATE reads (liveness probe + locked read each) are counted, so the zero cannot pass vacuously.
  • R2-29/R2-31: oneCommitKeepsTheLastActivationChange builds the two activations with distinct fields on every column and asserts all four columns plus the stamp on blank/restore.
  • R2-30 (declined): the twin width guard in backfillActivation is unreachable — the publication contract rejects an id wider than 512 bytes at reserve time (measured: the reserve fails with "Invalid ID encoding" before any journal row exists), so no binding can reference an oversized activation id for the backfill to re-check. Both guards share the MAX_ACTIVATION_* constants from R1-8.
  • R2-37: the publication journal fixture is extracted into PublicationJournalFixture (create(DataSource, journalHeadAuthorization) + the event/append/request/reserve helpers); ToolPublicationStoreTest and Issue13181QueryBudgetTest delegate, ending the drift between the two copies.

Snapshot gating

  • R2-18: the correlated per-tick snapshot lookup is gone — a deferred rewrite instead stamps snapshot_stale_since on the consumer-progress row (migration V36), and findMaterializationTargets re-selects on the marker alone; rewriteStaleSnapshot converges and clears it.
  • R2-19: the design doc now states the invariant precisely — readers key off the snapshot's own covered_sequence, so projection progress may lead the snapshot between rewrites — and §3 documents the marker.
  • R2-36: deferredSnapshotConvergesOnTheAgedOutReselection runs on the frozen-clock Fixture(Clock) like its siblings.

List pages

  • R2-16/R2-17: findLatestEnvironmentEvents selects each session's latest environment event on its latest turn in SQL (a MAX(sequence_id) derived table filtered by the (session, turn) pairs) — one row per session, no OR chain, no dropped limit.
  • R2-13: the close-state predicate has one spelling — hasCompletedWorkspaceClose delegates to completedWorkspaceCloses.
  • R2-14: the test-only singulars (findActiveTurn/findLatestTurn/findLatestEnvironmentEvent/findSnapshotCoveredSequence) are deleted from the store SPI.
  • R2-10: hasActions reads approval_mode off the page row; the actions != null clause and the ManagedActionStore collaborator are gone.
  • R2-11: the retention rule has one spelling again — a static retention(session, closed) helper feeding both capability assemblers.
  • R2-15: the batch reads pre-size their maps and bind lists from the known element count.
  • R2-35: the item-part read is pinned alongside the item read (partReads [1,0,…,2] on the burst).

Transcript

  • R2-7 (doc) + tail bounding: the transcript's event tail past the snapshot is bounded by the caller's limit (findNewestTailEvents, newest kept, olderCursor/hasMore reported), and §3 says precisely which view lags (the items array) and which does not (the event tail).

Docs and wording

  • R2-3: the supersession note no longer leaves the zero-event revocation guarantee standing — it states closure within one window plus one poll interval.
  • R2-4/R2-5/R2-6: the three zh-CN notes link the zh-CN design doc.
  • R2-8: §8 discloses the two re-fixtured ManagedAgentServerIntegrationTest snapshot tests instead of claiming the suite is unchanged.
  • R2-9: the README row says enforcement lands at the next chunk boundary after the window's end.
  • R2-12: the ReadGrant javadoc states the idle-stream closure bound (recheck window + poll interval).

Test internals

  • R2-26: the artifact window tests share readingPolicy/oneBytePerReadReader/flippingResponse helpers; the dead permission flag is gone (the flip lives in flippingResponse, still firing on the first write; the per-read sleep is untouched).
  • R2-27: the deletion test now observes the stage-1 transition — it asserts the session reached DELETING.
  • R2-28: the revocation window test retimed to 1000ms/3000ms margins.
  • R2-33: relaxationDefaultsMatchTheShippedConfiguration binds application.yml through YamlPropertySourceLoader + ApplicationContextRunner, so both halves of the invariant are guarded.

Full module suite: 503 tests, 0 failures, 2 errors — the two pre-existing ToolPublicationStoreTest timing flakes that fail identically on the pre-change commit under load (A/B-verified on a clean worktree this round). Flyway uniqueness gate: clean.

Post-commit audit rounds (undirected + adversarial over the committed diff):

  • Round 1: one Critical — a no-activation-change commit re-stamped carried-forward head columns to the new journal revision, laundering a pre-V34 writer's skew into a fresh-looking head (both auditors found it independently); fixed by advancing the stamp only when the preserved columns were current at the previous revision, pinned by noChangeCommitDoesNotRestampASkewedHead with the always-advance mutant going red. Three Minors fixed alongside: the width pre-check widened to long arithmetic (extreme exponents overflowed the int subtraction), the yml-mirror test now asserts the flattened keys exist (key drift cannot false-pass on Java defaults), and the transcript page-seam overlap (page 1's inlined control events reappear on the cursor path) is documented in §3 of the design doc — the WebShell client deduplicates on the event sequence, so the seam repeat is not user-visible. The head-path coverage split behind R2-32 was hardened by parameterizing the release-fence test over the flag.
  • Round 2: one Major found independently by both auditors — the millisLenient pre-check guarded only the giant-integer direction (1e+N), while 1e-N still forced toBigIntegerExact to expand 10^N before dividing (measured ~56s for a 12-byte input); the scale is now bounded too (at most 19 decimal places, which a representable long never needs), with a 1e-100000000 test arm. Two test-coverage Minors fixed alongside: transcriptTailIsBoundedByTheCallerLimit now pins the newest-kept property with exact sequences (the oldest-kept mutant goes red), and deferredSnapshotConvergesOnTheAgedOutReselection ages only the marker for the selection arm and pins that the tick's scan never touches managed_agent_snapshot. Everything else the adversarial pass attacked held (stamp conditional, readIntent, V36 marker lifecycle, env derived table, fixture extraction, Flyway).
  • Round 3: adversarial CLEAN (every fix re-attacked with fresh angles — backfill/commit interleavings, parser-exotic expiresAt forms probed empirically, the pins' discriminating power — all held; readIntent, the V36 marker lifecycle, the env derived table, and the fixture extraction all held). The undirected pass verified all six round-1/2 fixes as correct, complete, and non-vacuously pinned; its single Minor was a wrong lock attribution in the publish positive-control comment, corrected (claim authorize + retention put + verify-open lease + install authorize = the 4 head reads).
  • Round 4 (confirmation, both audit types over the final state): CLEAN — the undirected pass re-derived every fixed spot and found nothing reportable; the adversarial pass enumerated all stamp/column writers and readers (two writers, two gated readers, both under the head row lock), re-probed the millisLenient bounds empirically, and closed fresh attacks on readIntent's strict decode, the env derived table, and the marker lifecycle. Two residual notes are recorded as non-defects: readIntent's range read visits O(revisions-after-intent) rows (pre-existing; the pinned budget is the statement count, which holds), and the zh-CN doc renders "stamp" as 时间戳.

wenshao added 6 commits October 3, 2026 13:58
…5-V38

Main landed V34 (tool output collection) plus five managed-agent merges
since the review head. Resolution:

- Migrations renumbered: activation head columns V35, event-type index
  V36, snapshot deferral marker V37, journal sequence index V38; every
  code comment, test name, and both design-doc languages updated.
- ToolPublicationStore: the head read keeps #13192's SQL-side
  writer_live CASE while also selecting the activation columns and
  stamp; the head gate and the legacy fallback now run on the DB-clock
  epoch (#13192's nowEpoch) like the rest of the method.
- ManagedAgentService: #13112's per-row creator-submit capability is
  folded into the batch page assembly instead of paid per row — the
  registry gains createdSessions and a findReadable batch twin (both
  preserving the binary-safe CAST comparisons, the sargable plain IN
  kept alongside), and maySubmitWorkspaceTurn splits the row-shape
  checks from the registry reads so the singular mutation path is
  unchanged. Unbound pages still cost 3 queries; bound pages 4, with
  the creator batch only when submit-shaped sessions exist and the
  grant batch only for creator-owned ones.
- ToolPublicationStoreTest keeps both sides' new authorization tests;
  HarnessCoordinatorTest takes main's boundCancellingStore helper with
  the SessionRecord approval-mode argument.
Four Criticals and the accepted Suggestions from the third review:

- Restore the uncapped cursor-less transcript tail (merge-base
  behaviour): capping it at the caller's limit falsified the published
  webShellTranscript contract and broke the prefix invariant the stream
  resume relies on (a lagging snapshot's middle band was served by
  neither page nor stream), and the new backward walk re-served
  already-projected delta rows past the seam. The contract sentence is
  now pinned in ManagedAgentApiContractTest, and the budget suite pins
  the full tail (every event past the snapshot, hasMore=false, no
  cursor) instead of the cap.
- The commit-side activation capture applies the same sessionKey/v
  scope check both read paths enforce, so a foreign-scoped
  activation.changed is rejected at commit instead of being promoted
  into the durable head columns a later flag flip would trust.
- The head path's contiguity count carries the legacy walk's
  byte-length tripwire, and the single-revision locate is pinned
  against holes and overlapping ranges; both intent reads reject
  duplicate lines at one sequence.
- Pins added: the phase arm of the commit width guard, the deferral
  marker surviving a fresh empty tick, the per-event attributes read in
  the materialization budget (plus a whole-batch statement total), the
  idle-stream revocation closing at the next wake for both stream
  loops, the Range path's in-window exposure at the shipped 5s default,
  the configuration-to-store flag wiring, and the four-column
  last-activation-wins rotation.
- Housekeeping: the dead args field dropped from the extracted
  fixture's consumer, the provably dead width guard dropped from
  backfillActivation (its callers always pass contract-validated
  values), the yml-mirror test now neutralizes the ambient
  QWEN_MANAGED_AGENT_* environment, the properties test keeps the
  flattened-key assertions, and the event fixture gained a scope/version
  overload.
- Docs: the admission design's recheck sentence now states the windowed
  behaviour and the real closure bound (window plus one poll interval)
  in both languages; §2 qualifies path 4's relief as opt-in until the
  fleet runs the V35 code, with the operator flip tracked as #13295; §6
  states the locate's row cost honestly (constant statement count,
  distance-proportional rows, strictly lighter than the legacy walk);
  §8 composes the bound-page budgets; §9 records the chain proof's
  narrowed parse scope; the README's revalidation row states the real
  in-window exposure and its env-var table names all three new knobs.
…aths

Audit of the R3 wave found one Minor: the new commit-side check compared
the sessionKey field by field, so a key carrying extra fields passed the
commit but reads as foreign to both read paths' closed-key comparison.
The check now builds the closed three-field key and deep-equals it,
exactly the read-side semantics; the commit-rejection test gains the
extra-fielded arm. Also narrows the duplicate-intent test's comment to
the revisions a path actually reads (a stray out-of-range claim is
invisible to the head path's locate and unreachable through the commit
validation).
@wenshao

wenshao commented Oct 3, 2026

Copy link
Copy Markdown
Collaborator Author

Review round R3 addressed (4 Criticals + 20 Suggestions — all accepted)

Criticals

  • R3-1/R3-2 (transcript tail cap): reverted to the merge-base behaviour — the cursor-less page again serves every event after the snapshot (the published contract's exact words), so the prefix invariant the stream resume relies on holds and the backward-walk seam disappears with the cursor it came from. transcriptServesEveryEventPastTheSnapshot pins the full tail (every event past the snapshot, hasMore=false, no cursor, the reported watermark equals the session's last sequence); the capped variant would fail it. ManagedAgentApiContractTest now pins the webShellTranscript description's load-bearing sentences, so a future paging change that leaves the published text stale goes red (the R3-1 acceptance).
  • R3-3 (chain proof dropped the walk's tripwires): the contiguity COUNT now carries the same byte_length >= 1 AND byte_length <= MAX_TRANSACTION_BYTES bounds the legacy walk's transactions() applies per revision, and §6/§9 (both languages) state honestly what the count proves (gap-freeness with the tripwire; the per-line parse is kept for the located revision) — including the one residual: a pre-existing journal carrying a foreign-scoped line above the intent is caught by the legacy scan but not re-read after backfill, which is why the commit side now refuses such lines. aCorruptedJournalRowAboveTheIntentFencesBothPaths pins both arms (head path: "journal evidence is missing"; legacy: "failed verification"), going red without the bounds.
  • R3-4 (write-side scope check): ManagedExtensionRecordStore.apply applies the same sessionKey/v equality to the activation.changed capture that both read paths enforce — a foreign-scoped line is rejected at commit ("Journal event scope conflicts", same message) instead of being promoted into durable trusted columns. foreignScopedActivationChangeIsRejectedAtCommit (foreign key + unknown version arms) goes red with the check removed.
  • R3-6 (duplicate intents): both readIntent and the legacy walk now require(intent == null, "Intent sequence conflicts") before assigning, so the two paths fence ambiguous evidence identically; duplicateIntentLinesAtOneSequenceAreFenced (both flags) goes red with either guard removed.

Suggestions — test pins (each verified against its mutant)

  • R3-9: aJournalHoleAboveTheIntentFencesTheHeadPath (deleted revision above the intent) goes red without the contiguity require; anOverlappingJournalRangeFencesTheHeadPath (widened range) goes red with size() == 1 weakened.
  • R3-13: idleStreamClosesAfterRevocationAtTheNextWake (both loops, nothing published after revocation) completes within window + poll interval; deleting the loop-head recheck from streamWebShell turns its arm red while the public arm stays green.
  • R3-14: the item-read filter now also counts the per-event existingAttributes read (trailing where keeps the parts table out), and a whole-batch ledger.total() vector is pinned (measured [35×8, 17]).
  • R3-17: oversizedActivationPhaseBlanksTheHeadColumnsCleanly — a 46-char phase blanks the columns, advances the stamp, and the scan fences; without the phase arm the commit dies with DataIntegrityViolation (verified).
  • R2-20 residual: activationChangeBeyondTheDeclaredEventsIsRejectedAtCommit pins the position guard.
  • R2-29 residual: oneCommitKeepsTheLastActivationChange now varies activationId and epoch between the two payloads and asserts all four columns record the last.
  • R3-18: the fixture's event() gained a scope/version overload (the three-arg form is byte-identical); foreignScopedIntentLinesAreFenced and unknownVersionIntentLinesAreFenced pin the read-side scope guard under both flag settings.
  • R3-19: ToolPublicationConfigurationTest.theJournalHeadAuthorizationFlagReachesTheStore pins the flag's path from properties into the store via a public journalHeadAuthorization() accessor; negating the wiring goes red.
  • R3-10: revocationInsideTheShippedWindowStillShipsARange pins the shipped 5s default's actual Range-path guarantee (admission + first guard = 2 policy calls, the window covers the single write) next to the existing PT0S case.
  • R3-11: emptyBatchDoesNotRewriteAFreshLaggingSnapshot now also pins the deferral marker surviving the not-aged exit (value unchanged, and the session stays unselected).

Suggestions — housekeeping and docs

  • R3-16: the dead args field and its assignment are gone from ToolPublicationStoreTest.
  • R2-30 residual: the backfillActivation width guard is dropped as provably unreachable — both callers require an active phase and the binding's contract-capped id immediately before the call — and the javadoc now says so instead of describing a live path.
  • R2-33 residual: relaxationDefaultsMatchTheShippedConfiguration now replaces systemEnvironment with a scrubbed map (all QWEN_MANAGED_AGENT_* removed) before adding the yaml sources; the hostile-env rerun (QWEN_MANAGED_AGENT_TOOL_PUBLICATION_JOURNAL_HEAD_AUTHORIZATION=true plus the harness pair) stays green.
  • R3-15: the class javadoc restates all four snapshot-gate arms; the test is renamed materializationRewritesTheSnapshotAtCreationAndTerminalEvents.
  • R3-7: the admission design's recheck sentence now states the windowed behaviour and the real closure bound (next recheck, plus at most one poll interval for an idle stream), and the superseded note is scoped to the original per-event decision — both languages.
  • R3-8: §8 composes the bound-page budgets (6 with Turns: page, latest turns, environment events, close-state, creator-marker, grant; 5 when the turn-less page skips the environment batch) in both languages.
  • R2-9 residual: the README row now states the real in-window exposure (up to 1 MiB for a Range request, up to a full read-timeout of a streaming download; nothing written after a denial).
  • R2-23 residual: §6 and V38's comment no longer claim a constant-cost seek — the statement count is constant while the locate's row cost grows with the intent's distance behind the head (still strictly lighter than the legacy walk's locked row reads). On the suggested rows() pin: the ledger counts rows a statement RETURNS, so a COUNT(*) yields one row regardless of scan depth — a row-examined pin needs engine-level instrumentation H2 can't provide through the proxy, so the honesty fix is in the claims instead.
  • R2-38: the README names all three knobs (read-grant-recheck-interval, read-revalidation-interval, journal-head-authorization) with defaults and restore/rollout semantics.
  • R3-5: §2 qualifies path 4's relief as opt-in until the fleet runs the V35 code; the operator flip is tracked as follow-up issue Enable journal-head authorization after the fleet runs the V35 schema #13295 (filed, bilingual, with the acceptance checks).

The round's fixed-ruling confirmations (R1-1, R1-2, R2-1, R2-2, R2-3, R2-8, R2-20, R2-35, R2-36, R2-37) are acknowledged with thanks — no action needed.

Full module suite on this commit: 611 tests, 0 failures — the only errors are the three pre-existing, environment-bound failures reproduced on unmodified origin/main under the same load (the two ToolPublicationStoreTest timing flakes plus HarnessCoordinatorTest.runningOwnerObservesCancellationAfterStreamingStarts[accepted], all A/B-verified). Flyway uniqueness gate: 38 migrations, all unique. SpotBugs high-confidence gate: clean.

wenshao added 5 commits October 4, 2026 13:50
The Critical plus the accepted Suggestions from the fourth review:

- readIntent now answers a damaged journal with the session store's
  corruption fault (500 managed_session_journal_corrupt) instead of a
  client request fault (400): the locate, the contiguity count, and the
  verified-page checks all throw journalCorrupt, widened to
  package-private for the shared spelling. The fence tests assert the
  code and the 5xx status; the legacy path answered 500 already.
- requireLegacyActivation applies the closed-key scope check before
  promoting a scanned activation into the head columns, so a
  pre-existing journal carrying a foreign-scoped activation.changed is
  refused instead of laundered into trusted state.
- The commit-side scope check now covers every enveloped
  managed_session_event_v1 line except domain.committed ones (whose
  requireEnvelope keeps its pinned messages) — a misscoped or
  unknown-version event line is refused at commit instead of reaching
  the journal; bare event-subtype lines with no envelope stay inert and
  tolerated as before.
- tailEvents pages at SNAPSHOT_REFRESH_EVENTS (the gate's lag bound),
  so a maximally lagging snapshot's transcript tail is one event read
  instead of ten; the full-tail contract pins hold with the wider page.
- The empty materialization tick is pinned whole (4 statements), and
  the Range revocation test uses the shared readingPolicy helper.

The nine deferred observations from the review are recorded in the PR
thread.
The adversarial audit of the R4 wave found two Suggestions:

- readIntent answered a binding naming a never-committed intent
  sequence with the 500 corruption fault under the head path, where the
  legacy walk answers 400. The locate now splits the cases: an
  ambiguous range or a hole inside the committed span is corruption
  (500), while a sequence beyond the committed span is the requester's
  fault (400 "Committed publication evidence is missing"), matching the
  legacy walk. Pinned by aNeverCommittedIntentSequenceIsAClientFault
  under both flag settings (red without the split, verified).
- The commit-side scope check's domain.committed carve-out also exempt
  unknown-domain lines that requireEnvelope never sees; those now get
  the closed-key check too (pinned by unknownDomainLinesAreRejectedAtCommit,
  red under the kind-based carve-out, verified).

Plus the tail-read test now exercises the multi-page arm: a tail longer
than one 1000-event page is served completely across two reads.
…plification' into fix/13181-managed-agent-query-amplification
Main's #13301 persists Workspace session tool profiles and claimed
migration V35; this branch's four migrations move to V36-V39
(activation columns, event-type index, snapshot deferral marker,
journal sequence index) and SessionRecord carries both new fields
(approvalMode, toolProfile).
@wenshao

wenshao commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator Author

Review round R4 addressed (1 Critical + 5 Suggestions, all accepted; 9 deferred observations recorded)

Critical

  • R4-1 (corruption misclassified as a client fault): readIntent's three journal-damage checks (the locate, the contiguity count, the verified page) now throw the session store's corruption fault — 500 managed_session_journal_corrupt, shared via ManagedSessionStore.journalCorrupt() widened to package-private — instead of an IllegalArgumentException that the handler maps to 400 invalid_request. Both authorization paths now answer a damaged journal identically, and the reserve/renew caller's retry/alert semantics are right. aJournalHoleAboveTheIntentFencesTheHeadPath, anOverlappingJournalRangeFencesTheHeadPath, and aCorruptedJournalRowAboveTheIntentFencesBothPaths all assert the code and the 5xx status under both flag settings (the head arm goes red with the old require — verified).

Suggestions

  • R4-2 (scope-less legacy scan could launder a foreign line into the head): requireLegacyActivation now applies the closed-key scope check (sessionKey deep-equality + v == 1, the exact spelling the sibling readers use) before backfillActivation may promote the line. aForeignScopedActivationInAPreExistingJournalIsNotPromoted (SQL-planted poison, cold head) refuses under both flag settings and the head stays NULL; it goes red without the check (verified).
  • R4-3 (the fence tests planted the foreign line at the intent, the one position both paths parse): the commit-side scope check in ManagedExtensionRecordStore.apply now covers every managed_session_event_v1 line, so a misscoped line can never enter the journal — the head path's narrower parse scope is safe by construction. The two fence tests are retargeted to assert the commit rejection (foreignScopedIntentLinesAreRejectedAtCommit, unknownVersionIntentLinesAreRejectedAtCommit; both go red with the check narrowed back to activation lines — verified), and aForeignScopedLineAboveTheIntentIsRefusedByTheLegacyScan plants a misscoped line above the intent via SQL: the legacy scan refuses it (both gate settings, via a cold head), while the warm head path authorizes from the head columns without re-parsing the intermediate revision — the §9 residual, recorded.
  • R4-4 (the restored uncapped tail pays ten round trips at the lag ceiling): tailEvents now pages at SNAPSHOT_REFRESH_EVENTS (the deferral gate's own lag bound), so a transcript open on a maximally lagging session costs one event read instead of ten, and a transient longer burst still pages further. aMaximallyLaggingSnapshotTailIsOneRead pins the 999-event tail at exactly one managed_agent_event read (red at page size 100 — verified); the full-tail contract pins hold with the wider page.
  • R4-5 (the empty tick had no statement pin): emptyBatchDoesNotRewriteAFreshLaggingSnapshot now pins the whole empty batch at ledger.total() == 4 (the event read, the session lock, the progress lock, the snapshot read).
  • R3-10 follow-up: the Range-window test now uses the shared readingPolicy(...) helper like its siblings (the calls == 2 pin is unchanged and green).

Deferred (recorded in the review body, not requested this round): the nine round-4 deferred observations (wall-clock sleeps in the revocation test, the QueryLedger.matches negation grammar, the stream tests' repeated fixture, §8's acceptance list, the empty-batch rewriteStaleSnapshot call shape, fractional expiresAt shapes, the width-constant two-writer invariant, a lapping-window artifact case, the stream helper's third knob) stay deferred per the convergence posture — happy to take any of them in a follow-up if you'd like.

Full module suite on the final head: 633 tests, 0 real failures — the only failures are the pre-existing, environment-bound set that reproduces on unmodified origin/main under load (the two ToolPublicationStoreTest timing flakes, A/B-verified). Flyway uniqueness gate: clean (39 migrations). Checkstyle and SpotBugs high-confidence gates: clean. Post-commit audits: an undirected pass found only wording-level observations (folded into the commit message and §9); an adversarial pass found two Suggestions (the fault-class split for a never-committed intentSequence, and the unknown-domain carve-out gap) — both fixed with red-verified pins in the audit-response commit; the confirmation pass over the final state came back CLEAN.

Heads-up on the latest push: main's #13301 (persisted Workspace session tool profiles) landed its own V35 while this round was in flight, so this branch's four migrations are renumbered to V36–V39 and SessionRecord now carries both new fields (approvalMode, toolProfile); every textual V35–V38 reference in code, tests, README, and the design docs moved with them.

@wenshao

wenshao commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator Author

@qwen-code /triage

@wenshao

wenshao commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator Author

Real-stack verification — #13217 at 4d3b1351c7 (trial-merged onto main 6136786c0c)

Verdict for merge: supports merging. On a real MySQL 8.4.7 + Druid deployment driven through the public and WebShell APIs, the three hot paths the API can reach shrink as claimed:

  • List pages: 41–141 statements down to 3–6.
  • Snapshot rewrites: 18–22× fewer rewrites and ~18.5× fewer binlog bytes.
  • SSE read-grant queries: 139× fewer.

Both bounded-staleness relaxations stay inside their documented bounds, and read-grant-recheck-interval=PT0S restores main's revocation behaviour and query volume. On the upgrade path:

  • V36–V39 apply cleanly to a populated database.
  • The six list/detail responses are byte-identical before and after the upgrade.
  • The old binary keeps serving the migrated schema.
  • The activation_head_revision stamp falls behind after an old-binary commit, which is the rolling-window premise the design depends on.

Nothing blocking. The R5-3 redundancy is confirmed but small. Publication authorization (path 4) and the artifact-download throttle can't be reached through the public API with shipped defaults. I ran the PR's own budget test for them on real MySQL instead (§4).

How it was run

Arms main = 6136786c0c (includes the Druid pool swap #13363 and #13365) · PR = trial merge 8136a6be2c = PR head 4d3b1351c7 + main; its diff against main is exactly the PR's 41 files.
Stack per arm Spring Managed Agent Server fat jar (JDK 21.0.12, isolated Maven repo, embedded Runtime Broker) · native MySQL 8.4.7 (performance_schema on, ROW binlog, time_zone=+00:00, JVM TZ=UTC) · packaged Hosted Harness dist/cli.js serve --profile hosted-harness built from main (the PR changes no TypeScript) · deterministic fake OpenAI model
Traffic Only POST/GET /v1/agents/sessions… and /api/agent/web-shell/v1/…. Workspace-bound Sessions run through the real Broker worker and mount.
Measurement Statement counts and rows examined come from performance_schema.events_statements_summary_by_digest, filtered to the arm's schema. List counts subtract a 2 s idle window. Snapshot bytes are managed_agent_snapshot row-event bytes parsed from the binary log. Snapshot lag is sampled straight from MySQL. Nothing is instrumented inside the app.
Host 10-core macOS; load average 30–40 from other sessions throughout, so wall times are noisy and I make no latency claims.

1. The three API-reachable hot paths (figure 1)

Operation main PR
GET /v1/agents/sessions — 20 unbound 41 stmts · 54 rows 3 · 40
GET /v1/agents/sessions — 20 Workspace-bound 81 · 94 4 · 60
WebShell sessions/query — 20 unbound 61 · 254 3 · 90
WebShell sessions/query — 20 Workspace-bound 141 · 1146 6 · 131
GET /v1/agents/sessions/{id} / WebShell sessions/get 3 / 4 3 / 3
Burst: 5 Turns × 3000 deltas (15,025 events) 565 rewrites · 84.7 MiB binlog · 1817 ms SQL 26 · 4.6 MiB · 121 ms
60 s stream: 1500 deltas @ 40 ms (1,505 events, items_json ≈ 178 KB) 258 rewrites · 86.0 MiB · 862 ms 14 · 4.7 MiB · 58 ms
SSE: 4 subscribers on that stream (2 public + 2 WebShell) 6,943 managed_workspace_access reads (1.15 per subscriber-event) 50 (0.008)

Notes on these numbers:

  • On the real stack a Workspace-bound WebShell page costs 7 statements per row on main, more than the 4 per row behind the 41–81 range in the PR description: per-row approval_mode, close-state, create-command and registry reads, plus the turn and environment-event lookups. PR's 6 equals the bound in design §8 for a bound page with creator-owned sessions.
  • All four SSE subscribers on PR received 1505/1505 events in order.
  • History-Turn wall time was 23–33 s on main and 19–32 s on PR. Both arms spend that time mostly in the per-delta Session Store commit path (the 1,505-event stream made 1,608 vs 1,614 journal commits), so I see no end-to-end latency change on this host.

Figure 1 — database work per operation

2. Bounded staleness, measured (figure 2)

  • Snapshot / GET …/items. During the 60 s stream the snapshot was refreshed every 268 ms (median) on main and every 5058 ms on PR (max 5168 ms). On PR the oldest event missing from the snapshot was at most 4.9 s old, and the lag peaked at 120 events (main: 11). Both arms converge on the terminal batch. In the trickle-then-pause case PR lagged by at most 22 events and converged about 5 s after ingestion paused, through the snapshot_stale_since reselection; the marker was observed set and then cleared. The WebShell transcript showed every token throughout, because it tails the events past the snapshot.

  • Read-grant revocation (can_read flipped 4 s into a 15 s stream):

    Arm Stream closed after Events delivered after revoke
    main 123 ms 0
    PR (default 5 s) 4666 ms 45
    PR + read-grant-recheck-interval=PT0S 144 ms 0

    The public and WebShell streams behave identically. A new subscription after revocation returns 404 in every arm. PT0S also restores the per-event query volume (6,905 reads, against 6,943 on main).

Figure 2 — bounded staleness measured live

3. Upgrade, rollback and the rolling-window stamp (figure 3)

One MySQL database through four phases. Each phase is a fresh Spring + Harness process pair on the same deployment directories, as on a real host.

  1. main (V35) creates 41 Sessions, 29 of them with Turns.
  2. PR boots on that database. Flyway applies V36–V39 in 0.049 s. On the same rows, all six list and detail responses are byte-identical to main's (15–18 KB bodies), while statements per call fall from 41 / 81 / 61 / 141 / 3 / 4 to 3 / 4 / 3 / 6 / 3 / 3. New Turns completed on 6/6 pre-existing Sessions and 1/1 new Session.
  3. main again on the V39 schema (rollback, or the old half of a rolling fleet). Flyway logs Successfully validated 39 migrations and the server runs normally. 6/6 old Sessions plus 1 new completed. A 205-event stream reached both SSE subscribers in full. No Spring ERROR lines in any phase.
  4. PR again. 6/6 plus 1/1 completed.

Journal-head activation stamp on the 28 list Sessions that have journals:

  • Before the upgrade: 28 null.
  • After PR commits: 6 current.
  • After old-binary commits: those 6 are lagging. The old binary advances journal_revision without touching the columns, which is exactly the rolling-window skew the stamp is designed to expose, here produced by real traffic.
  • After the next PR commit carrying an activation change: 6 current again.
  • In the ab-pr run, real Hosted Turns filled the V36 columns on all 31 heads, each with a current stamp.

The reader side (refusing a lagging or null head, then rescanning and backfilling) needs tool publication, so it is covered in §4 instead.

Scaled database. I copied the main-arm data ×100 into a V35 schema: 4,300 Sessions, 1,736,300 events, 2.1 GiB.

  • V36–V39 applied in 4.87 s, of which the V37 event index took 4.74 s.
  • I then rebuilt V37 on 2.14M rows while a writer inserted an event every ~5 ms. The 474 inserts during the 3.3 s build had a max latency of 6.4 ms and a mean of 0.54 ms, so InnoDB's online DDL did not block ingestion.

After each restart, the first Turn on a pre-existing Session waited 32–35 s for the previous Harness's writer lease. This happened in every phase with both binaries, so it is unrelated to the PR.

Figure 3 — upgrade, rollback, rolling-window stamp

4. Paths the public API can't reach

Publication authorization (path 4) and the artifact re-validation window both need qwen.managed-agent.tool-publication.enabled (default false) plus an object store. They also need hosted-workspace-shell/1 Sessions, and the Managed Agent API only creates hosted-workspace-files/1, so neither path can be reached through the public API with shipped defaults. What I ran instead:

  • The PR's own Issue13181QueryBudgetTest on real MySQL. I patched only its fixture so that each test gets a fresh MySQL 8.4.7 schema migrated V1–V39 by Flyway (36-line patch, on the assets branch). Result: 30/30 pass on InnoDB, the same as on H2. That includes the publication budgets (verifyDispatch reads the locked journal 0 times, renew costs 4 journal statements, a fenced renew costs 0) and the stamp cases staleHeadStampRescansInsteadOfTrustingTheColumns, staleHeadStampRebackfillsAndRejoinsTheHeadPath, noChangeCommitDoesNotRestampASkewedHead and publicationAuthorizationRescansAndBackfillsPreMigrationHeads.
  • Full managed-agent-server unit suite on the trial merge (main with Druid, plus the PR): 633 run, 0 failures, 0 errors, 1 skipped. The two timing-sensitive ToolPublicationStoreTest cases named in the PR description passed here.
  • Plans on the scaled database (figure 4): a 20-row page drops from 41 / 81 / 61 / 121 statements to 3 / 4 / 3 / 5. Both new batch reads use the V37 index: findLatestTurns reads 39 index entries and findLatestEnvironmentEvents reads 78 rows. When Sessions have 500 Turns each, the optimizer switches the environment read to managed_agent_event_turn_idx by itself (40 rows).

Figure 4 — scaled database, online DDL, budget test on MySQL

Observations (non-blocking)

  1. findLatestTurns cost grows with Turns per listed Session (figure 4). It reads every turn.accepted index entry of each listed Session and then takes MAX; MySQL uses no loose index scan here. Server time for a 20-row page: 0.61 ms at 1 Turn per Session, 1.16 ms at 50, 4.49 ms at 500, 35.95 ms at 5,000. main's per-row read costs 0.10–0.30 ms × 20 plus 20 round trips, so the crossover lies somewhere between 500 and 5,000 Turns per Session; at 500 PR is still ahead. I don't think this blocks the PR, but it is worth remembering for very long-lived Sessions. I tried a per-Session ORDER BY … DESC LIMIT 1 rewrite and it was slower (117 ms, or 158 ms with FORCE INDEX), so I'm not proposing a fix.
  2. R5-3 is real but small. On PR, QwenHostedHarnessConnector still issues about 1.7 single-row SELECT approval_mode per Workspace Turn (41 across 24 Turns). main's 904 such reads came mostly from per-row list lookups, which this PR removes. Next to roughly 1,600 Session Store commits per 1,500-event Turn, the remainder is noise, so a follow-up is fine.
  3. API clients can see both documented trade-offs. In the trickle run, 9.8 s in, the WebShell transcript had 38 tokens while GET …/items showed 19. A revoked reader received 45 more events over 4.7 s. Both match the design. PT0S restores the strict behaviour at main's query cost. Operators who rely on prompt revocation may want a line about this in the release notes.

Not covered

Evidence: figures, raw results.json and samples for every run, the rig (rig.mjs, scale.mjs, configs) and the MySQL fixture patch are on assets-pr13217.

中文版

真实栈验证 — #13217 @ 4d3b1351c7(试合并到 main 6136786c0c)

合并参考结论:支持合并。 在真实 MySQL 8.4.7 + Druid 部署上,只通过公开 API 与 WebShell API 驱动,公开 API 能触达的三条热路径都按 PR 所述大幅下降:

  • 列表页:41–141 条语句 → 3–6 条。
  • 快照:重写次数降为 1/18–1/22,binlog 字节约降为 1/18.5。
  • SSE 授权查询:降为 1/139。

两项有界陈旧放宽都落在文档承诺的范围内;read-grant-recheck-interval=PT0S 能恢复 main 的撤销行为和查询量。升级路径方面:

  • V36–V39 在有数据的库上顺利应用。
  • 升级前后 6 个列表/详情接口的响应逐字节一致。
  • 旧二进制能继续在迁移后的 schema 上服务。
  • 旧二进制提交后 activation_head_revision stamp 会落后,这正是设计所依赖的滚动窗口前提,并在真实流量下得到确认。

无阻断项。R5-3 的冗余查询确认存在,但影响很小。发布授权(第 4 条路径)和 artifact 下载节流在默认配置下公开 API 无法触达,改为在真实 MySQL 上运行 PR 自带的预算测试来覆盖(第 4 节)。

运行方式

两臂 main = 6136786c0c(含 Druid 连接池替换 #13363、#13365)· PR = 试合并 8136a6be2c = PR head 4d3b1351c7 + main,相对 main 的 diff 恰为 PR 的 41 个文件
每臂的栈 Spring Managed Agent Server fat jar(JDK 21.0.12,隔离 Maven 仓库,内嵌 Runtime Broker)· 本机 MySQL 8.4.7(开启 performance_schema,ROW binlog,time_zone=+00:00,JVM TZ=UTC)· 从 main 构建的打包 Hosted Harness dist/cli.js serve --profile hosted-harness(PR 不改 TypeScript)· 确定性假 OpenAI 模型
流量 只调用 /v1/agents/sessions… 和 /api/agent/web-shell/v1/…;Workspace 绑定会话真实经过 Broker worker 与挂载目录
计量 语句数和扫描行数来自 performance_schema.events_statements_summary_by_digest,按该臂的 schema 过滤;列表计数扣除 2 秒空闲窗口内的后台语句;快照字节取 binlog 中 managed_agent_snapshot 的行事件字节;快照滞后直接从 MySQL 采样。应用内部不做任何插桩
宿主 10 核 macOS,全程有其他会话占用,负载 30–40,耗时数据噪声大,因此不对延迟下结论

1. 公开 API 可触达的三条热路径(图 1)

操作 main PR
GET /v1/agents/sessions,20 个未绑定会话 41 条语句 · 扫描 54 行 3 · 40
GET /v1/agents/sessions,20 个 Workspace 绑定会话 81 · 94 4 · 60
WebShell sessions/query,20 个未绑定会话 61 · 254 3 · 90
WebShell sessions/query,20 个 Workspace 绑定会话 141 · 1146 6 · 131
公开详情 / WebShell 详情 3 / 4 3 / 3
突发:5 个 Turn × 3000 delta(15,025 个事件) 565 次重写 · binlog 84.7 MiB · SQL 1817 ms 26 · 4.6 MiB · 121 ms
60 秒流:1500 delta、间隔 40 ms(1,505 个事件,items_json ≈ 178 KB) 258 次 · 86.0 MiB · 862 ms 14 · 4.7 MiB · 58 ms
该流上 4 个 SSE 订阅者(2 个公开流 + 2 个 WebShell 流) managed_workspace_access 查询 6,943 次(每订阅者每事件 1.15 次) 50 次(0.008)

关于这些数字:

  • 在真实栈上,main 的 Workspace 绑定 WebShell 列表页每行要 7 条语句(approval_mode、关闭状态、create-command、registry 各一次逐行读取,外加 turn 与环境事件查询),比 PR 描述中 41–81 区间所对应的每行 4 条还多。PR 的 6 条等于设计 §8 对「含创建者自有会话的绑定页」给出的上界。
  • PR 下 4 个 SSE 订阅者都按序收到了全部 1505 个事件。
  • 历史 Turn 的墙钟时间:main 23–33 秒,PR 19–32 秒。两臂的时间主要都花在每个 delta 一次的 Session Store 提交上(1,505 个事件的流分别提交了 1,608 / 1,614 次 journal),在这台机器上看不出端到端延迟的变化。

2. 实测有界陈旧(图 2)

  • 快照 / GET …/items:60 秒流式输出期间,main 的快照刷新间隔中位数为 268 ms,PR 为 5058 ms(最大 5168 ms)。PR 下未进入快照的最老事件最多 4.9 秒,最多落后 120 个事件(main 为 11 个)。两臂都在 terminal 批次时追平。「先慢流再暂停」场景里,PR 最多落后 22 个事件,在写入暂停约 5 秒后经 snapshot_stale_since 重新选中而收敛;实测 marker 先置位、后清除。WebShell transcript 因为会读取快照之后的尾部事件,全程显示所有 token。例如 9.8 秒时,transcript 已有 38 个 token,而 GET …/items 只有 19 个。

  • 读权限撤销(15 秒的流进行到第 4 秒时把 can_read 置为 FALSE):

    臂 流在撤销后关闭的时间 撤销后多送出的事件
    main 123 ms 0
    PR(默认 5 秒窗口) 4666 ms 45
    PR + read-grant-recheck-interval=PT0S 144 ms 0

    公开流与 WebShell 流表现一致;撤销后发起的新订阅在所有臂都返回 404。PT0S 同时也恢复了逐事件的查询量(6,905 次,main 为 6,943 次)。

3. 升级、回滚与滚动窗口 stamp(图 3)

同一个 MySQL 库依次经历四个阶段。每个阶段都是一对新的 Spring + Harness 进程,部署目录保持不变,与真实主机重启一致。

  1. main(V35) 创建 41 个会话,其中 29 个有 Turn。
  2. PR 在该库上启动。 Flyway 用 0.049 秒应用 V36–V39。同一批数据下,6 个列表/详情接口的响应与 main 逐字节一致(15–18 KB),每次调用的语句数从 41 / 81 / 61 / 141 / 3 / 4 降到 3 / 4 / 3 / 6 / 3 / 3。6/6 个旧会话和 1/1 个新会话的新 Turn 全部完成。
  3. main 回到 V39 库上(即回滚,或滚动发布中仍是旧版本的那一半实例)。Flyway 输出 Successfully validated 39 migrations,服务正常运行。6/6 个旧会话加 1 个新会话全部完成;一个 205 个事件的流完整送达两个 SSE 订阅者。所有阶段的 Spring 日志都没有 ERROR。
  4. PR 再次启动。 6/6 加 1/1 全部完成。

28 个带 journal 的列表会话上,journal head 激活 stamp 的变化:

  • 升级前:28 个 null。
  • PR 提交后:6 个 current。
  • 旧二进制提交后:这 6 个变为 lagging。旧二进制推进了 journal_revision,却不维护这些列;这正是 stamp 设计要识别的滚动窗口偏差,这里由真实流量产生。
  • PR 下一次携带激活变更的提交后:这 6 个重新变为 current。
  • 在 ab-pr 那一轮中,真实 Hosted Turn 把全部 31 个 head 的 V36 列都填上了,stamp 均为最新。

读侧行为(拒绝 lagging 或 null 的 head,然后回扫并回填)需要开启 tool publication,因此改在第 4 节覆盖。

放大库。 把 main 臂的数据复制 100 倍到一个 V35 schema:4,300 个会话、1,736,300 个事件、2.1 GiB。

  • V36–V39 用 4.87 秒完成,其中 V37 事件索引占 4.74 秒。
  • 随后在 214 万行上重建 V37,同时每约 5 ms 写入一个事件。建索引的 3.3 秒内完成了 474 次写入,最大延迟 6.4 ms、平均 0.54 ms,说明 InnoDB 的在线 DDL 没有阻塞事件写入。

每次重启后,旧会话上的第一个 Turn 都要等上一个 Harness 的 writer lease 过期,约 32–35 秒。这在每个阶段、两种二进制上都一样,与本 PR 无关。

4. 公开 API 触达不到的路径

发布授权(第 4 条路径)和 artifact 重新校验窗口都需要 qwen.managed-agent.tool-publication.enabled(默认 false)和对象存储,还需要 hosted-workspace-shell/1 类型的会话;而 Managed Agent API 只会创建 hosted-workspace-files/1 会话。因此在默认配置下,这两条路径都无法通过公开 API 触达。改为做了以下验证:

  • 在真实 MySQL 上运行 PR 自带的 Issue13181QueryBudgetTest。 只改了它的 fixture,让每个用例都使用一个新建的 MySQL 8.4.7 schema,并由 Flyway 完整执行 V1–V39(36 行补丁,已放在 assets 分支)。结果是 InnoDB 上 30/30 全部通过,与 H2 相同。其中包括发布路径的预算(verifyDispatch 持锁读 journal 0 次、renew 4 条 journal 语句、fenced renew 0 条),以及 stamp 相关用例:staleHeadStampRescansInsteadOfTrustingTheColumns、staleHeadStampRebackfillsAndRejoinsTheHeadPath、noChangeCommitDoesNotRestampASkewedHead、publicationAuthorizationRescansAndBackfillsPreMigrationHeads。
  • 在试合并上运行 managed-agent-server 全量单测(含 Druid 的 main 加上本 PR):共 633 个,0 失败、0 错误、1 跳过。PR 描述中提到的两个时序敏感的 ToolPublicationStoreTest 用例在这里也通过了。
  • 放大库上的执行计划(图 4):20 行列表页的语句数从 41 / 81 / 61 / 121 降到 3 / 4 / 3 / 5。两条新的批量查询都走 V37 索引:findLatestTurns 读 39 个索引项,findLatestEnvironmentEvents 读 78 行。当每个会话有 500 个 Turn 时,优化器会自动把环境事件查询切换到 managed_agent_event_turn_idx(读 40 行)。

观察(不阻断)

  1. findLatestTurns 的成本随列表中每个会话的 Turn 数增长(图 4)。它会读出每个会话的全部 turn.accepted 索引项再取 MAX,MySQL 在这里没有用 loose index scan。20 行列表页的服务端耗时:每会话 1 个 Turn 时 0.61 ms,50 个时 1.16 ms,500 个时 4.49 ms,5,000 个时 35.95 ms。main 的逐行查询每条 0.10–0.30 ms,共 20 条,另加 20 次往返,所以交叉点位于每会话 500 到 5,000 个 Turn 之间;500 个时 PR 仍更快。我认为这不阻断合并,但对超长生命周期的会话值得留意。我试过按会话 ORDER BY … DESC LIMIT 1 的改写,反而更慢(117 ms,加 FORCE INDEX 后 158 ms),所以没有提出修改方案。
  2. R5-3 确实存在,但影响很小。 PR 下 QwenHostedHarnessConnector 每个 Workspace Turn 仍会发出约 1.7 次 SELECT approval_mode 单行查询(24 个 Turn 共 41 次)。main 的 904 次主要来自列表页的逐行查询,本 PR 已经去掉。与每 1,500 个事件的 Turn 约 1,600 次 Session Store 提交相比,剩下的这些可以忽略,留作后续即可。
  3. 两项有文档说明的权衡,API 调用方都能直接看到。 慢流场景进行到 9.8 秒时,WebShell transcript 已有 38 个 token,而 GET …/items 只有 19 个。被撤销读权限的订阅者在 4.7 秒内又收到了 45 个事件。两者都与设计一致;PT0S 能以 main 的查询量为代价恢复严格行为。依赖及时撤销的运维方,可能需要在发布说明里看到这一点。

未覆盖

证据(图、每轮原始 results.json 与采样、装置 rig.mjs / scale.mjs / 配置、MySQL fixture 补丁)位于 assets-pr13217。

@qqqys qqqys left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Critical-only review at head 4d3b1351. No blocking issue found — approving.

Four rounds of this PR filed ten Criticals, so I did not treat the newest approval or the empty round-5 ledger as proof on its own. What I checked:

No Critical thread stands open at this head. I enumerated every review thread with isResolved == false through GraphQL — there are more than fifty, across the design docs, both store classes, the event-stream and artifact services and the new budget suite — and every one of them is graded [Suggestion]. Not one unresolved thread carries [Critical]. Round 5's ledger, whose sha is this exact commit, likewise files three findings and all three are sev:"S". So the R1-R4 Criticals are not merely unanswered; nothing blocking remains anchored here.

One of the re-pointed R3 Criticals I confirmed in the source myself. The write-time journal scope fence is present at ManagedExtensionRecordStore.java:190-196, building a sessionScope node from tenantId/workspaceId/sessionId with the comment that the closed-key check both read paths enforce now applies at write time too, so a misscoped line never enters the journal. I could not locate the second one's site (the cursor-less transcript-tail bound) inside this review's time box; I am relying on the round-5 pass at this head plus the absence of any open Critical thread for that one, and saying so rather than implying I traced it.

CI at this head really does cover the four new migrations. My first read of the checks was truncated at 100 of 493 and looked like almost nothing had run; paginating gives 26 successful, 455 skipped, 12 cancelled and zero failures, with Runtime Broker and Managed Agent MariaDB / Java 21 and Hosted process fault gates / MySQL 8.4 / Java 21 both green — those are the lanes that apply V36-V39 against a real database — plus the full Java matrix, Test (ubuntu-latest, Node 22.x) and Lint & Static. The Windows and macOS TypeScript lanes are skipped, which is this repo's normal routing and not a signal either way.

No migration version collision, verified against current main rather than assumed. Main's migration directory today tops out at V35__managed_session_tool_profile.sql, this head adds V36 through V39, and git merge-tree main head reports a clean merge with no conflicted paths.

Two non-blocking notes, neither a condition of this approval:

  • V36 is claimed by another open PR. #13355 adds V36__managed_task_event.sql and was, when I reviewed it earlier today, also clean against a main that stopped at V35. Only one of the two can keep the number: the Flyway uniqueness gate sees one branch at a time, so it cannot catch a cross-PR collision, and the loser fails at startup with "Found more than one migration with version 36". Whichever merges second needs renumbering to V40+. Worth settling before either lands.
  • The bounded-staleness cluster deserves a maintainer's eye even though it is graded Suggestion. The 5s SSE read-grant recheck window and the artifact revalidation window trade prompt revocation for the query savings this PR exists to deliver; the approving review records them as issue-sanctioned and reversible with PT0S, and several open Suggestions (R1-13, R2-12, R1-24, R1-14) argue the committed contract and design docs still state a stricter bound than the code now guarantees. That is a documentation-versus-behaviour gap on a security-relevant latency, so it is worth closing even though it does not block.

Coverage, so this approval is not read as broader than it is. I verified the items above and read the migration set, the scope fence and the thread state. I did not independently audit the remaining production surface — the snapshot snapshot_stale_since deferral marker and re-selection disjunct, the batched session-page assembly, the journal-head activation stamp and its journal-head-authorization flag (default off, so the shipped path is the pre-existing journal scan), or the millisLenient digit-width pre-check. Round 5 examined this head and filed nothing blocking there, which is why I am approving rather than deferring; if a maintainer wants one part attested independently, the head-authorization stamp and its self-healing rescan is where I would look first.

@yiliang114 yiliang114 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deep-review pass at 4d3b135 (commenting; qqqys's approval already stands).

Verified the four hot paths against the code, not the design text:

  • Snapshot gating: the four rewrite conditions match §3 exactly, and the deferral marker writes the snapshot's updated_at (so a fresh snapshot waits out the window; an aged one converges on the next tick). rewriteStaleSnapshot clears the marker on convergence and on recreation-by-retraction.
  • SSE recheck window: per-stream ReadGrant with a System.nanoTime window; initial grant check unchanged; session.deleted still terminates immediately.
  • V36 activation columns: commit extracts activation.changed during the existing parse pass, blanks oversized values instead of rejecting (the readers treat NULL as absent — checked against the freshness rule), and the stamp-equality rule makes a pre-V36 writer's commit self-detecting (journal_revision bumps without the stamp → rescan + backfill). Rolling window is genuinely self-healing as designed.
  • producerBindingLocked: O(1) only under the flag; legacy scan + backfill otherwise. requireEvidence's intent locate uses the V39 index — four statements regardless of journal depth, as §6 claims.
  • Migrations: V36–V39 are clean — main's head is V35, no collision.

Not line-verified: the QueryLedger budget test's exact per-endpoint counts (read the budgets, did not re-derive each query), and the E2E arms. Nothing blocking.

@wenshao
wenshao added this pull request to the merge queue Oct 4, 2026
Merged via the queue into main with commit 98b0255 Oct 4, 2026
519 of 531 checks passed
wenshao added a commit that referenced this pull request Oct 4, 2026
#13217 landed migrations V36 through V39 on main, so the task-event
outbox moves to V40.

ManagedExtensionRecordStore.apply() keeps this branch's per-line checks
and takes #13217's additions: the last activation.changed payload rides
back in ApplyResult for the journal head columns, and a non-Stage-H
event line scoped to another Session or version is still refused as
"Journal event scope conflicts", the message both publication read
paths give it. An event past the declared count now names its invalid
journal position as well.

#13217's tests built journal lines the commit now refuses: an untyped
{} in the commit marker's place, the unknown kinds tool.progress and
checkpoint.saved, and envelopes without eventId and occurredAt. They now
write real commit markers, known kinds and full envelopes through
PublicationJournalFixture. duplicateIntentLinesAtOneSequenceAreFenced
plants its duplicate in the stored revision, because the commit refuses
a line out of its sequence.
yiliang114 added a commit that referenced this pull request Oct 4, 2026
Resolve ManagedWorkspaceRegistry.java: main's #13217 anchored its new
createdSessions batch twin on isSessionCreator, which this PR deletes.
Keep the batch twin (ManagedAgentService still calls it) and keep the
deletion plus this PR's bindingCurrent probe; the merged tree has no
remaining isSessionCreator reference.

Co-authored-by: Qwen-Coder <[email protected]>
Patrol-Run: qwen-pr-conflict/jmutwkfe753
wenshao added a commit that referenced this pull request Oct 4, 2026
Reconciles the extension-store verification this branch adds with
main's #13217 (stop database amplification on session hot paths):

* The store keeps the transaction-request form of apply() so the
  declared digests and ranges stay verifiable, and returns the new
  ApplyResult (receipts + the last activation's payload) that #13217's
  commit uses instead of re-parsing the journal. The session-scope
  message unifies on "Journal event scope conflicts", pinned already
  by five sites on main; the closed-shape and position checks the
  shared event rules already covered are not duplicated.
* The migration V36 became V40 and V37 became V41: main landed its
  journal-activation V36 through snapshot-deferral V38 and journal-
  sequence V39 after this branch's numbering.
* The shared PublicationJournalFixture now commits in the journal
  shapes the store verifies — a real genesis, a commit marker ending
  every transaction and canonical content digests — instead of the
  loose "{}" fillers and raw text digests both sides previously used.
* #13217's own replay snapshots are reconciled to the store's closed
  event rules: phases, workers, lease fields and subjects completed
  where the commit still lands, and the oversized/mistyped-payload
  arms (id over maxIdBytes, a phase outside the vocabulary, a stringly
  typed or absent expiry, a misplaced second event) now pin the
  refusal where the stricter layer fires instead of assuming the
  commit stores the poison. The duplicate-intents case asserts the
  closed-domain/scope rule the line actually trips.

Verified on this tree: full Java unit suite 641 green (the three most
affected classes 133/133 first), the managed-runtime TypeScript suites
1933 green, and neutralizing both canonical digest guards turns the
declared-digest battery red before the guards were restored.
wenshao added a commit to wenshao/qwen-code that referenced this pull request Oct 7, 2026
…#13565)

Move the merged-PR history of the Managed Agent dual-path proposal
(QwenLM#12380) out of the issue body into a bilingual ledger under
docs/design/. The issue body had come within 10 KB of GitHub's 256 KiB
limit, so the body will keep only the delivery snapshot and the open
PRs, and merged rows move here rewritten to their merged final state.

The ledger carries every merged row the issue tracked up to its
2026-10-03 reconcile (adding the missing merge commit to the five
earliest rows and normalising the Chinese state cells), rewrites the
eight rows the issue still listed as open although their PRs had merged
(QwenLM#13141, QwenLM#13166, QwenLM#13174, QwenLM#13210, QwenLM#13214, QwenLM#13217, QwenLM#13218, QwenLM#13247), and
adds rows for the 48 managed-agent PRs merged between that reconcile and
main 0c13502 that had no row yet. PRs closed without merging (QwenLM#13087,
QwenLM#13336) sit in their own table. Later merges land at the next reconcile.

Co-authored-by: wenshao <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/self-reported The linked issue was opened by the PR author (self-reported)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(managed-agent): database amplification on session hot paths (snapshot rewrite, SSE, lists, publication)

3 participants