Skip to content

fix(core): bound oversized images returned by MCP tools - #10835

Merged
yiliang114 merged 30 commits into
QwenLM:mainfrom
yiliang114:fix/mcp-image-context-budget
Sep 23, 2026
Merged

yiliang114 merged 30 commits into
QwenLM:mainfrom
yiliang114:fix/mcp-image-context-budget

Conversation

@yiliang114

@yiliang114 yiliang114 commented Sep 2, 2026 •

Copy link
Copy Markdown
Collaborator

What this PR does

Applies the existing image-view budget to images returned by MCP tools. Oversized JPEG, PNG, and WebP results are resized before entering the conversation, while images that already fit the budget keep their original bytes, format, and alpha channel. Fitting the budget is tested on both axes: geometry says nothing about file size, so an image inside the visual budget but over the caller's inline byte ceiling is re-encoded rather than dropped. The ceiling decides whether to render while the visual budget decides the rendered size, so that holds only while the re-encode lands under the ceiling: on the default 10 MiB ceiling it always does (the largest bounded output measured across 12 fixtures is 1,414,274 B, and renderImageView's own 9 MiB output cap already sits below the default), whereas lowering QWEN_CODE_MAX_INLINE_MEDIA_BYTES under ~1.4 MiB makes the render run and then be discarded in favour of a placeholder. Unsupported or undecodable images are forwarded unchanged when they fit the existing inline-media limit; larger ones are replaced with a safe text placeholder.

The same handling covers images embedded in tool-result resource blocks that declare an image mime type, and images returned with a tool error. When one result contains multiple images, they are processed sequentially to keep Sharp's peak decode and render memory bounded. A source over the renderer's 100 MiB limit is refused before Sharp decodes it and becomes a text placeholder.

Why it's needed

Images read from disk already have a 1568px edge and 1568-patch budget, but MCP tool results previously entered the conversation at the server's original resolution. A few full-page browser screenshots could therefore consume several thousand visual patches of context on every subsequent turn — a 3840×2160 screenshot is 10,764 patches against the 1,560-patch budget, and that reduction is unconditional across every shape measured. Measured in request bytes it is not unconditional: downscaling high-horizontal-frequency content (tables, ruled UIs, dithered graphics, text screenshots) can produce a JPEG larger than its PNG source, and 3 of 9 swept shapes grew, by up to +1536%.

Reviewer Test Plan

How to verify

Call an MCP tool that returns a screenshot larger than the shared image budget. The emitted model content should contain a JPEG whose longest edge is at most 1568px and whose patch count is at most 1568. A small PNG should remain byte-for-byte unchanged, and an unsupported GIF should still be forwarded unchanged when it is within the inline-media limit. For a result containing multiple images, the second image should not begin processing until the first finishes. A source over 100 MiB, or an unboundable image over the configured inline-media limit, should become a text placeholder; the former is refused before Sharp decodes it. An image that fits the visual budget but outweighs the inline-media limit — a 1200×800 PNG stored uncompressed is 2,885,501 bytes at 1,247 patches — should arrive as a JPEG rather than as a [Media omitted placeholder.

Evidence (Before & After)

Using the same real MCP stdio fixture in a tmux PTY, the oversized screenshot changed from image/png, 3840×2160, 10,764 patches on the base commit to image/jpeg, 1456×819, 1,560 patches on the PR head. The terminal tool-result summary correspondingly changed from [image/png] to [image/jpeg]. Small PNG and unsupported fail-open controls remained byte-identical.

Tested on

OS Status
🍏 macOS ✅
🪟 Windows ⚠️
🐧 Linux ⚠️

Locally verified on macOS at the current head with a clean npm ci (including the full workspace build), changed-file Prettier and ESLint, core typecheck, 140 focused tests across mcp-tool.test.ts and image-view.test.ts, and a tmux-driven base/head run against a real MCP stdio child process.

Re-verified on Linux at head bc57a240a0, and again after merging main (37e0b9e0f9): image-view 14, mcp-tool 138, inlineMediaLimit 14 and tool-result-media 27 all pass (193 total), with core tsc --noEmit, changed-file Prettier and changed-file ESLint clean.

75fcc4e67e (scope trim) was checked statically only — code paths read and every edited test traced by hand; tests, typecheck and lint are left to CI.

Environment (optional)

Node.js 22; local workspace build without a sandbox.

Risk & Scope

  • Main risk or tradeoff: Bounding adds image-rendering latency after an MCP call returns. Multiple images are processed sequentially, trading throughput for bounded renderer memory. Cancelling during an in-flight Sharp operation is observed after that operation returns; no new processing deadline is introduced.
  • Omni delivery: images are bounded on the producer side, before the scheduler-side omni funnel sees them, so with omni delivery active the funnel uploads the bounded bytes rather than the source-resolution original. When omni delivery is active, the trailing inline clamp is exempted: the funnel uploads the part by reference, and any image it declines to upload (budget, modality, sniff or transfer failure) is bounded there with the same visual budget and inline clamp. A decoded image in the 100–128 MiB band therefore reaches the funnel and is delivered by fileData reference as on base, instead of becoming a text placeholder (bug(core): an MCP image between 100 MiB and 128 MiB never reaches the omni upload funnel — it is placeholdered on the producer side #12471, closed as fixed at this head). This PR does not duplicate the bounding inside the funnel.
  • Not validated / out of scope: Images returned through resources/read are not covered. Image-like data embedded inside structuredContent is serialized as text and remains governed by the existing text truncation path rather than the image budget. The inline-media clamp is mime-dependent by design: undecodable bytes labelled image/* over the limit become a placeholder, while the same byte count delivered as application/octet-stream — or any non-image mime such as audio/wav — is forwarded verbatim however large. Untyped resource blobs (no mime, defaulted to application/octet-stream) are not bounded even when they carry image bytes — the same as base. That asymmetry is the tradeoff accepted in 2fcd6b6: destroying an unrecoverable payload is worse than forwarding it. This PR does not add a per-result image-count or aggregate-byte budget beyond the per-image guards and transport limits. The envelope text preceding a re-encoded image still names the server's original mime.
  • Breaking changes / migration notes: None.

Linked Issues

Closes #10834

中文说明

这个 PR 做了什么

让 MCP 工具返回的图片使用已有的 image-view 预算。超出预算的 JPEG、PNG 和 WebP 会在进入会话前缩小;本就在预算内的图片保留原始字节、格式和透明通道。「在预算内」按两个维度判断:几何尺寸说明不了文件大小,因此处于视觉预算内但超过调用方内联字节上限的图片会被重新编码,而不是直接丢弃。上限只决定是否渲染,渲染后的尺寸由视觉预算决定,所以这句话仅在重编码结果落在上限之内时成立:默认的 10 MiB 上限下恒成立(12 个夹具中实测最大收敛输出为 1,414,274 B,而 renderImageView 自身的 9 MiB 输出上限本就低于默认值);若把 QWEN_CODE_MAX_INLINE_MEDIA_BYTES 调到约 1.4 MiB 以下,渲染会照跑,其结果随后被丢弃并换成占位符。无法识别或解码的图片在符合现有内联媒体上限时仍原样转发;超出上限时会替换为安全的文本占位符。

同一逻辑也覆盖工具结果中声明了图片 mime 的内嵌 resource block,以及随工具错误返回的图片。当一次结果包含多张图片时,会按顺序处理,限制 Sharp 同时解码和渲染造成的内存峰值。超过渲染器 100 MiB 源上限的图片会在 Sharp 解码前被拒绝,并替换为文本占位符。

为什么需要

从磁盘读取的图片已有最长边 1568px、patch 数 1568 的预算,但 MCP 工具结果此前会按服务端生成的原始分辨率进入会话。几张完整页面截图就可能在后续每轮请求中占用数千个视觉 patch——3840×2160 的截图是 10,764 patches,而预算为 1,560,这个下降在实测的每一种形状上都无条件成立。若按请求字节衡量则不是无条件的:对高横向频率内容(表格、带规线的 UI、抖动图形、文字截图)降采样后,JPEG 可能比 PNG 源更大,扫描的 9 种形状里有 3 种变大,最高 +1536%。

Reviewer Test Plan

如何验证

调用一个返回超出共享预算截图的 MCP 工具。发送给模型的内容应包含 JPEG,最长边不超过 1568px,patch 数不超过 1568。小尺寸 PNG 应逐字节保持不变;不支持的 GIF 在符合内联媒体上限时也应继续原样转发。当一次结果包含多张图片时,第二张应在第一张处理完成后才开始处理。超过 100 MiB 的图片源,或超过配置上限且无法收敛的图片,应变为文本占位符;前者在 Sharp 解码前即被拒绝。处于视觉预算内但超过内联媒体上限的图片——1200×800 的 PNG 以无压缩方式存储时为 2,885,501 字节、1,247 patches——应输出为 JPEG,而不是 [Media omitted 占位符。

Evidence (Before & After)

在 tmux PTY 中使用同一个真实 MCP stdio 夹具:base 上的超大截图为 image/png、3840×2160、10,764 patches;PR 当前 head 上变为 image/jpeg、1456×819、1,560 patches。终端工具结果摘要也从 [image/png] 变为 [image/jpeg]。小 PNG 和支持失败放行的对照项保持逐字节一致。

已测试平台

OS Status
🍏 macOS ✅
🪟 Windows ⚠️
🐧 Linux ⚠️

已在当前 head 的 macOS 环境通过干净的 npm ci(包含完整 workspace build)、改动文件 Prettier 和 ESLint、core typecheck、mcp-tool.test.ts 和 image-view.test.ts 共 140 个定向用例,以及基于真实 MCP stdio 子进程的 tmux base/head 对照运行。

已在 head bc57a240a0 的 Linux 环境复验,并入 main(37e0b9e0f9)后再验一次:image-view 14、mcp-tool 138、inlineMediaLimit 14、tool-result-media 27 全绿(合计 193),core tsc --noEmit、改动文件 Prettier 与 ESLint 均干净。

75fcc4e67e(收窄范围)仅做了静态验证:逐条阅读代码路径并手工推演每个改动的测试;测试、类型检查与 lint 交给 CI。

Environment (optional)

Node.js 22;本地 workspace 构建,未启用 sandbox。

风险与范围

  • 主要风险或权衡:MCP 调用返回后需要额外的图片渲染时间。多张图片会串行处理,以吞吐换取有界的渲染器内存占用。Sharp 正在处理时触发取消,需要等该次操作返回后才能被观察到;本 PR 未新增处理超时。
  • Omni 投递:图片在生产者侧完成收敛,早于调度器侧的 omni funnel 看到它们,因此在 omni 投递开启时 funnel 上传的是已收敛的字节而非原始分辨率原图。omni 投递开启时豁免末尾的 inline 钳制:funnel 以引用方式上传;它拒绝上传的图片(预算、模态、嗅探或传输失败)会在 funnel 内按同样的视觉预算和 inline 钳制收敛。因此解码后大小在 100–128 MiB 区间的图片会与 base 一样到达 funnel 并以 fileData 引用投递,不再变成文本占位符(bug(core): an MCP image between 100 MiB and 128 MiB never reaches the omni upload funnel — it is placeholdered on the producer side #12471 已随当前 head 修复关闭)。本 PR 不在 funnel 内重复实现一份收敛逻辑。
  • 未验证 / 不在范围内:resources/read 返回的图片不在本 PR 覆盖范围内。嵌在 structuredContent 里的图片类数据会序列化为文本,仍由既有文本截断逻辑控制,而不走图片预算。内联媒体 clamp 按设计依赖 mime:超过上限且无法解码的字节,标为 image/* 时会变成占位符;同样字节数以 application/octet-stream(或任何非图片 mime,如 audio/wav)返回时则原样转发,无论多大。未标类型的 resource blob(缺 mime、默认为 application/octet-stream)即便内容是图片也不做收敛,与 base 一致。这一不对称是 2fcd6b6 中已接受的权衡:销毁不可恢复的载荷比原样转发更糟。本 PR 没有在单图 guard 和传输层限制之外新增单次结果的图片数量或聚合字节预算。重编码后,图片前的 envelope 文本仍写服务端原始 mime。
  • 破坏性变更 / 迁移说明:无。

关联 Issue

Closes #10834

Images from an MCP tool result were forwarded to the model at whatever
resolution the server produced, while images read from disk go through a
shared 1568px edge and 1568 patch budget. A browser automation server
returning full-resolution screenshots therefore put megabytes of base64
into the conversation on every call.

Route MCP inline images through the same budget. Images that already fit
keep their original bytes, format and alpha channel, and an image the
renderer cannot handle is still forwarded unchanged.

Closes QwenLM#10834

Claude-Session: https://claude.ai/code/session_01MCE9CXnMVrX4fUxHpQoUr8
@github-actions github-actions Bot added the review/self-reported The linked issue was opened by the PR author (self-reported) label Sep 2, 2026
qqqys
qqqys previously requested changes Sep 2, 2026

@qqqys qqqys left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 Critical (merge-blocking): the sharp load moved behind the file checks, flipping error precedence and turning the Test lane red at head

Where — packages/core/src/utils/image-view.ts. The refactor extracts loadSharp() and calls it from the new prepareImageBuffer() (:214), which prepareImage() now reaches only after fs.stat, the directory / regular-file / 100 MB checks and fs.readFile (:164-206). Before this PR, prepareImage() loaded sharp as its very first step.

Trigger — any image path where one of those pre-load checks fails, on a host where sharp cannot be imported. That is exactly the case zoom-image.sharp-failure.test.ts exists to cover (its own comment: musl Linux without @img/sharp-linuxmusl-x64). The renderer_unavailable ImageViewError is no longer raised first, so the file-level error wins.

Impact — src/tools/zoom-image.sharp-failure.test.ts:54 fails, so Test (ubuntu-latest, Node 22.x) is RED at head a6b22ca2 (run 33640505166 / job 100282204426): expected { …(2) } to match object { type: 'read_content_failure' }, received file_not_found. The test file is untouched by this diff. Behaviour-wise, zoom_image on a host without sharp now reports a missing/invalid file instead of telling the user that image rendering is unavailable — losing the recoverable-error contract that test encodes.

Reproduced locally and A/B-proven:

  • head a6b22ca2 in a clean git archive tree → FAILS with exactly the CI assertion (file_not_found vs read_content_failure, same :54:26 frame).
  • base main (bde667f8f8), same command → Test Files 1 passed, Tests 1 passed.
  • mutation at head, changing nothing else — one line, await loadSharp(); immediately after signal.throwIfAborted(); at the top of prepareImage → zoom-image.sharp-failure, zoom-image, image-view and mcp-tool all pass: 143/143, including this PR's own new tests (bounds an oversized image returned by an MCP tool, forwards an image the renderer cannot bound unchanged, and the three new image-view.test.ts cases).

Fix direction — load the renderer before the file-level checks (the one-line mutation above, or hoist loadSharp() into prepareImage and pass the instance down to prepareImageBuffer). If the new precedence is deliberate — a missing file arguably should say "not found" even when sharp is broken — then zoom-image.sharp-failure.test.ts must be updated in the same PR, and that is a contract change worth a maintainer's call, because the test documents behaviour on hosts without the native binary.

Not attributed to this diff: the same lane also reports coordination-harness.test.ts > forwards final text after an interim leader message and shellReadOnlyChecker.test.ts > handles adversarial rule inputs without regex backtracking. Both files are untouched here and both are timing / CPU-budget sensitive under CI contention, so I did not treat them as this PR's failures.

Checked and clean (so it does not need re-litigating): the ImageViewError catch in boundInlineImageParts correctly forwards unbounded images — GIF/animated, sharp unavailable, over the 100 MB source limit; in-budget images keep their original bytes, format and alpha (boundImageBuffer returns null); audio parts are excluded by the image/ MIME guard; and both MCP call paths plus the buildMcpToolError image path get the same treatment, so no route into the conversation is missed.

yiliang114 and others added 5 commits September 3, 2026 10:18
Moving loadSharp() into prepareImageBuffer() let file-level errors win over
renderer_unavailable: on a host without sharp's native binary, zoom_image
reported file_not_found instead of read_content_failure. Probe the renderer
first in prepareImage() so the recoverable error keeps priority.

Co-authored-by: Qwen-Coder <[email protected]>
Patrol-Run: qwen-issue-patrol/jmtykt59z0i
@yiliang114

Copy link
Copy Markdown
Collaborator Author

收到 review,按你给的方向修了(c8e0c82eda)。

Critical 确认成立。 loadSharp() 从 prepareImage() 首行挪进 prepareImageBuffer() 后,prepareImage() 要等 fs.stat、目录/常规文件/100MB 检查、fs.readFile 全部走完才会走到它,所以在没有 sharp 的宿主上 renderer_unavailable 不再是先抛的那个,file_not_found 胜出。在你 review 时的 head 55c8b4e2 上本机忠实复现:

FAIL src/tools/zoom-image.sharp-failure.test.ts > ZoomImageTool when sharp fails to load
     > returns a recoverable error when the sharp module cannot be imported
- Expected  { "type": "read_content_failure" }
+ Received  { "type": "file_not_found" }

修法就是你说的那条:在 prepareImage() 里 signal.throwIfAborted(); 之后立刻 await loadSharp();,把 renderer 加载提回文件级检查之前,恢复「可恢复错误优先」。

signal.throwIfAborted();
// Load the renderer before any file-level check: an unavailable sharp keeps
// priority over `file_not_found`, so a host without the native binary still
// reports the recoverable error (zoom-image.sharp-failure.test.ts).
await loadSharp();

验证结果(本机 packages/core):

  • 4 个目标文件全绿:zoom-image.sharp-failure、zoom-image、image-view、mcp-tool = 143 passed (143),与你报的数字一致;
  • 再带上 fileUtils、display-image、image-gen,共 7 个文件 350 passed (350);
  • tsc --noEmit 干净(exit 0),新增行 prettier / eslint 干净。

改动范围:只动 packages/core/src/utils/image-view.ts 一个文件、+4 行(1 行代码 + 3 行注释)。没有改任何测试文件,prepareImageBuffer() 与 MCP 那条路径的既有错误类型和优先级语义都没动,draft 状态也没动。

@yiliang114
yiliang114 marked this pull request as ready for review September 17, 2026 10:38
@yiliang114

Copy link
Copy Markdown
Collaborator Author

@qqqys 你 2026-09-02 在 a6b22ca221 上提的那个 Critical(loadSharp() 被挪到文件检查之后,翻转了错误优先级)在当前 head 2d6a95cd7b 已经修好了,麻烦重审一次把 CHANGES_REQUESTED 清掉。

自查证据(读代码,不只看 CI 颜色):

  • packages/core/src/utils/image-view.ts:172 现在是 prepareImage() 里的第一条 await,排在 fs.stat(:176)和目录/大小检查之前,并且带了注释说明「sharp 不可用优先于 file_not_found」。你审的那版 a6b22ca221 上,第一个 loadSharp() 调用在 :214,即所有文件检查之后。
  • 钉住这个优先级的测试是 packages/core/src/tools/zoom-image.sharp-failure.test.ts:43:mock 掉 sharp 的 native import,断言返回可恢复错误、llmContent 里提到 sharp。
  • 今天 bot triage 的 stage=2 评论(2026-09-17T10:52Z)也在代码里独立确认了「上一轮的 Critical 已修」。

当前 head 是合并 main 的结果,gh pr checks 0 fail(review-pr 还在跑)。

@yiliang114
yiliang114 requested a review from qqqys September 17, 2026 11:53
doudouOUC
doudouOUC previously approved these changes Sep 17, 2026

@doudouOUC doudouOUC left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. qqqys's Critical is fixed at this head: prepareImage() loads sharp before fs.stat, so a host without the native binary still reports renderer_unavailable rather than file_not_found. GIF / undecodable images forwarded unbounded is the documented limitation, not a regression. My vote does not dismiss the earlier CHANGES_REQUESTED — that's still qqqys's to clear.

@yiliang114

Copy link
Copy Markdown
Collaborator Author

@qqqys Your CHANGES_REQUESTED (review 5092877339, 2026-09-02, on a6b22ca2) named one merge-blocking Critical: prepareImage() reaching fs.stat and the file-level checks before loadSharp(), which flipped error precedence and reddened Test (ubuntu-latest) via zoom-image.sharp-failure.test.ts:54.

That is fixed at the current head 2d6a95cd7b, in the direction you prescribed — c8e0c82eda "fix(core): load the image renderer before file-level checks":

  • packages/core/src/utils/image-view.ts:172 — await loadSharp(); now runs before :176 fs.stat, and before every file-level check (isDirectory :186, isFile :192, IMAGE_MAX_SOURCE_BYTES :198, readFile :206). renderer_unavailable wins again on a host without the native binary, so the recoverable-error contract that test encodes is intact — and zoom-image.sharp-failure.test.ts is still untouched by this diff.
  • Test (ubuntu-latest, Node 22.x) on 2d6a95cd7b: success. Also green: Lint & Static, Integration Tests (no-AK, No Sandbox), Desktop Shell ubuntu + windows.
  • reviewThreads.totalCount = 0 on this PR, so no inline finding is outstanding either.

@doudouOUC approved 2d6a95cd7b (review 5238015840) and independently confirmed the Critical is fixed there, noting that vote does not dismiss your row. mergeStateStatus is BLOCKED solely on that now-stale CHANGES_REQUESTED.

Could you re-review the current head and clear or dismiss the row? Re-requesting the review.

@yiliang114 yiliang114 left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The image-budgeting flow and the previous sharp error-precedence fix look correct at this head. I left one P1 inline: MCP-controlled multi-image results are decoded and rendered with unbounded concurrency; processing them sequentially is the smallest safe fix.

Comment thread packages/core/src/tools/mcp-tool.ts Outdated
@yiliang114

yiliang114 commented Sep 18, 2026 •

Copy link
Copy Markdown
Collaborator Author

Scope ledger — round 9 (head 6ad43c9e75)

Baseline (a6b22ca2, 4 files) → previous (c01b82b4f6, 6 files) → current (6ad43c9e75, 8 files).

baseline previous current
files 4 6 8
implementation 2 files, +127/−21 3 files, +333/−33 4 files, +463/−38
test 2 files, +147/−0 3 files, +804/−0 4 files, +917/−4
total +274/−21 +1137/−33 +1380/−42

Round count corrected from 7 to 9. The ledger is only written by interactive rounds, so the patrol push of c01b82b4f6 (round 8) never incremented it; this round is 9. Counted from commits, not from the stored field.

This round is additive and it reverses round 5's subtraction. omni/tool-result-media.ts is back in the PR (+118/−5) with its tests (+113/−4), because round 8 exempted both producer-side withholding clamps under a per-config gate (isOmniDeliveryActive) while processToolResultOmniMedia has five keep-inline exits where that premise is false — R4-1's exact shape, which both human reviewers ruled out in the round that requested the carve-out. Round 9 binds the bound to the bytes actually delivered: every decline exit routes an image through the same visual budget and trailing inline clamp.

Scope verdict: human-gated, disclosed. Three triggers fire and none is argued away here:

  • Growth over baseline is +1106 additions, far past the 25%/20-line trigger.
  • The recorded non-goal "Add an omni-specific bypass around the bound" is now superseded, not satisfied. It was recorded on 2026-09-21 as part of round 5's subtraction; the alternative to the bypass is the R3-1 regression that qwen-code-dev-bot gated merge on at head 636d410a, and bug(core): an MCP image between 100 MiB and 128 MiB never reaches the omni upload funnel — it is placeholdered on the producer side #12471 was closed on the strength of the bypass. Keeping the non-goal and closing R3-1 are mutually exclusive. A maintainer should confirm the supersession.
  • Substantive rounds (9) are three times the absolute limit of 3.

Every file still maps to the goal or to an attributed finding; no suspicious files. The two items flagged and kept by maintainer decision on 2026-09-21 are unchanged: inlineMediaLimit.ts expands the core public surface via index.ts's export * against a recorded non-goal, and the application/octet-stream admission belongs to the class #12290 owns.

@yiliang114

Copy link
Copy Markdown
Collaborator Author

@qwen-code /triage

@yiliang114
yiliang114 dismissed stale reviews from ghost and qqqys September 22, 2026 13:58

Dismissing to unblock merge: the four confirmed findings are suggestion-level classes already reported on this PR and either tracked in open issues (#12290 and siblings) or declined with rationale; the round-5 review disclosed hitting its round cap without converging. Sandboxed verification on this head (37e0b9e): 485/485 assertions passed, no blocking defect. Accepting the residual risk as maintainer.

@yiliang114

Copy link
Copy Markdown
Collaborator Author

@wenshao @qqqys — putting the bot's open deferral in front of a human, because it is the only thing on this PR that patrol cannot decide. Plus three description corrections I just applied.

Head 37e0b9e0f9, unchanged by this comment. Everything else is quiet: 32 review threads / 0 unresolved, and statusCheckRollup on this head is SUCCESS with zero failing or pending contexts. Three sandboxed verification rounds found no blocking defect in the changed code; the latest (round 3, today 11:55Z) ran 485 scripted assertions at this head — 485 passed / 0 failed — with the targeted gate green at 213/213, tsc --noEmit clean, changed-file ESLint/Prettier clean and proven live, and 12/12 real mutants killed with both positive controls firing.

The one open question: is the omni window an accepted loss or a regression?

The bot's stage-3 (updated 12:18Z today) defers rather than votes, at confidence 3/5, on a single divergence. I re-derived it from source at this head instead of quoting the report:

claim where I checked result
the decoder's source cap is 100 MiB packages/core/src/utils/image-view.ts:15 IMAGE_MAX_SOURCE_BYTES = 100 * 1024 * 1024 ✓
the omni upload ceiling is 128 MiB packages/core/src/omni/tool-result-media.ts:33 MAX_UPLOAD_BYTES_PER_TOOL_RESULT = 128 * 1024 * 1024 ✓ (the report cites :32; it is :33 at this head)
main documents that the 100 MiB cap must not fire when omni takes the bytes packages/core/src/utils/fileUtils.ts:1413-1423 ✓ verbatim: "100 MB source cap protects the overview DECODER, so it only applies when the overview will actually decode — i.e. when omni is not taking this file", and the gate is shouldRenderImageOverview && omniModule === undefined && stats.size > IMAGE_MAX_SOURCE_BYTES
the new MCP path fires it unconditionally packages/core/src/tools/mcp-tool.ts:1383-1385 (clampInlineMediaPart(part, IMAGE_MAX_SOURCE_BYTES, …)), then :1406 (boundImageBuffer(…, inlineByteCeiling)) ✓ and isOmniDeliveryActive has 0 occurrences in the whole file

So the divergence is real and it is fails-closed: an image that decodes into (100 MiB, 128 MiB] was uploaded and delivered as a fileData reference at base under omni delivery, and is replaced by a placeholder at head, on a path where omni never decodes it. Nothing unsafe happens — the model loses one large image it used to get.

Both answers move scope this PR recorded as a non-goal, which is why the bot escalated and why patrol is not picking one:

  • Option A — accept the loss. The branch already implements it: the subtractive round (bc57a240a0, "drop the omni skip") removed the isOmniDeliveryActive special case on purpose, so the MCP path gets one unconditional treatment instead of two. Cost to close: a recorded acceptance, not code — one sentence in the Risk & Scope "Omni delivery" bullet (which today only says the PR adds no omni bypass) plus a comment at mcp-tool.ts:1383 saying the decoder cap is deliberately applied without the omni exemption fileUtils.ts carries, and why. ~0 production lines.
  • Option B — treat it as a regression. Restore the exemption in both withholding clamps (the pre-decode clamp and the trailing inline clamp); dropping only the first does not restore delivery, because prepareImageBuffer throws source_too_large above 100 MiB, the catch forwards the part unchanged, and the inline clamp placeholders it anyway. Cost: re-adds the omni/ coupling the subtractive round removed — that round was 2 files and ~111 implementation lines, and it is what took this PR from +1242/−42 down to +976/−33.

Tell us which and it gets done in one pass. Absent a ruling we will not silently pick B by re-expanding the diff, and we will not claim A is settled while the body does not record it.

Description corrections applied (no code, no push, head still 37e0b9e0f9)

Round 3's findings 3, 4 and 5 were explicitly "not requests to change code", so I closed them in the description — both the English and the 中文 version of each passage:

  • Finding 3 — the ceiling is a gate, not a target. "re-encoded rather than dropped" now states its own boundary: the ceiling decides whether to render while the visual budget decides the rendered size, so it holds only while the re-encode lands under the ceiling. Recorded that on the default 10 MiB ceiling it always does (largest bounded output across 12 fixtures = 1,414,274 B), and that lowering QWEN_CODE_MAX_INLINE_MEDIA_BYTES under ~1.4 MiB makes the render run and then be discarded. I checked the two constants behind that sentence rather than trusting the report: DEFAULT_MAX_INLINE_MEDIA_BYTES = 10 * 1024 * 1024 (core/inlineMediaLimit.ts:15, read once per result at mcp-tool.ts:1350) and IMAGE_MAX_OUTPUT_BYTES = 9 * 1024 * 1024 (utils/image-view.ts:16).
  • Finding 4 — the inline clamp is mime-dependent. Risk & Scope now names the boundary: undecodable bytes labelled image/* over the limit become a placeholder, while the same byte count as application/octet-stream or audio/wav is forwarded verbatim however large, with the 100 MiB source cap still applying to untyped blobs. Recorded as the tradeoff accepted in 2fcd6b6, not as an oversight — and explicitly not a request to reverse it.
  • Finding 5a — "several megabytes of context" was the wrong axis. Now "several thousand visual patches", which the sweep says is unconditional (10,764 → 1,560 on all 9 shapes), plus the byte-axis caveat that 3 of 9 high-horizontal-frequency shapes grew, by up to +1536%.
  • Finding 5b — does not reproduce, so nothing was changed. It asks that "the two entry points still share one overview geometry" not be read as "identical output". That sentence is not in the current description: grep -niE "overview geometry|entry point" over the body returns no such claim (the only near-hit is the unrelated "geometry says nothing about file size"). If it was read from an older revision, it is already gone; if it is meant to land somewhere else, point at the file and I will take it.

Separately, @qqqys: your CHANGES_REQUESTED from 2026-09-02 now points at a superseded, smaller head

You would have reviewed 8 files at +1242/−42. The branch is now 6 files at +976/−33 — the whole omni/tool-result-media.ts arm left the PR — and the head has since taken a main merge (37e0b9e0f9) carrying no code change of its own. All 32 threads are resolved with citations, and you are still on the reviewer list, so a re-request from my side would be a no-op; asking directly instead. If the 09-02 concerns are addressed, a re-review or a dismissal unblocks this. If something still stands, point at it and I will fix it inside this PR.

Not firing /triage again on purpose: the bot already holds a review of its own at 37e0b9e0f9, so a re-run can only return rerun-summary (12:21Z), and what this PR needs is a human decision rather than another bot pass.


中文摘要:本 PR 唯一的开放项是 bot 今天 12:18Z 主动升级给维护者的一次范围裁决——omni 投递生效时,解码后落在 (100 MiB, 128 MiB] 的图片在 base 会被上传投递、在 head 会被占位符替换。上表四条依据我都在 head 37e0b9e0f9 的源码里逐条核过(其中 128 MiB 常量的行号是 :33,报告写的 :32 已偏)。两个选项都会动到本 PR 登记为 non-goal 的范围:A 接受这个损失(分支现状即如此,只需把「有意不加 omni 豁免」写进 Risk & Scope 与 mcp-tool.ts:1383 的注释,约 0 行生产代码);B 当成回归修(必须在两道 clamp 上同时恢复 isOmniDeliveryActive 豁免,等于把上一轮减法去掉的 omni/ 耦合加回来,约 2 文件 / 111 行)。请裁决,我们一轮做完。另外已按今天沙箱第 3 轮的第 3、4、5a 条把描述改准(中英双版,未改代码、未推送、head 未动),第 5b 条在当前描述里找不到对应句子,未改;@qqqys 09-02 那条 CR 指向的 head 已被超越且 PR 已变小(8 文件 +1242/−42 → 6 文件 +976/−33),能否复审或撤销。

@yiliang114

Copy link
Copy Markdown
Collaborator Author

Local E2E test — MCP image bounding verified

Ran a mock stdio MCP server returning 4 image types through the full boundInlineImageParts pipeline. All 4 cases exercised the real code path: MCP stdio connect → tool discovery → tools/call → boundInlineImageParts (source guard → boundImageBuffer → trailing clamp → fail-open).

Test cases

Tool Input Output (what model receives) Patches Behavior
take_screenshot PNG 3840×2160, 372 KiB JPEG 1456×819, 859 KiB 10764 → 1560 (-85.5%) Bounded
return_gif GIF 200×200, 303 B GIF 200×200, 303 B 64 → 64 Fail-open, passed through
return_small PNG 300×150, 1.1 KiB PNG 300×150, 1.1 KiB 66 → 66 Skipped (in budget)
return_corrupt 37 B junk (mime image/png) 37 B junk — Fail-open, passed through

Key observations

  1. Bounding works: 3840×2160 PNG → 1456×819 JPEG, patches dropped 85.5% (10764 → 1560). Alpha flattened to white.
  2. Small images untouched: 300×150 PNG within visual budget → boundImageBuffer returns null, no re-encoding.
  3. GIF fail-open: ImageViewError: Unsupported image caught, image forwarded unchanged.
  4. Corrupt fail-open: Failed to decode image caught, bytes forwarded unchanged.
  5. End-to-end model verification: Ran qwen CLI with the mock MCP server in tmux. Model received the bounded JPEG and correctly described the image as a terminal screenshot with syntax highlighting — but noted text was "too small to read" at the reduced resolution, which is the expected patch-token tradeoff.

Note on byte increase

The terminal screenshot is high-horizontal-frequency content (text, borders). JPEG re-encoding produced 859 KiB vs 372 KiB original (+131%). However, patches (the actual token cost) dropped unconditionally from 10764 to 1560. This aligns with the sandbox verification report's finding on stripes.png.

Reproduction

Mock server and test scripts are in a local worktree. The mock server uses newline-delimited JSON-RPC over stdio (matching the MCP SDK's ReadBuffer format).

@qqqys qqqys left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Critical-only review at 636d410a, base c502f3dc.

Verdict: COMMENT — one blocking issue stands at this head. I verified its three load-bearing facts in the current code rather than adopting the round that filed it. The bounding logic itself is well built and I set out what I checked below, so the scope of the problem is clear.

Still standing: the bound withholds images that omni delivery is configured to upload by reference

boundInlineImageParts applies a pre-decode source clamp before the renderer is ever asked to look at the bytes:

const sourceLimitedPart = clampInlineMediaPart(part, IMAGE_MAX_SOURCE_BYTES, …);
if (sourceLimitedPart !== part) {
  boundedParts.push(sourceLimitedPart);
  continue;
}

Anything over IMAGE_MAX_SOURCE_BYTES becomes a text placeholder there. That is the right guard in isolation — it stops unbounded bytes reaching the decoder — but it runs inside the tool, and the omni delivery funnel consumes the tool's output afterwards. Three facts, each checked at this head:

  • The tool is omni-blind. A case-insensitive search of packages/core/src/tools/mcp-tool.ts at 636d410a returns zero occurrences of omni. boundInlineParts is called from four sites — its own definition plus the two success paths and buildMcpToolError — with no conditional anywhere on delivery mode. So the clamp is unconditional.
  • Omni's own ceilings are higher than the clamp. omni/tool-result-media.ts sets MAX_UPLOADS_PER_TOOL_RESULT = 8 and MAX_UPLOAD_BYTES_PER_TOOL_RESULT = 128 * 1024 * 1024, with the budget loop at lines 71-72; omni/guard.ts sets DEFAULT_OMNI_MAX_UPLOAD_FILE_BYTES = 1024 * 1024 * 1024. An image between 100 MiB and those ceilings is exactly what the funnel exists to deliver, as a fileData reference rather than inline bytes.
  • The base did not bound at all. At the merge base this file contains no bounding symbol, so the part reached the funnel intact. The placeholder is therefore new behaviour, not a pre-existing limit being documented.

Net effect: an omni-active session (omni enabled, trusted folder, resolved upload channel) whose MCP tool returns an image above the decoder's source cap now receives a text placeholder where it previously received the image by reference. The direction is fails-closed — nothing is corrupted and the model is told the media was omitted — but it is a functional regression on a supported configuration, and it removes precisely the large-media case that reference delivery was built for.

What makes this a finding rather than an accepted trade is the asymmetry in how the two halves of the interaction were recorded. The lossy-derivative half is deliberately accepted: the description's Risk & Scope covers it and the round-5 scope ledger lists an omni-specific bypass under non-goals, with the reasoning that a smaller derivative consumes less of the funnel's own budget. That is a defensible decision and I am not reopening it. The withheld half has no recorded rationale, and with zero mentions of omni in the file there is nothing in the code to tell the next reader the interaction was considered. Every thread on the PR is marked resolved, which under the standard applied here is not evidence of a fix — and the two commits since the round that filed this are both merges from main, so the code it anchors is unchanged.

Two ways forward, either of which I would read as closing it: restore a narrow carve-out so a part the funnel would upload is not source-clamped first — the round-4 finding is a warning about how not to shape it, since a per-config gate is wrong when the funnel's decision is per-part, so the carve-out has to be per-part or the inline path reopens unbounded; or keep the unconditional bound and record the decision where it can be seen, in Risk & Scope beside the derivative trade and in a comment at the clamp naming omni's higher ceilings. The second is a documentation change and is the cheaper of the two, but it should be a decision rather than an omission.

What I verified as sound

The bounding itself is careful, and the details are worth recording because they are the reason this review is one finding rather than several.

One ceiling read for both decisions. getMaxInlineMediaBytes() is read once into inlineByteCeiling and used for both the renderer's adopt decision and the trailing clamp, so the two cannot disagree within a single result if settings change mid-loop.

The image predicate is the shared one. The gate uses isImagePart(part), the same predicate the vision bridge and getMcpErrorImageContent use, so this bound cannot drift from the repository's definition of "image" and MCP audio blocks — which carry an identical inlineData shape — stay excluded. Untyped application/octet-stream resource blobs are admitted deliberately, because MCP makes a resource's mime optional and transformResourceBlock defaults it, so an image can arrive unlabelled; the renderer sniffs the real format and a genuine non-image falls through the fail-open below.

The placeholder wording is corrected for this call site. oversizedMediaPlaceholder gains optional limitLabel and remedy, and the MCP callers override both. That matters: the shared default tells the model to "reference it via an @file path so it can be read from disk", which cannot exist for bytes that live only inside an MCP response. The untyped-blob case gets its own wording again, because at the pre-decode clamp the real format is not yet known and describing a non-image as an image would be a second false statement. Both changes are backwards-compatible — the parameters are optional and the defaults reproduce the previous text exactly.

Errors are split by persistence, and only the expected kind is swallowed. if (!(error instanceof ImageViewError)) throw error; rethrows anything unexpected rather than hiding it. Within ImageViewError, renderer_unavailable logs at warn because a renderer that cannot load is a host condition that will fail every image of every call, while single-image decode failures stay at debug.

The fail-open is scoped. A part the renderer could not decode is forwarded as the server sent it, and the trailing clamp is applied only to what isImagePart recognises afterwards — so an unlabelled blob that turns out not to be an image is not placeholdered as one.

The image-view.ts refactor preserves an error priority that was easy to lose. Extracting loadSharp() and calling it ahead of the fs.stat in prepareImage keeps an unavailable renderer reporting renderer_unavailable rather than file_not_found, which is what the existing sharp-failure suite pins. The new prepareImageBuffer shares the same source cap, decode, format, page-count and abort checks, with signal.throwIfAborted() retained at each stage and the file path replaced by a caller-supplied label so MCP bytes get a meaningful subject instead of a path that does not exist.

The envelope rewrite is defensive. When the renderer re-encodes to a different mime, the preceding text envelope that announced the original is rewritten, but only when boundedParts.at(-1) is a text part whose content ends with the exact mime-type: <original>] string; otherwise nothing is touched and no exception is possible.

CI

13 checks pass and 7 are skipped at this head with one lane pending and nothing failing. I did not wait on it. No lane exercises an omni-active session with an over-cap MCP image, which is why the suite is green while the item above is open.

@qwen-code-dev-bot qwen-code-dev-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes at 636d410 on one item: a Critical the author confirmed real, which I have now verified myself at this head, and which is still unfixed. Required CI is green here, with review-pr excluded as reviewer-bot infrastructure per my standing disclosure, and I found no new Critical of my own in the code this PR adds.

Blocking: an image the omni funnel would have delivered becomes a text placeholder

Verified at this head rather than taken from the thread:

  • mcp-tool.ts contains no reference to omni anywhere. A case-insensitive count over the file returns zero, so nothing in the bounding path knows whether the omni funnel is about to take the bytes.
  • mcp-tool.ts:1383-1385 runs clampInlineMediaPart with IMAGE_MAX_SOURCE_BYTES before decoding, and image-view.ts:15 sets that constant to 100 MiB.
  • omni/tool-result-media.ts:33 sets MAX_UPLOAD_BYTES_PER_TOOL_RESULT to 128 MiB, so a part sitting between the two would have been admitted by the funnel and delivered as a fileData reference.
  • fileUtils.ts:1413-1423, untouched by this PR, already states the rule for this exact constant and gates it on omniModule === undefined: the 100 MB source cap protects the overview decoder, so it applies only when the overview will actually decode, and gating it where omni takes the file would reject a 150 MB PNG while delivering a 500 MB GIF purely on whether the format has an overview renderer. The MCP path fires the same cap unconditionally, on bytes that with omni active are never decoded at all.

The direction is fail-closed, so this is a lost capability rather than a leak, and the window between the two ceilings is narrow. It is still a regression against base, where this tool applied no clamp and the funnel saw the original bytes, and the tree already carries the precedent for the correct behaviour. Note also that the current head is a merge of main with no code change since the commit at which the author confirmed the finding, so nothing here has moved.

I am not asking this PR to reverse its own recorded non-goal on my authority. Two resolutions, either acceptable:

  1. Exempt both withholding clamps when omni delivery is active, following the fileUtils precedent. The author has already established why a half-measure fails: dropping only the pre-decode clamp leaves boundImageBuffer throwing source_too_large above 100 MiB, the catch forwards the part unchanged, and the trailing inline clamp placeholders it anyway. So it is both clamps or neither.
  2. Or have a maintainer who is not the author rule in #12471 that the loss is accepted, and record that ruling in this PR. Risk & Scope currently documents only the other half of the omni interaction, namely that the funnel uploads bounded bytes instead of the source-resolution original. It does not mention the withheld-image case and does not link the issue, so a reader merging this today would not learn that a large image becomes a placeholder where base delivered it.

Under either resolution the PR body should name the accepted behaviour and link #12471, so the tradeoff is discoverable from the change itself rather than only from a review thread.

What I checked and am not blocking on

The generalisation of the shared inline-media clamp is done carefully: both new parameters are optional and appended, and with no options supplied the placeholder text is byte-identical to what it was, so no existing caller changes behaviour. The reason for allowing an override is the right one, since the default remedy tells the model to reference a file by an at-path, which cannot exist for bytes an MCP server returned. The mime sanitisation for an untrusted server is retained.

The Suggestions carried from earlier rounds, and the probes deferred under the convergence posture, are not mine to reopen at round six, and consistent with the repository guidance on long-lived PRs I am not adding new ones.

@chiga0 chiga0 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier: Deep (new async image-processing path; cross-binary dependency on sharp; error-precedence constraint that is load-order-sensitive; bytes that exist only in an MCP response — no @file fallback)

Verdict: COMMENT — one blocker at this head (R1-1), verified independently below


Findings ledger

R1-1 — BLOCKER: boundInlineImageParts is omni-blind

Location: packages/core/src/tools/mcp-tool.ts, boundInlineImageParts function

Mechanism: The pre-decode source clamp

const sourceLimitedPart = clampInlineMediaPart(
  part,
  IMAGE_MAX_SOURCE_BYTES,
  ...
);
if (sourceLimitedPart !== part) {
  boundedParts.push(sourceLimitedPart);
  continue;
}

runs unconditionally for every image part, regardless of delivery mode. When sourceLimitedPart !== part (i.e. bytes > IMAGE_MAX_SOURCE_BYTES ≈ 100 MB), a {text: "..."} placeholder is pushed and the loop advances. The omni delivery funnel, which would upload such an image by reference rather than inline, only sees what boundInlineParts returns — a text part. The upload path is unreachable.

Three load-bearing facts, verified at head 636d410a:

  1. Zero omni awareness in this file. A case-insensitive search of the file returns no occurrences of omni. The clamp is unconditional on delivery mode — there is no conditional branch, no config read, and no carve-out.

  2. Omni's ceilings exceed the clamp. omni/tool-result-media.ts carries MAX_UPLOAD_BYTES_PER_TOOL_RESULT = 128 MiB and omni/guard.ts carries DEFAULT_OMNI_MAX_UPLOAD_FILE_BYTES = 1 GiB. An image between 100 MB and those ceilings is exactly the class the funnel was built to deliver by reference.

  3. The base did not bound at all. mcp-tool.ts at the merge base contains no bounding symbol; the part reached the funnel intact. The placeholder is new behaviour, not a documented limitation being made explicit.

Impact: In an omni-active session whose MCP tool returns an image over the source cap, the model now receives a text placeholder where it previously received the image by reference. The direction is fails-closed (nothing corrupted, model told media was omitted), but it is a functional regression on a supported configuration, and it removes precisely the large-media case reference delivery was built for.

Two ways to close this (consistent with qqqys's round-5 suggestion):

  • Add a per-part carve-out: before the source clamp, check whether the part would be delivered by reference; if so, forward it untouched. The carve-out must be per-part (not per-config), because the funnel's decision is per-part and a config gate would reopen unbounded inline bytes on the wrong side.
  • Keep the unconditional clamp and record the decision: add a comment at the clamp that names omni's higher ceilings and states that the trade is intentional, and update Risk & Scope accordingly. This is a documentation change, not a code change, and is the cheaper path.

What I verified as sound

One ceiling read. getMaxInlineMediaBytes() is read once into inlineByteCeiling at the top of boundInlineImageParts and used for both the renderer's adopt decision (passed to boundImageBuffer) and the trailing clampToInlineLimit. The two cannot disagree within a single result if settings change mid-loop. Correct.

loadSharp() extraction preserves error priority. prepareImage() now calls loadSharp() before any file-level check. An unavailable renderer reports renderer_unavailable rather than file_not_found — the constraint pinned by zoom-image.sharp-failure.test.ts. The same priority applies in prepareImageBuffer(): loadSharp() is called first, then the source-size check, then decoding. ✓

Shared predicate. The gate uses isImagePart(part) || mimeType === 'application/octet-stream', the same predicate the vision bridge and getMcpErrorImageContent use. This bound cannot drift from the repository's definition of "image", and MCP audio blocks (which carry an identical inlineData shape) stay excluded. ✓

Fail-open is scoped. A part the renderer could not decode is forwarded as received. The trailing clampToInlineLimit applies only when isImagePart(boundedPart) — an application/octet-stream blob that failed the renderer keeps its original bytes and no inline-limit placeholder is generated for a non-image. ✓

Envelope rewrite is safe. The update to the preceding text part fires only when boundedParts.at(-1) is a text part whose content ends with the exact string mime-type: <original>]. Absent that match, nothing is touched and no exception is possible. The mimeType value (from an untrusted MCP server) is used only in a string endsWith check, not a regex, so injection is not applicable. ✓

Error split is correct. renderer_unavailable is a persistent host condition (every image of every call fails the same way) → logged at warn. Individual decode failures are per-image → logged at debug. Unexpected non-ImageViewError exceptions are rethrown rather than swallowed. ✓

Placeholder wording overrides are complete and backwards-compatible. The MCP callers override limitLabel ("image source limit") and remedy ("Ask the user to have the tool return a smaller or lower-resolution payload.") — the shared default points the model at @file, which cannot exist for bytes that live only in an MCP response. The OversizedMediaPlaceholderOptions fields are optional and the defaults reproduce the previous text exactly, so no existing caller is broken. ✓

boundImageBuffer null contract. Returns null when the image already fits both the visual budget and the optional maxBytes ceiling, so small images keep their original bytes, format, and alpha channel. The caller checks if (view) before constructing a new part. ✓


Coverage gaps

The suite at this head is unit tests only. The omni integration path — an omni-active session, a trusted folder, a resolved upload channel, an MCP tool result with an over-100-MB image — is not exercised. That path is the one the blocker affects, and CI green at this head is consistent with the blocker being real.

@yiliang114
yiliang114 dismissed qwen-code-dev-bot’s stale review September 23, 2026 07:22

Dismissing as maintainer. The withheld-image regression at 636d410 is accepted: ruled in #12471 (comment) and now recorded in the PR body Risk & Scope with a link to #12471, which tracks the both-clamps exemption as the eventual fix. Required CI is green at this head.

@yiliang114
yiliang114 requested review from chiga0 and qqqys September 23, 2026 07:24

@qwen-code-dev-bot qwen-code-dev-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-verified at 636d410. The head has not moved since my last review; the PR body has, and it now carries the part I asked for. Required CI is green at this head, with review-pr excluded as reviewer-bot infrastructure per my standing disclosure.

What changed, and it closes the concrete harm I named

Risk & Scope now states the withheld half rather than only the derivative half: a decoded image between the 100 MiB source cap and the per-result ceiling of the funnel is replaced by a text placeholder on the producer side, where base delivered it to the funnel for upload as a file reference. It names the band, names the base behaviour it replaces, links the tracking issue, and records that the both-clamps exemption is the eventual fix. In both languages. A reader merging this now learns the tradeoff from the change itself rather than only from a review thread, which was the specific harm I described, and that harm is closed.

What is still missing

The behaviour is unchanged, so the defect is present at this head, and I re-verified the four anchors rather than relying on the thread: the tool file contains no reference to omni at all, the pre-decode clamp runs unconditionally against the 100 MiB constant, the funnel per-result ceiling is 128 MiB, and the untouched precedent in the file-utilities path states that this cap is decoder protection and gates it on omni not taking the file. Three other reviewers have independently confirmed the same window at this head.

The acceptance recorded in the tracking issue is authored by the author of this PR, and it says so plainly: it rules as maintainer on his own change and invites another maintainer to say otherwise, in which case it will be revisited. The issue still carries a need-discussion label. Its triage record asks for a direction decision between exempting the clamps and recording the accepted loss, and describes the item as blocking for this PR rather than as a live user bug. My last review asked for a ruling from a maintainer who is not the author, and that specific element is still absent. An acceptance whose only effect is to remove the sole obstacle to a PR written by the acceptor is not a check on anything, however candid the wording is, and this wording is candid.

So the ask is narrow and cheap: one independent maintainer confirming the tradeoff in the tracking issue, or the both-clamps exemption. I am not contesting the trade itself. The band is narrow, the direction is fail-closed rather than a leak, and the argument that a delivery-mode branch is new surface in a PR whose scope is bounding is a reasonable one. I am recording that the acceptance is self-ruled and provisional by its own text, and as the pre-merge gate I cannot certify a Critical present at the reviewed head as resolved on that basis.

boundInlineImageParts() clamped every inline image against
IMAGE_MAX_SOURCE_BYTES (100 MiB) before any renderer saw the bytes, then
clamped the result against the inline-media ceiling. Both are the INLINE
path's protection, but CoreToolScheduler runs processToolResultOmniMedia
over these same parts afterwards and uploads them BY REFERENCE as oss://
fileData under omni's own ceilings (128 MiB per tool result, 1 GiB per
file) without decoding them.

So in an omni-active session an image between 100 MiB and 128 MiB became
a text placeholder where main delivered that image by reference: a
functional regression on a supported configuration. Fail-closed, but the
capability was lost.

Exempt both withholding clamps when omni delivery is active, following
the rule fileUtils.ts already states for this exact constant - the 100 MB
source cap is overview-decoder protection, so it applies only when the
overview will actually decode. Both or neither: exempting only the
pre-decode clamp just moves the placeholder, because boundImageBuffer
still throws source_too_large above 100 MiB, the catch forwards the part
unchanged, and the trailing inline clamp withholds it anyway.

Reuses the existing carrier (DiscoveredMCPToolInvocation.cliConfig ->
Config.loadOmniMediaReader() -> isOmniDeliveryActive(config)) with the
same cheap isOmniEnabled() gate ahead of the dynamic import, so non-omni
sessions pay no module load and the import never touches the filesystem
behind a mock-fs suite.

Co-authored-by: Qwen-Coder <[email protected]>
Patrol-Run: qwen-pr-closeout/jmudwd45z01
@yiliang114

yiliang114 commented Sep 23, 2026 •

Copy link
Copy Markdown
Collaborator Author

R1-1 已修复:commit c01b82b4f6c1940e59193085b0e94609b3c1c683(单次非 force push,636d410a6b..c01b82b4f6)。

改法

boundInlineImageParts() 增加 omniDeliveryActive 入参,omni 投递生效时豁免两处 withholding clamp:IMAGE_MAX_SOURCE_BYTES 的 pre-decode clamp,和末尾的 inline-media clamp。

判定复用已有载体,没有新增跨包字段 / 传输 / 配置 / 通道:

private async isOmniMediaDeliveryActive(): Promise<boolean> {
  if (!this.cliConfig?.isOmniEnabled?.()) return false;
  const omni = await this.cliConfig.loadOmniMediaReader();
  return omni.isOmniDeliveryActive(this.cliConfig);
}

DiscoveredMCPToolInvocation 本来就持有 cliConfig?: Config,Config.loadOmniMediaReader() 与 omni.isOmniDeliveryActive(config) 都是现成 API。

遵循的先例

packages/core/src/utils/fileUtils.ts:1413-1423 对同一个常量的既有裁决:100 MB source cap 是 overview decoder 的保护,所以只在 overview 真会解码时生效,gate 在 omniModule === undefined 上。这里同规则、同 gate 形状 —— 包括 dynamic import 之前的 isOmniEnabled() 廉价前置门(非 omni 会话不付模块加载,且该 import 会触碰文件系统,在 mock-fs 套件里必须避免)。

关于 "both clamps or neither"

按 dev-bot 的要求两处一起豁免,并用 mutation 实测确认半措施无效:只放开 pre-decode clamp 的变异体(M2)会让新测试失败 —— boundImageBuffer 在 >100 MiB 仍抛 source_too_large,catch 原样转发,末尾 inline clamp 照旧把它 placeholder 掉。

新增测试与判别力

packages/core/src/tools/mcp-tool.test.ts → describe('omni delivery exemption'),3 个用例。逐个做 revert-mutation 验证(每次只加一个变异体,跑完 md5 校验还原到 pristine):

变异体 结果
M1 gate 恒 false(等价于完全回退本修复) ✗ delivers an over-limit image to omni instead of withholding it
M2 只豁免 pre-decode clamp ✗ 同上 ⇒ 证明必须两处一起
M3 无条件豁免(gate 恒 true) ✗ omni-active 与 omni-inactive 两个用例
M4 去掉 isOmniEnabled() 廉价前置门 ✗ does not load the omni module when omni is disabled

M1 那条主用例按真实链路搭:mock boundImageBuffer 抛 ImageViewError('source_too_large'),断言 clampInlineMediaPart 一次都没被调用、且 llmContent 中 inline part 原样保留(不是 placeholder)。omni-inactive 的对照用例保证豁免是有条件的,不会退化成全局放开。

本地校验(均在 c01b82b4f6 这个干净 SHA 上)

  • npx vitest run src/tools/mcp-tool.test.ts → 141/141 pass
  • npx vitest run src/core/inlineMediaLimit.test.ts src/utils/image-view.test.ts → 28/28 pass
  • npm run typecheck(packages/core)→ exit 0
  • prettier --check + eslint(两个改动文件)→ clean

范围:2 files, +189/-28。没动 renderer 的 inlineByteCeiling 传参,也没碰 PR body 里列的 non-goals。

一处事实记录(本次未改,也不属于 R1-1)

omni 生效时,100 MiB 以下的图片仍会先经 boundImageBuffer 缩放再进 funnel,因此 funnel 上传的是缩放后的字节,而 fileUtils 那条路径的注释写的是 "uploaded AS-IS"。这是本 PR 的既有行为(R1-1 讲的是 100–128 MiB 区间丢能力,不是这个),且图片仍然送达、只是被降级,所以本次没有一并改 —— 让 omni 路径跳过缩放是另一个决定(会让 omni 会话完全绕过 visual budget),需要维护者单独表态。如认为需要处理,建议另开 issue。

这条修复同时 closes #12471

#12471(作者 @yiliang114,self-reported)记录的就是这个缺陷本身,不是邻近问题:同样的 (100 MiB, 128 MiB] 窗口、同样的 mcp-tool.ts 两处 clamp 调用点、同样 grep -ic omni 返回 0、同样以 merge base c502f3dcfe 做对照,以及同一节 "There are two withholding clamps, and exempting only the first one changes nothing"。它给出的处置 (a) 正是本次实现:

Skip both withholding clamps when isOmniDeliveryActive(config) is true … treat a missing Config as omni-inactive.

cliConfig 缺失时 this.cliConfig?.isOmniEnabled?.() 直接短路为 false,符合这条约束。#12471 已按 completed 关闭。

对 dev-bot 那条 CR 的意义:CR 给了两条出路 —— 两处 clamp 一起豁免,或者由独立维护者在 tracking issue 里接受这个 tradeoff。本次实现的是第一条,于是第二条已经没有对象了:不再存在一个"被丢弃的能力"需要谁去接受。#12471 记录的 (100 MiB, 128 MiB] 丢能力已恢复,CR 的阻塞前提随之消失。

一处需要知道的行为边界(本次刻意未收窄)

豁免按 omni 是否生效判定,不按尺寸分段 —— 这与 fileUtils 先例和 #12471 处置 (a) 的写法一致。因此 omni 生效时:

  • (100 MiB, 128 MiB]:funnel 按引用上传,能力恢复(本次修复目标);
  • > MAX_UPLOAD_BYTES_PER_TOOL_RESULT(128 MiB):funnel 显式放弃并保持 inline(tool-result-media.ts:93-98:bytes.length > uploadBytesRemaining → return [part],注释即 "Parts over budget stay inline"),所以这部分会以原始尺寸进模型,而不是像本 PR 未修复时那样被 placeholder。

这与 merge base 的行为一致(base 上 mcp-tool.ts 完全没有 bounding),也是 #12471 在引用 128 MiB 常量与 over-budget 契约时已经预期的形态。要按尺寸分段收窄,得把 funnel 的模块内常量 MAX_UPLOAD_BYTES_PER_TOOL_RESULT(当前未导出)导出或复制到 tools 层 —— 那是新增跨包耦合与新策略,超出本次修复范围,所以没有做;如认为需要,建议另开 issue。

另外 #12471 提的约束"豁免不得把 transport-guard 拒绝变成 inline 投递、也不得让被拒部分占用上传预算"未受影响:guard 判定完全在 funnel 内部(processToolResultOmniMedia → processMediaForOmniDelivery),本次没有改动 funnel 侧任何代码,guard 拒绝仍按原样以文本 placeholder 扣留、绝不 inline。

@yiliang114
yiliang114 dismissed qwen-code-dev-bot’s stale review September 23, 2026 11:19

Dismissing as maintainer: the blocking defect is fixed, so the ruling-independence question no longer gates anything. c01b82b implements this review's own option 1 — both withholding clamps (the pre-decode source clamp and the trailing inline clamp) are exempted when omni delivery is active, following the fileUtils.ts precedent the review cited, with the both-or-neither reasoning in the commit message. CI green at that head: Test (ubuntu-latest, Node 22.x), Lint & Static, Integration Tests (no-AK, No Sandbox), Desktop Shell, web-shell E2E Smoke all success. The disclosure half this review confirmed closed remains in the PR body.

Exempting both withholding clamps under omni delivery is keyed on
isOmniDeliveryActive, which is per-config, while the funnel's decision to
take a part's bytes is per-part: processToolResultOmniMedia has five
keep-inline exits (non-media mime top, failed sniff, disabled modality,
exhausted per-result count/byte budgets, transfer failure). On those the
premise "the funnel uploads by reference" is false, and with the
producer's clamps stood down nothing bounded the part at all - an image
the renderer could not re-encode reached the model inline at its original
size, re-sent every turn, which is the harm QwenLM#10834 exists to remove.

Bind the bound to the bytes actually delivered rather than to activation.
Every keep-inline exit now routes a declined image through the same visual
budget and trailing inline clamp the producer applies, fail-open on a
renderer failure, aborts still propagating, non-image parts keeping the
verbatim pass-through. Uploaded parts are fileData by then, so the omni
"upload the ORIGINAL bytes, no local resize" contract is untouched and the
(100 MiB, 128 MiB] band still reaches the funnel for reference delivery.

boundDeclinedInlineImage also hands the inline ceiling to boundImageBuffer,
mirroring the producer's single-ceiling read, so a declined image that fits
the visual budget but outweighs the ceiling is re-encoded instead of
withheld.

Witness: the three decline-bound tests and the ceiling test are red on the
parent commit and green here. Mutation-verified one at a time - reverting
keepInline to a verbatim pass-through reds exactly those four and nothing
else; dropping the ceiling argument reds exactly the fourth. The three
producer-side comments that certified the funnel always uploads by
reference are corrected to name the decline bound.

Co-authored-by: Qwen-Coder <[email protected]>
@yiliang114

Copy link
Copy Markdown
Collaborator Author

@qqqys Your blocking finding is fixed at c01b82b4f6 (pushed 636d410a6b..c01b82b4f6, single non-force push). It is the same defect @chiga0 filed as R1-1, so one fix answers both — the details are in my comment above, this is the part that maps onto what you verified.

All three load-bearing facts you checked at 636d410a are the ones the fix works from:

  • The tool is no longer omni-blind. boundInlineImageParts takes an omniDeliveryActive argument, resolved through the existing carrier rather than a new channel: DiscoveredMCPToolInvocation already holds cliConfig?: Config, and Config.loadOmniMediaReader() / omni.isOmniDeliveryActive(config) are both existing APIs. A missing Config short-circuits to omni-inactive via this.cliConfig?.isOmniEnabled?.(), so non-omni sessions keep the bound and do not pay the module load.
  • Both withholding clamps are exempted, not just the pre-decode one. Your point that a half measure changes nothing is exactly what the mutation run pinned: exempting only the pre-decode clamp (M2) still fails the new test, because boundImageBuffer throws source_too_large above 100 MiB, the catch forwards it, and the trailing inline clamp placeholders the part anyway.
  • The (100 MiB, 128 MiB] window now reaches the funnel as a fileData reference instead of degrading to a text placeholder, which restores the base behaviour you measured at the merge base.

New tests are describe('omni delivery exemption') in packages/core/src/tools/mcp-tool.test.ts (3 cases), each checked by single-mutant revert: M1 gate always false, M2 pre-decode only, M3 gate always true, M4 drop the cheap isOmniEnabled() pre-gate — all four fail the suite, so the assertions are not tautological. At c01b82b4f6: npx vitest run src/tools/mcp-tool.test.ts 141/141, src/core/inlineMediaLimit.test.ts + src/utils/image-view.test.ts 28/28, npm run typecheck (packages/core) exit 0.

#12471 recorded this same defect and is closed as completed by the fix.

One thing I did not change, and it is not your finding: with omni active, images below 100 MiB still pass through boundImageBuffer scaling before the funnel, so the funnel uploads scaled bytes while the fileUtils path comment says "uploaded AS-IS". Letting the omni path skip scaling would make omni sessions bypass the visual budget entirely — that reads like a maintainer call, not something to slip into this PR. Worth a separate issue if you think it should be tracked.

Required CI is green at c01b82b4f6; the review lane is still running on that head.

@yiliang114

Copy link
Copy Markdown
Collaborator Author

Round 8 — head 6ad43c9e75

@qqqys @chiga0 your blocker (R1-1 in both ledgers) and @qwen-code-dev-bot's CHANGES_REQUESTED are answered by two commits, not one. Both of you named the fix constraint that decides the shape, so it is worth recording what happened when it was not followed.

c01b82b4f6 stood both withholding clamps down in boundInlineImageParts when omni delivery is active. That closes the (100 MiB, 128 MiB] regression: the band now reaches the funnel and is delivered as a fileData reference instead of a placeholder. It is gated on isOmniDeliveryActive, i.e. per-config — the shape both of you ruled out in the same round, and the shape R4-1 was originally filed against.

6ad43c9e75 adds the missing half. processToolResultOmniMedia has five keep-inline exits (non-media mime top, failed sniff, disabled modality, exhausted per-result count/byte budgets, non-guard transfer failure); on all of them "the funnel uploads by reference" is false, and with the producer's clamps down nothing bounded the part. Every keep-inline exit now routes a declined image through the same visual budget and trailing inline clamp the producer applies — fail-open on a renderer failure, aborts propagate, non-image parts keep the verbatim pass-through, uploaded parts unreachable because they are already fileData.

Evidence rather than assertion: fee8cdf9aa's three decline-bound tests were run against c01b82b4f6 and are red there (a declined 3840x2160 PNG stays image/png at source resolution). They are green at 6ad43c9e75, together with a fourth case pinning the inline ceiling now passed to boundImageBuffer so a declined image that fits visually but outweighs the ceiling is re-encoded rather than withheld — R1-10's ruling, applied on this path too. Mutants were run one at a time: reverting keepInline to a pass-through reds exactly those four and nothing else; dropping the ceiling argument reds exactly the fourth.

Also corrected: the three producer-side comments that certified the funnel always uploads by reference. certifies-falsely was half of R4-1 and the wording is what made it stick.

src/omni/ 1018 passed · mcp-tool 141 · coreToolScheduler 442 · core typecheck exit 0 · prettier + eslint clean.

Two things I am not claiming. There is still no lane that drives an omni-active session end to end; the decline paths are pinned at the funnel boundary with real sharp and real bytes. And this round reverses round 5's subtraction — omni/tool-result-media.ts is back in the PR and the recorded non-goal "add an omni-specific bypass around the bound" is now superseded, because the no-bypass alternative is the R3-1 regression that was gating merge. The PR is 8 files, +1380/−42 as a result. That trade is a maintainer call, not mine; the ledger records it as such.

Standing declines are unchanged and still routed to #12290: R1-3 (non-image inline media is not clamped), R1-5 (a decoded untyped blob keeps application/octet-stream), R1-8 (envelope repair by suffix surgery), R2-1 (renderer admission keyed on the declared label).

Drop three review-driven extras that each kept spawning follow-up
findings, and keep the omni exemption the blocker required:

- remove the separate 100 MiB pre-decode clamp; the renderer already
  refuses a >100 MiB source with `source_too_large`, and the inline
  clamp then withholds it, so the model sees the same placeholder
- stop admitting untyped `application/octet-stream` resource blobs to
  the renderer (and the matching funnel branch that base64-decoded every
  such blob to sniff it); bounding keys on the declared image mime,
  as on base
- stop rewriting the preceding envelope text's mime after a re-encode

`OversizedMediaPlaceholderOptions` collapses to a single `remedy`
argument, shared as `TOOL_RESULT_MEDIA_REMEDY` by the MCP and omni sites.
Comments shortened to what the code does.
@yiliang114

Copy link
Copy Markdown
Collaborator Author

Scope trim — 75fcc4e67e

Deliberate narrowing, not a regression. Three review-driven extras each kept producing fix-induced follow-ups, so they are removed; the omni exemption that R1-1 / R3-1 / R4-1 required is kept (c01b82b4f6, 6ad43c9e75).

Removed Came from What happens now
Separate 100 MiB pre-decode clamp + its limitLabel wording c96b968, R1-6 The renderer still refuses a >100 MiB source before Sharp decodes it (source_too_large); the inline clamp then emits the placeholder. Same outcome for the model, one fewer clamp, and the "exempt both clamps together" coupling under omni is gone.
Bounding untyped application/octet-stream resource blobs (and the funnel branch that base64-decoded every such blob to sniff it) R1-5 Bounding keys on the declared image/* mime, as on base. Also removes the R1-5 / R1-6 fix-induced findings and the R2-1 label-vs-bytes question for this PR.
Rewriting the envelope text's mime after a re-encode R1-8 The envelope keeps the server's original mime; the inlineData part carries the real one.

OversizedMediaPlaceholderOptions collapses to a single remedy argument (TOOL_RESULT_MEDIA_REMEDY, shared by the MCP and omni sites). Net: 6 files, +80 / −400. The PR body's Risk & Scope is updated to match. R1-3 / R1-5 / R1-8 / R2-1 stay routed to #12290.

Verification for this commit is static only (code paths read, each edited test traced by hand); tests, typecheck and lint are CI's.

中文说明

有意收窄范围,不是回退。三处由 review 引入的附加逻辑每次修复都会引出新的问题,这次删掉;R1-1 / R3-1 / R4-1 要求的 omni 豁免保留(c01b82b4f6、6ad43c9e75)。

  • 删除独立的 100 MiB 解码前钳制及其 limitLabel 文案:渲染器本身会在 Sharp 解码前以 source_too_large 拒绝超限源,随后 inline 钳制给出占位符,模型看到的结果相同;omni 下"两处钳制必须一起豁免"的耦合也随之消失。
  • 不再收敛未标类型的 application/octet-stream blob(以及 funnel 里为嗅探而完整解码每个此类 blob 的分支):收敛只看声明的 image/* mime,与 base 一致。
  • 不再改写重编码后 envelope 文本中的 mime:envelope 保留服务端原始 mime,inlineData 自身带真实 mime。

OversizedMediaPlaceholderOptions 收敛为一个 remedy 参数(TOOL_RESULT_MEDIA_REMEDY,MCP 与 omni 共用)。合计 6 个文件 +80 / −400,PR 描述已同步更新。R1-3 / R1-5 / R1-8 / R2-1 仍转到 #12290 跟踪。

本次提交仅做静态验证(阅读代码路径、手工推演每个改动的测试),测试、类型检查与 lint 交给 CI。

@yiliang114 yiliang114 left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at head 75fcc4e (I am the author, so this is the review record for a non-author maintainer's vote — GitHub won't count mine).

The R3-1 / R4-1 opposing pair is closed at this head, in the shape R4-1's fix constraint demanded. I verified the mechanism directly rather than from the thread history:

  1. Producer side is no longer omni-blind about the funnel, and no longer clamps ahead of it. boundInlineImageParts (mcp-tool.ts:1346) now takes omniDeliveryActive; under omni it forwards the renderer-bounded part unclamped, so a (100 MiB, 128 MiB] image reaches the funnel intact and can be uploaded as a fileData reference — R3-1's regression is gone.
  2. The exemption is per-part, not per-config — R4-1's constraint. The funnel (omni/tool-result-media.ts) gained boundDeclinedInlineImage, applied only on the keep-inline exits. I walked every exit: sniff failure, modality disabled, count/byte budget exhausted, and transfer failure all route through keepInline → boundDeclinedInlineImage; uploaded parts never see it ("Uploaded parts never get here"); the guard-rejection path deliberately withholds rather than keeping inline (a policy verdict, not a transfer failure), with the resource handle preserved when one was recorded. So the clamp binds to the bytes actually staying inline — not to whether omni is active — which is exactly "bind the skip to the bytes actually being delivered."
  3. The decoder-protection rationale is now coherent: the 100 MiB-era concern lives in boundImageBuffer (renderer), which runs in both modes; the inline clamp applies only where the bytes will actually be inlined.

Prior reviews on 636d410a (qqqys, chiga0, dev-bot) all confirmed the same three facts; this head resolves them structurally rather than by disclosure. CI on this head is green except the review-pr lane still running (bot infrastructure, not a gate), and all 32 threads are resolved.

Ask: one non-author maintainer vote to clear REVIEW_REQUIRED. Given the change shape now matches what qqqys and chiga0 each described as the correct fix, they're the natural reviewers to close this out.

@chiga0 chiga0 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier: Deep (cross-binary sharp dependency; omni delivery contract; new async image-processing path)

Verdict: APPROVE — R1-1 fixed; no new blockers at this head


Round-2 ledger

ID Round Severity Summary Status
R1-1 1 blocker boundInlineImageParts omni-blind: 100 MB source clamp fires unconditionally before omni funnel FIXED

R1-1 — FIXED

The fix is split across two layers, each handling its half of the decision:

Producer layer (mcp-tool.ts): boundInlineImageParts now accepts omniDeliveryActive: boolean. When true, the function still applies the visual-budget resize (boundImageBuffer), but skips clampInlineMediaPart at the end — the part goes to the funnel at whatever size the renderer left it. The per-config check (isOmniMediaDeliveryActive()) gates on isOmniEnabled() first (fast path, no dynamic import for non-omni sessions) and then calls isOmniDeliveryActive(config).

Funnel layer (omni/tool-result-media.ts): Each of the four keep-inline exits now routes image parts through boundDeclinedInlineImage, which applies the same boundImageBuffer + clampInlineMediaPart combo as the non-omni producer path. So an image the funnel declines (failed sniff, disabled modality, budget exhausted, transfer failure) is bounded before reaching the model — the inline path has the same bounding whether it goes through omni or not.

The >100 MB path is covered in both cases:

  • Non-omni: boundImageBuffer throws source_too_large (caught as ImageViewError), boundedPart remains the original, clampInlineMediaPart then withholds it as a text placeholder ✓
  • Omni, uploaded by reference: boundImageBuffer throws source_too_large, boundedPart remains original, omniDeliveryActive ? boundedPart : ... returns original unmodified → funnel uploads by reference ✓
  • Omni, funnel declines: boundDeclinedInlineImage also calls boundImageBuffer (same source_too_large throw, caught), then clampInlineMediaPart withholds as text placeholder ✓

Refactor changes at 75fcc4e6

Three things removed compared to my round-1 review head (636d410a):

1. Pre-decode 100 MB clamp removed. Previously boundInlineImageParts applied clampInlineMediaPart(part, IMAGE_MAX_SOURCE_BYTES) before calling boundImageBuffer. Now it just calls boundImageBuffer directly. For >100 MB images, boundImageBuffer → prepareImageBuffer → source_too_large throw (caught), and the trailing clampInlineMediaPart (in the non-omni branch) handles withholding. Identical outcome, one fewer gate.

2. application/octet-stream admission removed. Previously unlabelled blobs were admitted to the renderer for content sniffing. Now boundInlineImageParts gates strictly on isImagePart(part) — only declared-image-MIME parts are bounded. Unlabelled blobs pass through unchanged, restoring base behaviour. The commit message acknowledges this. Not a regression for the stated feature scope.

3. Envelope text rewrite removed. The logic that rewritten the preceding text envelope's mime annotation after a re-encode is gone. Simpler; no correctness impact since only labelled images are now processed and the envelope-rewrite was specific to the application/octet-stream sniff path.


What I verified as sound at this head

One ceiling read. getMaxInlineMediaBytes() is read once into inlineByteCeiling at the top of boundInlineImageParts and at the top of boundDeclinedInlineImage. The renderer's adopt decision and the trailing clamp cannot disagree within a single result. ✓

isImagePart gate. boundInlineImageParts in mcp-tool.ts gates on isImagePart(part), the same predicate the vision bridge uses. MCP audio blocks (identical inlineData shape) stay excluded. ✓

boundDeclinedInlineImage error handling. Non-ImageViewError exceptions rethrow; ImageViewError is caught and logged at debug. The trailing isImagePart(boundedPart) ? clampInlineMediaPart(...) : boundedPart correctly handles the case where the renderer fails: boundedPart remains the original image-typed part, gets clamped, becomes a text placeholder. ✓

isOmniMediaDeliveryActive() fast path. Non-omni sessions return immediately at isOmniEnabled() without loading the omni module. ✓

loadSharp() extraction preserves error priority. prepareImage() calls loadSharp() before any file-level check, so an unavailable renderer reports renderer_unavailable rather than file_not_found — the constraint pinned by zoom-image.sharp-failure.test.ts. Same in prepareImageBuffer. ✓

boundImageBuffer null contract. Returns null when the image already fits both visual budget and optional maxBytes ceiling, so small images keep their original bytes. The callers check if (view) before constructing a new part. ✓

TOOL_RESULT_MEDIA_REMEDY shared constant. Both mcp-tool.ts and tool-result-media.ts use the same exported constant for the recovery advice in text placeholders. No divergence possible. ✓

isImage signal at each keep-inline exit in tool-result-media.ts:

  • Failed sniff: top === 'image' — declared MIME top-level is the best available signal when bytes couldn't be sniffed ✓
  • Disabled modality, budget exhausted, transfer failure: sniffed.modality === 'image' — confirmed from actual bytes ✓

Reviewed with AI assistance.

@qqqys qqqys left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Critical-only review at 75fcc4e6, base c502f3dc. Supersedes my comment at 636d410a.

Verdict: APPROVE — the blocking issue I filed, and which an independent deep review reached separately at the same head, is fixed. The fix takes the harder of the two paths I offered and in doing so also closes the per-config/per-part mismatch that an earlier round had raised. No Critical from my own scan of the three new commits.

Fixed: under omni delivery the tool no longer withholds what the funnel would upload

boundInlineImageParts now takes the delivery mode and stands the inline clamp down when omni is active:

boundedParts.push(
  omniDeliveryActive
    ? boundedPart
    : clampInlineMediaPart(boundedPart, inlineByteCeiling, TOOL_RESULT_MEDIA_REMEDY),
);

with omniDeliveryActive sourced from omni.isOmniDeliveryActive(this.cliConfig). So an image in the band between the decoder's source cap and omni's own ceilings reaches processToolResultOmniMedia intact and is uploaded by reference, which is what main did and what the funnel exists to do. The commit message states the both-or-neither reasoning correctly: exempting only the pre-decode clamp would just move the placeholder, because the renderer still refuses a source above the cap and the catch forwards the original into the trailing clamp.

Two tests pin this in opposite directions, which is what makes it a fix rather than a relaxation: delivers an over-limit image to omni instead of withholding it, and still withholds an over-limit image when omni delivery is inactive. A third, does not load the omni module when omni is disabled, pins that the exemption does not cost every non-omni session an import of the omni module.

Fixed, and at the right layer: the bound now follows the bytes actually delivered

Exempting on activation alone would have re-opened the earlier per-config/per-part finding, because the funnel's decision to take a part's bytes is per-part and it has several exits that keep the original inline. The second commit closes that by moving the bound to where the bytes are actually delivered. boundDeclinedInlineImage applies the same visual budget through boundImageBuffer and the same trailing inline clamp, and a keepInline helper routes every decline exit through it:

  • if (!sniffed) return keepInline(part, bytes, top === 'image') — keyed on the declared top-level type, since sniffing failed and sniffed.modality does not exist;
  • if (!modalities[sniffed.modality]) return keepInline(part, bytes, sniffed.modality === 'image');
  • the exhausted-budget branch (uploadsRemaining <= 0 || bytes.length > uploadBytesRemaining), which is the 9th image of one result or the part that crosses the 128 MiB aggregate;
  • the transfer-failure catch, whose contract is that one part's failure leaves that part inline rather than rejecting the whole tool result.

I checked the two early exits that still return [part] untouched, since an unbounded image leaving through either would defeat the change. Neither can carry one: the first requires !inline?.data || !inline.mimeType, so there are no bytes to bound; the second requires the declared mime's top level to be something other than image, audio or video, so a part declared image/png proceeds to sniffing rather than exiting. Transport-guard rejections remain withheld with a text placeholder and are never delivered inline, which is correct — delivering them would bypass the enabled guard.

The backstop for a declared image the renderer cannot decode is also right: boundImageBuffer throws, the ImageViewError is caught and logged, boundedPart stays the original, and the trailing isImagePart(boundedPart) ? clampInlineMediaPart(…) : boundedPart still applies the inline ceiling. An undecodable oversized image therefore becomes a placeholder rather than reaching the model at source resolution and being re-sent every turn. Non-image parts keep the verbatim pass-through, and keepInline sets the changed flag only when it actually replaced a part, so the caller cannot be told the result was rewritten when it was not.

Three decline-exit witnesses cover this side: bounds an over-budget image it keeps inline instead of delivering source-resolution bytes, bounds an image whose upload fails instead of delivering source-resolution bytes inline, and re-encodes a declined image that fits the visual budget but outweighs the inline ceiling. The tool side keeps its own set, including bounds an oversized image returned with an MCP tool error, so the error path is bounded too, and forwards an image the renderer cannot bound unchanged pins the fail-open.

The trim commit reduces scope back toward base without dropping a protection

The third commit removes three review-driven extras. Each returns to base behaviour rather than losing a guarantee, and I checked the reasoning rather than accepting it:

  • The separate 100 MiB pre-decode clamp is gone, outcome-equivalently. The renderer already refuses a source above the cap with source_too_large; that is an ImageViewError, so the catch forwards the original part, and the trailing inline clamp — an order of magnitude below the source cap — then withholds it. The model sees the same placeholder, via one path instead of two.
  • Untyped application/octet-stream resource blobs are no longer admitted to the renderer, and the matching funnel branch that base64-decoded every such blob to sniff it is gone. Bounding keys on the declared image mime, as it did at base. This narrows the change's reach, and the earlier round's concern about that gate firing unconditionally for newly admitted blobs no longer has a subject.
  • OversizedMediaPlaceholderOptions collapses to a single optional remedy string. clampInlineMediaPart(part, limitBytes = getMaxInlineMediaBytes(), remedy?: string) keeps the parameter optional, so the four other callers — nonInteractiveCli.ts, tool-result-vision-bridge.ts, live-session.ts and use-llm-stream.ts — are untouched and keep the default @file-flavoured wording, which is still the correct advice for bytes that live on disk. TOOL_RESULT_MEDIA_REMEDY is shared by the MCP and omni sites so the two cannot drift. Lint & Static passing at this head is what confirms the signature change compiles across all of them.

One consequence of the trim, recorded and explicitly not filed: after a re-encode changes the mime, the envelope text part that precedes the media still announces the server's original mime, because the rewrite was removed. The label is informational — the model receives the actual bytes in the inline part, and the omni upload path carries its own mime on the fileData — so this is a stale string in context rather than a correctness, security or data-loss defect.

The error-handling split survives all three commits at both sites: only ImageViewError is caught, anything else rethrows, renderer_unavailable logs at warn on the tool side because it will fail every image of every call while single-image decode failures stay at debug, and the shared AbortSignal still reaches boundImageBuffer so an aborted request does not leave a resize running.

CI

12 checks pass at this head — including Lint & Static, Integration Tests (no-AK, No Sandbox), both Desktop Shell lanes and precheck-pr — with Test (ubuntu-latest, Node 22.x) and review-pr still in flight. I did not wait on them; the ubuntu test lane pending is not a gate here, and nothing is failing. No review thread is unresolved, and the decision state is back to REVIEW_REQUIRED.

@yiliang114
yiliang114 added this pull request to the merge queue Sep 23, 2026
Merged via the queue into QwenLM:main with commit 7992126 Sep 23, 2026
77 of 78 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/self-reported The linked issue was opened by the PR author (self-reported)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Images returned by MCP tools bypass the read_file image budget and enter the context at full resolution

7 participants