feat: support drag and drop img in web-shell by water-in-stone · Pull Request #8696 · QwenLM/qwen-code

water-in-stone · 2026-08-07T15:24:45Z

What this PR does

This PR adds image drag-and-drop support to every Web Shell composer while reusing the existing paste, attachment preview, and multimodal prompt pipeline.

It supports PNG, JPEG, GIF, WebP, and BMP images, preserves attachment order across files and batches, and allows image-only prompts in the main chat, split panes, and side tasks. It also provides drop-state feedback, attachment removal, queued submission, edit, and retry behavior.

The prompt lifecycle has been hardened to:

Limit image reading to four concurrent readers and an estimated 8 MiB base64 ingestion budget.
Prevent submission while images are still being read.
Prevent stale readers from updating a cleared or replaced composer.
Correlate admission and queue lifecycle events by prompt ID.
Avoid duplicate or missing user messages when start, terminal, and HTTP response events arrive out of order.
Preserve text, images, and input annotations across safe edit and retry flows.
Avoid automatic recovery or retry when admission may have succeeded but the response was lost.
Distinguish prompts that were removed before dispatch from prompts that started and were later cancelled.

The daemon wire format, ACP/Core protocol, and public Web Shell API remain unchanged.

Why it's needed

Web Shell already supported pasting images, but dropping image files onto the composer was not handled. Browser default drop behavior could also insert unwanted content or navigate away from the page.

The existing asynchronous ingestion and queue lifecycle had additional edge cases: attachments could be reordered, an image could be omitted if the prompt was submitted before reading completed, and response/event races could produce duplicate transcript entries or lose recoverable payload data.

This PR closes those gaps and makes image prompts reliable across direct, queued, image-only, retry, and editing flows.

Reviewer Test Plan

How to verify

Open Web Shell with an image-capable model.
Drag PNG and BMP files onto the composer and confirm that:
- The page does not navigate away.
- Drop highlighting is cleared after the drop.
- Thumbnails appear in selection order.
Send the images without entering text and confirm the image-only prompt appears once in the transcript.
Remove one attachment before sending and confirm only the remaining image is submitted.
Paste PNG, JPEG, GIF, WebP, and BMP images and confirm paste behavior matches drag-and-drop behavior.
Drop unsupported or over-budget files and confirm they are skipped with aggregated feedback while valid files are retained.
Submit an image-only prompt while another turn is active and confirm it is queued, can be edited or removed, and does not disappear because its text is empty.
Confirm definite admission failures remain retryable, while ambiguous transport failures are marked as uncertain and are not automatically resent.
Run the focused automated checks:
- Web Shell: 7 focused test files, 530 tests passed.
- WebUI: 2 focused test files, 237 tests passed.
- Web Shell and WebUI lint passed.
- Root build and typecheck passed.
- Chromium Web Shell smoke tests passed for the drag-and-drop implementation.
- Prettier and git diff --check passed.

Evidence (Before & After)

Before	After
Dropping image files was not handled and could trigger browser default behavior.	Dropped files become ordered composer attachments without navigating away.
Image-only prompts could be treated as empty submissions.	Image-only prompts work in direct, split, side-task, and queued flows.
Image readers could complete out of order or update a stale composer.	Ingestion is ordered, bounded, and isolated by composer ownership.
Admission response races could duplicate messages or lose recoverable payloads.	Prompt-ID lifecycle correlation preserves complete payloads and appends each executed prompt once.

Automated Chromium coverage verifies PNG/BMP drop, preview ordering, image-only submission, deletion, request payload contents, transcript rendering, admission failure retry, and post-admission turn-error retry.

20260807-170544_._.s.video.meeting_0807_video.mp4

Tested on

OS	Status
🍏 macOS	✅ tested
🪟 Windows	⚠️ not tested
🐧 Linux	⚠️ not tested

Risk & Scope

Main risk or tradeoff: The main risk is asynchronous prompt admission and queue-event ordering. Focused tests cover response loss, early lifecycle events, cancellation, confirmed removal, owner changes, and duplicate prevention. The client-side 8 MiB budget protects browser resources but is intentionally separate from daemon or proxy request limits.
Not validated / out of scope: Manual Windows, Linux, and Firefox validation; restoring original image bytes from text-only daemon summaries after a page reload; SVG, TIFF, HEIC, PDF, directories, remote URLs, and cross-provider BMP image equivalence. Anthropic retains its existing unsupported-media text fallback for BMP.
Breaking changes / migration notes: None. The daemon wire format, ACP/Core protocol, and public Web Shell API are unchanged. Only internal WebUI action contracts were extended.

Linked Issues

Closes #8321

中文说明

What this PR does

本 PR 为所有 Web Shell composer 增加图片拖放能力，并复用现有的图片粘贴、附件预览和多模态 prompt 链路。

支持 PNG、JPEG、GIF、WebP 和 BMP，能够保持多文件及多批次附件顺序，并让主聊天、split pane 和 side task 都可以发送 image-only prompt。同时覆盖拖放状态反馈、附件删除、排队发送、编辑和重试。

本次还强化了 prompt 生命周期：

图片读取最多使用四个并发 reader，并设置估算为 8 MiB base64 数据的客户端摄取预算。
图片读取完成前禁止提交，避免发送不完整的 prompt。
composer 被清空或切换后，旧 reader 不再污染新状态。
admission 与队列生命周期事件通过 prompt ID 精确关联。
started、terminal 和 HTTP response 乱序时，不会重复或遗漏 user message。
安全编辑和重试时保留文字、图片及 input annotations。
admission 可能成功但 response 丢失时，不自动恢复或重试，避免重复发送。
区分“尚未 dispatch 就被删除”和“已经开始执行后被取消”的 prompt。

daemon wire format、ACP/Core 协议及公开 Web Shell API 均保持不变。

Why it's needed

Web Shell 已经支持粘贴图片，但将图片文件拖入 composer 时不会被接管，浏览器默认行为还可能插入无效内容或离开当前页面。

原有异步摄取和队列生命周期也存在一些边界问题：附件可能因为读取完成顺序不同而乱序；读取完成前提交会遗漏图片；response 与生命周期事件发生竞态时，可能产生重复 transcript message 或丢失可恢复的完整 payload。

本 PR 补齐这些缺口，使 direct、queued、image-only、retry 和 edit 等图片 prompt 流程更加可靠。

Reviewer Test Plan

How to verify

使用支持图片的模型打开 Web Shell。
将 PNG 和 BMP 文件拖入 composer，确认：
- 页面不会跳转。
- drop 完成后高亮状态被清理。
- 缩略图按照选择顺序显示。
不输入文字直接发送，确认 image-only prompt 在 transcript 中只出现一次。
发送前删除一张附件，确认只提交剩余图片。
分别粘贴 PNG、JPEG、GIF、WebP 和 BMP，确认 paste 与 drag-and-drop 行为一致。
拖入不支持或超过客户端预算的文件，确认这些文件被跳过并显示聚合提示，合法文件仍然保留。
在已有 turn 执行期间提交 image-only prompt，确认它会正常排队，可以编辑或删除，不会因为文字为空而丢失。
确认明确的 admission 拒绝仍可安全重试；结果不明确的 transport failure 会标记为 uncertain，且不会自动重新发送。
运行聚焦验证：
- Web Shell：7 个聚焦测试文件，共 530 tests，通过。
- WebUI：2 个聚焦测试文件，共 237 tests，通过。
- Web Shell 和 WebUI lint 通过。
- 全仓 build 和 typecheck 通过。
- 图片拖放实现的 Chromium Web Shell smoke tests 通过。
- Prettier 和 git diff --check 通过。

Evidence (Before & After)

Before	After
图片文件 drop 未被接管，可能触发浏览器默认行为。	drop 后图片按顺序成为 composer 附件，页面不会离开。
image-only prompt 可能被当作空提交。	main、split、side-task 和 queued 流程均支持 image-only prompt。
reader 可能乱序完成或写入已经失效的 composer。	摄取顺序稳定、资源有界，并按 composer owner 隔离。
admission response 竞态可能重复 message 或丢失完整 payload。	通过 prompt ID 关联生命周期，保留完整 payload，每个已执行 prompt 只追加一次。

Chromium 自动化覆盖 PNG/BMP drop、预览顺序、image-only 提交、附件删除、请求内容、transcript 渲染、admission failure retry 和 post-admission turn-error retry。

20260807-170544_._.s.video.meeting_0807_video.mp4

Tested on

OS	Status
🍏 macOS	✅ tested
🪟 Windows	⚠️ not tested
🐧 Linux	⚠️ not tested

Environment (optional)

macOS 26.0、Node.js v22.22.3、本地 Web Shell mock daemon，以及 Chromium Playwright smoke tests。

Risk & Scope

Main risk or tradeoff: 主要风险是异步 prompt admission 与队列事件的乱序。聚焦测试覆盖 response 丢失、生命周期事件早到、取消、确认删除、owner 切换及重复消息防护。客户端 8 MiB 预算用于保护浏览器资源，不等同于 daemon 或代理的请求上限。
Not validated / out of scope: 未进行 Windows、Linux 和 Firefox 人工验证；不支持页面重载后从 text-only daemon summary 恢复原图片；SVG、TIFF、HEIC、PDF、目录和远程 URL 不在范围内；不承诺 BMP 在所有 provider 上都保持图片语义，Anthropic 继续使用既有的 unsupported-media 文本降级。
Breaking changes / migration notes: 无 breaking change，无需迁移。daemon wire format、ACP/Core 协议和公开 Web Shell API 不变，仅扩展了内部 WebUI action contract。

Linked Issues

Closes #8321

Allow Web Shell composers to ingest image files reliably while preserving the existing multimodal prompt protocol. - Share ordered image ingestion across desktop and mobile editors - Support image-only prompts and BMP preview and provider-safe handling - Preserve queued payloads across retries and uncertain outcomes - Add lifecycle guards, user feedback, unit coverage, and browser tests

Preserve complete prompt payloads and prevent duplicate or uncertain delivery states when admission responses race with queue lifecycle events. - Correlate admission, queue, and terminal events by prompt ID - Restore images and input annotations across retry and edit flows - Bound image reader concurrency and encoded attachment memory - Reconcile confirmed removals and explain ambiguous queue entries

Document the reviewed admission, recovery, and resource invariants. Keep the design aligned with the hardened Web Shell implementation. - Record bounded image ingestion and encoded-data budgeting - Clarify prompt lifecycle correlation and confirmed removal behavior - Describe annotation restoration and internal action boundaries - Update focused validation evidence and acceptance criteria

water-in-stone · 2026-08-08T04:10:35Z

@wenshao @yiliang114 Please take a look

ytahdn · 2026-08-08T09:01:12Z

Review Summary — PR #8696

Reviewed: 37 files, +4482/−414 at head of feat/support-drag-drop-img

What this PR does

Adds image drag-and-drop to every Web Shell composer, reusing the existing paste/attachment/multimodal pipeline, and hardens the async prompt lifecycle (ordered bounded ingestion, prompt-ID correlation, dedup, safe retry/edit).

Strengths

imageIngestion.ts is well-engineered. Bounded concurrency (4 readers) with a safe shared-index pool; 8 MiB base64 budget using the correct ceil(size/3)*4 estimate; order preserved via an indexed settled array; MIME normalization with BMP aliases (image/x-bmp, image/x-ms-bmp); distinct rejection reasons (unsupported/unavailable/too-large/read-failed).
promptAdmission.ts correctly separates definite rejections (413/501) from ambiguous transport failures, so only safe failures stay retryable and uncertain ones are not auto-resent.
Lifecycle correlation is thorough. useQueuedPrompts adds prompt-ID tracking, an owner token for session isolation, a bounded completed-prompt-id LRU (prevents duplicate/missing user messages), and explicit restore/discard for unknown prompts.
Drag-drop uses capture-phase handlers with preventDefault() (onDragEnterCapture/onDragOverCapture/onDropCapture), preventing browser navigation and unwanted content insertion; dropEffect='copy' on dragover.
Strong test coverage. 530 web-shell + 237 webui focused tests plus Chromium e2e smoke covering drop, ordering, image-only submit, removal, retry, and payload contents.

Observations

[Info] Size/complexity. The PR is large and touches many lifecycle paths; the complexity is justified by the races described, but it's a lot to land at once.

[Info] ownerTokenRef reset-during-render is an unusual pattern but acceptable here for session scoping.

No blocking issues. CI green (Test, web-shell E2E, Capture visuals). Only ci-bot has approved so far.

Approve.

ytahdn

LGTM. Well-engineered drag-drop ingestion (bounded, ordered, budgeted) and thorough prompt-lifecycle hardening (prompt-ID correlation, dedup, safe retry). Capture-phase preventDefault prevents navigation. Strong test coverage, CI green.

water-in-stone · 2026-08-08T13:06:48Z

@yiliang114 Please take a look

yiliang114

LGTM. The hard parts check out: reader concurrency is genuinely bounded (worker-pool index handoff, batches serialized on a tail promise, cumulative 8 MiB budget inclusive at the boundary), stale readers are killed by lane identity + generation + abort with a post-await recheck, the race matrix is pinned by tests (terminal-before-response binding, no double append, removed-before-response no-append, exactly-once restore after definite rejection, unknown-payload restore/discard without resend), lost-response prompts are never auto-resent, and there is no new XSS surface (count-only notices, data-URL thumbnails, SVG stays blocked). The removed-before-dispatch vs started-then-cancelled distinction and the admission classifier are correct, and the core-package change is test-only.

One P2 worth fixing or testing away before merge: restoreQueuedPromptsToEditor drops the #7134 guard — images are now restored unconditionally after mergeRestoredPromptText, keyed by local row id. Same-row double restore is blocked, but if the payload text is already in the editor when a different row carrying the same payload is restored (re-typed/re-queued identical content, or a row re-materializing with a fresh id), the text dedupes while the images append again — exactly the case the old condition guarded. Suggested fix: skip image/annotation restore when nextText === currentText, or dedup on serverPromptId/payload hash; plus a restore-while-text-already-present regression test.

Minor notes: no content sniffing (extension/declared MIME trusted, parity with paste — the provider is the real validation boundary; worth a line in the design doc's trust-boundary section), drop protection is composer-scoped so drops on the message list still navigate (pre-existing), and the branch now conflicts with main and needs a rebase. Nothing blocks from my side.

…rop-img # Conflicts: # packages/web-shell/client/App.tsx

water-in-stone · 2026-08-09T14:37:58Z

@ytahdn @yiliang114 All the conflicts have been resolved. Please take another look.

water-in-stone · 2026-08-09T14:40:08Z

@qwen-code-ci-bot @qwen-code-dev-bot Please take a look

ytahdn · 2026-08-10T02:49:29Z

One P2 remains on the current head (e6c85df): restoreQueuedPromptsToEditor deduplicates restored text with mergeRestoredPromptText, but then restores images unconditionally. If the same payload text is already at the top of the editor—such as an identical prompt being retyped/requeued, or the payload reappearing under a fresh local row id—the text remains single while restoreImages() appends the attachments again. The new restoredPromptIdsRef only protects the same local row id, so it does not preserve the previous #7134 guard across equivalent rows.

I reproduced this against the current head with a focused regression test: prefill the editor with describe, enqueue describe with one image, then reject admission with HTTP 413. setText is correctly not called, but restoreImages is called once; all 35 existing tests in that file pass and the added regression fails.

Suggested minimal fix: when non-empty restored text is already present (nextText === currentText), skip restoring that payload’s images/annotations, while retaining the image-only restoration path. Please add the corresponding regression test before merge.

Skip payload attachments when restoring text is a no-op because the same prompt text already exists in the composer. - Restore images and annotations only when their text is inserted - Preserve image-only restoration regardless of the current draft - Add regression coverage for duplicate text with attachments

water-in-stone · 2026-08-10T03:21:07Z

@ytahdn @yiliang114 Fixed in dc75bf6.

restoreQueuedPromptsToEditor now restores images and input annotations for a non-empty payload only when its text is actually merged into the composer. Image-only payloads retain their existing restoration behavior.

I added the requested regression test using an existing describe draft, a queued describe payload with an image and annotation, and an HTTP 413 rejection. It fails before the fix and passes afterward.

Validation:

42 relevant Vitest tests passed
Web Shell build and typecheck passed
Targeted ESLint, Prettier, and diff checks passed

Thanks for catching this.

ytahdn

Re-reviewed at dc75bf6. The duplicate attachment restoration issue is fixed: attachments are skipped when non-empty restored text is already present, while image-only restoration remains intact. The focused queue restoration suites pass 42/42 locally. No remaining blockers from my review.

yiliang114

Re-approving at head dc75bf6 after my approval was dismissed by the new push. The P2 is fixed exactly as recommended: restoreQueuedPromptsToEditor now tracks textWasRestored (true only when mergeRestoredPromptText actually changes the editor text) and skips image/annotation restore when the payload text is already present — normal restores and image-only prompts are unaffected, annotation offset alignment holds in both branches, and the new dom test reproduces the exact duplicate scenario (payload text already in editor + 413 reject) asserting no restore calls while the queue drains. The accepted tradeoff is the safer direction: in the dedupe case a not-yet-restored attachment is silently dropped instead of duplicated, and it stays visible to the user. I also verified the main-merge resolutions in e6c85df are clean unions — all four conflicted files keep the full PR wiring (the fourth onImageIngestionNotice site correctly folded into main's artifactPanelSharedProps), so nothing from the original review was dropped. CI green on this head. Nothing blocks merge.

heyang.why added 3 commits August 7, 2026 16:40

water-in-stone requested review from LaZzyMan, doudouOUC, tanzhenxin, wenshao and yiliang114 as code owners August 7, 2026 15:24

water-in-stone changed the title ~~Feat/support drag drop img~~ feat: support drag drop img Aug 8, 2026

water-in-stone changed the title ~~feat: support drag drop img~~ feat: support drag drop img in web-shell Aug 8, 2026

ytahdn previously approved these changes Aug 8, 2026

View reviewed changes

wenshao mentioned this pull request Aug 8, 2026

perf(review): bake a soft tool-call budget into finder and auditor briefs #8708

Merged

wenshao enabled auto-merge August 8, 2026 22:45

water-in-stone changed the title ~~feat: support drag drop img in web-shell~~ feat: support drag and drop img in web-shell Aug 9, 2026

yiliang114 previously approved these changes Aug 9, 2026

View reviewed changes

Merge remote-tracking branch 'upstream/main' into feat/support-drag-d…

e6c85df

…rop-img # Conflicts: # packages/web-shell/client/App.tsx

water-in-stone dismissed stale reviews from yiliang114, ytahdn, and ghost via e6c85df August 9, 2026 14:35

github-actions Bot mentioned this pull request Aug 9, 2026

📊 AI CLI 工具社区动态日报 2026-08-10 litang9/big_model_radar#341

Open

ytahdn approved these changes Aug 10, 2026

View reviewed changes

yiliang114 approved these changes Aug 10, 2026

View reviewed changes

wenshao added this pull request to the merge queue Aug 10, 2026

Merged via the queue into QwenLM:main with commit e465867 Aug 10, 2026
52 of 53 checks passed

qwen-code-dev-bot mentioned this pull request Aug 10, 2026

fix(web-shell): reconcile mid-turn messages with daemon state #8798

Merged

Navigation Menu

Repository navigation

feat: support drag and drop img in web-shell - #8696

feat: support drag and drop img in web-shell#8696
wenshao merged 5 commits into
QwenLM:mainfrom
water-in-stone:feat/support-drag-drop-img

water-in-stone commented Aug 7, 2026 •

edited

Loading

Uh oh!

water-in-stone commented Aug 8, 2026

Uh oh!

ytahdn commented Aug 8, 2026

Uh oh!

ytahdn left a comment

Uh oh!

water-in-stone commented Aug 8, 2026

Uh oh!

yiliang114 left a comment

Uh oh!

water-in-stone commented Aug 9, 2026 •

edited

Loading

Uh oh!

water-in-stone commented Aug 9, 2026

Uh oh!

ytahdn commented Aug 10, 2026

Uh oh!

water-in-stone commented Aug 10, 2026

Uh oh!

ytahdn left a comment

Uh oh!

yiliang114 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

water-in-stone commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does

Why it's needed

Reviewer Test Plan

How to verify

Evidence (Before & After)

Tested on

Risk & Scope

Linked Issues

What this PR does

Why it's needed

Reviewer Test Plan

How to verify

Evidence (Before & After)

Tested on

Environment (optional)

Risk & Scope

Linked Issues

Uh oh!

water-in-stone commented Aug 8, 2026

Uh oh!

ytahdn commented Aug 8, 2026

Review Summary — PR #8696

What this PR does

Strengths

Observations

Uh oh!

ytahdn left a comment

Choose a reason for hiding this comment

Uh oh!

water-in-stone commented Aug 8, 2026

Uh oh!

yiliang114 left a comment

Choose a reason for hiding this comment

Uh oh!

water-in-stone commented Aug 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

water-in-stone commented Aug 9, 2026

Uh oh!

ytahdn commented Aug 10, 2026

Uh oh!

water-in-stone commented Aug 10, 2026

Uh oh!

ytahdn left a comment

Choose a reason for hiding this comment

Uh oh!

yiliang114 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

water-in-stone commented Aug 7, 2026 •

edited

Loading

water-in-stone commented Aug 9, 2026 •

edited

Loading