Repository navigation
fix(core): recover function-style XML tool calls - #13437
Conversation
Co-authored-by: Qwen-Coder <[email protected]>
|
The final built CLI for #10692 used an isolated loopback SSE provider and safe fixtures. Both final-candidate scenarios passed.
Parser SHA-256: Four direct-source checks passed. The final two-file unit run had 622 passes and two LlmChat timeouts out of 624 tests; all 71 parser tests passed. A serial single-worker rerun of the four matching cases passed without raising timeouts. The original full-suite failure logs remain preserved. Affected build/typecheck, bundle, lint and formatting passed. Earlier CLI evidence covers the main/candidate comparison, attribute-bearing and fenced examples, failed streams and partial-write protection on preceding artifacts. These are actual built headless CLI runs with a controlled provider. They do not establish production-model or TUI behavior, or removal of XML already painted during streaming. The broader issue remains open for behavior outside this patch. 中文说明#10692 的最终真实构建 CLI 使用隔离本地 SSE provider 和安全 fixture,两个候选场景均通过。
解析器 SHA-256 为 四项直接源码检查通过。最终两文件单测 624 项中 622 项通过、两项 LlmChat 超时,全部 71 项 parser 测试通过。两个失败名对应的四项用例在串行、单 worker 下隔离复跑全部通过,未提高超时限值;原整组失败日志保留。受影响包 build/typecheck、bundle、lint 和格式检查通过。较早产物的 CLI 证据另行覆盖主分支/候选对比、带属性及围栏示例、失败流和半截写入保护。 这些是受控 provider 下真实构建 headless CLI 的结果,不等于生产模型或 TUI 验证,也不宣称消除实时流已经显示的 XML。宽泛 issue 继续跟进补丁范围之外的行为。 |
Current-head tmux / TUI verification — 2026-10-05Tool recovery is verified on 2a064ee. One real interactive The raw XML block remains visible in the final terminal above the successful Read step. Recovery removes the recovered span from persisted model text; this PR does not remove text already rendered in the TUI. #10692 remains open for that display boundary and the other documented unsupported forms.
The first harness returned FAIL / exit 1: its broad classifier counted the managed-memory background request as a third primary request. The original failed result, script, requests and journal are preserved. Offline validation of the same recordings corrects that classification and passes 11 scoped assertions; the product was not rerun. No product failure was dismissed. Environment: macOS, Node 22.22.0, CLI 0.24.7, anonymous local workspace and isolated configuration/runtime. All model responses came from an owned loopback SSE provider; production-model behavior is unverified. Entrypoint SHA256: This screenshot comes from the actual live tmux attach stream through the repository's terminal renderer. It shows both successful tool execution and the remaining XML display boundary. The path and identifiers belong only to this anonymous local test. Image assets are hosted in Complete readable tmux capture: startup, typed prompt, streaming checkpoint, final pane中文测试报告当前提交 2a064ee 的工具恢复验收通过。实际在 tmux 中启动编译 CLI,用真实键盘输入提交1个读取匿名文件的用户流程。受控 provider 只把完整 function/parameter XML 分块放进 content,没有原生 tool_calls,也没有伪造工具回执。CLI 实际执行1次 read_file,真实文件字节和相同调用ID进入下一次主请求,终端出现成功 Read 步骤和最终回答;匿名文件未被修改。 最终终端仍显示原始 XML 块,位于成功 Read 步骤上方。 持久化模型文本已移除恢复部分,但已经绘制的终端文本尚未移除。工具恢复成功不能算作界面泄漏全部解决;#10692 保持开放,其余不支持的格式也继续保留在 issue 范围内。 第一次脚本确实 FAIL/exit1:它把带有历史 marker/read_file 定义的后台记忆维护请求误算作第3次主请求。原始失败结果、脚本、3份请求和真实 journal 均保留。对同一份记录离线核对后,实际是2次主请求、1次后台请求,1次工具及1次真实回执,11个限定断言通过;没有重启或重跑产品,也没有掩盖真实产品失败。 环境为 macOS、Node 22.22.0、CLI 0.24.7,使用匿名目录、隔离配置/runtime以及自有 loopback SSE provider,未验证生产模型。HEAD、源码和全部入口/chunk/native产物前后稳定,入口及657文件清单摘要见上,实际加载371文件/369chunks。CLI正常退出0,自有provider、tmux/CLI、截图renderer及临时目录已清理。 上图来自同一个真实 tmux pane 的实时终端渲染;完整4步 capture-pane 可读记录也已贴在上方,涵盖启动、输入、流式检查点和最终界面。图中目录/标识均来自匿名本地测试,图片仅上传到 img-host,已逐字节读回确认。此轮只覆盖1个真实TUI流程,未重复旧headless矩阵或声称所有瞬时绘制行为已修复。 |
|
The current-head unit and lint/static checks passed. The The PR remains ready for review. Current-head terminal evidence is in #13437 (comment) ; the remaining hosted-process/WebShell checks and maintainer review are still tracked by GitHub. 中文:当前提交的单元测试及静态检查通过;自动审查进程OOM、退出134,没有提交审查结论,不能视为代码缺陷判定或审核通过。PR保持Ready,实际终端截图及完整capture已发布,剩余检查和维护者审查继续按页面状态跟进。 |
|
UI verification of the current head (2a064ee, branch merged with latest main; build + typecheck green, focused core suites 624/624 passing). Mode: merge-reference. Same harness both arms — the repo's loopback fake OpenAI server ( Before (merge-base fallback): the XML stays dead assistant text. No tool row, and the request ledger shows the next request going out without any tool result ( After (this PR): the split XML is recovered into a real Result: PASS — the user-visible contract change is exactly the one this PR claims: a wrapped/split function-style XML call now executes instead of remaining text. |
yiliang114
left a comment
There was a problem hiding this comment.
Reviewed the taught-dialect recovery at 2a064ee999a4 (base f2e069061ab4), parser source read from the head revision and exercised directly on the head build of xml-tool-call-fallback.ts (node 24, marked 15.0.12), with main as the control.
Verdict: request changes — 3 findings, no approval-blocking complaint about the feature itself.
The feature does what it claims: the full taught shape from prompts.ts (<tool_call> / <function=…> / <parameter=k>v</parameter>, prose interleaved, values on their own lines) recovers every call with the prose preserved, <example>-wrapped system-prompt echoes stay text, inline-code mentions do not suppress a later call, fenced documentation is skipped, edit payloads containing fence lines survive, __proto__ stays a real null-prototype key, and the invoke dialect is unchanged. The example scanner also held up under bold/blockquote/list/heading wrapping, quoted attributes containing >, nesting, and self-closing tags.
What needs work:
- [P1] Truncated-parameter borrowing is still fail-open (inline,
xml-tool-call-fallback.ts:207) — a block whose</parameter>and</function>are both missing swallows the next complete call's closing tags, so the guard sees an empty residue and dispatches a corruptedwrite_filewhosecontentis the following call's markup; that call never runs andremainingTextis empty. New versus main, which recovered nothing. - [P1] Unbounded synchronous lexer cost (inline,
:155) —Lexer.lexInlineruns on every candidate with no<example>early-out, no timeout and notry/catch; 4 KB / 8 KB of[a](prose in a turn that also carries a call measured 3.1 s / 23 s. - [P3] Rejected blocks still feed the example scan (inline,
:245) — masking uses only accepted blocks' ranges.
Every finding has a self-contained repro in the comment. Not reviewed here: the daemon/WebShell end-to-end path (your tmux evidence covers it) and the llm-chat.ts integration wiring, which is untouched by this diff.
中文
结论:建议改一版再合(3 条,均不否定功能方向)。
功能本身是对的:prompts.ts 里教的完整格式(含散文交错、参数值独占行)能全部恢复且散文保留;<example> 包裹的系统提示回显保持为文本;inline code 里的字面标签不会压掉后续真实调用;围栏文档被跳过;含围栏行的 edit payload 正常;__proto__ 仍是 null-prototype 的真实键;invoke 方言无回归。example 扫描在粗体/引用/列表/标题包裹、带 > 的属性、嵌套、自闭合等形态下也都正常。
需要改的:① 【P1】未闭合参数仍会「借用」下一个完整调用的闭合标签 → 守卫看到空残留,于是派发一个 content 被污染的 write_file,而真正的调用不执行、remainingText 为空(main 上不恢复任何东西,属新引入);② 【P1】Lexer.lexInline 无条件跑在全量文本上,无早退、无超时、无 try/catch:本机实测 4 KB / 8 KB 的 [a]( 散文 + 一个调用要 3.1 s / 23 s;③ 【P3】被拒块的参数范围没进掩码集合,仍会影响 example 扫描。每条都在行内评论里给了可复现片段。未覆盖:daemon/WebShell 端到端路径(你已有 tmux 证据)与本次 diff 未触及的 llm-chat.ts 接线。
Scope baseline — resolve-pr-comments round 1 (2026-10-05)
2026-10-06 scope reconciliation: reconstructed three prior substantive repair rounds (merge-only commits excluded); this is round four. The delivered change stays in the same two parser files, adds no dependencies or public surface, and changes the current implementation delta from +224/-62 to +223/-62. Optional rejected-block performance refactoring remains outside this round. |
… lexer cost Three review findings on the recovery path: 1. A block whose parameter was never closed borrowed the close tags of a later block. PARAMETER_PATTERN consumed them, the residual-tag guard saw an empty string, and the truncated call was dispatched with the next call's raw markup as a parameter value while the real call never ran and remainingText came back empty. Count the parameter open tags a block body contains against the ones the accepted matches closed, and reject on mismatch. A rejected block may have swallowed an intact later block, so rescan from just after its open tag; that block is then recovered on its own merits (still skipped when it sits in a fence or an example). 2. The markdown lexer ran over the whole response for every recovery candidate even when the text contained no example tag at all, and it is super-linear on unterminated link/emphasis runs: 4 KB of repeated link openers cost ~3.0 s of synchronous work before the intent-ratio guard. Skip the lexer when no example tag can produce a range, and catch a lexer throw so it degrades to "no example ranges" instead of aborting the turn. 3. Parameter spans were recorded only for accepted blocks, so a rejected block's parameter data was never masked out of the prose the example scan reads; a literal example opener in it swallowed a later valid call. Derive the spans from the raw parameter matches over the whole text. Co-authored-by: Qwen-Coder <[email protected]> Patrol-Run: qwen-pr-closeout/jmuv5xnpc3q
The Lint & Static lane failed on 2ce7b0c at its gate-freshness pre-flight, not on any lint rule. Its own output: "The lint gate changed on 'main' after this branch last incorporated it: scripts/lint.js: 6b878e1 (#12650) (2026-10-05)". That commit landed on main at 12:44:27Z, seven seconds before this job started at 12:44:34Z, and the lane checks out the branch head alone, so it requires the branch to carry the current gate definition. Merging main re-validates it. No source conflicts: main has not touched xml-tool-call-fallback.ts or its test since this branch last merged it. Co-authored-by: Qwen-Coder <[email protected]> Patrol-Run: qwen-pr-closeout/jmuv5xnpc3q
|
The red That job ran 12:44:34Z -> 12:45:42Z and failed at its gate-freshness pre-flight, before linting anything. Its own output:
Re-verified on the merged head:
The fix SHA cited in the three thread replies ( |
|
@qwen-code /triage |
qwen-code-review-bot
left a comment
There was a problem hiding this comment.
APPROVE at c13c2d61, with one non-blocking suggestion about a degradation direction. Scope: I read xml-tool-call-fallback.ts in full at this head plus the whole of 2ce7b0c1's production diff. I did not work through the ~600 lines of test changes beyond confirming the parser suite runs in the green Test lane.
Worth stating first: 2ce7b0c1 (12:42) has no review coverage. @qwen-code-ci-bot's review was dismissed at 09:37, before that commit landed, so the hardening is unreviewed until now. 3 threads, 0 unresolved; CI is 22 pass / 0 fail with nothing in flight.
Finding 1's fix is an arithmetic invariant, and the rescan cannot loop
The borrowed-closer test is:
paramsBlock.match(PARAM_OPEN_PATTERN)?.length !==
(outsideParameters.match(PARAM_OPEN_PATTERN)?.length ?? 0) +
(paramsBlock.match(PARAMETER_PATTERN)?.length ?? 0)Each complete PARAMETER_PATTERN match consumes exactly one open and one close tag, and outsideParameters is the body with those matches removed, so totalOpens === leftoverOpens + completeMatches holds for any well-formed block. It breaks precisely in the borrowed case: one match spans two open tags, so the left side exceeds the right and the block is rejected. Counting rather than pattern-matching the residual is what makes this work, and the comment says why the older residual-tag test could not see it — a borrowed close leaves no residual tag behind.
The rescan position is match.index + match[0].indexOf('>') + 1, which is strictly greater than match.index for any match (indexOf('>') cannot be negative here, since both alternatives in TOOL_CALL_PATTERN begin with a tag that contains one). So lastIndex always advances and the loop terminates; and because the assignment sits in the reject branch only, accepted blocks do not rescan. Letting a swallowed later block be found on its own merits is the right consequence — the alternative was losing a valid call because a preceding invalid one had eaten its close tag.
Finding 3's fix moves the masking input to the right place
parameterRanges is now built by scanning PARAMETER_PATTERN over the whole text before the block loop, instead of accumulating spans from accepted blocks. The comment states the failure it closes: a rejected block's parameter data went unmasked, so a literal example tag inside it could swallow a later valid call. Parameter spans are a property of the raw text, so deriving them from the raw text is the correct dependency direction — and it makes the masking independent of the guard's verdicts, which is what lets findings 1 and 3 compose instead of interact.
Finding 2's early return is sound
if (!text.includes('<example') && !text.includes('</example')) return [];The tag scan that follows matches <\/?example(?=[\s/>])… and every candidate is re-checked against tagPositions, so with no literal example tag in the text the function could only ever return an empty list anyway. Skipping the lexer therefore changes nothing semantically and removes the super-linear cost on example-free responses, which is the common case. The stated measurement — 4 KB of repeated link openers costing ~3.0 s of synchronous work, with nothing bounding the model's output length — is the right reason to gate it.
One non-blocking suggestion: the lexer catch degrades in the unsafe direction
try {
collectTags(Lexer.lexInline(prose), prose, 0);
} catch {
tagPositions.clear();
}Clearing yields "no example ranges", and exampleRanges is consumed as an exclusion filter (:282-287):
!exampleRanges.some(([exampleStart, exampleEnd]) => …)With an empty list, some is false and no block is excluded. So on a lexer throw, a call written inside an explicit example wrapper becomes recoverable and executable, which is the opposite of the body's promise that "fenced or explicit example documentation … remains text".
The blast radius is narrower than it first looks, and this is why I am filing it as a suggestion rather than a blocker: the other half of that filter, positionInsideFence(text, …), does not depend on the lexer, so fenced content stays excluded even when the lexer throws. Only the explicit-wrapper half fails open. And reaching it requires Lexer.lexInline to throw at all — which the author judged possible enough to add the catch for, so the consequence of the catch is worth choosing deliberately rather than inheriting.
The conservative alternative costs nothing relative to today's main: when the lexer throws, bail out of recovery for this response entirely rather than continuing with no example ranges. "Recover nothing this turn" is exactly the behaviour main has now, so it cannot be a regression, whereas "recover everything, including documentation" is a new and silent one. If bailing out of the whole response is too blunt, treating the throw as "the entire text is unverifiable, so exclude all blocks" is the same shape.
A second note, also non-blocking
The rescan guarantees progress but not bounded total work: each rejected block restarts the block scan from just after its own open tag, so a response containing many rejected blocks is quadratic in the block loop. That is much milder than the lexer's super-linearity and the intent-ratio guard bounds how much text reaches here, but since finding 2 exists precisely because synchronous cost on model-controlled input mattered, the interaction is worth knowing about rather than discovering later.
Two things done right that are easy to miss
args is Object.create(null), so a model-supplied parameter name cannot pollute a prototype — consistent with how the rest of the repo handles network-derived keys. And the catch's comment records why a catch is needed now when it was not before ("the regex-only implementation this replaced could not throw"), which is the kind of provenance that stops the next reader deleting it as dead defensiveness.
CI and vote effect
22 pass, 0 fail, 9 skipped, nothing in progress — including Test (ubuntu-latest, Node 22.x), Lint & Static, Integration Tests (no-AK, No Sandbox), web-shell E2E Smoke and both Desktop Shell legs. The body discloses a local run of 622/624 with two LlmChat cases timing out and passing on a serial single-worker rerun with unchanged limits; the green Test lane at this head is consistent with that being local contention rather than a defect, so I am not carrying it forward — but the two-file unit run is the one that exercises this parser, and it is worth knowing the full-suite result rests on CI rather than on a clean local run.
All three changed files are under packages/core/src/core/, which is CODEOWNERS-covered, so require_code_owner_review applies: one of wenshao, tanzhenxin, LaZzyMan, doudouOUC or qqqys must approve. This approval satisfies required_approving_review_count: 1 but not the code-owner requirement, so reviewDecision stays REVIEW_REQUIRED until one of them does.
yiliang114
left a comment
There was a problem hiding this comment.
Main flow looks right — the taught-dialect recovery, fence/<example> shields, native-call priority, and span removal all verified independently (details below). Two follow-ups inline, one worth closing before merge; nothing here questions the direction.
Verification against this head (c13c2d61a0), independent of the PR-thread evidence:
- Merge-reference TUI run, real built CLI in tmux against the repo's loopback fake OpenAI server, before/after differing only in this file (merge-base vs head, same build everywhere else). After: one real
read_fileexecuted, the next model request carried the matchingtoolreceipt with the actual file bytes; before: same XML stayed dead text with zero tools. Fenced and<example>-wrapped variants of the same call executed zero tools and stayed intact text. - Regression pin: the new taught-dialect suite is red on merge-base source (16 failed / 13 passed) and green at this head; full run of both changed test files 630/630; eslint clean on all three files.
- The three earlier round findings (borrowed closers, no-marker lexer cost, rejected-block masking) are confirmed fixed at this head, empirically where reproducible.
Count: 0 P0, 1 P1, 1 P2, 1 suggestion (a failed-stream no-dispatch pin is still missing — the streamError === null gate is pre-existing and correct by construction, but the PR body lists it as a test-plan item, so pinning it would close the loop; non-blocking).
Note for the gate: contributor PR touching packages/core, so per AGENTS.md this still needs a maintainer's review before merge regardless of my verdict.
The invoke branch of TOOL_CALL_PATTERN admits `>` inside the quoted name,
so deriving the rescan offset with indexOf('>') could land in the middle
of the open tag. The rescan then matched a complete call embedded in that
name attribute and dispatched it out of a block the guard had just
rejected, leaving corrupted residual text behind.
Scan for the first `>` outside a quoted run instead, and skip the whole
block when no unquoted tag end exists.
Co-authored-by: Qwen-Coder <[email protected]>
Patrol-Run: qwen-pr-closeout/jmuvn2xwd0e
marked's inline lexer is quadratic on unterminated emphasis runs, and the early return only skipped it when no example marker appeared anywhere -- so any turn merely mentioning the marker paid a full lexInline pass over the whole masked prose. Measured on this build, `'*a '.repeat(n)` after one marker costs 238ms at 2k units, 794ms at 4k, 3.0s at 8k and 11.7s at 16k (~64KB): a synchronous stall inside a streaming turn, with nothing upstream bounding model output length. Past MAX_LEXER_SCAN_LENGTH the lexer pass is now skipped and example detection degrades to the regex-only tag scan, which bounds the synchronous work deterministically. Co-authored-by: Qwen-Coder <[email protected]> Patrol-Run: qwen-pr-closeout/jmuvn2xwd0e
Counting the parameter open tags a block body contains against the ones the accepted matches closed fired on an open tag that sits inside an accepted value — the same geometry as a borrowed closer, but there the swallowed region holds no nested call. The intact call was rejected, so it never ran and its raw markup stayed visible, where the merge base recovered it with exactly the intended arguments. A closer is only borrowed when the value swallowed a later call, so key the guard on a call opener the parameters did not consume; removing that term makes the borrowed-closer witness fail again. Co-authored-by: Qwen-Coder <[email protected]> Patrol-Run: qwen-pr-closeout/jmuw232ch95
qwen-code-review-bot
left a comment
There was a problem hiding this comment.
Re-approving at 838292de. My approval at c13c2d61 was dismissed by the push. I reviewed the delta only — the three commits and their 82 lines of new tests — and two of them fix defects in code I approved. I verified both by executing the guards rather than by reading them, so the record here is measured rather than asserted.
I was wrong about the borrowed-closer arithmetic, and the new test is strictly better
In my previous review I called the open/close tag count "an arithmetic invariant" and wrote that counting rather than pattern-matching the residual was what made it work. That was wrong. I ran both guards over three inputs:
| input | expected | old arithmetic | new nested-opener test |
|---|---|---|---|
parameter value that literally contains <parameter name="x"> |
accept | reject | accept |
| unclosed parameter that borrows a nested later call's close tag | reject | reject | reject |
| well-formed single parameter | accept | accept | accept |
So the arithmetic had a false positive on exactly the case the new commit names — a value documenting this syntax — and the consequence was the bad one: an intact call was rejected, never ran, and left its raw markup visible. The replacement /<(?:function|invoke)(?:[\s=>]|$)/.test(paramsBlock) still rejects the borrowed-closer case, because a borrowed close only happens when the swallowed region contains a nested call, and a nested call carries its own opener. Same coverage of the case the arithmetic was added for, without firing on legitimate literal text. The new comment states the model correctly: an open tag inside an accepted value is that value's own text, not evidence of an unclosed parameter. The arithmetic was a proxy for "a nested call was swallowed", and proxies fire on things that merely resemble the target.
The openTagEnd fix closes a hole I had the observation for and did not draw the conclusion from
I noted last round that TOOL_CALL_PATTERN's invoke branch admits > inside the quoted name, and used that only to argue resumeAt is strictly greater than match.index, so the loop terminates. Progress was the wrong thing to be checking. Measured on <invoke name="a>b"> followed by further markup: indexOf('>') + 1 is 16, inside the quoted name, while openTagEnd returns 19, the real tag end. A rescan from 16 runs inside a name attribute the guard had just rejected, and can match and dispatch a complete call embedded there — which is what does not dispatch a complete call embedded in a rejected block name now pins.
The -1 fallback is also the right shape: when no unquoted > exists the tag end is not derivable, so resumeAt becomes match.index + match[0].length and the block is skipped whole rather than rescanned. "Do not rescan inside a rejected block unless the tag end is known" is a safer default than any computed guess.
The lexer cap, and what it changes about my earlier suggestion
The measured numbers in the comment are the justification, and they are the right kind: '*a '.repeat(n) after one example marker costs 238 ms at 2k units, 794 ms at 4k, 3.0 s at 8k and 11.7 s at 16k — a synchronous stall inside a streaming turn, with nothing upstream bounding a model turn's length.
The part I want to record is that the cap path fails closed. (!skipLexer && !tagPositions.has(tagPosition)) waives the lexer-confirmation requirement when the lexer was skipped, so every regex-matched tag is taken at face value and example ranges still form. The test pins this with a spy and a positive control — not.toHaveBeenCalled() over the cap, toHaveBeenCalled() under it — and asserts recovered === false on both sides, so "still honouring examples" is checked rather than assumed. A test that only asserted the over-cap behaviour would have passed with the lexer never called at all.
That narrows but does not close the suggestion from my last review. The catch branch still runs tagPositions.clear() while skipLexer stays false, so on a lexer throw the requirement is not waived and every tag is skipped: no example ranges, and the explicit-wrapper exclusion is lost while the fence exclusion survives. The exposure is smaller now, since the inputs most likely to make the lexer choke are the ones the cap diverts away from lexing. And this PR has demonstrated what the conservative degradation looks like, which makes the fix small: have the throw set the same waiver the cap sets, instead of clearing positions and keeping the requirement. Two lines, and it makes both degradations behave identically. Still non-blocking — it needs a throw to reach.
My other note is unchanged and still non-blocking: the reject path still rescans (lastIndex = resumeAt), so the position is now correct but the total work is still unbounded across many rejected blocks. Correctness was the thing that needed fixing; the shape can wait.
Gates
5 threads, 0 unresolved. CI 21 pass, 0 fail, 8 skipped, with only review-pr still running — Test (ubuntu-latest, Node 22.x), Lint & Static, Integration Tests (no-AK, No Sandbox) and web-shell E2E Smoke are green at this head, so the three new tests have executed in CI rather than only locally.
Both files remain under packages/core/src/core/, so require_code_owner_review applies and one of wenshao, tanzhenxin, LaZzyMan, doudouOUC or qqqys must approve. This re-approval satisfies required_approving_review_count: 1 but not the code-owner requirement, so reviewDecision stays REVIEW_REQUIRED until one of them submits.
chiga0
left a comment
There was a problem hiding this comment.
No blocking findings. Approval blockers: none.
Scope
Read xml-tool-call-fallback.ts (head) in full. Read callers in llm-chat.ts (lines 6417–6419). Verified marked is already a declared dependency. Diffed against all existing inline comments and review bodies.
NOT reviewed: test changes (~600 lines) beyond confirming the new tests exist and exercise the stated cases; Windows/Linux runtime behaviour (macOS tested by author per PR body).
What I checked
API/compatibility (class 2). extractXmlToolCalls and tryRecoverXmlToolCalls have unchanged signatures. containsXmlToolCalls broadens its match set (now also detects function-style XML), which is the intended semantic change. extractXmlToolCalls is not re-exported from the package's index.ts barrel, so no external consumer is affected. The two llm-chat.ts call sites at lines 6417–6419 pass through unchanged.
Contract internals. TOOL_CALL_PATTERN.lastIndex is reset to 0 at every entry point: containsXmlToolCalls, recoverableToolCallBlocks, and the prose-ratio check in tryRecoverXmlToolCalls. No re-entrancy risk in single-threaded JS.
Prose-ratio guard. TOOL_CALL_PATTERN.replace on the full text (including fenced/example blocks) computes the guard the same way as before for invoke-style and extends it to function-style. Blocks excluded by fence/example detection are still available for the ratio denominator.
Cross-check against existing reviews. Three P1 issues and one P2 found by yiliang114 and addressed in subsequent commits:
| Finding | My assessment at current head |
|---|---|
| P1: Borrowed-closer dispatch (truncated parameter swallowing next block's close tags) | Confirmed fixed. /<(?:function|invoke)(?:[\s=>]|$)/.test(paramsBlock) correctly rejects a block whose params body contains a nested opener — covering the borrowed case without false-positives on values that document the syntax literally. |
P1: Unbounded lexer cost (Lexer.lexInline quadratic on long runs) |
Confirmed fixed. MAX_LEXER_SCAN_LENGTH = 64 * 1024 cap documented with measured timings; !text.includes('<example') early return skips the lexer on the common case. |
P3: parameterRanges scope (rejected block data unmasked) |
Confirmed fixed. Ranges now derived from a full-text PARAMETER_PATTERN scan before the block loop, decoupled from guard verdicts. |
P2: indexOf('>') landing inside invoke name attribute |
Confirmed fixed. openTagEnd() walks the open tag respecting quoted runs; returns -1 when no unquoted > exists, causing the block to be skipped whole rather than rescanned at an interior position. |
Non-blocking note from existing reviews (re-confirming): When Lexer.lexInline throws, the catch clears tagPositions and skipLexer remains false, so the tag scan finds no confirmed positions and produces no example ranges. Explicit-wrapper exclusion is silently lost on a lexer throw, though fence exclusion survives. The conservative fix (treat a lexer throw the same as the over-cap path: skip the lexer requirement rather than clear the confirmed set) remains a two-line change and the regression direction is safe — "recover nothing inside explicit examples this turn" rather than "recover everything including documentation". Non-blocking as before.
Quadratic reject path (still non-blocking): TOOL_CALL_PATTERN.lastIndex = resumeAt is strictly greater than match.index after openTagEnd fix, so the loop terminates. Total work across many rejected blocks is O(n²) in the block count, milder than the lexer's super-linearity and bounded by the intent-ratio guard. No action required here.
Reviewed with AI assistance.
qqqys
left a comment
There was a problem hiding this comment.
Critical-only scan at head 838292de1171b9d51dde86f65b6df40c05226df8 — partial, budget-limited.
Verdict: COMMENT. No Critical is asserted against this diff. This is not an approval: the review ran out of budget before the historical-blocker gate could be closed, and the unconfirmed gates are listed below so the author and maintainer know exactly what was and was not checked.
What was completed
The whole production diff was read at this head — packages/core/src/core/xml-tool-call-fallback.ts, +224/-62, all 395 patch lines. Nothing blocking was found in it, and three properties were checked directly rather than assumed:
- The rescan loop terminates.
resumeAtismatch.index + tagEndwhenopenTagEndderives a tag end, elsematch.index + match[0].length. Both are strictly greater thanmatch.index, and the assignment sits only in the reject branch, soTOOL_CALL_PATTERN.lastIndexalways advances and accepted blocks do not rescan. - The offset remap for
remainingTextis sound.removedRangesinherits ascending order from the match loop, and the prefix-sum walk with thestart > originalOffsetbreak maps a wrapper's post-strip offset back to its original position correctly — which is what lets an originally-empty wrapper survive while a wrapper emptied by stripping does not. - Parameter spans are derived from raw text, not from accepted blocks.
parameterRangesis built by scanningPARAMETER_PATTERNover the whole text before the block loop, so masking and fence tracking no longer depend on the guard's verdicts. That is the right dependency direction and is what lets the borrowed-closer rejection and the masking compose.
Gates not confirmed — the reason this is not an approval
- The three earlier Criticals were not verified at this head. The
qwen-code-review-botreview that assesses them (5418141544) was written againstc13c2d61, which is two commits behind the current head, and it isDISMISSED. Its reasoning — the borrowed-closer count invariant, the rescan advancement, the early example return — reads as sound, and the rescan and masking halves are independently confirmed above, but inheriting a verdict from an older head over a dismissed review is exactly what this channel's rules do not permit. Theqwen-code-ci-botreview that originally filed those findings was dismissed before2ce7b0c1landed, so its findings are not readable from the review collection either. - The two approvals at this head were not read.
qwen-code-review-bot(5424307028) andchiga0(5424322087) both approved at838292de. Their scope disclosures would say what was covered; without them I cannot borrow their coverage. - The ~539 lines of test changes were not read, beyond noting that
Test (ubuntu-latest, Node 22.x)is green. For a parser whose whole job is deciding which model-authored markup becomes an executable call, a false green in the new cases would be the most likely place for a defect to hide. - Ten inline author threads on this file were not read.
One item worth a maintainer's eye, explicitly not filed as a Critical
computeExampleRanges degrades in the unsafe direction on a lexer throw: the catch clears tagPositions, the tag scan re-checks every candidate against that now-empty set, so exampleRanges comes back empty and no block is excluded — a call written inside an explicit example wrapper becomes recoverable and executable, which is the opposite of the documented promise. qwen-code-review-bot already graded this non-blocking at c13c2d61 and it is unchanged at this head. I am not filing it as a Critical because I could not establish within budget that Lexer.lexInline can actually throw here, and an unproven reachability chain is not a reportable finding. Recording it so the choice is visible rather than implicit. Note the direction is not uniform: the skipLexer path over MAX_LEXER_SCAN_LENGTH leaves tagPositions empty too, but there the regex-matched tags are taken at face value, which yields more exclusions rather than fewer.
CI
Healthy at this head — Test (ubuntu-latest, Node 22.x), Lint & Static and the integration and desktop lanes report pass, with no failure attributable to this PR.
Next step
Nothing here asks for a code change. A reviewer with budget to spare should close gate 1 by re-checking the three earlier findings against 838292de in source, and gate 3 by confirming the new parser cases actually fail when the guards are reverted; once those two are closed this is approvable on the strength of the production read above.
Use the existing bounded regex scan when real nested emphasis overflows the lexer, preserving example text while recovering following calls. Co-authored-by: Qwen-Coder <[email protected]>
f3dd8cf
|
Delivered A real input reproduced the failure with the core-resolved The catch now reuses the existing regex-only example scan used for oversized input. This is intentionally conservative: if unlexable text contains an ambiguous inline-code Validation:
中文:最新建议中的 lexer 异常并非只靠 mock 推测,真实依赖和真实文本已能复现。修复后示例不会被误恢复为调用,后续正常调用仍可恢复;新的回归测试覆盖了这个行为。异常时沿用已有的保守回退,歧义内容可能继续显示为文本;已显示 XML 的清理和通用性能改写继续留在本 PR 范围之外。新提交仍需 CI 和复审。 |
qqqys
left a comment
There was a problem hiding this comment.
Critical-only scan at head f3dd8cf89b14d0a3374d50f59dd410bf00c313a0 — complete. Verdict: APPROVE.
My previous review on this PR was a budget-limited COMMENT at 838292de. It asserted no Critical and listed four gates it had not closed. One commit has landed since; this review closes those gates at the current head.
The delta since the last review is the fix for the one item I declined to file
f3dd8cf8 is +3/-4 in xml-tool-call-fallback.ts. It is exactly the conservative degradation both prior reviewers specified for the lexer catch:
let skipLexer = text.length > MAX_LEXER_SCAN_LENGTH;
…
} catch {
// Preserve example filtering with the same fallback as oversized text.
skipLexer = true;
}The old branch ran tagPositions.clear() while skipLexer stayed false, so the tag scan's (!skipLexer && !tagPositions.has(tagPosition)) was true for every candidate, exampleRanges came back empty, and the explicit-wrapper exclusion was silently lost — a call documented inside <example> became recoverable and executable. Setting the waiver instead of clearing the set makes the throw path behave identically to the over-cap path: regex-matched tags are taken at face value, example ranges still form, and the wrapper exclusion survives.
Three properties of the replacement checked directly:
- No stale-partial-set hazard. If
collectTagsthrows midway,tagPositionsmay hold partial entries.skipLexer = trueshort-circuits the&&, so the set is not consulted at all — partial state is inert. - The mutation is visible where it is read.
skipLexeris function-scopedletand the consumingwhileloop runs after thetry/catch, so the assignment lands before any read. - The degradation direction is safe. Face-value tag acceptance can only yield more example ranges, hence more blocks excluded. "Recover nothing inside explicit examples this turn" matches
main's behaviour and cannot be a regression; the old direction could have been.
The rewritten test is stronger than the mock it replaces. It drives a real lexer overflow ('*'.repeat(4096) either side of prose) instead of a stubbed throw, bounds the example range with a real </example>, and asserts the safe outcome on both sides: only read_file is recovered with {file_path: 'b.ts'}, and remainingText still contains the full documentation. expect(text.length).toBeLessThan(64 * 1024) is the load-bearing control — it proves the test exercises the catch path rather than the cap path, which now behave identically and would otherwise be indistinguishable.
Gate 1 — the five historical blocking findings are fixed at this head
All five review threads are resolved, and I verified each in source at f3dd8cf8 rather than inheriting a verdict from an older head:
| Finding | Evidence at current head |
|---|---|
| P1 borrowed-closer dispatch | Reject branch tests /<(?:function|invoke)(?:[\s=>]|$)/.test(paramsBlock). A borrowed close only occurs when the swallowed region holds a nested call, which carries its own opener — so the case is covered without firing on a value that documents this syntax literally. |
| P1 unbounded lexer cost | MAX_LEXER_SCAN_LENGTH = 64 * 1024 cap with measured timings in the comment, plus the !text.includes('<example') && !text.includes('</example') early return. The early return is semantically neutral: every candidate is re-checked against tagPositions, so an example-free text could only ever return []. |
| P1 single-marker re-opening | Same cap governs the marker-present path; the cap path fails closed, as above. |
P3 parameterRanges scope |
Built by scanning PARAMETER_PATTERN over the whole text before the block loop, so masking no longer depends on the guard's verdicts and a rejected block's data cannot swallow a later valid call. |
P2 indexOf('>') inside a name |
openTagEnd() walks the open tag tracking quote runs and returns -1 when no unquoted > exists; resumeAt then becomes match.index + match[0].length, skipping the block whole rather than rescanning into a rejected name attribute. |
Independently re-confirmed at this head: the rescan loop terminates, since resumeAt is match.index + tagEnd (tagEnd ≥ 1) or match.index + match[0].length, both strictly greater than match.index, and the assignment sits only in the reject branch so accepted blocks do not rescan. The remainingText offset remap is sound — removedRanges inherits ascending order from the match loop and the prefix-sum walk breaks on start > originalOffset, which is what lets an originally-empty wrapper survive while one emptied by stripping does not. args remains Object.create(null), so a model-supplied parameter name cannot pollute a prototype.
Gates 2 and 3 — coverage I previously could not borrow
The two approvals at 838292de were read in full. qwen-code-review-bot executed the guards over concrete inputs rather than reading them, and correctly retracted its own earlier "arithmetic invariant" claim by showing the old count rejected a parameter value that literally contained <parameter name="x"> — a false positive in the bad direction, since an intact call was rejected and left raw markup visible. chiga0 independently reached the same fixed/not-fixed verdict on all four findings and disclosed its own scope, including that it did not review the ~600 lines of tests. Their coverage is consistent with what I verified in source above; nothing in it is contradicted.
On tests: the production file is read in full at this head and I found no provable blocking defect in an uncovered path, and Test (ubuntu-latest, Node 22.x) is green at this head, so the new parser cases executed in CI rather than only locally. Per this channel's rules a missing test is not itself a Critical, and I have no evidence of a false green here.
Gate 4 — inline threads
All five inline threads read; zero unresolved. None was closed on a resolved flag alone — each was checked against the current source as tabulated above.
CI
Healthy at this head. Test (ubuntu-latest, Node 22.x), Lint & Static, Integration Tests (no-AK, No Sandbox), web-shell E2E Smoke and both Desktop Shell legs pass. The only failures are review-pr and fallback-comment, which are this repository's review-automation lane and not product checks; no failure is attributable to this PR's code.
Note for the maintainer, not a blocker
The reject path still rescans, so total work across many rejected blocks is quadratic in the block count. Position is now correct and the intent-ratio guard bounds how much text reaches here, so this is a shape concern rather than a correctness one. Both prior reviewers graded it non-blocking and I agree.
All changed files are under packages/core/src/core/, so require_code_owner_review applies. This approval is submitted as qqqys.
chiga0
left a comment
There was a problem hiding this comment.
No blocking findings. Approval blockers: none.
Re-review delta
My round-1 review (838292de) was dismissed when head moved to f3dd8cf89b14. The delta is one commit: +3/-4 in xml-tool-call-fallback.ts, the catch block fix I listed as a non-blocking recommendation.
Prior non-blocking note — FIXED. The catch branch in computeExampleRanges now sets skipLexer = true rather than calling tagPositions.clear(). As I noted last round, clearing the set while skipLexer remains false caused every subsequent tag to fail the tagPositions.has() check, producing empty example ranges and silently losing explicit-wrapper exclusion on a lexer throw. The fix makes the throw path behave identically to the over-cap path: example ranges still form from the regex scan, the wrapper exclusion survives. Confirmed at head by reading xml-tool-call-fallback.ts lines 752–757.
Three additional properties confirmed at this head:
- No stale-partial-set hazard:
skipLexer = trueshort-circuits the&&, so any partial entries intagPositionsfrom a midway throw are never consulted. skipLexeris function-scopedlet; the assignment lands before the consuming loop.- Degradation direction is safe: "treat this turn as over-cap" is strictly more conservative than "silently lose all example ranges".
Round-1 findings status
All prior findings from yiliang114 (borrowed-closer dispatch, unbounded lexer cost, rejected-block masking, openTagEnd attribute parsing) and the one non-blocking note from my round-1 review are confirmed fixed at this head. No new issues introduced by the delta.
Scope (cumulative)
Read xml-tool-call-fallback.ts at head in full (round 1). Read llm-chat.ts callers at lines 6417–6419 (round 1). Verified marked ^15.0.12 is a production dep (round 1). Delta commit read in full this round.
NOT reviewed: test changes beyond confirming they exercise the stated cases; Windows/Linux runtime behaviour.
Reviewed with AI assistance.
yiliang114
left a comment
There was a problem hiding this comment.
Verdict: request changes — one blocking finding in the new recovery path. The rest of this head checks out.
Verified clean at f3dd8cf8 before the blocker: taught-dialect recovery for wrapped/bare/split calls; span removal (strictly ascending, non-overlapping ranges — no prose eaten); the llm-chat.ts gating (recovery only on a clean stream with no native calls, and the recovered functionCall flows through the normal scheduler/confirmation path — no approval bypass); fence/example shields under the real lexer; and the f3dd8cf8 lexer-failure change itself, which is strictly better than the tagPositions.clear() it replaced (that one dropped example shielding entirely). CI is green at this head.
The blocker is inline at line 282, reproduced by executing this head's extractXmlToolCalls directly:
<function=write_file><parameter=file_path>a.txt</parameter><parameter=content>How to call: <function=read_file><parameter=file_path>b.ts</parameter></function> done</parameter></function>
→ [{ name: "read_file", args: { file_path: "b.ts" } }] // write_file never runs
Trace: the outer match ends at the documented call's </function> → the guard sees a <function opener anywhere in paramsBlock (values included) and rejects the intact block → the rescan restarts at resumeAt, just past the outer open tag, i.e. back inside the parameter values → the documented read_file matches clean and is accepted. None of the three documentation guards cover that spot: fence tracking skips lines inside parameter ranges, computeExampleRanges skips <example> tags inside parameter ranges, and the fence check runs on outsideParameters with values stripped. The block's own comment ("an open tag inside an accepted value is that value's own text") describes exactly the case the guard doesn't distinguish — the pinned test only covers a <parameter> mention, not a complete call, and fencing the documented call inside the value doesn't protect it either.
This blocks merge because the natural "write_file documents the dialect" case executes the wrong tool: the intended write is silently lost, its markup is left mangled in remainingText, and an arbitrary documented call is dispatched instead (approval-gated, but no longer reading as documentation to the approver).
Two non-blocking notes: the skipLexer face-value example pairing has a specific inline-code hole (inline at line 190 — sharper instance of the regex-only degradation direction flagged in an earlier round), and positionInsideFence re-scans text[0..index] per example tag and per block while the 64KiB cap covers only the lexer pass — a very large turn dense with <example> tags multiplies that O(N) rescan. Follow-up material, not blockers.
| // this syntax literally — and counting it as unclosed rejected the intact | ||
| // call, which then never ran and left its raw markup visible. | ||
| if ( | ||
| /<(?:function|invoke)(?:[\s=>]|$)/.test(paramsBlock) || |
There was a problem hiding this comment.
[P1] A complete call documented inside a parameter value is dispatched, and the intended outer call is silently dropped.
Reproduced against this head's source:
input: <function=write_file>…<parameter=content>How to call: <function=read_file><parameter=file_path>b.ts</parameter></function> done</parameter>…
output: [{ name: "read_file", args: { file_path: "b.ts" } }] — write_file dropped
This guard tests paramsBlock — parameter values included — for a <function/invoke opener, so a value that documents a complete call rejects the intact outer block. The rescan then restarts at resumeAt (just past the outer open tag, back inside the values), where the documented call matches clean and is accepted. The invoke-style equivalent behaves identically, and fencing the documented call inside the value does not protect it (parameter-interior lines are masked from fence tracking).
Might be worth testing outsideParameters here instead of paramsBlock for function/invoke openers — that matches the comment above ("an open tag inside an accepted value is that value's own text") — and/or refusing a rescanned block whose start falls inside a raw parameter range, so the envelope-swallow recovery keeps working while documentation-in-value is never dispatched.
| while ((match = tags.exec(text)) !== null) { | ||
| const tagPosition = match.index; | ||
| if ( | ||
| (!skipLexer && !tagPositions.has(tagPosition)) || |
There was a problem hiding this comment.
[P2] Non-blocking. In skipLexer mode (over the 64KiB cap, or the new lexer-failure fallback) every regex-matched <example> tag is taken at face value — including one inside a single-backtick inline code span, which the real lexer would not tokenize as html. A backticked `</example>` inside an example block closes the range early, and documented call markup after it becomes recoverable. Execution stays approval-gated and the degraded mode is documented above, so this doesn't block merge — but a bounded line-local check (odd backtick-run count before the tag ⇒ code span) plus a regression test at text.length > 64KiB would close the asymmetry.



What this PR does
Recover complete function/parameter XML through the existing tool-call fallback, including wrapped, bare and split responses. Remove only the spans actually recovered. Fenced or explicit example documentation, including example attributes and tag whitespace, remains text. Literal tag mentions in inline code do not suppress a later real call. Incomplete blocks and originally empty examples stay intact; parameter data keeps its original values.
Why it's needed
A complete call in the XML format taught to Qwen Coder currently remains assistant text and never executes its tool. A matched built-CLI reproduction confirms zero reads on main; the candidate executes the real read once and sends its result on the next model request.
Reviewer Test Plan
How to verify
Evidence (Before & After)
The final built CLI passed two scenarios: a literal
<example>in inline code followed by a real function-style call executes one read and carries its matching receipt into the next primary request; a genuine unclosed example executes zero tools and preserves the complete text. Four direct-source boundary checks passed. The final two-file unit run executed 624 tests: 622 passed, including all 71 parser tests, while two LlmChat cases timed out without parser assertion failures. A serial single-worker rerun of the four cases matching those two test names passed, with unchanged timeout limits. The original 622/624 full-suite result and failure logs remain preserved. Affected build/typecheck, bundle, lint and formatting passed. Earlier matched main/candidate CLI evidence separately established the zero-read baseline, native-call authority, failed-stream protection, fenced documentation, partial writes and attribute-bearing examples; those runs used preceding artifacts.Current-head actual tmux/TUI verification at
2a064ee999a4confirms one real read, one matching receipt and the final answer. The raw XML remains visible in the terminal; persisted model text and rendered terminal text have distinct outcomes. Screenshots, full readable capture and bilingual test report.Tested on
Environment (optional)
Controlled loopback provider, isolated session and safe fixture workspace. No customer traces, credentials or private endpoints are included.
Risk & Scope
Existing prose, fence, finish-reason and native-call guards remain in place. XML already rendered during streaming, other notation families, truncated or namespaced markup, scalar schema coercion and provider capability policy are outside this patch. No protocol or configuration migration.
Ready for review. Current-head terminal acceptance verifies tool recovery; removing already-rendered XML remains outside this patch. Outstanding GitHub checks and maintainer review remain visible in the PR.
Linked Issues
Refs #10692. The broader malformed-output issue remains open.
中文说明
本 PR 的改动
通过既有工具调用 fallback 恢复完整 function/parameter XML,包含带外层、裸格式和跨 chunk 的响应。只删除实际恢复的范围。围栏或显式 example 中的文档(含 example 属性及标签空白)保留为文本。inline code 中字面提到的标签不会阻止后续真实调用。不完整块和原有空示例保持完整;参数数据保留原值。
为什么需要
按 Qwen Coder 已教授 XML 格式生成的完整调用,目前仍作为助手文本保存,不执行工具。真实构建 CLI 的配对复现确认主分支读取次数为零;候选实际读取一次,并在下一次模型请求发送结果。
评审验证计划
如何验证
前后证据
最终构建 CLI 的两个场景通过:inline code 中字面
<example>后的合法 function 风格调用真实读取一次,并将匹配回执带入下一次主请求;真正未闭合 example 不执行工具,完整原文保留。四项直接源码边界检查通过。最终两文件单测执行 624 项,622 项通过(包含全部 71 项 parser 测试),两项 LlmChat 用例超时,没有 parser 断言失败。两个失败名对应的四项用例在串行、单 worker 的隔离复跑中全部通过,超时限值未调整;原 622/624 整组结果和失败日志保留。受影响包 build/typecheck、bundle、lint 和格式检查通过。较早的主分支/候选配对 CLI 证据另行确认了零读取基线、原生调用优先、失败流保护、围栏文档、半截写入及带属性示例;这些运行对应前一份产物。测试平台
环境
受控本地 provider、隔离会话和安全 fixture 工作区。不包含客户 trace、凭据或私有服务地址。
风险与范围
保留既有说明文本、围栏、finish-reason 和原生调用保护。实时流已显示的 XML、其他格式族、截断或带命名空间的 markup、标量按 schema 转型,以及 provider 能力策略不在本补丁范围内。无协议或配置迁移。
当前已转为 Ready。最新提交
2a064ee999a4的实际 tmux/TUI 流程确认1次真实读取、1次匹配回执及最终回答;持久化模型文本已移除恢复部分,但终端仍显示原始XML。首次脚本的后台请求误分类失败完整保留,同一记录离线复验11/11通过,没有重跑产品。截图、完整可读capture和双语测试报告已发布;GitHub检查和维护者审查仍按PR页面状态跟进。关联 Issue
Refs #10692。更广的异常输出 issue 保持打开。