Skip to content

fix(core): use UTF-8 offsets for web-fetch citations - #29440

Closed
WenJing95 wants to merge 1 commit into
google-gemini:mainfrom
WenJing95:fix/web-fetch-utf8-citations
Closed

WenJing95 wants to merge 1 commit into
google-gemini:mainfrom
WenJing95:fix/web-fetch-utf8-citations

Conversation

@WenJing95

Copy link
Copy Markdown

Summary

Fix misplaced citations in web-fetch for non-ASCII responses by handling UTF-8 byte offsets properly (matching the existing web-search logic). Added regression tests for multibyte text, emoji, and out-of-order citations.

Related Issues

Fixes #29039

How to Validate

npm run generate
npm exec --workspace @google/gemini-cli-core -- vitest run src/tools/web-fetch.test.ts src/tools/web-search.test.ts --coverage.enabled=false
npm run build
npm run typecheck
npm run lint

Tool tests, workspace build, typecheck, and lint pass locally.

Pre-Merge Checklist

  • Added/updated tests (if needed)
  • Windows
  • npm run

@WenJing95
WenJing95 requested a review from a team as a code owner September 21, 2026 23:36
@google-cla

google-cla Bot commented Sep 21, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses an issue where citations were being incorrectly placed in web-fetch responses containing non-ASCII characters. By transitioning from character-based splitting to byte-level manipulation using TextEncoder and TextDecoder, the tool now correctly aligns with the grounding metadata's UTF-8 byte indices, consistent with existing web-search implementation.

Highlights

  • UTF-8 Byte Offset Handling: Updated the citation insertion logic in web-fetch to use UTF-8 byte offsets instead of character indices, ensuring correct placement for non-ASCII characters like emojis and multi-byte scripts.
  • Regression Testing: Added comprehensive test cases covering ASCII, Chinese, Japanese, and emoji-containing text to verify citation placement accuracy.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@github-actions github-actions Bot added the size/m A medium sized PR label Sep 21, 2026
@github-actions

Copy link
Copy Markdown

📊 PR Size: size/M

  • Lines changed: 99
  • Additions: +94
  • Deletions: -5
  • Files changed: 2

@github-actions

Copy link
Copy Markdown

🛑 Action Required: Evaluation Approval

Steering changes have been detected in this PR. To prevent regressions, a maintainer must approve the evaluation run before this PR can be merged.

Maintainers:

  1. Go to the Workflow Run Summary.
  2. Click the yellow 'Review deployments' button.
  3. Select the 'eval-gate' environment and click 'Approve'.

Once approved, the evaluation results will be posted here automatically.

@gemini-cli gemini-cli Bot added the area/agent Issues related to Core Agent, Tools, Memory, Sub-Agents, Hooks, Agent Quality label Sep 21, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the citation insertion logic in WebFetchTool to handle UTF-8 byte offsets correctly. Instead of splitting the response text by characters, it now uses TextEncoder and TextDecoder to manipulate the raw byte array, ensuring accurate placement of citation markers for multi-byte characters like Chinese, Japanese, and emojis. Additionally, comprehensive unit tests have been added to verify this behavior across various text encodings. There are no review comments, and I have no additional feedback to provide.

@gemini-cli

gemini-cli Bot commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

Hi there! Thank you for your interest in contributing to Gemini CLI.

To ensure we maintain high code quality and focus on our prioritized roadmap, we only guarantee review and consideration of pull requests for issues that are explicitly labeled as 'help wanted'.

This PR will be closed in 7 days if it remains without that designation. We encourage you to find and contribute to existing 'help wanted' issues in our backlog! Thank you for your understanding.

@gemini-cli

gemini-cli Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

This pull request is being closed as it has been open for 14 days without a 'help wanted' designation. We encourage you to find and contribute to existing 'help wanted' issues in our backlog! Thank you for your understanding.

This branch is waiting to be deployed

1 waiting deployment
eval-gate — a1889f49 Waiting Sep 21, 2026 by WenJing95 via Evaluate Steering & Regressions #2034
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/agent Issues related to Core Agent, Tools, Memory, Sub-Agents, Hooks, Agent Quality size/m A medium sized PR status/pr-nudge-sent

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: web-fetch grounding citation markers inserted at wrong positions (UTF-16 splice of UTF-8 byte offsets)

1 participant