Skip to content
This repository was archived by the owner on Sep 23, 2026. It is now read-only.

fix(fetch): suppress duplicated extracted comment text - #1863

Closed
SherlockShemol wants to merge 1 commit into
MoonshotAI:mainfrom
SherlockShemol:fix/fetch-url-duplicate-comments
Closed

SherlockShemol wants to merge 1 commit into
MoonshotAI:mainfrom
SherlockShemol:fix/fetch-url-duplicate-comments

Conversation

@SherlockShemol

@SherlockShemol SherlockShemol commented Apr 13, 2026 •

Copy link
Copy Markdown

Summary

  • switch the HTML extraction path in FetchURL to inspect Trafilatura's main text and comments separately
  • suppress extracted comments when they normalize to the same content as the main body
  • add a regression test for duplicated GitHub issue extraction output

Root cause

trafilatura with include_comments=True can misclassify GitHub issue body content as comments and append a second malformed copy of the same text.

For pages like https://github.com/MoonshotAI/Moonlight/issues/4, this caused FetchURL to return the issue body twice:

  • once as the normal extracted main content
  • once again as a malformed "comment" block with missing spacing around inline code

Testing

  • uv run pytest tests/tools/test_fetch_url.py -q
  • uv run pytest tests/tools/test_tool_descriptions.py -q
  • uv run pytest tests/tools/test_tool_schemas.py -q

Related issue

Notes

  • make test still reports an unrelated existing failure in tests/e2e/test_shell_modal_e2e.py::test_approval_consecutive_across_turns
  • the fetch_url regression covered by this change passes

Open with Devin

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 4 additional findings.

Open in Devin Review

chatgpt-codex-connector[bot]

This comment was marked as resolved.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FetchURL duplicates GitHub issue body when comments extraction misfires

1 participant