Skip to content

fix(web): fall back to a non-browser fetch on Cloudflare bot challenges #89

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Sarath1018 wants to merge 2 commits into
base: feature/darwin
Choose a base branch
from
Open

fix(web): fall back to a non-browser fetch on Cloudflare bot challenges #89

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c69183c
fix(web): fall back to a non-browser fetch on Cloudflare bot challenges
Sarath1018 Sep 23, 2026
a180c1b
fix(web): always stop Playwright so a failed crawl can't poison the w…
Sarath1018 Sep 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
fix(web): always stop Playwright so a failed crawl can't poison the w… 
…orker

load_from_state raised "No valid pages" / "Skipped indexing ..." without
stopping Playwright, and also leaked an instance restarted after a batch
boundary when the remaining pages produced no documents. A sync_playwright
instance that isn't stopped leaves its asyncio loop marked running on the
thread, so every later start_playwright() on that thread fails with "It
looks like you are using Playwright Sync API inside the asyncio loop".

Dask workers run --nthreads=1, so one leaked instance broke every later web
connector on that pod until it restarted: yesterday's w3.org Cloudflare 403
failures (attempts 583631/583632) poisoned both workers, and today's
attempts 584050/584051 failed immediately.

Wrap the crawl in try/finally so the current instance is always stopped
(stop() is idempotent), and stop the instance in start_playwright if
browser launch / context / OAuth setup fails after start().

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
  • Loading branch information
@Sarath1018 @claude
Sarath1018 and claude committed Sep 23, 2026
commit a180c1b61193fe1dca7905fdeaec4c0c28023d50
Loading
Loading