dgtlmoonandClaude Opus 5 b61b83c35d Fetch favicon candidates in parallel under one deadline, and stop the two ways it could hang (#4435) ...

Each candidate icon got its own fresh 2s AbortController, so the cost was n x 2s: a site
declaring five <link rel="icon"> variants spent 10s+ in here, sequentially, inside the
page - holding a browser, a worker and a proxy connection the whole time - to fetch
decoration.

Capping the total is not enough, and is a trap. Giving up early returns no icon, so
nothing is saved, favicon_is_expired() stays true, and the same cost is paid again on the
very next check - forever. Measured against a page with five hanging icons, one 404 and
one good one:

  sequential, 2s each : 10.0s, found the icon
  total cap only      :  3.0s, found NOTHING (and repeats every check)
  parallel + deadline :  3.0s, found the icon

So: fetch them all concurrently under one shared 3s AbortController and take the first
success in preference order - the array is already sorted largest-first then
apple-touch-icon, so this still returns the preferred icon rather than merely the
quickest to answer.

Two hangs fixed while in here:

- clearTimeout() fired before `await resp.blob()`, leaving the body read unguarded. The
  shared signal now covers it (aborting a signal errors the body stream too), so a slow
  or never-ending body is bounded like the headers.

- the FileReader promise had no reject path and resolved only from onloadend, reading
  reader.result unguarded. A FileReader failure threw inside the callback and left the
  promise permanently pending, with the timer already cleared - the favicon fetch then
  hung forever with nothing to stop it. It now always resolves.

Also skips an oversized icon from Content-Length before pulling its body down the wire,
where the server declares it.

Unchanged: the candidate collection and sort, the data: URI shortcut, the 1MB limit that
matches bump_favicon(). Verified no regression on the ordinary paths - a page with one
working icon still returns it in 0.0s, and a page with no <link> still falls back to
/favicon.ico.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-09-15 09:43:43 +02:00

..

exceptions

…

res

Fetch favicon candidates in parallel under one deadline, and stop the two ways it could hang (#4435)

2026-09-15 09:43:43 +02:00

__init__.py

…

base.py

Worker id/available fetcher (#4415)

2026-09-12 16:04:39 +02:00

playwright.py

Browser fetchers - Follow re-navigation, and one content-ready deadline (re-targets #4421 + #4422 at master) (#4426)

2026-09-12 17:08:28 +02:00

puppeteer.py

Puppeteer fetcher - Re-navigation cap must not mean "extract right now" (#4439)

2026-09-14 20:31:48 +02:00

requests.py

…

screenshot_handler.py

…

webdriver_selenium.py

…