mirror of
https://github.com/dgtlmoon/changedetection.io.git
synced 2026-10-11 14:31:25 +00:00
dgtlmoonandClaude Opus 5
e93e45d7cf
The goal is to compare the text of the page the browser lands on, even when the site navigates
again after the first response. Both fetchers were bound to the first navigation, which shows up
as two different bugs:
1. pyppeteer hangs until the hard processing timeout. Its navigation watcher is bound to the
loaderId of the navigation it started, so when the site replaces that document the 'load' it
waits for never arrives for that loaderId. With timeout=0 and setDefaultNavigationTimeout(0)
there is nothing to break the wait, so goto() blocks until
PUPPETEER_MAX_PROCESSING_TIMEOUT_SECONDS (180s) kills the fetch and the watch records an empty
xpath_data - while the browser is sitting on a fully loaded page. Traced on slated.com:
0.24s goto start
0.74s main frame networkIdle loaderId=A0E0E0B2 <- never gets 'load'
2.72s main frame init loaderId=56D2B5EF <- re-navigated to get.slated.com
3.49s main frame load loaderId=56D2B5EF <- fires for the new document
25.2s goto still hanging, frame._loaderId is now 56D2B5EF
Now the navigation races goto() against the main frame firing 'load', bounded by
BROWSER_NAVIGATION_TIMEOUT_SECONDS (default 30), and falls back to the document we can see.
slated.com / getastra.com / addupsolutions.com went from a 180s timeout with no content to
200 with full content in 5-35s.
2. Both fetchers reported the status of the interstitial. A site that gates unseen visitors with
an error status plus a client-side redirect (reported against fotokoch.de: 503 + meta refresh,
then a 200 with the real page) failed the watch even though the content was present, and the
only workaround was ignore_status_codes, which also hides genuine 404s and 500s forever.
The fetchers now keep the latest main-frame document response and judge on that - the refresh
lands during the existing extra_wait, so the 200 wins.
Playwright also waits for a settled load state before extracting, which is what produced
"Execution context was destroyed, most likely because of a navigation" when the refresh collided
with extraction.
The navigation-response tracker is installed once per page and shared between the fetcher and
action_goto_url() rather than each navigation adding its own listener - 'response' fires once per
HTTP response, hundreds of times on a heavy page, so the callbacks are worth not duplicating.
Verified one listener remains after an install plus four navigations.
Selenium is unaffected either way - it hardcodes status_code = 200 because WebDriver cannot see
the HTTP status.
Tested: new test_renavigation.py covers the interstitial case end to end and was checked to fail
without the fix and pass with it, on both fetchers. The test endpoint gates on last-seen time
rather than a hit count, because a counter lets the second check see a clean 200 and the test
then passes without the fix. Full browser suite 11 passed on playwright and on pyppeteer, 494
unit tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>