mirror of
https://github.com/dgtlmoon/changedetection.io.git
synced 2026-09-19 11:55:59 +00:00
dgtlmoonandClaude Opus 5
8d938b5966
* Browser fetchers - Judge a fetch on the document we end up extracting, not the first navigation
The goal is to compare the text of the page the browser lands on, even when the site navigates
again after the first response. Both fetchers were bound to the first navigation, which shows up
as two different bugs:
1. pyppeteer hangs until the hard processing timeout. Its navigation watcher is bound to the
loaderId of the navigation it started, so when the site replaces that document the 'load' it
waits for never arrives for that loaderId. With timeout=0 and setDefaultNavigationTimeout(0)
there is nothing to break the wait, so goto() blocks until
PUPPETEER_MAX_PROCESSING_TIMEOUT_SECONDS (180s) kills the fetch and the watch records an empty
xpath_data - while the browser is sitting on a fully loaded page. Traced on slated.com:
0.24s goto start
0.74s main frame networkIdle loaderId=A0E0E0B2 <- never gets 'load'
2.72s main frame init loaderId=56D2B5EF <- re-navigated to get.slated.com
3.49s main frame load loaderId=56D2B5EF <- fires for the new document
25.2s goto still hanging, frame._loaderId is now 56D2B5EF
Now the navigation races goto() against the main frame firing 'load', bounded by
BROWSER_NAVIGATION_TIMEOUT_SECONDS (default 30), and falls back to the document we can see.
slated.com / getastra.com / addupsolutions.com went from a 180s timeout with no content to
200 with full content in 5-35s.
2. Both fetchers reported the status of the interstitial. A site that gates unseen visitors with
an error status plus a client-side redirect (reported against fotokoch.de: 503 + meta refresh,
then a 200 with the real page) failed the watch even though the content was present, and the
only workaround was ignore_status_codes, which also hides genuine 404s and 500s forever.
The fetchers now keep the latest main-frame document response and judge on that - the refresh
lands during the existing extra_wait, so the 200 wins.
Playwright also waits for a settled load state before extracting, which is what produced
"Execution context was destroyed, most likely because of a navigation" when the refresh collided
with extraction.
The navigation-response tracker is installed once per page and shared between the fetcher and
action_goto_url() rather than each navigation adding its own listener - 'response' fires once per
HTTP response, hundreds of times on a heavy page, so the callbacks are worth not duplicating.
Verified one listener remains after an install plus four navigations.
Selenium is unaffected either way - it hardcodes status_code = 200 because WebDriver cannot see
the HTTP status.
Tested: new test_renavigation.py covers the interstitial case end to end and was checked to fail
without the fix and pass with it, on both fetchers. The test endpoint gates on last-seen time
rather than a hit count, because a counter lets the second check see a clean 200 and the test
then passes without the fix. Full browser suite 11 passed on playwright and on pyppeteer, 494
unit tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Puppeteer fetcher - One content-ready deadline instead of a stopLoading watchdog per frame event
Page.stopLoading is what stops a page that would otherwise load forever waiting on a subresource
that never answers, so that we can still screenshot and scrape what rendered. That intent was
right, but it was implemented as a fire-and-forget task armed by every frame event, which measured
on a single fetch of an iframe-heavy page came to:
14 watchdog tasks spawned
11 page-wide Page.stopLoading calls
3 tasks outliving the fetch and firing against a closed page
Page.stopLoading takes no frame or loader argument - it is the Stop button, and it stops the whole
page. Verified directly: one call stopped a pending main frame and a pending iframe in the same
instant. So the other 10 calls were redundant, and because they landed at arbitrary later times
they could stop a *subsequent* navigation we actually wanted - which is the likeliest reason the
same URL fetched in 8s on one run and 35s on the next.
Replaced with a single deadline, awaited inline so nothing can outlive the fetch (there is no
create_task left in this file at all):
navigate (bounded) -> wait the configured delay -> Page.stopLoading -> extract
The delay is measured from when navigation finished, not from when it started. Anchoring it to the
start would quietly rob a slow-loading page of its settle time, and letting JS-rendered content
appear after load is the whole point of the setting. Verified with a server that takes 5s to answer
and renders via JS 2s after load: total 9.6s for a 4s delay, and the late content is captured.
Because a page is never reliably "finished" - many sites navigate as part of their normal design -
the delay restarts when the MAIN frame replaces its document, so a redirect or interstitial gets
the same settle time the first document got. Iframes do not restart it, and it is capped by
BROWSER_CONTENT_READY_MAX_RESETS (default 2).
Only the existing "wait n seconds before extracting text" stays user-facing;
BROWSER_NAVIGATION_TIMEOUT_SECONDS is a safety net with a sane default rather than a second knob
for users to reason about. This matches what other scrapers do: bound the navigation, do not fail
when it times out, settle, then extract.
Timings are also more predictable now - the four reported URLs went from 5-35s of variance to
4.6-7.2s at a 3s delay, all with full content and a 200.
Tested: 11 passed pyppeteer browser suite, 7 passed playwright, 494 unit + llm.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
461 lines
16 KiB
Python
461 lines
16 KiB
Python
#!/usr/bin/env python3
|
|
from operator import truediv
|
|
|
|
from flask import make_response, request, current_app
|
|
from flask import url_for
|
|
import logging
|
|
import time
|
|
import os
|
|
import threading
|
|
|
|
# Thread-safe global storage for test endpoint content
|
|
# Avoids filesystem cache issues in parallel tests
|
|
_test_endpoint_content_lock = threading.Lock()
|
|
_test_endpoint_content = {}
|
|
|
|
def write_test_file_and_sync(filepath, content, mode='w'):
|
|
"""
|
|
Write test data to file and ensure it's synced to disk.
|
|
Also stores in thread-safe global dict to bypass filesystem cache.
|
|
|
|
Critical for parallel tests where workers may read files immediately after write.
|
|
Without fsync(), data may still be in OS buffers when workers try to read,
|
|
causing race conditions where old data is seen.
|
|
|
|
Args:
|
|
filepath: Full path to file
|
|
content: Content to write (str or bytes)
|
|
mode: File mode ('w' for text, 'wb' for binary)
|
|
"""
|
|
# Convert content to bytes if needed
|
|
if isinstance(content, str):
|
|
content_bytes = content.encode('utf-8')
|
|
else:
|
|
content_bytes = content
|
|
|
|
# Store in thread-safe global dict for instant access
|
|
with _test_endpoint_content_lock:
|
|
_test_endpoint_content[os.path.basename(filepath)] = content_bytes
|
|
|
|
# Also write to file for compatibility
|
|
with open(filepath, mode) as f:
|
|
f.write(content)
|
|
f.flush() # Flush Python buffer to OS
|
|
os.fsync(f.fileno()) # Force OS to write to disk
|
|
|
|
def set_original_response(datastore_path, extra_title=''):
|
|
test_return_data = f"""<html>
|
|
<head><title>head title{extra_title}</title></head>
|
|
<body>
|
|
Some initial text<br>
|
|
<p>Which is across multiple lines</p>
|
|
<br>
|
|
So let's see what happens. <br>
|
|
<span class="foobar-detection" style='display:none'></span>
|
|
</body>
|
|
</html>
|
|
"""
|
|
|
|
write_test_file_and_sync(os.path.join(datastore_path, "endpoint-content.txt"), test_return_data)
|
|
return None
|
|
|
|
def set_modified_response(datastore_path):
|
|
test_return_data = """<html>
|
|
<head><title>modified head title</title></head>
|
|
<body>
|
|
Some initial text<br>
|
|
<p>which has this one new line</p>
|
|
<br>
|
|
So let's see what happens. <br>
|
|
</body>
|
|
</html>
|
|
"""
|
|
|
|
write_test_file_and_sync(os.path.join(datastore_path, "endpoint-content.txt"), test_return_data)
|
|
return None
|
|
def set_longer_modified_response(datastore_path):
|
|
test_return_data = """<html>
|
|
<head><title>modified head title</title></head>
|
|
<body>
|
|
Some initial text<br>
|
|
<p>which has this one new line</p>
|
|
<br>
|
|
So let's see what happens. <br>
|
|
So let's see what happens. <br>
|
|
So let's see what happens. <br>
|
|
So let's see what happens. <br>
|
|
</body>
|
|
</html>
|
|
"""
|
|
|
|
write_test_file_and_sync(os.path.join(datastore_path, "endpoint-content.txt"), test_return_data)
|
|
return None
|
|
|
|
def set_more_modified_response(datastore_path):
|
|
test_return_data = """<html>
|
|
<head><title>modified head title</title></head>
|
|
<body>
|
|
Some initial text<br>
|
|
<p>which has this one new line</p>
|
|
<br>
|
|
So let's see what happens. <br>
|
|
Ohh yeah awesome<br>
|
|
</body>
|
|
</html>
|
|
"""
|
|
|
|
write_test_file_and_sync(os.path.join(datastore_path, "endpoint-content.txt"), test_return_data)
|
|
return None
|
|
|
|
|
|
def set_empty_text_response(datastore_path):
|
|
test_return_data = """<html><body></body></html>"""
|
|
|
|
write_test_file_and_sync(os.path.join(datastore_path, "endpoint-content.txt"), test_return_data)
|
|
|
|
return None
|
|
|
|
def wait_for_notification_endpoint_output(datastore_path):
|
|
'''Apprise can take a few seconds to fire'''
|
|
#@todo - could check the apprise object directly instead of looking for this file
|
|
from os.path import isfile
|
|
notification_file = os.path.join(datastore_path, "notification.txt")
|
|
for i in range(1, 20):
|
|
time.sleep(1)
|
|
if isfile(notification_file):
|
|
return True
|
|
|
|
return False
|
|
|
|
# kinda funky, but works for now
|
|
def get_UUID_for_tag_name(client, name):
|
|
app_config = client.application.config.get('DATASTORE').data
|
|
for uuid, tag in app_config['settings']['application'].get('tags', {}).items():
|
|
if name == tag.get('title', '').lower().strip():
|
|
return uuid
|
|
return None
|
|
|
|
|
|
# kinda funky, but works for now
|
|
def extract_rss_token_from_UI(client):
|
|
return client.application.config.get('DATASTORE').data['settings']['application'].get('rss_access_token')
|
|
# import re
|
|
# res = client.get(
|
|
# url_for("watchlist.index"),
|
|
# )
|
|
# m = re.search('token=(.+?)"', str(res.data))
|
|
# token_key = m.group(1)
|
|
# return token_key.strip()
|
|
|
|
# kinda funky, but works for now
|
|
def extract_UUID_from_client(client):
|
|
import re
|
|
res = client.get(
|
|
url_for("watchlist.index"),
|
|
)
|
|
# <span id="api-key">{{api_key}}</span>
|
|
|
|
m = re.search('edit/(.+?)[#"]', str(res.data))
|
|
uuid = m.group(1)
|
|
return uuid.strip()
|
|
|
|
def delete_all_watches(client=None):
|
|
wait_for_all_checks(client)
|
|
|
|
uuids = list(client.application.config.get('DATASTORE').data['watching'])
|
|
for uuid in uuids:
|
|
client.application.config.get('DATASTORE').delete(uuid)
|
|
from changedetectionio.flask_app import update_q
|
|
|
|
# Clear the queue to prevent leakage to next test
|
|
# Use clear() method to ensure both priority_items and notification_queue are drained
|
|
if hasattr(update_q, 'clear'):
|
|
update_q.clear()
|
|
else:
|
|
# Fallback for old implementation
|
|
while not update_q.empty():
|
|
try:
|
|
update_q.get_nowait()
|
|
except:
|
|
break
|
|
|
|
time.sleep(0.2)
|
|
|
|
# Delete any old watch metadata
|
|
from pathlib import Path
|
|
|
|
base_path = Path(
|
|
client.application.config.get('DATASTORE').datastore_path
|
|
).resolve()
|
|
|
|
max_depth = 2
|
|
|
|
for file in base_path.rglob("*.json"):
|
|
# Calculate depth relative to base path
|
|
depth = len(file.relative_to(base_path).parts) - 1
|
|
|
|
if depth <= max_depth and file.is_file():
|
|
file.unlink()
|
|
|
|
|
|
def wait_for_all_checks(client=None):
|
|
"""
|
|
Waits until the queue is empty and workers are idle.
|
|
Delegates to worker_pool.wait_for_all_checks for shared logic.
|
|
"""
|
|
from changedetectionio.flask_app import update_q as global_update_q
|
|
from changedetectionio import worker_pool
|
|
return worker_pool.wait_for_all_checks(global_update_q, timeout=150)
|
|
|
|
|
|
def wait_for_watch_history(client, min_history_count=2, timeout=10):
|
|
"""
|
|
Wait for watches to have sufficient history entries.
|
|
Useful after wait_for_all_checks() when you need to ensure history is populated.
|
|
|
|
Args:
|
|
client: Test client with access to datastore
|
|
min_history_count: Minimum number of history entries required
|
|
timeout: Maximum time to wait in seconds
|
|
"""
|
|
datastore = client.application.config.get('DATASTORE')
|
|
start_time = time.time()
|
|
|
|
while time.time() - start_time < timeout:
|
|
all_have_history = True
|
|
for uuid, watch in datastore.data['watching'].items():
|
|
history_count = len(watch.history.keys())
|
|
if history_count < min_history_count:
|
|
all_have_history = False
|
|
break
|
|
|
|
if all_have_history:
|
|
return True
|
|
|
|
time.sleep(0.2)
|
|
|
|
# Timeout - return False
|
|
return False
|
|
|
|
|
|
# Replaced by new_live_server_setup and calling per function scope in conftest.py
|
|
def live_server_setup(live_server):
|
|
return True
|
|
|
|
def new_live_server_setup(live_server):
|
|
|
|
@live_server.app.route('/test-random-content-endpoint')
|
|
def test_random_content_endpoint():
|
|
import secrets
|
|
return "Random content - {}\n".format(secrets.token_hex(64))
|
|
|
|
# Re-navigation gate: the first hit answers with an error status AND a client-side meta
|
|
# refresh, the second serves the real page. This mirrors sites that gate visitors they have
|
|
# not seen recently (reported against fotokoch.de, which answers 503 + meta refresh and then
|
|
# serves a 200). The browser follows the refresh, so the fetch has to be judged on the
|
|
# document we actually end up extracting rather than on the interstitial.
|
|
# Keyed on last-seen time rather than a hit count, so EVERY fresh check starts out gated -
|
|
# a counter would serve a clean 200 to the second check and let the test pass without the fix.
|
|
_interstitial_last_seen = {}
|
|
|
|
@live_server.app.route('/test-interstitial')
|
|
def test_interstitial():
|
|
key = request.args.get('key', 'default')
|
|
now = time.time()
|
|
seen_recently = (now - _interstitial_last_seen.get(key, 0)) < 10
|
|
_interstitial_last_seen[key] = now
|
|
if not seen_recently:
|
|
resp = make_response(
|
|
'<html><head><meta http-equiv="refresh" content="1"></head>'
|
|
'<body>Browser check in progress, you will be redirected</body></html>', 503)
|
|
else:
|
|
resp = make_response(
|
|
'<html><body><h1>The real page content is here</h1></body></html>', 200)
|
|
resp.headers['Content-Type'] = 'text/html'
|
|
return resp
|
|
|
|
@live_server.app.route('/test-endpoint2')
|
|
def test_endpoint2():
|
|
return "<html><body>some basic content</body></html>"
|
|
|
|
@live_server.app.route('/test-endpoint')
|
|
def test_endpoint():
|
|
# REMOVED: logger.debug() causes file locking between test process and Flask server process
|
|
# Flask server runs in separate multiprocessing.Process and inherited loguru tries to
|
|
# write to same log files, causing request handlers to block on file locks
|
|
# from loguru import logger
|
|
# logger.debug(f"/test-endpoint hit {request}")
|
|
ctype = request.args.get('content_type')
|
|
status_code = request.args.get('status_code')
|
|
content = request.args.get('content') or None
|
|
delay = int(request.args.get('delay', 0))
|
|
|
|
if delay:
|
|
time.sleep(delay)
|
|
|
|
# Used to just try to break the header detection
|
|
uppercase_headers = request.args.get('uppercase_headers')
|
|
|
|
try:
|
|
if content is not None:
|
|
resp = make_response(content, status_code)
|
|
if uppercase_headers:
|
|
ctype=ctype.upper()
|
|
resp.headers['CONTENT-TYPE'] = ctype if ctype else 'text/html'
|
|
else:
|
|
resp.headers['Content-Type'] = ctype if ctype else 'text/html'
|
|
return resp
|
|
|
|
# Check thread-safe global dict first (instant, no cache issues)
|
|
# Fall back to file if not in dict (for tests that write directly)
|
|
with _test_endpoint_content_lock:
|
|
content_data = _test_endpoint_content.get("endpoint-content.txt")
|
|
|
|
if content_data is None:
|
|
# Not in global dict, read from file
|
|
datastore_path = current_app.config.get('TEST_DATASTORE_PATH', 'test-datastore')
|
|
filepath = os.path.join(datastore_path, "endpoint-content.txt")
|
|
|
|
# REMOVED: os.sync() was blocking for many seconds during parallel tests
|
|
# With -n 6+ parallel tests, heavy I/O causes os.sync() to wait for ALL
|
|
# system writes to complete, causing "Read timed out" errors
|
|
# File writes from test code are already flushed by the time workers fetch
|
|
|
|
try:
|
|
with open(filepath, "rb") as f:
|
|
content_data = f.read()
|
|
except Exception as e:
|
|
# REMOVED: logger.error() causes file locking in multiprocess context
|
|
# Just raise the exception directly for debugging
|
|
raise
|
|
|
|
resp = make_response(content_data, status_code)
|
|
if uppercase_headers:
|
|
resp.headers['CONTENT-TYPE'] = ctype if ctype else 'text/html'
|
|
else:
|
|
resp.headers['Content-Type'] = ctype if ctype else 'text/html'
|
|
return resp
|
|
except FileNotFoundError:
|
|
return make_response('', status_code)
|
|
|
|
# Just return the headers in the request
|
|
@live_server.app.route('/test-headers')
|
|
def test_headers():
|
|
|
|
output = []
|
|
|
|
for header in request.headers:
|
|
output.append("{}:{}".format(str(header[0]), str(header[1])))
|
|
|
|
content = "\n".join(output)
|
|
|
|
resp = make_response(content, 200)
|
|
resp.headers['server'] = 'custom'
|
|
return resp
|
|
|
|
# Just return the body in the request
|
|
@live_server.app.route('/test-body', methods=['POST', 'GET'])
|
|
def test_body():
|
|
print ("TEST-BODY GOT", request.data, "returning")
|
|
return request.data
|
|
|
|
# Just return the verb in the request
|
|
@live_server.app.route('/test-method', methods=['POST', 'GET', 'PATCH'])
|
|
def test_method():
|
|
return request.method
|
|
|
|
# Where we POST to as a notification, also use a space here to test URL escaping is OK across all tests that use this. ( #2868 )
|
|
@live_server.app.route('/test_notification_endpoint', methods=['POST', 'GET'])
|
|
def test_notification_endpoint():
|
|
datastore_path = current_app.config.get('TEST_DATASTORE_PATH', 'test-datastore')
|
|
|
|
with open(os.path.join(datastore_path, "notification.txt"), "wb") as f:
|
|
# Debug method, dump all POST to file also, used to prove #65
|
|
data = request.stream.read()
|
|
if data != None:
|
|
f.write(data)
|
|
|
|
with open(os.path.join(datastore_path, "notification-url.txt"), "w") as f:
|
|
f.write(request.url)
|
|
|
|
with open(os.path.join(datastore_path, "notification-headers.txt"), "w") as f:
|
|
f.write(str(request.headers))
|
|
|
|
if request.content_type:
|
|
with open(os.path.join(datastore_path, "notification-content-type.txt"), "w") as f:
|
|
f.write(request.content_type)
|
|
|
|
print("\n>> Test notification endpoint was hit.\n", data)
|
|
|
|
content = "Text was set"
|
|
status_code = request.args.get('status_code',200)
|
|
resp = make_response(content, status_code)
|
|
return resp
|
|
|
|
# Just return the verb in the request
|
|
@live_server.app.route('/test-basicauth', methods=['GET'])
|
|
def test_basicauth_method():
|
|
auth = request.authorization
|
|
ret = " ".join([auth.username, auth.password, auth.type])
|
|
return ret
|
|
|
|
# Just return some GET var
|
|
@live_server.app.route('/test-return-query', methods=['GET'])
|
|
def test_return_query():
|
|
return request.query_string
|
|
|
|
|
|
@live_server.app.route('/endpoint-test.pdf')
|
|
def test_pdf_endpoint():
|
|
datastore_path = current_app.config.get('TEST_DATASTORE_PATH', 'test-datastore')
|
|
|
|
# Force filesystem sync before reading to ensure fresh data
|
|
try:
|
|
os.sync()
|
|
except (AttributeError, PermissionError):
|
|
pass
|
|
|
|
# Tried using a global var here but didn't seem to work, so reading from a file instead.
|
|
with open(os.path.join(datastore_path, "endpoint-test.pdf"), "rb") as f:
|
|
resp = make_response(f.read(), 200)
|
|
resp.headers['Content-Type'] = 'application/pdf'
|
|
return resp
|
|
|
|
@live_server.app.route('/test-interactive-html-endpoint')
|
|
def test_interactive_html_endpoint():
|
|
header_text=""
|
|
for k,v in request.headers.items():
|
|
header_text += f"{k}: {v}<br>"
|
|
|
|
resp = make_response(f"""
|
|
<html>
|
|
<body>
|
|
Primitive JS check for <pre>changedetectionio/tests/visualselector/test_fetch_data.py</pre>
|
|
<p id="remove">This text should be removed</p>
|
|
<form onsubmit="event.preventDefault();">
|
|
<!-- obfuscated text so that we dont accidentally get a false positive due to conversion of the source :) --->
|
|
<button name="test-button" onclick="
|
|
getElementById('remove').remove();
|
|
getElementById('some-content').innerHTML = atob('SSBzbWVsbCBKYXZhU2NyaXB0IGJlY2F1c2UgdGhlIGJ1dHRvbiB3YXMgcHJlc3NlZCE=');
|
|
getElementById('reflect-text').innerHTML = getElementById('test-input-text').value;
|
|
">Click here</button>
|
|
|
|
<div id="some-content"></div>
|
|
|
|
<pre>
|
|
{header_text.lower()}
|
|
</pre>
|
|
|
|
<br>
|
|
<!-- used for testing that the jinja2 compiled here --->
|
|
<input type="text" value="" id="test-input-text" /><br>
|
|
<div id="reflect-text">Waiting to reflect text from #test-input-text here</div>
|
|
</form>
|
|
|
|
</body>
|
|
</html>""", 200)
|
|
resp.headers['Content-Type'] = 'text/html'
|
|
return resp
|
|
|
|
live_server.start()
|