An "extra browser" was a name + a ws(s):// endpoint in settings.requests.extra_browsers,
selected by a watch as the magic string 'extra_browser_<name>'. That string resolved to
html_webdriver plus a custom connection URL, which meant the protocol the endpoint was
spoken to came from env vars rather than from the entry: CDP over a WebSocket with
PLAYWRIGHT_DRIVER_URL set, CDP via pyppeteer with FAST_PUPPETEER_CHROME_FETCHER, and the
W3C WebDriver protocol over HTTP on a Selenium-only install - where a wss:// URL cannot
work at all. The form only ever accepted ws:// / wss://, so the feature was silently
broken on exactly the installs that could not honour it.
So it becomes an engine, html_external_cdp, which pins the protocol: a subclass of the
Playwright fetcher that takes its endpoint from the watch's browser config
(FetcherConfig.connection_url) instead of the environment. It is base-only
(ready_to_use=False) because an endpoint is required, so each endpoint is one browser
config ("variation") on the Browsers page - which is what the old settings list was.
update_36 migrates each extra_browsers row to such a variation, keyed by the SAME
'extra_browser_<name>' string watches already hold, so no watch, group override, API value
or global default needs rewriting; the legacy selector simply becomes a real browser-config
id. A row whose endpoint the model rejects is logged and skipped rather than taking the
update chain, and with it startup, down.
Knock-on cleanups, all of which delete a special case rather than add one:
- The proxy opt-out for custom endpoints is now Fetcher.ignores_proxy_setting, asked of
the engine, instead of a string-prefix test in call_browser().
- A live browser-steps / visual-selector session asks the engine where to connect
(Fetcher.browser_steps_connection_url, overridden by html_external_cdp) and refuses an
engine whose supports_browser_steps is False, instead of reading the env var itself and
silently stepping a browser the watch does not check with. That refusal is real: on a
Selenium install html_webdriver cannot drive a live session.
- is_valid_browser_selector() answers "may a watch store this in fetch_backend?" in one
place; the API (create/update/import), the quick-add form validator and the bulk "set
browser" operation each had their own copy, which is how they came to disagree about
whether a browser-config id was acceptable.
- api-spec.yaml's fetch_backend pattern enumerated extra_browser_* while rejecting
browser-config ids and every engine newer than html_webdriver. Valid values are
per-install, so the schema now bounds the string and the handlers do the real check.
- html_external_cdp registers unconditionally (unlike html_playwright_builtin): migrated
configs name it, so it must resolve even without the playwright library, or those
watches would quietly fetch with the plain HTTP client. The library is imported lazily
inside run(), and an unavailable engine now warns instead of falling back silently.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Set Cache-Control: no-store on dynamic responses by default
Pages with per-session content (settings, CSRF-token-bearing forms,
watch data) had no Cache-Control header, so a misconfigured CDN or
reverse proxy sitting in front of the app could cache and replay them
across requests/sessions — most commonly surfacing as "CSRF tokens do
not match" after the edge served a stale cached page. Routes that
already set their own Cache-Control (static assets, screenshots,
favicons) are left untouched.
* Adding tests and moving function
---------
Co-authored-by: dgtlmoon <dgtlmoon@gmail.com>
* fix(http): send a single Date header on werkzeug built-in server
Static resources served via werkzeug send_from_directory/send_file get a
Date header injected into the WSGI response by make_conditional()
(werkzeug/wrappers/response.py:752-757). When the app runs on Werkzeug's
built-in server -- the default path started through
socketio.run(..., allow_unsafe_werkzeug=True) in changedetectionio/__init__.py:694
and used by the docker entrypoint -- BaseHTTPRequestHandler.send_response()
(werkzeug/serving.py:271) emits its own Date header line as well, so the
wire response carries two Date headers. RFC 9110 forbids this and nginx
rejects the response with "upstream sent duplicate header line" (issue
#4299, see also #4101).
Fix: a global after_request hook pops the application-side Date copy so
only the server's single header reaches the wire. Verified safe on
gunicorn too, which also emits its own Date header.
Test: new tests/test_duplicate_date_header.py hits the live_server over
real HTTP with http.client (the Flask test client talks to the WSGI app
directly and never sees the server-added header) and asserts the Date
header appears exactly once, on the exact static resources named in the
issue. Fails on unfixed code with two identical Date lines; passes with
the fix.
Fixes#4299
* Apply suggestion from @dgtlmoon
* Tidy the #4299 Date header fix and its test
flask_app.py: the applied suggestion landed with a 3-space indent and
trailing whitespace - the latter was the only W291 in the file, which
.ruff.toml selects.
test_duplicate_date_header.py:
- drop the 10s socket wait loop, pytest-flask's live_server already
blocks until the port accepts connections
- drop the unused `app` fixture argument (live_server depends on it)
- stop hardcoding jquery-3.6.0.min.js: asserting 200 on a vendored
filename turns a jQuery bump into a failure in a file about HTTP
headers. Any send_from_directory() response exercises the same path,
so styles.css alone is enough.
Still red before the fix (two identical Date lines) and green after.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: dgtlmoon <leigh@morresi.net>
Co-authored-by: dgtlmoon <dgtlmoon@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every watch whose xPath filter used contains() started reporting "Warning, no filters were
found" on pages whose HTML plainly contained the target. 18 unrelated watches broke in the same
hour, browser and plain-requests fetchers alike, and the saved snapshot was perfect every time.
elementpath implements the XPath string functions on top of locale.strxfrm:
def contains(self, a, b): return self.strxfrm(b) in self.strxfrm(a)
Under LC_COLLATE=C, strxfrm() is the identity function and that is an ordinary substring test.
Under a real locale it returns a binary collation key, and a substring of a collation key is not
the collation key of the substring - so contains(), starts-with(), ends-with() and
substring-before/after() return false for EVERY input. Reduced to one line, no document needed:
LC_COLLATE=C contains("xx month xx", "month") -> True
LC_COLLATE=en_US.UTF-8 contains("xx month xx", "month") -> False
Name tests, axes and '=' are untouched, which is exactly why it read as "did the page change
layout?" - //div kept working while //div[contains(.,"month")] returned nothing.
No code change caused this. Generating the image's locales (#4429) made ENV LC_ALL=en_US.UTF-8
satisfiable for the first time; flask_app's setlocale(LC_ALL, ...) had been raising locale.Error
and leaving us in C, and once it succeeded it took LC_COLLATE with it. That is why the bug
survived a bisect to 0.60.2 and why reinstalling the exact pip set from a working install did not
shift it - it could only be found by diffing the two containers. Same image base, same Python
3.11.16, lxml 6.1.3, libxml2 2.14.6, elementpath 5.1.1, same HTML:
good-old 0.60.4 setlocale FAILED: unsupported locale setting 67 matches
bad-new 0.60.5 setlocale en_US.UTF-8 0 matches
Fixed in both places, because either alone leaves a hole:
- flask_app sets LC_CTYPE/LC_NUMERIC/LC_MONETARY/LC_TIME individually instead of LC_ALL. This
block exists to make prices render correctly and it still does - 1234567 is still "1,234,567"
- it just no longer touches collation.
- html_tools.xpath_filter() pins the Unicode codepoint collation per evaluation, so a filter
means the same thing whatever an operator puts in LANG/LC_ALL, and does not depend on a
distant module's locale bookkeeping. forms.py's XPath validation pins it too, so validation
cannot accept an expression that then behaves differently at check time.
Per XPath 3.1 the default collation is codepoint and must not consult LC_COLLATE, so the
underlying behaviour is arguably an elementpath bug; the pin above holds regardless.
Verified end to end inside the failing container: LC_COLLATE=C, LC_NUMERIC=en_US.UTF-8,
thousands separators intact, and the reported filter back from 0 to 1190797 chars of output.
Tested: new unit test covers contains/starts-with/ends-with under a UTF-8 collation and was
checked to fail without the fix; 325 unit tests pass.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Scheduler+API Bug - if an invalid timezone was set (through edit of watch or API) it could have crashed the scheduler, Added `timezone` to the official API docs
* adding missing files
* No need to add imported items to the check queue, the scheduler will do this #3762
* Tests - Faster recheck/reschedule loop under pytest environment
* More wait time under test
* Bunch up some tests a little
* fix typo
* woops
* If they want to queue one thats already running, thats up to them.
* WIP
* Fixing queue limit size
* Increase max queue size and many CPU performance fixes
Multi-language / Translations Support (#3696)
- Complete internationalization system implemented
- Support for 7 languages: Czech (cs), German (de), French (fr), Italian (it), Korean (ko), Chinese Simplified (zh), Chinese Traditional (zh_TW)
- Language selector with localized flags and theming
- Flash message translations
- Multiple translation fixes and improvements across all languages
- Language setting preserved across redirects
Pluggable Content Fetchers (#3653)
- New architecture for extensible content fetcher system
- Allows custom fetcher implementations
Image / Screenshot Comparison Processor (#3680)
- New processor for visual change detection (disabled for this release)
- Supporting CSS/JS infrastructure added
UI Improvements
Design & Layout
- Auto-generated tag color schemes
- Simplified login form styling
- Removed hard-coded CSS, moved to SCSS variables
- Tag UI cleanup and improvements
- Automatic tab wrapper functionality
- Menu refactoring for better organization
- Cleanup of offset settings
- Hide sticky tabs on narrow viewports
- Improved responsive layout (#3702)
User Experience
- Modal alerts/confirmations on delete/clear operations (#3693, #3598, #3382)
- Auto-add https:// to URLs in quickwatch form if not present
- Better redirect handling on login (#3699)
- 'Recheck all' now returns to correct group/tag (#3673)
- Language set redirect keeps hash fragment
- More friendly human-readable text throughout UI
Performance & Reliability
Scheduler & Processing
- Soft delays instead of blocking time.sleep() calls (#3710)
- More resilient handling of same UUID being processed (#3700)
- Better Puppeteer timeout handling
- Improved Puppeteer shutdown/cleanup (#3692)
- Requests cleanup now properly async
History & Rendering
- Faster server-side "difference" rendering on History page (#3442)
- Show ignored/triggered rows in history
- API: Retry watch data if watch dict changed (more reliable)
API Improvements
- Watch get endpoint: retry mechanism for changed watch data
- WatchHistoryDiff API endpoint includes extra format args (#3703)
Testing Improvements
- Replace time.sleep with wait_for_notification_endpoint_output (#3716)
- Test for mode switching (#3701)
- Test for #3720 added (#3725)
- Extract-text difference test fixes
- Improved dev workflow
Bug Fixes
- Notification error text output (#3672, #3669, #3280)
- HTML validation fixes (#3704)
- Template discovery path fixes
- Notification debug log now uses system locale for dates/times
- Puppeteer spelling mistake in log output
- Recalculation on anchor change
- Queue bubble update disabled temporarily
Dependency Updates
- beautifulsoup4 updated (#3724)
- psutil 7.1.0 → 7.2.1 (#3723)
- python-engineio ~=4.12.3 → ~=4.13.0 (#3707)
- python-socketio ~=5.14.3 → ~=5.16.0 (#3706)
- flask-socketio ~=5.5.1 → ~=5.6.0 (#3691)
- brotli ~=1.1 → ~=1.2 (#3687)
- lxml updated (#3590)
- pytest ~=7.2 → ~=9.0 (#3676)
- jsonschema ~=4.0 → ~=4.25 (#3618)
- pluggy ~=1.5 → ~=1.6 (#3616)
- cryptography 44.0.1 → 46.0.3 (security) (#3589)
Documentation
- README updated with viewport size setup information
Development Infrastructure
- Dev container only built on dev branch
- Improved dev workflow tooling
