dgtlmoon and Claude Opus 5
ba2280ba81
External CDP Browser engine - replaces the extra_browser_* hackery
...
An "extra browser" was a name + a ws(s):// endpoint in settings.requests.extra_browsers,
selected by a watch as the magic string 'extra_browser_<name>'. That string resolved to
html_webdriver plus a custom connection URL, which meant the protocol the endpoint was
spoken to came from env vars rather than from the entry: CDP over a WebSocket with
PLAYWRIGHT_DRIVER_URL set, CDP via pyppeteer with FAST_PUPPETEER_CHROME_FETCHER, and the
W3C WebDriver protocol over HTTP on a Selenium-only install - where a wss:// URL cannot
work at all. The form only ever accepted ws:// / wss://, so the feature was silently
broken on exactly the installs that could not honour it.
So it becomes an engine, html_external_cdp, which pins the protocol: a subclass of the
Playwright fetcher that takes its endpoint from the watch's browser config
(FetcherConfig.connection_url) instead of the environment. It is base-only
(ready_to_use=False) because an endpoint is required, so each endpoint is one browser
config ("variation") on the Browsers page - which is what the old settings list was.
update_36 migrates each extra_browsers row to such a variation, keyed by the SAME
'extra_browser_<name>' string watches already hold, so no watch, group override, API value
or global default needs rewriting; the legacy selector simply becomes a real browser-config
id. A row whose endpoint the model rejects is logged and skipped rather than taking the
update chain, and with it startup, down.
Knock-on cleanups, all of which delete a special case rather than add one:
- The proxy opt-out for custom endpoints is now Fetcher.ignores_proxy_setting, asked of
the engine, instead of a string-prefix test in call_browser().
- A live browser-steps / visual-selector session asks the engine where to connect
(Fetcher.browser_steps_connection_url, overridden by html_external_cdp) and refuses an
engine whose supports_browser_steps is False, instead of reading the env var itself and
silently stepping a browser the watch does not check with. That refusal is real: on a
Selenium install html_webdriver cannot drive a live session.
- is_valid_browser_selector() answers "may a watch store this in fetch_backend?" in one
place; the API (create/update/import), the quick-add form validator and the bulk "set
browser" operation each had their own copy, which is how they came to disagree about
whether a browser-config id was acceptable.
- api-spec.yaml's fetch_backend pattern enumerated extra_browser_* while rejecting
browser-config ids and every engine newer than html_webdriver. Valid values are
per-install, so the schema now bounds the string and the handlers do the real check.
- html_external_cdp registers unconditionally (unlike html_playwright_builtin): migrated
configs name it, so it must resolve even without the playwright library, or those
watches would quietly fetch with the plain HTTP client. The library is imported lazily
inside run(), and an unavailable engine now warns instead of falling back silently.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-17 19:39:16 +02:00
dgtlmoon
00a00b664f
Merge branch 'master' into browser-ui
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Build distribution 📦 (push) Canceled after 0s
ChangeDetection.io Container Build Test / Build linux/amd64 (alpine) (push) Canceled after 0s
ChangeDetection.io Container Build Test / Build linux/arm64 (alpine) (push) Canceled after 0s
ChangeDetection.io Container Build Test / Build linux/amd64 (main) (push) Canceled after 0s
ChangeDetection.io Container Build Test / Build linux/arm/v7 (main) (push) Canceled after 0s
ChangeDetection.io Container Build Test / Build linux/arm/v8 (main) (push) Canceled after 0s
ChangeDetection.io Container Build Test / Build linux/arm64 (main) (push) Canceled after 0s
ChangeDetection.io App Test / lint-code (push) Canceled after 0s
ChangeDetection.io App Test / lint-translations (push) Canceled after 0s
ChangeDetection.io App Test / lint-template-i18n (push) Canceled after 0s
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Test the built package works basically. (push) Canceled after 0s
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Publish Python 🐍 distribution 📦 to PyPI (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-11 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-12 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-13 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-14 (push) Canceled after 0s
2026-09-17 17:26:16 +02:00
dgtlmoon
3a71777499
UI - Fixing lots of actions that should be a POST style action which led to a 404
...
Also solves some of GHSA-56fq-63vj-9992 Add-watch-UI should be POST/CSRF protected
2026-09-04 12:38:20 +02:00
dgtlmoon
f42da28dab
GHSA-23mp-8222-96fr - reflected XSS in /diff/<uuid>/download-patch via from_version - Using built-in plain text response instead. Reported-by: Mayssare Amakhtari (@cy3erm)
...
Build and push containers / metadata (push) Canceled after 0s
Build and push containers / build-push-containers (push) Canceled after 0s
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Build distribution 📦 (push) Canceled after 0s
ChangeDetection.io Container Build Test / Build linux/amd64 (alpine) (push) Canceled after 0s
ChangeDetection.io Container Build Test / Build linux/arm64 (alpine) (push) Canceled after 0s
ChangeDetection.io Container Build Test / Build linux/amd64 (main) (push) Canceled after 0s
ChangeDetection.io Container Build Test / Build linux/arm/v7 (main) (push) Canceled after 0s
ChangeDetection.io Container Build Test / Build linux/arm/v8 (main) (push) Canceled after 0s
ChangeDetection.io Container Build Test / Build linux/arm64 (main) (push) Canceled after 0s
ChangeDetection.io App Test / lint-code (push) Canceled after 0s
ChangeDetection.io App Test / lint-translations (push) Canceled after 0s
ChangeDetection.io App Test / lint-template-i18n (push) Canceled after 0s
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Test the built package works basically. (push) Canceled after 0s
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Publish Python 🐍 distribution 📦 to PyPI (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-10 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-11 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-12 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-13 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-14 (push) Canceled after 0s
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com > (#4324 )
2026-08-24 16:24:22 +02:00
dgtlmoon
50389b0778
GHSA-56fq-63vj-9992 Improved URL filtering ( #4315 )
2026-08-20 15:49:41 +02:00
dgtlmoon
20e4641d25
WIP
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Build distribution 📦 (push) Canceled after 0s
ChangeDetection.io App Test / lint-code (push) Canceled after 0s
ChangeDetection.io App Test / lint-translations (push) Canceled after 0s
ChangeDetection.io App Test / lint-template-i18n (push) Canceled after 0s
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Test the built package works basically. (push) Canceled after 0s
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Publish Python 🐍 distribution 📦 to PyPI (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-10 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-11 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-12 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-13 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-14 (push) Canceled after 0s
2026-07-22 17:13:00 +02:00
dgtlmoon
c596cf6c14
WIP
2026-07-17 15:29:24 +02:00
dgtlmoon
71d332d5a0
UI full refactor / upgrade ( #4183 )
Build and push containers / metadata (push) Has been cancelled
Build and push containers / build-push-containers (push) Has been cancelled
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Build distribution 📦 (push) Has been cancelled
ChangeDetection.io App Test / lint-code (push) Has been cancelled
ChangeDetection.io App Test / lint-translations (push) Has been cancelled
ChangeDetection.io App Test / lint-template-i18n (push) Has been cancelled
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Test the built package works basically. (push) Has been cancelled
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Publish Python 🐍 distribution 📦 to PyPI (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-10 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-11 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-12 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-13 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-14 (push) Has been cancelled
2026-07-16 15:08:29 +02:00
dgtlmoon
31a760c214
CVE-2026-35490 - Authentication Bypass via Decorator Ordering
2026-04-04 05:58:53 +02:00
dgtlmoon
5a4266069b
Content Fetchers / Browsers - Improvements for pluggable extra fetchers/browsers. ( #3981 )
Build and push containers / metadata (push) Has been cancelled
Build and push containers / build-push-containers (push) Has been cancelled
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Build distribution 📦 (push) Has been cancelled
ChangeDetection.io App Test / lint-code (push) Has been cancelled
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Test the built package works basically. (push) Has been cancelled
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Publish Python 🐍 distribution 📦 to PyPI (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-10 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-11 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-12 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-13 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-14 (push) Has been cancelled
2026-03-15 17:35:46 +01:00
dgtlmoon
3558e9ee10
Browser Steps - Minor code cleanup
2026-02-16 13:22:54 +01:00
dgtlmoon
fe465de73c
Browser Steps - Clean off empty fields on save/update (UI and API), small refactor Re #3874 , #3879 ( #3880 )
2026-02-16 13:05:46 +01:00
dgtlmoon
892b645147
Refactor for Tags storage ( #3848 )
Build and push containers / metadata (push) Has been cancelled
Build and push containers / build-push-containers (push) Has been cancelled
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Build distribution 📦 (push) Has been cancelled
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Test the built package works basically. (push) Has been cancelled
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Publish Python 🐍 distribution 📦 to PyPI (push) Has been cancelled
ChangeDetection.io App Test / lint-code (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-10 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-11 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-12 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-13 (push) Has been cancelled
2026-02-07 13:13:02 +01:00
dgtlmoon
c2ee84b753
Browser Steps UI async_loop bug, refactored startup of BrowserSteps, increased test coverage. Re #3734 ( #3735 )
2026-01-14 11:27:01 +01:00
Voczi
1c2c22b8df
Specify UTF-8 encoding for xpath_element_js ( #3650 )
2025-11-23 19:55:26 +01:00
dgtlmoon
64350a2e78
Replace jinja2-time with arrow and improve timedate timezone integration, fixes timezones in templates such as {% now 'Europe/London', '%Y-%m-%d' %} etc ( #3496 )
2025-10-13 11:52:02 +02:00
dgtlmoon
c7374245e1
Browser Steps - Fix for source: URLs fix ( #3254 )
2025-06-11 09:43:48 +02:00
dgtlmoon
d7e24f64a5
Browser Steps - Better support for sites that redirect on click/login etc
2025-06-04 09:43:27 +02:00
dgtlmoon
a8e7f8236e
BrowserSteps - remove unsupported exception class
2025-06-04 07:33:09 +02:00
dgtlmoon
73f3beda00
Realtime UI - Socketio tweaks and refactor ( #3220 )
2025-06-03 10:17:19 +02:00
dgtlmoon
162f573967
Fixes to ensure proxy errors are handled correctly ( #3168 )
2025-05-03 16:05:40 +02:00
dgtlmoon
a75f57de43
Browser Steps - <Select> by Option Text - #1224 , #1228 ( #3138 )
2025-04-22 14:33:35 +02:00
dgtlmoon
72a1c3dda1
Browser Steps - error reporting and session shutdown improvements ( #3137 )
2025-04-22 12:18:51 +02:00
dgtlmoon
4269079c54
Refactor image saving with forked process to reduce memory usage, improvements to xpath scraper handling ( #3099 )
2025-04-11 09:04:51 +02:00
Sean Kelly
9f326783e5
Memory fixes for large playwright screenshots ( #3092 )
Build and push containers / metadata (push) Has been cancelled
Build and push containers / build-push-containers (push) Has been cancelled
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Build distribution 📦 (push) Has been cancelled
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Test the built 📦 package works basically. (push) Has been cancelled
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Publish Python 🐍 distribution 📦 to PyPI (push) Has been cancelled
ChangeDetection.io App Test / lint-code (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-10 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-11 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-12 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-13 (push) Has been cancelled
CodeQL / Analyze (javascript) (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
2025-04-09 17:02:44 +02:00
dgtlmoon
97291ce6d0
Code - Tidy up lint errors ( #3074 )
Build and push containers / metadata (push) Has been cancelled
Build and push containers / build-push-containers (push) Has been cancelled
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Build distribution 📦 (push) Has been cancelled
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Test the built 📦 package works basically. (push) Has been cancelled
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Publish Python 🐍 distribution 📦 to PyPI (push) Has been cancelled
ChangeDetection.io App Test / lint-code (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-10 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-11 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-12 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-13 (push) Has been cancelled
2025-03-31 13:05:48 +02:00
dgtlmoon
ff00417bc5
Browser Steps - Should use the Watch URL/link after any Jinja2 type templates are applied
Build and push containers / metadata (push) Has been cancelled
Build and push containers / build-push-containers (push) Has been cancelled
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Build distribution 📦 (push) Has been cancelled
ChangeDetection.io App Test / lint-code (push) Has been cancelled
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Test the built 📦 package works basically. (push) Has been cancelled
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Publish Python 🐍 distribution 📦 to PyPI (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-10 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-11 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-12 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-13 (push) Has been cancelled
CodeQL / Analyze (javascript) (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
2025-02-27 17:22:39 +01:00
dgtlmoon
849c5b2293
BrowserSteps - Speed up scraping, refactor screenshot handling for very long pages ( #2999 )
Build and push containers / metadata (push) Waiting to run
Build and push containers / build-push-containers (push) Waiting to run
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Build distribution 📦 (push) Waiting to run
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Test the built 📦 package works basically. (push) Blocked by required conditions
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Publish Python 🐍 distribution 📦 to PyPI (push) Blocked by required conditions
ChangeDetection.io App Test / lint-code (push) Waiting to run
ChangeDetection.io App Test / test-application-3-10 (push) Blocked by required conditions
ChangeDetection.io App Test / test-application-3-11 (push) Blocked by required conditions
ChangeDetection.io App Test / test-application-3-12 (push) Blocked by required conditions
ChangeDetection.io App Test / test-application-3-13 (push) Blocked by required conditions
ChangeDetection.io Container Build Test / test-container-build (push) Has been cancelled
2025-02-27 16:52:38 +01:00
dgtlmoon
4bf560256b
Browser Steps - Added new "Make all child elements visible" action
Build and push containers / metadata (push) Waiting to run
Build and push containers / build-push-containers (push) Waiting to run
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Build distribution 📦 (push) Waiting to run
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Test the built 📦 package works basically. (push) Blocked by required conditions
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Publish Python 🐍 distribution 📦 to PyPI (push) Blocked by required conditions
ChangeDetection.io App Test / lint-code (push) Waiting to run
ChangeDetection.io App Test / test-application-3-10 (push) Blocked by required conditions
ChangeDetection.io App Test / test-application-3-11 (push) Blocked by required conditions
ChangeDetection.io App Test / test-application-3-12 (push) Blocked by required conditions
ChangeDetection.io App Test / test-application-3-13 (push) Blocked by required conditions
CodeQL / Analyze (javascript) (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
2025-02-26 23:12:02 +01:00
dgtlmoon
7903b03a0c
Browser Steps - Added new "Remove elements" action
2025-02-26 22:37:06 +01:00
dgtlmoon
5d9380609c
Browser Steps - Increasing timeout for actions and unifying timeout values
Build and push containers / metadata (push) Waiting to run
Build and push containers / build-push-containers (push) Waiting to run
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Build distribution 📦 (push) Waiting to run
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Test the built 📦 package works basically. (push) Blocked by required conditions
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Publish Python 🐍 distribution 📦 to PyPI (push) Blocked by required conditions
ChangeDetection.io App Test / lint-code (push) Waiting to run
ChangeDetection.io App Test / test-application-3-10 (push) Blocked by required conditions
ChangeDetection.io App Test / test-application-3-11 (push) Blocked by required conditions
ChangeDetection.io App Test / test-application-3-12 (push) Blocked by required conditions
ChangeDetection.io App Test / test-application-3-13 (push) Blocked by required conditions
2025-02-10 10:56:44 +01:00
dgtlmoon
a8b3918fca
Browser Steps - Fixing 'Uncheck checkbox' #2958
2025-02-10 10:49:40 +01:00
Dawid Wróbel
a5ff1cd1d7
browser_steps: add "click element containing text if exists" ( #2629 )
2024-09-17 18:30:54 +02:00
Mike Splain
bf0d410d32
Browser Steps UI - Interactive UI wasn't sending headers but was when the check ran ( #2551 )
2024-08-19 10:21:05 +02:00
Kenny Root
d23a301826
Use #!/usr/bin/env to support virtualenv ( #2518 )
2024-07-25 14:39:01 +02:00
dgtlmoon
ed38012c6e
Code - Fixing deprecation warning ( #2477 )
2024-07-09 17:38:17 +02:00
dgtlmoon
343e359b39
Now saving last two HTML snapshots for future reference, refactor, dont write screenshots and xpath to disk when no change detected (saves disk IO) ( #2431 )
2024-06-23 09:19:32 +02:00
dgtlmoon
cc147be76e
Prefer pythons built in "importlib" over pkg_resources+setuptools ( #2424 )
2024-06-18 09:08:48 +02:00
dgtlmoon
e4504fee49
Browser Steps - Fixing "goto site" step #2330 #2337 ( #2364 )
2024-05-15 10:49:30 +02:00
dgtlmoon
bd6eda696c
Merge pull request from GHSA-4r7v-whpg-8rx3
...
* CVE-2024-32651 - Security fix - Server Side Template Injection in Jinja2 allows Remote Command Execution
* use ImmutableSandboxedEnvironment also in validation
2024-04-25 22:06:09 +02:00
dgtlmoon
9449c59fbb
Code - Getting ready for newer python versions - packing our own strtobool ( #2291 )
2024-04-03 16:17:15 +02:00
dgtlmoon
14e632bc19
Custom headers fix in Browser Steps and Playwright/Puppeteer fetchers ( #2197 )
2024-02-26 18:02:45 +01:00
dgtlmoon
823a0c99f4
Code - Split content fetcher code up (playwright, puppeteer and requests), fix puppeteer direct chrome support ( #2169 )
2024-02-11 00:09:12 +01:00
dgtlmoon
c5a4e0aaa3
Fetching - Prefer to use SockPuppetBrowser ( #2163 )
2024-02-07 20:58:21 +01:00
Constantin Hong
4be0fafa93
Support Loguru as a logger ( #2036 )
2024-01-16 09:48:16 +01:00
dgtlmoon
8be0029260
Browser Steps - Fixing "'Response' object is not subscriptable" where quotes were used in connection URL - Quote wrapped URL for browserstep url was breaking the connection #1627 #1823 #2099 ( #2100 )
2024-01-11 10:12:00 +01:00
dgtlmoon
98f56736c1
Improve handling of SIGTERM shutdown in containers, remove unnecessary multi-process handler for pip installs, tidy up modules ( #2014 )
2023-12-01 17:31:09 +01:00
dgtlmoon
7fdae82e46
Browser Steps - Adding validation for "Click X,Y" step
2023-11-28 12:36:15 +01:00
dgtlmoon
d8ee5472f1
Update playwright fetcher library and API calls
2023-11-28 11:20:06 +01:00
dgtlmoon
349111eb35
Fetching/BrowserSteps - Going to a page was using slightly logic to the main way - make them use the same methods ( #1890 )
2023-10-26 20:19:22 +02:00
