36 Commits

Include renames

Author	SHA1	Message	Date
dgtlmoonandClaude Opus 5	ba2280ba81	External CDP Browser engine - replaces the extra_browser_* hackery ... An "extra browser" was a name + a ws(s):// endpoint in settings.requests.extra_browsers, selected by a watch as the magic string 'extra_browser_<name>'. That string resolved to html_webdriver plus a custom connection URL, which meant the protocol the endpoint was spoken to came from env vars rather than from the entry: CDP over a WebSocket with PLAYWRIGHT_DRIVER_URL set, CDP via pyppeteer with FAST_PUPPETEER_CHROME_FETCHER, and the W3C WebDriver protocol over HTTP on a Selenium-only install - where a wss:// URL cannot work at all. The form only ever accepted ws:// / wss://, so the feature was silently broken on exactly the installs that could not honour it. So it becomes an engine, html_external_cdp, which pins the protocol: a subclass of the Playwright fetcher that takes its endpoint from the watch's browser config (FetcherConfig.connection_url) instead of the environment. It is base-only (ready_to_use=False) because an endpoint is required, so each endpoint is one browser config ("variation") on the Browsers page - which is what the old settings list was. update_36 migrates each extra_browsers row to such a variation, keyed by the SAME 'extra_browser_<name>' string watches already hold, so no watch, group override, API value or global default needs rewriting; the legacy selector simply becomes a real browser-config id. A row whose endpoint the model rejects is logged and skipped rather than taking the update chain, and with it startup, down. Knock-on cleanups, all of which delete a special case rather than add one: - The proxy opt-out for custom endpoints is now Fetcher.ignores_proxy_setting, asked of the engine, instead of a string-prefix test in call_browser(). - A live browser-steps / visual-selector session asks the engine where to connect (Fetcher.browser_steps_connection_url, overridden by html_external_cdp) and refuses an engine whose supports_browser_steps is False, instead of reading the env var itself and silently stepping a browser the watch does not check with. That refusal is real: on a Selenium install html_webdriver cannot drive a live session. - is_valid_browser_selector() answers "may a watch store this in fetch_backend?" in one place; the API (create/update/import), the quick-add form validator and the bulk "set browser" operation each had their own copy, which is how they came to disagree about whether a browser-config id was acceptable. - api-spec.yaml's fetch_backend pattern enumerated extra_browser_* while rejecting browser-config ids and every engine newer than html_webdriver. Valid values are per-install, so the schema now bounds the string and the handlers do the real check. - html_external_cdp registers unconditionally (unlike html_playwright_builtin): migrated configs name it, so it must resolve even without the playwright library, or those watches would quietly fetch with the plain HTTP client. The library is imported lazily inside run(), and an unavailable engine now warns instead of falling back silently. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-17 19:39:16 +02:00
dgtlmoon	00a00b664f	Merge branch 'master' into browser-ui	2026-09-17 17:26:16 +02:00
dgtlmoonandClaude Opus 5	fbd9472218	fix(api): accept an existing tag UUID in the watch tag field, and deprecate it (#4361) ... `tag` on POST /watch was documented as taking a tag UUID, but the value went to add_tag(title): a UUID silently created a junk tag *titled* with that UUID and never applied the tag the caller asked for. `tags` (UUIDs) was the only thing that worked. - `tag=` now resolves an existing tag UUID to that tag, still falling back to title matching/creation for names. A UUID-shaped value matching nothing is skipped with a warning rather than becoming a group named after a UUID. - Blank tokens ("One,,Two,") no longer store False in watch['tags'] - add_tag() returns False for an empty title and it was appended unguarded. Consumers tolerate it (get_all_tags_for_watch() dictfilt()s over known tags) but it is not valid data. - add_tag()'s title search is extracted to tag_uuid_for_title(), so existence can be tested without creating as a side effect. add_tag()'s contract is unchanged. - api-spec: `tag` is marked `deprecated: true` (so Redoc renders the badge) and states plainly that it takes names, not UUIDs. `tags` now says what it really does - applied verbatim, never creates, unknown UUIDs stored as dangling refs. Rendered docs rebuilt. Every claim in the new field docs is asserted in test_api_tags.py against the real API. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 16:46:27 +02:00
dgtlmoon	68a445df1b	Env var - PAGE_WATCH_LIMIT enhancements (#4359)	2026-09-02 13:37:55 +02:00
dgtlmoon	af234636b9	Watch loading - Dont crawl whole disk, just check each directory for a watch.json/tag.json (#4358)	2026-09-02 11:10:49 +02:00
dgtlmoon	bb2e6adece	Add watch UI should show which browser(s) are available and set default browser (#4334)	2026-08-25 20:15:59 +02:00
dgtlmoon	acc4533f51	Security - GHSA-mh42-m7cg-49fr - Authenticated path traversal to arbitrary file write via the watch processor field (enum bypassed through /imports/import)	2026-08-20 10:56:13 +02:00
dgtlmoon	6f2e4aa9af	UI - Move rendering of each watchlist row out of the watchlist main template so it can be re-used by the realtime update (#4305)	2026-08-16 18:07:34 +02:00
snowyukittyandsnowyukitty	21f19b239a	fix: avoid division by zero when logging watch load rate (#4289) ... Co-authored-by: snowyukitty <270071858+snowyukitty@users.noreply.github.com>	2026-08-16 14:26:39 +02:00
dgtlmoon	8a12bb82b0	WIP	2026-07-17 16:45:59 +02:00
dgtlmoon	c596cf6c14	WIP	2026-07-17 15:29:24 +02:00
dgtlmoon	ee35b3da1a	UI - Another attempt at configurable browsers	2026-07-16 21:30:52 +02:00
dgtlmoon	71d332d5a0	UI full refactor / upgrade (#4183)	2026-07-16 15:08:29 +02:00
dgtlmoon	24da66f92c	Restock - Threshold change since "first check" was working really as "since last check", update UI, tests, field name.	2026-06-18 12:01:41 +02:00
dgtlmoon	0e3f1941b3	Code - LLM settings pydantic refactor (#4181)	2026-05-25 10:43:52 +02:00
skkzsh	281c1ea7e1	i18n: Clear pre-existing dennis warnings in messages.pot (#4112)	2026-05-02 04:57:13 +02:00
dgtlmoon	238d6ba72d	Feature - Groups/tag - Apply a group by specifying a wildcard, ie *.mysite.com* (#4032)	2026-04-10 02:45:23 +02:00
dgtlmoon	5669509255	API - Processors configuration is now part of the API (#3902)	2026-02-25 11:30:39 +01:00
dgtlmoon	fe7aa38c65	CVE-2026-27696 - Server-Side Request Forgery (SSRF) via Watch URLs, set env var ALLOW_IANA_RESTRICTED_ADDRESSES to true to access IANA reserved URLs such as http://169.254.169.254, http://10.0.0.1/, http://127.0.0.1/, etc.	2026-02-23 21:56:43 +01:00
dgtlmoon	fafea1b5c6	Updates/migration - Re-run tag update, re-save to cleanup changedetection.json, code refactor (#3898)	2026-02-18 14:05:23 +01:00
dgtlmoon	549e167746	Datastore - On fresh installs, also scan for existing watch.json watches in subdirectories	2026-02-16 15:56:46 +01:00
dgtlmoon	b6d3d63372	Avoid reprocessing if the page was the same (#3867)	2026-02-14 21:24:28 +01:00
dgtlmoon	ef41dd304c	Refactoring upgrade path (#3861)	2026-02-11 16:13:08 +01:00
dgtlmoon	987789425d	Tags update fix (#3849)	2026-02-07 17:13:41 +01:00
dgtlmoon	892b645147	Refactor for Tags storage (#3848)	2026-02-07 13:13:02 +01:00
dgtlmoon	c577bd700c	Refactor watch saving backend, closes #3846 (#3847)	2026-02-07 03:41:35 +01:00
dgtlmoon	84d26640cc	Adding more tests and Watch object improvements (#3841)	2026-02-05 17:01:08 +01:00
dgtlmoon	2349344d9e	Improved watch global settings handling (#3839)	2026-02-05 16:40:00 +01:00
dgtlmoon	bdc2916c07	New datastore message should be warning not critical	2026-02-05 16:25:22 +01:00
dgtlmoon	4fd477a60c	Improving upgrade path	2026-02-05 13:00:01 +01:00
dgtlmoon	dc8b387f40	History length limit size option (#3834)	2026-02-05 12:29:20 +01:00
dgtlmoon	30dc4ac23b	Refactor of queue systenm and improve tests, improves multiple workers (#3826)	2026-02-02 22:28:27 +01:00
dgtlmoon	2658f81f02	Ability to limit total number of watches with env var PAGE_WATCH_LIMIT (#3828)	2026-02-02 11:21:44 +01:00
dgtlmoon	fd820c9330	Remove deprecated call to strtobool	2026-01-28 17:27:20 +01:00
dgtlmoon	a239480272	DB data migration upgrade fixes (#3811)	2026-01-28 12:02:19 +01:00
dgtlmoon	fceb3cf39f	Big refactor to save watches as their own datafile with some agnostic data store backend, saves writing a huge JSON file every time (#3775)	2026-01-28 10:18:21 +01:00