* API - Reject broken Jinja2 in url, link_to_open, body and headers, shared validation with the edit form
The edit form validated Jinja2 templates on save but the API did not, so a broken body or
header template saved through the API only failed at fetch time, on every check.
- New jinja2_custom/validate.py: validate_jinja2_fields() renders each value that matches
JINJA2_MARKER_PATTERN and returns {field: translated error}
- The edit form uses it instead of four copies of the same try/except
- API create and update return 400 naming the field
Translations: messages moved to the new file (msgids unchanged); also translated the header
error for fr/it and fixed the cs/es wording.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* rebuild
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The old site's icon was kept until it aged past the 24h refetch threshold, or forever
when the new site has no icon or the watch uses a fetcher that doesn't fetch favicons.
Watch.update() now clears favicon.* and the filename cache when the URL changes, which
covers both the UI edit form and the API. favicon_is_expired() then sees no file and
the next check fetches the new icon. A change from an empty URL is ignored, since every
fresh Watch object starts with url '' and is filled from storage via update().
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Add tests for Jinja2 template detection in URL validation (RED)
* Fix Jinja2 false-positive when URL contains `{%` (GREEN)
* Consolidate Jinja2 template detection across URL/body/headers validation
* Propagate `FORCE_COLOR` to test containers
* Apply suggestion from @dgtlmoon
* Apply suggestion from @dgtlmoon
* Only render request headers/body at fetch time when they contain Jinja2, drop FORCE_COLOR changes
The edit form now skips Jinja2 validation for text like {%22 that is not a template,
but processors/base.py still rendered every header and the body unconditionally, so
a body such as q={%22key%22:1} was accepted on save and then failed every check with
a TemplateSyntaxError. Use JINJA2_MARKER_PATTERN there too, so it is sent as-is.
Also reset the CI workflow and run_*_tests.sh scripts to master, the FORCE_COLOR
changes are unrelated to this fix.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: dgtlmoon <dgtlmoon@gmail.com>
Co-authored-by: dgtlmoon <leigh@morresi.net>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Judge favicon expiry on the icon the watch actually has, not whatever glob lists first
bump_favicon() names the saved file after the icon's type (favicon.png, favicon.ico, ...)
and did not remove a previous save under a different extension, so a watch whose icon
changed type ends up holding more than one favicon.*.
favicon_is_expired() then picked next(iter(glob.glob("favicon.*"))). glob order is
os.scandir order - stable, but decided by filename hash - so for some extension pairs the
stale file wins every single time, its age always exceeds the 24h threshold, and the
favicon is refetched on every check forever. Measured across realistic pairs, 3 of 10
lose that way:
old=.png new=.ico -> glob lists favicon.png first -> always expired
old=.ico new=.svg -> glob lists favicon.ico first -> always expired
old=.png new=.webp -> glob lists favicon.png first -> always expired
That is an in-page network fetch through the watch's proxy on every check, plus the
favicon fetcher's own sequential per-icon timeouts, for a favicon that is already saved
and current.
Fixed at both ends: pick the newest candidate by mtime, and have bump_favicon() drop
superseded files so the ambiguity stops arising.
Also tidied two things in the same function:
- os.path.isfile(None) raised a TypeError when the module-level filename cache said yes
after the file had been removed, which the broad except turned into a logger.critical()
for an entirely benign condition. The empty case is now handled explicitly.
- the broad except stays (this runs inline on the check path as an argument to
fetcher.run(), so it must never fail a watch) but logs at warning, since the only cost
of getting this wrong is one extra favicon fetch.
Tests fail on master (favicon.ico left behind; fresh icon reported expired) and pass here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Favicon - write the new icon before removing superseded ones, so a failed write keeps the old icon
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Proxy usernames and passwords with special characters have to be URL-escaped
(#2062). The requests fetcher decodes them, but the Playwright and Puppeteer
fetchers and Browser Steps passed the escaped values to the browser as-is.
The proxy URL, password included, was also logged at DEBUG, the default level.
Add proxy_credentials() and redact_proxy_url() in changedetectionio/proxy_url.py
and use them at those sites, with unit tests.
* Log puremagic 'Could not identify file' at debug, not warning
puremagic raises PureError when no signature matches, which is the normal
result for plain text and most HTML pages. guess_stream_type() caught it
with every other exception and logged a warning on many checks, although
the content-based detection that follows handles it as designed. Other
puremagic failures still log a warning.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Skip puremagic on empty content and catch PureError directly
puremagic.magic_string() raises ValueError('Input was empty') for empty
input, which is as normal as an unidentified signature, so skip the call
then. Import puremagic before the try (it is a hard dependency) so the
except clause can name PureError instead of comparing the class name.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Keep the puremagic import inside the try so a missing library falls back
ImportError is caught before puremagic.PureError, so that clause is only
evaluated once the import has succeeded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* LLM restock: parse string prices with price_parser instead of stripping separators
If the model returns the price as page text, e.g. "1.299,95 €", stripping
everything but digits and dots gave 1.29995. price_parser (already a
dependency) handles the locale formats the prompt lists.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* LLM restock: use Restock.parse_currency() like the other price extractors
Same parser c8afea1b (#4494) moved the structured-data extractors to, so
there's one place price text becomes a number. parse_currency() now lists
every caller, and the call sites point back to it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* LLM restock: expose prompt/cache/parsing helpers so other LLM backends can reuse them
Split the hook body into build_restock_messages(), restock_cache_key/get/put()
and parse_restock_reply(), so a plugin talking to a provider without litellm
only has to supply the HTTP call. accumulate_global_tokens() takes an optional
cost_usd, since the default estimate imports litellm's pricing (~200MB that's
never released). The built-in plugin behaves exactly as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Test: give the no-litellm subprocess check a PYTHONPATH
CI runs pytest from inside the package dir without installing it, so the
child interpreter couldn't import changedetectionio.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Restock plugins: pass html_to_text() output to the restock hook, let third-party backends go first
get_itemprop_availability_override now also gets `text` (the fetched page through
html_to_text()), only computed when an implementation declares it. The hook is
firstresult and the built-in litellm plugin is trylast, so a plugin that talks to an
LLM with plain requests answers first and the built-in one is never billed as well.
Also fixes the built-in plugin never receiving llm_intent: pluggy silently doesn't
pass hookimpl arguments that have a default value, and it declared llm_intent=None.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Watch.history ran os.path.realpath() twice and os.path.exists() once for every
line in history.txt (~10 syscalls per snapshot), on every watch at startup and
on every history read after that.
basename() already reduces each entry to a plain filename, so the only way out
of the watch data dir is a symlink. One os.scandir() of the data dir now
answers "does it exist", and only symlinks pay for a realpath() containment
check. scandir() never lists '.' or '..', so those are still rejected.
~13x faster on 200 watches x 500 snapshots (1.18s -> 0.09s, warm cache),
identical output. Adds tests for symlinks escaping/inside the data dir,
dangling symlinks, and missing snapshots/blank lines.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The extractors stripped everything but [\d.] before converting, so the
decimal comma was thrown away before Restock.parse_currency(), which
already understands it, ever saw the value. Route the values through
parse_currency() instead, and teach it that a lone "1,299" is a
thousands separator so that case does not regress.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Every watch whose xPath filter used contains() started reporting "Warning, no filters were
found" on pages whose HTML plainly contained the target. 18 unrelated watches broke in the same
hour, browser and plain-requests fetchers alike, and the saved snapshot was perfect every time.
elementpath implements the XPath string functions on top of locale.strxfrm:
def contains(self, a, b): return self.strxfrm(b) in self.strxfrm(a)
Under LC_COLLATE=C, strxfrm() is the identity function and that is an ordinary substring test.
Under a real locale it returns a binary collation key, and a substring of a collation key is not
the collation key of the substring - so contains(), starts-with(), ends-with() and
substring-before/after() return false for EVERY input. Reduced to one line, no document needed:
LC_COLLATE=C contains("xx month xx", "month") -> True
LC_COLLATE=en_US.UTF-8 contains("xx month xx", "month") -> False
Name tests, axes and '=' are untouched, which is exactly why it read as "did the page change
layout?" - //div kept working while //div[contains(.,"month")] returned nothing.
No code change caused this. Generating the image's locales (#4429) made ENV LC_ALL=en_US.UTF-8
satisfiable for the first time; flask_app's setlocale(LC_ALL, ...) had been raising locale.Error
and leaving us in C, and once it succeeded it took LC_COLLATE with it. That is why the bug
survived a bisect to 0.60.2 and why reinstalling the exact pip set from a working install did not
shift it - it could only be found by diffing the two containers. Same image base, same Python
3.11.16, lxml 6.1.3, libxml2 2.14.6, elementpath 5.1.1, same HTML:
good-old 0.60.4 setlocale FAILED: unsupported locale setting 67 matches
bad-new 0.60.5 setlocale en_US.UTF-8 0 matches
Fixed in both places, because either alone leaves a hole:
- flask_app sets LC_CTYPE/LC_NUMERIC/LC_MONETARY/LC_TIME individually instead of LC_ALL. This
block exists to make prices render correctly and it still does - 1234567 is still "1,234,567"
- it just no longer touches collation.
- html_tools.xpath_filter() pins the Unicode codepoint collation per evaluation, so a filter
means the same thing whatever an operator puts in LANG/LC_ALL, and does not depend on a
distant module's locale bookkeeping. forms.py's XPath validation pins it too, so validation
cannot accept an expression that then behaves differently at check time.
Per XPath 3.1 the default collation is codepoint and must not consult LC_COLLATE, so the
underlying behaviour is arguably an elementpath bug; the pin above holds regardless.
Verified end to end inside the failing container: LC_COLLATE=C, LC_NUMERIC=en_US.UTF-8,
thousands separators intact, and the reported filter back from 0 to 1190797 chars of output.
Tested: new unit test covers contains/starts-with/ends-with under a UTF-8 collation and was
checked to fail without the fix; 325 unit tests pass.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Compare against a resolved data_dir in the history path-traversal test
Watch.history resolves entries with os.path.realpath, so
test_normal_snapshot_entry_is_accepted compared a resolved path against an
unresolved data_dir. On macOS the datastore lives under /tmp, which is a
symlink to /private/tmp, so the assertion fails for a path that is in fact
inside the directory. The guard is correct; the test was not.
Resolve both sides, matching what the production code does.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Add a test that actually exercises the history containment check
Disabling the containment check in Watch.history left every test in
TestHistoryPathTraversal passing. os.path.basename() reduces both traversal
fixtures ('/etc/passwd', '../../etc/passwd') to 'passwd', so neither reaches
the check — they stop at the os.path.exists() test below it.
A bare '..' survives basename() and resolves to the parent of data_dir, which
exists, so the containment check is what rejects it. With that check disabled
this new test is the only one in the class that fails.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: GG5533 <285285461+GG5533@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* fix(conditions): support zero values in condition filtering and json logic conversion
- Fix filter_complete_rules dropping rules where value is 0/0.0 due to 0 == False in Python
- Fix convert_to_jsonlogic raising EmptyConditionRuleRowNotUsable on value=0 due to truthiness check
- Fix str != 'None' type comparison typo in convert_to_jsonlogic
- Add comprehensive unit tests covering zero value condition filtering, conversion, and execution
* chore: re-trigger CI checks
---------
Co-authored-by: Andrew Peabody <apeabody@users.noreply.github.com>
- Define _NO_TEMPERATURE_MODEL_KEYWORDS constant and omit temperature=0 on initial request
- Omit thinkingConfig in _thinking_extra_body for flash-lite models
- Unconditionally strip rejected sampling params and extra_body thinkingConfig on BadRequestError (handling Gemini 400 INVALID_ARGUMENT without named details)
- Add comprehensive unit tests in test_llm_client_gemini.py
* Scheduler+API Bug - if an invalid timezone was set (through edit of watch or API) it could have crashed the scheduler, Added `timezone` to the official API docs
* adding missing files
* Restock detection - fix inverted condition that skipped the OpenGraph availability fallback
get_itemprop_availability() only dug through OpenGraph properties when
price was missing OR when availability was ALREADY found - so on pages
that expose price via JSON-LD but stock state only via the OpenGraph
commerce tags (<meta property="product:availability">), the availability
(and currency) was silently dropped.
The inner loop already guards each field with 'if not value.get(...)',
so the outer gate now matches that intent: dig OpenGraph when any of
price/availability/currency is still missing.
Co-Authored-By: Claude <noreply@anthropic.com>
* Normalise availability after the OpenGraph fallback, not before
An availability found via OpenGraph was left raw, so the Facebook
commerce spelling 'in stock' never matched the 'instock' test and the
product read as out of stock.
---------
Co-authored-by: Claude <noreply@anthropic.com>
