wtforms~=3.2 now installs 3.3.0, which changed FloatField's widget from a text input
to a number input, so test_itemprop_percent_threshold failed on every branch. The
value is still saved and rendered correctly; only check that.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The old site's icon was kept until it aged past the 24h refetch threshold, or forever
when the new site has no icon or the watch uses a fetcher that doesn't fetch favicons.
Watch.update() now clears favicon.* and the filename cache when the URL changes, which
covers both the UI edit form and the API. favicon_is_expired() then sees no file and
the next check fetches the new icon. A change from an empty URL is ignored, since every
fresh Watch object starts with url '' and is filled from storage via update().
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Add tests for Jinja2 template detection in URL validation (RED)
* Fix Jinja2 false-positive when URL contains `{%` (GREEN)
* Consolidate Jinja2 template detection across URL/body/headers validation
* Propagate `FORCE_COLOR` to test containers
* Apply suggestion from @dgtlmoon
* Apply suggestion from @dgtlmoon
* Only render request headers/body at fetch time when they contain Jinja2, drop FORCE_COLOR changes
The edit form now skips Jinja2 validation for text like {%22 that is not a template,
but processors/base.py still rendered every header and the body unconditionally, so
a body such as q={%22key%22:1} was accepted on save and then failed every check with
a TemplateSyntaxError. Use JINJA2_MARKER_PATTERN there too, so it is sent as-is.
Also reset the CI workflow and run_*_tests.sh scripts to master, the FORCE_COLOR
changes are unrelated to this fix.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: dgtlmoon <dgtlmoon@gmail.com>
Co-authored-by: dgtlmoon <leigh@morresi.net>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Notifications - |regex_replace ReDoS timeout now works in any thread, use the 'regex' package timeout= instead of SIGALRM
signal.alarm() can only be set from the main thread, but notifications are rendered
in the notification runner threads and previews in the Flask request threads, so
since #4577 the timeout was effectively never applied in production.
Re #3501, follow-up to #4577
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* regex_replace - Allow the timeout to be set with the REGEX_TIMEOUT_SECONDS env var
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* regex_replace - Import 'regex' lazily inside the filter, saves ~2MB RSS per instance that never uses it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Judge favicon expiry on the icon the watch actually has, not whatever glob lists first
bump_favicon() names the saved file after the icon's type (favicon.png, favicon.ico, ...)
and did not remove a previous save under a different extension, so a watch whose icon
changed type ends up holding more than one favicon.*.
favicon_is_expired() then picked next(iter(glob.glob("favicon.*"))). glob order is
os.scandir order - stable, but decided by filename hash - so for some extension pairs the
stale file wins every single time, its age always exceeds the 24h threshold, and the
favicon is refetched on every check forever. Measured across realistic pairs, 3 of 10
lose that way:
old=.png new=.ico -> glob lists favicon.png first -> always expired
old=.ico new=.svg -> glob lists favicon.ico first -> always expired
old=.png new=.webp -> glob lists favicon.png first -> always expired
That is an in-page network fetch through the watch's proxy on every check, plus the
favicon fetcher's own sequential per-icon timeouts, for a favicon that is already saved
and current.
Fixed at both ends: pick the newest candidate by mtime, and have bump_favicon() drop
superseded files so the ambiguity stops arising.
Also tidied two things in the same function:
- os.path.isfile(None) raised a TypeError when the module-level filename cache said yes
after the file had been removed, which the broad except turned into a logger.critical()
for an entirely benign condition. The empty case is now handled explicitly.
- the broad except stays (this runs inline on the check path as an argument to
fetcher.run(), so it must never fail a watch) but logs at warning, since the only cost
of getting this wrong is one extra favicon fetch.
Tests fail on master (favicon.ico left behind; fresh icon reported expired) and pass here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Favicon - write the new icon before removing superseded ones, so a failed write keeps the old icon
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Proxy usernames and passwords with special characters have to be URL-escaped
(#2062). The requests fetcher decodes them, but the Playwright and Puppeteer
fetchers and Browser Steps passed the escaped values to the browser as-is.
The proxy URL, password included, was also logged at DEBUG, the default level.
Add proxy_credentials() and redact_proxy_url() in changedetectionio/proxy_url.py
and use them at those sites, with unit tests.
extract_json_blob_from_html() and extract_json_as_string() both peeked at
content_start[0] to decide whether the body looks like JSON. A blank body
leaves content_start empty, so the peek raised IndexError instead of the
watch reporting that no JSON was found.
Guard the index in both places so empty and whitespace-only content takes
the same JSONNotFound path as any other non-JSON body.
Fixes#4531
Co-authored-by: Yi-111-a <yi-111-a@users.noreply.github.com>
* Log puremagic 'Could not identify file' at debug, not warning
puremagic raises PureError when no signature matches, which is the normal
result for plain text and most HTML pages. guess_stream_type() caught it
with every other exception and logged a warning on many checks, although
the content-based detection that follows handles it as designed. Other
puremagic failures still log a warning.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Skip puremagic on empty content and catch PureError directly
puremagic.magic_string() raises ValueError('Input was empty') for empty
input, which is as normal as an unidentified signature, so skip the call
then. Import puremagic before the try (it is a hard dependency) so the
except clause can name PureError instead of comparing the class name.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Keep the puremagic import inside the try so a missing library falls back
ImportError is caught before puremagic.PureError, so that clause is only
evaluated once the import has succeeded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* LLM restock: parse string prices with price_parser instead of stripping separators
If the model returns the price as page text, e.g. "1.299,95 €", stripping
everything but digits and dots gave 1.29995. price_parser (already a
dependency) handles the locale formats the prompt lists.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* LLM restock: use Restock.parse_currency() like the other price extractors
Same parser c8afea1b (#4494) moved the structured-data extractors to, so
there's one place price text becomes a number. parse_currency() now lists
every caller, and the call sites point back to it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* LLM restock: expose prompt/cache/parsing helpers so other LLM backends can reuse them
Split the hook body into build_restock_messages(), restock_cache_key/get/put()
and parse_restock_reply(), so a plugin talking to a provider without litellm
only has to supply the HTTP call. accumulate_global_tokens() takes an optional
cost_usd, since the default estimate imports litellm's pricing (~200MB that's
never released). The built-in plugin behaves exactly as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Test: give the no-litellm subprocess check a PYTHONPATH
CI runs pytest from inside the package dir without installing it, so the
child interpreter couldn't import changedetectionio.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Restock plugins: pass html_to_text() output to the restock hook, let third-party backends go first
get_itemprop_availability_override now also gets `text` (the fetched page through
html_to_text()), only computed when an implementation declares it. The hook is
firstresult and the built-in litellm plugin is trylast, so a plugin that talks to an
LLM with plain requests answers first and the built-in one is never billed as well.
Also fixes the built-in plugin never receiving llm_intent: pluggy silently doesn't
pass hookimpl arguments that have a default value, and it declared llm_intent=None.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
SCREENSHOT_MAX_HEIGHT_DEFAULT was raised to 20000 in #3092
(content_fetchers/__init__.py), the example comment still said 16000.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Watch.history ran os.path.realpath() twice and os.path.exists() once for every
line in history.txt (~10 syscalls per snapshot), on every watch at startup and
on every history read after that.
basename() already reduces each entry to a plain filename, so the only way out
of the watch data dir is a symlink. One os.scandir() of the data dir now
answers "does it exist", and only symlinks pay for a realpath() containment
check. scandir() never lists '.' or '..', so those are still rejected.
~13x faster on 200 watches x 500 snapshots (1.18s -> 0.09s, warm cache),
identical output. Adds tests for symlinks escaping/inside the data dir,
dangling symlinks, and missing snapshots/blank lines.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Currently dark mode is only available to users that have opted-in by
selecting a cookie. Users that have set their device preference to
dark still see light mode until they've specifically clicked the button
to set the cookie.
Without a css_dark_mode cookie the page now renders data-darkmode="auto",
and the dark styles apply under @media (prefers-color-scheme: dark), so a
device in dark mode gets the dark theme without opting in from the menu.
Pure CSS - no flash of the wrong theme and no JS needed.
The cookie set by the menu toggle still wins either way, so light-mode
devices can choose dark and vice versa. In "auto" the toggle reads the
current effective theme via matchMedia so the first click always flips it.
A dark-mode SCSS mixin emits each dark block for both
html[data-darkmode="true"] and the auto+media-query case; the processor
badge and tag colour rules generated outside SCSS get the same pair.
* PDF - Keep the original PDF of the newest two versions
The history snapshot of a PDF watch is only the pdftohtml text, and the "last fetched" .html.br
for a PDF is just a checksum, so the actual documents were never kept. Store {timestamp}.pdf
alongside the snapshot (same as {timestamp}.html.br does for HTML) and prune to the newest two
history entries, so the latest two versions are always available for a visual PDF compare.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Test - Browser PDF test, fix race with an immediate fallback recheck, check the original PDFs are kept
A duplicate queue entry (scheduler + import) can run the html_requests fallback check right
after the failed browser check, so by the time the test looks the error may already be cleared.
Accept both outcomes, the error message itself is covered in tests/test_pdf.py.
Also check both original PDF versions are stored, whichever fetcher got them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* PDF - Checksum the original document not the browser's PDF viewer HTML, browser PDF test
The "unchanged since last check" checksum was taken from fetcher.content, for a PDF in a browser
that is the viewer's HTML wrapper which is identical for every version. Prefer raw_content.
Also check that a PDF is actually available before the checksum skip; previously the viewer HTML
checksum was saved before the PDF error was raised, so the next check skipped as "unchanged",
cleared the error, and the watch silently never recorded anything.
Adds a playwright + puppeteer container test (PDF detected, unchanged PDF is not a change,
new PDF version is), puppeteer must fetch it itself, playwright must fall back to html_requests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* PDF - Only use raw_content for the document checksum when it is a PDF
Every other content type keeps checksumming .content exactly as before, so existing
watches keep their checksum basis.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Revert the raw_content document checksum change
Choosing between .content and .raw_content belongs in the general binary content handling
(#1334), which will be done separately. The browser PDF test and the early "PDF not available"
check stay.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Fetchers - Define raw_content on the base fetcher, give a clear error when a fetcher can't supply the PDF
Fetchers that never set raw_content (puppeteer, playwright, selenium) crashed with
"'fetcher' object has no attribute 'raw_content'" when the response turned out to be a PDF.
Browsers render PDFs in their built-in viewer, so all they have is the viewer HTML; now the
processor reports that plainly instead of crashing or feeding HTML to pdftohtml.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Puppeteer - Capture the real PDF body when the page turns out to be a PDF
Chrome hands a PDF straight to its built-in viewer, after which page.content and
Network.getResponseBody only return the viewer's HTML. Pause main-frame document responses
with the CDP Fetch domain and, when the Content-Type is application/pdf, copy the body out
before it is continued - a single download, and nothing needs to be known in advance.
content/raw_content then match the requests fetcher's binary handling (md5 + original bytes)
so the existing PDF processing and checksum skip work unchanged. Normal HTML pages are
continued immediately, no measurable difference in fetch time.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Fetchers - Add supports_pdf capability, only force html_requests for PDFs when the fetcher can't supply them
PDF watches were always forced onto html_requests. Now that puppeteer can capture the original
PDF (useful when the session needs a real browser, Cloudflare etc), only fetchers without
supports_pdf fall back - both at fetch time (resolve_content_fetcher) and in Watch.get_fetch_backend.
Also save the content-type when a fetcher couldn't supply the PDF (it was only saved on success),
otherwise watch.is_pdf never became true for URLs not ending in .pdf and the watch failed on
every check instead of falling back to html_requests on the next one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* LLM restock: parse string prices with price_parser instead of stripping separators
If the model returns the price as page text, e.g. "1.299,95 €", stripping
everything but digits and dots gave 1.29995. price_parser (already a
dependency) handles the locale formats the prompt lists.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* LLM restock: use Restock.parse_currency() like the other price extractors
Same parser c8afea1b (#4494) moved the structured-data extractors to, so
there's one place price text becomes a number. parse_currency() now lists
every caller, and the call sites point back to it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* ci: pin third party actions to commit SHAs
A tag or branch ref is a movable pointer, whoever controls the action
repository can repoint it and the next run executes their code with
this repo's secrets, here the Docker Hub tokens and the container
publish path. The March 2025 tj-actions/changed-files incident
(CVE-2025-30066) worked exactly that way. Each pin keeps the resolved
tag in a comment and dependabot's weekly github-actions config bumps
SHA pins like tags, so they self-maintain. The pypi publish action
moves from the floating release/v1 branch to its current release.
* ci: declare least-privilege workflow permissions
Without a permissions block every job token inherits the repository
default scope. Each workflow now declares what it uses. The test and
build workflows get read-only tokens, CodeQL keeps its code scanning
upload, the container publish job keeps packages write for the ghcr
push, and the pypi publish job already carried its own id-token block
for trusted publishing, untouched.
* ci: pass release metadata to inline scripts through env
The two debug steps interpolated the release tag name and the
workflow_run branch directly into bash. Those strings are event
metadata that can carry shell syntax, and the workflow runs on
release events with the publish credentials in reach. Routed through
env they arrive as plain data, output unchanged.
* ci: stop persisting the checkout token in .git/config
actions/checkout writes the job token into .git/config so later steps can
push. None of these 21 jobs push with git: the containers are published
with registry credentials, PyPI uses trusted publishing, and the test
and CodeQL jobs only read the tree. A token left in .git/config is one
more credential for a compromised step or a packed artifact to pick up,
so each checkout now sets persist-credentials: false.
