The old site's icon was kept until it aged past the 24h refetch threshold, or forever
when the new site has no icon or the watch uses a fetcher that doesn't fetch favicons.
Watch.update() now clears favicon.* and the filename cache when the URL changes, which
covers both the UI edit form and the API. favicon_is_expired() then sees no file and
the next check fetches the new icon. A change from an empty URL is ignored, since every
fresh Watch object starts with url '' and is filled from storage via update().
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Add tests for Jinja2 template detection in URL validation (RED)
* Fix Jinja2 false-positive when URL contains `{%` (GREEN)
* Consolidate Jinja2 template detection across URL/body/headers validation
* Propagate `FORCE_COLOR` to test containers
* Apply suggestion from @dgtlmoon
* Apply suggestion from @dgtlmoon
* Only render request headers/body at fetch time when they contain Jinja2, drop FORCE_COLOR changes
The edit form now skips Jinja2 validation for text like {%22 that is not a template,
but processors/base.py still rendered every header and the body unconditionally, so
a body such as q={%22key%22:1} was accepted on save and then failed every check with
a TemplateSyntaxError. Use JINJA2_MARKER_PATTERN there too, so it is sent as-is.
Also reset the CI workflow and run_*_tests.sh scripts to master, the FORCE_COLOR
changes are unrelated to this fix.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: dgtlmoon <dgtlmoon@gmail.com>
Co-authored-by: dgtlmoon <leigh@morresi.net>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Judge favicon expiry on the icon the watch actually has, not whatever glob lists first
bump_favicon() names the saved file after the icon's type (favicon.png, favicon.ico, ...)
and did not remove a previous save under a different extension, so a watch whose icon
changed type ends up holding more than one favicon.*.
favicon_is_expired() then picked next(iter(glob.glob("favicon.*"))). glob order is
os.scandir order - stable, but decided by filename hash - so for some extension pairs the
stale file wins every single time, its age always exceeds the 24h threshold, and the
favicon is refetched on every check forever. Measured across realistic pairs, 3 of 10
lose that way:
old=.png new=.ico -> glob lists favicon.png first -> always expired
old=.ico new=.svg -> glob lists favicon.ico first -> always expired
old=.png new=.webp -> glob lists favicon.png first -> always expired
That is an in-page network fetch through the watch's proxy on every check, plus the
favicon fetcher's own sequential per-icon timeouts, for a favicon that is already saved
and current.
Fixed at both ends: pick the newest candidate by mtime, and have bump_favicon() drop
superseded files so the ambiguity stops arising.
Also tidied two things in the same function:
- os.path.isfile(None) raised a TypeError when the module-level filename cache said yes
after the file had been removed, which the broad except turned into a logger.critical()
for an entirely benign condition. The empty case is now handled explicitly.
- the broad except stays (this runs inline on the check path as an argument to
fetcher.run(), so it must never fail a watch) but logs at warning, since the only cost
of getting this wrong is one extra favicon fetch.
Tests fail on master (favicon.ico left behind; fresh icon reported expired) and pass here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Favicon - write the new icon before removing superseded ones, so a failed write keeps the old icon
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Watch.history ran os.path.realpath() twice and os.path.exists() once for every
line in history.txt (~10 syscalls per snapshot), on every watch at startup and
on every history read after that.
basename() already reduces each entry to a plain filename, so the only way out
of the watch data dir is a symlink. One os.scandir() of the data dir now
answers "does it exist", and only symlinks pay for a realpath() containment
check. scandir() never lists '.' or '..', so those are still rejected.
~13x faster on 200 watches x 500 snapshots (1.18s -> 0.09s, warm cache),
identical output. Adds tests for symlinks escaping/inside the data dir,
dangling symlinks, and missing snapshots/blank lines.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* PDF - Keep the original PDF of the newest two versions
The history snapshot of a PDF watch is only the pdftohtml text, and the "last fetched" .html.br
for a PDF is just a checksum, so the actual documents were never kept. Store {timestamp}.pdf
alongside the snapshot (same as {timestamp}.html.br does for HTML) and prune to the newest two
history entries, so the latest two versions are always available for a visual PDF compare.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Test - Browser PDF test, fix race with an immediate fallback recheck, check the original PDFs are kept
A duplicate queue entry (scheduler + import) can run the html_requests fallback check right
after the failed browser check, so by the time the test looks the error may already be cleared.
Accept both outcomes, the error message itself is covered in tests/test_pdf.py.
Also check both original PDF versions are stored, whichever fetcher got them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Fetchers - Define raw_content on the base fetcher, give a clear error when a fetcher can't supply the PDF
Fetchers that never set raw_content (puppeteer, playwright, selenium) crashed with
"'fetcher' object has no attribute 'raw_content'" when the response turned out to be a PDF.
Browsers render PDFs in their built-in viewer, so all they have is the viewer HTML; now the
processor reports that plainly instead of crashing or feeding HTML to pdftohtml.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Puppeteer - Capture the real PDF body when the page turns out to be a PDF
Chrome hands a PDF straight to its built-in viewer, after which page.content and
Network.getResponseBody only return the viewer's HTML. Pause main-frame document responses
with the CDP Fetch domain and, when the Content-Type is application/pdf, copy the body out
before it is continued - a single download, and nothing needs to be known in advance.
content/raw_content then match the requests fetcher's binary handling (md5 + original bytes)
so the existing PDF processing and checksum skip work unchanged. Normal HTML pages are
continued immediately, no measurable difference in fetch time.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Fetchers - Add supports_pdf capability, only force html_requests for PDFs when the fetcher can't supply them
PDF watches were always forced onto html_requests. Now that puppeteer can capture the original
PDF (useful when the session needs a real browser, Cloudflare etc), only fetchers without
supports_pdf fall back - both at fetch time (resolve_content_fetcher) and in Watch.get_fetch_backend.
Also save the content-type when a fetcher couldn't supply the PDF (it was only saved on success),
otherwise watch.is_pdf never became true for URLs not ending in .pdf and the watch failed on
every check instead of falling back to html_requests on the next one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Several places call gc.collect() after a check to keep C-level memory
(pyppeteer buffers, libxml2 documents, PIL, brotli) from accumulating.
Individually each is reasonable; run concurrently by many fetch workers they
become a storm. Every gc.collect() is a full stop-the-world pass that walks the
whole heap holding the GIL, so at FETCH_WORKERS=50 the process spends most of
its time stopped in the collector - which also starves each worker's asyncio
loop, leaving its CDP websocket unread.
Measured on 153 real puppeteer checks of a live site at FETCH_WORKERS=50, with
an `//div` include filter so the lxml document tree is realistic:
collects objects freed gc time checks/sec CPU/check RSS
one per call site 790 2,594,098 91.3s 0.766 1.373s 279.6MB
debounced to 1s 48 2,219,010 7.3s 1.433 0.731s 275.3MB
none at all 0 0 0.0s 1.503 0.685s 287.7MB
Debouncing keeps 86% of the reclamation for 6% of the collections: 1.9x the
throughput, half the CPU per check, gc down from 30.8% to 6.4% of wall time,
and a lower resident plateau than collecting every time. It works because the
collector is process-wide - any worker's collection breaks every other worker's
cycles too, so with many workers the calls are overwhelmingly redundant
duplicates rather than independently necessary.
Removing them entirely is slightly faster still, but it was the only
configuration whose RSS had not plateaued by the end of the run, so it is not
the default. EXPLICIT_GC_MIN_INTERVAL=0 restores the previous behaviour.
Collecting a younger generation was measured and rejected: gen 0 freed 1,136
objects against the full pass's 2,594,098, because objects surviving a 10-30s
fetch have already been promoted out of gen 0.
Two call sites are additionally fixed because they could never reclaim anything:
- Watch.py brotli: brotli.Compressor is not gc-tracked, so the collector cannot
see it - `del` frees it by refcount. Over 60 x 2.2MB compressions, RSS growth
was +0.9MB with neither mechanism, +0.2MB with gc.collect() alone, and +0.0MB
with malloc_trim() alone or with both. malloc_trim is the load-bearing line
and is kept; the collect cost ~31ms of stop-the-world per snapshot save for no
reclamation.
- puppeteer quit(): runs twice per check (run()'s finally, then the worker's
safety net) and nulls self.page/self.browser in its own finally blocks, so the
second call closes nothing and breaks no cycles yet still paid for a full
collection - 88 calls across 51 checks. Now only collects when it actually
closed something.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Multi-language / Translations Support (#3696)
- Complete internationalization system implemented
- Support for 7 languages: Czech (cs), German (de), French (fr), Italian (it), Korean (ko), Chinese Simplified (zh), Chinese Traditional (zh_TW)
- Language selector with localized flags and theming
- Flash message translations
- Multiple translation fixes and improvements across all languages
- Language setting preserved across redirects
Pluggable Content Fetchers (#3653)
- New architecture for extensible content fetcher system
- Allows custom fetcher implementations
Image / Screenshot Comparison Processor (#3680)
- New processor for visual change detection (disabled for this release)
- Supporting CSS/JS infrastructure added
UI Improvements
Design & Layout
- Auto-generated tag color schemes
- Simplified login form styling
- Removed hard-coded CSS, moved to SCSS variables
- Tag UI cleanup and improvements
- Automatic tab wrapper functionality
- Menu refactoring for better organization
- Cleanup of offset settings
- Hide sticky tabs on narrow viewports
- Improved responsive layout (#3702)
User Experience
- Modal alerts/confirmations on delete/clear operations (#3693, #3598, #3382)
- Auto-add https:// to URLs in quickwatch form if not present
- Better redirect handling on login (#3699)
- 'Recheck all' now returns to correct group/tag (#3673)
- Language set redirect keeps hash fragment
- More friendly human-readable text throughout UI
Performance & Reliability
Scheduler & Processing
- Soft delays instead of blocking time.sleep() calls (#3710)
- More resilient handling of same UUID being processed (#3700)
- Better Puppeteer timeout handling
- Improved Puppeteer shutdown/cleanup (#3692)
- Requests cleanup now properly async
History & Rendering
- Faster server-side "difference" rendering on History page (#3442)
- Show ignored/triggered rows in history
- API: Retry watch data if watch dict changed (more reliable)
API Improvements
- Watch get endpoint: retry mechanism for changed watch data
- WatchHistoryDiff API endpoint includes extra format args (#3703)
Testing Improvements
- Replace time.sleep with wait_for_notification_endpoint_output (#3716)
- Test for mode switching (#3701)
- Test for #3720 added (#3725)
- Extract-text difference test fixes
- Improved dev workflow
Bug Fixes
- Notification error text output (#3672, #3669, #3280)
- HTML validation fixes (#3704)
- Template discovery path fixes
- Notification debug log now uses system locale for dates/times
- Puppeteer spelling mistake in log output
- Recalculation on anchor change
- Queue bubble update disabled temporarily
Dependency Updates
- beautifulsoup4 updated (#3724)
- psutil 7.1.0 → 7.2.1 (#3723)
- python-engineio ~=4.12.3 → ~=4.13.0 (#3707)
- python-socketio ~=5.14.3 → ~=5.16.0 (#3706)
- flask-socketio ~=5.5.1 → ~=5.6.0 (#3691)
- brotli ~=1.1 → ~=1.2 (#3687)
- lxml updated (#3590)
- pytest ~=7.2 → ~=9.0 (#3676)
- jsonschema ~=4.0 → ~=4.25 (#3618)
- pluggy ~=1.5 → ~=1.6 (#3616)
- cryptography 44.0.1 → 46.0.3 (security) (#3589)
Documentation
- README updated with viewport size setup information
Development Infrastructure
- Dev container only built on dev branch
- Improved dev workflow tooling
