25 Commits

Include renames

Author	SHA1	Message	Date
dgtlmoonandClaude Opus 5.5	ce777a6d60	PDF - Keep the original PDF of the newest two versions (#4549) ... * PDF - Keep the original PDF of the newest two versions The history snapshot of a PDF watch is only the pdftohtml text, and the "last fetched" .html.br for a PDF is just a checksum, so the actual documents were never kept. Store {timestamp}.pdf alongside the snapshot (same as {timestamp}.html.br does for HTML) and prune to the newest two history entries, so the latest two versions are always available for a visual PDF compare. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Test - Browser PDF test, fix race with an immediate fallback recheck, check the original PDFs are kept A duplicate queue entry (scheduler + import) can run the html_requests fallback check right after the failed browser check, so by the time the test looks the error may already be cleared. Accept both outcomes, the error message itself is covered in tests/test_pdf.py. Also check both original PDF versions are stored, whichever fetcher got them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-13 15:53:18 +02:00
dgtlmoonandClaude Opus 5.5	44fd71679e	Puppeteer fetcher - PDF improvements, Handle PDF fetching by Puppeteer directly, cleanup supports (#4547) ... * Fetchers - Define raw_content on the base fetcher, give a clear error when a fetcher can't supply the PDF Fetchers that never set raw_content (puppeteer, playwright, selenium) crashed with "'fetcher' object has no attribute 'raw_content'" when the response turned out to be a PDF. Browsers render PDFs in their built-in viewer, so all they have is the viewer HTML; now the processor reports that plainly instead of crashing or feeding HTML to pdftohtml. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Puppeteer - Capture the real PDF body when the page turns out to be a PDF Chrome hands a PDF straight to its built-in viewer, after which page.content and Network.getResponseBody only return the viewer's HTML. Pause main-frame document responses with the CDP Fetch domain and, when the Content-Type is application/pdf, copy the body out before it is continued - a single download, and nothing needs to be known in advance. content/raw_content then match the requests fetcher's binary handling (md5 + original bytes) so the existing PDF processing and checksum skip work unchanged. Normal HTML pages are continued immediately, no measurable difference in fetch time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Fetchers - Add supports_pdf capability, only force html_requests for PDFs when the fetcher can't supply them PDF watches were always forced onto html_requests. Now that puppeteer can capture the original PDF (useful when the session needs a real browser, Cloudflare etc), only fetchers without supports_pdf fall back - both at fetch time (resolve_content_fetcher) and in Watch.get_fetch_backend. Also save the content-type when a fetcher couldn't supply the PDF (it was only saved on success), otherwise watch.is_pdf never became true for URLs not ending in .pdf and the watch failed on every check instead of falling back to html_requests on the next one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-13 11:50:43 +02:00
dgtlmoonandClaude Opus 5	06446fa26b	Debounce the explicit gc.collect() storm on the watch-check path (#4431) ... Several places call gc.collect() after a check to keep C-level memory (pyppeteer buffers, libxml2 documents, PIL, brotli) from accumulating. Individually each is reasonable; run concurrently by many fetch workers they become a storm. Every gc.collect() is a full stop-the-world pass that walks the whole heap holding the GIL, so at FETCH_WORKERS=50 the process spends most of its time stopped in the collector - which also starves each worker's asyncio loop, leaving its CDP websocket unread. Measured on 153 real puppeteer checks of a live site at FETCH_WORKERS=50, with an `//div` include filter so the lxml document tree is realistic: collects objects freed gc time checks/sec CPU/check RSS one per call site 790 2,594,098 91.3s 0.766 1.373s 279.6MB debounced to 1s 48 2,219,010 7.3s 1.433 0.731s 275.3MB none at all 0 0 0.0s 1.503 0.685s 287.7MB Debouncing keeps 86% of the reclamation for 6% of the collections: 1.9x the throughput, half the CPU per check, gc down from 30.8% to 6.4% of wall time, and a lower resident plateau than collecting every time. It works because the collector is process-wide - any worker's collection breaks every other worker's cycles too, so with many workers the calls are overwhelmingly redundant duplicates rather than independently necessary. Removing them entirely is slightly faster still, but it was the only configuration whose RSS had not plateaued by the end of the run, so it is not the default. EXPLICIT_GC_MIN_INTERVAL=0 restores the previous behaviour. Collecting a younger generation was measured and rejected: gen 0 freed 1,136 objects against the full pass's 2,594,098, because objects surviving a 10-30s fetch have already been promoted out of gen 0. Two call sites are additionally fixed because they could never reclaim anything: - Watch.py brotli: brotli.Compressor is not gc-tracked, so the collector cannot see it - `del` frees it by refcount. Over 60 x 2.2MB compressions, RSS growth was +0.9MB with neither mechanism, +0.2MB with gc.collect() alone, and +0.0MB with malloc_trim() alone or with both. malloc_trim is the load-bearing line and is kept; the collect cost ~31ms of stop-the-world per snapshot save for no reclamation. - puppeteer quit(): runs twice per check (run()'s finally, then the worker's safety net) and nulls self.page/self.browser in its own finally blocks, so the second call closes nothing and breaks no cycles yet still paid for a full collection - 88 calls across 51 checks. Now only collects when it actually closed something. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-14 00:11:47 +02:00
dgtlmoon	900a77e828	Worker id/available fetcher (#4415) ... * Worker ID # should be available to the content fetchers * Actually init with the worker ID	2026-09-12 16:04:39 +02:00
dgtlmoon	97317398ac	LLM - Per watch/group settings and UI tidyup (#4354) ... * UI - LLM section tidyup * Rebuild translatiosn * UI - Fixing language for prompt adjustement to be more clear * UI - clarify field action * UI - clarify layout for sections * WIP * test tweaks	2026-09-02 10:07:34 +02:00
dgtlmoon	71d332d5a0	UI full refactor / upgrade (#4183)	2026-07-16 15:08:29 +02:00
Ehsan	5eb2638ffc	fix: report the actual 1-based failed browser-step number (#4200) (#4258)	2026-07-14 14:48:36 +02:00
dgtlmoon	0e3f1941b3	Code - LLM settings pydantic refactor (#4181)	2026-05-25 10:43:52 +02:00
dgtlmoon	701833b6ed	UI - LLM - Flag LLM_FEATURES_DISABLED to disable all LLM from the UI/system (#4171)	2026-05-21 12:51:56 +02:00
dgtlmoon	9d9a58e763	LLM - Master on/off switch (enable/disable) (#4162)	2026-05-19 19:05:13 +02:00
dgtlmoon	c765285026	LLM - UI & Ollama tweaks (#4148)	2026-05-16 10:18:24 +02:00
dgtlmoon	bf5efc4c31	LLM - Fixing summary cache miss-hit (#4136)	2026-05-12 17:44:36 +02:00
dgtlmoon	fbe59f89b6	UI - Make LLM status sticky (#4135)	2026-05-12 15:28:33 +02:00
dgtlmoon	e9e8c8d218	LLM / AI Change detection rules and Summaries	2026-04-24 15:21:09 +02:00
dgtlmoon	722644b111	Handle inline favicons (#4047 #3891 )	2026-04-11 07:35:27 +02:00
dgtlmoon	c4b890f4fa	last_error should be cleared if page content was the same and there was no error (#3997)	2026-03-21 20:18:05 +01:00
dgtlmoon	2343ddd88a	Minor code tidy	2026-02-17 02:46:22 +01:00
dgtlmoon	23063ad8a1	UI - More fixes for realtime updates	2026-02-17 02:37:03 +01:00
dgtlmoon	27b8a2d178	UI - Fixing realtime updates for status updates when checking (#3889)	2026-02-17 02:26:38 +01:00
dgtlmoon	a53f2a784d	Pluggy plugin hook for before and after a watch is processed (#3888)	2026-02-17 01:58:41 +01:00
dgtlmoon	b6d3d63372	Avoid reprocessing if the page was the same (#3867)	2026-02-14 21:24:28 +01:00
dgtlmoon	dc8b387f40	History length limit size option (#3834)	2026-02-05 12:29:20 +01:00
dgtlmoon	83d7ce0fcf	Processor plugin improvements - Now supports creating your own processor (for example, monitor DNS changes) (#3739)	2026-02-04 14:23:08 +01:00
dgtlmoon	1aabf967ef	Puppeteer and Playwright browser close/shutdown improvements (#3830)	2026-02-03 11:14:51 +01:00
dgtlmoon	30dc4ac23b	Refactor of queue systenm and improve tests, improves multiple workers (#3826)	2026-02-02 22:28:27 +01:00