23 Commits

Include renames

Author	SHA1	Message	Date
dgtlmoonandClaude Opus 5	06446fa26b	Debounce the explicit gc.collect() storm on the watch-check path (#4431) ... Several places call gc.collect() after a check to keep C-level memory (pyppeteer buffers, libxml2 documents, PIL, brotli) from accumulating. Individually each is reasonable; run concurrently by many fetch workers they become a storm. Every gc.collect() is a full stop-the-world pass that walks the whole heap holding the GIL, so at FETCH_WORKERS=50 the process spends most of its time stopped in the collector - which also starves each worker's asyncio loop, leaving its CDP websocket unread. Measured on 153 real puppeteer checks of a live site at FETCH_WORKERS=50, with an `//div` include filter so the lxml document tree is realistic: collects objects freed gc time checks/sec CPU/check RSS one per call site 790 2,594,098 91.3s 0.766 1.373s 279.6MB debounced to 1s 48 2,219,010 7.3s 1.433 0.731s 275.3MB none at all 0 0 0.0s 1.503 0.685s 287.7MB Debouncing keeps 86% of the reclamation for 6% of the collections: 1.9x the throughput, half the CPU per check, gc down from 30.8% to 6.4% of wall time, and a lower resident plateau than collecting every time. It works because the collector is process-wide - any worker's collection breaks every other worker's cycles too, so with many workers the calls are overwhelmingly redundant duplicates rather than independently necessary. Removing them entirely is slightly faster still, but it was the only configuration whose RSS had not plateaued by the end of the run, so it is not the default. EXPLICIT_GC_MIN_INTERVAL=0 restores the previous behaviour. Collecting a younger generation was measured and rejected: gen 0 freed 1,136 objects against the full pass's 2,594,098, because objects surviving a 10-30s fetch have already been promoted out of gen 0. Two call sites are additionally fixed because they could never reclaim anything: - Watch.py brotli: brotli.Compressor is not gc-tracked, so the collector cannot see it - `del` frees it by refcount. Over 60 x 2.2MB compressions, RSS growth was +0.9MB with neither mechanism, +0.2MB with gc.collect() alone, and +0.0MB with malloc_trim() alone or with both. malloc_trim is the load-bearing line and is kept; the collect cost ~31ms of stop-the-world per snapshot save for no reclamation. - puppeteer quit(): runs twice per check (run()'s finally, then the worker's safety net) and nulls self.page/self.browser in its own finally blocks, so the second call closes nothing and breaks no cycles yet still paid for a full collection - 88 calls across 51 checks. Now only collects when it actually closed something. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-14 00:11:47 +02:00
dgtlmoon	900a77e828	Worker id/available fetcher (#4415) ... * Worker ID # should be available to the content fetchers * Actually init with the worker ID	2026-09-12 16:04:39 +02:00
dgtlmoon	97317398ac	LLM - Per watch/group settings and UI tidyup (#4354) ... * UI - LLM section tidyup * Rebuild translatiosn * UI - Fixing language for prompt adjustement to be more clear * UI - clarify field action * UI - clarify layout for sections * WIP * test tweaks	2026-09-02 10:07:34 +02:00
dgtlmoon	71d332d5a0	UI full refactor / upgrade (#4183)	2026-07-16 15:08:29 +02:00
Ehsan	5eb2638ffc	fix: report the actual 1-based failed browser-step number (#4200) (#4258)	2026-07-14 14:48:36 +02:00
dgtlmoon	0e3f1941b3	Code - LLM settings pydantic refactor (#4181)	2026-05-25 10:43:52 +02:00
dgtlmoon	701833b6ed	UI - LLM - Flag LLM_FEATURES_DISABLED to disable all LLM from the UI/system (#4171)	2026-05-21 12:51:56 +02:00
dgtlmoon	9d9a58e763	LLM - Master on/off switch (enable/disable) (#4162)	2026-05-19 19:05:13 +02:00
dgtlmoon	c765285026	LLM - UI & Ollama tweaks (#4148)	2026-05-16 10:18:24 +02:00
dgtlmoon	bf5efc4c31	LLM - Fixing summary cache miss-hit (#4136)	2026-05-12 17:44:36 +02:00
dgtlmoon	fbe59f89b6	UI - Make LLM status sticky (#4135)	2026-05-12 15:28:33 +02:00
dgtlmoon	e9e8c8d218	LLM / AI Change detection rules and Summaries	2026-04-24 15:21:09 +02:00
dgtlmoon	722644b111	Handle inline favicons (#4047 #3891 )	2026-04-11 07:35:27 +02:00
dgtlmoon	c4b890f4fa	last_error should be cleared if page content was the same and there was no error (#3997)	2026-03-21 20:18:05 +01:00
dgtlmoon	2343ddd88a	Minor code tidy	2026-02-17 02:46:22 +01:00
dgtlmoon	23063ad8a1	UI - More fixes for realtime updates	2026-02-17 02:37:03 +01:00
dgtlmoon	27b8a2d178	UI - Fixing realtime updates for status updates when checking (#3889)	2026-02-17 02:26:38 +01:00
dgtlmoon	a53f2a784d	Pluggy plugin hook for before and after a watch is processed (#3888)	2026-02-17 01:58:41 +01:00
dgtlmoon	b6d3d63372	Avoid reprocessing if the page was the same (#3867)	2026-02-14 21:24:28 +01:00
dgtlmoon	dc8b387f40	History length limit size option (#3834)	2026-02-05 12:29:20 +01:00
dgtlmoon	83d7ce0fcf	Processor plugin improvements - Now supports creating your own processor (for example, monitor DNS changes) (#3739)	2026-02-04 14:23:08 +01:00
dgtlmoon	1aabf967ef	Puppeteer and Playwright browser close/shutdown improvements (#3830)	2026-02-03 11:14:51 +01:00
dgtlmoon	30dc4ac23b	Refactor of queue systenm and improve tests, improves multiple workers (#3826)	2026-02-02 22:28:27 +01:00