Compare commits

...

24 Commits

Author SHA1 Message Date
dgtlmoon
fe5beac2b2 Small fix for 3.14 setup
Some checks failed
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Build distribution 📦 (push) Has been cancelled
ChangeDetection.io Container Build Test / Build linux/amd64 (alpine) (push) Has been cancelled
ChangeDetection.io Container Build Test / Build linux/arm64 (alpine) (push) Has been cancelled
ChangeDetection.io Container Build Test / Build linux/amd64 (main) (push) Has been cancelled
ChangeDetection.io Container Build Test / Build linux/arm/v7 (main) (push) Has been cancelled
ChangeDetection.io Container Build Test / Build linux/arm/v8 (main) (push) Has been cancelled
ChangeDetection.io Container Build Test / Build linux/arm64 (main) (push) Has been cancelled
ChangeDetection.io App Test / lint-code (push) Has been cancelled
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Test the built package works basically. (push) Has been cancelled
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Publish Python 🐍 distribution 📦 to PyPI (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-10 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-11 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-12 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-13 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-14 (push) Has been cancelled
2026-01-16 09:40:07 +01:00
dgtlmoon
5a1d44dc62 Merge branch 'master' into python-314 2026-01-16 09:23:23 +01:00
dependabot[bot]
6db1085337 Bump elementpath from 5.0.4 to 5.1.0 (#3754) 2026-01-16 09:22:10 +01:00
吾爱分享
66553e106d Update zh translations with improved, consistent Simplified Chinese UI copy. (#3752) 2026-01-16 09:21:29 +01:00
dependabot[bot]
5b01dbd9f8 Bump apprise from 1.9.5 to 1.9.6 (#3753) 2026-01-16 09:09:02 +01:00
dgtlmoon
c86f214fc3 0.52.6
Some checks failed
Build and push containers / metadata (push) Has been cancelled
Build and push containers / build-push-containers (push) Has been cancelled
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Build distribution 📦 (push) Has been cancelled
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Test the built package works basically. (push) Has been cancelled
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Publish Python 🐍 distribution 📦 to PyPI (push) Has been cancelled
ChangeDetection.io App Test / lint-code (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-10 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-11 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-12 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-13 (push) Has been cancelled
2026-01-15 22:28:58 +01:00
dgtlmoon
32149640d9 Selenium fetcher - Small fix for #3748 RGB error on transparent screenshots or similar (#3749) 2026-01-15 20:56:53 +01:00
dgtlmoon
15f16455fc UI - Show queue size above watch table in realtime
Some checks failed
Build and push containers / metadata (push) Has been cancelled
Build and push containers / build-push-containers (push) Has been cancelled
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Build distribution 📦 (push) Has been cancelled
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Test the built package works basically. (push) Has been cancelled
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Publish Python 🐍 distribution 📦 to PyPI (push) Has been cancelled
ChangeDetection.io App Test / lint-code (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-10 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-11 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-12 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-13 (push) Has been cancelled
ChangeDetection.io Container Build Test / Build linux/amd64 (alpine) (push) Has been cancelled
ChangeDetection.io Container Build Test / Build linux/arm64 (alpine) (push) Has been cancelled
ChangeDetection.io Container Build Test / Build linux/amd64 (main) (push) Has been cancelled
ChangeDetection.io Container Build Test / Build linux/arm/v7 (main) (push) Has been cancelled
ChangeDetection.io Container Build Test / Build linux/arm/v8 (main) (push) Has been cancelled
ChangeDetection.io Container Build Test / Build linux/arm64 (main) (push) Has been cancelled
2026-01-15 17:28:09 +01:00
dgtlmoon
15cdfac9d9 0.52.5 2026-01-15 14:07:09 +01:00
dgtlmoon
04de397916 Revert sub-process brotli saving because it could fork-bomb/use up too many system resources (#3747) 2026-01-15 13:56:08 +01:00
dgtlmoon
4643082c5b i18n: Recompile zh_Hant_TW/LC_MESSAGES/messages.mo 2026-01-15 13:21:49 +01:00
滅ü
3b2b74e62d i18n: Update zh_Hant_TW translations (#3745) 2026-01-15 13:12:25 +01:00
dependabot[bot]
68354cf53d Update jsonschema requirement from ~=4.25 to ~=4.26 (#3743) 2026-01-15 13:03:16 +01:00
dgtlmoon
3e364e0eba Translations - ZH_Hant_TW - Fixing timeago string handling #3737
Some checks failed
Build and push containers / metadata (push) Has been cancelled
Build and push containers / build-push-containers (push) Has been cancelled
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Build distribution 📦 (push) Has been cancelled
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Test the built package works basically. (push) Has been cancelled
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Publish Python 🐍 distribution 📦 to PyPI (push) Has been cancelled
ChangeDetection.io Container Build Test / Build linux/amd64 (alpine) (push) Has been cancelled
ChangeDetection.io Container Build Test / Build linux/arm64 (alpine) (push) Has been cancelled
ChangeDetection.io Container Build Test / Build linux/amd64 (main) (push) Has been cancelled
ChangeDetection.io Container Build Test / Build linux/arm/v7 (main) (push) Has been cancelled
ChangeDetection.io Container Build Test / Build linux/arm/v8 (main) (push) Has been cancelled
ChangeDetection.io Container Build Test / Build linux/arm64 (main) (push) Has been cancelled
ChangeDetection.io App Test / lint-code (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-10 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-11 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-12 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-13 (push) Has been cancelled
2026-01-15 12:24:53 +01:00
dgtlmoon
06ea29bfc7 Translations - Fixing zh_TW to zh_Hant_TW , adding tests #3737 (#3744) 2026-01-15 12:01:12 +01:00
dependabot[bot]
f4e178955c Bump pyppeteer-ng from 2.0.0rc10 to 2.0.0rc11 (#3742) 2026-01-15 10:31:42 +01:00
dgtlmoon
51d531d732 0.52.4
Some checks failed
Build and push containers / metadata (push) Has been cancelled
Build and push containers / build-push-containers (push) Has been cancelled
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Build distribution 📦 (push) Has been cancelled
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Test the built package works basically. (push) Has been cancelled
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Publish Python 🐍 distribution 📦 to PyPI (push) Has been cancelled
ChangeDetection.io App Test / lint-code (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-10 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-11 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-12 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-13 (push) Has been cancelled
CodeQL / Analyze (javascript) (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
2026-01-14 13:26:23 +01:00
dgtlmoon
e40c4ca97d Fixing Traditional Chinese locale mapping #3737 (#3738) 2026-01-14 13:26:07 +01:00
dgtlmoon
b8ede70f3a Languages - Pypi/pip package was missing translations 2026-01-14 13:09:23 +01:00
dgtlmoon
50b349b464 0.52.3
Some checks failed
Build and push containers / metadata (push) Has been cancelled
Build and push containers / build-push-containers (push) Has been cancelled
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Build distribution 📦 (push) Has been cancelled
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Test the built package works basically. (push) Has been cancelled
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Publish Python 🐍 distribution 📦 to PyPI (push) Has been cancelled
ChangeDetection.io App Test / lint-code (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-10 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-11 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-12 (push) Has been cancelled
ChangeDetection.io App Test / test-application-3-13 (push) Has been cancelled
2026-01-14 12:00:54 +01:00
dgtlmoon
67d097cca7 UI - Groups - Adding 'Recheck' button from groups overview page 2026-01-14 11:59:42 +01:00
dgtlmoon
494385a379 Minor playwright memory cleanup improvements (#3736) 2026-01-14 11:54:53 +01:00
dgtlmoon
c2ee84b753 Browser Steps UI async_loop bug, refactored startup of BrowserSteps, increased test coverage. Re #3734 (#3735) 2026-01-14 11:27:01 +01:00
dgtlmoon
9421f7e279 Python 3.14 test #3662 2025-11-30 18:13:24 +01:00
30 changed files with 1509 additions and 2148 deletions

View File

@@ -52,4 +52,13 @@ jobs:
uses: ./.github/workflows/test-stack-reusable-workflow.yml uses: ./.github/workflows/test-stack-reusable-workflow.yml
with: with:
python-version: '3.13' python-version: '3.13'
skip-pypuppeteer: true skip-pypuppeteer: true
test-application-3-14:
#if: github.event_name == 'push' && github.ref == 'refs/heads/master'
needs: lint-code
uses: ./.github/workflows/test-stack-reusable-workflow.yml
with:
python-version: '3.14'
skip-pypuppeteer: false

View File

@@ -11,6 +11,7 @@ recursive-include changedetectionio/realtime *
recursive-include changedetectionio/static * recursive-include changedetectionio/static *
recursive-include changedetectionio/templates * recursive-include changedetectionio/templates *
recursive-include changedetectionio/tests * recursive-include changedetectionio/tests *
recursive-include changedetectionio/translations *
recursive-include changedetectionio/widgets * recursive-include changedetectionio/widgets *
prune changedetectionio/static/package-lock.json prune changedetectionio/static/package-lock.json
prune changedetectionio/static/styles/node_modules prune changedetectionio/static/styles/node_modules

View File

@@ -2,7 +2,7 @@
# Read more https://github.com/dgtlmoon/changedetection.io/wiki # Read more https://github.com/dgtlmoon/changedetection.io/wiki
# Semver means never use .01, or 00. Should be .1. # Semver means never use .01, or 00. Should be .1.
__version__ = '0.52.2' __version__ = '0.52.6'
from changedetectionio.strtobool import strtobool from changedetectionio.strtobool import strtobool
from json.decoder import JSONDecodeError from json.decoder import JSONDecodeError
@@ -41,9 +41,10 @@ from loguru import logger
# #
# IMPLEMENTATION: # IMPLEMENTATION:
# 1. Explicit contexts everywhere (primary protection): # 1. Explicit contexts everywhere (primary protection):
# - Watch.py: ctx = multiprocessing.get_context('spawn')
# - playwright.py: ctx = multiprocessing.get_context('spawn') # - playwright.py: ctx = multiprocessing.get_context('spawn')
# - puppeteer.py: ctx = multiprocessing.get_context('spawn') # - puppeteer.py: ctx = multiprocessing.get_context('spawn')
# - isolated_opencv.py: ctx = multiprocessing.get_context('spawn')
# - isolated_libvips.py: ctx = multiprocessing.get_context('spawn')
# #
# 2. Global default (defense-in-depth, below): # 2. Global default (defense-in-depth, below):
# - Safety net if future code forgets explicit context # - Safety net if future code forgets explicit context

View File

@@ -1,5 +1,4 @@
from blinker import signal from blinker import signal
from .processors.exceptions import ProcessorException from .processors.exceptions import ProcessorException
import changedetectionio.content_fetchers.exceptions as content_fetchers_exceptions import changedetectionio.content_fetchers.exceptions as content_fetchers_exceptions
from changedetectionio.processors.text_json_diff.processor import FilterNotFoundInResponse from changedetectionio.processors.text_json_diff.processor import FilterNotFoundInResponse
@@ -9,7 +8,6 @@ from changedetectionio.flask_app import watch_check_update
import asyncio import asyncio
import importlib import importlib
import os import os
import queue
import time import time
from loguru import logger from loguru import logger

View File

@@ -21,31 +21,154 @@ from changedetectionio.flask_app import login_optionally_required
from loguru import logger from loguru import logger
browsersteps_sessions = {} browsersteps_sessions = {}
browsersteps_watch_to_session = {} # Maps watch_uuid -> browsersteps_session_id
io_interface_context = None io_interface_context = None
import json import json
import hashlib import hashlib
from flask import Response from flask import Response
import asyncio import asyncio
import threading import threading
import time
def run_async_in_browser_loop(coro): # Dedicated event loop for ALL browser steps sessions
"""Run async coroutine using the existing async worker event loop""" _browser_steps_loop = None
from changedetectionio import worker_handler _browser_steps_thread = None
_browser_steps_loop_lock = threading.Lock()
# Use the existing async worker event loop instead of creating a new one
if worker_handler.USE_ASYNC_WORKERS and worker_handler.async_loop and not worker_handler.async_loop.is_closed(): def _start_browser_steps_loop():
logger.debug("Browser steps using existing async worker event loop") """Start a dedicated event loop for browser steps in its own thread"""
future = asyncio.run_coroutine_threadsafe(coro, worker_handler.async_loop) global _browser_steps_loop
return future.result()
else: # Create and set the event loop for this thread
# Fallback: create a new event loop (for sync workers or if async loop not available) loop = asyncio.new_event_loop()
logger.debug("Browser steps creating temporary event loop") asyncio.set_event_loop(loop)
loop = asyncio.new_event_loop() _browser_steps_loop = loop
asyncio.set_event_loop(loop)
logger.debug("Browser steps event loop started")
try:
# Run the loop forever - handles all browsersteps sessions
loop.run_forever()
except Exception as e:
logger.error(f"Browser steps event loop error: {e}")
finally:
try: try:
return loop.run_until_complete(coro) # Cancel all remaining tasks
pending = asyncio.all_tasks(loop)
for task in pending:
task.cancel()
# Wait for tasks to finish cancellation
if pending:
loop.run_until_complete(asyncio.gather(*pending, return_exceptions=True))
except Exception as e:
logger.debug(f"Error during browser steps loop cleanup: {e}")
finally: finally:
loop.close() loop.close()
logger.debug("Browser steps event loop closed")
def _ensure_browser_steps_loop():
"""Ensure the browser steps event loop is running"""
global _browser_steps_loop, _browser_steps_thread
with _browser_steps_loop_lock:
if _browser_steps_thread is None or not _browser_steps_thread.is_alive():
logger.debug("Starting browser steps event loop thread")
_browser_steps_thread = threading.Thread(
target=_start_browser_steps_loop,
daemon=True,
name="BrowserStepsEventLoop"
)
_browser_steps_thread.start()
# Wait for the loop to be ready
timeout = 5.0
start_time = time.time()
while _browser_steps_loop is None:
if time.time() - start_time > timeout:
raise RuntimeError("Browser steps event loop failed to start")
time.sleep(0.01)
logger.debug("Browser steps event loop thread started and ready")
def run_async_in_browser_loop(coro):
"""Run async coroutine using the dedicated browser steps event loop"""
_ensure_browser_steps_loop()
if _browser_steps_loop and not _browser_steps_loop.is_closed():
logger.debug("Browser steps using dedicated event loop")
future = asyncio.run_coroutine_threadsafe(coro, _browser_steps_loop)
return future.result()
else:
raise RuntimeError("Browser steps event loop is not available")
def cleanup_expired_sessions():
"""Remove expired browsersteps sessions and cleanup their resources"""
global browsersteps_sessions, browsersteps_watch_to_session
expired_session_ids = []
# Find expired sessions
for session_id, session_data in browsersteps_sessions.items():
browserstepper = session_data.get('browserstepper')
if browserstepper and browserstepper.has_expired:
expired_session_ids.append(session_id)
# Cleanup expired sessions
for session_id in expired_session_ids:
logger.debug(f"Cleaning up expired browsersteps session {session_id}")
session_data = browsersteps_sessions[session_id]
# Cleanup playwright resources asynchronously
browserstepper = session_data.get('browserstepper')
if browserstepper:
try:
run_async_in_browser_loop(browserstepper.cleanup())
except Exception as e:
logger.error(f"Error cleaning up session {session_id}: {e}")
# Remove from sessions dict
del browsersteps_sessions[session_id]
# Remove from watch mapping
for watch_uuid, mapped_session_id in list(browsersteps_watch_to_session.items()):
if mapped_session_id == session_id:
del browsersteps_watch_to_session[watch_uuid]
break
if expired_session_ids:
logger.info(f"Cleaned up {len(expired_session_ids)} expired browsersteps session(s)")
def cleanup_session_for_watch(watch_uuid):
"""Cleanup a specific browsersteps session for a watch UUID"""
global browsersteps_sessions, browsersteps_watch_to_session
session_id = browsersteps_watch_to_session.get(watch_uuid)
if not session_id:
logger.debug(f"No browsersteps session found for watch {watch_uuid}")
return
logger.debug(f"Cleaning up browsersteps session {session_id} for watch {watch_uuid}")
session_data = browsersteps_sessions.get(session_id)
if session_data:
browserstepper = session_data.get('browserstepper')
if browserstepper:
try:
run_async_in_browser_loop(browserstepper.cleanup())
except Exception as e:
logger.error(f"Error cleaning up session {session_id} for watch {watch_uuid}: {e}")
# Remove from sessions dict
del browsersteps_sessions[session_id]
# Remove from watch mapping
del browsersteps_watch_to_session[watch_uuid]
logger.debug(f"Cleaned up session for watch {watch_uuid}")
# Opportunistically cleanup any other expired sessions
cleanup_expired_sessions()
def construct_blueprint(datastore: ChangeDetectionStore): def construct_blueprint(datastore: ChangeDetectionStore):
browser_steps_blueprint = Blueprint('browser_steps', __name__, template_folder="templates") browser_steps_blueprint = Blueprint('browser_steps', __name__, template_folder="templates")
@@ -123,6 +246,9 @@ def construct_blueprint(datastore: ChangeDetectionStore):
if not watch_uuid: if not watch_uuid:
return make_response('No Watch UUID specified', 500) return make_response('No Watch UUID specified', 500)
# Cleanup any existing session for this watch
cleanup_session_for_watch(watch_uuid)
logger.debug("Starting connection with playwright") logger.debug("Starting connection with playwright")
logger.debug("browser_steps.py connecting") logger.debug("browser_steps.py connecting")
@@ -131,6 +257,10 @@ def construct_blueprint(datastore: ChangeDetectionStore):
browsersteps_sessions[browsersteps_session_id] = run_async_in_browser_loop( browsersteps_sessions[browsersteps_session_id] = run_async_in_browser_loop(
start_browsersteps_session(watch_uuid) start_browsersteps_session(watch_uuid)
) )
# Store the mapping of watch_uuid -> browsersteps_session_id
browsersteps_watch_to_session[watch_uuid] = browsersteps_session_id
except Exception as e: except Exception as e:
if 'ECONNREFUSED' in str(e): if 'ECONNREFUSED' in str(e):
return make_response('Unable to start the Playwright Browser session, is sockpuppetbrowser running? Network configuration is OK?', 401) return make_response('Unable to start the Playwright Browser session, is sockpuppetbrowser running? Network configuration is OK?', 401)

View File

@@ -50,7 +50,8 @@
<td>{{ "{:,}".format(tag_count[uuid]) if uuid in tag_count else 0 }}</td> <td>{{ "{:,}".format(tag_count[uuid]) if uuid in tag_count else 0 }}</td>
<td class="title-col inline"> <a href="{{url_for('watchlist.index', tag=uuid) }}">{{ tag.title }}</a></td> <td class="title-col inline"> <a href="{{url_for('watchlist.index', tag=uuid) }}">{{ tag.title }}</a></td>
<td> <td>
<a class="pure-button pure-button-primary" href="{{ url_for('tags.form_tag_edit', uuid=uuid) }}">{{ _('Edit') }}</a>&nbsp; <a class="pure-button pure-button-primary" href="{{ url_for('tags.form_tag_edit', uuid=uuid) }}">{{ _('Edit') }}</a>
<a href="{{ url_for('ui.form_watch_checknow', tag=uuid) }}" class="pure-button pure-button-primary" >{{ _('Recheck') }}</a>
<a class="pure-button button-error" <a class="pure-button button-error"
href="{{ url_for('tags.delete', uuid=uuid) }}" href="{{ url_for('tags.delete', uuid=uuid) }}"
data-requires-confirm data-requires-confirm

View File

@@ -238,6 +238,13 @@ def construct_blueprint(datastore: ChangeDetectionStore, update_q, queuedWatchMe
datastore.data['watching'][uuid] = watch_class(datastore_path=datastore.datastore_path, default=datastore.data['watching'][uuid]) datastore.data['watching'][uuid] = watch_class(datastore_path=datastore.datastore_path, default=datastore.data['watching'][uuid])
flash(gettext("Updated watch - unpaused!") if request.args.get('unpause_on_save') else gettext("Updated watch.")) flash(gettext("Updated watch - unpaused!") if request.args.get('unpause_on_save') else gettext("Updated watch."))
# Cleanup any browsersteps session for this watch
try:
from changedetectionio.blueprint.browser_steps import cleanup_session_for_watch
cleanup_session_for_watch(uuid)
except Exception as e:
logger.debug(f"Error cleaning up browsersteps session: {e}")
# Re #286 - We wait for syncing new data to disk in another thread every 60 seconds # Re #286 - We wait for syncing new data to disk in another thread every 60 seconds
# But in the case something is added we should save straight away # But in the case something is added we should save straight away
datastore.needs_write_urgent = True datastore.needs_write_urgent = True
@@ -325,8 +332,6 @@ def construct_blueprint(datastore: ChangeDetectionStore, update_q, queuedWatchMe
'url': url_for('rss.rss_single_watch', uuid=watch['uuid'], token=app_rss_token) 'url': url_for('rss.rss_single_watch', uuid=watch['uuid'], token=app_rss_token)
}, },
'settings_application': datastore.data['settings']['application'], 'settings_application': datastore.data['settings']['application'],
'system_has_playwright_configured': os.getenv('PLAYWRIGHT_DRIVER_URL'),
'system_has_webdriver_configured': os.getenv('WEBDRIVER_URL'),
'ui_edit_stats_extras': collect_ui_edit_stats_extras(watch), 'ui_edit_stats_extras': collect_ui_edit_stats_extras(watch),
'visual_selector_data_ready': datastore.visualselector_data_is_ready(watch_uuid=uuid), 'visual_selector_data_ready': datastore.visualselector_data_is_ready(watch_uuid=uuid),
'timezone_default_config': datastore.data['settings']['application'].get('scheduler_timezone_default'), 'timezone_default_config': datastore.data['settings']['application'].get('scheduler_timezone_default'),

View File

@@ -206,7 +206,7 @@ Math: {{ 1 + 1 }}") }}
<div class="tab-pane-inner" id="browser-steps"> <div class="tab-pane-inner" id="browser-steps">
{% if capabilities.supports_browser_steps %} {% if capabilities.supports_browser_steps %}
{% if visual_selector_data_ready %} {% if true %}
<img class="beta-logo" src="{{url_for('static_content', group='images', filename='beta-logo.png')}}" alt="New beta functionality"> <img class="beta-logo" src="{{url_for('static_content', group='images', filename='beta-logo.png')}}" alt="New beta functionality">
<fieldset> <fieldset>
<div class="pure-control-group"> <div class="pure-control-group">

View File

@@ -2,7 +2,6 @@ import os
import time import time
from flask import Blueprint, request, make_response, render_template, redirect, url_for, flash, session from flask import Blueprint, request, make_response, render_template, redirect, url_for, flash, session
from flask_login import current_user
from flask_paginate import Pagination, get_page_parameter from flask_paginate import Pagination, get_page_parameter
from changedetectionio import forms from changedetectionio import forms
@@ -85,6 +84,7 @@ def construct_blueprint(datastore: ChangeDetectionStore, update_q, queuedWatchMe
app_rss_token=datastore.data['settings']['application'].get('rss_access_token'), app_rss_token=datastore.data['settings']['application'].get('rss_access_token'),
datastore=datastore, datastore=datastore,
errored_count=errored_count, errored_count=errored_count,
extra_classes='has-queue' if len(update_q.queue) else '',
form=form, form=form,
generate_tag_colors=processors.generate_processor_badge_colors, generate_tag_colors=processors.generate_processor_badge_colors,
guid=datastore.data['app_guid'], guid=datastore.data['app_guid'],
@@ -92,9 +92,10 @@ def construct_blueprint(datastore: ChangeDetectionStore, update_q, queuedWatchMe
hosted_sticky=os.getenv("SALTED_PASS", False) == False, hosted_sticky=os.getenv("SALTED_PASS", False) == False,
now_time_server=round(time.time()), now_time_server=round(time.time()),
pagination=pagination, pagination=pagination,
processor_badge_css=processors.get_processor_badge_css(),
processor_badge_texts=processors.get_processor_badge_texts(), processor_badge_texts=processors.get_processor_badge_texts(),
processor_descriptions=processors.get_processor_descriptions(), processor_descriptions=processors.get_processor_descriptions(),
processor_badge_css=processors.get_processor_badge_css(), queue_size=len(update_q.queue),
queued_uuids=[q_uuid.item['uuid'] for q_uuid in update_q.queue], queued_uuids=[q_uuid.item['uuid'] for q_uuid in update_q.queue],
search_q=request.args.get('q', '').strip(), search_q=request.args.get('q', '').strip(),
sort_attribute=request.args.get('sort') if request.args.get('sort') else request.cookies.get('sort'), sort_attribute=request.args.get('sort') if request.args.get('sort') else request.cookies.get('sort'),

View File

@@ -99,9 +99,14 @@ html[data-darkmode="true"] .watch-tag-list.tag-{{ class_name }} {
data-confirm-message="{{ _('<p>Are you sure you want to delete the selected watches?</strong></p><p>This action cannot be undone.</p>') }}" data-confirm-message="{{ _('<p>Are you sure you want to delete the selected watches?</strong></p><p>This action cannot be undone.</p>') }}"
data-confirm-button="{{ _('Delete') }}"><i data-feather="trash" style="width: 14px; height: 14px; stroke: white; margin-right: 4px;"></i>{{ _('Delete') }}</button> data-confirm-button="{{ _('Delete') }}"><i data-feather="trash" style="width: 14px; height: 14px; stroke: white; margin-right: 4px;"></i>{{ _('Delete') }}</button>
</div> </div>
{%- if watches|length >= pagination.per_page -%}
{{ pagination.info }} <div id="stats_row">
{%- endif -%} <div class="left">{%- if watches|length >= pagination.per_page -%}{{ pagination.info }}{%- endif -%}</div>
<div class="right" >{{ _('Queued size') }}: <span id="queue-size-int">{{ queue_size }}</span></div>
</div>
{%- if search_q -%}<div id="search-result-info">{{ _('Searching') }} "<strong><i>{{search_q}}</i></strong>"</div>{%- endif -%} {%- if search_q -%}<div id="search-result-info">{{ _('Searching') }} "<strong><i>{{search_q}}</i></strong>"</div>{%- endif -%}
<div> <div>
<a href="{{url_for('watchlist.index')}}" class="pure-button button-tag {{'active' if not active_tag_uuid }}">{{ _('All') }}</a> <a href="{{url_for('watchlist.index')}}" class="pure-button button-tag {{'active' if not active_tag_uuid }}">{{ _('All') }}</a>

View File

@@ -1,3 +1,4 @@
import gc
import json import json
import os import os
from urllib.parse import urlparse from urllib.parse import urlparse
@@ -185,20 +186,33 @@ class fetcher(Fetcher):
super().screenshot_step(step_n=step_n) super().screenshot_step(step_n=step_n)
screenshot = await capture_full_page_async(page=self.page, screenshot_format=self.screenshot_format) screenshot = await capture_full_page_async(page=self.page, screenshot_format=self.screenshot_format)
# Request GC immediately after screenshot to free memory
# Screenshots can be large and browser steps take many of them
await self.page.request_gc()
if self.browser_steps_screenshot_path is not None: if self.browser_steps_screenshot_path is not None:
destination = os.path.join(self.browser_steps_screenshot_path, 'step_{}.jpeg'.format(step_n)) destination = os.path.join(self.browser_steps_screenshot_path, 'step_{}.jpeg'.format(step_n))
logger.debug(f"Saving step screenshot to {destination}") logger.debug(f"Saving step screenshot to {destination}")
with open(destination, 'wb') as f: with open(destination, 'wb') as f:
f.write(screenshot) f.write(screenshot)
# Clear local reference to allow screenshot bytes to be collected
del screenshot
gc.collect()
async def save_step_html(self, step_n): async def save_step_html(self, step_n):
super().save_step_html(step_n=step_n) super().save_step_html(step_n=step_n)
content = await self.page.content() content = await self.page.content()
# Request GC after getting page content
await self.page.request_gc()
destination = os.path.join(self.browser_steps_screenshot_path, 'step_{}.html'.format(step_n)) destination = os.path.join(self.browser_steps_screenshot_path, 'step_{}.html'.format(step_n))
logger.debug(f"Saving step HTML to {destination}") logger.debug(f"Saving step HTML to {destination}")
with open(destination, 'w', encoding='utf-8') as f: with open(destination, 'w', encoding='utf-8') as f:
f.write(content) f.write(content)
# Clear local reference
del content
gc.collect()
async def run(self, async def run(self,
fetch_favicon=True, fetch_favicon=True,
@@ -305,6 +319,12 @@ class fetcher(Fetcher):
if self.status_code != 200 and not ignore_status_codes: if self.status_code != 200 and not ignore_status_codes:
screenshot = await capture_full_page_async(self.page, screenshot_format=self.screenshot_format) screenshot = await capture_full_page_async(self.page, screenshot_format=self.screenshot_format)
# Cleanup before raising to prevent memory leak
await self.page.close()
await context.close()
await browser.close()
# Force garbage collection to release Playwright resources immediately
gc.collect()
raise Non200ErrorCodeReceived(url=url, status_code=self.status_code, screenshot=screenshot) raise Non200ErrorCodeReceived(url=url, status_code=self.status_code, screenshot=screenshot)
if not empty_pages_are_a_change and len((await self.page.content()).strip()) == 0: if not empty_pages_are_a_change and len((await self.page.content()).strip()) == 0:
@@ -313,48 +333,52 @@ class fetcher(Fetcher):
await browser.close() await browser.close()
raise EmptyReply(url=url, status_code=response.status) raise EmptyReply(url=url, status_code=response.status)
# Run Browser Steps here # Wrap remaining operations in try/finally to ensure cleanup
if self.browser_steps_get_valid_steps():
await self.iterate_browser_steps(start_url=url)
await self.page.wait_for_timeout(extra_wait * 1000)
now = time.time()
# So we can find an element on the page where its selector was entered manually (maybe not xPath etc)
if current_include_filters is not None:
await self.page.evaluate("var include_filters={}".format(json.dumps(current_include_filters)))
else:
await self.page.evaluate("var include_filters=''")
await self.page.request_gc()
# request_gc before and after evaluate to free up memory
# @todo browsersteps etc
MAX_TOTAL_HEIGHT = int(os.getenv("SCREENSHOT_MAX_HEIGHT", SCREENSHOT_MAX_HEIGHT_DEFAULT))
self.xpath_data = await self.page.evaluate(XPATH_ELEMENT_JS, {
"visualselector_xpath_selectors": visualselector_xpath_selectors,
"max_height": MAX_TOTAL_HEIGHT
})
await self.page.request_gc()
self.instock_data = await self.page.evaluate(INSTOCK_DATA_JS)
await self.page.request_gc()
self.content = await self.page.content()
await self.page.request_gc()
logger.debug(f"Scrape xPath element data in browser done in {time.time() - now:.2f}s")
# Bug 3 in Playwright screenshot handling
# Some bug where it gives the wrong screenshot size, but making a request with the clip set first seems to solve it
# JPEG is better here because the screenshots can be very very large
# Screenshots also travel via the ws:// (websocket) meaning that the binary data is base64 encoded
# which will significantly increase the IO size between the server and client, it's recommended to use the lowest
# acceptable screenshot quality here
try: try:
# Run Browser Steps here
if self.browser_steps_get_valid_steps():
await self.iterate_browser_steps(start_url=url)
await self.page.wait_for_timeout(extra_wait * 1000)
now = time.time()
# So we can find an element on the page where its selector was entered manually (maybe not xPath etc)
if current_include_filters is not None:
await self.page.evaluate("var include_filters={}".format(json.dumps(current_include_filters)))
else:
await self.page.evaluate("var include_filters=''")
await self.page.request_gc()
# request_gc before and after evaluate to free up memory
# @todo browsersteps etc
MAX_TOTAL_HEIGHT = int(os.getenv("SCREENSHOT_MAX_HEIGHT", SCREENSHOT_MAX_HEIGHT_DEFAULT))
self.xpath_data = await self.page.evaluate(XPATH_ELEMENT_JS, {
"visualselector_xpath_selectors": visualselector_xpath_selectors,
"max_height": MAX_TOTAL_HEIGHT
})
await self.page.request_gc()
self.instock_data = await self.page.evaluate(INSTOCK_DATA_JS)
await self.page.request_gc()
self.content = await self.page.content()
await self.page.request_gc()
logger.debug(f"Scrape xPath element data in browser done in {time.time() - now:.2f}s")
# Bug 3 in Playwright screenshot handling
# Some bug where it gives the wrong screenshot size, but making a request with the clip set first seems to solve it
# JPEG is better here because the screenshots can be very very large
# Screenshots also travel via the ws:// (websocket) meaning that the binary data is base64 encoded
# which will significantly increase the IO size between the server and client, it's recommended to use the lowest
# acceptable screenshot quality here
# The actual screenshot - this always base64 and needs decoding! horrible! huge CPU usage # The actual screenshot - this always base64 and needs decoding! horrible! huge CPU usage
self.screenshot = await capture_full_page_async(page=self.page, screenshot_format=self.screenshot_format) self.screenshot = await capture_full_page_async(page=self.page, screenshot_format=self.screenshot_format)
except ScreenshotUnavailable:
# Re-raise screenshot unavailable exceptions
raise
except Exception as e: except Exception as e:
# It's likely the screenshot was too long/big and something crashed # It's likely the screenshot was too long/big and something crashed
raise ScreenshotUnavailable(url=url, status_code=self.status_code) raise ScreenshotUnavailable(url=url, status_code=self.status_code)
@@ -389,6 +413,10 @@ class fetcher(Fetcher):
pass pass
browser = None browser = None
# Force Python GC to release Playwright resources immediately
# Playwright objects can have circular references that delay cleanup
gc.collect()
# Plugin registration for built-in fetcher # Plugin registration for built-in fetcher
class PlaywrightFetcherPlugin: class PlaywrightFetcherPlugin:

View File

@@ -15,7 +15,7 @@ class fetcher(Fetcher):
proxy_url = None proxy_url = None
# Capability flags # Capability flags
supports_browser_steps = True supports_browser_steps = False
supports_screenshots = True supports_screenshots = True
supports_xpath_element_data = True supports_xpath_element_data = True
@@ -156,6 +156,9 @@ class fetcher(Fetcher):
from PIL import Image from PIL import Image
import io import io
img = Image.open(io.BytesIO(screenshot_png)) img = Image.open(io.BytesIO(screenshot_png))
# Convert to RGB if needed (JPEG doesn't support transparency)
if img.mode != 'RGB':
img = img.convert('RGB')
jpeg_buffer = io.BytesIO() jpeg_buffer = io.BytesIO()
img.save(jpeg_buffer, format='JPEG', quality=int(os.getenv("SCREENSHOT_QUALITY", 72))) img.save(jpeg_buffer, format='JPEG', quality=int(os.getenv("SCREENSHOT_QUALITY", 72)))
self.screenshot = jpeg_buffer.getvalue() self.screenshot = jpeg_buffer.getvalue()

View File

@@ -9,6 +9,7 @@ import threading
import time import time
import timeago import timeago
from blinker import signal from blinker import signal
from pathlib import Path
from changedetectionio.strtobool import strtobool from changedetectionio.strtobool import strtobool
from threading import Event from threading import Event
@@ -84,6 +85,10 @@ app.config['NEW_VERSION_AVAILABLE'] = False
if os.getenv('FLASK_SERVER_NAME'): if os.getenv('FLASK_SERVER_NAME'):
app.config['SERVER_NAME'] = os.getenv('FLASK_SERVER_NAME') app.config['SERVER_NAME'] = os.getenv('FLASK_SERVER_NAME')
# Babel/i18n configuration
app.config['BABEL_TRANSLATION_DIRECTORIES'] = str(Path(__file__).parent / 'translations')
app.config['BABEL_DEFAULT_LOCALE'] = 'en_GB'
#app.config["EXPLAIN_TEMPLATE_LOADING"] = True #app.config["EXPLAIN_TEMPLATE_LOADING"] = True
@@ -395,13 +400,9 @@ def changedetection_app(config=None, datastore_o=None):
def get_locale(): def get_locale():
# 1. Try to get locale from session (user explicitly selected) # 1. Try to get locale from session (user explicitly selected)
if 'locale' in session: if 'locale' in session:
locale = session['locale'] return session['locale']
logger.trace(f"DEBUG: get_locale() returning from session: {locale}")
return locale
# 2. Fall back to Accept-Language header # 2. Fall back to Accept-Language header
locale = request.accept_languages.best_match(language_codes) return request.accept_languages.best_match(language_codes)
logger.trace(f"DEBUG: get_locale() returning from Accept-Language: {locale}")
return locale
# Initialize Babel with locale selector # Initialize Babel with locale selector
babel = Babel(app, locale_selector=get_locale) babel = Babel(app, locale_selector=get_locale)
@@ -518,9 +519,20 @@ def changedetection_app(config=None, datastore_o=None):
@app.route('/set-language/<locale>') @app.route('/set-language/<locale>')
def set_language(locale): def set_language(locale):
"""Set the user's preferred language in the session""" """Set the user's preferred language in the session"""
if not request.cookies:
logger.error("Cannot set language without session cookie")
flash("Cannot set language without session cookie", 'error')
return redirect(url_for('watchlist.index'))
# Validate the locale against available languages # Validate the locale against available languages
if locale in language_codes: if locale in language_codes:
session['locale'] = locale session['locale'] = locale
# CRITICAL: Flask-Babel caches the locale in the request context (ctx.babel_locale)
# We must refresh to clear this cache so the new locale takes effect immediately
# This is especially important for tests where multiple requests happen rapidly
from flask_babel import refresh
refresh()
else: else:
logger.error(f"Invalid locale {locale}, available: {language_codes}") logger.error(f"Invalid locale {locale}, available: {language_codes}")

View File

@@ -29,6 +29,9 @@ def get_timeago_locale(flask_locale):
""" """
locale_map = { locale_map = {
'zh': 'zh_CN', # Chinese Simplified 'zh': 'zh_CN', # Chinese Simplified
# timeago library just hasn't been updated to use the more modern locale naming convention, before BCP 47 / RFC 5646.
'zh_TW': 'zh_TW', # Chinese Traditional (timeago uses zh_TW)
'zh_Hant_TW': 'zh_TW', # Flask-Babel normalizes zh_TW to zh_Hant_TW, map back to timeago's zh_TW
'pt': 'pt_PT', # Portuguese (Portugal) 'pt': 'pt_PT', # Portuguese (Portugal)
'sv': 'sv_SE', # Swedish 'sv': 'sv_SE', # Swedish
'no': 'nb_NO', # Norwegian Bokmål 'no': 'nb_NO', # Norwegian Bokmål
@@ -53,7 +56,7 @@ LANGUAGE_DATA = {
'it': {'flag': 'fi fi-it fis', 'name': 'Italiano'}, 'it': {'flag': 'fi fi-it fis', 'name': 'Italiano'},
'ja': {'flag': 'fi fi-jp fis', 'name': '日本語'}, 'ja': {'flag': 'fi fi-jp fis', 'name': '日本語'},
'zh': {'flag': 'fi fi-cn fis', 'name': '中文 (简体)'}, 'zh': {'flag': 'fi fi-cn fis', 'name': '中文 (简体)'},
'zh_TW': {'flag': 'fi fi-tw fis', 'name': '繁體中文'}, 'zh_Hant_TW': {'flag': 'fi fi-tw fis', 'name': '繁體中文'},
'ru': {'flag': 'fi fi-ru fis', 'name': 'Русский'}, 'ru': {'flag': 'fi fi-ru fis', 'name': 'Русский'},
'pl': {'flag': 'fi fi-pl fis', 'name': 'Polski'}, 'pl': {'flag': 'fi fi-pl fis', 'name': 'Polski'},
'nl': {'flag': 'fi fi-nl fis', 'name': 'Nederlands'}, 'nl': {'flag': 'fi fi-nl fis', 'name': 'Nederlands'},

View File

@@ -18,21 +18,31 @@ BROTLI_COMPRESS_SIZE_THRESHOLD = int(os.getenv('SNAPSHOT_BROTLI_COMPRESSION_THRE
minimum_seconds_recheck_time = int(os.getenv('MINIMUM_SECONDS_RECHECK_TIME', 3)) minimum_seconds_recheck_time = int(os.getenv('MINIMUM_SECONDS_RECHECK_TIME', 3))
mtable = {'seconds': 1, 'minutes': 60, 'hours': 3600, 'days': 86400, 'weeks': 86400 * 7} mtable = {'seconds': 1, 'minutes': 60, 'hours': 3600, 'days': 86400, 'weeks': 86400 * 7}
def _brotli_compress_worker(conn, filepath, mode=None): def _brotli_save(contents, filepath, mode=None, fallback_uncompressed=False):
""" """
Worker function to compress data with brotli in a separate process. Save compressed data using native brotli.
This isolates memory - when process exits, OS reclaims all memory. Testing shows no memory leak when using gc.collect() after compression.
Args: Args:
conn: multiprocessing.Pipe connection to receive data contents: data to compress (str or bytes)
filepath: destination file path filepath: destination file path
mode: brotli compression mode (e.g., brotli.MODE_TEXT) mode: brotli compression mode (e.g., brotli.MODE_TEXT)
fallback_uncompressed: if True, save uncompressed on failure; if False, raise exception
Returns:
str: actual filepath saved (may differ from input if fallback used)
Raises:
Exception: if compression fails and fallback_uncompressed is False
""" """
import brotli import brotli
import gc
# Ensure contents are bytes
if isinstance(contents, str):
contents = contents.encode('utf-8')
try: try:
# Receive data from parent process via pipe (avoids pickle overhead)
contents = conn.recv()
logger.debug(f"Starting brotli compression of {len(contents)} bytes.") logger.debug(f"Starting brotli compression of {len(contents)} bytes.")
if mode is not None: if mode is not None:
@@ -43,111 +53,25 @@ def _brotli_compress_worker(conn, filepath, mode=None):
with open(filepath, 'wb') as f: with open(filepath, 'wb') as f:
f.write(compressed_data) f.write(compressed_data)
# Send success status back
conn.send(True)
logger.debug(f"Finished brotli compression - From {len(contents)} to {len(compressed_data)} bytes.") logger.debug(f"Finished brotli compression - From {len(contents)} to {len(compressed_data)} bytes.")
# No need for explicit cleanup - process exit frees all memory
except Exception as e:
logger.critical(f"Brotli compression worker failed: {e}")
conn.send(False)
finally:
conn.close()
# Force garbage collection to prevent memory buildup
gc.collect()
def _brotli_subprocess_save(contents, filepath, mode=None, timeout=30, fallback_uncompressed=False): return filepath
"""
Save compressed data using subprocess to isolate memory.
Uses Pipe to avoid pickle overhead for large data.
Args:
contents: data to compress (str or bytes)
filepath: destination file path
mode: brotli compression mode (e.g., brotli.MODE_TEXT)
timeout: subprocess timeout in seconds
fallback_uncompressed: if True, save uncompressed on failure; if False, raise exception
Returns:
str: actual filepath saved (may differ from input if fallback used)
Raises:
Exception: if compression fails and fallback_uncompressed is False
"""
import multiprocessing
import sys
# Ensure contents are bytes
if isinstance(contents, str):
contents = contents.encode('utf-8')
# Use explicit spawn context for thread safety (avoids fork() with multi-threaded parent)
# Always use spawn - consistent behavior in tests and production
ctx = multiprocessing.get_context('spawn')
parent_conn, child_conn = ctx.Pipe()
# Run compression in subprocess using spawn (not fork)
proc = ctx.Process(target=_brotli_compress_worker, args=(child_conn, filepath, mode))
# Windows-safe: Set daemon=False explicitly to avoid issues with process cleanup
proc.daemon = False
proc.start()
try:
# Send data to subprocess via pipe (avoids pickle)
parent_conn.send(contents)
# Wait for result with timeout
if parent_conn.poll(timeout):
success = parent_conn.recv()
else:
success = False
logger.warning(f"Brotli compression subprocess timed out after {timeout}s")
# Graceful termination with platform-aware cleanup
try:
proc.terminate()
except Exception as term_error:
logger.debug(f"Process termination issue (may be normal on Windows): {term_error}")
parent_conn.close()
proc.join(timeout=5)
# Force kill if still alive after graceful termination
if proc.is_alive():
try:
if sys.platform == 'win32':
# Windows: use kill() which is more forceful
proc.kill()
else:
# Unix: terminate() already sent SIGTERM, now try SIGKILL
proc.kill()
proc.join(timeout=2)
except Exception as kill_error:
logger.warning(f"Failed to kill brotli compression process: {kill_error}")
# Check if file was created successfully
if success and os.path.exists(filepath):
return filepath
except Exception as e: except Exception as e:
logger.error(f"Brotli compression error: {e}") logger.error(f"Brotli compression error: {e}")
try:
parent_conn.close()
except:
pass
try:
proc.terminate()
proc.join(timeout=2)
except:
pass
# Compression failed # Compression failed
if fallback_uncompressed: if fallback_uncompressed:
logger.warning(f"Brotli compression failed for {filepath}, saving uncompressed") logger.warning(f"Brotli compression failed for {filepath}, saving uncompressed")
fallback_path = filepath.replace('.br', '') fallback_path = filepath.replace('.br', '')
with open(fallback_path, 'wb') as f: with open(fallback_path, 'wb') as f:
f.write(contents) f.write(contents)
return fallback_path return fallback_path
else: else:
raise Exception(f"Brotli compression subprocess failed for {filepath}") raise Exception(f"Brotli compression failed for {filepath}: {e}")
class model(watch_base): class model(watch_base):
@@ -523,7 +447,7 @@ class model(watch_base):
if not os.path.exists(dest): if not os.path.exists(dest):
try: try:
actual_dest = _brotli_subprocess_save(contents, dest, mode=brotli.MODE_TEXT, fallback_uncompressed=True) actual_dest = _brotli_save(contents, dest, mode=brotli.MODE_TEXT, fallback_uncompressed=True)
if actual_dest != dest: if actual_dest != dest:
snapshot_fname = os.path.basename(actual_dest) snapshot_fname = os.path.basename(actual_dest)
except Exception as e: except Exception as e:
@@ -949,13 +873,13 @@ class model(watch_base):
def save_last_text_fetched_before_filters(self, contents): def save_last_text_fetched_before_filters(self, contents):
import brotli import brotli
filepath = os.path.join(self.watch_data_dir, 'last-fetched.br') filepath = os.path.join(self.watch_data_dir, 'last-fetched.br')
_brotli_subprocess_save(contents, filepath, mode=brotli.MODE_TEXT, fallback_uncompressed=False) _brotli_save(contents, filepath, mode=brotli.MODE_TEXT, fallback_uncompressed=False)
def save_last_fetched_html(self, timestamp, contents): def save_last_fetched_html(self, timestamp, contents):
self.ensure_data_dir_exists() self.ensure_data_dir_exists()
snapshot_fname = f"{timestamp}.html.br" snapshot_fname = f"{timestamp}.html.br"
filepath = os.path.join(self.watch_data_dir, snapshot_fname) filepath = os.path.join(self.watch_data_dir, snapshot_fname)
_brotli_subprocess_save(contents, filepath, mode=None, fallback_uncompressed=True) _brotli_save(contents, filepath, mode=None, fallback_uncompressed=True)
self._prune_last_fetched_html_snapshots() self._prune_last_fetched_html_snapshots()
def get_fetched_html(self, timestamp): def get_fetched_html(self, timestamp):

View File

@@ -13,14 +13,9 @@ Research: https://github.com/libvips/pyvips/issues/234
import multiprocessing import multiprocessing
# CRITICAL: Use 'spawn' instead of 'fork' to avoid inheriting parent's # CRITICAL: Use 'spawn' context instead of 'fork' to avoid inheriting parent's
# LibVIPS threading state which can cause hangs in gaussblur operations # LibVIPS threading state which can cause hangs in gaussblur operations
# https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods # https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods
try:
multiprocessing.set_start_method('spawn', force=False)
except RuntimeError:
# Already set, ignore
pass
def _worker_generate_diff(conn, img_bytes_from, img_bytes_to, threshold, blur_sigma, max_width, max_height): def _worker_generate_diff(conn, img_bytes_from, img_bytes_to, threshold, blur_sigma, max_width, max_height):
@@ -95,9 +90,10 @@ def generate_diff_isolated(img_bytes_from, img_bytes_to, threshold, blur_sigma,
Returns: Returns:
bytes: JPEG diff image or None on failure bytes: JPEG diff image or None on failure
""" """
parent_conn, child_conn = multiprocessing.Pipe() ctx = multiprocessing.get_context('spawn')
parent_conn, child_conn = ctx.Pipe()
p = multiprocessing.Process( p = ctx.Process(
target=_worker_generate_diff, target=_worker_generate_diff,
args=(child_conn, img_bytes_from, img_bytes_to, threshold, blur_sigma, max_width, max_height) args=(child_conn, img_bytes_from, img_bytes_to, threshold, blur_sigma, max_width, max_height)
) )
@@ -140,7 +136,8 @@ def calculate_change_percentage_isolated(img_bytes_from, img_bytes_to, threshold
Returns: Returns:
float: Change percentage float: Change percentage
""" """
parent_conn, child_conn = multiprocessing.Pipe() ctx = multiprocessing.get_context('spawn')
parent_conn, child_conn = ctx.Pipe()
def _worker_calculate(conn): def _worker_calculate(conn):
try: try:
@@ -185,7 +182,7 @@ def calculate_change_percentage_isolated(img_bytes_from, img_bytes_to, threshold
finally: finally:
conn.close() conn.close()
p = multiprocessing.Process(target=_worker_calculate, args=(child_conn,)) p = ctx.Process(target=_worker_calculate, args=(child_conn,))
p.start() p.start()
result = 0.0 result = 0.0
@@ -233,7 +230,8 @@ def compare_images_isolated(img_bytes_from, img_bytes_to, threshold, blur_sigma,
tuple: (changed_detected, change_percentage) tuple: (changed_detected, change_percentage)
""" """
print(f"[Parent] Starting compare_images_isolated subprocess", flush=True) print(f"[Parent] Starting compare_images_isolated subprocess", flush=True)
parent_conn, child_conn = multiprocessing.Pipe() ctx = multiprocessing.get_context('spawn')
parent_conn, child_conn = ctx.Pipe()
def _worker_compare(conn): def _worker_compare(conn):
try: try:
@@ -301,7 +299,7 @@ def compare_images_isolated(img_bytes_from, img_bytes_to, threshold, blur_sigma,
finally: finally:
conn.close() conn.close()
p = multiprocessing.Process(target=_worker_compare, args=(child_conn,)) p = ctx.Process(target=_worker_compare, args=(child_conn,))
print(f"[Parent] Starting subprocess (pid will be assigned)", flush=True) print(f"[Parent] Starting subprocess (pid will be assigned)", flush=True)
p.start() p.start()
print(f"[Parent] Subprocess started (pid={p.pid}), waiting for result (30s timeout)", flush=True) print(f"[Parent] Subprocess started (pid={p.pid}), waiting for result (30s timeout)", flush=True)

View File

@@ -76,7 +76,7 @@ $(document).ready(function () {
// Cache DOM elements for performance // Cache DOM elements for performance
const queueBubble = document.getElementById('queue-bubble'); const queueBubble = document.getElementById('queue-bubble');
const queueSizePagerInfoText = document.getElementById('queue-size-int');
// Only try to connect if authentication isn't required or user is authenticated // Only try to connect if authentication isn't required or user is authenticated
// The 'is_authenticated' variable will be set in the template // The 'is_authenticated' variable will be set in the template
if (typeof is_authenticated !== 'undefined' ? is_authenticated : true) { if (typeof is_authenticated !== 'undefined' ? is_authenticated : true) {
@@ -118,6 +118,10 @@ $(document).ready(function () {
socket.on('queue_size', function (data) { socket.on('queue_size', function (data) {
console.log(`${data.event_timestamp} - Queue size update: ${data.q_length}`); console.log(`${data.event_timestamp} - Queue size update: ${data.q_length}`);
if(queueSizePagerInfoText) {
queueSizePagerInfoText.textContent = parseInt(data.q_length).toLocaleString() || 'None';
}
document.body.classList.toggle('has-queue', parseInt(data.q_length) > 0);
// Update queue bubble in action sidebar // Update queue bubble in action sidebar
//if (queueBubble) { //if (queueBubble) {

View File

@@ -1,6 +1,4 @@
.pagination-page-info { .pagination-page-info {
color: #fff;
font-size: 0.85rem;
text-transform: capitalize; text-transform: capitalize;
} }

View File

@@ -1,4 +1,32 @@
/* table related */ /* table related */
#stats_row {
display: flex;
align-items: center;
width: 100%;
color: #fff;
font-size: 0.85rem;
>* {
padding-bottom: 0.5rem;
}
.left {
text-align: left;
}
.right {
opacity: 0.5;
transition: opacity 0.6s ease;
margin-left: auto; /* pushes it to the far right */
text-align: right;
}
}
body.has-queue {
#stats_row {
.right {
opacity: 1.0;
}
}
}
.watch-table { .watch-table {
width: 100%; width: 100%;
font-size: 80%; font-size: 80%;

File diff suppressed because one or more lines are too long

View File

@@ -1,5 +1,5 @@
<!DOCTYPE html> <!DOCTYPE html>
<html lang="{{ get_locale() }}" data-darkmode="{{ get_darkmode_state() }}"> <html lang="{{ get_locale()|replace('_', '-') }}" data-darkmode="{{ get_darkmode_state() }}">
<head> <head>
<meta charset="utf-8" > <meta charset="utf-8" >

View File

@@ -2,6 +2,7 @@
import psutil import psutil
import time import time
from threading import Thread from threading import Thread
import multiprocessing
import pytest import pytest
import arrow import arrow
@@ -97,6 +98,34 @@ def cleanup(datastore_path):
if os.path.isfile(f): if os.path.isfile(f):
os.unlink(f) os.unlink(f)
def pytest_configure(config):
"""Configure pytest environment before tests run.
CRITICAL: Set multiprocessing start method to 'fork' for Python 3.14+ compatibility.
Python 3.14 changed the default start method from 'fork' to 'forkserver' on Linux.
The forkserver method requires all objects to be picklable, but pytest-flask's
LiveServer uses nested functions that can't be pickled.
Setting 'fork' explicitly:
- Maintains compatibility with Python 3.10-3.13 (where 'fork' was already default)
- Fixes Python 3.14 pickling errors
- Only affects Unix-like systems (Windows uses 'spawn' regardless)
See: https://github.com/python/cpython/issues/126831
See: https://docs.python.org/3/whatsnew/3.14.html
"""
# Only set if not already set (respects existing configuration)
if multiprocessing.get_start_method(allow_none=True) is None:
try:
# 'fork' is available on Unix-like systems (Linux, macOS)
# On Windows, this will have no effect as 'spawn' is the only option
multiprocessing.set_start_method('fork', force=False)
logger.debug("Set multiprocessing start method to 'fork' for Python 3.14+ compatibility")
except (ValueError, RuntimeError):
# Already set, not available on this platform, or context already created
pass
def pytest_addoption(parser): def pytest_addoption(parser):
"""Add custom command-line options for pytest. """Add custom command-line options for pytest.

View File

@@ -1,7 +1,71 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
from flask import url_for from flask import url_for
from .util import live_server_setup from .util import live_server_setup, wait_for_all_checks
def test_zh_TW(client, live_server, measure_memory_usage, datastore_path):
import time
test_url = url_for('test_endpoint', _external=True)
# Be sure we got a session cookie
res = client.get(url_for("watchlist.index"), follow_redirects=True)
res = client.get(
url_for("set_language", locale="zh_Hant_TW"), # Traditional
follow_redirects=True
)
# HTML follows BCP 47 language tag rules, not underscore-based locale formats.
assert b'<html lang="zh-Hant-TW"' in res.data
assert b'Cannot set language without session cookie' not in res.data
assert '選擇語言'.encode() in res.data
# Check second set works
res = client.get(
url_for("set_language", locale="en_GB"),
follow_redirects=True
)
assert b'Cannot set language without session cookie' not in res.data
res = client.get(url_for("watchlist.index"), follow_redirects=True)
assert b"Select Language" in res.data, "Second set of language worked"
# Check arbitration between zh_Hant_TW<->zh
res = client.get(
url_for("set_language", locale="zh"), # Simplified chinese
follow_redirects=True
)
res = client.get(url_for("watchlist.index"), follow_redirects=True)
assert "选择语言".encode() in res.data, "Simplified chinese worked and it means the flask-babel cache worked"
# timeago library just hasn't been updated to use the more modern locale naming convention, before BCP 47 / RFC 5646.
# The Python timeago library (https://github.com/hustcc/timeago) supports 48 locales but uses different naming conventions than Flask-Babel.
def test_zh_Hant_TW_timeago_integration():
"""Test that zh_Hant_TW mapping works and timeago renders Traditional Chinese correctly"""
import timeago
from datetime import datetime, timedelta
from changedetectionio.languages import get_timeago_locale
# 1. Test the mapping
mapped_locale = get_timeago_locale('zh_Hant_TW')
assert mapped_locale == 'zh_TW', "zh_Hant_TW should map to timeago's zh_TW"
assert get_timeago_locale('zh_TW') == 'zh_TW', "zh_TW should also map to zh_TW"
# 2. Test timeago library renders Traditional Chinese with the mapped locale
now = datetime.now()
# Test various time periods with Traditional Chinese strings
result_15s = timeago.format(now - timedelta(seconds=15), now, mapped_locale)
assert '秒前' in result_15s, f"Expected '秒前' in '{result_15s}'"
result_5m = timeago.format(now - timedelta(minutes=5), now, mapped_locale)
assert '分鐘前' in result_5m, f"Expected '分鐘前' in '{result_5m}'"
result_2h = timeago.format(now - timedelta(hours=2), now, mapped_locale)
assert '小時前' in result_2h, f"Expected '小時前' in '{result_2h}'"
result_3d = timeago.format(now - timedelta(days=3), now, mapped_locale)
assert '天前' in result_3d, f"Expected '天前' in '{result_3d}'"
def test_language_switching(client, live_server, measure_memory_usage, datastore_path): def test_language_switching(client, live_server, measure_memory_usage, datastore_path):
@@ -13,6 +77,9 @@ def test_language_switching(client, live_server, measure_memory_usage, datastore
3. Switch back to English and verify English text appears 3. Switch back to English and verify English text appears
""" """
# Establish session cookie
client.get(url_for("watchlist.index"), follow_redirects=True)
# Step 1: Set the language to Italian using the /set-language endpoint # Step 1: Set the language to Italian using the /set-language endpoint
res = client.get( res = client.get(
url_for("set_language", locale="it"), url_for("set_language", locale="it"),
@@ -61,6 +128,9 @@ def test_invalid_locale(client, live_server, measure_memory_usage, datastore_pat
The app should ignore invalid locales and continue working. The app should ignore invalid locales and continue working.
""" """
# Establish session cookie
client.get(url_for("watchlist.index"), follow_redirects=True)
# First set to English # First set to English
res = client.get( res = client.get(
url_for("set_language", locale="en"), url_for("set_language", locale="en"),
@@ -93,6 +163,9 @@ def test_language_persistence_in_session(client, live_server, measure_memory_usa
within the same session. within the same session.
""" """
# Establish session cookie
client.get(url_for("watchlist.index"), follow_redirects=True)
# Set language to Italian # Set language to Italian
res = client.get( res = client.get(
url_for("set_language", locale="it"), url_for("set_language", locale="it"),
@@ -119,6 +192,9 @@ def test_set_language_with_redirect(client, live_server, measure_memory_usage, d
""" """
from flask import url_for from flask import url_for
# Establish session cookie
client.get(url_for("watchlist.index"), follow_redirects=True)
# Set language with a redirect parameter (simulating language change from /settings) # Set language with a redirect parameter (simulating language change from /settings)
res = client.get( res = client.get(
url_for("set_language", locale="de", redirect="/settings"), url_for("set_language", locale="de", redirect="/settings"),

View File

@@ -144,7 +144,6 @@ def test_basic_browserstep(client, live_server, measure_memory_usage, datastore_
def test_non_200_errors_report_browsersteps(client, live_server, measure_memory_usage, datastore_path): def test_non_200_errors_report_browsersteps(client, live_server, measure_memory_usage, datastore_path):
four_o_four_url = url_for('test_endpoint', status_code=404, _external=True) four_o_four_url = url_for('test_endpoint', status_code=404, _external=True)
four_o_four_url = four_o_four_url.replace('localhost.localdomain', 'cdio') four_o_four_url = four_o_four_url.replace('localhost.localdomain', 'cdio')
four_o_four_url = four_o_four_url.replace('localhost', 'cdio') four_o_four_url = four_o_four_url.replace('localhost', 'cdio')
@@ -186,3 +185,65 @@ def test_non_200_errors_report_browsersteps(client, live_server, measure_memory_
url_for("ui.form_delete", uuid="all"), url_for("ui.form_delete", uuid="all"),
follow_redirects=True follow_redirects=True
) )
def test_browsersteps_edit_UI_startsession(client, live_server, measure_memory_usage, datastore_path):
assert os.getenv('PLAYWRIGHT_DRIVER_URL'), "Needs PLAYWRIGHT_DRIVER_URL set for this test"
# Add a watch first
test_url = url_for('test_interactive_html_endpoint', _external=True)
test_url = test_url.replace('localhost.localdomain', 'cdio')
test_url = test_url.replace('localhost', 'cdio')
uuid = client.application.config.get('DATASTORE').add_watch(url=test_url, extras={'fetch_backend': 'html_webdriver', 'paused': True})
# Test starting a browsersteps session
res = client.get(
url_for("browser_steps.browsersteps_start_session", uuid=uuid),
follow_redirects=True
)
assert res.status_code == 200
assert res.is_json
json_data = res.get_json()
assert 'browsersteps_session_id' in json_data
assert json_data['browsersteps_session_id'] # Not empty
browsersteps_session_id = json_data['browsersteps_session_id']
# Verify the session exists in browsersteps_sessions
from changedetectionio.blueprint.browser_steps import browsersteps_sessions, browsersteps_watch_to_session
assert browsersteps_session_id in browsersteps_sessions
assert uuid in browsersteps_watch_to_session
assert browsersteps_watch_to_session[uuid] == browsersteps_session_id
# Verify browsersteps UI shows up on edit page
res = client.get(url_for("ui.ui_edit.edit_page", uuid=uuid))
assert b'browsersteps-click-start' in res.data, "Browsersteps manual UI shows up"
# Session should still exist after GET (not cleaned up yet)
assert browsersteps_session_id in browsersteps_sessions
assert uuid in browsersteps_watch_to_session
# Test cleanup happens on save (POST)
res = client.post(
url_for("ui.ui_edit.edit_page", uuid=uuid),
data={
"url": test_url,
"tags": "",
'fetch_backend': "html_webdriver",
"time_between_check_use_default": "y",
},
follow_redirects=True
)
assert b"Updated watch" in res.data
# NOW verify the session was cleaned up after save
assert browsersteps_session_id not in browsersteps_sessions
assert uuid not in browsersteps_watch_to_session
# Cleanup
client.get(
url_for("ui.form_delete", uuid="all"),
follow_redirects=True
)

File diff suppressed because it is too large Load Diff

View File

@@ -42,7 +42,7 @@ orjson~=3.11
# jq not available on Windows so must be installed manually # jq not available on Windows so must be installed manually
# Notification library # Notification library
apprise==1.9.5 apprise==1.9.6
diff_match_patch diff_match_patch
@@ -70,7 +70,7 @@ lxml >=4.8.0,!=5.2.0,!=5.2.1,<7
# XPath 2.0-3.1 support - 4.2.0 had issues, 4.1.5 stable # XPath 2.0-3.1 support - 4.2.0 had issues, 4.1.5 stable
# Consider updating to latest stable version periodically # Consider updating to latest stable version periodically
elementpath==5.0.4 elementpath==5.1.0
# For fast image comparison in screenshot change detection # For fast image comparison in screenshot change detection
# opencv-python-headless is OPTIONAL (excluded from requirements.txt) # opencv-python-headless is OPTIONAL (excluded from requirements.txt)
@@ -91,7 +91,7 @@ jq~=1.3; python_version >= "3.8" and sys_platform == "linux"
# playwright is installed at Dockerfile build time because it's not available on all platforms # playwright is installed at Dockerfile build time because it's not available on all platforms
pyppeteer-ng==2.0.0rc10 pyppeteer-ng==2.0.0rc11
pyppeteerstealth>=0.0.4 pyppeteerstealth>=0.0.4
# Include pytest, so if theres a support issue we can ask them to run these tests on their setup # Include pytest, so if theres a support issue we can ask them to run these tests on their setup
@@ -100,7 +100,7 @@ pytest-flask ~=1.3
pytest-mock ~=3.15 pytest-mock ~=3.15
# Anything 4.0 and up but not 5.0 # Anything 4.0 and up but not 5.0
jsonschema ~= 4.25 jsonschema ~= 4.26
# OpenAPI validation support # OpenAPI validation support
openapi-core[flask] >= 0.19.0 openapi-core[flask] >= 0.19.0