lee/changedetection.io
1
0 Fork 0
mirror of https://github.com/dgtlmoon/changedetection.io.git synced 2026-10-05 11:38:22 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
Files
e4c75bf486c8d9dfd32ea8b206ce48e788d61ea1
changedetection.io/changedetectionio/blueprint/ui/__init__.py
T
dgtlmoonandClaude Opus 5 ba2280ba81 External CDP Browser engine - replaces the extra_browser_* hackery 
An "extra browser" was a name + a ws(s):// endpoint in settings.requests.extra_browsers,
selected by a watch as the magic string 'extra_browser_<name>'. That string resolved to
html_webdriver plus a custom connection URL, which meant the protocol the endpoint was
spoken to came from env vars rather than from the entry: CDP over a WebSocket with
PLAYWRIGHT_DRIVER_URL set, CDP via pyppeteer with FAST_PUPPETEER_CHROME_FETCHER, and the
W3C WebDriver protocol over HTTP on a Selenium-only install - where a wss:// URL cannot
work at all. The form only ever accepted ws:// / wss://, so the feature was silently
broken on exactly the installs that could not honour it.

So it becomes an engine, html_external_cdp, which pins the protocol: a subclass of the
Playwright fetcher that takes its endpoint from the watch's browser config
(FetcherConfig.connection_url) instead of the environment. It is base-only
(ready_to_use=False) because an endpoint is required, so each endpoint is one browser
config ("variation") on the Browsers page - which is what the old settings list was.

update_36 migrates each extra_browsers row to such a variation, keyed by the SAME
'extra_browser_<name>' string watches already hold, so no watch, group override, API value
or global default needs rewriting; the legacy selector simply becomes a real browser-config
id. A row whose endpoint the model rejects is logged and skipped rather than taking the
update chain, and with it startup, down.

Knock-on cleanups, all of which delete a special case rather than add one:

 - The proxy opt-out for custom endpoints is now Fetcher.ignores_proxy_setting, asked of
   the engine, instead of a string-prefix test in call_browser().
 - A live browser-steps / visual-selector session asks the engine where to connect
   (Fetcher.browser_steps_connection_url, overridden by html_external_cdp) and refuses an
   engine whose supports_browser_steps is False, instead of reading the env var itself and
   silently stepping a browser the watch does not check with. That refusal is real: on a
   Selenium install html_webdriver cannot drive a live session.
 - is_valid_browser_selector() answers "may a watch store this in fetch_backend?" in one
   place; the API (create/update/import), the quick-add form validator and the bulk "set
   browser" operation each had their own copy, which is how they came to disagree about
   whether a browser-config id was acceptable.
 - api-spec.yaml's fetch_backend pattern enumerated extra_browser_* while rejecting
   browser-config ids and every engine newer than html_webdriver. Valid values are
   per-install, so the schema now bounds the string and the handlers do the real check.
 - html_external_cdp registers unconditionally (unlike html_playwright_builtin): migrated
   configs name it, so it must resolve even without the playwright library, or those
   watches would quietly fetch with the plain HTTP client. The library is imported lazily
   inside run(), and an unavailable engine now warns instead of falling back silently.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 19:39:16 +02:00

484 lines
23 KiB
Python
Raw Blame History

import time
import threading
from blinker import signal
from flask import Blueprint, request, redirect, url_for, flash, render_template, session, current_app, abort
from flask_babel import gettext
from loguru import logger
from changedetectionio.store import ChangeDetectionStore
from changedetectionio.blueprint.ui.edit import construct_blueprint as construct_edit_blueprint
from changedetectionio.blueprint.ui.notification import construct_blueprint as construct_notification_blueprint
from changedetectionio.blueprint.ui.views import construct_blueprint as construct_views_blueprint
from changedetectionio.blueprint.ui.queue import construct_blueprint as construct_queue_blueprint
from changedetectionio.blueprint.ui import diff, preview, browser_config
def _handle_operations(op, uuids, datastore, worker_pool, update_q, queuedWatchMetaData, watch_check_update, extra_data=None, emit_flash=True):
"""Apply a bulk operation to the given watch uuids.
Returns a {'message': str, 'type': 'success'|'error'} result describing the outcome so callers
(e.g. the socket.io handler) can surface server-driven feedback. When emit_flash is True the same
message is also flashed for the classic form-POST + redirect path.
"""
from flask import flash
# Result feedback - 'success' unless a branch overrides it (e.g. invalid input)
result_message = None
result_type = 'success'
if op == 'delete':
for uuid in uuids:
if datastore.data['watching'].get(uuid):
datastore.delete(uuid)
result_message = gettext("{} watches deleted").format(len(uuids))
elif op == 'pause':
for uuid in uuids:
if datastore.data['watching'].get(uuid):
datastore.data['watching'][uuid]['paused'] = True
datastore.data['watching'][uuid].commit()
result_message = gettext("{} watches paused").format(len(uuids))
elif op == 'unpause':
for uuid in uuids:
if datastore.data['watching'].get(uuid):
datastore.data['watching'][uuid.strip()]['paused'] = False
datastore.data['watching'][uuid].commit()
result_message = gettext("{} watches unpaused").format(len(uuids))
elif (op == 'mark-viewed'):
for uuid in uuids:
if datastore.data['watching'].get(uuid):
datastore.set_last_viewed(uuid, int(time.time()))
result_message = gettext("{} watches updated").format(len(uuids))
elif (op == 'mute'):
for uuid in uuids:
if datastore.data['watching'].get(uuid):
datastore.data['watching'][uuid]['notification_muted'] = True
datastore.data['watching'][uuid].commit()
result_message = gettext("{} watches muted").format(len(uuids))
elif (op == 'unmute'):
for uuid in uuids:
if datastore.data['watching'].get(uuid):
datastore.data['watching'][uuid]['notification_muted'] = False
datastore.data['watching'][uuid].commit()
result_message = gettext("{} watches un-muted").format(len(uuids))
elif (op == 'recheck'):
for uuid in uuids:
if datastore.data['watching'].get(uuid):
# Recheck and require a full reprocessing
worker_pool.queue_item_async_safe(update_q, queuedWatchMetaData.PrioritizedItem(priority=1, item={'uuid': uuid}))
result_message = gettext("{} watches queued for rechecking").format(len(uuids))
elif (op == 'clear-errors'):
for uuid in uuids:
if datastore.data['watching'].get(uuid):
datastore.data['watching'][uuid]["last_error"] = False
datastore.data['watching'][uuid].commit()
result_message = gettext("{} watches errors cleared").format(len(uuids))
elif (op == 'clear-history'):
for uuid in uuids:
if datastore.data['watching'].get(uuid):
datastore.clear_watch_history(uuid)
result_message = gettext("{} watches cleared/reset.").format(len(uuids))
elif (op == 'notification-default'):
from changedetectionio.notification import (
USE_SYSTEM_DEFAULT_NOTIFICATION_FORMAT_FOR_WATCH
)
for uuid in uuids:
if datastore.data['watching'].get(uuid):
datastore.data['watching'][uuid]['notification_title'] = None
datastore.data['watching'][uuid]['notification_body'] = None
datastore.data['watching'][uuid]['notification_urls'] = []
datastore.data['watching'][uuid]['notification_format'] = USE_SYSTEM_DEFAULT_NOTIFICATION_FORMAT_FOR_WATCH
datastore.data['watching'][uuid].commit()
result_message = gettext("{} watches set to use default notification settings").format(len(uuids))
elif (op == 'set-fetch-backend'):
# extra_data = the chosen "Browser": 'system', a built-in engine name, or a saved
# browser-config id - the same set as the edit picker.
from changedetectionio.model.browser_config import is_valid_browser_selector
if is_valid_browser_selector(extra_data, datastore, allow_empty=False):
for uuid in uuids:
if datastore.data['watching'].get(uuid):
datastore.data['watching'][uuid]['fetch_backend'] = extra_data
datastore.data['watching'][uuid].commit()
result_message = gettext("Browser / fetch method updated on {} watches").format(len(uuids))
else:
result_message = gettext("Invalid browser / fetch method selected")
result_type = 'error'
elif (op == 'set-proxy'):
# extra_data = proxy key; '' means the system default proxy
valid = [''] + (list(datastore.proxy_list.keys()) if datastore.proxy_list else [])
if (extra_data or '') in valid:
for uuid in uuids:
if datastore.data['watching'].get(uuid):
datastore.data['watching'][uuid]['proxy'] = (extra_data or '')
datastore.data['watching'][uuid].commit()
result_message = gettext("Proxy updated on {} watches").format(len(uuids))
else:
result_message = gettext("Invalid proxy selected")
result_type = 'error'
elif (op == 'assign-tag'):
op_extradata = extra_data
if op_extradata:
tag_uuid = datastore.add_tag(title=op_extradata)
if op_extradata and tag_uuid:
for uuid in uuids:
if datastore.data['watching'].get(uuid):
# Bug in old versions caused by bad edit page/tag handler
if isinstance(datastore.data['watching'][uuid]['tags'], str):
datastore.data['watching'][uuid]['tags'] = []
datastore.data['watching'][uuid]['tags'].append(tag_uuid)
datastore.data['watching'][uuid].commit()
result_message = gettext("{} watches were tagged").format(len(uuids))
if uuids:
for uuid in uuids:
watch_check_update.send(watch_uuid=uuid)
# Classic form-POST path flashes (shown after the redirect); the socket path uses the returned value
if result_message and emit_flash:
if result_type == 'error':
flash(result_message, 'error')
else:
flash(result_message)
return {'message': result_message, 'type': result_type} if result_message else None
def construct_blueprint(datastore: ChangeDetectionStore, update_q, worker_pool, queuedWatchMetaData, watch_check_update):
ui_blueprint = Blueprint('ui', __name__, template_folder="templates")
# Register the edit blueprint
edit_blueprint = construct_edit_blueprint(datastore, update_q, queuedWatchMetaData)
ui_blueprint.register_blueprint(edit_blueprint)
# Register the notification blueprint
notification_blueprint = construct_notification_blueprint(datastore)
ui_blueprint.register_blueprint(notification_blueprint)
# Register the views blueprint
views_blueprint = construct_views_blueprint(datastore, update_q, queuedWatchMetaData, watch_check_update)
ui_blueprint.register_blueprint(views_blueprint)
# Register diff and preview blueprints
diff_blueprint = diff.construct_blueprint(datastore)
ui_blueprint.register_blueprint(diff_blueprint)
preview_blueprint = preview.construct_blueprint(datastore)
ui_blueprint.register_blueprint(preview_blueprint)
queue_blueprint = construct_queue_blueprint(datastore, update_q)
ui_blueprint.register_blueprint(queue_blueprint)
# Register the browser configs ("Browsers") blueprint
browser_config_blueprint = browser_config.construct_blueprint(datastore)
ui_blueprint.register_blueprint(browser_config_blueprint)
# Import the login decorator
from changedetectionio.auth_decorator import login_optionally_required
@ui_blueprint.route("/clear_history/<uuid_str:uuid>", methods=['POST'])
@login_optionally_required
def clear_watch_history(uuid):
try:
datastore.clear_watch_history(uuid)
except KeyError:
flash(gettext('Watch not found'), 'error')
else:
flash(gettext("Cleared snapshot history for watch {}").format(uuid))
return redirect(url_for('watchlist.index'))
@ui_blueprint.route("/clear_history", methods=['GET', 'POST'])
@login_optionally_required
def clear_all_history():
if request.method == 'POST':
confirmtext = request.form.get('confirmtext', '')
if confirmtext.strip().lower() == gettext('clear').strip().lower():
# Run in background thread to avoid blocking
def clear_history_background():
# Capture UUIDs first to avoid race conditions
watch_uuids = list(datastore.data['watching'].keys())
logger.info(f"Background: Clearing history for {len(watch_uuids)} watches")
for uuid in watch_uuids:
try:
datastore.clear_watch_history(uuid)
except Exception as e:
logger.error(f"Error clearing history for watch {uuid}: {e}")
logger.info("Background: Completed clearing history")
# Start daemon thread
threading.Thread(target=clear_history_background, daemon=True).start()
flash(gettext("History clearing started in background"))
else:
flash(gettext('Incorrect confirmation text.'), 'error')
return redirect(url_for('watchlist.index'))
output = render_template("clear_all_history.html")
return output
# Clear all statuses, so we do not see the 'unviewed' class
@ui_blueprint.route("/form/mark-all-viewed", methods=['POST'])
@login_optionally_required
def mark_all_viewed():
# Save the current newest history as the most recently viewed. Operate on
# the SAME set the watch list is currently showing (tag/processor/status/
# search) — via the shared filter — so a filtered "Mark all viewed" only
# touches what's in view, not every watch.
from changedetectionio.blueprint.watchlist import filters as wl_filters
list_filters = wl_filters.list_filters_from_args(datastore, request.args)
now = int(time.time())
# Runs SYNCHRONOUSLY, and must stay that way. Re #4021: this used to hand the work to a
# background thread and redirect immediately, so the watch list re-rendered from a
# datastore that was still being marked and showed rows as unviewed until a manual
# refresh. The realtime events that would have corrected it were emitted while the
# browser was mid-navigation with no socket connected, so they went nowhere.
# It is cheap enough to do inline: the per-watch signal is suppressed below (that was
# the actual cost, not the disk write, which measures ~0.05ms per watch).
marked_count = 0
try:
for watch_uuid, watch in datastore.data['watching'].items():
if not wl_filters.watch_matches_filters(datastore, watch, list_filters):
continue
datastore.set_last_viewed(watch_uuid, now, send_signal=False)
marked_count += 1
logger.info(f"Marking complete: {marked_count} watches marked as viewed")
except Exception as e:
logger.error(f"Error marking as viewed: {e}")
# One summary event instead of one per watch, so other open tabs refresh their counters.
# This page doesn't need it - the redirect below re-renders it from the marked datastore.
if marked_count:
general_stats_update = signal('general_stats_update')
if general_stats_update:
general_stats_update.send()
return redirect(url_for('watchlist.index', **wl_filters.filter_query_args(request.args)))
@ui_blueprint.route("/delete", methods=['POST'])
@login_optionally_required
def form_delete():
uuid = request.args.get('uuid')
# More for testing, possible to return the first/only
if uuid == 'first':
uuid = list(datastore.data['watching'].keys()).pop()
if uuid != 'all' and not uuid in datastore.data['watching'].keys():
flash(gettext('The watch by UUID {} does not exist.').format(uuid), 'error')
return redirect(url_for('watchlist.index'))
datastore.delete(uuid)
flash(gettext('Deleted.'))
return redirect(url_for('watchlist.index'))
@ui_blueprint.route("/clone", methods=['POST'])
@login_optionally_required
def form_clone():
uuid = request.args.get('uuid')
if uuid == 'first':
uuid = list(datastore.data['watching'].keys()).pop()
new_uuid = datastore.clone(uuid)
if not new_uuid:
# Refused (e.g. PAGE_WATCH_LIMIT) - the reason is already flashed
return redirect(url_for('watchlist.index'))
if not datastore.data['watching'].get(uuid).get('paused'):
worker_pool.queue_item_async_safe(update_q, queuedWatchMetaData.PrioritizedItem(priority=5, item={'uuid': new_uuid}))
flash(gettext('Cloned, you are editing the new watch.'))
return redirect(url_for("ui.ui_edit.edit_page", uuid=new_uuid))
@ui_blueprint.route("/checknow", methods=['POST'])
@login_optionally_required
def form_watch_checknow():
# Forced recheck will skip the 'skip if content is the same' rule (, 'reprocess_existing_data': True})))
from changedetectionio.blueprint.watchlist import filters as wl_filters
uuid = request.args.get('uuid')
if uuid:
# Single watch - check if already queued or running
if worker_pool.is_watch_running(uuid) or uuid in update_q.get_queued_uuids():
flash(gettext("Watch is already queued or being checked."))
else:
worker_pool.queue_item_async_safe(update_q, queuedWatchMetaData.PrioritizedItem(priority=1, item={'uuid': uuid}))
flash(gettext("Queued 1 watch for rechecking."))
else:
# Multiple watches - operate on the SAME set the watch list is showing
# (tag/processor/status/search) via the shared filter, skipping paused.
list_filters = wl_filters.list_filters_from_args(datastore, request.args)
watches_to_queue = []
for k in sorted(datastore.data['watching'].items(), key=lambda item: item[1].get('last_checked', 0)):
watch_uuid = k[0]
watch = k[1]
if not watch['paused'] and watch_uuid and wl_filters.watch_matches_filters(datastore, watch, list_filters):
watches_to_queue.append(watch_uuid)
# If less than 20 watches, queue synchronously for immediate feedback
if len(watches_to_queue) < 20:
# Get already queued/running UUIDs once (efficient)
queued_uuids = set(update_q.get_queued_uuids())
running_uuids = set(worker_pool.get_running_uuids())
# Filter out watches that are already queued or running
watches_to_queue_filtered = []
for watch_uuid in watches_to_queue:
if watch_uuid not in queued_uuids and watch_uuid not in running_uuids:
watches_to_queue_filtered.append(watch_uuid)
# Queue only the filtered watches
for watch_uuid in watches_to_queue_filtered:
worker_pool.queue_item_async_safe(update_q, queuedWatchMetaData.PrioritizedItem(priority=1, item={'uuid': watch_uuid}))
# Provide feedback about skipped watches
skipped_count = len(watches_to_queue) - len(watches_to_queue_filtered)
if skipped_count > 0:
flash(gettext("Queued {count} watches for rechecking ({skipped_count} already queued or running).").format(
count=len(watches_to_queue_filtered), skipped_count=skipped_count))
else:
if len(watches_to_queue_filtered) == 1:
flash(gettext("Queued 1 watch for rechecking."))
else:
flash(gettext("Queued {} watches for rechecking.").format(len(watches_to_queue_filtered)))
else:
# 20+ watches - queue in background thread to avoid blocking HTTP response
# Capture queued/running state before background thread
queued_uuids = set(update_q.get_queued_uuids())
running_uuids = set(worker_pool.get_running_uuids())
def queue_watches_background():
"""Background thread to queue watches - discarded after completion."""
try:
queued_count = 0
skipped_count = 0
for watch_uuid in watches_to_queue:
# Check if already queued or running (state captured at start)
if watch_uuid not in queued_uuids and watch_uuid not in running_uuids:
worker_pool.queue_item_async_safe(update_q, queuedWatchMetaData.PrioritizedItem(priority=1, item={'uuid': watch_uuid}))
queued_count += 1
else:
skipped_count += 1
logger.info(f"Background queueing complete: {queued_count} watches queued, {skipped_count} skipped (already queued/running)")
except Exception as e:
logger.error(f"Error in background queueing: {e}")
# Start background thread and return immediately
thread = threading.Thread(target=queue_watches_background, daemon=True, name="QueueWatches-Background")
thread.start()
# Return immediately with approximate message
flash(gettext("Queueing watches for rechecking in background..."))
return redirect(url_for('watchlist.index', **wl_filters.filter_query_args(request.args)))
@ui_blueprint.route("/form/checkbox-operations", methods=['POST'])
@login_optionally_required
def form_watch_list_checkbox_operations():
op = request.form['op']
uuids = [u.strip() for u in request.form.getlist('uuids') if u]
extra_data = request.form.get('op_extradata', '').strip()
_handle_operations(
datastore=datastore,
extra_data=extra_data,
queuedWatchMetaData=queuedWatchMetaData,
uuids=uuids,
worker_pool=worker_pool,
update_q=update_q,
watch_check_update=watch_check_update,
op=op,
)
return redirect(url_for('watchlist.index'))
@ui_blueprint.route("/share-url/<uuid_str:uuid>", methods=['POST'])
@login_optionally_required
def form_share_put_watch(uuid):
if not datastore.data['settings']['application']['ui'].get('use_share_watch'):
abort(403, description="Access denied")
"""Given a watch UUID, upload the info and return a share-link
the share-link can be imported/added"""
import requests
import json
from copy import deepcopy
# copy it to memory as trim off what we dont need (history)
watch = deepcopy(datastore.data['watching'].get(uuid))
# For older versions that are not a @property
if (watch.get('history')):
del (watch['history'])
# for safety/privacy
for k in list(watch.keys()):
if k.startswith('notification_'):
del watch[k]
for r in['uuid', 'last_checked', 'last_changed']:
if watch.get(r):
del (watch[r])
# Add the global stuff which may have an impact
watch['ignore_text'] += datastore.data['settings']['application']['global_ignore_text']
watch['subtractive_selectors'] += datastore.data['settings']['application']['global_subtractive_selectors']
watch_json = json.dumps(watch)
try:
r = requests.request(method="POST",
data={'watch': watch_json},
url="https://changedetection.io/share/share",
headers={'App-Guid': datastore.data['app_guid']})
res = r.json()
# Add to the flask session
session['share-link'] = f"https://changedetection.io/share/{res['share_key']}"
except Exception as e:
logger.error(f"Error sharing -{str(e)}")
flash(gettext("Could not share, something went wrong while communicating with the share server - {}").format(str(e)), 'error')
return redirect(url_for('watchlist.index'))
@ui_blueprint.route("/language/auto-detect", methods=['POST'])
def delete_locale_language_session_var_if_it_exists():
"""Clear the session locale preference to auto-detect from browser Accept-Language header"""
if 'locale' in session:
session.pop('locale', None)
# Refresh Flask-Babel to clear cached locale
from flask_babel import refresh
refresh()
flash(gettext("Language set to auto-detect from browser"))
# Check if there's a redirect parameter to return to the same page
redirect_url = request.args.get('redirect')
# If redirect is provided and safe, use it
from changedetectionio.is_safe_url import is_safe_url
if redirect_url and is_safe_url(redirect_url, current_app):
return redirect(redirect_url)
# Otherwise redirect to watchlist
return redirect(url_for('watchlist.index'))
return ui_blueprint
Reference in new issue View Git Blame Copy Permalink