Be sure visual-selector data is set when filter is not found

.github/ISSUE_TEMPLATE/bug_report.md

@@ -7,20 +7,6 @@ assignees: 'dgtlmoon'
 
 ---
 
-**DO NOT USE THIS FORM TO REPORT THAT A PARTICULAR WEBSITE IS NOT SCRAPING/WATCHING AS EXPECTED**
-
-This form is only for direct bugs and feature requests todo directly with the software.
-
-Please report watched websites (full URL and _any_ settings) that do not work with changedetection.io as expected [**IN THE DISCUSSION FORUMS**](https://github.com/dgtlmoon/changedetection.io/discussions) or your report will be deleted
-
-CONSIDER TAKING OUT A SUBSCRIPTION FOR A SMALL PRICE PER MONTH, YOU GET THE BENEFIT OF USING OUR PAID PROXIES AND FURTHERING THE DEVELOPMENT OF CHANGEDETECTION.IO
-
-THANK YOU
-
-
-
-
-
 **Describe the bug**
 A clear and concise description of what the bug is.
 

README-pip.md

@@ -1,45 +1,42 @@
-## Web Site Change Detection, Monitoring and Notification.
+#  changedetection.io
+![changedetection.io](https://github.com/dgtlmoon/changedetection.io/actions/workflows/test-only.yml/badge.svg?branch=master)
+<a href="https://hub.docker.com/r/dgtlmoon/changedetection.io" target="_blank" title="Change detection docker hub">
+  <img src="https://img.shields.io/docker/pulls/dgtlmoon/changedetection.io" alt="Docker Pulls"/>
+</a>
+<a href="https://hub.docker.com/r/dgtlmoon/changedetection.io" target="_blank" title="Change detection docker hub">
+  <img src="https://img.shields.io/github/v/release/dgtlmoon/changedetection.io" alt="Change detection latest tag version"/> 
+</a>
 
-Live your data-life pro-actively, track website content changes and receive notifications via Discord, Email, Slack, Telegram and 70+ more
+## Self-hosted open source change monitoring of web pages.
 
-[<img src="https://raw.githubusercontent.com/dgtlmoon/changedetection.io/master/docs/screenshot.png" style="max-width:100%;" alt="Self-hosted web page change monitoring"  title="Self-hosted web page change monitoring"  />](https://lemonade.changedetection.io/start?src=pip)
+_Know when web pages change! Stay ontop of new information!_ 
+
+Live your data-life *pro-actively* instead of *re-actively*, do not rely on manipulative social media for consuming important information.
 
 
-[**Don't have time? Let us host it for you! try our extremely affordable subscription use our proxies and support!**](https://lemonade.changedetection.io/start) 
+<img src="https://raw.githubusercontent.com/dgtlmoon/changedetection.io/master/screenshot.png" style="max-width:100%;" alt="Self-hosted web page change monitoring"  title="Self-hosted web page change monitoring"  />
+
+
+**Get your own private instance now! Let us host it for you!**
+
+[**Try our $6.99/month subscription - unlimited checks, watches and notifications!**](https://lemonade.changedetection.io/start), choose from different geographical locations, let us handle everything for you. 
+
 
 
 #### Example use cases
 
-- Products and services have a change in pricing
+Know when ...
-- _Out of stock notification_ and _Back In stock notification_
+
-- Governmental department updates (changes are often only on their websites)
+- Government department updates (changes are often only on their websites)
+- Local government news (changes are often only on their websites)
 - New software releases, security advisories when you're not on their mailing list.
 - Festivals with changes
 - Realestate listing changes
-- Know when your favourite whiskey is on sale, or other special deals are announced before anyone else
 - COVID related news from government websites
-- University/organisation news from their website
 - Detect and monitor changes in JSON API responses 
-- JSON API monitoring and alerting
+- API monitoring and alerting
-- Changes in legal and other documents
-- Trigger API calls via notifications when text appears on a website
-- Glue together APIs using the JSON filter and JSON notifications
-- Create RSS feeds based on changes in web content
-- Monitor HTML source code for unexpected changes, strengthen your PCI compliance
-- You have a very sensitive list of URLs to watch and you do _not_ want to use the paid alternatives. (Remember, _you_ are the product)
-
-_Need an actual Chrome runner with Javascript support? We support fetching via WebDriver and Playwright!</a>_
-
-#### Key Features
-
-- Lots of trigger filters, such as "Trigger on text", "Remove text by selector", "Ignore text", "Extract text", also using regular-expressions!
-- Target elements with xPath and CSS Selectors, Easily monitor complex JSON with JsonPath rules
-- Switch between fast non-JS and Chrome JS based "fetchers"
-- Easily specify how often a site should be checked
-- Execute JS before extracting text (Good for logging in, see examples in the UI!)
-- Override Request Headers, Specify `POST` or `GET` and other methods
-- Use the "Visual Selector" to help target specific elements
 
+**Get monitoring now!**
 
 ```bash
 $ pip3 install changedetection.io   
@@ -54,5 +51,17 @@ $ changedetection.io -d /path/to/empty/data/dir -p 5000
 
 Then visit http://127.0.0.1:5000 , You should now be able to access the UI.
 
+### Features
+- Website monitoring
+- Change detection of content and analyses
+- Filters on change (Select by CSS or JSON)
+- Triggers (Wait for text, wait for regex)
+- Notification support
+- JSON API Monitoring
+- Parse JSON embedded in HTML
+- (Reverse) Proxy support
+- Javascript support via WebDriver
+- RaspberriPi (arm v6/v7/64 support)
+
 See https://github.com/dgtlmoon/changedetection.io for more information.
 

README.md

@@ -1,16 +1,23 @@
-## Web Site Change Detection, Monitoring and Notification.
+#  changedetection.io
-
-Live your data-life pro-actively, track website content changes and receive notifications via Discord, Email, Slack, Telegram and 70+ more
-
-[<img src="https://raw.githubusercontent.com/dgtlmoon/changedetection.io/master/docs/screenshot.png" style="max-width:100%;" alt="Self-hosted web page change monitoring"  title="Self-hosted web page change monitoring"  />](https://lemonade.changedetection.io/start?src=github)
-
 [![Release Version][release-shield]][release-link] [![Docker Pulls][docker-pulls]][docker-link] [![License][license-shield]](LICENSE.md)
 
 ![changedetection.io](https://github.com/dgtlmoon/changedetection.io/actions/workflows/test-only.yml/badge.svg?branch=master)
 
-Know when important content changes, we support notifications via Discord, Telegram, Home-Assistant, Slack, Email and 70+ more
+## Web Site Change Detection, Monitoring and Notification - Self-Hosted or SaaS.
 
-[**Don't have time? Let us host it for you! try our $6.99/month subscription - use our proxies and support!**](https://lemonade.changedetection.io/start) , _half the price of other website change monitoring services and comes with unlimited watches & checks!_
+_Know when web pages change! Stay ontop of new information! get notifications when important website content changes_ 
+
+Live your data-life *pro-actively* instead of *re-actively*.
+
+Free, Open-source web page monitoring, notification and change detection. Don't have time? [**Try our $6.99/month subscription - unlimited checks and watches!**](https://lemonade.changedetection.io/start)
+
+
+[<img src="https://raw.githubusercontent.com/dgtlmoon/changedetection.io/master/docs/screenshot.png" style="max-width:100%;" alt="Self-hosted web page change monitoring"  title="Self-hosted web page change monitoring"  />](https://lemonade.changedetection.io/start)
+
+
+**Get your own private instance now! Let us host it for you!**
+
+[**Try our $6.99/month subscription - unlimited checks and watches!**](https://lemonade.changedetection.io/start) , _half the price of other website change monitoring services and comes with unlimited watches & checks!_
 
 
 
@@ -39,18 +46,7 @@ Know when important content changes, we support notifications via Discord, Teleg
 - Monitor HTML source code for unexpected changes, strengthen your PCI compliance
 - You have a very sensitive list of URLs to watch and you do _not_ want to use the paid alternatives. (Remember, _you_ are the product)
 
-_Need an actual Chrome runner with Javascript support? We support fetching via WebDriver and Playwright!</a>_
+_Need an actual Chrome runner with Javascript support? We support fetching via WebDriver!</a>_
-
-#### Key Features
-
-- Lots of trigger filters, such as "Trigger on text", "Remove text by selector", "Ignore text", "Extract text", also using regular-expressions!
-- Target elements with xPath and CSS Selectors, Easily monitor complex JSON with JsonPath rules
-- Switch between fast non-JS and Chrome JS based "fetchers"
-- Easily specify how often a site should be checked
-- Execute JS before extracting text (Good for logging in, see examples in the UI!)
-- Override Request Headers, Specify `POST` or `GET` and other methods
-- Use the "Visual Selector" to help target specific elements
-
 
 ## Screenshots
 

changedetectionio/__init__.py

@@ -1,5 +1,16 @@
 #!/usr/bin/python3
 
+
+# @todo logging
+# @todo extra options for url like , verify=False etc.
+# @todo enable https://urllib3.readthedocs.io/en/latest/user-guide.html#ssl as option?
+# @todo option for interval day/6 hour/etc
+# @todo on change detected, config for calling some API
+# @todo fetch title into json
+# https://distill.io/features
+# proxy per check
+#  - flask_cors, itsdangerous,MarkupSafe
+
 import datetime
 import os
 import queue
@@ -33,7 +44,7 @@ from flask_wtf import CSRFProtect
 from changedetectionio import html_tools
 from changedetectionio.api import api_v1
 
-__version__ = '0.39.19.1'
+__version__ = '0.39.17.2'
 
 datastore = None
 
@@ -492,7 +503,7 @@ def changedetection_app(config=None, datastore_o=None):
         from changedetectionio import fetch_site_status
 
         # Get the most recent one
-        newest_history_key = datastore.data['watching'][uuid].get('newest_history_key')
+        newest_history_key = datastore.get_val(uuid, 'newest_history_key')
 
         # 0 means that theres only one, so that there should be no 'unviewed' history available
         if newest_history_key == 0:
@@ -541,6 +552,10 @@ def changedetection_app(config=None, datastore_o=None):
         # be sure we update with a copy instead of accidently editing the live object by reference
         default = deepcopy(datastore.data['watching'][uuid])
 
+        # Show system wide default if nothing configured
+        if datastore.data['watching'][uuid]['fetch_backend'] is None:
+            default['fetch_backend'] = datastore.data['settings']['application']['fetch_backend']
+
         # Show system wide default if nothing configured
         if all(value == 0 or value == None for value in datastore.data['watching'][uuid]['time_between_check'].values()):
             default['time_between_check'] = deepcopy(datastore.data['settings']['requests']['time_between_check'])
@@ -583,6 +598,8 @@ def changedetection_app(config=None, datastore_o=None):
             if form.fetch_backend.data == datastore.data['settings']['application']['fetch_backend']:
                 extra_update_obj['fetch_backend'] = None
 
+            # Notification URLs
+            datastore.data['watching'][uuid]['notification_urls'] = form.notification_urls.data
 
             # Ignore text
             form_ignore_text = form.ignore_text.data
@@ -638,11 +655,9 @@ def changedetection_app(config=None, datastore_o=None):
                                      watch=datastore.data['watching'][uuid],
                                      form=form,
                                      has_empty_checktime=using_default_check_time,
-                                     has_default_notification_urls=True if len(datastore.data['settings']['application']['notification_urls']) else False,
                                      using_global_webdriver_wait=default['webdriver_delay'] is None,
                                      current_base_url=datastore.data['settings']['application']['base_url'],
                                      emailprefix=os.getenv('NOTIFICATION_MAIL_BUTTON_PREFIX', False),
-                                     settings_application=datastore.data['settings']['application'],
                                      visualselector_data_is_ready=visualselector_data_is_ready,
                                      visualselector_enabled=visualselector_enabled,
                                      playwright_enabled=os.getenv('PLAYWRIGHT_DRIVER_URL', False)
@@ -672,10 +687,6 @@ def changedetection_app(config=None, datastore_o=None):
         form = forms.globalSettingsForm(formdata=request.form if request.method == 'POST' else None,
                                         data=default
                                         )
-
-        # Remove the last option 'System default'
-        form.application.form.notification_format.choices.pop()
-
         if datastore.proxy_list is None:
             # @todo - Couldn't get setattr() etc dynamic addition working, so remove it instead
             del form.requests.form.proxy
@@ -721,8 +732,7 @@ def changedetection_app(config=None, datastore_o=None):
                                  current_base_url = datastore.data['settings']['application']['base_url'],
                                  hide_remove_pass=os.getenv("SALTED_PASS", False),
                                  api_key=datastore.data['settings']['application'].get('api_access_token'),
-                                 emailprefix=os.getenv('NOTIFICATION_MAIL_BUTTON_PREFIX', False),
+                                 emailprefix=os.getenv('NOTIFICATION_MAIL_BUTTON_PREFIX', False))
-                                 settings_application=datastore.data['settings']['application'])
 
         return output
 
@@ -1176,63 +1186,6 @@ def changedetection_app(config=None, datastore_o=None):
         flash("{} watches are queued for rechecking.".format(i))
         return redirect(url_for('index', tag=tag))
 
-    @app.route("/form/checkbox-operations", methods=['POST'])
-    @login_required
-    def form_watch_list_checkbox_operations():
-        op = request.form['op']
-        uuids = request.form.getlist('uuids')
-
-        if (op == 'delete'):
-            for uuid in uuids:
-                uuid = uuid.strip()
-                if datastore.data['watching'].get(uuid):
-                    datastore.delete(uuid.strip())
-            flash("{} watches deleted".format(len(uuids)))
-
-        elif (op == 'pause'):
-            for uuid in uuids:
-                uuid = uuid.strip()
-                if datastore.data['watching'].get(uuid):
-                    datastore.data['watching'][uuid.strip()]['paused'] = True
-
-            flash("{} watches paused".format(len(uuids)))
-
-        elif (op == 'unpause'):
-            for uuid in uuids:
-                uuid = uuid.strip()
-                if datastore.data['watching'].get(uuid):
-                    datastore.data['watching'][uuid.strip()]['paused'] = False
-            flash("{} watches unpaused".format(len(uuids)))
-
-        elif (op == 'mute'):
-            for uuid in uuids:
-                uuid = uuid.strip()
-                if datastore.data['watching'].get(uuid):
-                    datastore.data['watching'][uuid.strip()]['notification_muted'] = True
-            flash("{} watches muted".format(len(uuids)))
-
-        elif (op == 'unmute'):
-            for uuid in uuids:
-                uuid = uuid.strip()
-                if datastore.data['watching'].get(uuid):
-                    datastore.data['watching'][uuid.strip()]['notification_muted'] = False
-            flash("{} watches un-muted".format(len(uuids)))
-
-        elif (op == 'notification-default'):
-            from changedetectionio.notification import (
-                default_notification_format_for_watch
-            )
-            for uuid in uuids:
-                uuid = uuid.strip()
-                if datastore.data['watching'].get(uuid):
-                    datastore.data['watching'][uuid.strip()]['notification_title'] = None
-                    datastore.data['watching'][uuid.strip()]['notification_body'] = None
-                    datastore.data['watching'][uuid.strip()]['notification_urls'] = []
-                    datastore.data['watching'][uuid.strip()]['notification_format'] = default_notification_format_for_watch
-            flash("{} watches set to use default notification settings".format(len(uuids)))
-
-        return redirect(url_for('index'))
-
     @app.route("/api/share-url", methods=['GET'])
     @login_required
     def form_share_put_watch():
@@ -1432,18 +1385,13 @@ def ticker_thread_check_time_launch_checks():
             seconds_since_last_recheck = now - watch['last_checked']
             if seconds_since_last_recheck >= (threshold + watch.jitter_seconds) and seconds_since_last_recheck >= recheck_time_minimum_seconds:
                 if not uuid in running_uuids and uuid not in [q_uuid for p,q_uuid in update_q.queue]:
-                    # Use Epoch time as priority, so we get a "sorted" PriorityQueue, but we can still push a priority 1 into it.
+                    print("> Queued watch UUID {} last checked at {} queued at {:0.2f} jitter {:0.2f}s, {:0.2f}s since last checked".format(uuid,
-                    priority = int(time.time())
-                    print(
-                        "> Queued watch UUID {} last checked at {} queued at {:0.2f} priority {} jitter {:0.2f}s, {:0.2f}s since last checked".format(
-                            uuid,
                                                                                                          watch['last_checked'],
                                                                                                          now,
-                            priority,
                                                                                                          watch.jitter_seconds,
                                                                                                          now - watch['last_checked']))
                     # Into the queue with you
-                    update_q.put((priority, uuid))
+                    update_q.put((5, uuid))
 
                     # Reset for next time
                     watch.jitter_seconds = 0

changedetectionio/content_fetcher.py

@@ -31,12 +31,11 @@ class JSActionExceptions(Exception):
         return
 
 class PageUnloadable(Exception):
-    def __init__(self, status_code, url, screenshot=False, message=False):
+    def __init__(self, status_code, url, screenshot=False):
         # Set this so we can use it in other parts of the app
         self.status_code = status_code
         self.url = url
         self.screenshot = screenshot
-        self.message = message
         return
 
 class EmptyReply(Exception):
@@ -293,15 +292,7 @@ class base_html_playwright(Fetcher):
 
         # allow per-watch proxy selection override
         if proxy_override:
-            # https://playwright.dev/docs/network#http-proxy
+            self.proxy = {'server': proxy_override}
-            from urllib.parse import urlparse
-            parsed = urlparse(proxy_override)
-            proxy_url = "{}://{}:{}".format(parsed.scheme, parsed.hostname, parsed.port)
-            self.proxy = {'server': proxy_url}
-            if parsed.username:
-                self.proxy['username'] = parsed.username
-            if parsed.password:
-                self.proxy['password'] = parsed.password
 
     def run(self,
             url,
@@ -365,7 +356,7 @@ class base_html_playwright(Fetcher):
                 print(str(e))
                 context.close()
                 browser.close()
-                raise PageUnloadable(url=url, status_code=None, message=e.message)
+                raise PageUnloadable(url=url, status_code=None)
 
             if response is None:
                 context.close()

changedetectionio/fetch_site_status.py

@@ -63,11 +63,13 @@ class perform_site_check():
 
 
     def run(self, uuid):
+        timestamp = int(time.time())  # used for storage etc too
+
         changed_detected = False
         screenshot = False  # as bytes
         stripped_text_from_html = ""
 
-        watch = self.datastore.data['watching'].get(uuid)
+        watch = self.datastore.data['watching'][uuid]
 
         # Protect against file:// access
         if re.search(r'^file', watch['url'], re.IGNORECASE) and not os.getenv('ALLOW_FILE_URI', False):
@@ -78,7 +80,7 @@ class perform_site_check():
         # Unset any existing notification error
         update_obj = {'last_notification_error': False, 'last_error': False}
 
-        extra_headers =self.datastore.data['watching'][uuid].get('headers')
+        extra_headers = self.datastore.get_val(uuid, 'headers')
 
         # Tweak the base config with the per-watch ones
         request_headers = self.datastore.data['settings']['headers'].copy()
@@ -91,9 +93,9 @@ class perform_site_check():
             request_headers['Accept-Encoding'] = request_headers['Accept-Encoding'].replace(', br', '')
 
         timeout = self.datastore.data['settings']['requests']['timeout']
-        url = watch.get('url')
+        url = self.datastore.get_val(uuid, 'url')
-        request_body = self.datastore.data['watching'][uuid].get('body')
+        request_body = self.datastore.get_val(uuid, 'body')
-        request_method = self.datastore.data['watching'][uuid].get('method')
+        request_method = self.datastore.get_val(uuid, 'method')
         ignore_status_codes = self.datastore.data['watching'][uuid].get('ignore_status_codes', False)
 
         # source: support

changedetectionio/forms.py

@@ -314,14 +314,14 @@ class quickWatchForm(Form):
 
 # Common to a single watch and the global settings
 class commonSettingsForm(Form):
-    notification_urls = StringListField('Notification URL list', validators=[validators.Optional(), ValidateAppRiseServers()])
+
-    notification_title = StringField('Notification title', validators=[validators.Optional(), ValidateTokensList()])
+    notification_urls = StringListField('Notification URL list', validators=[validators.Optional(), ValidateNotificationBodyAndTitleWhenURLisSet(), ValidateAppRiseServers()])
-    notification_body = TextAreaField('Notification body', validators=[validators.Optional(), ValidateTokensList()])
+    notification_title = StringField('Notification title', default=default_notification_title, validators=[validators.Optional(), ValidateTokensList()])
-    notification_format = SelectField('Notification format', choices=valid_notification_formats.keys())
+    notification_body = TextAreaField('Notification body', default=default_notification_body, validators=[validators.Optional(), ValidateTokensList()])
+    notification_format = SelectField('Notification format', choices=valid_notification_formats.keys(), default=default_notification_format)
     fetch_backend = RadioField(u'Fetch method', choices=content_fetcher.available_fetchers(), validators=[ValidateContentFetcherIsReady()])
     extract_title_as_title = BooleanField('Extract <title> from document and use as watch title', default=False)
-    webdriver_delay = IntegerField('Wait seconds before extracting text', validators=[validators.Optional(), validators.NumberRange(min=1,
+    webdriver_delay = IntegerField('Wait seconds before extracting text', validators=[validators.Optional(), validators.NumberRange(min=1, message="Should contain one or more seconds")] )
-                                                                                                                                    message="Should contain one or more seconds")])
 
 class watchForm(commonSettingsForm):
 
@@ -355,8 +355,6 @@ class watchForm(commonSettingsForm):
     filter_failure_notification_send = BooleanField(
         'Send a notification when the filter can no longer be found on the page', default=False)
 
-    notification_muted = BooleanField('Notifications Muted / Off', default=False)
-
     def validate(self, **kwargs):
         if not super().validate():
             return False

changedetectionio/model/Watch.py

@@ -6,7 +6,9 @@ minimum_seconds_recheck_time = int(os.getenv('MINIMUM_SECONDS_RECHECK_TIME', 60)
 mtable = {'seconds': 1, 'minutes': 60, 'hours': 3600, 'days': 86400, 'weeks': 86400 * 7}
 
 from changedetectionio.notification import (
-    default_notification_format_for_watch
+    default_notification_body,
+    default_notification_format,
+    default_notification_title,
 )
 
 
@@ -30,9 +32,9 @@ class model(dict):
             'ignore_text': [],  # List of text to ignore when calculating the comparison checksum
             # Custom notification content
             'notification_urls': [],  # List of URLs to add to the notification Queue (Usually AppRise)
-            'notification_title': None,
+            'notification_title': default_notification_title,
-            'notification_body': None,
+            'notification_body': default_notification_body,
-            'notification_format': default_notification_format_for_watch,
+            'notification_format': default_notification_format,
             'notification_muted': False,
             'css_filter': '',
             'last_error': False,

changedetectionio/notification.py

@@ -14,19 +14,16 @@ valid_tokens = {
     'current_snapshot': ''
 }
 
-default_notification_format_for_watch = 'System default'
-default_notification_format = 'Text'
-default_notification_body = '{watch_url} had a change.\n---\n{diff}\n---\n'
-default_notification_title = 'ChangeDetection.io Notification - {watch_url}'
-
 valid_notification_formats = {
     'Text': NotifyFormat.TEXT,
     'Markdown': NotifyFormat.MARKDOWN,
     'HTML': NotifyFormat.HTML,
-    # Used only for editing a watch (not for global)
-    default_notification_format_for_watch: default_notification_format_for_watch
 }
 
+default_notification_format = 'Text'
+default_notification_body = '{watch_url} had a change.\n---\n{diff}\n---\n'
+default_notification_title = 'ChangeDetection.io Notification - {watch_url}'
+
 def process_notification(n_object, datastore):
 
     # Get the notification body from datastore

changedetectionio/static/images/notice.svg

@@ -1,51 +0,0 @@
-<?xml version="1.0" encoding="UTF-8" standalone="no"?>
-<!-- Created with Inkscape (http://www.inkscape.org/) -->
-
-<svg
-   width="20.108334mm"
-   height="21.43125mm"
-   viewBox="0 0 20.108334 21.43125"
-   version="1.1"
-   id="svg5"
-   xmlns:xlink="http://www.w3.org/1999/xlink"
-   xmlns="http://www.w3.org/2000/svg"
-   xmlns:svg="http://www.w3.org/2000/svg">
-  <defs
-     id="defs2" />
-  <g
-     id="layer1"
-     transform="translate(-141.05873,-76.816635)">
-    <image
-       width="20.108334"
-       height="21.43125"
-       preserveAspectRatio="none"
-       style="image-rendering:optimizeQuality"
-       xlink:href="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAEwAAABRCAYAAAB430BuAAAABHNCSVQICAgIfAhkiAAABLxJREFU
-eJztnN2Z2jgUhl8Z7petIGwF0WMXsFBBoIKwFWS2gmQryKSCJRXsTAUDBTDRVBCmgkAB9tkLexh+
-bIONLGwP7xU2RjafpaOjoyNBCxHNQAJEfG5sl+3ZLrAWeAyST5/sF91mFH3bRbZbsAq4ClaQq2B7
-iKYnmg9Z318F20ICRnj8pMOd6E3HscNVsATxmQD/oeghPCnDLO26q2AkYin+TQ7XREyyrn3zgu2J
-BSEjZTBZ179pwQ7EEv7KaoovvFnBUsV6ZHrsd+0WTHhKPV1SLGivYEsA1KEtEs2grFitRjQ65VxP
-fH5JgEjAKsvXupKwFfYxaYJeSeHcWqVSCuwD7/HQQD8lRHLWDStBWG3slbAElkTc5/lTZdkIJhpN
-h6/UUZDyzAgZK8PKVoEKErE8HlD0bBVcI2ZqwdBWYbFgAT+g1UZwrBbcvRyIpofHJ1Sh1rQCZt1k
-lN5msQAm8CoYoFF8KVHOsFtQ5aayExBUhpnopJl6J/3/FREGWCrxmaH40/4z1oyQ320Yf5dDozXC
-P4QMCRkCY4S5w/tbMTtd4L2Ngo6wJmSQ4hfdScAU+OjgGazgOXEl8oJyof3Z6Spx0iTzgnLKsMoK
-w9SRuoR3rHniVVMXwRpDXQR7d+kHOJV6CFZB0khVOBGsTcE6VzWsNVGQizfJptU+N4LlD3AbVfsu
-XsOahhvB8nrB08IrtcGNYNIct+EYl2+S6mr0D8kLUMrV6BfFRTzOGs4Ey8p1aNrUnssaliaMO/vV
-sfNi3AmW5j54DgUTO/dyJ1hab9iwHhLcNskP23ZMND0kewFBXek6vZvHg/hMiUPSN00z+OBasFig
-y8wSRfnZ0adSBz+sUVwFK4jbJhnPP06To1ETczpcCnavHhltHd82LU0AXDbJMGXBU8PSBAA8Jxk0
-wnNaqlGSJuAyg+dsXIV38iZqXU3iWsmodhetSNlDQgJGriZxbWVSe1hS/gQ+S/C6j4QEfES21vxU
-icXsoC4vC5mqJvbybyXgduucG/YWaYmmj+IdHvpoxFdt8ltRP5h3iZjRqfBh60C4t1rNY7rxAU95
-aYnhEp+/u8pgxGfeRCfyJIR5SkLfFOHYXMMzu63PEDF9WQnSo8MUmhduyUWYEzGyvnRmU3683ugG
-GAG/2bqJU4RnFDNCpsfWb5chswUnwb5Xg+hxiyo9w7MGJoSVpmYulam+A8scS+5nPYtf+s9mpZw7
-J1nayDnCVuu4Ck+E6DqIBYDHHR1+is/n8kVUhfBExMBFMzm4taafkXcWL9BSfBG/nNN8sutYcE3S
-d7XI3o6lSpIe/xcAIX/svzDxMVu22BAyLNKL2q9hwrdLiZWwXbP6B99GDLaGSpoOD6JPn4yxK1i8
-B0StY1zKsCJiQNxzQ0HRbAm2BsZN2TBDGVaE5USzIVjsNix2VrzWHmUwB6J5fD32uyKCzQ7OxG5D
-vzZuQ0E2osXjRlBMjvWe5WtYPE4b2BynXQJlMEToTUegmEiwM1mzQ1nBvqvH5ov1wlZHcA+AZHdc
-xQW7vNuQS9kBtzKs1IIRMM7b0q/YvGTzto4qbFutdV5FnLtLk2x3JVWUfXKTbIu9Opc2J6Osj19S
-HLfJKO64r6rg/wFBX3+2ZapW8wAAAABJRU5ErkJggg==
-"
-       id="image832"
-       x="141.05873"
-       y="76.816635" />
-  </g>
-</svg>

changedetectionio/static/images/play.svg

@@ -1,122 +0,0 @@
-<?xml version="1.0" encoding="UTF-8" standalone="no"?>
-<svg
-   version="1.1"
-   id="Capa_1"
-   x="0px"
-   y="0px"
-   viewBox="0 0 15 14.998326"
-   xml:space="preserve"
-   width="15"
-   height="14.998326"
-   sodipodi:docname="play.svg"
-   inkscape:version="1.1.1 (1:1.1+202109281949+c3084ef5ed)"
-   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
-   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
-   xmlns="http://www.w3.org/2000/svg"
-   xmlns:svg="http://www.w3.org/2000/svg"
-   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
-   xmlns:cc="http://creativecommons.org/ns#"
-   xmlns:dc="http://purl.org/dc/elements/1.1/"><sodipodi:namedview
-   id="namedview21"
-   pagecolor="#ffffff"
-   bordercolor="#666666"
-   borderopacity="1.0"
-   inkscape:pageshadow="2"
-   inkscape:pageopacity="0.0"
-   inkscape:pagecheckerboard="0"
-   showgrid="false"
-   inkscape:zoom="45.47174"
-   inkscape:cx="7.4991632"
-   inkscape:cy="7.4991632"
-   inkscape:window-width="1554"
-   inkscape:window-height="896"
-   inkscape:window-x="3048"
-   inkscape:window-y="227"
-   inkscape:window-maximized="0"
-   inkscape:current-layer="Capa_1" /><metadata
-   id="metadata39"><rdf:RDF><cc:Work
-       rdf:about=""><dc:format>image/svg+xml</dc:format><dc:type
-         rdf:resource="http://purl.org/dc/dcmitype/StillImage" /></cc:Work></rdf:RDF></metadata><defs
-   id="defs37" />
-<path
-   id="path2"
-   style="fill:#1b98f8;fill-opacity:1;stroke-width:0.0292893"
-   d="M 7.4980469,0 C 4.5496028,-0.04093755 1.7047721,1.8547661 0.58789062,4.5800781 -0.57819305,7.2574082 0.02636631,10.583252 2.0703125,12.671875 4.0368718,14.788335 7.2754393,15.560096 9.9882812,14.572266 12.800219,13.617028 14.874915,10.855516 14.986328,7.8847656 15.172991,4.9968456 13.497714,2.109448 10.910156,0.8203125 9.858961,0.28011352 8.6796569,-0.00179908 7.4980469,0 Z"
-   sodipodi:nodetypes="ccccccc" />
-<g
-   id="g4"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g6"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g8"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g10"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g12"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g14"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g16"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g18"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g20"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g22"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g24"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g26"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g28"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g30"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g32"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<path
-   sodipodi:type="star"
-   style="fill:#ffffff;fill-opacity:1;stroke-width:37.7953;paint-order:stroke fill markers"
-   id="path1203"
-   inkscape:flatsided="false"
-   sodipodi:sides="3"
-   sodipodi:cx="7.2964563"
-   sodipodi:cy="7.3240671"
-   sodipodi:r1="3.805218"
-   sodipodi:r2="1.9026089"
-   sodipodi:arg1="-0.0017436774"
-   sodipodi:arg2="1.0454539"
-   inkscape:rounded="0"
-   inkscape:randomized="0"
-   d="M 11.101669,7.317432 8.2506324,8.9701135 5.3995964,10.622795 5.3938504,7.3273846 5.3881041,4.0319742 8.2448863,5.6747033 Z"
-   inkscape:transform-center-x="-0.94843001"
-   inkscape:transform-center-y="0.0033175346" /></svg>

changedetectionio/static/js/watch-overview.js

@@ -22,18 +22,5 @@ $(function () {
       });
   });
 
-    // checkboxes - check all
-    $("#check-all").click(function (e) {
-        $('input[type=checkbox]').not(this).prop('checked', this.checked);
-    });
-    // checkboxes - show/hide buttons
-    $("input[type=checkbox]").click(function (e) {
-        if ($('input[type=checkbox]:checked').length) {
-            $('#checkbox-operations').slideDown();
-        } else {
-            $('#checkbox-operations').slideUp();
-        }
-    });
-
 });
 

changedetectionio/static/js/watch-settings.js

@@ -30,11 +30,4 @@ $(document).ready(function() {
     });
     toggle();
 
-    $('#notification-setting-reset-to-default').click(function (e) {
-        $('#notification_title').val('');
-        $('#notification_body').val('');
-        $('#notification_format').val('System default');
-        $('#notification_urls').val('');
-        e.preventDefault();
-    });
 });

changedetectionio/static/styles/styles.css

@@ -555,26 +555,3 @@ ul {
   .snapshot-age.error {
     background-color: #ff0000;
     color: #fff; }
-
-#checkbox-operations {
-  background: rgba(0, 0, 0, 0.05);
-  padding: 1em;
-  border-radius: 10px;
-  margin-bottom: 1em;
-  display: none; }
-
-.checkbox-uuid > * {
-  vertical-align: middle; }
-
-.inline-warning {
-  border: 1px solid #ff3300;
-  padding: 0.5rem;
-  border-radius: 5px;
-  color: #ff3300; }
-  .inline-warning > span {
-    display: inline-block;
-    vertical-align: middle; }
-  .inline-warning img.inline-warning-icon {
-    display: inline;
-    height: 26px;
-    vertical-align: middle; }

changedetectionio/static/styles/styles.scss

@@ -774,33 +774,3 @@ ul {
   }
 }
 
-#checkbox-operations {
-  background: rgba(0, 0, 0, 0.05);
-  padding: 1em;
-  border-radius: 10px;
-  margin-bottom: 1em;
-  display: none;
-}
-.checkbox-uuid {
-  > * {
-    vertical-align: middle;
-  }
-}
-
-.inline-warning {
-  > span {
-    display: inline-block;
-    vertical-align: middle;
-  }
-
-  img.inline-warning-icon {
-    display: inline;
-    height: 26px;
-    vertical-align: middle;
-  }
-
-  border: 1px solid #ff3300;
-  padding: 0.5rem;
-  border-radius: 5px;
-  color: #ff3300;
-}

changedetectionio/store.py

@@ -244,6 +244,10 @@ class ChangeDetectionStore:
 
         return False
 
+    def get_val(self, uuid, val):
+        # Probably their should be dict...
+        return self.data['watching'][uuid].get(val)
+
     # Remove a watchs data but keep the entry (URL etc)
     def clear_watch_history(self, uuid):
         import pathlib
@@ -337,8 +341,6 @@ class ChangeDetectionStore:
 
     # Save as PNG, PNG is larger but better for doing visual diff in the future
     def save_screenshot(self, watch_uuid, screenshot: bytes, as_error=False):
-        if not self.data['watching'].get(watch_uuid):
-            return
 
         if as_error:
             target_path = os.path.join(self.datastore_path, watch_uuid, "last-error-screenshot.png")
@@ -352,16 +354,14 @@ class ChangeDetectionStore:
             f.close()
 
     def save_error_text(self, watch_uuid, contents):
-        if not self.data['watching'].get(watch_uuid):
+
-            return
         target_path = os.path.join(self.datastore_path, watch_uuid, "last-error.txt")
 
         with open(target_path, 'w') as f:
             f.write(contents)
 
     def save_xpath_data(self, watch_uuid, data, as_error=False):
-        if not self.data['watching'].get(watch_uuid):
+
-            return
         if as_error:
             target_path = os.path.join(self.datastore_path, watch_uuid, "elements-error.json")
         else:
@@ -536,24 +536,3 @@ class ChangeDetectionStore:
             except:
                 continue
         return
-
-    def update_5(self):
-        # If the watch notification body, title look the same as the global one, unset it, so the watch defaults back to using the main settings
-        # In other words - the watch notification_title and notification_body are not needed if they are the same as the default one
-        current_system_body = self.data['settings']['application']['notification_body'].translate(str.maketrans('', '', "\r\n "))
-        current_system_title = self.data['settings']['application']['notification_body'].translate(str.maketrans('', '', "\r\n "))
-        for uuid, watch in self.data['watching'].items():
-            try:
-                watch_body = watch.get('notification_body', '')
-                if watch_body and watch_body.translate(str.maketrans('', '', "\r\n ")) == current_system_body:
-                    # Looks the same as the default one, so unset it
-                    watch['notification_body'] = None
-
-                watch_title = watch.get('notification_title', '')
-                if watch_title and watch_title.translate(str.maketrans('', '', "\r\n ")) == current_system_title:
-                    # Looks the same as the default one, so unset it
-                    watch['notification_title'] = None
-            except Exception as e:
-                continue
-        return
-

changedetectionio/templates/_common_fields.jinja

@@ -1,14 +1,13 @@
 
 {% from '_helpers.jinja' import render_field %}
 
-{% macro render_common_settings_form(form, emailprefix, settings_application) %}
+{% macro render_common_settings_form(form, current_base_url, emailprefix) %}
                         <div class="pure-control-group">
                             {{ render_field(form.notification_urls, rows=5, placeholder="Examples:
     Gitter - gitter://token/room
     Office365 - o365://TenantID:AccountEmail/ClientID/ClientSecret/TargetEmail
     AWS SNS - sns://AccessKeyID/AccessSecretKey/RegionName/+PhoneNo
-    SMTPS - mailtos://user:pass@mail.domain.com?to=receivingAddress@example.com",
+    SMTPS - mailtos://user:pass@mail.domain.com?to=receivingAddress@example.com", class="notification-urls")
-    class="notification-urls" )
                             }}
                             <div class="pure-form-message-inline">
                               <ul>
@@ -27,16 +26,15 @@
                         </div>
                         <div id="notification-customisation" class="pure-control-group">
                             <div class="pure-control-group">
-                                {{ render_field(form.notification_title, class="m-d notification-title", placeholder=settings_application['notification_title']) }}
+                                {{ render_field(form.notification_title, class="m-d notification-title") }}
                                 <span class="pure-form-message-inline">Title for all notifications</span>
                             </div>
                             <div class="pure-control-group">
-                                {{ render_field(form.notification_body , rows=5, class="notification-body", placeholder=settings_application['notification_body']) }}
+                                {{ render_field(form.notification_body , rows=5, class="notification-body") }}
                                 <span class="pure-form-message-inline">Body for all notifications</span>
                             </div>
                             <div class="pure-control-group">
-                            <!-- unsure -->
+                                {{ render_field(form.notification_format , rows=5, class="notification-format") }}
-                                {{ render_field(form.notification_format , class="notification-format") }}
                                 <span class="pure-form-message-inline">Format for all notifications</span>
                             </div>
                             <div class="pure-controls">
@@ -96,7 +94,7 @@
                                 </table>
                                 <br/>
                                 URLs generated by changedetection.io (such as <code>{diff_url}</code>) require the <code>BASE_URL</code> environment variable set.<br/>
-                                Your <code>BASE_URL</code> var is currently "{{settings_application['current_base_url']}}"
+                                Your <code>BASE_URL</code> var is currently "{{current_base_url}}"
                             </span>
                             </div>
                         </div>

changedetectionio/templates/edit.html

@@ -135,20 +135,10 @@ User-Agent: wonderbra 1.0") }}
             </div>
 
             <div class="tab-pane-inner" id="notifications">
+                <strong>Note: <i>These settings override the global settings for this watch.</i></strong>
                 <fieldset>
-                    <div  class="pure-control-group inline-radio">
+                    <div class="field-group">
-                      {{ render_checkbox_field(form.notification_muted) }}
+                        {{ render_common_settings_form(form, current_base_url, emailprefix) }}
-                    </div>
-                    <div class="field-group" id="notification-field-group">
-                        {% if has_default_notification_urls %}
-                        <div class="inline-warning">
-                            <img class="inline-warning-icon" src="{{url_for('static_content', group='images', filename='notice.svg')}}" alt="Look out!" title="Lookout!"/>
-                            There are <a href="{{ url_for('settings_page')}}#notifications">system-wide notification URLs enabled</a>, this form will override notification settings for this watch only &dash; an empty Notification URL list here will still send notifications.
-                        </div>
-                        {% endif %}
-                        <a href="#notifications" id="notification-setting-reset-to-default" class="pure-button button-xsmall" style="right: 20px; top: 20px; position: absolute; background-color: #5f42dd; border-radius: 4px; font-size: 70%; color: #fff">Use system defaults</a>
-
-                        {{ render_common_settings_form(form, emailprefix, settings_application) }}
                     </div>
                 </fieldset>
             </div>

changedetectionio/templates/preview.html

@@ -57,7 +57,6 @@
          </br>
          {% if is_html_webdriver %}
            {% if screenshot %}
-             <div class="snapshot-age">{{watch.snapshot_screenshot_ctime|format_timestamp_timeago}}</div>
              <img style="max-width: 80%" id="screenshot-img" alt="Current screenshot from most recent request"/>
            {% else %}
               No screenshot available just yet! Try rechecking the page.

changedetectionio/templates/settings.html

@@ -60,7 +60,7 @@
                         {{ render_field(form.application.form.base_url, placeholder="http://yoursite.com:5000/",
                         class="m-d") }}
                         <span class="pure-form-message-inline">
-                            Base URL used for the <code>{base_url}</code> token in notifications and RSS links.<br/>Default value is the ENV var 'BASE_URL' (Currently "{{settings_application['current_base_url']}}"),
+                            Base URL used for the {base_url} token in notifications and RSS links.<br/>Default value is the ENV var 'BASE_URL' (Currently "{{current_base_url}}"),
                             <a href="https://github.com/dgtlmoon/changedetection.io/wiki/Configurable-BASE_URL-setting">read more here</a>.
                         </span>
                     </div>
@@ -87,7 +87,7 @@
             <div class="tab-pane-inner" id="notifications">
                 <fieldset>
                     <div class="field-group">
-                        {{ render_common_settings_form(form.application.form, emailprefix, settings_application) }}
+                        {{ render_common_settings_form(form.application.form, current_base_url, emailprefix) }}
                     </div>
                 </fieldset>
             </div>

changedetectionio/templates/watch-overview.html

@@ -24,17 +24,6 @@
         </fieldset>
         <span style="color:#eee; font-size: 80%;"><img style="height: 1em;display:inline-block;" src="{{url_for('static_content', group='images', filename='spread-white.svg')}}" /> Tip: You can also add 'shared' watches. <a href="https://github.com/dgtlmoon/changedetection.io/wiki/Sharing-a-Watch">More info</a></a></span>
     </form>
-
-    <form class="pure-form" action="{{ url_for('form_watch_list_checkbox_operations') }}" method="POST" id="watch-list-form">
-    <input type="hidden" name="csrf_token" value="{{ csrf_token() }}"/>
-    <div id="checkbox-operations">
-        <button class="pure-button button-secondary button-xsmall" style="font-size: 70%"  name="op" value="pause">Pause</button>
-        <button class="pure-button button-secondary button-xsmall" style="font-size: 70%"  name="op" value="unpause">UnPause</button>
-        <button class="pure-button button-secondary button-xsmall" style="font-size: 70%"  name="op" value="mute">Mute</button>
-        <button class="pure-button button-secondary button-xsmall" style="font-size: 70%"  name="op" value="unmute">UnMute</button>
-        <button class="pure-button button-secondary button-xsmall" style="font-size: 70%" name="op" value="notification-default">Use default notification</button>
-        <button class="pure-button button-secondary button-xsmall" style="background: #dd4242; font-size: 70%" name="op" value="delete">Delete</button>
-    </div>
     <div>
         <a href="{{url_for('index')}}" class="pure-button button-tag {{'active' if not active_tag }}">All</a>
         {% for tag in tags %}
@@ -52,7 +41,7 @@
         <table class="pure-table pure-table-striped watch-table">
             <thead>
             <tr>
-                <th><input style="vertical-align: middle" type="checkbox" id="check-all"/> #</th>
+                <th>#</th>
                 <th></th>
                 {% set link_order = "desc" if sort_order else "asc" %}
                 {% set arrow_span = "" %}
@@ -77,13 +66,9 @@
                 {% if watch.paused is defined and watch.paused != False %}paused{% endif %}
                 {% if watch.newest_history_key| int > watch.last_viewed and watch.history_n>=2 %}unviewed{% endif %}
                 {% if watch.uuid in queued_uuids %}queued{% endif %}">
-                <td class="inline checkbox-uuid" ><input name="uuids"  type="checkbox" value="{{ watch.uuid}} "/> <span>{{ loop.index }}</span></td>
+                <td class="inline">{{ loop.index }}</td>
                 <td class="inline watch-controls">
-                    {% if not watch.paused %}
+                    <a class="state-{{'on' if watch.paused }}" href="{{url_for('index', op='pause', uuid=watch.uuid, tag=active_tag)}}"><img src="{{url_for('static_content', group='images', filename='pause.svg')}}" alt="Pause checks" title="Pause checks"/></a>
-                    <a class="state-off" href="{{url_for('index', op='pause', uuid=watch.uuid, tag=active_tag)}}"><img src="{{url_for('static_content', group='images', filename='pause.svg')}}" alt="Pause checks" title="Pause checks"/></a>
-                    {% else %}
-                    <a class="state-on" href="{{url_for('index', op='pause', uuid=watch.uuid, tag=active_tag)}}"><img src="{{url_for('static_content', group='images', filename='play.svg')}}" alt="UnPause checks" title="UnPause checks"/></a>
-                    {% endif %}
                     <a class="state-{{'on' if watch.notification_muted}}" href="{{url_for('index', op='mute', uuid=watch.uuid, tag=active_tag)}}"><img src="{{url_for('static_content', group='images', filename='bell-off.svg')}}" alt="Mute notifications" title="Mute notifications"/></a>
                 </td>
                 <td class="title-col inline">{{watch.title if watch.title is not none and watch.title|length > 0 else watch.url}}
@@ -144,6 +129,5 @@
          #}
 
     </div>
-    </form>
 </div>
 {% endblock %}

changedetectionio/tests/test_notification.py

@@ -4,13 +4,7 @@ import re
 from flask import url_for
 from . util import set_original_response, set_modified_response, set_more_modified_response, live_server_setup
 import logging
-
+from changedetectionio.notification import default_notification_body, default_notification_title
-from changedetectionio.notification import (
-    default_notification_body,
-    default_notification_format,
-    default_notification_title,
-    valid_notification_formats,
-)
 
 def test_setup(live_server):
     live_server_setup(live_server)
@@ -26,26 +20,9 @@ def test_check_notification(client, live_server):
 
     # Re 360 - new install should have defaults set
     res = client.get(url_for("settings_page"))
-    notification_url = url_for('test_notification_endpoint', _external=True).replace('http', 'json')
-
     assert default_notification_body.encode() in res.data
     assert default_notification_title.encode() in res.data
 
-    #####################
-    # Set this up for when we remove the notification from the watch, it should fallback with these details
-    res = client.post(
-        url_for("settings_page"),
-        data={"application-notification_urls": notification_url,
-              "application-notification_title": "fallback-title "+default_notification_title,
-              "application-notification_body": "fallback-body "+default_notification_body,
-              "application-notification_format": default_notification_format,
-              "requests-time_between_check-minutes": 180,
-              'application-fetch_backend': "html_requests"},
-        follow_redirects=True
-    )
-
-    assert b"Settings updated." in res.data
-
     # When test mode is in BASE_URL env mode, we should see this already configured
     env_base_url = os.getenv('BASE_URL', '').strip()
     if len(env_base_url):
@@ -70,6 +47,8 @@ def test_check_notification(client, live_server):
 
     # Goto the edit page, add our ignore text
     # Add our URL to the import page
+    url = url_for('test_notification_endpoint', _external=True)
+    notification_url = url.replace('http', 'json')
 
     print (">>>> Notification URL: "+notification_url)
 
@@ -179,30 +158,6 @@ def test_check_notification(client, live_server):
     # be sure we see it in the output log
     assert b'New ChangeDetection.io Notification - ' + test_url.encode('utf-8') in res.data
 
-    set_original_response()
-    res = client.post(
-        url_for("edit_page", uuid="first"),
-        data={
-        "url": test_url,
-        "tag": "my tag",
-        "title": "my title",
-        "notification_urls": '',
-        "notification_title": '',
-        "notification_body": '',
-        "notification_format": default_notification_format,
-        "fetch_backend": "html_requests"},
-        follow_redirects=True
-    )
-    assert b"Updated watch." in res.data
-
-    time.sleep(2)
-
-    # Verify what was sent as a notification, this file should exist
-    with open("test-datastore/notification.txt", "r") as f:
-        notification_submission = f.read()
-    assert "fallback-title" in notification_submission
-    assert "fallback-body" in notification_submission
-
     # cleanup for the next
     client.get(
         url_for("form_delete", uuid="all"),
@@ -225,20 +180,20 @@ def test_notification_validation(client, live_server):
     assert b"Watch added" in res.data
 
     # Re #360 some validation
-#    res = client.post(
+    res = client.post(
-#        url_for("edit_page", uuid="first"),
+        url_for("edit_page", uuid="first"),
-#        data={"notification_urls": 'json://localhost/foobar',
+        data={"notification_urls": 'json://localhost/foobar',
-#              "notification_title": "",
+              "notification_title": "",
-#              "notification_body": "",
+              "notification_body": "",
-#              "notification_format": "Text",
+              "notification_format": "Text",
-#              "url": test_url,
+              "url": test_url,
-#              "tag": "my tag",
+              "tag": "my tag",
-#              "title": "my title",
+              "title": "my title",
-#              "headers": "",
+              "headers": "",
-#              "fetch_backend": "html_requests"},
+              "fetch_backend": "html_requests"},
-#        follow_redirects=True
+        follow_redirects=True
-#    )
+    )
-#    assert b"Notification Body and Title is required when a Notification URL is used" in res.data
+    assert b"Notification Body and Title is required when a Notification URL is used" in res.data
 
     # Now adding a wrong token should give us an error
     res = client.post(
@@ -260,5 +215,3 @@ def test_notification_validation(client, live_server):
         url_for("form_delete", uuid="all"),
         follow_redirects=True
     )
-
-

changedetectionio/tests/visualselector/test_fetch_data.py

@@ -7,7 +7,6 @@ from ..util import live_server_setup, wait_for_all_checks, extract_UUID_from_cli
 # Add a site in paused mode, add an invalid filter, we should still have visual selector data ready
 def test_visual_selector_content_ready(client, live_server):
     import os
-    import json
 
     assert os.getenv('PLAYWRIGHT_DRIVER_URL'), "Needs PLAYWRIGHT_DRIVER_URL set for this test"
     live_server_setup(live_server)
@@ -34,7 +33,3 @@ def test_visual_selector_content_ready(client, live_server):
     uuid = extract_UUID_from_client(client)
     assert os.path.isfile(os.path.join('test-datastore', uuid, 'last-screenshot.png')), "last-screenshot.png should exist"
     assert os.path.isfile(os.path.join('test-datastore', uuid, 'elements.json')), "xpath elements.json data should exist"
-
-    # Open it and see if it roughly looks correct
-    with open(os.path.join('test-datastore', uuid, 'elements.json'), 'r') as f:
-        json.load(f)

changedetectionio/update_worker.py

@@ -11,14 +11,11 @@ from changedetectionio.html_tools import FilterNotFoundInResponse
 # Requests for checking on a single site(watch) from a queue of watches
 # (another process inserts watches into the queue that are time-ready for checking)
 
-import logging
-import sys
 
 class update_worker(threading.Thread):
     current_uuid = None
 
     def __init__(self, q, notification_q, app, datastore, *args, **kwargs):
-        logging.basicConfig(stream=sys.stderr, level=logging.DEBUG)
         self.q = q
         self.app = app
         self.notification_q = notification_q
@@ -29,10 +26,6 @@ class update_worker(threading.Thread):
 
         from changedetectionio import diff
 
-        from changedetectionio.notification import (
-            default_notification_format_for_watch
-        )
-
         n_object = {}
         watch = self.datastore.data['watching'].get(watch_uuid, False)
         if not watch:
@@ -47,27 +40,33 @@ class update_worker(threading.Thread):
                 "History index had 2 or more, but only 1 date loaded, timestamps were not unique? maybe two of the same timestamps got written, needs more delay?"
             )
 
-        n_object['notification_urls'] = watch['notification_urls'] if len(watch['notification_urls']) else \
+        # Did it have any notification alerts to hit?
-            self.datastore.data['settings']['application']['notification_urls']
+        if len(watch['notification_urls']):
-
+            print(">>> Notifications queued for UUID from watch {}".format(watch_uuid))
-        n_object['notification_title'] = watch['notification_title'] if watch['notification_title'] else \
+            n_object['notification_urls'] = watch['notification_urls']
-            self.datastore.data['settings']['application']['notification_title']
+            n_object['notification_title'] = watch['notification_title']
-
+            n_object['notification_body'] = watch['notification_body']
-        n_object['notification_body'] = watch['notification_body'] if watch['notification_body'] else \
+            n_object['notification_format'] = watch['notification_format']
-            self.datastore.data['settings']['application']['notification_body']
-
-        n_object['notification_format'] = watch['notification_format'] if watch['notification_format'] != default_notification_format_for_watch else \
-            self.datastore.data['settings']['application']['notification_format']
 
+        # No? maybe theres a global setting, queue them all
+        elif len(self.datastore.data['settings']['application']['notification_urls']):
+            print(">>> Watch notification URLs were empty, using GLOBAL notifications for UUID: {}".format(watch_uuid))
+            n_object['notification_urls'] = self.datastore.data['settings']['application']['notification_urls']
+            n_object['notification_title'] = self.datastore.data['settings']['application']['notification_title']
+            n_object['notification_body'] = self.datastore.data['settings']['application']['notification_body']
+            n_object['notification_format'] = self.datastore.data['settings']['application']['notification_format']
+        else:
+            print(">>> NO notifications queued, watch and global notification URLs were empty.")
 
         # Only prepare to notify if the rules above matched
-        if 'notification_urls' in n_object and n_object['notification_urls']:
+        if 'notification_urls' in n_object:
             # HTML needs linebreak, but MarkDown and Text can use a linefeed
             if n_object['notification_format'] == 'HTML':
                 line_feed_sep = "</br>"
             else:
                 line_feed_sep = "\n"
 
+            snapshot_contents = ''
             with open(watch_history[dates[-1]], 'rb') as f:
                 snapshot_contents = f.read()
 
@@ -78,10 +77,8 @@ class update_worker(threading.Thread):
                 'diff': diff.render_diff(watch_history[dates[-2]], watch_history[dates[-1]], line_feed_sep=line_feed_sep),
                 'diff_full': diff.render_diff(watch_history[dates[-2]], watch_history[dates[-1]], True, line_feed_sep=line_feed_sep)
             })
-            logging.info (">> SENDING NOTIFICATION")
+
             self.notification_q.put(n_object)
-        else:
-            logging.info (">> NO Notification sent, notification_url was empty in both watch and system")
 
     def send_filter_failure_notification(self, watch_uuid):
 
@@ -186,9 +183,6 @@ class update_worker(threading.Thread):
                         process_changedetection_results = False
 
                     except FilterNotFoundInResponse as e:
-                        if not self.datastore.data['watching'].get(uuid):
-                            continue
-
                         err_text = "Warning, filter '{}' not found".format(str(e))
                         self.datastore.update_watch(uuid=uuid, update_obj={'last_error': err_text,
                                                                            # So that we get a trigger when the content is added again
@@ -228,10 +222,8 @@ class update_worker(threading.Thread):
                         self.datastore.update_watch(uuid=uuid, update_obj={'last_error': err_text,
                                                                            'last_check_status': e.status_code})
                     except content_fetcher.PageUnloadable as e:
+                        # @todo connection-refused ?
                         err_text = "Page request from server didnt respond correctly"
-                        if e.message:
-                            err_text = "{} - {}".format(err_text, e.message)
-
                         if e.screenshot:
                             self.datastore.save_screenshot(watch_uuid=uuid, screenshot=e.screenshot, as_error=True)
 
@@ -243,9 +235,6 @@ class update_worker(threading.Thread):
                         # Other serious error
                         process_changedetection_results = False
                     else:
-                        # Crash protection, the watch entry could have been removed by this point (during a slow chrome fetch etc)
-                        if not self.datastore.data['watching'].get(uuid):
-                            continue
 
                         # Mark that we never had any failures
                         if not self.datastore.data['watching'][uuid].get('ignore_status_codes'):
@@ -253,6 +242,10 @@ class update_worker(threading.Thread):
 
                         self.cleanup_error_artifacts(uuid)
 
+                    # Crash protection, the watch entry could have been removed by this point (during a slow chrome fetch etc)
+                    if not self.datastore.data['watching'].get(uuid):
+                        continue
+
                     # Different exceptions mean that we may or may not want to bump the snapshot, trigger notifications etc
                     if process_changedetection_results:
                         try:

docker-compose.yml

@@ -30,7 +30,7 @@ services:
   #
   #             https://playwright.dev/python/docs/api/class-browsertype#browser-type-launch-option-proxy
   #
-  #        Plain requests - proxy support example.
+  #        Plain requsts - proxy support example.
   #      - HTTP_PROXY=socks5h://10.10.1.10:1080
   #      - HTTPS_PROXY=socks5h://10.10.1.10:1080
   #