WIP - remove column, use a smart 'data-int' to sort on instead

* Update CONTRIBUTING.md

* Add option for tags on import (#377)

* Add option for tags on import and backup

* .add_watch() can accept empty tag
Use https://changedetection.io/CHANGELOG.txt as a nice default page to watch

* plaintext mime type fix - Don't attempt to extract HTML content from plaintext, this will remove lines and break changedetection (#391)

* #323 Adding note about discord:// 2000 char limit (#392)

* Adding note about discord:// 2000 char limit

* Ability to use a generated salted password in deployments as env var SALTED_PASS (#397)

* Ability to use a generated salted password in deployments as env var SALTED_PASS

* Offer instance on Lemonade
Tidy README

* Update README - Tidy up sections

* Update README - Fix docker section

* Update README.md

* /preview format doesnt need <pre> - fixing too many returnlines in content on diff/preview page

* fixed the reference to wiki for rpi section (#402)

* Add notification note - tgram:// bots cant send messages to other bots, so you should specify chat ID of non-bot user.

* Notification error log handler (#403)

* Add a notifications debug/error log interface (Link available under the notification URLs list)

* Refactor tests for notification error log handler (#404)

* Introduce -h option to allow listening not on 0.0.0.0. (#406)

* Fix typo in the startup create-directory command suggestion (#405)

* Use flask url_for() for webdriver chrome icon instead of relative path

* merging latest upstream changes

Co-authored-by: dgtlmoon <dgtlmoon@gmail.com>
Co-authored-by: Tim Loderhose <timlod@users.noreply.github.com>
Co-authored-by: Radu Ursache <3800336+rursache@users.noreply.github.com>
Co-authored-by: Alexander Aleksandrovič Klimov <al2klimov@gmail.com>

.github/workflows/containers.yml

@@ -2,20 +2,16 @@ name: Build and push containers
 
 on:
   # Automatically triggered by a testing workflow passing, but this is only checked when it lands in the `master`/default branch
-#  workflow_run:
-#    workflows: ["ChangeDetection.io Test"]
-#    branches: [master]
-#    tags: ['0.*']
-#    types: [completed]
+  workflow_run:
+    workflows: ["ChangeDetection.io Test"]
+    branches: [master]
+    tags: ['0.*']
+    types: [completed]
 
   # Or a new tagged release
   release:
     types: [published, edited]
 
-  push:
-    branches:
-      - master
-
 jobs:
   metadata:
     runs-on: ubuntu-latest
@@ -95,7 +91,8 @@ jobs:
           file: ./Dockerfile
           push: true
           tags: |
-            ${{ secrets.DOCKER_HUB_USERNAME }}/changedetection.io:latest,ghcr.io/${{ github.repository }}:latest
+            ${{ secrets.DOCKER_HUB_USERNAME }}/changedetection.io:latest
+            ghcr.io/${{ github.repository }}:latest
           platforms: linux/amd64,linux/arm64,linux/arm/v6,linux/arm/v7
           cache-from: type=local,src=/tmp/.buildx-cache
           cache-to: type=local,dest=/tmp/.buildx-cache
@@ -110,7 +107,8 @@ jobs:
           file: ./Dockerfile
           push: true
           tags: |
-            ${{ secrets.DOCKER_HUB_USERNAME }}/changedetection.io:${{ github.event.release.tag_name }},ghcr.io/dgtlmoon/changedetection.io:${{ github.event.release.tag_name }}
+            ${{ secrets.DOCKER_HUB_USERNAME }}/changedetection.io:${{ github.event.release.tag_name }}
+            ghcr.io/dgtlmoon/changedetection.io:${{ github.event.release.tag_name }}
           platforms: linux/amd64,linux/arm64,linux/arm/v6,linux/arm/v7
           cache-from: type=local,src=/tmp/.buildx-cache
           cache-to: type=local,dest=/tmp/.buildx-cache

.gitignore

@@ -7,6 +7,4 @@ __pycache__
 .pytest_cache
 build
 dist
-venv
-*.egg-info*
 .vscode/settings.json

Dockerfile

@@ -20,11 +20,6 @@ COPY requirements.txt /requirements.txt
 
 RUN pip install --target=/dependencies -r /requirements.txt
 
-# Playwright is an alternative to Selenium
-# Excluded this package from requirements.txt to prevent arm/v6 and arm/v7 builds from failing
-RUN pip install --target=/dependencies playwright~=1.20 \
-    || echo "WARN: Failed to install Playwright. The application can still run, but the Playwright option will be disabled."
-
 # Final image stage
 FROM python:3.8-slim
 

MANIFEST.in

@@ -1,7 +1,6 @@
 recursive-include changedetectionio/templates *
 recursive-include changedetectionio/static *
-recursive-include changedetectionio/model *
 include changedetection.py
 global-exclude *.pyc
-global-exclude node_modules
+global-exclude *node_modules*
 global-exclude venv

README-pip.md

@@ -16,13 +16,6 @@ Live your data-life *pro-actively* instead of *re-actively*, do not rely on mani
 
 <img src="https://raw.githubusercontent.com/dgtlmoon/changedetection.io/master/screenshot.png" style="max-width:100%;" alt="Self-hosted web page change monitoring"  title="Self-hosted web page change monitoring"  />
 
-
-**Get your own private instance now! Let us host it for you!**
-
-[**Try our $6.99/month subscription - unlimited checks, watches and notifications!**](https://lemonade.changedetection.io/start), choose from different geographical locations, let us handle everything for you. 
-
-
-
 #### Example use cases
 
 Know when ...
@@ -65,3 +58,14 @@ Then visit http://127.0.0.1:5000 , You should now be able to access the UI.
 
 See https://github.com/dgtlmoon/changedetection.io for more information.
 
+
+
+### Support us
+
+Do you use changedetection.io to make money? does it save you time or money? Does it make your life easier? less stressful? Remember, we write this software when we should be doing actual paid work, we have to buy food and pay rent just like you.
+
+Please support us, even small amounts help a LOT.
+
+BTC `1PLFN327GyUarpJd7nVe7Reqg9qHx5frNn`
+
+<img src="https://raw.githubusercontent.com/dgtlmoon/changedetection.io/master/btc-support.png" style="max-width:50%;" alt="Support us!"  />

README.md

@@ -9,15 +9,18 @@ _Know when web pages change! Stay ontop of new information!_
 
 Live your data-life *pro-actively* instead of *re-actively*.
 
-Free, Open-source web page monitoring, notification and change detection. Don't have time? [**Try our $6.99/month subscription - unlimited checks and watches!**](https://lemonade.changedetection.io/start)
+Open source web page monitoring, notification and change detection.
 
 
-[<img src="https://raw.githubusercontent.com/dgtlmoon/changedetection.io/master/screenshot.png" style="max-width:100%;" alt="Self-hosted web page change monitoring"  title="Self-hosted web page change monitoring"  />](https://lemonade.changedetection.io/start)
+<img src="https://raw.githubusercontent.com/dgtlmoon/changedetection.io/master/screenshot.png" style="max-width:100%;" alt="Self-hosted web page change monitoring"  title="Self-hosted web page change monitoring"  />
 
 
 **Get your own private instance now! Let us host it for you!**
 
-[**Try our $6.99/month subscription - unlimited checks and watches!**](https://lemonade.changedetection.io/start) , _half the price of other website change monitoring services and comes with unlimited watches & checks!_
+[![Deploy to Lemonade](https://lemonade.changedetection.io/static/images/lemonade.svg)](https://lemonade.changedetection.io/start)
+
+
+[_Let us host your own private instance - We accept PayPal and Bitcoin, Support the further development of changedetection.io!_](https://lemonade.changedetection.io/start)
 
 
 
@@ -36,14 +39,13 @@ Free, Open-source web page monitoring, notification and change detection. Don't
 - COVID related news from government websites
 - University/organisation news from their website
 - Detect and monitor changes in JSON API responses 
-- JSON API monitoring and alerting
+- API monitoring and alerting
 - Changes in legal and other documents
 - Trigger API calls via notifications when text appears on a website
 - Glue together APIs using the JSON filter and JSON notifications
 - Create RSS feeds based on changes in web content
-- Monitor HTML source code for unexpected changes, strengthen your PCI compliance
 - You have a very sensitive list of URLs to watch and you do _not_ want to use the paid alternatives. (Remember, _you_ are the product)
-
+ 
 _Need an actual Chrome runner with Javascript support? We support fetching via WebDriver!</a>_
 
 ## Screenshots
@@ -68,10 +70,6 @@ Docker standalone
 $ docker run -d --restart always -p "127.0.0.1:5000:5000" -v datastore-volume:/datastore --name changedetection.io dgtlmoon/changedetection.io
 ```
 
-### Windows
-
-See the install instructions at the wiki https://github.com/dgtlmoon/changedetection.io/wiki/Microsoft-Windows
-
 ### Python Pip
 
 Check out our pypi page https://pypi.org/project/changedetection.io/
@@ -165,17 +163,17 @@ See the wiki https://github.com/dgtlmoon/changedetection.io/wiki/Proxy-configura
 
 Raspberry Pi and linux/arm/v6 linux/arm/v7 arm64 devices are supported! See the wiki for [details](https://github.com/dgtlmoon/changedetection.io/wiki/Fetching-pages-with-WebDriver)
 
+## Windows native support?
+
+Sorry not yet :( https://github.com/dgtlmoon/changedetection.io/labels/windows
 
 ## Support us
 
 Do you use changedetection.io to make money? does it save you time or money? Does it make your life easier? less stressful? Remember, we write this software when we should be doing actual paid work, we have to buy food and pay rent just like you.
 
+Please support us, even small amounts help a LOT.
 
-Firstly, consider taking out a [change detection monthly subscription - unlimited checks and watches](https://lemonade.changedetection.io/start) , even if you don't use it, you still get the warm fuzzy feeling of helping out the project. (And who knows, you might just use it!)
-
-Or directly donate an amount PayPal [![Donate](https://img.shields.io/badge/Donate-PayPal-green.svg)](https://www.paypal.com/donate/?hosted_button_id=7CP6HR9ZCNDYJ)
-
-Or BTC `1PLFN327GyUarpJd7nVe7Reqg9qHx5frNn`
+BTC `1PLFN327GyUarpJd7nVe7Reqg9qHx5frNn`
 
 <img src="https://raw.githubusercontent.com/dgtlmoon/changedetection.io/master/btc-support.png" style="max-width:50%;" alt="Support us!"  />
 

changedetection.py

@@ -1,11 +1,110 @@
 #!/usr/bin/python3
 
-# Entry-point for running from the CLI when not installed via Pip, Pip will handle the console_scripts entry_points's from setup.py
-# It's recommended to use `pip3 install changedetection.io` and start with `changedetection.py` instead, it will be linkd to your global path.
-# or Docker.
-# Read more https://github.com/dgtlmoon/changedetection.io/wiki
+# Launch as a eventlet.wsgi server instance.
+
+import getopt
+import os
+import sys
+
+import eventlet
+import eventlet.wsgi
+import changedetectionio
+
+from changedetectionio import store
+
+def main():
+    ssl_mode = False
+    host = ''
+    port = os.environ.get('PORT') or 5000
+    do_cleanup = False
+
+    # Must be absolute so that send_from_directory doesnt try to make it relative to backend/
+    datastore_path = os.path.join(os.getcwd(), "datastore")
+
+    try:
+        opts, args = getopt.getopt(sys.argv[1:], "Ccsd:h:p:", "port")
+    except getopt.GetoptError:
+        print('backend.py -s SSL enable -h [host] -p [port] -d [datastore path]')
+        sys.exit(2)
+
+    create_datastore_dir = False
+
+    for opt, arg in opts:
+        #        if opt == '--purge':
+        # Remove history, the actual files you need to delete manually.
+        #            for uuid, watch in datastore.data['watching'].items():
+        #                watch.update({'history': {}, 'last_checked': 0, 'last_changed': 0, 'previous_md5': None})
+
+        if opt == '-s':
+            ssl_mode = True
+
+        if opt == '-h':
+            host = arg
+
+        if opt == '-p':
+            port = int(arg)
+
+        if opt == '-d':
+            datastore_path = arg
+
+        # Cleanup (remove text files that arent in the index)
+        if opt == '-c':
+            do_cleanup = True
+
+        # Create the datadir if it doesnt exist
+        if opt == '-C':
+            create_datastore_dir = True
+
+    # isnt there some @thingy to attach to each route to tell it, that this route needs a datastore
+    app_config = {'datastore_path': datastore_path}
+
+    if not os.path.isdir(app_config['datastore_path']):
+        if create_datastore_dir:
+            os.mkdir(app_config['datastore_path'])
+        else:
+            print ("ERROR: Directory path for the datastore '{}' does not exist, cannot start, please make sure the directory exists.\n"
+                   "Alternatively, use the -C parameter.".format(app_config['datastore_path']),file=sys.stderr)
+            sys.exit(2)
+
+    datastore = store.ChangeDetectionStore(datastore_path=app_config['datastore_path'], version_tag=changedetectionio.__version__)
+    app = changedetectionio.changedetection_app(app_config, datastore)
+
+    # Go into cleanup mode
+    if do_cleanup:
+        datastore.remove_unused_snapshots()
+
+    app.config['datastore_path'] = datastore_path
+
+
+    @app.context_processor
+    def inject_version():
+        return dict(right_sticky="v{}".format(datastore.data['version_tag']),
+                    new_version_available=app.config['NEW_VERSION_AVAILABLE'],
+                    has_password=datastore.data['settings']['application']['password'] != False
+                    )
+
+    # Proxy sub-directory support
+    # Set environment var USE_X_SETTINGS=1 on this script
+    # And then in your proxy_pass settings
+    #
+    #         proxy_set_header Host "localhost";
+    #         proxy_set_header X-Forwarded-Prefix /app;
+
+    if os.getenv('USE_X_SETTINGS'):
+        print ("USE_X_SETTINGS is ENABLED\n")
+        from werkzeug.middleware.proxy_fix import ProxyFix
+        app.wsgi_app = ProxyFix(app.wsgi_app, x_prefix=1, x_host=1)
+
+    if ssl_mode:
+        # @todo finalise SSL config, but this should get you in the right direction if you need it.
+        eventlet.wsgi.server(eventlet.wrap_ssl(eventlet.listen((host, port)),
+                                               certfile='cert.pem',
+                                               keyfile='privkey.pem',
+                                               server_side=True), app)
+
+    else:
+        eventlet.wsgi.server(eventlet.listen((host, int(port))), app)
 
-from changedetectionio import changedetection
 
 if __name__ == '__main__':
-    changedetection.main()
+    main()

changedetectionio/.gitignore

@@ -1 +0,0 @@
-test-datastore

changedetectionio/changedetection.py

@@ -1,114 +0,0 @@
-#!/usr/bin/python3
-
-# Launch as a eventlet.wsgi server instance.
-
-import getopt
-import os
-import sys
-
-import eventlet
-import eventlet.wsgi
-from . import store, changedetection_app, content_fetcher
-from . import __version__
-
-def main():
-    ssl_mode = False
-    host = ''
-    port = os.environ.get('PORT') or 5000
-    do_cleanup = False
-    datastore_path = None
-
-    # On Windows, create and use a default path.
-    if os.name == 'nt':
-        datastore_path = os.path.expandvars(r'%APPDATA%\changedetection.io')
-        os.makedirs(datastore_path, exist_ok=True)
-    else:
-        # Must be absolute so that send_from_directory doesnt try to make it relative to backend/
-        datastore_path = os.path.join(os.getcwd(), "../datastore")
-
-    try:
-        opts, args = getopt.getopt(sys.argv[1:], "Ccsd:h:p:", "port")
-    except getopt.GetoptError:
-        print('backend.py -s SSL enable -h [host] -p [port] -d [datastore path]')
-        sys.exit(2)
-
-    create_datastore_dir = False
-
-    for opt, arg in opts:
-        #        if opt == '--purge':
-        # Remove history, the actual files you need to delete manually.
-        #            for uuid, watch in datastore.data['watching'].items():
-        #                watch.update({'history': {}, 'last_checked': 0, 'last_changed': 0, 'previous_md5': None})
-
-        if opt == '-s':
-            ssl_mode = True
-
-        if opt == '-h':
-            host = arg
-
-        if opt == '-p':
-            port = int(arg)
-
-        if opt == '-d':
-            datastore_path = arg
-
-        # Cleanup (remove text files that arent in the index)
-        if opt == '-c':
-            do_cleanup = True
-
-        # Create the datadir if it doesnt exist
-        if opt == '-C':
-            create_datastore_dir = True
-
-    # isnt there some @thingy to attach to each route to tell it, that this route needs a datastore
-    app_config = {'datastore_path': datastore_path}
-
-    if not os.path.isdir(app_config['datastore_path']):
-        if create_datastore_dir:
-            os.mkdir(app_config['datastore_path'])
-        else:
-            print(
-                "ERROR: Directory path for the datastore '{}' does not exist, cannot start, please make sure the directory exists or specify a directory with the -d option.\n"
-                "Or use the -C parameter to create the directory.".format(app_config['datastore_path']), file=sys.stderr)
-            sys.exit(2)
-
-    datastore = store.ChangeDetectionStore(datastore_path=app_config['datastore_path'], version_tag=__version__)
-    app = changedetection_app(app_config, datastore)
-
-    # Go into cleanup mode
-    if do_cleanup:
-        datastore.remove_unused_snapshots()
-
-    app.config['datastore_path'] = datastore_path
-
-
-    @app.context_processor
-    def inject_version():
-        return dict(right_sticky="v{}".format(datastore.data['version_tag']),
-                    new_version_available=app.config['NEW_VERSION_AVAILABLE'],
-                    has_password=datastore.data['settings']['application']['password'] != False
-                    )
-
-    # Proxy sub-directory support
-    # Set environment var USE_X_SETTINGS=1 on this script
-    # And then in your proxy_pass settings
-    #
-    #         proxy_set_header Host "localhost";
-    #         proxy_set_header X-Forwarded-Prefix /app;
-
-    if os.getenv('USE_X_SETTINGS'):
-        print ("USE_X_SETTINGS is ENABLED\n")
-        from werkzeug.middleware.proxy_fix import ProxyFix
-        app.wsgi_app = ProxyFix(app.wsgi_app, x_prefix=1, x_host=1)
-
-    if ssl_mode:
-        # @todo finalise SSL config, but this should get you in the right direction if you need it.
-        eventlet.wsgi.server(eventlet.wrap_ssl(eventlet.listen((host, port)),
-                                               certfile='cert.pem',
-                                               keyfile='privkey.pem',
-                                               server_side=True), app)
-
-    else:
-        eventlet.wsgi.server(eventlet.listen((host, int(port))), app)
-
-

changedetectionio/content_fetcher.py

@@ -1,9 +1,12 @@
-from abc import ABC, abstractmethod
-import chardet
 import os
-import requests
 import time
-import sys
+from abc import ABC, abstractmethod
+from selenium import webdriver
+from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
+from selenium.webdriver.common.proxy import Proxy as SeleniumProxy
+from selenium.common.exceptions import WebDriverException
+import urllib3.exceptions
+
 
 class EmptyReply(Exception):
     def __init__(self, status_code, url):
@@ -11,50 +14,26 @@ class EmptyReply(Exception):
         self.status_code = status_code
         self.url = url
         return
-    pass
 
-class ReplyWithContentButNoText(Exception):
-    def __init__(self, status_code, url):
-        # Set this so we can use it in other parts of the app
-        self.status_code = status_code
-        self.url = url
-        return
     pass
 
-
 class Fetcher():
     error = None
     status_code = None
-    content = None
+    content = None # Should always be bytes.
     headers = None
-    # Will be needed in the future by the VisualSelector, always get this where possible.
-    screenshot = False
-    fetcher_description = "No description"
-    system_http_proxy = os.getenv('HTTP_PROXY')
-    system_https_proxy = os.getenv('HTTPS_PROXY')
 
-    # Time ONTOP of the system defined env minimum time
-    render_extract_delay=0
+    fetcher_description ="No description"
 
     @abstractmethod
     def get_error(self):
         return self.error
 
     @abstractmethod
-    def run(self,
-            url,
-            timeout,
-            request_headers,
-            request_body,
-            request_method,
-            ignore_status_codes=False):
+    def run(self, url, timeout, request_headers, request_body, request_method):
         # Should set self.error, self.status_code and self.content
         pass
 
-    @abstractmethod
-    def quit(self):
-        return
-
     @abstractmethod
     def get_last_status_code(self):
         return self.status_code
@@ -64,121 +43,29 @@ class Fetcher():
     def is_ready(self):
         return True
 
-
 #   Maybe for the future, each fetcher provides its own diff output, could be used for text, image
 #   the current one would return javascript output (as we use JS to generate the diff)
 #
+#   Returns tuple(mime_type, stream)
+#    @abstractmethod
+#    def return_diff(self, stream_a, stream_b):
+#        return
+
 def available_fetchers():
-    # See the if statement at the bottom of this file for how we switch between playwright and webdriver
-    import inspect
-    p = []
-    for name, obj in inspect.getmembers(sys.modules[__name__], inspect.isclass):
-        if inspect.isclass(obj):
-            # @todo html_ is maybe better as fetcher_ or something
-            # In this case, make sure to edit the default one in store.py and fetch_site_status.py
-            if name.startswith('html_'):
-                t = tuple([name, obj.fetcher_description])
-                p.append(t)
+        import inspect
+        from changedetectionio import content_fetcher
+        p=[]
+        for name, obj in inspect.getmembers(content_fetcher):
+            if inspect.isclass(obj):
+                # @todo html_ is maybe better as fetcher_ or something
+                # In this case, make sure to edit the default one in store.py and fetch_site_status.py
+                if "html_" in name:
+                    t=tuple([name,obj.fetcher_description])
+                    p.append(t)
 
-    return p
+        return p
 
-
-class base_html_playwright(Fetcher):
-    fetcher_description = "Playwright {}/Javascript".format(
-        os.getenv("PLAYWRIGHT_BROWSER_TYPE", 'chromium').capitalize()
-    )
-    if os.getenv("PLAYWRIGHT_DRIVER_URL"):
-        fetcher_description += " via '{}'".format(os.getenv("PLAYWRIGHT_DRIVER_URL"))
-
-    browser_type = ''
-    command_executor = ''
-
-    # Configs for Proxy setup
-    # In the ENV vars, is prefixed with "playwright_proxy_", so it is for example "playwright_proxy_server"
-    playwright_proxy_settings_mappings = ['bypass', 'server', 'username', 'password']
-
-    proxy = None
-
-    def __init__(self, proxy_override=None):
-
-        # .strip('"') is going to save someone a lot of time when they accidently wrap the env value
-        self.browser_type = os.getenv("PLAYWRIGHT_BROWSER_TYPE", 'chromium').strip('"')
-        self.command_executor = os.getenv(
-            "PLAYWRIGHT_DRIVER_URL",
-            'ws://playwright-chrome:3000'
-        ).strip('"')
-
-        # If any proxy settings are enabled, then we should setup the proxy object
-        proxy_args = {}
-        for k in self.playwright_proxy_settings_mappings:
-            v = os.getenv('playwright_proxy_' + k, False)
-            if v:
-                proxy_args[k] = v.strip('"')
-
-        if proxy_args:
-            self.proxy = proxy_args
-
-        # allow per-watch proxy selection override
-        if proxy_override:
-            self.proxy = {'server': proxy_override}
-
-    def run(self,
-            url,
-            timeout,
-            request_headers,
-            request_body,
-            request_method,
-            ignore_status_codes=False):
-
-        from playwright.sync_api import sync_playwright
-        import playwright._impl._api_types
-        from playwright._impl._api_types import Error, TimeoutError
-
-        with sync_playwright() as p:
-            browser_type = getattr(p, self.browser_type)
-
-            # Seemed to cause a connection Exception even tho I can see it connect
-            # self.browser = browser_type.connect(self.command_executor, timeout=timeout*1000)
-            browser = browser_type.connect_over_cdp(self.command_executor, timeout=timeout * 1000)
-
-            # Set user agent to prevent Cloudflare from blocking the browser
-            # Use the default one configured in the App.py model that's passed from fetch_site_status.py
-            context = browser.new_context(
-                user_agent=request_headers['User-Agent'] if request_headers.get('User-Agent') else 'Mozilla/5.0',
-                proxy=self.proxy
-            )
-            page = context.new_page()
-            page.set_viewport_size({"width": 1280, "height": 1024})
-            try:
-                response = page.goto(url, timeout=timeout * 1000, wait_until='commit')
-                # Wait_until = commit
-                # - `'commit'` - consider operation to be finished when network response is received and the document started loading.
-                # Better to not use any smarts from Playwright and just wait an arbitrary number of seconds
-                # This seemed to solve nearly all 'TimeoutErrors'
-                extra_wait = int(os.getenv("WEBDRIVER_DELAY_BEFORE_CONTENT_READY", 5)) + self.render_extract_delay
-                page.wait_for_timeout(extra_wait * 1000)
-            except playwright._impl._api_types.TimeoutError as e:
-                raise EmptyReply(url=url, status_code=None)
-
-            if response is None:
-                raise EmptyReply(url=url, status_code=None)
-
-            if len(page.content().strip()) == 0:
-                raise EmptyReply(url=url, status_code=None)
-
-            self.status_code = response.status
-            self.content = page.content()
-            self.headers = response.all_headers()
-
-            # Some bug where it gives the wrong screenshot size, but making a request with the clip set first seems to solve it
-            # JPEG is better here because the screenshots can be very very large
-            page.screenshot(type='jpeg', clip={'x': 1.0, 'y': 1.0, 'width': 1280, 'height': 1024})
-            self.screenshot = page.screenshot(type='jpeg', full_page=True, quality=90)
-            context.close()
-            browser.close()
-
-
-class base_html_webdriver(Fetcher):
+class html_webdriver(Fetcher):
     if os.getenv("WEBDRIVER_URL"):
         fetcher_description = "WebDriver Chrome/Javascript via '{}'".format(os.getenv("WEBDRIVER_URL"))
     else:
@@ -191,11 +78,12 @@ class base_html_webdriver(Fetcher):
     selenium_proxy_settings_mappings = ['proxyType', 'ftpProxy', 'httpProxy', 'noProxy',
                                         'proxyAutoconfigUrl', 'sslProxy', 'autodetect',
                                         'socksProxy', 'socksVersion', 'socksUsername', 'socksPassword']
-    proxy = None
 
-    def __init__(self, proxy_override=None):
-        from selenium.webdriver.common.proxy import Proxy as SeleniumProxy
 
+
+    proxy=None
+
+    def __init__(self):
         # .strip('"') is going to save someone a lot of time when they accidently wrap the env value
         self.command_executor = os.getenv("WEBDRIVER_URL", 'http://browser-chrome:4444/wd/hub').strip('"')
 
@@ -206,43 +94,24 @@ class base_html_webdriver(Fetcher):
             if v:
                 proxy_args[k] = v.strip('"')
 
-        # Map back standard HTTP_ and HTTPS_PROXY to webDriver httpProxy/sslProxy
-        if not proxy_args.get('webdriver_httpProxy') and self.system_http_proxy:
-            proxy_args['httpProxy'] = self.system_http_proxy
-        if not proxy_args.get('webdriver_sslProxy') and self.system_https_proxy:
-            proxy_args['httpsProxy'] = self.system_https_proxy
-
-        # Allows override the proxy on a per-request basis
-        if proxy_override is not None:
-            proxy_args['httpProxy'] = proxy_override
-
         if proxy_args:
             self.proxy = SeleniumProxy(raw=proxy_args)
 
-    def run(self,
-            url,
-            timeout,
-            request_headers,
-            request_body,
-            request_method,
-            ignore_status_codes=False):
+    def run(self, url, timeout, request_headers, request_body, request_method):
 
-        from selenium import webdriver
-        from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
-        from selenium.common.exceptions import WebDriverException
         # request_body, request_method unused for now, until some magic in the future happens.
 
         # check env for WEBDRIVER_URL
-        self.driver = webdriver.Remote(
+        driver = webdriver.Remote(
             command_executor=self.command_executor,
             desired_capabilities=DesiredCapabilities.CHROME,
             proxy=self.proxy)
 
         try:
-            self.driver.get(url)
+            driver.get(url)
         except WebDriverException as e:
             # Be sure we close the session window
-            self.quit()
+            driver.quit()
             raise
 
         # @todo - how to check this? is it possible?
@@ -251,91 +120,51 @@ class base_html_webdriver(Fetcher):
         # raise EmptyReply(url=url, status_code=r.status_code)
 
         # @todo - dom wait loaded?
-        time.sleep(int(os.getenv("WEBDRIVER_DELAY_BEFORE_CONTENT_READY", 5)) + self.render_extract_delay)
-        self.content = self.driver.page_source
+        time.sleep(int(os.getenv("WEBDRIVER_DELAY_BEFORE_CONTENT_READY", 5)))
+        self.content = driver.page_source
         self.headers = {}
-        self.screenshot = self.driver.get_screenshot_as_png()
-        self.quit()
 
-    # Does the connection to the webdriver work? run a test connection.
+        driver.quit()
+
+
     def is_ready(self):
         from selenium import webdriver
         from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
         from selenium.common.exceptions import WebDriverException
 
-        self.driver = webdriver.Remote(
+        driver = webdriver.Remote(
             command_executor=self.command_executor,
             desired_capabilities=DesiredCapabilities.CHROME)
 
         # driver.quit() seems to cause better exceptions
-        self.quit()
+        driver.quit()
+
         return True
 
-    def quit(self):
-        if self.driver:
-            try:
-                self.driver.quit()
-            except Exception as e:
-                print("Exception in chrome shutdown/quit" + str(e))
-
-
 # "html_requests" is listed as the default fetcher in store.py!
 class html_requests(Fetcher):
     fetcher_description = "Basic fast Plaintext/HTTP Client"
 
-    def __init__(self, proxy_override=None):
-        self.proxy_override = proxy_override
-
-    def run(self,
-            url,
-            timeout,
-            request_headers,
-            request_body,
-            request_method,
-            ignore_status_codes=False):
-
-        proxies={}
-
-        # Allows override the proxy on a per-request basis
-        if self.proxy_override:
-            proxies = {'http': self.proxy_override, 'https': self.proxy_override, 'ftp': self.proxy_override}
-        else:
-            if self.system_http_proxy:
-                proxies['http'] = self.system_http_proxy
-            if self.system_https_proxy:
-                proxies['https'] = self.system_https_proxy
+    def run(self, url, timeout, request_headers, request_body, request_method):
+        import requests
 
         r = requests.request(method=request_method,
-                             data=request_body,
-                             url=url,
-                             headers=request_headers,
-                             timeout=timeout,
-                             proxies=proxies,
-                             verify=False)
+                         data=request_body,
+                         url=url,
+                         headers=request_headers,
+                         timeout=timeout,
+                         verify=False)
 
-        # If the response did not tell us what encoding format to expect, Then use chardet to override what `requests` thinks.
-        # For example - some sites don't tell us it's utf-8, but return utf-8 content
-        # This seems to not occur when using webdriver/selenium, it seems to detect the text encoding more reliably.
-        # https://github.com/psf/requests/issues/1604 good info about requests encoding detection
-        if not r.headers.get('content-type') or not 'charset=' in r.headers.get('content-type'):
-            encoding = chardet.detect(r.content)['encoding']
-            if encoding:
-                r.encoding = encoding
+        # https://stackoverflow.com/questions/44203397/python-requests-get-returns-improperly-decoded-text-instead-of-utf-8
+        # Return bytes here
+        html = r.text
 
         # @todo test this
         # @todo maybe you really want to test zero-byte return pages?
-        if (not ignore_status_codes and not r) or not r.content or not len(r.content):
+        if not r or not html or not len(html):
             raise EmptyReply(url=url, status_code=r.status_code)
 
         self.status_code = r.status_code
-        self.content = r.text
+        self.content = html
         self.headers = r.headers
 
-
-# Decide which is the 'real' HTML webdriver, this is more a system wide config
-# rather than site-specific.
-use_playwright_as_chrome_fetcher = os.getenv('PLAYWRIGHT_DRIVER_URL', False)
-if use_playwright_as_chrome_fetcher:
-    html_webdriver = base_html_playwright
-else:
-    html_webdriver = base_html_webdriver

changedetectionio/diff.py

@@ -2,31 +2,22 @@
 
 import difflib
 
-
-def same_slicer(l, a, b):
-    if a == b:
-        return [l[a]]
-    else:
-        return l[a:b]
-
 # like .compare but a little different output
 def customSequenceMatcher(before, after, include_equal=False):
     cruncher = difflib.SequenceMatcher(isjunk=lambda x: x in " \\t", a=before, b=after)
 
-    # @todo Line-by-line mode instead of buncghed, including `after` that is not in `before` (maybe unset?)
     for tag, alo, ahi, blo, bhi in cruncher.get_opcodes():
         if include_equal and tag == 'equal':
             g = before[alo:ahi]
             yield g
         elif tag == 'delete':
-            g = ["(removed) " + i for i in same_slicer(before, alo, ahi)]
+            g = "(removed) {}".format(before[alo])
             yield g
         elif tag == 'replace':
-            g = ["(changed) " + i for i in same_slicer(before, alo, ahi)]
-            g += ["(into   ) " + i for i in same_slicer(after, blo, bhi)]
+            g = ["(changed) {}".format(before[alo]), "(-> into) {}".format(after[blo])]
             yield g
         elif tag == 'insert':
-            g = ["(added  ) " + i for i in same_slicer(after, blo, bhi)]
+            g = "(added) {}".format(after[blo])
             yield g
 
 # only_differences - only return info about the differences, no context

changedetectionio/fetch_site_status.py

@@ -1,10 +1,11 @@
-import hashlib
-import os
-import re
 import time
+from changedetectionio import content_fetcher
+from changedetectionio import html_tools
+import hashlib
+from inscriptis import get_text
 import urllib3
-
-from changedetectionio import content_fetcher, html_tools
+from . import html_tools
+import re
 
 urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
 
@@ -16,50 +17,15 @@ class perform_site_check():
         super().__init__(*args, **kwargs)
         self.datastore = datastore
 
-    # If there was a proxy list enabled, figure out what proxy_args/which proxy to use
-    # if watch.proxy use that
-    # fetcher.proxy_override = watch.proxy or main config proxy
-    # Allows override the proxy on a per-request basis
-    # ALWAYS use the first one is nothing selected
-
-    def set_proxy_from_list(self, watch):
-        proxy_args = None
-        if self.datastore.proxy_list is None:
-            return None
-
-        # If its a valid one
-        if any([watch['proxy'] in p for p in self.datastore.proxy_list]):
-            proxy_args = watch['proxy']
-
-        # not valid (including None), try the system one
-        else:
-            system_proxy = self.datastore.data['settings']['requests']['proxy']
-            # Is not None and exists
-            if any([system_proxy in p for p in self.datastore.proxy_list]):
-                proxy_args = system_proxy
-
-        # Fallback - Did not resolve anything, use the first available
-        if proxy_args is None:
-            proxy_args = self.datastore.proxy_list[0][0]
-
-        return proxy_args
-
     def run(self, uuid):
         timestamp = int(time.time())  # used for storage etc too
 
         changed_detected = False
-        screenshot = False  # as bytes
         stripped_text_from_html = ""
 
         watch = self.datastore.data['watching'][uuid]
-
-        # Protect against file:// access
-        if re.search(r'^file', watch['url'], re.IGNORECASE) and not os.getenv('ALLOW_FILE_URI', False):
-            raise Exception(
-                "file:// type access is denied for security reasons."
-            )
-
         # Unset any existing notification error
+
         update_obj = {'last_notification_error': False, 'last_error': False}
 
         extra_headers = self.datastore.get_val(uuid, 'headers')
@@ -74,166 +40,122 @@ class perform_site_check():
         if 'Accept-Encoding' in request_headers and "br" in request_headers['Accept-Encoding']:
             request_headers['Accept-Encoding'] = request_headers['Accept-Encoding'].replace(', br', '')
 
-        timeout = self.datastore.data['settings']['requests']['timeout']
-        url = self.datastore.get_val(uuid, 'url')
-        request_body = self.datastore.get_val(uuid, 'body')
-        request_method = self.datastore.get_val(uuid, 'method')
-        ignore_status_code = self.datastore.get_val(uuid, 'ignore_status_codes')
+        # @todo check the failures are really handled how we expect
 
-        # source: support
-        is_source = False
-        if url.startswith('source:'):
-            url = url.replace('source:', '')
-            is_source = True
-
-        # Pluggable content fetcher
-        prefer_backend = watch['fetch_backend']
-        if hasattr(content_fetcher, prefer_backend):
-            klass = getattr(content_fetcher, prefer_backend)
         else:
-            # If the klass doesnt exist, just use a default
-            klass = getattr(content_fetcher, "html_requests")
+            timeout = self.datastore.data['settings']['requests']['timeout']
+            url = self.datastore.get_val(uuid, 'url')
+            request_body = self.datastore.get_val(uuid, 'body')
+            request_method = self.datastore.get_val(uuid, 'method')
 
-        proxy_args = self.set_proxy_from_list(watch)
-        fetcher = klass(proxy_override=proxy_args)
-
-        # Configurable per-watch or global extra delay before extracting text (for webDriver types)
-        system_webdriver_delay = self.datastore.data['settings']['application'].get('webdriver_delay', None)
-        if watch['webdriver_delay'] is not None:
-            fetcher.render_extract_delay = watch['webdriver_delay']
-        elif system_webdriver_delay is not None:
-            fetcher.render_extract_delay = system_webdriver_delay
-
-        fetcher.run(url, timeout, request_headers, request_body, request_method, ignore_status_code)
-
-        # Fetching complete, now filters
-        # @todo move to class / maybe inside of fetcher abstract base?
-
-        # @note: I feel like the following should be in a more obvious chain system
-        #  - Check filter text
-        #  - Is the checksum different?
-        #  - Do we convert to JSON?
-        # https://stackoverflow.com/questions/41817578/basic-method-chaining ?
-        # return content().textfilter().jsonextract().checksumcompare() ?
-
-        is_json = 'application/json' in fetcher.headers.get('Content-Type', '')
-        is_html = not is_json
-
-        # source: support, basically treat it as plaintext
-        if is_source:
-            is_html = False
-            is_json = False
-
-        css_filter_rule = watch['css_filter']
-        subtractive_selectors = watch.get(
-            "subtractive_selectors", []
-        ) + self.datastore.data["settings"]["application"].get(
-            "global_subtractive_selectors", []
-        )
-
-        has_filter_rule = css_filter_rule and len(css_filter_rule.strip())
-        has_subtractive_selectors = subtractive_selectors and len(subtractive_selectors[0].strip())
-
-        if is_json and not has_filter_rule:
-            css_filter_rule = "json:$"
-            has_filter_rule = True
-
-        if has_filter_rule:
-            if 'json:' in css_filter_rule:
-                stripped_text_from_html = html_tools.extract_json_as_string(content=fetcher.content, jsonpath_filter=css_filter_rule)
-                is_html = False
-
-        if is_html or is_source:
-            # CSS Filter, extract the HTML that matches and feed that into the existing inscriptis::get_text
-            html_content = fetcher.content
-
-            # If not JSON,  and if it's not text/plain..
-            if 'text/plain' in fetcher.headers.get('Content-Type', '').lower():
-                # Don't run get_text or xpath/css filters on plaintext
-                stripped_text_from_html = html_content
+            # Pluggable content fetcher
+            prefer_backend = watch['fetch_backend']
+            if hasattr(content_fetcher, prefer_backend):
+                klass = getattr(content_fetcher, prefer_backend)
             else:
-                # Then we assume HTML
-                if has_filter_rule:
-                    # For HTML/XML we offer xpath as an option, just start a regular xPath "/.."
-                    if css_filter_rule[0] == '/' or css_filter_rule.startswith('xpath:'):
-                        html_content = html_tools.xpath_filter(xpath_filter=css_filter_rule.replace('xpath:', ''),
-                                                               html_content=fetcher.content)
-                    else:
-                        # CSS Filter, extract the HTML that matches and feed that into the existing inscriptis::get_text
-                        html_content = html_tools.css_filter(css_filter=css_filter_rule, html_content=fetcher.content)
+                # If the klass doesnt exist, just use a default
+                klass = getattr(content_fetcher, "html_requests")
 
-                if has_subtractive_selectors:
-                    html_content = html_tools.element_removal(subtractive_selectors, html_content)
 
-                if not is_source:
-                    # extract text
-                    stripped_text_from_html = \
-                        html_tools.html_to_text(
-                            html_content,
-                            render_anchor_tag_content=self.datastore.data["settings"][
-                                "application"].get(
-                                "render_anchor_tag_content", False)
-                        )
+            fetcher = klass()
+            fetcher.run(url, timeout, request_headers, request_body, request_method)
+            # Fetching complete, now filters
+            # @todo move to class / maybe inside of fetcher abstract base?
 
-                elif is_source:
+            # @note: I feel like the following should be in a more obvious chain system
+            #  - Check filter text
+            #  - Is the checksum different?
+            #  - Do we convert to JSON?
+            # https://stackoverflow.com/questions/41817578/basic-method-chaining ?
+            # return content().textfilter().jsonextract().checksumcompare() ?
+
+            is_json = 'application/json' in fetcher.headers.get('Content-Type', '')
+            is_html = not is_json
+            css_filter_rule = watch['css_filter']
+
+            has_filter_rule = css_filter_rule and len(css_filter_rule.strip())
+            if is_json and not has_filter_rule:
+                css_filter_rule = "json:$"
+                has_filter_rule = True
+
+            if has_filter_rule:
+                if 'json:' in css_filter_rule:
+                    stripped_text_from_html = html_tools.extract_json_as_string(content=fetcher.content, jsonpath_filter=css_filter_rule)
+                    is_html = False
+
+            if is_html:
+                # CSS Filter, extract the HTML that matches and feed that into the existing inscriptis::get_text
+                html_content = fetcher.content
+
+                # If not JSON,  and if it's not text/plain..
+                if 'text/plain' in fetcher.headers.get('Content-Type', '').lower():
+                    # Don't run get_text or xpath/css filters on plaintext
                     stripped_text_from_html = html_content
+                else:
+                    # Then we assume HTML
+                    if has_filter_rule:
+                        # For HTML/XML we offer xpath as an option, just start a regular xPath "/.."
+                        if css_filter_rule[0] == '/':
+                            html_content = html_tools.xpath_filter(xpath_filter=css_filter_rule, html_content=fetcher.content)
+                        else:
+                            # CSS Filter, extract the HTML that matches and feed that into the existing inscriptis::get_text
+                            html_content = html_tools.css_filter(css_filter=css_filter_rule, html_content=fetcher.content)
+
+                    # get_text() via inscriptis
+                    stripped_text_from_html = get_text(html_content)
+
 
             # Re #340 - return the content before the 'ignore text' was applied
             text_content_before_ignored_filter = stripped_text_from_html.encode('utf-8')
 
-        # Re #340 - return the content before the 'ignore text' was applied
-        text_content_before_ignored_filter = stripped_text_from_html.encode('utf-8')
+            # We rely on the actual text in the html output.. many sites have random script vars etc,
+            # in the future we'll implement other mechanisms.
 
-        # Treat pages with no renderable text content as a change? No by default
-        empty_pages_are_a_change = self.datastore.data['settings']['application'].get('empty_pages_are_a_change', False)
-        if not is_json and not empty_pages_are_a_change and len(stripped_text_from_html.strip()) == 0:
-            raise content_fetcher.ReplyWithContentButNoText(url=url, status_code=200)
+            update_obj["last_check_status"] = fetcher.get_last_status_code()
 
-        # We rely on the actual text in the html output.. many sites have random script vars etc,
-        # in the future we'll implement other mechanisms.
+            # If there's text to skip
+            # @todo we could abstract out the get_text() to handle this cleaner
+            text_to_ignore = watch.get('ignore_text', []) + self.datastore.data['settings']['application'].get('global_ignore_text', [])
+            if len(text_to_ignore):
+                stripped_text_from_html = html_tools.strip_ignore_text(stripped_text_from_html, text_to_ignore)
+            else:
+                stripped_text_from_html = stripped_text_from_html.encode('utf8')
 
-        update_obj["last_check_status"] = fetcher.get_last_status_code()
+            # Re #133 - if we should strip whitespaces from triggering the change detected comparison
+            if self.datastore.data['settings']['application'].get('ignore_whitespace', False):
+                fetched_md5 = hashlib.md5(stripped_text_from_html.translate(None, b'\r\n\t ')).hexdigest()
+            else:
+                fetched_md5 = hashlib.md5(stripped_text_from_html).hexdigest()
 
-        # If there's text to skip
-        # @todo we could abstract out the get_text() to handle this cleaner
-        text_to_ignore = watch.get('ignore_text', []) + self.datastore.data['settings']['application'].get('global_ignore_text', [])
-        if len(text_to_ignore):
-            stripped_text_from_html = html_tools.strip_ignore_text(stripped_text_from_html, text_to_ignore)
-        else:
-            stripped_text_from_html = stripped_text_from_html.encode('utf8')
+            # On the first run of a site, watch['previous_md5'] will be an empty string, set it the current one.
+            if not len(watch['previous_md5']):
+                watch['previous_md5'] = fetched_md5
+                update_obj["previous_md5"] = fetched_md5
 
-        # Re #133 - if we should strip whitespaces from triggering the change detected comparison
-        if self.datastore.data['settings']['application'].get('ignore_whitespace', False):
-            fetched_md5 = hashlib.md5(stripped_text_from_html.translate(None, b'\r\n\t ')).hexdigest()
-        else:
-            fetched_md5 = hashlib.md5(stripped_text_from_html).hexdigest()
+            blocked_by_not_found_trigger_text = False
 
-        # On the first run of a site, watch['previous_md5'] will be None, set it the current one.
-        if not watch.get('previous_md5'):
-            watch['previous_md5'] = fetched_md5
-            update_obj["previous_md5"] = fetched_md5
+            if len(watch['trigger_text']):
+                # Yeah, lets block first until something matches
+                blocked_by_not_found_trigger_text = True
+                # Filter and trigger works the same, so reuse it
+                result = html_tools.strip_ignore_text(content=str(stripped_text_from_html),
+                                                      wordlist=watch['trigger_text'],
+                                                      mode="line numbers")
+                if result:
+                    blocked_by_not_found_trigger_text = False
 
-        blocked_by_not_found_trigger_text = False
 
-        if len(watch['trigger_text']):
-            # Yeah, lets block first until something matches
-            blocked_by_not_found_trigger_text = True
-            # Filter and trigger works the same, so reuse it
-            result = html_tools.strip_ignore_text(content=str(stripped_text_from_html),
-                                                  wordlist=watch['trigger_text'],
-                                                  mode="line numbers")
-            if result:
-                blocked_by_not_found_trigger_text = False
+            if not blocked_by_not_found_trigger_text and watch['previous_md5'] != fetched_md5:
+                changed_detected = True
+                update_obj["previous_md5"] = fetched_md5
+                update_obj["last_changed"] = timestamp
 
-        if not blocked_by_not_found_trigger_text and watch['previous_md5'] != fetched_md5:
-            changed_detected = True
-            update_obj["previous_md5"] = fetched_md5
-            update_obj["last_changed"] = timestamp
 
-        # Extract title as title
-        if is_html:
-            if self.datastore.data['settings']['application']['extract_title_as_title'] or watch['extract_title_as_title']:
-                if not watch['title'] or not len(watch['title']):
-                    update_obj['title'] = html_tools.extract_element(find='title', html_content=fetcher.content)
+            # Extract title as title
+            if is_html:
+                if self.datastore.data['settings']['application']['extract_title_as_title'] or watch['extract_title_as_title']:
+                    if not watch['title'] or not len(watch['title']):
+                        update_obj['title'] = html_tools.extract_element(find='title', html_content=fetcher.content)
 
-        return changed_detected, update_obj, text_content_before_ignored_filter, fetcher.screenshot
+
+        return changed_detected, update_obj, text_content_before_ignored_filter

changedetectionio/forms.py

@@ -1,31 +1,13 @@
+from wtforms import Form, SelectField, RadioField, BooleanField, StringField, PasswordField, validators, IntegerField, fields, TextAreaField, \
+    Field
+
+from wtforms import widgets, SubmitField
+from wtforms.validators import ValidationError
+from wtforms.fields import html5
+from changedetectionio import content_fetcher
 import re
 
-from wtforms import (
-    BooleanField,
-    Field,
-    Form,
-    IntegerField,
-    PasswordField,
-    RadioField,
-    SelectField,
-    StringField,
-    SubmitField,
-    TextAreaField,
-    fields,
-    validators,
-    widgets,
-)
-from wtforms.validators import ValidationError
-
-from changedetectionio import content_fetcher
-from changedetectionio.notification import (
-    default_notification_body,
-    default_notification_format,
-    default_notification_title,
-    valid_notification_formats,
-)
-
-from wtforms.fields import FormField
+from changedetectionio.notification import default_notification_format, valid_notification_formats, default_notification_body, default_notification_title
 
 valid_method = {
     'GET',
@@ -37,38 +19,34 @@ valid_method = {
 
 default_method = 'GET'
 
-
 class StringListField(StringField):
     widget = widgets.TextArea()
 
     def _value(self):
         if self.data:
-            # ignore empty lines in the storage
-            data = list(filter(lambda x: len(x.strip()), self.data))
-            # Apply strip to each line
-            data = list(map(lambda x: x.strip(), data))
-            return "\r\n".join(data)
+            return "\r\n".join(self.data)
         else:
             return u''
 
     # incoming
     def process_formdata(self, valuelist):
-        if valuelist and len(valuelist[0].strip()):
-            # Remove empty strings, stripping and splitting \r\n, only \n etc.
-            self.data = valuelist[0].splitlines()
-            # Remove empty lines from the final data
-            self.data = list(filter(lambda x: len(x.strip()), self.data))
+        if valuelist:
+            # Remove empty strings
+            cleaned = list(filter(None, valuelist[0].split("\n")))
+            self.data = [x.strip() for x in cleaned]
+            p = 1
         else:
             self.data = []
 
 
+
 class SaltyPasswordField(StringField):
     widget = widgets.PasswordInput()
     encrypted_password = ""
 
     def build_password(self, password):
-        import base64
         import hashlib
+        import base64
         import secrets
 
         # Make a new salt on every new password and store it with the password
@@ -89,13 +67,6 @@ class SaltyPasswordField(StringField):
         else:
             self.data = False
 
-class TimeBetweenCheckForm(Form):
-    weeks = IntegerField('Weeks', validators=[validators.Optional(), validators.NumberRange(min=0, message="Should contain zero or more seconds")])
-    days = IntegerField('Days', validators=[validators.Optional(), validators.NumberRange(min=0, message="Should contain zero or more seconds")])
-    hours = IntegerField('Hours', validators=[validators.Optional(), validators.NumberRange(min=0, message="Should contain zero or more seconds")])
-    minutes = IntegerField('Minutes', validators=[validators.Optional(), validators.NumberRange(min=0, message="Should contain zero or more seconds")])
-    seconds = IntegerField('Seconds', validators=[validators.Optional(), validators.NumberRange(min=0, message="Should contain zero or more seconds")])
-    # @todo add total seconds minimum validatior = minimum_seconds_recheck_time
 
 # Separated by  key:value
 class StringDictKeyValue(StringField):
@@ -133,8 +104,8 @@ class ValidateContentFetcherIsReady(object):
         self.message = message
 
     def __call__(self, form, field):
-        import urllib3.exceptions
         from changedetectionio import content_fetcher
+        import urllib3.exceptions
 
         # Better would be a radiohandler that keeps a reference to each class
         if field.data is not None:
@@ -249,98 +220,79 @@ class ValidateCSSJSONXPATHInput(object):
     @todo CSS validator ;)
     """
 
-    def __init__(self, message=None, allow_xpath=True, allow_json=True):
+    def __init__(self, message=None):
         self.message = message
-        self.allow_xpath = allow_xpath
-        self.allow_json = allow_json
 
     def __call__(self, form, field):
 
-        if isinstance(field.data, str):
-            data = [field.data]
-        else:
-            data = field.data
-
-        for line in data:
         # Nothing to see here
-            if not len(line.strip()):
-                return
+        if not len(field.data.strip()):
+            return
 
-            # Does it look like XPath?
-            if line.strip()[0] == '/':
-                if not self.allow_xpath:
-                    raise ValidationError("XPath not permitted in this field!")
-                from lxml import etree, html
-                tree = html.fromstring("<html></html>")
+        # Does it look like XPath?
+        if field.data.strip()[0] == '/':
+            from lxml import html, etree
+            tree = html.fromstring("<html></html>")
 
-                try:
-                    tree.xpath(line.strip())
-                except etree.XPathEvalError as e:
-                    message = field.gettext('\'%s\' is not a valid XPath expression. (%s)')
-                    raise ValidationError(message % (line, str(e)))
-                except:
-                    raise ValidationError("A system-error occurred when validating your XPath expression")
+            try:
+                tree.xpath(field.data.strip())
+            except etree.XPathEvalError as e:
+                message = field.gettext('\'%s\' is not a valid XPath expression. (%s)')
+                raise ValidationError(message % (field.data, str(e)))
+            except:
+                raise ValidationError("A system-error occurred when validating your XPath expression")
 
-            if 'json:' in line:
-                if not self.allow_json:
-                    raise ValidationError("JSONPath not permitted in this field!")
+        if 'json:' in field.data:
+            from jsonpath_ng.exceptions import JsonPathParserError, JsonPathLexerError
+            from jsonpath_ng.ext import parse
 
-                from jsonpath_ng.exceptions import (
-                    JsonPathLexerError,
-                    JsonPathParserError,
-                )
-                from jsonpath_ng.ext import parse
+            input = field.data.replace('json:', '')
 
-                input = line.replace('json:', '')
-
-                try:
-                    parse(input)
-                except (JsonPathParserError, JsonPathLexerError) as e:
-                    message = field.gettext('\'%s\' is not a valid JSONPath expression. (%s)')
-                    raise ValidationError(message % (input, str(e)))
-                except:
-                    raise ValidationError("A system-error occurred when validating your JSONPath expression")
-
-                # Re #265 - maybe in the future fetch the page and offer a
-                # warning/notice that its possible the rule doesnt yet match anything?
+            try:
+                parse(input)
+            except (JsonPathParserError, JsonPathLexerError) as e:
+                message = field.gettext('\'%s\' is not a valid JSONPath expression. (%s)')
+                raise ValidationError(message % (input, str(e)))
+            except:
+                raise ValidationError("A system-error occurred when validating your JSONPath expression")
 
+            # Re #265 - maybe in the future fetch the page and offer a
+            # warning/notice that its possible the rule doesnt yet match anything?
 
 class quickWatchForm(Form):
-    url = fields.URLField('URL', validators=[validateURL()])
+    # https://wtforms.readthedocs.io/en/2.3.x/fields/#module-wtforms.fields.html5
+    # `require_tld` = False is needed even for the test harness "http://localhost:5005.." to run
+    url = html5.URLField('URL', validators=[validateURL()])
     tag = StringField('Group tag', [validators.Optional(), validators.Length(max=35)])
 
-# Common to a single watch and the global settings
 class commonSettingsForm(Form):
 
-    notification_urls = StringListField('Notification URL list', validators=[validators.Optional(), ValidateNotificationBodyAndTitleWhenURLisSet(), ValidateAppRiseServers()])
-    notification_title = StringField('Notification title', default=default_notification_title, validators=[validators.Optional(), ValidateTokensList()])
-    notification_body = TextAreaField('Notification body', default=default_notification_body, validators=[validators.Optional(), ValidateTokensList()])
-    notification_format = SelectField('Notification format', choices=valid_notification_formats.keys(), default=default_notification_format)
-    fetch_backend = RadioField(u'Fetch method', choices=content_fetcher.available_fetchers(), validators=[ValidateContentFetcherIsReady()])
+    notification_urls = StringListField('Notification URL List', validators=[validators.Optional(), ValidateNotificationBodyAndTitleWhenURLisSet(), ValidateAppRiseServers()])
+    notification_title = StringField('Notification Title', default=default_notification_title, validators=[validators.Optional(), ValidateTokensList()])
+    notification_body = TextAreaField('Notification Body', default=default_notification_body, validators=[validators.Optional(), ValidateTokensList()])
+    notification_format = SelectField('Notification Format', choices=valid_notification_formats.keys(), default=default_notification_format)
+    trigger_check = BooleanField('Send test notification on save')
+    fetch_backend = RadioField(u'Fetch Method', choices=content_fetcher.available_fetchers(), validators=[ValidateContentFetcherIsReady()])
     extract_title_as_title = BooleanField('Extract <title> from document and use as watch title', default=False)
-    webdriver_delay = IntegerField('Wait seconds before extracting text', validators=[validators.Optional(), validators.NumberRange(min=1, message="Should contain one or more seconds")] )
 
 class watchForm(commonSettingsForm):
 
-    url = fields.URLField('URL', validators=[validateURL()])
-    tag = StringField('Group tag', [validators.Optional(), validators.Length(max=35)], default='')
+    url = html5.URLField('URL', validators=[validateURL()])
+    tag = StringField('Group tag', [validators.Optional(), validators.Length(max=35)])
 
-    time_between_check = FormField(TimeBetweenCheckForm)
+    minutes_between_check = html5.IntegerField('Maximum time in minutes until recheck',
+                                               [validators.Optional(), validators.NumberRange(min=1)])
+    css_filter = StringField('CSS/JSON/XPATH Filter', [ValidateCSSJSONXPATHInput()])
+    title = StringField('Title')
 
-    css_filter = StringField('CSS/JSON/XPATH Filter', [ValidateCSSJSONXPATHInput()], default='')
-
-    subtractive_selectors = StringListField('Remove elements', [ValidateCSSJSONXPATHInput(allow_xpath=False, allow_json=False)])
-    title = StringField('Title', default='')
-
-    ignore_text = StringListField('Ignore text', [ValidateListRegex()])
-    headers = StringDictKeyValue('Request headers')
-    body = TextAreaField('Request body', [validators.Optional()])
-    method = SelectField('Request method', choices=valid_method, default=default_method)
-    ignore_status_codes = BooleanField('Ignore status codes (process non-2xx status codes as normal)', default=False)
+    ignore_text = StringListField('Ignore Text', [ValidateListRegex()])
+    headers = StringDictKeyValue('Request Headers')
+    body = TextAreaField('Request Body', [validators.Optional()])
+    method = SelectField('Request Method', choices=valid_method, default=default_method)
     trigger_text = StringListField('Trigger/wait for text', [validators.Optional(), ValidateListRegex()])
+
     save_button = SubmitField('Save', render_kw={"class": "pure-button pure-button-primary"})
     save_and_preview_button = SubmitField('Save & Preview', render_kw={"class": "pure-button pure-button-primary"})
-    proxy = RadioField('Proxy')
 
     def validate(self, **kwargs):
         if not super().validate():
@@ -355,33 +307,12 @@ class watchForm(commonSettingsForm):
 
         return result
 
+class globalSettingsForm(commonSettingsForm):
 
-# datastore.data['settings']['requests']..
-class globalSettingsRequestForm(Form):
-    time_between_check = FormField(TimeBetweenCheckForm)
-    proxy = RadioField('Proxy')
-
-
-# datastore.data['settings']['application']..
-class globalSettingsApplicationForm(commonSettingsForm):
-
-    base_url = StringField('Base URL', validators=[validators.Optional()])
-    global_subtractive_selectors = StringListField('Remove elements', [ValidateCSSJSONXPATHInput(allow_xpath=False, allow_json=False)])
-    global_ignore_text = StringListField('Ignore Text', [ValidateListRegex()])
-    ignore_whitespace = BooleanField('Ignore whitespace')
-    real_browser_save_screenshot = BooleanField('Save last screenshot when using Chrome?')
-    removepassword_button = SubmitField('Remove password', render_kw={"class": "pure-button pure-button-primary"})
-    empty_pages_are_a_change =  BooleanField('Treat empty pages as a change?', default=False)
-    render_anchor_tag_content = BooleanField('Render anchor tag content', default=False)
-    fetch_backend = RadioField('Fetch Method', default="html_requests", choices=content_fetcher.available_fetchers(), validators=[ValidateContentFetcherIsReady()])
     password = SaltyPasswordField()
-
-
-class globalSettingsForm(Form):
-    # Define these as FormFields/"sub forms", this way it matches the JSON storage
-    # datastore.data['settings']['application']..
-    # datastore.data['settings']['requests']..
-
-    requests = FormField(globalSettingsRequestForm)
-    application = FormField(globalSettingsApplicationForm)
-    save_button = SubmitField('Save', render_kw={"class": "pure-button pure-button-primary"})
+    minutes_between_check = html5.IntegerField('Maximum time in minutes until recheck',
+                                               [validators.NumberRange(min=1)])
+    extract_title_as_title = BooleanField('Extract <title> from document and use as watch title')
+    base_url = StringField('Base URL', validators=[validators.Optional()])
+    global_ignore_text = StringListField('Ignore Text', [ValidateListRegex()])
+    ignore_whitespace = BooleanField('Ignore whitespace')

changedetectionio/html_tools.py

@@ -1,13 +1,7 @@
 import json
-import re
-from typing import List
-
 from bs4 import BeautifulSoup
 from jsonpath_ng.ext import parse
 import re
-from inscriptis import get_text
-from inscriptis.model.config import ParserConfig
-
 
 class JSONNotFound(ValueError):
     def __init__(self, msg):
@@ -22,22 +16,11 @@ def css_filter(css_filter, html_content):
 
     return html_block + "\n"
 
-def subtractive_css_selector(css_selector, html_content):
-    soup = BeautifulSoup(html_content, "html.parser")
-    for item in soup.select(css_selector):
-        item.decompose()
-    return str(soup)
-
-
-def element_removal(selectors: List[str], html_content):
-    """Joins individual filters into one css filter."""
-    selector = ",".join(selectors)
-    return subtractive_css_selector(selector, html_content)
-
 
 # Return str Utf-8 of matched rules
 def xpath_filter(xpath_filter, html_content):
-    from lxml import etree, html
+    from lxml import html
+    from lxml import etree
 
     tree = html.fromstring(html_content)
     html_block = ""
@@ -81,8 +64,7 @@ def _parse_json(json_data, jsonpath_filter):
         # Re 265 - Just return an empty string when filter not found
         return ''
 
-    # Ticket #462 - allow the original encoding through, usually it's UTF-8 or similar
-    stripped_text_from_html = json.dumps(s, indent=4, ensure_ascii=False)
+    stripped_text_from_html = json.dumps(s, indent=4)
 
     return stripped_text_from_html
 
@@ -169,36 +151,4 @@ def strip_ignore_text(content, wordlist, mode="content"):
     if mode == "line numbers":
         return ignored_line_numbers
 
-    return "\n".encode('utf8').join(output)
-
-
-def html_to_text(html_content: str, render_anchor_tag_content=False) -> str:
-    """Converts html string to a string with just the text. If ignoring
-    rendering anchor tag content is enable, anchor tag content are also
-    included in the text
-
-    :param html_content: string with html content
-    :param render_anchor_tag_content: boolean flag indicating whether to extract
-    hyperlinks (the anchor tag content) together with text. This refers to the
-    'href' inside 'a' tags.
-    Anchor tag content is rendered in the following manner:
-    '[ text ](anchor tag content)'
-    :return: extracted text from the HTML
-    """
-    #  if anchor tag content flag is set to True define a config for
-    #  extracting this content
-    if render_anchor_tag_content:
-
-        parser_config = ParserConfig(
-            annotation_rules={"a": ["hyperlink"]}, display_links=True
-        )
-
-    # otherwise set config to None
-    else:
-        parser_config = None
-
-    # get text and annotations via inscriptis
-    text_content = get_text(html_content, config=parser_config)
-
-    return text_content
-
+    return "\n".encode('utf8').join(output)

changedetectionio/importer.py

@@ -1,133 +0,0 @@
-from abc import ABC, abstractmethod
-import time
-import validators
-
-
-class Importer():
-    remaining_data = []
-    new_uuids = []
-    good = 0
-
-    def __init__(self):
-        self.new_uuids = []
-        self.good = 0
-        self.remaining_data = []
-
-    @abstractmethod
-    def run(self,
-            data,
-            flash,
-            datastore):
-        pass
-
-
-class import_url_list(Importer):
-    """
-    Imports a list, can be in <code>https://example.com tag1, tag2, last tag</code> format
-    """
-    def run(self,
-            data,
-            flash,
-            datastore,
-            ):
-
-        urls = data.split("\n")
-        good = 0
-        now = time.time()
-
-        if (len(urls) > 5000):
-            flash("Importing 5,000 of the first URLs from your list, the rest can be imported again.")
-
-        for url in urls:
-            url = url.strip()
-            if not len(url):
-                continue
-
-            tags = ""
-
-            # 'tags' should be a csv list after the URL
-            if ' ' in url:
-                url, tags = url.split(" ", 1)
-
-            # Flask wtform validators wont work with basic auth, use validators package
-            # Up to 5000 per batch so we dont flood the server
-            if len(url) and validators.url(url.replace('source:', '')) and good < 5000:
-                new_uuid = datastore.add_watch(url=url.strip(), tag=tags, write_to_disk_now=False)
-                if new_uuid:
-                    # Straight into the queue.
-                    self.new_uuids.append(new_uuid)
-                    good += 1
-                    continue
-
-            # Worked past the 'continue' above, append it to the bad list
-            if self.remaining_data is None:
-                self.remaining_data = []
-            self.remaining_data.append(url)
-
-        flash("{} Imported from list in {:.2f}s, {} Skipped.".format(good, time.time() - now, len(self.remaining_data)))
-
-
-class import_distill_io_json(Importer):
-    def run(self,
-            data,
-            flash,
-            datastore,
-            ):
-
-        import json
-        good = 0
-        now = time.time()
-        self.new_uuids=[]
-
-
-        try:
-            data = json.loads(data.strip())
-        except json.decoder.JSONDecodeError:
-            flash("Unable to read JSON file, was it broken?", 'error')
-            return
-
-        if not data.get('data'):
-            flash("JSON structure looks invalid, was it broken?", 'error')
-            return
-
-        for d in data.get('data'):
-            d_config = json.loads(d['config'])
-            extras = {'title': d['name']}
-
-            if len(d['uri']) and good < 5000:
-                try:
-                    # @todo we only support CSS ones at the moment
-                    if d_config['selections'][0]['frames'][0]['excludes'][0]['type'] == 'css':
-                        extras['subtractive_selectors'] = d_config['selections'][0]['frames'][0]['excludes'][0]['expr']
-                except KeyError:
-                    pass
-                except IndexError:
-                    pass
-
-                try:
-                    extras['css_filter'] = d_config['selections'][0]['frames'][0]['includes'][0]['expr']
-                    if d_config['selections'][0]['frames'][0]['includes'][0]['type'] == 'xpath':
-                        extras['css_filter'] = 'xpath:' + extras['css_filter']
-
-                except KeyError:
-                    pass
-                except IndexError:
-                    pass
-
-                try:
-                    extras['tag'] = ", ".join(d['tags'])
-                except KeyError:
-                    pass
-                except IndexError:
-                    pass
-
-                new_uuid = datastore.add_watch(url=d['uri'].strip(),
-                                               extras=extras,
-                                               write_to_disk_now=False)
-
-                if new_uuid:
-                    # Straight into the queue.
-                    self.new_uuids.append(new_uuid)
-                    good += 1
-
-        flash("{} Imported from Distill.io in {:.2f}s, {} Skipped.".format(len(self.new_uuids), time.time() - now, len(self.remaining_data)))

changedetectionio/model/App.py

@@ -1,53 +0,0 @@
-import collections
-import os
-
-import uuid as uuid_builder
-
-from changedetectionio.notification import (
-    default_notification_body,
-    default_notification_format,
-    default_notification_title,
-)
-
-class model(dict):
-    base_config = {
-            'note': "Hello! If you change this file manually, please be sure to restart your changedetection.io instance!",
-            'watching': {},
-            'settings': {
-                'headers': {
-                    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36',
-                    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
-                    'Accept-Encoding': 'gzip, deflate',  # No support for brolti in python requests yet.
-                    'Accept-Language': 'en-GB,en-US;q=0.9,en;'
-                },
-                'requests': {
-                    'timeout': 15,  # Default 15 seconds
-                    'time_between_check': {'weeks': None, 'days': None, 'hours': 3, 'minutes': None, 'seconds': None},
-                    'workers': 10,  # Number of threads, lower is better for slow connections
-                    'proxy': None # Preferred proxy connection
-                },
-                'application': {
-                    'password': False,
-                    'base_url' : None,
-                    'extract_title_as_title': False,
-                    'empty_pages_are_a_change': False,
-                    'fetch_backend': os.getenv("DEFAULT_FETCH_BACKEND", "html_requests"),
-                    'global_ignore_text': [], # List of text to ignore when calculating the comparison checksum
-                    'global_subtractive_selectors': [],
-                    'ignore_whitespace': False,
-                    'render_anchor_tag_content': False,
-                    'notification_urls': [], # Apprise URL list
-                    # Custom notification content
-                    'notification_title': default_notification_title,
-                    'notification_body': default_notification_body,
-                    'notification_format': default_notification_format,
-                    'real_browser_save_screenshot': True,
-                    'schema_version' : 0,
-                    'webdriver_delay': None  # Extra delay in seconds before extracting text
-                }
-            }
-        }
-
-    def __init__(self, *arg, **kw):
-        super(model, self).__init__(*arg, **kw)
-        self.update(self.base_config)

changedetectionio/model/Watch.py

@@ -1,70 +0,0 @@
-import os
-
-import uuid as uuid_builder
-
-minimum_seconds_recheck_time = int(os.getenv('MINIMUM_SECONDS_RECHECK_TIME', 60))
-
-from changedetectionio.notification import (
-    default_notification_body,
-    default_notification_format,
-    default_notification_title,
-)
-
-
-class model(dict):
-    base_config = {
-            'url': None,
-            'tag': None,
-            'last_checked': 0,
-            'last_changed': 0,
-            'paused': False,
-            'last_viewed': 0,  # history key value of the last viewed via the [diff] link
-            'newest_history_key': 0,
-            'title': None,
-            'previous_md5': False,
-#           UUID not needed, should be generated only as a key
-#            'uuid':
-            'headers': {},  # Extra headers to send
-            'body': None,
-            'method': 'GET',
-            'history': {},  # Dict of timestamp and output stripped filename
-            'ignore_text': [],  # List of text to ignore when calculating the comparison checksum
-            # Custom notification content
-            'notification_urls': [],  # List of URLs to add to the notification Queue (Usually AppRise)
-            'notification_title': default_notification_title,
-            'notification_body': default_notification_body,
-            'notification_format': default_notification_format,
-            'css_filter': "",
-            'subtractive_selectors': [],
-            'trigger_text': [],  # List of text or regex to wait for until a change is detected
-            'fetch_backend': None,
-            'extract_title_as_title': False,
-            'proxy': None, # Preferred proxy connection
-            # Re #110, so then if this is set to None, we know to use the default value instead
-            # Requires setting to None on submit if it's the same as the default
-            # Should be all None by default, so we use the system default in this case.
-            'time_between_check': {'weeks': None, 'days': None, 'hours': None, 'minutes': None, 'seconds': None},
-            'webdriver_delay': None
-        }
-
-    def __init__(self, *arg, **kw):
-        self.update(self.base_config)
-        # goes at the end so we update the default object with the initialiser
-        super(model, self).__init__(*arg, **kw)
-
-
-    @property
-    def has_empty_checktime(self):
-        # using all() + dictionary comprehension
-        # Check if all values are 0 in dictionary
-        res = all(x == None or x == False or x==0 for x in self.get('time_between_check', {}).values())
-        return res
-
-    def threshold_seconds(self):
-        seconds = 0
-        mtable = {'seconds': 1, 'minutes': 60, 'hours': 3600, 'days': 86400, 'weeks': 86400 * 7}
-        for m, n in mtable.items():
-            x = self.get('time_between_check', {}).get(m, None)
-            if x:
-                seconds += x * n
-        return seconds

changedetectionio/notification.py

@@ -26,6 +26,13 @@ default_notification_title = 'ChangeDetection.io Notification - {watch_url}'
 
 def process_notification(n_object, datastore):
 
+    apobj = apprise.Apprise(debug=True)
+
+    for url in n_object['notification_urls']:
+        url = url.strip()
+        print (">> Process Notification: AppRise notifying {}".format(url))
+        apobj.add(url)
+
     # Get the notification body from datastore
     n_body = n_object.get('notification_body', default_notification_body)
     n_title = n_object.get('notification_title', default_notification_title)
@@ -47,55 +54,19 @@ def process_notification(n_object, datastore):
     # https://github.com/caronc/apprise/wiki/Development_LogCapture
     # Anything higher than or equal to WARNING (which covers things like Connection errors)
     # raise it as an exception
-    apobjs=[]
-    for url in n_object['notification_urls']:
 
-        apobj = apprise.Apprise(debug=True)
-        url = url.strip()
-        if len(url):
-            print(">> Process Notification: AppRise notifying {}".format(url))
-            with apprise.LogCapture(level=apprise.logging.DEBUG) as logs:
-                # Re 323 - Limit discord length to their 2000 char limit total or it wont send.
-                # Because different notifications may require different pre-processing, run each sequentially :(
-                # 2000 bytes minus -
-                #     200 bytes for the overhead of the _entire_ json payload, 200 bytes for {tts, wait, content} etc headers
-                #     Length of URL - Incase they specify a longer custom avatar_url
+    with apprise.LogCapture(level=apprise.logging.DEBUG) as logs:
+        apobj.notify(
+        body=n_body,
+        title=n_title,
+        body_format=n_format)
 
-                # So if no avatar_url is specified, add one so it can be correctly calculated into the total payload
-                k = '?' if not '?' in url else '&'
-                if not 'avatar_url' in url:
-                    url += k + 'avatar_url=https://raw.githubusercontent.com/dgtlmoon/changedetection.io/master/changedetectionio/static/images/avatar-256x256.png'
+        # Returns empty string if nothing found, multi-line string otherwise
+        log_value = logs.getvalue()
+        if log_value and 'WARNING' in log_value or 'ERROR' in log_value:
+            raise Exception(log_value)
 
-                if url.startswith('tgram://'):
-                    # real limit is 4096, but minus some for extra metadata
-                    payload_max_size = 3600
-                    body_limit = max(0, payload_max_size - len(n_title))
-                    n_title = n_title[0:payload_max_size]
-                    n_body = n_body[0:body_limit]
 
-                elif url.startswith('discord://'):
-                    # real limit is 2000, but minus some for extra metadata
-                    payload_max_size = 1700
-                    body_limit = max(0, payload_max_size - len(n_title))
-                    n_title = n_title[0:payload_max_size]
-                    n_body = n_body[0:body_limit]
-
-                apobj.add(url)
-
-                apobj.notify(
-                    title=n_title,
-                    body=n_body,
-                    body_format=n_format)
-
-                apobj.clear()
-
-                # Incase it needs to exist in memory for a while after to process(?)
-                apobjs.append(apobj)
-
-                # Returns empty string if nothing found, multi-line string otherwise
-                log_value = logs.getvalue()
-                if log_value and 'WARNING' in log_value or 'ERROR' in log_value:
-                    raise Exception(log_value)
 
 # Notification title + body content parameters get created here.
 def create_notification_parameters(n_object, datastore):

changedetectionio/static/images/copy.svg

@@ -1,40 +0,0 @@
-<?xml version="1.0" encoding="UTF-8" standalone="no"?>
-<svg
-   version="1.1"
-   id="Layer_1"
-   x="0px"
-   y="0px"
-   viewBox="0 0 115.77 122.88"
-   style="enable-background:new 0 0 115.77 122.88"
-   xml:space="preserve"
-   sodipodi:docname="copy.svg"
-   inkscape:version="1.1.1 (1:1.1+202109281949+c3084ef5ed)"
-   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
-   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
-   xmlns="http://www.w3.org/2000/svg"
-   xmlns:svg="http://www.w3.org/2000/svg"><defs
-     id="defs11" /><sodipodi:namedview
-     id="namedview9"
-     pagecolor="#ffffff"
-     bordercolor="#666666"
-     borderopacity="1.0"
-     inkscape:pageshadow="2"
-     inkscape:pageopacity="0.0"
-     inkscape:pagecheckerboard="0"
-     showgrid="false"
-     inkscape:zoom="5.5501303"
-     inkscape:cx="57.83648"
-     inkscape:cy="61.439999"
-     inkscape:window-width="1920"
-     inkscape:window-height="1056"
-     inkscape:window-x="1920"
-     inkscape:window-y="0"
-     inkscape:window-maximized="1"
-     inkscape:current-layer="g6" /><style
-     type="text/css"
-     id="style2">.st0{fill-rule:evenodd;clip-rule:evenodd;}</style><g
-     id="g6"><path
-       class="st0"
-       d="M89.62,13.96v7.73h12.19h0.01v0.02c3.85,0.01,7.34,1.57,9.86,4.1c2.5,2.51,4.06,5.98,4.07,9.82h0.02v0.02 v73.27v0.01h-0.02c-0.01,3.84-1.57,7.33-4.1,9.86c-2.51,2.5-5.98,4.06-9.82,4.07v0.02h-0.02h-61.7H40.1v-0.02 c-3.84-0.01-7.34-1.57-9.86-4.1c-2.5-2.51-4.06-5.98-4.07-9.82h-0.02v-0.02V92.51H13.96h-0.01v-0.02c-3.84-0.01-7.34-1.57-9.86-4.1 c-2.5-2.51-4.06-5.98-4.07-9.82H0v-0.02V13.96v-0.01h0.02c0.01-3.85,1.58-7.34,4.1-9.86c2.51-2.5,5.98-4.06,9.82-4.07V0h0.02h61.7 h0.01v0.02c3.85,0.01,7.34,1.57,9.86,4.1c2.5,2.51,4.06,5.98,4.07,9.82h0.02V13.96L89.62,13.96z M79.04,21.69v-7.73v-0.02h0.02 c0-0.91-0.39-1.75-1.01-2.37c-0.61-0.61-1.46-1-2.37-1v0.02h-0.01h-61.7h-0.02v-0.02c-0.91,0-1.75,0.39-2.37,1.01 c-0.61,0.61-1,1.46-1,2.37h0.02v0.01v64.59v0.02h-0.02c0,0.91,0.39,1.75,1.01,2.37c0.61,0.61,1.46,1,2.37,1v-0.02h0.01h12.19V35.65 v-0.01h0.02c0.01-3.85,1.58-7.34,4.1-9.86c2.51-2.5,5.98-4.06,9.82-4.07v-0.02h0.02H79.04L79.04,21.69z M105.18,108.92V35.65v-0.02 h0.02c0-0.91-0.39-1.75-1.01-2.37c-0.61-0.61-1.46-1-2.37-1v0.02h-0.01h-61.7h-0.02v-0.02c-0.91,0-1.75,0.39-2.37,1.01 c-0.61,0.61-1,1.46-1,2.37h0.02v0.01v73.27v0.02h-0.02c0,0.91,0.39,1.75,1.01,2.37c0.61,0.61,1.46,1,2.37,1v-0.02h0.01h61.7h0.02 v0.02c0.91,0,1.75-0.39,2.37-1.01c0.61-0.61,1-1.46,1-2.37h-0.02V108.92L105.18,108.92z"
-       id="path4"
-       style="fill:#ffffff;fill-opacity:1" /></g></svg>

changedetectionio/static/images/notviewed.svg

@@ -0,0 +1 @@
+<svg id="svg2" xmlns="http://www.w3.org/2000/svg" width="9.5" height="15" viewBox="0 0 9.5 15"><path id="path3740" d="M2.2,0A2.41,2.41,0,0,0,0,1.5V2.8H2.2V0Z" transform="translate(0 0)" style="fill:#0078e7"/><path id="rect3728" d="M.3,1.7l-.2,1H2.2V.2A2.76,2.76,0,0,0,.3,1.7Z" transform="translate(0 0)" style="fill:#0078e7"/><path id="path3655" d="M9.5,2.6h0L2.3,0,2,.2,9.2,2.8v.1" transform="translate(0 0)" style="fill:#0078e7"/><path id="path3645" d="M9.2,2.8V13.3l.2-.3V2.5" transform="translate(0 0)" style="fill:#0078e7"/><path id="rect3517" d="M2,.2,9.2,2.8V13.4L2,10.8Z" transform="translate(0 0)" style="fill:#0078e7"/><path id="path3657" d="M2.1.2,9.2,2.8" transform="translate(0 0)" style="fill:#0078e7;stroke:#0078e7;stroke-miterlimit:4.660399913787842;stroke-width:0.10000000149011612px"/><path id="path3684" d="M.5,9.6.6,2.4.4,2.3S1.2,1.1,2,1L8.8,3.4v.1" transform="translate(0 0)" style="fill:#fff;fill-rule:evenodd"/><path id="path3679" d="M8.9,3.4,8.7,13,7.3,14.3.5,11.9V9.5" transform="translate(0 0)" style="fill:#fff;fill-rule:evenodd"/><path id="path3669" d="M7.5,4.2h0L.3,1.6,0,1.9,7.2,4.5v.1" transform="translate(0 0)" style="fill:#0078e7"/><path id="path3671" d="M7.2,4.5V15l.2-.3V4.2" transform="translate(0 0)" style="fill:#0078e7"/><path id="path3673" d="M0,1.9,7.2,4.5V15L0,12.4Z" transform="translate(0 0)" style="fill:#0078e7"/><path id="path3675" d="M.1,1.9,7.2,4.5" transform="translate(0 0)" style="fill:#0078e7;stroke:#0078e7;stroke-miterlimit:4.660399913787842;stroke-width:0.10000000149011612px"/></svg>

changedetectionio/static/images/pause.svg

@@ -1,84 +1 @@
-<?xml version="1.0" encoding="UTF-8" standalone="no"?>
-<svg
-   xmlns:dc="http://purl.org/dc/elements/1.1/"
-   xmlns:cc="http://creativecommons.org/ns#"
-   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
-   xmlns:svg="http://www.w3.org/2000/svg"
-   xmlns="http://www.w3.org/2000/svg"
-   version="1.1"
-   id="Capa_1"
-   x="0px"
-   y="0px"
-   viewBox="0 0 15 14.998326"
-   xml:space="preserve"
-   width="15"
-   height="14.998326"><metadata
-   id="metadata39"><rdf:RDF><cc:Work
-       rdf:about=""><dc:format>image/svg+xml</dc:format><dc:type
-         rdf:resource="http://purl.org/dc/dcmitype/StillImage" /><dc:title></dc:title></cc:Work></rdf:RDF></metadata><defs
-   id="defs37" />
-<path
-   id="path2"
-   style="fill:#1b98f8;fill-opacity:1;stroke-width:0.0292893"
-   d="M 7.4975161,6.5052867e-4 C 4.549072,-0.04028702 1.7055675,1.8548221 0.58868606,4.5801341 -0.57739762,7.2574642 0.02596981,10.583326 2.069916,12.671949 4.0364753,14.788409 7.2763651,15.56067 9.989207,14.57284 12.801145,13.617602 14.87442,10.855325 14.985833,7.8845744 15.172496,4.9966544 13.49856,2.1100704 10.911002,0.8209349 9.8598067,0.28073592 8.6791261,-0.00114855 7.4975161,6.5052867e-4 Z M 6.5602569,10.251923 c -0.00509,0.507593 -0.5693885,0.488472 -0.9352002,0.468629 -0.3399386,0.0018 -0.8402048,0.07132 -0.9297965,-0.374189 -0.015842,-1.8973128 -0.015872,-3.7979649 0,-5.6952784 0.1334405,-0.5224315 0.7416869,-0.3424086 1.1377562,-0.374189 0.3969969,-0.084515 0.8245634,0.1963256 0.7272405,0.6382917 0,1.7789118 0,3.5578239 0,5.3367357 z m 3.7490371,0 c -0.0051,0.507593 -0.5693888,0.488472 -0.9352005,0.468629 -0.3399386,0.0018 -0.8402048,0.07132 -0.9297965,-0.374189 -0.015842,-1.8973128 -0.015872,-3.7979649 0,-5.6952784 0.1334405,-0.5224315 0.7416869,-0.3424086 1.1377562,-0.374189 0.3969969,-0.084515 0.8245638,0.1963256 0.7272408,0.6382917 0,1.7789118 0,3.5578239 0,5.3367357 z" />
-<g
-   id="g4"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g6"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g8"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g10"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g12"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g14"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g16"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g18"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g20"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g22"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g24"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g26"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g28"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g30"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g32"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-</svg>
+<svg id="Capa_1" data-name="Capa 1" xmlns="http://www.w3.org/2000/svg" width="15.03" height="15.03" viewBox="0 0 15.03 15.03"><path id="path2" d="M7.5,0A7.56,7.56,0,0,0,.6,4.6a7.37,7.37,0,0,0,1.5,8.1A7.52,7.52,0,0,0,10,14.6,7.53,7.53,0,0,0,10.9.8,7.73,7.73,0,0,0,7.5,0ZM6.6,10.3c0,.5-.6.5-.9.5s-.8.1-.9-.4V4.7c.1-.5.7-.3,1.1-.4a.53.53,0,0,1,.7.6Zm3.7,0c0,.5-.6.5-.9.5s-.8.1-.9-.4V4.7c.1-.5.7-.3,1.1-.4a.53.53,0,0,1,.7.6Z" transform="translate(0.01 0)" style="fill:#0078e7"/></svg>

changedetectionio/static/images/play.svg

@@ -0,0 +1 @@
+<svg id="Capa_1" data-name="Capa 1" xmlns="http://www.w3.org/2000/svg" width="15" height="15" viewBox="0 0 15 15"><path id="path2" d="M7.5,0A7.62,7.62,0,0,0,0,7.5,7.55,7.55,0,0,0,7.5,15,7.55,7.55,0,0,0,15,7.5,7.6,7.6,0,0,0,10.9.8,9.42,9.42,0,0,0,7.5,0Z" transform="translate(0 0)" style="fill:#0078e7"/><polygon points="11.4 8 5.8 4.8 5.8 11.3 11.4 8" style="fill:#fff"/></svg>

changedetectionio/static/images/search.svg

@@ -0,0 +1 @@
+<svg id="Layer_1" data-name="Layer 1" xmlns="http://www.w3.org/2000/svg" width="32" height="32" viewBox="0 0 32 32"><circle cx="16" cy="16" r="16" style="fill:#0078e7"/><path d="M24,26.85l-4.93-5a8.53,8.53,0,0,1-4.71,1.41,8.63,8.63,0,1,1,7.32-4.12l5,5c.26.26,0,.9-.49,1.44l-.74.74C24.86,26.88,24.23,27.11,24,26.85Zm-3.9-12.23a5.75,5.75,0,1,0-5.74,5.79A5.76,5.76,0,0,0,20.07,14.62Z" style="fill:#fff"/></svg>

changedetectionio/static/images/sortable.svg

@@ -0,0 +1 @@
+<svg id="Layer_1" data-name="Layer 1" xmlns="http://www.w3.org/2000/svg" width="10.93" height="14.99" viewBox="0 0 10.93 14.99"><path d="M5.5,1,9.6,6.1H1.3Zm0,13L1.3,8.9H9.5Z" transform="translate(0.02 -0.01)" style="fill:#0078e7;stroke:#0078e7;stroke-miterlimit:10;stroke-width:1.25px"/></svg>

changedetectionio/static/images/spread.svg

@@ -1,46 +0,0 @@
-<?xml version="1.0" encoding="UTF-8" standalone="no"?>
-<svg
-   width="18"
-   height="19.92"
-   viewBox="0 0 18 19.92"
-   version="1.1"
-   id="svg6"
-   sodipodi:docname="spread.svg"
-   inkscape:version="1.1.1 (1:1.1+202109281949+c3084ef5ed)"
-   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
-   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
-   xmlns="http://www.w3.org/2000/svg"
-   xmlns:svg="http://www.w3.org/2000/svg">
-  <defs
-     id="defs10" />
-  <sodipodi:namedview
-     id="namedview8"
-     pagecolor="#ffffff"
-     bordercolor="#666666"
-     borderopacity="1.0"
-     inkscape:pageshadow="2"
-     inkscape:pageopacity="0.0"
-     inkscape:pagecheckerboard="0"
-     showgrid="false"
-     fit-margin-top="0"
-     fit-margin-left="0"
-     fit-margin-right="0"
-     fit-margin-bottom="0"
-     inkscape:zoom="28.416667"
-     inkscape:cx="9.0087975"
-     inkscape:cy="9.9941348"
-     inkscape:window-width="1920"
-     inkscape:window-height="1056"
-     inkscape:window-x="1920"
-     inkscape:window-y="0"
-     inkscape:window-maximized="1"
-     inkscape:current-layer="svg6" />
-  <path
-     d="M -3,-2 H 21 V 22 H -3 Z"
-     fill="none"
-     id="path2" />
-  <path
-     d="m 15,14.08 c -0.76,0 -1.44,0.3 -1.96,0.77 L 5.91,10.7 C 5.96,10.47 6,10.24 6,10 6,9.76 5.96,9.53 5.91,9.3 L 12.96,5.19 C 13.5,5.69 14.21,6 15,6 16.66,6 18,4.66 18,3 18,1.34 16.66,0 15,0 c -1.66,0 -3,1.34 -3,3 0,0.24 0.04,0.47 0.09,0.7 L 5.04,7.81 C 4.5,7.31 3.79,7 3,7 1.34,7 0,8.34 0,10 c 0,1.66 1.34,3 3,3 0.79,0 1.5,-0.31 2.04,-0.81 l 7.12,4.16 c -0.05,0.21 -0.08,0.43 -0.08,0.65 0,1.61 1.31,2.92 2.92,2.92 1.61,0 2.92,-1.31 2.92,-2.92 0,-1.61 -1.31,-2.92 -2.92,-2.92 z"
-     id="path4"
-     style="fill:#0078e7;fill-opacity:1" />
-</svg>

changedetectionio/static/images/viewed.svg

@@ -0,0 +1 @@
+<svg id="svg2" xmlns="http://www.w3.org/2000/svg" width="16.33" height="11.64" viewBox="0 0 16.33 11.64"><path id="path2416" d="M14.2,6.5l1.4,3.1-6.2.8s0,.2-.4.1a.65.65,0,0,1-.6-.6L1.8,11.1.6,4.2H.9v.1H.8L2,10.9,8.4,9.8V10h.4s0,.4.5.4l.1-.1,6-.8-1.2-3Z" transform="translate(-0.01 -0.04)" style="fill:#0078e7;stroke:#007ec0;fill-rule:evenodd"/><path id="path2400" d="M1,4.3H.9L2,10.9,8.4,9.8s-1.7-.9-6.1,1L1,4.3Z" transform="translate(-0.01 -0.04)" style="fill:#1187c3;fill-rule:evenodd"/><path id="path2402" d="M1,4.1V3.6h.1V3.2l.2-.3h.1V2.5L2.8,8.9l-.1.3v.2l-.1.1-.2.1-.1,1.1L1,4.1Z" transform="translate(-0.01 -0.04)" style="fill:#fff;stroke:#0078e7;stroke-width:0.5px;fill-rule:evenodd"/><path id="path2388" d="M2.3,10.8l.1-1.2.3-.3V9.2l.2-.3s5.5-.2,5.9.8l-.4.1s-1.7-.9-6.1,1Z" transform="translate(-0.01 -0.04)" style="fill:#fff;stroke:#0078e7;stroke-width:0.5px;fill-rule:evenodd"/><path id="path2394" d="M2.2,2.5H1.5L2.9,8.9s5.3-.2,5.9.8c0,0-.1-1.1-5.4-1.6L2.2,2.5Z" transform="translate(-0.01 -0.04)" style="fill:#fff;stroke:#0078e7;stroke-width:0.5px;fill-rule:evenodd"/><path id="path2398" d="M2,1.3,3.4,8s5,.5,5.4,1.7l-2-6.3c0-.1,0-1.1-4.8-2.1Z" transform="translate(-0.01 -0.04)" style="fill:#fff;stroke:#0078e7;stroke-width:0.5px;fill-rule:evenodd"/><path id="path2403" d="M6.9,3.3l5-2.9,2.5,5.9L8.8,9.7,6.9,3.3Z" transform="translate(-0.01 -0.04)" style="fill:#fff;stroke:#0078e7;stroke-width:0.5px;fill-rule:evenodd"/><path id="path2411" d="M8.5,10V9.8l.3-.1s.2.6.6.6v.1a.49.49,0,0,1-.5-.5l-.4.1Z" transform="translate(-0.01 -0.04)" style="fill:#1187c3;stroke:#007ec0;stroke-width:0.5px;fill-rule:evenodd"/><path id="path2415" d="M8.8,9.7s.2.6.6.5l6-.8-.3-.6h-.3c.1,0-5,.2-6,.9Z" transform="translate(-0.01 -0.04)" style="fill:#fff;stroke:#0078e7;stroke-width:0.5px;fill-rule:evenodd"/><path id="path2404" d="M15.1,8.9l-1-2.3L8.8,9.7v.1s.5-.6,6-.9V9Z" transform="translate(-0.01 -0.04)" style="fill:#fff;stroke:#0078e7;stroke-width:0.5px;fill-rule:evenodd"/></svg>

changedetectionio/static/js/global-settings.js

@@ -1,16 +0,0 @@
-$(document).ready(function() {
-    function toggle() {
-        if ($('input[name="application-fetch_backend"]:checked').val() != 'html_requests') {
-            $('#requests-override-options').hide();
-            $('#webdriver-override-options').show();
-        } else {
-            $('#requests-override-options').show();
-            $('#webdriver-override-options').hide();
-        }
-    }
-    $('input[name="application-fetch_backend"]').click(function (e) {
-        toggle();
-    });
-    toggle();
-
-});

changedetectionio/static/js/notifications.js

@@ -1,53 +0,0 @@
-$(document).ready(function() {
-
-  $('#add-email-helper').click(function (e) {
-    e.preventDefault();
-    email = prompt("Destination email");
-    if(email) {
-      var n = $(".notification-urls");
-      var p=email_notification_prefix;
-      $(n).val( $.trim( $(n).val() )+"\n"+email_notification_prefix+email );
-    }
-  });
-
-  $('#send-test-notification').click(function (e) {
-    e.preventDefault();
-
-    // this can be global
-    var csrftoken = $('input[name=csrf_token]').val();
-    $.ajaxSetup({
-        beforeSend: function(xhr, settings) {
-            if (!/^(GET|HEAD|OPTIONS|TRACE)$/i.test(settings.type) && !this.crossDomain) {
-                xhr.setRequestHeader("X-CSRFToken", csrftoken)
-            }
-        }
-    })
-
-    data = {
-        window_url : window.location.href,
-        notification_urls : $('.notification-urls').val(),
-        notification_title : $('.notification-title').val(),
-        notification_body : $('.notification-body').val(),
-        notification_format : $('.notification-format').val(),
-    }
-    for (key in data) {
-      if (!data[key].length) {
-        alert(key+" is empty, cannot send test.")
-        return;
-      }
-    }
-
-    $.ajax({
-      type: "POST",
-      url: notification_base_url,
-      data : data
-    }).done(function(data){
-      console.log(data);
-      alert('Sent');
-    }).fail(function(data){
-      console.log(data);
-      alert('Error: '+data.responseJSON.error);
-    })
-  });
-});
-

changedetectionio/static/js/settings.js

@@ -0,0 +1,13 @@
+window.addEventListener("load", (event) => {
+  // just an example for now
+  function toggleVisible(elem) {
+    // theres better ways todo this
+    var x = document.getElementById(elem);
+    if (x.style.display === "block") {
+      x.style.display = "none";
+    } else {
+      x.style.display = "block";
+    }
+  }
+});
+

changedetectionio/static/js/tabs.js

@@ -1,15 +1,13 @@
 // Rewrite this is a plugin.. is all this JS really 'worth it?'
 
-
-if(!window.location.hash) {
-  var tab=document.querySelectorAll("#default-tab a");
-  tab[0].click();
-}
-
+// display correct label and messages for minutes or seconds 
+document.addEventListener("DOMContentLoaded", function(event) {
+	use_seconds_change();
+});
 window.addEventListener('hashchange', function() {
   var tabs = document.getElementsByClassName('active');
   while (tabs[0]) {
-    tabs[0].classList.remove('active')
+    tabs[0].classList.remove('active');
   }
   set_active_tab();
 }, false);
@@ -26,6 +24,7 @@ if (!has_errors.length) {
   focus_error_tab();
 }
 
+
 function set_active_tab() {
   var tab=document.querySelectorAll("a[href='"+location.hash+"']");
   if (tab.length) {
@@ -41,7 +40,7 @@ function focus_error_tab() {
     var tabs = document.querySelectorAll('.tabs li a'),i;
     for (i = 0; i < tabs.length; ++i) {
       var tab_name=tabs[i].hash.replace('#','');
-      var pane_errors=document.querySelectorAll('#'+tab_name+' .error')
+      var pane_errors=document.querySelectorAll('#'+tab_name+' .error');
       if (pane_errors.length) {
         document.location.hash = '#'+tab_name;
         return true;
@@ -49,7 +48,3 @@ function focus_error_tab() {
     }
     return false;
 }
-
-
-
-

changedetectionio/static/js/tbltools.js

@@ -0,0 +1,406 @@
+// table tools
+
+// must be a var for keyChar and keyCode use
+var CONSTANT_ESCAPE_KEY = 27;
+var CONSTANT_S_KEY = 83;
+var CONSTANT_s_KEY = 115;
+
+// globals
+var loading;
+var sort_column; // new window or tab is always last_changed
+var sort_order;  // new window or tab is always descending
+
+// restore scroll position on submit/reload 
+document.addEventListener("DOMContentLoaded", function(event) {
+	load_functions();
+	var scrollpos = sessionStorage.getItem('scrollpos');
+	if (scrollpos) window.scrollTo(0, scrollpos);
+});
+
+// mobile scroll position retention 
+if (/Android|webOS|iPhone|iPad|iPod|BlackBerry|IEMobile|Opera Mini/i.test(navigator.userAgent)) {
+	document.addEventListener("visibilitychange", function() {
+		storeScrollAndSearch();
+	});
+} else {
+	// non-mobile scroll position retention 
+	window.onbeforeunload = function(e) {
+		storeScrollAndSearch();
+	};
+}
+function storeScrollAndSearch() {
+	sessionStorage.setItem('scrollpos', window.pageYOffset);
+	sessionStorage.setItem('searchtxt', document.getElementById("txtInput").value);
+}
+
+// mobile positioning of checkbox-controls grid popup
+document.addEventListener("touchstart", touchStartHandler, false);
+var touchXY = {};
+function touchStartHandler(event) {
+	var touches = event.changedTouches;
+	touchXY = {
+		clientX : touches[0].clientX,
+		clientY : touches[0].clientY
+	};
+}
+
+// (ctl)-alt-s search hotkey
+document.onkeyup = function(e) {
+	var e = e || window.event; // for IE to cover IEs window event-object
+	if (e.altKey && (e.which == CONSTANT_S_KEY || e.which == CONSTANT_s_KEY)) {
+		document.getElementById("txtInput").focus();
+		return false;
+	}
+}
+
+// new window or tab loading
+function load_functions() {
+	// loading
+	loading = true;
+	// retain checked items
+	checkChange();
+	// retrieve saved sorting
+	getSort();
+	// sort if not default
+	sortTable(sort_column);
+	// search
+	if (isSessionStorageSupported()) {
+		// retrieve search
+		if (sessionStorage.getItem("searchtxt") != null) {
+			document.getElementById("txtInput").value = sessionStorage.getItem("searchtxt");
+			tblSearch(this);
+		}
+	}
+}
+
+// sorting
+function sortTable(n) {
+	var table, rows, switching, i, x, y, shouldSwitch, dir, switchcount = 0,
+		sortimgs, sortableimgs;
+	table = document.getElementById("watch-table");
+	switching = true;
+	//Set the sorting direction, either default 9, 1 or saved
+	if (loading) {
+		getSort();
+		dir = (sort_order == 0) ? "asc" : "desc";
+		loading = false;
+	} else {
+		dir = "asc";
+	}
+	/*Make a loop that will continue until
+	no switching has been done:*/
+	while (switching) {
+		//start by saying: no switching is done:
+		switching = false;
+		rows = table.rows;
+		/*Loop through all table rows (except the
+		first, which contains table headers):*/
+		for (i = 1; i < (rows.length - 1); i++) {
+			//start by saying there should be no switching:
+			shouldSwitch = false;
+			/*Get the two elements you want to compare,
+			one from current row and one from the next:*/
+			x = rows[i].getElementsByTagName("TD")[n];
+			y = rows[i + 1].getElementsByTagName("TD")[n];
+			x = x.innerHTML.toLowerCase();
+			y = y.innerHTML.toLowerCase();
+			if (!isNaN(x)) { // handle numeric columns
+				x = parseFloat(x);
+				y = parseFloat(y);
+			}
+			if (n == 1) { // handle play/pause column
+				x = rows[i].getElementsByTagName("TD")[n].getElementsByTagName("img")[0].src;
+				y = rows[i + 1].getElementsByTagName("TD")[n].getElementsByTagName("img")[0].src;
+			}
+			/*check if the two rows should switch place,
+			based on the direction, asc or desc:*/
+			if (dir == "asc") {
+				if (x > y) {
+					//if so, mark as a switch and break the loop:
+					shouldSwitch = true;
+					break;
+				}
+			} else if (dir == "desc") {
+				if (x < y) {
+					//if so, mark as a switch and break the loop:
+					shouldSwitch = true;
+					break;
+				}
+			}
+		}
+		if (shouldSwitch) {
+			/*If a switch has been marked, make the switch
+			and mark that a switch has been done:*/
+			rows[i].parentNode.insertBefore(rows[i + 1], rows[i]);
+			switching = true;
+			//Each time a switch is done, increase this count by 1:
+			switchcount++;
+		} else {
+			/*If no switching has been done AND the direction is "asc",
+			set the direction to "desc" and run the while loop again.*/
+			if (switchcount == 0 && dir == "asc") {
+				dir = "desc";
+				switching = true;
+			}
+		}
+	}
+	// hide all asc/desc sort arrows
+	sortimgs = document.querySelectorAll('[id^="sort-"]');
+	for (i = 0; i < sortimgs.length; i++) {
+		sortimgs[i].style.display = "none";
+	}
+	// show current asc/desc sort arrow and set sort_order var
+	if (dir == "asc") {
+		document.getElementById("sort-" + n + "a").style.display = "";
+	} else {
+		document.getElementById("sort-" + n + "d").style.display = "";
+	}
+	// show all sortable indicators
+	sortableimgs = document.querySelectorAll('[id^="sortable-"]');
+	for (i = 0; i < sortableimgs.length; i++) {
+		sortableimgs[i].style.display = "";
+	}
+	// hide sortable indicator from current column
+	document.getElementById("sortable-" + n).style.display = "none";
+	// save sorting
+	sessionStorage.setItem("sort_column", n);
+	sessionStorage.setItem("sort_order", (dir == "asc") ? 0 : 1);
+	// restripe rows
+	restripe();
+}
+
+// check/uncheck all checkboxes
+function checkAll(e) {
+	var elemID = event.srcElement.id;
+	if (!elemID) return;
+	var elem = document.getElementById(elemID);
+	var rect = elem.getBoundingClientRect();
+	var offsetLeft = document.documentElement.scrollLeft + rect.left;
+	var offsetTop;
+	if (/Android|webOS|iPhone|iPad|iPod|BlackBerry|IEMobile|Opera Mini/i.test(navigator.userAgent)) {
+		offsetTop = touchXY.clientY; // + rect.top;
+	}
+	else {
+		offsetTop = document.documentElement.scrollTop + rect.top;
+	}
+	var i;
+	var checkboxes = document.getElementsByName('check');
+	var checkboxFunctions = document.getElementById('checkbox-functions');
+	if (e.checked) {
+		for (i = 0; i < checkboxes.length; i++) {
+			checkboxes[i].checked = true;
+		}
+		checkboxFunctions.style.display = "";
+		checkboxFunctions.style.left = offsetLeft + 30 + "px";
+		checkboxFunctions.style.top = offsetTop + "px";
+	} else {
+		for (i = 0; i < checkboxes.length; i++) {
+			checkboxes[i].checked = false;
+		}
+		checkboxFunctions.style.display = "none";
+	}
+}
+
+// show/hide checkbox controls grid popup and check/uncheck checkall checkbox if all other checkboxes are checked/unchecked
+function checkChange(e) {
+	var elemID = event.srcElement.id;
+	if (!elemID) return;
+	var elem = document.getElementById(elemID);
+	var rect = elem.getBoundingClientRect();
+	var offsetLeft = document.documentElement.scrollLeft + rect.left;
+	var offsetTop;
+	if (/Android|webOS|iPhone|iPad|iPod|BlackBerry|IEMobile|Opera Mini/i.test(navigator.userAgent)) {
+		offsetTop = touchXY.clientY; // + rect.top;
+	}
+	else {
+		offsetTop = document.documentElement.scrollTop + rect.top;
+	}
+	var i;
+	var totalCheckbox = document.querySelectorAll('input[name="check"]').length;
+	var totalChecked = document.querySelectorAll('input[name="check"]:checked').length;
+	var checkboxFunctions = document.getElementById('checkbox-functions');
+	if(totalCheckbox == totalChecked) {
+		document.getElementsByName("showhide")[0].checked=true;
+	}
+	else {
+		document.getElementsByName("showhide")[0].checked=false;
+	}
+	if (totalChecked > 0) {
+		checkboxFunctions.style.display = "";
+		checkboxFunctions.style.left = offsetLeft + 30 + "px";
+		if ( offsetTop > ( window.innerHeight - checkboxFunctions.offsetHeight) ) {
+			checkboxFunctions.style.top = (window.innerHeight - checkboxFunctions.offsetHeight) + "px";
+		}
+		else {
+			checkboxFunctions.style.top = offsetTop + "px";
+		}
+	} else {
+		checkboxFunctions.style.display = "none";
+	}
+}
+
+// search watches in Title column
+function tblSearch(evt) {
+	var code = evt.charCode || evt.keyCode;
+	if (code == CONSTANT_ESCAPE_KEY) {
+		document.getElementById("txtInput").value = '';
+	}
+	var input, filter, table, tr, td, i, txtValue;
+	input = document.getElementById("txtInput");
+	filter = input.value.toUpperCase();
+	table = document.getElementById("watch-table");
+	tr = table.getElementsByTagName("tr");
+	for (i = 1; i < tr.length; i++) { // skip header
+		td = tr[i].getElementsByTagName("td")[3]; // col 3 is the hidden title/url column
+		if (td) {
+			txtValue = td.textContent || td.innerText;
+			if (txtValue.toUpperCase().indexOf(filter) > -1) {
+				tr[i].style.display = "";
+			} else {
+				tr[i].style.display = "none";
+			}
+		}
+	}
+	// restripe rows
+	restripe();
+	if (code == CONSTANT_ESCAPE_KEY) {
+		document.getElementById("watch-table-wrapper").focus();
+	}
+}
+
+// restripe after searching or sorting
+function restripe() {
+	var i, visrows = [];
+	var table = document.getElementById("watch-table");
+	var rows = table.getElementsByTagName("tr");
+
+	for (i = 1; i < rows.length; i++) { // skip header
+		if (rows[i].style.display !== "none") {
+			visrows.push(rows[i]);
+		}
+	}
+	for (i = 0; i < visrows.length; i++) {
+		var row = visrows[i];
+		if (i % 2 == 0) {
+			row.classList.remove('pure-table-odd');
+			row.classList.add('pure-table-even');
+		} else {
+			row.classList.remove('pure-table-even');
+			row.classList.add('pure-table-odd');
+		}
+		var cells = row.getElementsByTagName("td");
+		for (var j = 0; j < cells.length; j++) {
+			if (i % 2 == 0) {
+				cells[j].style.background = "#f2f2f2";
+			} else {
+				cells[j].style.background = "#ffffff";
+			}
+		}
+		// uncomment to renumber rows ascending:    var cells = row.getElementsByTagName("td");
+		// uncomment to renumber rows ascending:    cells[0].innerText = i+1;
+	}
+}
+
+// get checked or all uuids
+function getChecked(items) {
+	var i, checkedArr, uuids = '';
+
+	if (items === undefined) {
+		checkedArr = document.querySelectorAll('input[name="check"]:checked');
+	} else {
+		checkedArr = document.querySelectorAll('input[name="check"]');
+	}
+	if (checkedArr.length > 0) {
+		let output = [];
+		for (i = 0; i < checkedArr.length; i++) {
+			output.push(checkedArr[i].parentNode.parentNode.getAttribute("id"));
+		}
+		for (i = 0; i < checkedArr.length; i++) {
+			if (i < checkedArr.length - 1) {
+				uuids += output[i] + ",";
+			} else {
+				uuids += output[i];
+			}
+		}
+	}
+	return uuids;
+}
+
+// process selected watches 
+function processChecked(func, tag) {
+	var uuids, result;
+
+	if (func == 'mark_all_notviewed') {
+		uuids = getChecked('all');
+	} else {
+		uuids = getChecked();
+	}
+	// confirm if deleting
+	if (func == 'delete_selected' && uuids.length > 0) {
+		result = confirm('Deletions cannot be undone.\n\nAre you sure you want to continue?');
+		if (result == false) {
+			return;
+		}
+	}
+	// href locations
+	var currenturl = window.location;
+	var posturl = location.protocol + '//' + location.host + '/api/process-selected';
+	// posting vars
+	const XHR = new XMLHttpRequest(),
+		FD = new FormData();
+	// fill form data
+	FD.append('func', func);
+	FD.append('tag', tag);
+	FD.append('uuids', uuids);
+	// success
+	XHR.addEventListener('load', function(event) {
+		window.location = currenturl;
+	});
+	// error
+	XHR.addEventListener(' error', function(event) {
+		alert('Error posting request.');
+	});
+	// set up request
+	XHR.open('POST', posturl);
+	// send
+	XHR.send(FD);
+}
+
+function clearSearch() {
+	document.getElementById("txtInput").value = '';
+	tblSearch(CONSTANT_ESCAPE_KEY);
+}
+
+function isSessionStorageSupported() {
+	var storage = window.sessionStorage;
+	try {
+		storage.setItem('test', 'test');
+		storage.removeItem('test');
+		return true;
+	} catch (e) {
+		return false;
+	}
+}
+
+function getSort() {
+	if (isSessionStorageSupported()) {
+		// retrieve sort settings if set
+		if (sessionStorage.getItem("sort_column") != null) {
+			sort_column = sessionStorage.getItem("sort_column");
+			sort_order = sessionStorage.getItem("sort_order");
+		} else {
+			sort_column = 7; // last changed
+			sort_order = 1; // desc
+			//alert("Your web browser does not support retaining sorting and page position.");
+		}
+	}
+}
+
+function closeGridDisplay() {
+	document.getElementsByName("showhide")[0].checked = false;
+	var checkboxes = document.getElementsByName('check');
+	for (i = 0; i < checkboxes.length; i++) {
+		checkboxes[i].checked = false;
+	}
+	document.getElementById("checkbox-functions").style.display = "none";
+}

changedetectionio/static/js/watch-overview.js

@@ -1,24 +0,0 @@
-$(function () {
-  // Remove unviewed status when normally clicked
-  $('.diff-link').click(function () {
-    $(this).closest('.unviewed').removeClass('unviewed');
-  });
-
-  $('.with-share-link > *').click(function () {
-      $("#copied-clipboard").remove();
-
-      var range = document.createRange();
-      var n=$("#share-link")[0];
-      range.selectNode(n);
-      window.getSelection().removeAllRanges();
-      window.getSelection().addRange(range);
-      document.execCommand("copy");
-      window.getSelection().removeAllRanges();
-
-      $('.with-share-link').append('<span style="font-size: 80%; color: #fff;" id="copied-clipboard">Copied to clipboard</span>');
-      $("#copied-clipboard").fadeOut(2500, function() {
-       $(this).remove();
-      });
-  });
-});
-

changedetectionio/static/js/watch-settings.js

@@ -1,16 +0,0 @@
-$(document).ready(function() {
-    function toggle() {
-        if ($('input[name="fetch_backend"]:checked').val() != 'html_requests') {
-            $('#requests-override-options').hide();
-            $('#webdriver-override-options').show();
-        } else {
-            $('#requests-override-options').show();
-            $('#webdriver-override-options').hide();
-        }
-    }
-    $('input[name="fetch_backend"]').click(function (e) {
-        toggle();
-    });
-    toggle();
-
-});

changedetectionio/static/styles/diff.css

@@ -1,8 +1,7 @@
 #diff-ui {
   background: #fff;
   padding: 2em;
-  margin-left: 1em;
-  margin-right: 1em;
+  margin: 1em;
   border-radius: 5px;
   font-size: 11px; }
   #diff-ui table {
@@ -71,8 +70,3 @@ td#diff-col div {
 /* ignored and triggered? make it obvious error */
 .ignored.triggered {
   background-color: #ff0000; }
-
-.tab-pane-inner#screenshot {
-  text-align: center; }
-  .tab-pane-inner#screenshot img {
-    max-width: 99%; }

changedetectionio/static/styles/diff.scss

@@ -2,8 +2,7 @@
 
     background: #fff;
     padding: 2em;
-    margin-left: 1em;
-    margin-right: 1em;
+    margin: 1em;
     border-radius: 5px;
     font-size: 11px;
 
@@ -86,11 +85,4 @@ td#diff-col div {
 /* ignored and triggered? make it obvious error */
 .ignored.triggered {
   background-color: #ff0000;
-}
-
-.tab-pane-inner#screenshot {
-  text-align: center;
-  img {
-    max-width: 99%;
-  }
 }

changedetectionio/static/styles/styles.css

@@ -31,24 +31,21 @@ a.github-link {
 
 section.content {
   padding-top: 5em;
-  padding-bottom: 1em;
+  padding-bottom: 5em;
   flex-direction: column;
   display: flex;
   align-items: center;
   justify-content: center; }
 
-code {
-  background: #eee; }
-
 /* table related */
 .watch-table {
-  width: 100%;
-  font-size: 80%; }
+  width: 100%; }
   .watch-table tr.unviewed {
     font-weight: bold; }
   .watch-table .error {
     color: #a00; }
   .watch-table td {
+    font-size: 80%;
     white-space: nowrap; }
   .watch-table td.title-col {
     word-break: break-all;
@@ -83,11 +80,11 @@ code {
 
 body:after {
   content: "";
-  background: linear-gradient(130deg, #5ad8f7, #2f50af 41.07%, #9150bf 84.05%); }
+  background: linear-gradient(130deg, #ff7a18, #af002d 41.07%, #319197 76.05%); }
 
 body:after, body:before {
   display: block;
-  height: 650px;
+  height: 600px;
   position: absolute;
   top: 0;
   left: 0;
@@ -99,6 +96,9 @@ body::after {
 
 body::before {
   content: "";
+  background-image: url(/static/images/gradient-border.png); }
+
+body:before {
   background-size: cover; }
 
 body:after, body:before {
@@ -180,19 +180,10 @@ body:after, body:before {
   .messages li.notice {
     background: rgba(255, 255, 255, 0.5); }
 
-.messages.with-share-link > *:hover {
-  cursor: pointer; }
-
 #notification-customisation {
-  border: 1px solid #ccc;
-  padding: 0.5rem;
-  border-radius: 5px; }
-
-#notification-error-log {
   border: 1px solid #ccc;
   padding: 1rem;
-  border-radius: 5px;
-  overflow-wrap: break-word; }
+  border-radius: 5px; }
 
 #token-table.pure-table td, #token-table.pure-table th {
   font-size: 80%; }
@@ -208,8 +199,7 @@ body:after, body:before {
   #new-watch-form .label {
     display: none; }
   #new-watch-form legend {
-    color: #fff;
-    font-weight: bold; }
+    color: #fff; }
 
 #diff-col {
   padding-left: 40px; }
@@ -253,7 +243,7 @@ footer {
 .sticky-tab {
   position: absolute;
   top: 60px;
-  font-size: 65%;
+  font-size: 8px;
   background: #fff;
   padding: 10px; }
   .sticky-tab#left-sticky {
@@ -288,11 +278,6 @@ footer {
     padding-bottom: 1em; }
     .pure-form .pure-control-group div, .pure-form .pure-group div, .pure-form .pure-controls div {
       margin: 0px; }
-    .pure-form .pure-control-group .checkbox > *, .pure-form .pure-group .checkbox > *, .pure-form .pure-controls .checkbox > * {
-      display: inline;
-      vertical-align: middle; }
-    .pure-form .pure-control-group .checkbox > label, .pure-form .pure-group .checkbox > label, .pure-form .pure-controls .checkbox > label {
-      padding-left: 5px; }
   .pure-form .error input {
     background-color: #ffebeb; }
   .pure-form ul.errors {
@@ -309,10 +294,10 @@ footer {
     font-weight: bold; }
   .pure-form textarea {
     width: 100%; }
-  .pure-form .inline-radio ul {
+  .pure-form ul#fetch_backend {
     margin: 0px;
     list-style: none; }
-    .pure-form .inline-radio ul li > * {
+    .pure-form ul#fetch_backend > li > * {
       display: inline-block; }
 
 @media only screen and (max-width: 760px), (min-device-width: 768px) and (max-device-width: 1024px) {
@@ -324,23 +309,14 @@ footer {
   #nav-menu {
     overflow-x: scroll; } }
 
-@media only screen and (max-width: 760px), (min-device-width: 768px) and (max-device-width: 800px) {
-  div.sticky-tab#hosted-sticky {
-    top: 60px;
-    left: 0px;
-    right: auto; }
-  section.content {
-    padding-top: 110px; }
-  div.tabs.collapsable ul li {
-    display: block;
-    border-radius: 0px; }
-  input[type='text'] {
-    width: 100%; }
-  /*
+/*
 Max width before this PARTICULAR table gets nasty
 This query will take effect for any screen smaller than 760px
 and also iPads specifically.
 */
+@media only screen and (max-width: 760px), (min-device-width: 768px) and (max-device-width: 1024px) {
+  input[type='text'] {
+    width: 100%; }
   .watch-table {
     /* Force table to not be like tables anymore */
     /* Force table to not be like tables anymore */
@@ -412,22 +388,14 @@ and also iPads specifically.
 .pure-form-stacked > div:first-child {
   display: block; }
 
-.login-form .inner {
-  background: #fff;
-  padding: 20px;
-  border-radius: 5px; }
-
-.tab-pane-inner {
-  padding: 0px; }
-  .tab-pane-inner:not(:target) {
-    display: none; }
-  .tab-pane-inner:target {
-    display: block; }
-
 .edit-form {
-  min-width: 70%;
-  /* so it cant overflow */
-  max-width: 95%; }
+  min-width: 70%; }
+  .edit-form .tab-pane-inner {
+    padding: 0px; }
+    .edit-form .tab-pane-inner:not(:target) {
+      display: none; }
+    .edit-form .tab-pane-inner:target {
+      display: block; }
   .edit-form .box-wrap {
     position: relative; }
   .edit-form .inner {
@@ -436,15 +404,8 @@ and also iPads specifically.
   .edit-form #actions {
     display: block;
     background: #fff; }
-  .edit-form .pure-form-message-inline {
-    padding-left: 0; }
 
 ul {
   padding-left: 1em;
   padding-top: 0px;
   margin-top: 4px; }
-
-.time-check-widget tr {
-  display: inline; }
-  .time-check-widget tr input[type="number"] {
-    width: 4em; }

changedetectionio/static/styles/styles.scss

@@ -35,21 +35,16 @@ a.github-link {
 
 section.content {
   padding-top: 5em;
-  padding-bottom: 1em;
+  padding-bottom: 5em;
   flex-direction: column;
   display: flex;
   align-items: center;
   justify-content: center;
 }
 
-code {
-  background: #eee;
-}
-
 /* table related */
 .watch-table {
   width: 100%;
-  font-size: 80%;
 
   tr.unviewed {
     font-weight: bold;
@@ -60,6 +55,7 @@ code {
   }
 
   td {
+    font-size: 80%;
     white-space: nowrap;
   }
 
@@ -111,12 +107,12 @@ code {
 
 body:after {
   content: "";
-  background: linear-gradient(130deg, #5ad8f7, #2f50af 41.07%, #9150bf 84.05%);
+  background: linear-gradient(130deg, #ff7a18, #af002d 41.07%, #319197 76.05%)
 }
 
 body:after, body:before {
   display: block;
-  height: 650px;
+  height: 600px;
   position: absolute;
   top: 0;
   left: 0;
@@ -129,8 +125,11 @@ body::after {
 }
 
 body::before {
-  // background-image set in base.html so it works with reverse proxies etc
   content: "";
+  background-image: url(/static/images/gradient-border.png);
+}
+
+body:before {
   background-size: cover
 }
 
@@ -237,25 +236,14 @@ body:after, body:before {
             background: rgba(255, 255, 255, .5);
         }
     }
-    &.with-share-link {
-     > *:hover {
-       cursor:pointer;
-     }
-    }
 }
 
 #notification-customisation {
     border: 1px solid #ccc;
-    padding: 0.5rem;
+    padding: 1rem;
     border-radius: 5px;
 }
 
-#notification-error-log {
-    border: 1px solid #ccc;
-    padding: 1rem;
-    border-radius: 5px;
-    overflow-wrap: break-word;
-}
 
 #token-table {
     &.pure-table td, &.pure-table th {
@@ -277,7 +265,6 @@ body:after, body:before {
   }
   legend {
     color: #fff;
-    font-weight: bold;
   }
 }
 
@@ -333,7 +320,7 @@ footer {
 .sticky-tab {
   position: absolute;
   top: 60px;
-  font-size: 65%;
+  font-size: 8px;
   background: #fff;
   padding: 10px;
   &#left-sticky {
@@ -380,15 +367,6 @@ footer {
         div {
             margin: 0px;
         }
-        .checkbox {
-            > * {
-              display: inline;
-              vertical-align: middle;
-            }
-            > label {
-               padding-left: 5px;
-            }
-        }
     }
   /* The input fields with errors */
   .error {
@@ -418,16 +396,14 @@ footer {
   textarea {
     width: 100%;
   }
-  .inline-radio {
-      ul {
-        margin: 0px;
-        list-style: none;
-        li {
-            > * {
-                display: inline-block;
-            }
+  ul#fetch_backend {
+    margin: 0px;
+    list-style: none;
+    > li {
+        > * {
+            display: inline-block;
         }
-      }
+    }
   }
 }
 
@@ -444,35 +420,18 @@ footer {
   }
 }
 
-
-
-@media only screen and (max-width: 760px), (min-device-width: 768px) and (max-device-width: 800px) {
-
-  div.sticky-tab#hosted-sticky {
-    top: 60px;
-    left: 0px;
-    right: auto;
-  }
-
-  section.content {
-    padding-top: 110px;
-  }
-
-  // Make the tabs easier to hit, they will be all nice and horizontal
-  div.tabs.collapsable ul li {
-    display: block;
-    border-radius: 0px;
-  }
-
-  input[type='text'] {
-    width: 100%;
-  }
-
 /*
 Max width before this PARTICULAR table gets nasty
 This query will take effect for any screen smaller than 760px
 and also iPads specifically.
 */
+
+@media only screen and (max-width: 760px), (min-device-width: 768px) and (max-device-width: 1024px) {
+
+  input[type='text'] {
+    width: 100%;
+  }
+    
   .watch-table {
     /* Force table to not be like tables anymore */
     thead, tbody, th, td, tr {
@@ -586,16 +545,9 @@ $form-edge-padding: 20px;
     display: block;
   }
 }
-
-.login-form {
-  .inner {
-    background: #fff;;
-    padding: $form-edge-padding;
-    border-radius: 5px;
-  }
-}
-
-.tab-pane-inner {
+.edit-form {
+  min-width: 70%;
+  .tab-pane-inner {
     &:not(:target) {
         display: none;
     }
@@ -604,12 +556,7 @@ $form-edge-padding: 20px;
     }
     // doesnt need padding because theres another row of buttons/activity
     padding: 0px;
-}
-
-.edit-form {
-  min-width: 70%;
-  /* so it cant overflow */
-  max-width: 95%;
+  }
   .box-wrap {
     position: relative;
   }
@@ -621,23 +568,10 @@ $form-edge-padding: 20px;
     display: block;
     background: #fff;
   }
-
-  .pure-form-message-inline {
-    padding-left: 0;
-  }
 }
 
 ul {
     padding-left: 1em;
     padding-top: 0px;
     margin-top: 4px;
-}
-
-.time-check-widget {
-    tr {
-        display: inline;
-        input[type="number"] {
-            width: 4em;
-        }
-    }
 }

changedetectionio/store.py

@@ -1,30 +1,21 @@
-from flask import (
-    flash
-)
+from os import unlink, path, mkdir
 import json
-import logging
-import os
-import threading
-import time
 import uuid as uuid_builder
-from copy import deepcopy
-from os import mkdir, path, unlink
 from threading import Lock
-import re
-import requests
+from copy import deepcopy
 
-from . model import App, Watch
+import logging
+import time
+import threading
+import os
+
+from changedetectionio.notification import default_notification_format, default_notification_body, default_notification_title
 
 # Is there an existing library to ensure some data store (JSON etc) is in sync with CRUD methods?
 # Open a github issue if you know something :)
 # https://stackoverflow.com/questions/6190468/how-to-trigger-function-on-value-change
 class ChangeDetectionStore:
     lock = Lock()
-    # For general updates/writes that can wait a few seconds
-    needs_write = False
-
-    # For when we edit, we should write to disk
-    needs_write_urgent = False
 
     def __init__(self, datastore_path="/datastore", include_default_watches=True, version_tag="0.0.0"):
         # Should only be active for docker
@@ -32,14 +23,69 @@ class ChangeDetectionStore:
         self.needs_write = False
         self.datastore_path = datastore_path
         self.json_store_path = "{}/url-watches.json".format(self.datastore_path)
-        self.proxy_list = None
         self.stop_thread = False
 
-        self.__data = App.model()
+        self.__data = {
+            'note': "Hello! If you change this file manually, please be sure to restart your changedetection.io instance!",
+            'watching': {},
+            'settings': {
+                'headers': {
+                    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36',
+                    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
+                    'Accept-Encoding': 'gzip, deflate',  # No support for brolti in python requests yet.
+                    'Accept-Language': 'en-GB,en-US;q=0.9,en;'
+                },
+                'requests': {
+                    'timeout': 15,  # Default 15 seconds
+                    'minutes_between_check': 3 * 60,  # Default 3 hours
+                    'workers': 10  # Number of threads, lower is better for slow connections
+                },
+                'application': {
+                    'password': False,
+                    'base_url' : None,
+                    'extract_title_as_title': False,
+                    'fetch_backend': 'html_requests',
+                    'global_ignore_text': [], # List of text to ignore when calculating the comparison checksum
+                    'ignore_whitespace': False,
+                    'notification_urls': [], # Apprise URL list
+                    # Custom notification content
+                    'notification_title': default_notification_title,
+                    'notification_body': default_notification_body,
+                    'notification_format': default_notification_format,
+                }
+            }
+        }
 
         # Base definition for all watchers
-        # deepcopy part of #569 - not sure why its needed exactly
-        self.generic_definition = deepcopy(Watch.model())
+        self.generic_definition = {
+            'url': None,
+            'tag': None,
+            'last_checked': 0,
+            'last_changed': 0,
+            'paused': False,
+            'last_viewed': 0,  # history key value of the last viewed via the [diff] link
+            'newest_history_key': "",
+            'title': None,
+            # Re #110, so then if this is set to None, we know to use the default value instead
+            # Requires setting to None on submit if it's the same as the default
+            'minutes_between_check': None,
+            'previous_md5': "",
+            'uuid': str(uuid_builder.uuid4()),
+            'headers': {},  # Extra headers to send
+            'body': None,
+            'method': 'GET',
+            'history': {},  # Dict of timestamp and output stripped filename
+            'ignore_text': [], # List of text to ignore when calculating the comparison checksum
+            # Custom notification content
+            'notification_urls': [], # List of URLs to add to the notification Queue (Usually AppRise)
+            'notification_title': default_notification_title,
+            'notification_body': default_notification_body,
+            'notification_format': default_notification_format,
+            'css_filter': "",
+            'trigger_text': [],  # List of text or regex to wait for until a change is detected
+            'fetch_backend': None,
+            'extract_title_as_title': False
+        }
 
         if path.isfile('changedetectionio/source.txt'):
             with open('changedetectionio/source.txt') as f:
@@ -98,8 +144,8 @@ class ChangeDetectionStore:
             unlink(password_reset_lockfile)
 
         if not 'app_guid' in self.__data:
-            import os
             import sys
+            import os
             if "pytest" in sys.modules or "PYTEST_CURRENT_TEST" in os.environ:
                 self.__data['app_guid'] = "test-" + str(uuid_builder.uuid4())
             else:
@@ -111,17 +157,6 @@ class ChangeDetectionStore:
             secret = secrets.token_hex(16)
             self.__data['settings']['application']['rss_access_token'] = secret
 
-
-        # Proxy list support - available as a selection in settings when text file is imported
-        # CSV list
-        # "name, address", or just "name"
-        proxy_list_file = "{}/proxies.txt".format(self.datastore_path)
-        if path.isfile(proxy_list_file):
-            self.import_proxy_list(proxy_list_file)
-
-        # Bump the update version by running updates
-        self.run_updates()
-
         self.needs_write = True
 
         # Finally start the thread that will manage periodic data saves to JSON
@@ -147,16 +182,8 @@ class ChangeDetectionStore:
         self.data['watching'][uuid].update({'last_viewed': int(timestamp)})
         self.needs_write = True
 
-    def remove_password(self):
-        self.__data['settings']['application']['password'] = False
-        self.needs_write = True
-
     def update_watch(self, uuid, update_obj):
 
-        # It's possible that the watch could be deleted before update
-        if not self.__data['watching'].get(uuid):
-            return
-
         with self.lock:
 
             # In python 3.9 we have the |= dict operator, but that still will lose data on nested structures...
@@ -171,20 +198,10 @@ class ChangeDetectionStore:
 
         self.needs_write = True
 
-    @property
-    def threshold_seconds(self):
-        seconds = 0
-        mtable = {'seconds': 1, 'minutes': 60, 'hours': 3600, 'days': 86400, 'weeks': 86400 * 7}
-        minimum_seconds_recheck_time = int(os.getenv('MINIMUM_SECONDS_RECHECK_TIME', 60))
-        for m, n in mtable.items():
-            x = self.__data['settings']['requests']['time_between_check'].get(m)
-            if x:
-                seconds += x * n
-        return max(seconds, minimum_seconds_recheck_time)
-
     @property
     def data(self):
         has_unviewed = False
+        unviewed_count = 0
         for uuid, v in self.__data['watching'].items():
             self.__data['watching'][uuid]['newest_history_key'] = self.get_newest_history_key(uuid)
             if int(v['newest_history_key']) <= int(v['last_viewed']):
@@ -193,6 +210,7 @@ class ChangeDetectionStore:
             else:
                 self.__data['watching'][uuid]['viewed'] = False
                 has_unviewed = True
+                unviewed_count += 1
 
             # #106 - Be sure this is None on empty string, False, None, etc
             # Default var for fetch_backend
@@ -205,6 +223,7 @@ class ChangeDetectionStore:
           self.__data['settings']['application']['base_url'] = env_base_url.strip('" ')
 
         self.__data['has_unviewed'] = has_unviewed
+        self.__data['unviewed_count'] = unviewed_count
 
         return self.__data
 
@@ -244,7 +263,7 @@ class ChangeDetectionStore:
 
                 del self.data['watching'][uuid]
 
-            self.needs_write_urgent = True
+            self.needs_write = True
 
     # Clone a watch by UUID
     def clone(self, uuid):
@@ -268,64 +287,69 @@ class ChangeDetectionStore:
         return self.data['watching'][uuid].get(val)
 
     # Remove a watchs data but keep the entry (URL etc)
-    def scrub_watch(self, uuid):
-        import pathlib
+    def scrub_watch(self, uuid, limit_timestamp = False):
 
-        self.__data['watching'][uuid].update({'history': {}, 'last_checked': 0, 'last_changed': 0, 'newest_history_key': 0, 'previous_md5': False})
-        self.needs_write_urgent = True
+        import hashlib
+        del_timestamps = []
 
-        for item in pathlib.Path(self.datastore_path).rglob(uuid+"/*.txt"):
-            unlink(item)
+        changes_removed = 0
 
-    def add_watch(self, url, tag="", extras=None, write_to_disk_now=True):
+        for timestamp, path in self.data['watching'][uuid]['history'].items():
+            if not limit_timestamp or (limit_timestamp is not False and int(timestamp) > limit_timestamp):
+                self.unlink_history_file(path)
+                del_timestamps.append(timestamp)
+                changes_removed += 1
+
+        if not limit_timestamp:
+            self.data['watching'][uuid]['last_checked'] = 0
+            self.data['watching'][uuid]['last_changed'] = 0
+            self.data['watching'][uuid]['previous_md5'] = ""
+
+
+        for timestamp in del_timestamps:
+            del self.data['watching'][uuid]['history'][str(timestamp)]
+
+            # If there was a limitstamp, we need to reset some meta data about the entry
+            # This has to happen after we remove the others from the list
+            if limit_timestamp:
+                newest_key = self.get_newest_history_key(uuid)
+                if newest_key:
+                    self.data['watching'][uuid]['last_checked'] = int(newest_key)
+                    # @todo should be the original value if it was less than newest key
+                    self.data['watching'][uuid]['last_changed'] = int(newest_key)
+                    try:
+                        with open(self.data['watching'][uuid]['history'][str(newest_key)], "rb") as fp:
+                            content = fp.read()
+                        self.data['watching'][uuid]['previous_md5'] = hashlib.md5(content).hexdigest()
+                    except (FileNotFoundError, IOError):
+                        self.data['watching'][uuid]['previous_md5'] = ""
+                        pass
+
+        self.needs_write = True
+        return changes_removed
+
+    def add_watch(self, url, tag="", extras=None):
         if extras is None:
             extras = {}
-        # Incase these are copied across, assume it's a reference and deepcopy()
-        apply_extras = deepcopy(extras)
-
-        # Was it a share link? try to fetch the data
-        if (url.startswith("https://changedetection.io/share/")):
-            try:
-                r = requests.request(method="GET",
-                                     url=url,
-                                     # So we know to return the JSON instead of the human-friendly "help" page
-                                     headers={'App-Guid': self.__data['app_guid']})
-                res = r.json()
-
-                # List of permisable stuff we accept from the wild internet
-                for k in ['url', 'tag',
-                                   'paused', 'title',
-                                   'previous_md5', 'headers',
-                                   'body', 'method',
-                                   'ignore_text', 'css_filter',
-                                   'subtractive_selectors', 'trigger_text',
-                                   'extract_title_as_title']:
-                    if res.get(k):
-                        apply_extras[k] = res[k]
-
-            except Exception as e:
-                logging.error("Error fetching metadata for shared watch link", url, str(e))
-                flash("Error fetching metadata for {}".format(url), 'error')
-                return False
 
         with self.lock:
             # @todo use a common generic version of this
             new_uuid = str(uuid_builder.uuid4())
-            # #Re 569
-            # Not sure why deepcopy was needed here, sometimes new watches would appear to already have 'history' set
-            # I assumed this would instantiate a new object but somehow an existing dict was getting used
-            new_watch = deepcopy(Watch.model({
+            _blank = deepcopy(self.generic_definition)
+            _blank.update({
                 'url': url,
                 'tag': tag
-            }))
-
+            })
 
+            # Incase these are copied across, assume it's a reference and deepcopy()
+            apply_extras = deepcopy(extras)
             for k in ['uuid', 'history', 'last_checked', 'last_changed', 'newest_history_key', 'previous_md5', 'viewed']:
                 if k in apply_extras:
                     del apply_extras[k]
 
-            new_watch.update(apply_extras)
-            self.__data['watching'][new_uuid]=new_watch
+            _blank.update(apply_extras)
+
+            self.data['watching'][new_uuid] = _blank
 
         # Get the directory ready
         output_path = "{}/{}".format(self.datastore_path, new_uuid)
@@ -334,8 +358,7 @@ class ChangeDetectionStore:
         except FileExistsError:
             print(output_path, "already exists.")
 
-        if write_to_disk_now:
-            self.sync_to_json()
+        self.sync_to_json()
         return new_uuid
 
     # Save some text file to the appropriate path and bump the history
@@ -355,25 +378,9 @@ class ChangeDetectionStore:
 
         return fname
 
-    def get_screenshot(self, watch_uuid):
-        output_path = "{}/{}".format(self.datastore_path, watch_uuid)
-        fname = "{}/last-screenshot.png".format(output_path)
-        if path.isfile(fname):
-            return fname
-
-        return False
-
-    # Save as PNG, PNG is larger but better for doing visual diff in the future
-    def save_screenshot(self, watch_uuid, screenshot: bytes):
-        output_path = "{}/{}".format(self.datastore_path, watch_uuid)
-        fname = "{}/last-screenshot.png".format(output_path)
-        with open(fname, 'wb') as f:
-            f.write(screenshot)
-            f.close()
-
     def sync_to_json(self):
         logging.info("Saving JSON..")
-        print("Saving JSON..")
+
         try:
             data = deepcopy(self.__data)
         except RuntimeError as e:
@@ -390,12 +397,11 @@ class ChangeDetectionStore:
                 # system was out of memory, out of RAM etc
                 with open(self.json_store_path+".tmp", 'w') as json_file:
                     json.dump(data, json_file, indent=4)
-                os.replace(self.json_store_path+".tmp", self.json_store_path)
+                os.rename(self.json_store_path+".tmp", self.json_store_path)
             except Exception as e:
                 logging.error("Error writing JSON!! (Main JSON file save was skipped) : %s", str(e))
 
             self.needs_write = False
-            self.needs_write_urgent = False
 
     # Thread runner, this helps with thread/write issues when there are many operations that want to update the JSON
     # by just running periodically in one thread, according to python, dict updates are threadsafe.
@@ -406,14 +412,14 @@ class ChangeDetectionStore:
                 print("Shutting down datastore thread")
                 return
 
-            if self.needs_write or self.needs_write_urgent:
+            if self.needs_write:
                 self.sync_to_json()
 
             # Once per minute is enough, more and it can cause high CPU usage
             # better here is to use something like self.app.config.exit.wait(1), but we cant get to 'app' from here
-            for i in range(120):
-                time.sleep(0.5)
-                if self.stop_thread or self.needs_write_urgent:
+            for i in range(30):
+                time.sleep(2)
+                if self.stop_thread:
                     break
 
     # Go through the datastore path and remove any snapshots that are not mentioned in the index
@@ -427,71 +433,8 @@ class ChangeDetectionStore:
                 index.append(self.data['watching'][uuid]['history'][str(id)])
 
         import pathlib
-
         # Only in the sub-directories
-        for uuid in self.data['watching']:
-            for item in pathlib.Path(self.datastore_path).rglob(uuid+"/*.txt"):
-                if not str(item) in index:
-                    print ("Removing",item)
-                    unlink(item)
-
-    def import_proxy_list(self, filename):
-        import csv
-        with open(filename, newline='') as f:
-            reader = csv.reader(f, skipinitialspace=True)
-            # @todo This loop can could be improved
-            l = []
-            for row in reader:
-                if len(row):
-                    if len(row)>=2:
-                        l.append(tuple(row[:2]))
-                    else:
-                        l.append(tuple([row[0], row[0]]))
-            self.proxy_list = l if len(l) else None
-
-
-    # Run all updates
-    # IMPORTANT - Each update could be run even when they have a new install and the schema is correct
-    #             So therefor - each `update_n` should be very careful about checking if it needs to actually run
-    #             Probably we should bump the current update schema version with each tag release version?
-    def run_updates(self):
-        import inspect
-        import shutil
-
-        updates_available = []
-        for i, o in inspect.getmembers(self, predicate=inspect.ismethod):
-            m = re.search(r'update_(\d+)$', i)
-            if m:
-                updates_available.append(int(m.group(1)))
-        updates_available.sort()
-
-        for update_n in updates_available:
-            if update_n > self.__data['settings']['application']['schema_version']:
-                print ("Applying update_{}".format((update_n)))
-                # Wont exist on fresh installs
-                if os.path.exists(self.json_store_path):
-                    shutil.copyfile(self.json_store_path, self.datastore_path+"/url-watches-before-{}.json".format(update_n))
-
-                try:
-                    update_method = getattr(self, "update_{}".format(update_n))()
-                except Exception as e:
-                    print("Error while trying update_{}".format((update_n)))
-                    print(e)
-                    # Don't run any more updates
-                    return
-                else:
-                    # Bump the version, important
-                    self.__data['settings']['application']['schema_version'] = update_n
-
-    # Convert minutes to seconds on settings and each watch
-    def update_1(self):
-        if self.data['settings']['requests'].get('minutes_between_check'):
-            self.data['settings']['requests']['time_between_check']['minutes'] = self.data['settings']['requests']['minutes_between_check']
-            # Remove the default 'hours' that is set from the model
-            self.data['settings']['requests']['time_between_check']['hours'] = None
-
-        for uuid, watch in self.data['watching'].items():
-            if 'minutes_between_check' in watch:
-                # Only upgrade individual watch time if it was set
-                if watch.get('minutes_between_check', False):
-                    self.data['watching'][uuid]['time_between_check']['minutes'] = watch['minutes_between_check']
+        for item in pathlib.Path(self.datastore_path).rglob("*/*txt"):
+            if not str(item) in index:
+                print ("Removing",item)
+                unlink(item)

changedetectionio/templates/_common_fields.jinja

@@ -1,45 +1,42 @@
 
 {% from '_helpers.jinja' import render_field %}
 
-{% macro render_common_settings_form(form, current_base_url, emailprefix) %}
+{% macro render_common_settings_form(form, current_base_url) %}
+
                         <div class="pure-control-group">
                             {{ render_field(form.notification_urls, rows=5, placeholder="Examples:
     Gitter - gitter://token/room
     Office365 - o365://TenantID:AccountEmail/ClientID/ClientSecret/TargetEmail
     AWS SNS - sns://AccessKeyID/AccessSecretKey/RegionName/+PhoneNo
-    SMTPS - mailtos://user:pass@mail.domain.com?to=receivingAddress@example.com", class="notification-urls")
+    SMTPS - mailtos://user:pass@mail.domain.com?to=receivingAddress@example.com")
                             }}
                             <div class="pure-form-message-inline">
                               <ul>
                                 <li>Use <a target=_new href="https://github.com/caronc/apprise">AppRise URLs</a> for notification to just about any service! <i><a target=_new href="https://github.com/dgtlmoon/changedetection.io/wiki/Notification-configuration-notes">Please read the notification services wiki here for important configuration notes</a></i>.</li>
-                                <li><code>discord://</code> only supports a maximum <strong>2,000 characters</strong> of notification text, including the title.</li>
+                                <li><code>discord://</code> will silently fail if the total message length is more than 2000 chars.</li>
                                 <li><code>tgram://</code> bots cant send messages to other bots, so you should specify chat ID of non-bot user.</li>
-                                <li>Go here for <a href="{{url_for('notification_logs')}}">notification debug logs</a></li>
+                                <li>Go here for <a href="{{url_for('notification_logs')}}">Notification debug logs</a></li>
                               </ul>
                             </div>
-                            <br/>
-                            <a id="send-test-notification" class="pure-button button-secondary button-xsmall" style="font-size: 70%">Send test notification</a>
-{% if emailprefix %}
-                            <a id="add-email-helper" class="pure-button button-secondary button-xsmall" style="font-size: 70%">Add email</a>
-{% endif %}
                         </div>
-                        <div id="notification-customisation" class="pure-control-group">
+                        <div id="notification-customisation">
                             <div class="pure-control-group">
-                                {{ render_field(form.notification_title, class="m-d notification-title") }}
+                                {{ render_field(form.notification_title, class="m-d") }}
                                 <span class="pure-form-message-inline">Title for all notifications</span>
                             </div>
                             <div class="pure-control-group">
-                                {{ render_field(form.notification_body , rows=5, class="notification-body") }}
+                                {{ render_field(form.notification_body , rows=5) }}
                                 <span class="pure-form-message-inline">Body for all notifications</span>
                             </div>
                             <div class="pure-control-group">
-                                {{ render_field(form.notification_format , rows=5, class="notification-format") }}
+                                {{ render_field(form.notification_format , rows=5) }}
                                 <span class="pure-form-message-inline">Format for all notifications</span>
                             </div>
                             <div class="pure-controls">
                             <span class="pure-form-message-inline">
-                                These tokens can be used in the notification body and title to customise the notification text.
-
+                                These tokens can be used in the notification body and title to
+                                customise the notification text.
+                            </span>
                                 <table class="pure-table" id="token-table">
                                     <thead>
                                     <tr>
@@ -91,10 +88,13 @@
                                     </tr>
                                     </tbody>
                                 </table>
-                                <br/>
+                                <span class="pure-form-message-inline">
                                 URLs generated by changedetection.io (such as <code>{diff_url}</code>) require the <code>BASE_URL</code> environment variable set.<br/>
                                 Your <code>BASE_URL</code> var is currently "{{current_base_url}}"
                             </span>
                             </div>
                         </div>
+                        <div class="pure-control-group">
+                            {{ render_field(form.trigger_check) }}
+                        </div>
 {% endmacro %}

changedetectionio/templates/_helpers.jinja

@@ -1,30 +1,3 @@
-{% macro render_field(field) %}
-  <div {% if field.errors %} class="error" {% endif %}>{{ field(**kwargs)|safe }}
-  <div {% if field.errors %} class="error" {% endif %}>{{ field.label }}</div>
-
-  {% if field.errors %}
-    <ul class=errors>
-    {% for error in field.errors %}
-      <li>{{ error }}</li>
-    {% endfor %}
-    </ul>
-  {% endif %}
-  </div>
-{% endmacro %}
-
-{% macro render_checkbox_field(field) %}
-  <div class="checkbox {% if field.errors %} error {% endif %}">
-  {{ field(**kwargs)|safe }} {{ field.label }}
-  {% if field.errors %}
-    <ul class=errors>
-    {% for error in field.errors %}
-      <li>{{ error }}</li>
-    {% endfor %}
-    </ul>
-  {% endif %}
-  </div>
-{% endmacro %}
-
 {% macro render_field(field) %}
   <div {% if field.errors %} class="error" {% endif %}>{{ field.label }}</div>
   <div {% if field.errors %} class="error" {% endif %}>{{ field(**kwargs)|safe }}

changedetectionio/templates/base.html

@@ -5,7 +5,6 @@
     <meta name="viewport" content="width=device-width, initial-scale=1.0">
     <meta name="description" content="Self hosted website change detection.">
     <title>Change Detection{{extra_title}}</title>
-    <link rel="alternate" type="application/rss+xml" title="Changedetection.io » Feed{% if active_tag %}- {{active_tag}}{% endif %}" href="{{ url_for('rss', tag=active_tag , token=app_rss_token)}}" />
     <link rel="stylesheet" href="{{url_for('static_content', group='styles', filename='pure-min.css')}}">
     <link rel="stylesheet" href="{{url_for('static_content', group='styles', filename='styles.css')}}">
     {% if extra_stylesheets %}
@@ -13,15 +12,7 @@
         <link rel="stylesheet" href="{{ m }}?ver=1000">
         {% endfor %}
     {% endif %}
-    <style>
-    body::before {
-        background-image: url({{url_for('static_content', group='images', filename='gradient-border.png')}});
-    }
-    </style>
-    <script type="text/javascript" src="{{url_for('static_content', group='js', filename='jquery-3.6.0.min.js')}}"></script>
-
 </head>
-
 <body>
 
 <div class="header">
@@ -44,13 +35,16 @@
         {% if current_user.is_authenticated or not has_password %}
             {% if not current_diff_url %}
             <li class="pure-menu-item">
-                <a href="{{ url_for('settings_page')}}" class="pure-menu-link">SETTINGS</a>
+                <span class="search-box"><input type="text" id="txtInput" onkeyup="tblSearch(event)" onmouseup="clearSearch(event)" placeholder="Title..." /></span>
+            </li>
+            <li class="pure-menu-item">
+                <a href="{{ url_for('get_backup')}}" class="pure-menu-link">BACKUP</a>
             </li>
             <li class="pure-menu-item">
                 <a href="{{ url_for('import_page')}}" class="pure-menu-link">IMPORT</a>
             </li>
             <li class="pure-menu-item">
-                <a href="{{ url_for('get_backup')}}" class="pure-menu-link">BACKUP</a>
+                <a href="{{ url_for('settings_page')}}" class="pure-menu-link">SETTINGS</a>
             </li>
             {% else %}
             <li class="pure-menu-item">
@@ -94,13 +88,6 @@
         </ul>
       {% endif %}
     {% endwith %}
-
-    {% if session['share-link'] %}
-        <ul class="messages with-share-link">
-          <li class="message">Share this link: <span id="share-link">{{ session['share-link'] }}</span> <img style="height: 1em;display:inline-block;" src="{{url_for('static_content', group='images', filename='copy.svg')}}" /></li>
-        </ul>
-    {% endif %}
-
     {% block content %}
 
     {% endblock %}

changedetectionio/templates/diff.html

@@ -1,6 +1,7 @@
 {% extends 'base.html' %}
 
 {% block content %}
+
 <div id="settings">
     <h1>Differences</h1>
     <form class="pure-form " action="" method="GET">
@@ -34,45 +35,21 @@
 <div id="diff-jump">
     <a onclick="next_diff();">Jump</a>
 </div>
-
-<script type="text/javascript" src="{{url_for('static_content', group='js', filename='tabs.js')}}" defer></script>
-<div class="tabs">
-    <ul>
-        <li class="tab" id="default-tab"><a href="#text">Text</a></li>
-{% if screenshot %}
-        <li class="tab"><a href="#screenshot">Current screenshot</a></li>
-{% endif %}
-    </ul>
-</div>
-
 <div id="diff-ui">
-     <div class="tab-pane-inner" id="text">
-         <div class="tip">Pro-tip: Use <strong>show current snapshot</strong> tab to visualise what will be ignored.
-         </div>
-         <table>
-             <tbody>
-             <tr>
-                 <!-- just proof of concept copied straight from github.com/kpdecker/jsdiff -->
-                 <td id="a" style="display: none;">{{previous}}</td>
-                 <td id="b" style="display: none;">{{newest}}</td>
-                 <td id="diff-col">
-                     <span id="result"></span>
-                 </td>
-             </tr>
-             </tbody>
-         </table>
-         Diff algorithm from the amazing <a href="https://github.com/kpdecker/jsdiff">github.com/kpdecker/jsdiff</a>
-     </div>
-
-{% if screenshot %}
-     <div class="tab-pane-inner" id="screenshot">
-         <p>
-         <i>For now, only the most recent screenshot is saved and displayed.</i>
-             </p>
-
-        <img src="{{url_for('static_content', group='screenshot', filename=uuid)}}">
-     </div>
-{% endif %}
+    <div class="tip">Pro-tip: Use <strong>show current snapshot</strong> tab to visualise what will be ignored.</div>
+    <table>
+        <tbody>
+        <tr>
+            <!-- just proof of concept copied straight from github.com/kpdecker/jsdiff -->
+            <td id="a" style="display: none;">{{previous}}</td>
+            <td id="b" style="display: none;">{{newest}}</td>
+            <td id="diff-col">
+                <span id="result"></span>
+            </td>
+        </tr>
+        </tbody>
+    </table>
+    Diff algorithm from the amazing <a href="https://github.com/kpdecker/jsdiff">github.com/kpdecker/jsdiff</a>
 
 </div>
 

changedetectionio/templates/edit.html

@@ -1,20 +1,13 @@
 {% extends 'base.html' %}
 {% block content %}
-{% from '_helpers.jinja' import render_field, render_checkbox_field, render_button %}
+{% from '_helpers.jinja' import render_field %}
+{% from '_helpers.jinja' import render_button %}
 {% from '_common_fields.jinja' import render_common_settings_form %}
 <script type="text/javascript" src="{{url_for('static_content', group='js', filename='tabs.js')}}" defer></script>
-<script>
-    const notification_base_url="{{url_for('ajax_callback_send_notification_test')}}";
-{% if emailprefix %}
-    const email_notification_prefix=JSON.parse('{{ emailprefix|tojson }}');
-{% endif %}
-</script>
-<script type="text/javascript" src="{{url_for('static_content', group='js', filename='watch-settings.js')}}" defer></script>
-<script type="text/javascript" src="{{url_for('static_content', group='js', filename='notifications.js')}}" defer></script>
 
 <div class="edit-form monospaced-textarea">
 
-    <div class="tabs collapsable">
+    <div class="tabs">
         <ul>
             <li class="tab" id="default-tab"><a href="#general">General</a></li>
             <li class="tab"><a href="#request">Request</a></li>
@@ -26,7 +19,6 @@
     <div class="box-wrap inner">
         <form class="pure-form pure-form-stacked"
               action="{{ url_for('edit_page', uuid=uuid, next = request.args.get('next') ) }}" method="POST">
-             <input type="hidden" name="csrf_token" value="{{ csrf_token() }}"/>
 
             <div class="tab-pane-inner" id="general">
                 <fieldset>
@@ -42,8 +34,8 @@
                         <span class="pure-form-message-inline">Organisational tag/group name used in the main listing page</span>
                     </div>
                     <div class="pure-control-group">
-                        {{ render_field(form.time_between_check, class="time-check-widget") }}
-                        {% if has_empty_checktime %}
+                        {{ render_field(form.minutes_between_check) }}
+                        {% if using_default_minutes %}
                         <span class="pure-form-message-inline">Currently using the <a
                                 href="{{ url_for('settings_page', uuid=uuid) }}">default global settings</a>, change to another value if you want to be specific.</span>
                         {% else %}
@@ -52,74 +44,45 @@
                         {% endif %}
                     </div>
                     <div class="pure-control-group">
-                        {{ render_checkbox_field(form.extract_title_as_title) }}
+                        {{ render_field(form.extract_title_as_title) }}
                     </div>
                 </fieldset>
             </div>
 
             <div class="tab-pane-inner" id="request">
-                    <div class="pure-control-group inline-radio">
-                        {{ render_field(form.fetch_backend, class="fetch-backend") }}
+                    <div class="pure-control-group">
+                        {{ render_field(form.fetch_backend) }}
                         <span class="pure-form-message-inline">
                             <p>Use the <strong>Basic</strong> method (default) where your watched site doesn't need Javascript to render.</p>
                             <p>The <strong>Chrome/Javascript</strong> method requires a network connection to a running WebDriver+Chrome server, set by the ENV var 'WEBDRIVER_URL'. </p>
                         </span>
                     </div>
-                {% if form.proxy %}
-                    <div class="pure-control-group inline-radio">
-                        {{ render_field(form.proxy, class="fetch-backend-proxy") }}
-                        <span class="pure-form-message-inline">
-                        Choose a proxy for this watch
-                        </span>
-                    </div>
-                {% endif %}
-                <fieldset class="pure-group" id="webdriver-override-options">
-                    <div class="pure-form-message-inline">
-                        <strong>If you're having trouble waiting for the page to be fully rendered (text missing etc), try increasing the 'wait' time here.</strong>
-                        <br/>
-                        This will wait <i>n</i> seconds before extracting the text.
-                    </div>
-                    <div class="pure-control-group">
-                        {{ render_field(form.webdriver_delay) }}
-                    </div>
-                    {% if using_global_webdriver_wait %}
-                    <div class="pure-form-message-inline">
-                        <strong>Using the current global default settings</strong>
-                    </div>
-                    {% endif %}
-                </fieldset>
-                <fieldset class="pure-group" id="requests-override-options">
-                    <div class="pure-form-message-inline">
-                        <strong>Request override is currently only used by the <i>Basic fast Plaintext/HTTP Client</i> method.</strong>
-                    </div>
-                    <div class="pure-control-group">
-                        {{ render_field(form.method) }}
-                    </div>
-                    <div class="pure-control-group">
-{{ render_field(form.headers, rows=5, placeholder="Example
+
+                <fieldset class="pure-group">
+                                    <div class="pure-control-group">
+                    {{ render_field(form.method) }}
+                </div>
+                    <strong>Note: <i>Request Headers and Body settings are ONLY used by Basic fast Plaintext/HTTP Client fetch method.</i></strong>
+                    {{ render_field(form.headers, rows=5, placeholder="Example
 Cookie: foobar
 User-Agent: wonderbra 1.0") }}
-                    </div>
-                    <div class="pure-control-group">
-                                        {{ render_field(form.body, rows=5, placeholder="Example
+                </fieldset>
+                <div class="pure-control-group">
+                    {{ render_field(form.body, rows=5, placeholder="Example
 {
    \"name\":\"John\",
    \"age\":30,
    \"car\":null
 }") }}
-                    </div>
-                    <div>
-                        {{ render_checkbox_field(form.ignore_status_codes) }}
-                    </div>
-                </fieldset>
-                <br/>
+                </div>
+
             </div>
 
             <div class="tab-pane-inner" id="notifications">
                 <strong>Note: <i>These settings override the global settings for this watch.</i></strong>
                 <fieldset>
                     <div class="field-group">
-                        {{ render_common_settings_form(form, current_base_url, emailprefix) }}
+                        {{ render_common_settings_form(form, current_base_url) }}
                     </div>
                 </fieldset>
             </div>
@@ -144,27 +107,16 @@ User-Agent: wonderbra 1.0") }}
                         <span class="pure-form-message-inline">
                     <ul>
                         <li>CSS - Limit text to this CSS rule, only text matching this CSS rule is included.</li>
-                        <li>JSON - Limit text to this JSON rule, using <a href="https://pypi.org/project/jsonpath-ng/">JSONPath</a>, prefix with <code>"json:"</code>, use <code>json:$</code> to force re-formatting if required,  <a
+                        <li>JSON - Limit text to this JSON rule, using <a href="https://pypi.org/project/jsonpath-ng/">JSONPath</a>, prefix with <b>"json:"</b>, <a
                                 href="https://jsonpath.com/" target="new">test your JSONPath here</a></li>
-                        <li>XPath - Limit text to this XPath rule, simply start with a forward-slash, example  <code>//*[contains(@class, 'sametext')]</code> or <code>xpath://*[contains(@class, 'sametext')]</code>, <a
+                        <li>XPATH - Limit text to this XPath rule, simply start with a forward-slash, example  <b>//*[contains(@class, 'sametext')]</b>, <a
                                 href="http://xpather.com/" target="new">test your XPath here</a></li>
                     </ul>
                     Please be sure that you thoroughly understand how to write CSS or JSONPath, XPath selector rules before filing an issue on GitHub! <a
                                 href="https://github.com/dgtlmoon/changedetection.io/wiki/CSS-Selector-help">here for more CSS selector help</a>.<br/>
                 </span>
                     </div>
-                    <fieldset class="pure-group">
-                      {{ render_field(form.subtractive_selectors, rows=5, placeholder="header
-footer
-nav
-.stockticker") }}
-                      <span class="pure-form-message-inline">
-                        <ul>
-                          <li> Remove HTML element(s) by CSS selector before text conversion. </li>
-                          <li> Add multiple elements or CSS selectors per line to ignore multiple parts of the HTML. </li>
-                        </ul>
-                      </span>
-                    </fieldset>
+
                 </fieldset>
                 <fieldset class="pure-group">
                     {{ render_field(form.ignore_text, rows=5, placeholder="Some text to ignore in a line
@@ -173,7 +125,7 @@ nav
                     <span class="pure-form-message-inline">
                         <ul>
                             <li>Each line processed separately, any line matching will be ignored (removed before creating the checksum)</li>
-                            <li>Regular Expression support, wrap the line in forward slash <code>/regex/</code></li>
+                            <li>Regular Expression support, wrap the line in forward slash <b>/regex/</b></li>
                             <li>Changing this will affect the comparison checksum which may trigger an alert</li>
                             <li>Use the preview/show current tab to see ignores</li>
                         </ul>
@@ -189,8 +141,8 @@ nav
                     <ul>
                         <li>Text to wait for before triggering a change/notification, all text and regex are tested <i>case-insensitive</i>.</li>
                         <li>Trigger text is processed from the result-text that comes out of any CSS/JSON Filters for this watch</li>
-                        <li>Each line is processed separately (think of each line as "OR")</li>
-                        <li>Note: Wrap in forward slash / to use regex  example: <code>/foo\d/</code></li>
+                        <li>Each line is process separately (think of each line as "OR")</li>
+                        <li>Note: Wrap in forward slash / to use regex  example: <span style="font-family: monospace; background: #eee">/foo\d/</span></li>
                     </ul>
                         </span>
                     </div>

changedetectionio/templates/import.html

@@ -1,86 +1,29 @@
 {% extends 'base.html' %}
+
 {% block content %}
-<script type="text/javascript" src="{{url_for('static_content', group='js', filename='tabs.js')}}" defer></script>
-<div class="edit-form monospaced-textarea">
-
-    <div class="tabs collapsable">
-        <ul>
-            <li class="tab" id="default-tab"><a href="#url-list">URL List</a></li>
-            <li class="tab"><a href="#distill-io">Distill.io</a></li>
-        </ul>
-    </div>
-
-    <div class="box-wrap inner">
+<div class="edit-form">
+     <div class="inner">
         <form class="pure-form pure-form-aligned" action="{{url_for('import_page')}}" method="POST">
-            <input type="hidden" name="csrf_token" value="{{ csrf_token() }}"/>
-            <div class="tab-pane-inner" id="url-list">
-                <fieldset class="pure-group">
-                    <legend>
-                        Enter one URL per line, and optionally add tags for each URL after a space, delineated by comma
-                        (,):
-                        <br>
-                        <code>https://example.com tag1, tag2, last tag</code>
-                        <br>
-                        URLs which do not pass validation will stay in the textarea.
-                    </legend>
+            <fieldset class="pure-group">
+              <legend>
+                Enter one URL per line, and optionally add tags for each URL after a space, delineated by comma (,):
+                <br>
+                <code>https://example.com tag1, tag2, last tag</code>
+                <br>
+                URLs which do not pass validation will stay in the textarea.
+              </legend>
+              
 
-
-                    <textarea name="urls" class="pure-input-1-2" placeholder="https://"
-                              style="width: 100%;
+                <textarea name="urls" class="pure-input-1-2" placeholder="https://"
+                          style="width: 100%;
                                 font-family:monospace;
                                 white-space: pre;
                                 overflow-wrap: normal;
-                                overflow-x: scroll;" rows="25">{{ import_url_list_remaining }}</textarea>
-                </fieldset>
-
-
-            </div>
-
-            <div class="tab-pane-inner" id="distill-io">
-
-
-                <fieldset class="pure-group">
-                    <legend>
-                        Copy and Paste your Distill.io watch 'export' file, this should be a JSON file.</br>
-                        This is <i>experimental</i>, supported fields are <code>name</code>, <code>uri</code>, <code>tags</code>, <code>config:selections</code>, the rest (including <code>schedule</code>) are ignored.
-                        <br/>
-                        <p>
-                        How to export? <a href="https://distill.io/docs/web-monitor/how-export-and-import-monitors/">https://distill.io/docs/web-monitor/how-export-and-import-monitors/</a><br/>
-                        Be sure to set your default fetcher to Chrome if required.</br>
-                        </p>
-                    </legend>
-
-
-                    <textarea name="distill-io" class="pure-input-1-2" style="width: 100%;
-                                font-family:monospace;
-                                white-space: pre;
-                                overflow-wrap: normal;
-                                overflow-x: scroll;" placeholder="Example Distill.io JSON export file
-
-{
-    &quot;client&quot;: {
-        &quot;local&quot;: 1
-    },
-    &quot;data&quot;: [
-        {
-            &quot;name&quot;: &quot;Unraid | News&quot;,
-            &quot;uri&quot;: &quot;https://unraid.net/blog&quot;,
-            &quot;config&quot;: &quot;{\&quot;selections\&quot;:[{\&quot;frames\&quot;:[{\&quot;index\&quot;:0,\&quot;excludes\&quot;:[],\&quot;includes\&quot;:[{\&quot;type\&quot;:\&quot;xpath\&quot;,\&quot;expr\&quot;:\&quot;(//div[@id='App']/div[contains(@class,'flex')]/main[contains(@class,'relative')]/section[contains(@class,'relative')]/div[@class='container']/div[contains(@class,'flex')]/div[contains(@class,'w-full')])[1]\&quot;}]}],\&quot;dynamic\&quot;:true,\&quot;delay\&quot;:2}],\&quot;ignoreEmptyText\&quot;:true,\&quot;includeStyle\&quot;:false,\&quot;dataAttr\&quot;:\&quot;text\&quot;}&quot;,
-            &quot;tags&quot;: [],
-            &quot;content_type&quot;: 2,
-            &quot;state&quot;: 40,
-            &quot;schedule&quot;: &quot;{\&quot;type\&quot;:\&quot;INTERVAL\&quot;,\&quot;params\&quot;:{\&quot;interval\&quot;:4447}}&quot;,
-            &quot;ts&quot;: &quot;2022-03-27T15:51:15.667Z&quot;
-        }
-    ]
-}
-" rows="25">{{ original_distill_json }}</textarea>
-                </fieldset>
-            </div>
+                                overflow-x: scroll;" rows="25">{{ remaining }}</textarea>
+            </fieldset>
             <button type="submit" class="pure-button pure-input-1-2 pure-button-primary">Import</button>
         </form>
-
-    </div>
+     </div>
 </div>
 
 {% endblock %}

changedetectionio/templates/login.html

@@ -1,15 +1,15 @@
 {% extends 'base.html' %}
 
 {% block content %}
-<div class="login-form">
+<div class="edit-form">
+
  <div class="inner">
     <form class="pure-form pure-form-stacked" action="{{url_for('login')}}" method="POST">
-        <input type="hidden" name="csrf_token" value="{{ csrf_token() }}"/>
         <fieldset>
             <div class="pure-control-group">
                 <label for="password">Password</label>
                 <input type="password" id="password" required="" name="password" value=""
-                       size="15" autofocus />
+                       size="15"/>
                 <input type="hidden" id="email" name="email" value="defaultuser@changedetection.io" />
             </div>
             <div class="pure-control-group">

changedetectionio/templates/notification-log.html

@@ -5,7 +5,7 @@
      <div class="inner">
 
          <h4 style="margin-top: 0px;">The following issues were detected when sending notifications</h4>
-                <div id="notification-error-log">
+                <div id="notification-customisation">
                 <ul style="font-size: 80%; margin:0px; padding: 0 0 0 7px">
                 {% for log in logs|reverse %}
                     <li>{{log}}</li>

changedetectionio/templates/preview.html

@@ -6,40 +6,18 @@
     <h1>Current - {{watch.last_checked|format_timestamp_timeago}}</h1>
 </div>
 
-<script type="text/javascript" src="{{url_for('static_content', group='js', filename='tabs.js')}}" defer></script>
-<div class="tabs">
-    <ul>
-        <li class="tab" id="default-tab"><a href="#text">Text</a></li>
-{% if screenshot %}
-        <li class="tab"><a href="#screenshot">Current screenshot</a></li>
-{% endif %}
-    </ul>
-</div>
-
 <div id="diff-ui">
-    <div class="tab-pane-inner" id="text">
-        <span class="ignored">Grey lines are ignored</span> <span class="triggered">Blue lines are triggers</span>
-        <table>
-            <tbody>
-            <tr>
-                <td id="diff-col">
+    <span class="ignored">Grey lines are ignored</span> <span class="triggered">Blue lines are triggers</span>
+    <table>
+        <tbody>
+        <tr>
+            <td id="diff-col">
                     {% for row in content %}
                     <div class="{{row.classes}}">{{row.line}}</div>
                     {% endfor %}
-                </td>
-            </tr>
-            </tbody>
-        </table>
-    </div>
-
-{% if screenshot %}
-     <div class="tab-pane-inner" id="screenshot">
-         <p>
-         <i>For now, only the most recent screenshot is saved and displayed.</i>
-             </p>
-
-        <img src="{{url_for('static_content', group='screenshot', filename=uuid)}}">
-     </div>
-{% endif %}
+            </td>
+        </tr>
+        </tbody>
+    </table>
 </div>
 {% endblock %}

changedetectionio/templates/scrub.html

@@ -4,10 +4,9 @@
 <div class="edit-form">
     <div class="box-wrap inner">
     <form class="pure-form pure-form-stacked" action="{{url_for('scrub_page')}}" method="POST">
-        <input type="hidden" name="csrf_token" value="{{ csrf_token() }}"/>
         <fieldset>
             <div class="pure-control-group">
-                This will remove ALL version snapshots/data, but keep your list of URLs. <br/>
+                This will remove all version snapshots/data, but keep your list of URLs. <br/>
                 You may like to use the <strong>BACKUP</strong> link first.<br/>
             </div>
             <br/>
@@ -17,6 +16,12 @@
                 <span class="pure-form-message-inline">Type in the word <strong>scrub</strong> to confirm that you understand!</span>
             </div>
             <br/>
+            <div class="pure-control-group">
+                <label for="confirmtext">Optional: Limit deletion of snapshots to snapshots <i>newer</i> than date/time</label>
+                <input type="datetime-local" id="limit_date" name="limit_date"  />
+                <span class="pure-form-message-inline">dd/mm/yyyy hh:mm (24 hour format)</span>
+            </div>
+            <br/>
             <div class="pure-control-group">
                 <button type="submit" class="pure-button pure-button-primary">Scrub!</button>
             </div>

changedetectionio/templates/settings.html

@@ -1,20 +1,14 @@
 {% extends 'base.html' %}
 
 {% block content %}
-{% from '_helpers.jinja' import render_field, render_checkbox_field, render_button %}
+{% from '_helpers.jinja' import render_field %}
 {% from '_common_fields.jinja' import render_common_settings_form %}
-<script>
-    const notification_base_url="{{url_for('ajax_callback_send_notification_test')}}";
-{% if emailprefix %}
-    const email_notification_prefix=JSON.parse('{{emailprefix|tojson}}');
-{% endif %}
-</script>
-<script type="text/javascript" src="{{url_for('static_content', group='js', filename='tabs.js')}}" defer></script>
-<script type="text/javascript" src="{{url_for('static_content', group='js', filename='notifications.js')}}" defer></script>
 
-<script type="text/javascript" src="{{url_for('static_content', group='js', filename='global-settings.js')}}" defer></script>
+<script type="text/javascript" src="{{url_for('static_content', group='js', filename='settings.js')}}" defer></script>
+<script type="text/javascript" src="{{url_for('static_content', group='js', filename='tabs.js')}}" defer></script>
+
 <div class="edit-form">
-    <div class="tabs collapsable">
+    <div class="tabs">
         <ul>
             <li class="tab" id="default-tab"><a href="#general">General</a></li>
             <li class="tab"><a href="#notifications">Notifications</a></li>
@@ -24,19 +18,19 @@
     </div>
     <div class="box-wrap inner">
         <form class="pure-form pure-form-stacked settings" action="{{url_for('settings_page')}}" method="POST">
-            <input type="hidden" name="csrf_token" value="{{ csrf_token() }}"/>
             <div class="tab-pane-inner" id="general">
                 <fieldset>
                     <div class="pure-control-group">
-                        {{ render_field(form.requests.form.time_between_check, class="time-check-widget") }}
+                        {{ render_field(form.minutes_between_check) }}
                         <span class="pure-form-message-inline">Default time for all watches, when the watch does not have a specific time setting.</span>
                     </div>
                     <div class="pure-control-group">
                         {% if not hide_remove_pass %}
                             {% if current_user.is_authenticated %}
-                                {{ render_button(form.application.form.removepassword_button) }}
+                            <a href="{{url_for('settings_page', removepassword='yes')}}"
+                               class="pure-button pure-button-primary">Remove password</a>
                             {% else %}
-                            {{ render_field(form.application.form.password) }}
+                            {{ render_field(form.password) }}
                             <span class="pure-form-message-inline">Password protection for your changedetection.io application.</span>
                             {% endif %}
                         {% else %}
@@ -44,7 +38,7 @@
                         {% endif %}
                     </div>
                     <div class="pure-control-group">
-                        {{ render_field(form.application.form.base_url, placeholder="http://yoursite.com:5000/",
+                        {{ render_field(form.base_url, placeholder="http://yoursite.com:5000/",
                         class="m-d") }}
                         <span class="pure-form-message-inline">
                             Base URL used for the {base_url} token in notifications and RSS links.<br/>Default value is the ENV var 'BASE_URL' (Currently "{{current_base_url}}"),
@@ -53,88 +47,45 @@
                     </div>
 
                     <div class="pure-control-group">
-                        {{ render_checkbox_field(form.application.form.extract_title_as_title) }}
+                        {{ render_field(form.extract_title_as_title) }}
                         <span class="pure-form-message-inline">Note: This will automatically apply to all existing watches.</span>
                     </div>
-
-                    <div class="pure-control-group">
-                        {{ render_checkbox_field(form.application.form.real_browser_save_screenshot) }}
-                        <span class="pure-form-message-inline">When using a Chrome browser, a screenshot from the last check will be available on the Diff page</span>
-                    </div>
-
-                    <div class="pure-control-group">
-                        {{ render_checkbox_field(form.application.form.empty_pages_are_a_change) }}
-                        <span class="pure-form-message-inline">When a page contains HTML, but no renderable text appears (empty page), is this considered a change?</span>
-                    </div>
-                {% if form.requests.proxy %}
-                    <div class="pure-control-group inline-radio">
-                        {{ render_field(form.requests.form.proxy, class="fetch-backend-proxy") }}
-                        <span class="pure-form-message-inline">
-                        Choose a default proxy for all watches
-                        </span>
-                    </div>
-                {% endif %}
                 </fieldset>
             </div>
 
             <div class="tab-pane-inner" id="notifications">
                 <fieldset>
                     <div class="field-group">
-                        {{ render_common_settings_form(form.application.form, current_base_url, emailprefix) }}
+                        {{ render_common_settings_form(form, current_base_url) }}
                     </div>
                 </fieldset>
+                <a href="{{url_for('notification_logs')}}">Notification debug logs</a>
+
             </div>
 
             <div class="tab-pane-inner" id="fetching">
-                <div class="pure-control-group inline-radio">
-                    {{ render_field(form.application.form.fetch_backend, class="fetch-backend") }}
+                <div class="pure-control-group">
+                    {{ render_field(form.fetch_backend) }}
                     <span class="pure-form-message-inline">
                         <p>Use the <strong>Basic</strong> method (default) where your watched sites don't need Javascript to render.</p>
                         <p>The <strong>Chrome/Javascript</strong> method requires a network connection to a running WebDriver+Chrome server, set by the ENV var 'WEBDRIVER_URL'. </p>
                     </span>
                 </div>
-                <fieldset class="pure-group" id="webdriver-override-options">
-                    <div class="pure-form-message-inline">
-                        <strong>If you're having trouble waiting for the page to be fully rendered (text missing etc), try increasing the 'wait' time here.</strong>
-                        <br/>
-                        This will wait <i>n</i> seconds before extracting the text.
-                    </div>
-                    <div class="pure-control-group">
-                        {{ render_field(form.application.form.webdriver_delay) }}
-                    </div>
-                </fieldset>
             </div>
 
 
             <div class="tab-pane-inner" id="filters">
 
                     <fieldset class="pure-group">
-                    {{ render_checkbox_field(form.application.form.ignore_whitespace) }}
+                    {{ render_field(form.ignore_whitespace) }}
                     <span class="pure-form-message-inline">Ignore whitespace, tabs and new-lines/line-feeds when considering if a change was detected.<br/>
-                    <i>Note:</i> Changing this will change the status of your existing watches, possibly trigger alerts etc.
-                    </span>
-                    </fieldset>
-                <fieldset class="pure-group">
-                    {{ render_checkbox_field(form.application.form.render_anchor_tag_content) }}
-                    <span class="pure-form-message-inline">Render anchor tag content, default disabled, when enabled renders links as <code>(link text)[https://somesite.com]</code>
-                        <br/>
-                    <i>Note:</i> Changing this could affect the content of your existing watches, possibly trigger alerts etc.
+                    <i>Note:</i> Changing this will change the status of your existing watches, possibily trigger alerts etc.
                     </span>
                     </fieldset>
+
+
                     <fieldset class="pure-group">
-                      {{ render_field(form.application.form.global_subtractive_selectors, rows=5, placeholder="header
-footer
-nav
-.stockticker") }}
-                      <span class="pure-form-message-inline">
-                        <ul>
-                          <li> Remove HTML element(s) by CSS selector before text conversion. </li>
-                          <li> Add multiple elements or CSS selectors per line to ignore multiple parts of the HTML. </li>
-                        </ul>
-                      </span>
-                    </fieldset>
-                    <fieldset class="pure-group">
-                    {{ render_field(form.application.form.global_ignore_text, rows=5, placeholder="Some text to ignore in a line
+                    {{ render_field(form.global_ignore_text, rows=5, placeholder="Some text to ignore in a line
 /some.regex\d{2}/ for case-INsensitive regex
                     ") }}
                     <span class="pure-form-message-inline">Note: This is applied globally in addition to the per-watch rules.</span><br/>
@@ -142,7 +93,7 @@ nav
                         <ul>
                             <li>Note: This is applied globally in addition to the per-watch rules.</li>
                             <li>Each line processed separately, any line matching will be ignored (removed before creating the checksum)</li>
-                            <li>Regular Expression support, wrap the line in forward slash <code>/regex/</code></li>
+                            <li>Regular Expression support, wrap the line in forward slash <b>/regex/</b></li>
                             <li>Changing this will affect the comparison checksum which may trigger an alert</li>
                             <li>Use the preview/show current tab to see ignores</li>
                         </ul>
@@ -152,9 +103,11 @@ nav
 
             <div id="actions">
                 <div class="pure-control-group">
-                    {{ render_button(form.save_button) }}
-                    <a href="{{url_for('index')}}" class="pure-button button-small button-cancel">Back</a>
-                    <a href="{{url_for('scrub_page')}}" class="pure-button button-small button-cancel">Delete History Snapshot Data</a>
+                    <button type="submit" class="pure-button pure-button-primary">Save</button>
+                                           <a href="{{url_for('index')}}" class="pure-button button-small button-cancel">Back</a>
+                        <a href="{{url_for('scrub_page')}}" class="pure-button button-small button-cancel">Delete
+                            History
+                            Snapshot Data</a>
                 </div>
             </div>
         </form>

changedetectionio/templates/watch-overview.html

@@ -1,108 +1,147 @@
 {% extends 'base.html' %}
 {% block content %}
 {% from '_helpers.jinja' import render_simple_field %}
-<script type="text/javascript" src="{{url_for('static_content', group='js', filename='jquery-3.6.0.min.js')}}"></script>
-<script type="text/javascript" src="{{url_for('static_content', group='js', filename='watch-overview.js')}}" defer></script>
+
+<script src="{{url_for('static_content', group='js', filename='tbltools.js')}}"></script>
+
 <div class="box">
+	<form class="pure-form" action="{{ url_for('api_watch_add') }}" method="POST" id="new-watch-form">
+		<fieldset>
+			<legend>Add a new change detection watch</legend>
+				{{ render_simple_field(form.url, placeholder="https://...", required=true) }}
+				{{ render_simple_field(form.tag, value=active_tag if active_tag else '', placeholder="tag") }}
+			<div id="watch-actions"><button type="submit" class="pure-button pure-button-primary" name="action" value="watch">Watch</button>&nbsp;
+			<span id="add-paused"><label><input type="checkbox" name="add-paused">&nbsp;Add Paused</label></span></div>
+	   </fieldset>
+		<!-- add extra stuff, like do a http POST and send headers -->
+		<!-- user/pass r = requests.get('https://api.github.com/user', auth=('user', 'pass')) -->
+	</form>
+    <div id="watch-table-wrapper" tabindex="-1">
+		<div id="categories">
+			<a href="{{url_for('index')}}" class="pure-button button-tag {{'active' if not active_tag }}">All</a>
+			{% for tag in tags %}
+				{% if tag != "" %}
+					<a href="{{url_for('index', tag=tag) }}" class="pure-button button-tag {{'active' if active_tag == tag }}">{{ tag }}</a>
+				{% endif %}
+			{% endfor %}
+		</div>
+			
+		<div id="controls-top">
+			<div id="checkbox-functions" style="display: none;">
+				<ul id="post-list-buttons-top-grid">
+					<li id="grid-item-1">
+						<a href="javascript:processChecked('recheck_selected', '{{ active_tag }}');" class="pure-button button-tag " title="Recheck Selected{%if active_tag%} in &quot;{{active_tag}}&quot;{%endif%}">Recheck&nbsp;</a>
+					</li>
+					<li id="grid-item-2">
+						<a href="javascript:processChecked('mark_selected_notviewed', '{{ active_tag }}');" class="pure-button button-tag " title="Mark Selected as Unviewed{%if active_tag%} in &quot;{{active_tag}}&quot;{%endif%}">Unviewed</a>
+					</li>
+					<li id="grid-item-3">
+						<a href="javascript:processChecked('mark_selected_viewed', '{{ active_tag }}');" class="pure-button button-tag " title="Mark Selected as Viewed{%if active_tag%} in &quot;{{active_tag}}&quot;{%endif%}">Viewed&nbsp;&nbsp;</a>
+					</li>
+					<li id="grid-item-4">
+						<a href="javascript:processChecked('delete_selected', '{{ active_tag }}');" class="pure-button button-tag danger " title="Delete Selected{%if active_tag%} in &quot;{{active_tag}}&quot;{%endif%}">Delete&nbsp;&nbsp;</a>
+					</li>
+					<li id="grid-item-5">
+					   <a href="javascript:closeGridDisplay();" class="pure-button button-tag ">Cancel&nbsp;&nbsp;</a>
+					</li>
+				</ul>
+			</div>
+		</div>
 
-    <form class="pure-form" action="{{ url_for('api_watch_add') }}" method="POST" id="new-watch-form">
-        <input type="hidden" name="csrf_token" value="{{ csrf_token() }}"/>
-        <fieldset>
-            <legend>Add a new change detection watch</legend>
-                {{ render_simple_field(form.url, placeholder="https://...", required=true) }}
-                {{ render_simple_field(form.tag, value=active_tag if active_tag else '', placeholder="watch group") }}
-            <button type="submit" class="pure-button pure-button-primary">Watch</button>
-        </fieldset>
-        <span style="color:#eee; font-size: 80%;"><img style="height: 1em;display:inline-block;" src="{{url_for('static_content', group='images', filename='spread.svg')}}" /> Tip: You can also add 'shared' watches. <a href="https://github.com/dgtlmoon/changedetection.io/wiki/Sharing-a-Watch">More info</a></a></span>
-    </form>
-    <div>
-        <a href="{{url_for('index')}}" class="pure-button button-tag {{'active' if not active_tag }}">All</a>
-        {% for tag in tags %}
-            {% if tag != "" %}
-                <a href="{{url_for('index', tag=tag) }}" class="pure-button button-tag {{'active' if active_tag == tag }}">{{ tag }}</a>
-            {% endif %}
-        {% endfor %}
-    </div>
+		<div>
+			<table class="pure-table pure-table-striped watch-table" id="watch-table">
+				<thead>
+				<tr id="header">
+					<th class="inline chkbox-header"><input id="chk-all" type="checkbox" name="showhide" onchange="checkAll(this)" title="Check/Uncheck All">&nbsp;&nbsp;#</th>
+					<th class="pause-resume-header" onclick="sortTable(1)">
+						<span class="clickable"
+																				 title="Sort by Pause/Resume"><a
+							href="{{url_for('index', pause='pause-all', tag=active_tag)}}"><img
+							src="{{url_for('static_content', group='images', filename='pause.svg')}}" alt="Pause"
+							title="Pause All {%if active_tag%}in &quot;{{active_tag}}&quot; {%endif%}"/></a>&nbsp;<a
+							href="{{url_for('index', pause='resume-all', tag=active_tag)}}"><img
+							src="{{url_for('static_content', group='images', filename='play.svg')}}" alt="Resume"
+							title="Resume All {%if active_tag%}in &quot;{{active_tag}}&quot; {%endif%}"/></a>&nbsp;&nbsp;<span
+							id="sortable-1"><img
+							src="{{url_for('static_content', group='images', filename='sortable.svg')}}"
+							alt="sort"/></span><span class="sortarrow"><span id="sort-1a"
+																			 style="display:none;">&#9650;</span><span
+							id="sort-1d" style="display:none;">&#9660;</span></span></span></th>
+					<th onclick="sortTable(3)"><span class="clickable" title="Sort by Title">Title&nbsp;&nbsp;<span id="sortable-3"><img src="{{url_for('static_content', group='images', filename='sortable.svg')}}" alt="sort" /></span><span class="sortarrow"><span id="sort-3a" style="display:none;">&#9650;</span><span id="sort-3d" style="display:none;">&#9660;</span></span></span></th>
+					<th onclick="sortTable(5)"><span class="clickable" title="Sort by Last Checked">Checked&nbsp;&nbsp;<span id="sortable-5"><img src="{{url_for('static_content', group='images', filename='sortable.svg')}}" alt="sort" /></span><span class="sortarrow"><span id="sort-5a" style="display:none;">&#9650;</span><span id="sort-5d" style="display:none;">&#9660;</span></span></span></th>
+					<th onclick="sortTable(7)"><span class="clickable" title="Sort by Last Changed">Changed&nbsp;&nbsp;<span id="sortable-7" style="display:none;"><img src="{{url_for('static_content', group='images', filename='sortable.svg')}}" alt="sort" /></span><span class="sortarrow"><span id="sort-7a" style="display:none;">&#9650;</span><span id="sort-7d">&#9660;</span></span></span></th>
+					<th>Actions</th>
+				</tr>
+				</thead>
+				<tbody>
+					{% for watch in watches %}
+						<tr id="{{ watch.uuid }}"
+							class="{{ loop.cycle('pure-table-odd', 'pure-table-even') }}
+																								  
+							{% if watch.last_error is defined and watch.last_error != False %}error{% endif %}
+							{% if watch.last_notification_error is defined and watch.last_notification_error != False %}error{% endif %}
+							{% if watch.paused is defined and watch.paused != False %}paused{% endif %}
+							{% if watch.newest_history_key| int > watch.last_viewed| int %}unviewed{% endif %}">
+							<td class="inline chkbox"><input id="chk-{{ loop.index }}" type="checkbox" name="check" onchange="checkChange(this);">&nbsp;&nbsp;{{ loop.index }}</td>
+							<td class="inline pause-resume">
+							{% if watch.paused %}
+								<a href="{{url_for('index', pause=watch.uuid, tag=active_tag)}}"><img src="{{url_for('static_content', group='images', filename='play.svg')}}" alt="Resume" title="Resume"/></a>
+							{% else %}
+								<a href="{{url_for('index', pause=watch.uuid, tag=active_tag)}}"><img src="{{url_for('static_content', group='images', filename='pause.svg')}}" alt="Pause" title="Pause"/></a>
+							{% endif %}
+							</td>
+							<td class="title-col inline">{{watch.title if watch.title is not none and watch.title|length > 0 else watch.url}}
+								<a class="external inline-hover-img" target="_blank" rel="noopener" href="{{ watch.url }}"></a>
+								{%if watch.fetch_backend == "html_webdriver" %}<img style="height: 1em; display:inline-block;" src="{{url_for('static_content', group='images', filename='Google-Chrome-icon.png')}}" />{% endif %}
 
-    <div id="watch-table-wrapper">
-        <table class="pure-table pure-table-striped watch-table">
-            <thead>
-            <tr>
-                <th>#</th>
-                <th></th>
-                <th></th>
-                <th>Last Checked</th>
-                <th>Last Changed</th>
-                <th></th>
-            </tr>
-            </thead>
-            <tbody>
-
-
-            {% for watch in watches %}
-            <tr id="{{ watch.uuid }}"
-                class="{{ loop.cycle('pure-table-odd', 'pure-table-even') }}
-                {% if watch.last_error is defined and watch.last_error != False %}error{% endif %}
-                {% if watch.last_notification_error is defined and watch.last_notification_error != False %}error{% endif %}
-                {% if watch.paused is defined and watch.paused != False %}paused{% endif %}
-                {% if watch.newest_history_key| int > watch.last_viewed| int %}unviewed{% endif %}
-                {% if watch.uuid in queued_uuids %}queued{% endif %}">
-                <td class="inline">{{ loop.index }}</td>
-                <td class="inline paused-state state-{{watch.paused}}"><a href="{{url_for('index', pause=watch.uuid, tag=active_tag)}}"><img src="{{url_for('static_content', group='images', filename='pause.svg')}}" alt="Pause" title="Pause"/></a></td>
-
-                <td class="title-col inline">{{watch.title if watch.title is not none and watch.title|length > 0 else watch.url}}
-                    <a class="external" target="_blank" rel="noopener" href="{{ watch.url.replace('source:','') }}"></a>
-                    <a href="{{url_for('api_share_put_watch', uuid=watch.uuid)}}"><img style="height: 1em;display:inline-block;" src="{{url_for('static_content', group='images', filename='spread.svg')}}" /></a>
-
-                    {%if watch.fetch_backend == "html_webdriver" %}<img style="height: 1em; display:inline-block;" src="{{url_for('static_content', group='images', filename='Google-Chrome-icon.png')}}" />{% endif %}
-
-                    {% if watch.last_error is defined and watch.last_error != False %}
-                    <div class="fetch-error">{{ watch.last_error }}</div>
-                    {% endif %}
-                    {% if watch.last_notification_error is defined and watch.last_notification_error != False %}
-                    <div class="fetch-error notification-error">{{ watch.last_notification_error }}</div>
-                    {% endif %}
-                    {% if not active_tag %}
-                    <span class="watch-tag-list">{{ watch.tag}}</span>
-                    {% endif %}
-                </td>
-                <td class="last-checked">{{watch|format_last_checked_time}}</td>
-                <td class="last-changed">{% if watch.history|length >= 2 and watch.last_changed %}
-                    {{watch.last_changed|format_timestamp_timeago}}
-                    {% else %}
-                    Not yet
-                    {% endif %}
-                </td>
-                <td>
-                    <a {% if watch.uuid in queued_uuids %}disabled="true"{% endif %} href="{{ url_for('api_watch_checknow', uuid=watch.uuid, tag=request.args.get('tag')) }}"
-                       class="recheck pure-button button-small pure-button-primary">{% if watch.uuid in queued_uuids %}Queued{% else %}Recheck{% endif %}</a>
-                    <a href="{{ url_for('edit_page', uuid=watch.uuid)}}" class="pure-button button-small pure-button-primary">Edit</a>
-                    {% if watch.history|length >= 2 %}
-                    <a href="{{ url_for('diff_history_page', uuid=watch.uuid) }}" target="{{watch.uuid}}" class="pure-button button-small pure-button-primary diff-link">Diff</a>
-                    {% else %}
-                        {% if watch.history|length == 1 %}
-                            <a href="{{ url_for('preview_page', uuid=watch.uuid)}}" target="{{watch.uuid}}" class="pure-button button-small pure-button-primary">Preview</a>
-                        {% endif %}
-                    {% endif %}
-                </td>
-            </tr>
-            {% endfor %}
-            </tbody>
-        </table>
-        <ul id="post-list-buttons">
-            {% if has_unviewed %}
-            <li>
-                <a href="{{url_for('mark_all_viewed', tag=request.args.get('tag')) }}" class="pure-button button-tag ">Mark all viewed</a>
-            </li>
-            {% endif %}
-            <li>
-               <a href="{{ url_for('api_watch_checknow', tag=active_tag) }}" class="pure-button button-tag ">Recheck
-                all {% if active_tag%}in "{{active_tag}}"{%endif%}</a>
-            </li>
-            <li>
-                <a href="{{ url_for('rss', tag=active_tag , token=app_rss_token)}}"><img alt="RSS Feed" id="feed-icon" src="{{url_for('static_content', group='images', filename='Generic_Feed-icon.svg')}}" height="15"></a>
-            </li>
-        </ul>
-    </div>
+								{% if watch.last_error is defined and watch.last_error != False %}
+								<div class="fetch-error">{{ watch.last_error }}</div>
+								{% endif %}
+								{% if watch.last_notification_error is defined and watch.last_notification_error != False %}
+								<div class="fetch-error notification-error">{{ watch.last_notification_error }}</div>
+								{% endif %}
+								{% if not active_tag %}
+								<span class="watch-tag-list">{{ watch.tag}}</span>
+								{% endif %}
+							</td>
+							<td class="last-checked">{{watch|format_last_checked_time}}</td>
+							<td class="last-changed">{% if watch.history|length >= 2 and watch.last_changed %}
+								{{watch.last_changed|format_timestamp_timeago}}
+								{% else %}
+								Not yet
+								{% endif %}
+							</td>
+							<td>
+								<a href="{{ url_for('api_watch_checknow', uuid=watch.uuid, tag=request.args.get('tag')) }}"
+								   class="pure-button button-small pure-button-primary">Recheck</a>
+								<a href="{{ url_for('edit_page', uuid=watch.uuid)}}" class="pure-button button-small pure-button-primary">Edit</a>
+								{% if watch.history|length >= 2 %}
+								<a href="{{ url_for('diff_history_page', uuid=watch.uuid) }}" target="{{watch.uuid}}" class="pure-button button-small pure-button-primary">Diff</a>
+								{% else %}
+									{% if watch.history|length == 1 %}
+										<a href="{{ url_for('preview_page', uuid=watch.uuid)}}" target="{{watch.uuid}}" class="pure-button button-small pure-button-primary">Preview</a>
+									{% endif %}
+								{% endif %}
+							</td>
+						</tr>
+					{% endfor %}
+				</tbody>
+			</table>
+			<ul id="post-list-buttons">
+				{% if has_unviewed %}
+				<li>
+					<a href="{{url_for('mark_all_viewed', tag=request.args.get('tag')) }}" class="pure-button button-tag ">Mark all viewed</a>
+				</li>
+				{% endif %}
+				<li>
+				   <a href="{{ url_for('api_watch_checknow', tag=active_tag) }}" class="pure-button button-tag ">Recheck
+					all {% if active_tag%}in "{{active_tag}}"{%endif%}</a>
+				</li>
+				<li>
+					<a href="{{ url_for('rss', tag=active_tag , token=app_rss_token)}}"><img alt="RSS Feed" id="feed-icon" src="{{url_for('static_content', group='images', filename='Generic_Feed-icon.svg')}}" height="15"></a>
+				</li>
+			</ul>
+		</div>
+	</div>
 </div>
 {% endblock %}

changedetectionio/tests/conftest.py

@@ -16,7 +16,6 @@ def cleanup(datastore_path):
     # Unlink test output files
     files = ['output.txt',
              'url-watches.json',
-             'secret.txt',
              'notification.txt',
              'count.txt',
              'endpoint-content.txt'
@@ -43,9 +42,6 @@ def app(request):
     cleanup(app_config['datastore_path'])
     datastore = store.ChangeDetectionStore(datastore_path=app_config['datastore_path'], include_default_watches=False)
     app = changedetection_app(app_config, datastore)
-
-    # Disable CSRF while running tests
-    app.config['WTF_CSRF_ENABLED'] = False
     app.config['STOP_THREADS'] = True
 
     def teardown():

changedetectionio/tests/test_access_control.py

@@ -1,20 +1,20 @@
 from flask import url_for
-from . util import live_server_setup
+
 
 def test_check_access_control(app, client):
     # Still doesnt work, but this is closer.
 
-    with app.test_client(use_cookies=True) as c:
-        # Check we don't have any password protection enabled yet.
+    with app.test_client() as c:
+        # Check we dont have any password protection enabled yet.
         res = c.get(url_for("settings_page"))
         assert b"Remove password" not in res.data
 
         # Enable password check.
         res = c.post(
             url_for("settings_page"),
-            data={"application-password": "foobar",
-                  "requests-time_between_check-minutes": 180,
-                  'application-fetch_backend': "html_requests"},
+            data={"password": "foobar",
+                  "minutes_between_check": 180,
+                  'fetch_backend': "html_requests"},
             follow_redirects=True
         )
 
@@ -46,34 +46,63 @@ def test_check_access_control(app, client):
         assert b"BACKUP" in res.data
         assert b"IMPORT" in res.data
         assert b"LOG OUT" in res.data
-        assert b"time_between_check-minutes" in res.data
-        assert b"fetch_backend" in res.data
 
-        ##################################################
-        # Remove password button, and check that it worked
-        ##################################################
-        res = c.post(
-            url_for("settings_page"),
-            data={
-                "requests-time_between_check-minutes": 180,
-                "application-fetch_backend": "html_webdriver",
-                "application-removepassword_button": "Remove password"
-            },
-            follow_redirects=True,
-        )
+        # Now remove the password so other tests function, @todo this should happen before each test automatically
+        res = c.get(url_for("settings_page", removepassword="yes"),
+              follow_redirects=True)
         assert b"Password protection removed." in res.data
+
+        res = c.get(url_for("index"))
         assert b"LOG OUT" not in res.data
 
-        ############################################################
-        # Be sure a blank password doesnt setup password protection
-        ############################################################
+
+# There was a bug where saving the settings form would submit a blank password
+def test_check_access_control_no_blank_password(app, client):
+    # Still doesnt work, but this is closer.
+
+    with app.test_client() as c:
+        # Check we dont have any password protection enabled yet.
+        res = c.get(url_for("settings_page"))
+        assert b"Remove password" not in res.data
+
+        # Enable password check.
         res = c.post(
             url_for("settings_page"),
-            data={"application-password": "",
-                  "requests-time_between_check-minutes": 180,
-                  'application-fetch_backend': "html_requests"},
+            data={"password": "",
+                  "minutes_between_check": 180,
+                  'fetch_backend': "html_requests"},
+
+        follow_redirects=True
+        )
+
+        assert b"Password protection enabled." not in res.data
+        assert b"Login" not in res.data
+
+
+# There was a bug where saving the settings form would submit a blank password
+def test_check_access_no_remote_access_to_remove_password(app, client):
+    # Still doesnt work, but this is closer.
+
+    with app.test_client() as c:
+        # Check we dont have any password protection enabled yet.
+        res = c.get(url_for("settings_page"))
+        assert b"Remove password" not in res.data
+
+        # Enable password check.
+        res = c.post(
+            url_for("settings_page"),
+            data={"password": "password", "minutes_between_check": 180,
+                  'fetch_backend': "html_requests"},
             follow_redirects=True
         )
 
-        assert b"Password protection enabled" not in res.data
+        assert b"Password protection enabled." in res.data
+        assert b"Login" in res.data
 
+        res = c.get(url_for("settings_page", removepassword="yes"),
+              follow_redirects=True)
+        assert b"Password protection removed." not in res.data
+
+        res = c.get(url_for("index"),
+              follow_redirects=True)
+        assert b"watch-table-wrapper" not in res.data

changedetectionio/tests/test_api.py

@@ -26,8 +26,7 @@ def test_snapshot_api_detects_change(client, live_server):
     time.sleep(1)
 
     # Add our URL to the import page
-    test_url = url_for('test_endpoint', content_type="text/plain",
-                       _external=True)
+    test_url = url_for('test_endpoint', content_type="text/plain", _external=True)
     res = client.post(
         url_for("import_page"),
         data={"urls": test_url},

changedetectionio/tests/test_backend.py

@@ -25,7 +25,6 @@ def test_check_basic_change_detection_functionality(client, live_server):
         data={"urls": url_for('test_endpoint', _external=True)},
         follow_redirects=True
     )
-
     assert b"1 Imported" in res.data
 
     time.sleep(sleep_time_for_fetch_thread)
@@ -50,14 +49,6 @@ def test_check_basic_change_detection_functionality(client, live_server):
 
     #####################
 
-    # Check HTML conversion detected and workd
-    res = client.get(
-        url_for("preview_page", uuid="first"),
-        follow_redirects=True
-    )
-    # Check this class does not appear (that we didnt see the actual source)
-    assert b'foobar-detection' not in res.data
-
     # Make a change
     set_modified_response()
 
@@ -78,11 +69,6 @@ def test_check_basic_change_detection_functionality(client, live_server):
     res = client.get(url_for("rss"))
     expected_url = url_for('test_endpoint', _external=True)
     assert b'<rss' in res.data
-
-    # re #16 should have the diff in here too
-    assert b'(into   ) which has this one new line' in res.data
-    assert b'CDATA' in res.data
-    
     assert expected_url.encode('utf-8') in res.data
 
     # Following the 'diff' link, it should no longer display as 'unviewed' even after we recheck it a few times
@@ -109,7 +95,7 @@ def test_check_basic_change_detection_functionality(client, live_server):
     # Enable auto pickup of <title> in settings
     res = client.post(
         url_for("settings_page"),
-        data={"application-extract_title_as_title": "1", "requests-time_between_check-minutes": 180, 'application-fetch_backend': "html_requests"},
+        data={"extract_title_as_title": "1", "minutes_between_check": 180, 'fetch_backend': "html_requests"},
         follow_redirects=True
     )
 

changedetectionio/tests/test_element_removal.py

@@ -1,168 +0,0 @@
-#!/usr/bin/python3
-
-import time
-
-from flask import url_for
-
-from ..html_tools import *
-from .util import live_server_setup
-
-
-def test_setup(live_server):
-    live_server_setup(live_server)
-
-
-def set_original_response():
-    test_return_data = """<html>
-    <header>
-    <h2>Header</h2>
-    </header>
-    <nav>
-    <ul>
-      <li><a href="#">A</a></li>
-      <li><a href="#">B</a></li>
-      <li><a href="#">C</a></li>
-    </ul>
-    </nav>
-       <body>
-     Some initial text</br>
-     <p>Which is across multiple lines</p>
-     </br>
-     So let's see what happens.  </br>
-    <div id="changetext">Some text that will change</div>
-     </body>
-    <footer>
-    <p>Footer</p>
-    </footer>
-     </html>
-    """
-
-    with open("test-datastore/endpoint-content.txt", "w") as f:
-        f.write(test_return_data)
-
-
-def set_modified_response():
-    test_return_data = """<html>
-    <header>
-    <h2>Header changed</h2>
-    </header>
-    <nav>
-    <ul>
-      <li><a href="#">A changed</a></li>
-      <li><a href="#">B</a></li>
-      <li><a href="#">C</a></li>
-    </ul>
-    </nav>
-       <body>
-     Some initial text</br>
-     <p>Which is across multiple lines</p>
-     </br>
-     So let's see what happens.  </br>
-    <div id="changetext">Some text that changes</div>
-     </body>
-    <footer>
-    <p>Footer changed</p>
-    </footer>
-     </html>
-    """
-
-    with open("test-datastore/endpoint-content.txt", "w") as f:
-        f.write(test_return_data)
-
-
-def test_element_removal_output():
-    from changedetectionio import fetch_site_status
-    from inscriptis import get_text
-
-    # Check text with sub-parts renders correctly
-    content = """<html>
-    <header>
-    <h2>Header</h2>
-    </header>
-    <nav>
-    <ul>
-      <li><a href="#">A</a></li>
-    </ul>
-    </nav>
-       <body>
-     Some initial text</br>
-     <p>across multiple lines</p>
-     <div id="changetext">Some text that changes</div>
-     </body>
-    <footer>
-    <p>Footer</p>
-    </footer>
-     </html>
-    """
-    html_blob = element_removal(
-        ["header", "footer", "nav", "#changetext"], html_content=content
-    )
-    text = get_text(html_blob)
-    assert (
-        text
-        == """Some initial text
-
-across multiple lines
-"""
-    )
-
-
-def test_element_removal_full(client, live_server):
-    sleep_time_for_fetch_thread = 3
-
-    set_original_response()
-
-    # Give the endpoint time to spin up
-    time.sleep(1)
-
-    # Add our URL to the import page
-    test_url = url_for("test_endpoint", _external=True)
-    res = client.post(
-        url_for("import_page"), data={"urls": test_url}, follow_redirects=True
-    )
-    assert b"1 Imported" in res.data
-
-    # Goto the edit page, add the filter data
-    # Not sure why \r needs to be added - absent of the #changetext this is not necessary
-    subtractive_selectors_data = "header\r\nfooter\r\nnav\r\n#changetext"
-    res = client.post(
-        url_for("edit_page", uuid="first"),
-        data={
-            "subtractive_selectors": subtractive_selectors_data,
-            "url": test_url,
-            "tag": "",
-            "headers": "",
-            "fetch_backend": "html_requests",
-        },
-        follow_redirects=True,
-    )
-    assert b"Updated watch." in res.data
-
-    # Check it saved
-    res = client.get(
-        url_for("edit_page", uuid="first"),
-    )
-    assert bytes(subtractive_selectors_data.encode("utf-8")) in res.data
-
-    # Trigger a check
-    client.get(url_for("api_watch_checknow"), follow_redirects=True)
-
-    # Give the thread time to pick it up
-    time.sleep(sleep_time_for_fetch_thread)
-
-    # No change yet - first check
-    res = client.get(url_for("index"))
-    assert b"unviewed" not in res.data
-
-    #  Make a change to header/footer/nav
-    set_modified_response()
-
-    # Trigger a check
-    client.get(url_for("api_watch_checknow"), follow_redirects=True)
-
-    # Give the thread time to pick it up
-    time.sleep(sleep_time_for_fetch_thread)
-
-    # There should not be an unviewed change, as changes should be removed
-    res = client.get(url_for("index"))
-    assert b"unviewed" not in res.data

changedetectionio/tests/test_encoding.py

@@ -1,87 +0,0 @@
-#!/usr/bin/python3
-# coding=utf-8
-
-import time
-from flask import url_for
-from .util import live_server_setup
-import pytest
-
-
-def test_setup(live_server):
-    live_server_setup(live_server)
-
-
-def set_html_response():
-    test_return_data = """
-<html><body><span class="nav_second_img_text">
-                  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;铸大国重器，挺制造脊梁，致力能源未来，赋能美好生活。
-                                  </span>
-</body></html>
-    """
-    with open("test-datastore/endpoint-content.txt", "w") as f:
-        f.write(test_return_data)
-    return None
-
-
-# In the case the server does not issue a charset= or doesnt have content_type header set
-def test_check_encoding_detection(client, live_server):
-    set_html_response()
-
-    # Give the endpoint time to spin up
-    time.sleep(1)
-
-    # Add our URL to the import page
-    test_url = url_for('test_endpoint', content_type="text/html", _external=True)
-    client.post(
-        url_for("import_page"),
-        data={"urls": test_url},
-        follow_redirects=True
-    )
-
-    # Trigger a check
-    client.get(url_for("api_watch_checknow"), follow_redirects=True)
-
-    # Give the thread time to pick it up
-    time.sleep(2)
-
-    res = client.get(
-        url_for("preview_page", uuid="first"),
-        follow_redirects=True
-    )
-
-    # Should see the proper string
-    assert "铸大国重".encode('utf-8') in res.data
-    # Should not see the failed encoding
-    assert b'\xc2\xa7' not in res.data
-
-
-# In the case the server does not issue a charset= or doesnt have content_type header set
-def test_check_encoding_detection_missing_content_type_header(client, live_server):
-    set_html_response()
-
-    # Give the endpoint time to spin up
-    time.sleep(1)
-
-    # Add our URL to the import page
-    test_url = url_for('test_endpoint', _external=True)
-    client.post(
-        url_for("import_page"),
-        data={"urls": test_url},
-        follow_redirects=True
-    )
-
-    # Trigger a check
-    client.get(url_for("api_watch_checknow"), follow_redirects=True)
-
-    # Give the thread time to pick it up
-    time.sleep(2)
-
-    res = client.get(
-        url_for("preview_page", uuid="first"),
-        follow_redirects=True
-    )
-
-    # Should see the proper string
-    assert "铸大国重".encode('utf-8') in res.data
-    # Should not see the failed encoding
-    assert b'\xc2\xa7' not in res.data

changedetectionio/tests/test_errorhandling.py

@@ -1,7 +1,6 @@
 #!/usr/bin/python3
 
 import time
-
 from flask import url_for
 from . util import live_server_setup
 
@@ -18,9 +17,7 @@ def test_error_handler(client, live_server):
     time.sleep(1)
 
     # Add our URL to the import page
-    test_url = url_for('test_endpoint',
-                       status_code=403,
-                       _external=True)
+    test_url = url_for('test_endpoint_403_error', _external=True)
     res = client.post(
         url_for("import_page"),
         data={"urls": test_url},

changedetectionio/tests/test_html_to_text.py

@@ -1,38 +0,0 @@
-#!/usr/bin/python3
-"""Test suite for the method to extract text from an html string"""
-from ..html_tools import html_to_text
-
-
-def test_html_to_text_func():
-    test_html = """<html>
-       <body>
-     Some initial text</br>
-     <p>Which is across multiple lines</p>
-     <a href="/first_link"> More Text </a>
-     </br>
-     So let's see what happens.  </br>
-     <a href="second_link.com"> Even More Text </a>
-     </body>
-     </html>
-    """
-
-    # extract text, with 'render_anchor_tag_content' set to False
-    text_content = html_to_text(test_html, render_anchor_tag_content=False)
-
-    no_links_text = \
-        "Some initial text\n\nWhich is across multiple " \
-        "lines\n\nMore Text So let's see what happens. Even More Text"
-
-    # check that no links are in the extracted text
-    assert text_content == no_links_text
-
-    # extract text, with 'render_anchor_tag_content' set to True
-    text_content = html_to_text(test_html, render_anchor_tag_content=True)
-
-    links_text = \
-        "Some initial text\n\nWhich is across multiple lines\n\n[ More Text " \
-        "](/first_link) So let's see what happens. [ Even More Text ]" \
-        "(second_link.com)"
-
-    # check that links are present in the extracted text
-    assert text_content == links_text

changedetectionio/tests/test_ignore_text.py

@@ -171,24 +171,11 @@ def test_check_ignore_text_functionality(client, live_server):
 def test_check_global_ignore_text_functionality(client, live_server):
     sleep_time_for_fetch_thread = 3
 
-    # Give the endpoint time to spin up
-    time.sleep(1)
-
     ignore_text = "XXXXX\r\nYYYYY\r\nZZZZZ"
     set_original_ignore_response()
 
-    # Goto the settings page, add our ignore text
-    res = client.post(
-        url_for("settings_page"),
-        data={
-            "requests-time_between_check-minutes": 180,
-            "application-global_ignore_text": ignore_text,
-            'application-fetch_backend': "html_requests"
-        },
-        follow_redirects=True
-    )
-    assert b"Settings updated." in res.data
-
+    # Give the endpoint time to spin up
+    time.sleep(1)
 
     # Add our URL to the import page
     test_url = url_for('test_endpoint', _external=True)
@@ -205,6 +192,17 @@ def test_check_global_ignore_text_functionality(client, live_server):
     # Give the thread time to pick it up
     time.sleep(sleep_time_for_fetch_thread)
 
+    # Goto the settings page, add our ignore text
+    res = client.post(
+        url_for("settings_page"),
+        data={
+            "minutes_between_check": 180,
+            "global_ignore_text": ignore_text,
+            'fetch_backend': "html_requests"
+        },
+        follow_redirects=True
+    )
+    assert b"Settings updated." in res.data
 
     # Goto the edit page of the item, add our ignore text
     # Add our URL to the import page
@@ -227,16 +225,12 @@ def test_check_global_ignore_text_functionality(client, live_server):
     # Give the thread time to pick it up
     time.sleep(sleep_time_for_fetch_thread)
 
-    # so that we are sure everything is viewed and in a known 'nothing changed' state
-    res = client.get(url_for("diff_history_page", uuid="first"))
-
     # It should report nothing found (no new 'unviewed' class)
     res = client.get(url_for("index"))
     assert b'unviewed' not in res.data
     assert b'/test-endpoint' in res.data
 
-
-    #  Make a change which includes the ignore text
+    #  Make a change
     set_modified_ignore_response()
 
     # Trigger a check
@@ -249,7 +243,7 @@ def test_check_global_ignore_text_functionality(client, live_server):
     assert b'unviewed' not in res.data
     assert b'/test-endpoint' in res.data
 
-    # Just to be sure.. set a regular modified change that will trigger it
+    # Just to be sure.. set a regular modified change..
     set_modified_original_ignore_response()
     client.get(url_for("api_watch_checknow"), follow_redirects=True)
     time.sleep(sleep_time_for_fetch_thread)

changedetectionio/tests/test_ignorehyperlinks.py

@@ -1,125 +0,0 @@
-#!/usr/bin/python3
-"""Test suite for the render/not render anchor tag content functionality"""
-
-import time
-from flask import url_for
-from .util import live_server_setup
-
-
-def test_setup(live_server):
-    live_server_setup(live_server)
-
-def set_original_ignore_response():
-    test_return_data = """<html>
-       <body>
-     Some initial text</br>
-     <a href="/original_link"> Some More Text </a>
-     </br>
-     So let's see what happens.  </br>
-     </body>
-     </html>
-    """
-
-    with open("test-datastore/endpoint-content.txt", "w") as f:
-        f.write(test_return_data)
-
-
-# Should be the same as set_original_ignore_response() but with a different
-# link
-def set_modified_ignore_response():
-    test_return_data = """<html>
-       <body>
-     Some initial text</br>
-     <a href="/modified_link"> Some More Text </a>
-     </br>
-     So let's see what happens.  </br>
-     </body>
-     </html>
-    """
-
-    with open("test-datastore/endpoint-content.txt", "w") as f:
-        f.write(test_return_data)
-
-def test_render_anchor_tag_content_true(client, live_server):
-    """Testing that the link changes are detected when
-    render_anchor_tag_content setting is set to true"""
-    sleep_time_for_fetch_thread = 3
-
-    # Give the endpoint time to spin up
-    time.sleep(1)
-
-    # set original html text
-    set_original_ignore_response()
-
-    # Goto the settings page, choose to ignore links (dont select/send "application-render_anchor_tag_content")
-    res = client.post(
-        url_for("settings_page"),
-        data={
-            "requests-time_between_check-minutes": 180,
-            "application-fetch_backend": "html_requests",
-        },
-        follow_redirects=True,
-    )
-    assert b"Settings updated." in res.data
-
-    # Add our URL to the import page
-    test_url = url_for("test_endpoint", _external=True)
-    res = client.post(
-        url_for("import_page"), data={"urls": test_url},
-        follow_redirects=True
-    )
-    assert b"1 Imported" in res.data
-
-    time.sleep(sleep_time_for_fetch_thread)
-    # Trigger a check
-    client.get(url_for("api_watch_checknow"), follow_redirects=True)
-
-    # set a new html text with a modified link
-    set_modified_ignore_response()
-    time.sleep(sleep_time_for_fetch_thread)
-
-    # Trigger a check
-    client.get(url_for("api_watch_checknow"), follow_redirects=True)
-
-    # Give the thread time to pick it up
-    time.sleep(sleep_time_for_fetch_thread)
-
-    # We should not see the rendered anchor tag
-    res = client.get(url_for("preview_page", uuid="first"))
-    assert '(/modified_link)' not in res.data.decode()
-
-    # Goto the settings page, ENABLE render anchor tag
-    res = client.post(
-        url_for("settings_page"),
-        data={
-            "requests-time_between_check-minutes": 180,
-            "application-render_anchor_tag_content": "true",
-            "application-fetch_backend": "html_requests",
-        },
-        follow_redirects=True,
-    )
-    assert b"Settings updated." in res.data
-
-    # Trigger a check
-    client.get(url_for("api_watch_checknow"), follow_redirects=True)
-
-    # Give the thread time to pick it up
-    time.sleep(sleep_time_for_fetch_thread)
-
-
-
-    # check that the anchor tag content is rendered
-    res = client.get(url_for("preview_page", uuid="first"))
-    assert '(/modified_link)' in res.data.decode()
-
-    # since the link has changed, and we chose to render anchor tag content,
-    # we should detect a change (new 'unviewed' class)
-    res = client.get(url_for("index"))
-    assert b"unviewed" in res.data
-    assert b"/test-endpoint" in res.data
-
-    # Cleanup everything
-    res = client.get(url_for("api_delete", uuid="all"),
-                     follow_redirects=True)
-    assert b'Deleted' in res.data
-

changedetectionio/tests/test_ignorestatuscode.py

@@ -1,190 +0,0 @@
-#!/usr/bin/python3
-
-import time
-from flask import url_for
-from . util import live_server_setup
-
-
-def test_setup(live_server):
-    live_server_setup(live_server)
-
-
-def set_original_response():
-    test_return_data = """<html>
-       <body>
-     Some initial text</br>
-     <p>Which is across multiple lines</p>
-     </br>
-     So let's see what happens.  </br>
-     </body>
-     </html>
-    """
-
-    with open("test-datastore/endpoint-content.txt", "w") as f:
-        f.write(test_return_data)
-
-
-def set_some_changed_response():
-    test_return_data = """<html>
-       <body>
-     Some initial text</br>
-     <p>Which is across multiple lines, and a new thing too.</p>
-     </br>
-     So let's see what happens.  </br>
-     </body>
-     </html>
-    """
-
-    with open("test-datastore/endpoint-content.txt", "w") as f:
-        f.write(test_return_data)
-
-
-def test_normal_page_check_works_with_ignore_status_code(client, live_server):
-    sleep_time_for_fetch_thread = 3
-
-    # Give the endpoint time to spin up
-    time.sleep(1)
-
-    set_original_response()
-
-    # Goto the settings page, add our ignore text
-    res = client.post(
-        url_for("settings_page"),
-        data={
-            "requests-time_between_check-minutes": 180,
-            "application-ignore_status_codes": "y",
-            'application-fetch_backend': "html_requests"
-        },
-        follow_redirects=True
-    )
-    assert b"Settings updated." in res.data
-
-    # Add our URL to the import page
-    test_url = url_for('test_endpoint', _external=True)
-    res = client.post(
-        url_for("import_page"),
-        data={"urls": test_url},
-        follow_redirects=True
-    )
-    assert b"1 Imported" in res.data
-
-    time.sleep(sleep_time_for_fetch_thread)
-    # Trigger a check
-    client.get(url_for("api_watch_checknow"), follow_redirects=True)
-
-    set_some_changed_response()
-    time.sleep(sleep_time_for_fetch_thread)
-    # Trigger a check
-    client.get(url_for("api_watch_checknow"), follow_redirects=True)
-
-    # Give the thread time to pick it up
-    time.sleep(sleep_time_for_fetch_thread)
-
-    # It should report nothing found (no new 'unviewed' class)
-    res = client.get(url_for("index"))
-    assert b'unviewed' in res.data
-    assert b'/test-endpoint' in res.data
-
-
-# Tests the whole stack works with staus codes ignored
-def test_403_page_check_works_with_ignore_status_code(client, live_server):
-    sleep_time_for_fetch_thread = 3
-
-    set_original_response()
-
-    # Give the endpoint time to spin up
-    time.sleep(1)
-
-    # Add our URL to the import page
-    test_url = url_for('test_endpoint', status_code=403, _external=True)
-    res = client.post(
-        url_for("import_page"),
-        data={"urls": test_url},
-        follow_redirects=True
-    )
-    assert b"1 Imported" in res.data
-
-    # Trigger a check
-    client.get(url_for("api_watch_checknow"), follow_redirects=True)
-
-    # Give the thread time to pick it up
-    time.sleep(sleep_time_for_fetch_thread)
-
-    # Goto the edit page, check our ignore option
-    # Add our URL to the import page
-    res = client.post(
-        url_for("edit_page", uuid="first"),
-        data={"ignore_status_codes": "y", "url": test_url, "tag": "", "headers": "", 'fetch_backend': "html_requests"},
-        follow_redirects=True
-    )
-    assert b"Updated watch." in res.data
-
-    # Trigger a check
-    client.get(url_for("api_watch_checknow"), follow_redirects=True)
-
-    # Give the thread time to pick it up
-    time.sleep(sleep_time_for_fetch_thread)
-    #  Make a change
-    set_some_changed_response()
-
-    # Trigger a check
-    client.get(url_for("api_watch_checknow"), follow_redirects=True)
-    # Give the thread time to pick it up
-    time.sleep(sleep_time_for_fetch_thread)
-
-    # It should have 'unviewed' still
-    # Because it should be looking at only that 'sametext' id
-    res = client.get(url_for("index"))
-    assert b'unviewed' in res.data
-
-
-# Tests the whole stack works with staus codes ignored
-def test_403_page_check_fails_without_ignore_status_code(client, live_server):
-    sleep_time_for_fetch_thread = 3
-
-    set_original_response()
-
-    # Give the endpoint time to spin up
-    time.sleep(1)
-
-    # Add our URL to the import page
-    test_url = url_for('test_endpoint', status_code=403, _external=True)
-    res = client.post(
-        url_for("import_page"),
-        data={"urls": test_url},
-        follow_redirects=True
-    )
-    assert b"1 Imported" in res.data
-
-    # Trigger a check
-    client.get(url_for("api_watch_checknow"), follow_redirects=True)
-
-    # Give the thread time to pick it up
-    time.sleep(sleep_time_for_fetch_thread)
-
-    # Goto the edit page, check our ignore option
-    # Add our URL to the import page
-    res = client.post(
-        url_for("edit_page", uuid="first"),
-        data={"url": test_url, "tag": "", "headers": "", 'fetch_backend': "html_requests"},
-        follow_redirects=True
-    )
-    assert b"Updated watch." in res.data
-
-    # Trigger a check
-    client.get(url_for("api_watch_checknow"), follow_redirects=True)
-
-    # Give the thread time to pick it up
-    time.sleep(sleep_time_for_fetch_thread)
-    #  Make a change
-    set_some_changed_response()
-
-    # Trigger a check
-    client.get(url_for("api_watch_checknow"), follow_redirects=True)
-    # Give the thread time to pick it up
-    time.sleep(sleep_time_for_fetch_thread)
-
-    # It should have 'unviewed' still
-    # Because it should be looking at only that 'sametext' id
-    res = client.get(url_for("index"))
-    assert b'Status Code 403' in res.data

changedetectionio/tests/test_ignorewhitespace.py

@@ -61,9 +61,9 @@ def test_check_ignore_whitespace(client, live_server):
     res = client.post(
         url_for("settings_page"),
         data={
-            "requests-time_between_check-minutes": 180,
-            "application-ignore_whitespace": "y",
-            "application-fetch_backend": "html_requests"
+            "minutes_between_check": 180,
+            "ignore_whitespace": "y",
+            'fetch_backend': "html_requests"
         },
         follow_redirects=True
     )

changedetectionio/tests/test_import.py

@@ -5,17 +5,18 @@ import time
 from flask import url_for
 
 from .util import live_server_setup
-def test_setup(client, live_server):
-    live_server_setup(live_server)
+
 
 def test_import(client, live_server):
+
+    live_server_setup(live_server)
+
     # Give the endpoint time to spin up
     time.sleep(1)
 
     res = client.post(
         url_for("import_page"),
         data={
-            "distill-io": "",
             "urls": """https://example.com
 https://example.com tag1
 https://example.com tag1, other tag"""
@@ -25,96 +26,3 @@ https://example.com tag1, other tag"""
     assert b"3 Imported" in res.data
     assert b"tag1" in res.data
     assert b"other tag" in res.data
-    res = client.get(url_for("api_delete", uuid="all"), follow_redirects=True)
-
-    # Clear flask alerts
-    res = client.get( url_for("index"))
-    res = client.get( url_for("index"))
-
-def xtest_import_skip_url(client, live_server):
-
-
-    # Give the endpoint time to spin up
-    time.sleep(1)
-
-    res = client.post(
-        url_for("import_page"),
-        data={
-            "distill-io": "",
-            "urls": """https://example.com
-:ht000000broken
-"""
-        },
-        follow_redirects=True,
-    )
-    assert b"1 Imported" in res.data
-    assert b"ht000000broken" in res.data
-    assert b"1 Skipped" in res.data
-    res = client.get(url_for("api_delete", uuid="all"), follow_redirects=True)
-    # Clear flask alerts
-    res = client.get( url_for("index"))
-
-def test_import_distillio(client, live_server):
-
-    distill_data='''
-{
-    "client": {
-        "local": 1
-    },
-    "data": [
-        {
-            "name": "Unraid | News",
-            "uri": "https://unraid.net/blog",
-            "config": "{\\"selections\\":[{\\"frames\\":[{\\"index\\":0,\\"excludes\\":[],\\"includes\\":[{\\"type\\":\\"xpath\\",\\"expr\\":\\"(//div[@id='App']/div[contains(@class,'flex')]/main[contains(@class,'relative')]/section[contains(@class,'relative')]/div[@class='container']/div[contains(@class,'flex')]/div[contains(@class,'w-full')])[1]\\"}]}],\\"dynamic\\":true,\\"delay\\":2}],\\"ignoreEmptyText\\":true,\\"includeStyle\\":false,\\"dataAttr\\":\\"text\\"}",
-            "tags": ["nice stuff", "nerd-news"],
-            "content_type": 2,
-            "state": 40,
-            "schedule": "{\\"type\\":\\"INTERVAL\\",\\"params\\":{\\"interval\\":4447}}",
-            "ts": "2022-03-27T15:51:15.667Z"
-        }
-    ]
-}		   
-
-'''
-
-    # Give the endpoint time to spin up
-    time.sleep(1)
-    client.get(url_for("api_delete", uuid="all"), follow_redirects=True)
-    res = client.post(
-        url_for("import_page"),
-        data={
-            "distill-io": distill_data,
-            "urls" : ''
-        },
-        follow_redirects=True,
-    )
-
-
-    assert b"Unable to read JSON file, was it broken?" not in res.data
-    assert b"1 Imported from Distill.io" in res.data
-
-    res = client.get( url_for("edit_page", uuid="first"))
-
-    assert b"https://unraid.net/blog" in res.data
-    assert b"Unraid | News" in res.data
-
-
-    # flask/wtforms should recode this, check we see it
-    # wtforms encodes it like id=&#39 ,but html.escape makes it like id=&#x27
-    # - so just check it manually :(
-    #import json
-    #import html
-    #d = json.loads(distill_data)
-    # embedded_d=json.loads(d['data'][0]['config'])
-    # x=html.escape(embedded_d['selections'][0]['frames'][0]['includes'][0]['expr']).encode('utf-8')
-    assert b"xpath:(//div[@id='App']/div[contains(@class,'flex')]/main[contains(@class,'relative')]/section[contains(@class,'relative')]/div[@class='container']/div[contains(@class,'flex')]/div[contains(@class,'w-full')])[1]" in res.data
-
-    # did the tags work?
-    res = client.get( url_for("index"))
-
-    assert b"nice stuff" in res.data
-    assert b"nerd-news" in res.data
-
-    res = client.get(url_for("api_delete", uuid="all"), follow_redirects=True)
-    # Clear flask alerts
-    res = client.get(url_for("index"))

changedetectionio/tests/test_jsonpath_selector.py

@@ -1,5 +1,4 @@
 #!/usr/bin/python3
-# coding=utf-8
 
 import time
 from flask import url_for
@@ -143,7 +142,7 @@ def set_modified_response():
         }
       ],
       "boss": {
-        "name": "Örnsköldsvik"
+        "name": "Foobar"
       },
       "available": false
     }
@@ -247,10 +246,8 @@ def test_check_json_filter(client, live_server):
 
     # Should not see this, because its not in the JSONPath we entered
     res = client.get(url_for("diff_history_page", uuid="first"))
-
     # But the change should be there, tho its hard to test the change was detected because it will show old and new versions
-    # And #462 - check we see the proper utf-8 string there
-    assert "Örnsköldsvik".encode('utf-8') in res.data
+    assert b'Foobar' in res.data
 
 
 def test_check_json_filter_bool_val(client, live_server):
@@ -270,7 +267,6 @@ def test_check_json_filter_bool_val(client, live_server):
     )
     assert b"1 Imported" in res.data
 
-    time.sleep(3)
     # Goto the edit page, add our ignore text
     # Add our URL to the import page
     res = client.post(
@@ -285,7 +281,6 @@ def test_check_json_filter_bool_val(client, live_server):
     )
     assert b"Updated watch." in res.data
 
-    time.sleep(3)
 
     # Trigger a check
     client.get(url_for("api_watch_checknow"), follow_redirects=True)

changedetectionio/tests/test_nonrenderable_pages.py

@@ -1,102 +0,0 @@
-#!/usr/bin/python3
-
-import time
-from flask import url_for
-from urllib.request import urlopen
-from .util import set_original_response, set_modified_response, live_server_setup
-
-sleep_time_for_fetch_thread = 3
-
-
-def set_nonrenderable_response():
-    test_return_data = """<html>
-    <head><title>modified head title</title></head>
-    <!-- like when some angular app was broken and doesnt render or whatever -->
-    <body>
-     </body>
-     </html>
-    """
-
-    with open("test-datastore/endpoint-content.txt", "w") as f:
-        f.write(test_return_data)
-
-    return None
-
-def test_check_basic_change_detection_functionality(client, live_server):
-    set_original_response()
-    live_server_setup(live_server)
-
-    # Add our URL to the import page
-    res = client.post(
-        url_for("import_page"),
-        data={"urls": url_for('test_endpoint', _external=True)},
-        follow_redirects=True
-    )
-
-    assert b"1 Imported" in res.data
-
-    time.sleep(sleep_time_for_fetch_thread)
-
-    # Do this a few times.. ensures we dont accidently set the status
-    for n in range(3):
-        client.get(url_for("api_watch_checknow"), follow_redirects=True)
-
-        # Give the thread time to pick it up
-        time.sleep(sleep_time_for_fetch_thread)
-
-        # It should report nothing found (no new 'unviewed' class)
-        res = client.get(url_for("index"))
-        assert b'unviewed' not in res.data
-
-
-    #####################
-    client.post(
-        url_for("settings_page"),
-        data={"application-empty_pages_are_a_change": "",
-              "requests-time_between_check-minutes": 180,
-              'application-fetch_backend': "html_requests"},
-        follow_redirects=True
-    )
-
-    # this should not trigger a change, because no good text could be converted from the HTML
-    set_nonrenderable_response()
-
-    client.get(url_for("api_watch_checknow"), follow_redirects=True)
-
-    # Give the thread time to pick it up
-    time.sleep(sleep_time_for_fetch_thread)
-
-    # It should report nothing found (no new 'unviewed' class)
-    res = client.get(url_for("index"))
-    assert b'unviewed' not in res.data
-
-
-    # ok now do the opposite
-
-    client.post(
-        url_for("settings_page"),
-        data={"application-empty_pages_are_a_change": "y",
-              "requests-time_between_check-minutes": 180,
-              'application-fetch_backend': "html_requests"},
-        follow_redirects=True
-    )
-    set_modified_response()
-
-
-    client.get(url_for("api_watch_checknow"), follow_redirects=True)
-
-    # Give the thread time to pick it up
-    time.sleep(sleep_time_for_fetch_thread)
-
-    # It should report nothing found (no new 'unviewed' class)
-    res = client.get(url_for("index"))
-    assert b'unviewed' in res.data
-
-
-
-
-    #
-    # Cleanup everything
-    res = client.get(url_for("api_delete", uuid="all"), follow_redirects=True)
-    assert b'Deleted' in res.data
-

changedetectionio/tests/test_notification.py

@@ -2,17 +2,15 @@ import os
 import time
 import re
 from flask import url_for
-from . util import set_original_response, set_modified_response, set_more_modified_response, live_server_setup
+from . util import set_original_response, set_modified_response, live_server_setup
 import logging
 from changedetectionio.notification import default_notification_body, default_notification_title
 
-def test_setup(live_server):
-    live_server_setup(live_server)
-
 # Hard to just add more live server URLs when one test is already running (I think)
 # So we add our test here (was in a different file)
 def test_check_notification(client, live_server):
 
+    live_server_setup(live_server)
     set_original_response()
 
     # Give the endpoint time to spin up
@@ -51,76 +49,84 @@ def test_check_notification(client, live_server):
     notification_url = url.replace('http', 'json')
 
     print (">>>> Notification URL: "+notification_url)
-
-    notification_form_data = {"notification_urls": notification_url,
-                              "notification_title": "New ChangeDetection.io Notification - {watch_url}",
-                              "notification_body": "BASE URL: {base_url}\n"
-                                                   "Watch URL: {watch_url}\n"
-                                                   "Watch UUID: {watch_uuid}\n"
-                                                   "Watch title: {watch_title}\n"
-                                                   "Watch tag: {watch_tag}\n"
-                                                   "Preview: {preview_url}\n"
-                                                   "Diff URL: {diff_url}\n"
-                                                   "Snapshot: {current_snapshot}\n"
-                                                   "Diff: {diff}\n"
-                                                   "Diff Full: {diff_full}\n"
-                                                   ":-)",
-                              "notification_format": "Text"}
-
-    notification_form_data.update({
-        "url": test_url,
-        "tag": "my tag",
-        "title": "my title",
-        "headers": "",
-        "fetch_backend": "html_requests"})
-
     res = client.post(
         url_for("edit_page", uuid="first"),
-        data=notification_form_data,
+        data={"notification_urls": notification_url,
+              "notification_title": "New ChangeDetection.io Notification - {watch_url}",
+              "notification_body": "BASE URL: {base_url}\n"
+                                   "Watch URL: {watch_url}\n"
+                                   "Watch UUID: {watch_uuid}\n"
+                                   "Watch title: {watch_title}\n"
+                                   "Watch tag: {watch_tag}\n"
+                                   "Preview: {preview_url}\n"
+                                   "Diff URL: {diff_url}\n"
+                                   "Snapshot: {current_snapshot}\n"
+                                   "Diff: {diff}\n"
+                                   "Diff Full: {diff_full}\n"
+                                   ":-)",
+              "notification_format": "Text",
+              "url": test_url,
+              "tag": "my tag",
+              "title": "my title",
+              "headers": "",
+              "fetch_backend": "html_requests",
+              "trigger_check": "y"},
         follow_redirects=True
     )
     assert b"Updated watch." in res.data
-
+    assert b"Test notification queued" in res.data
 
     # Hit the edit page, be sure that we saved it
-    # Re #242 - wasnt saving?
     res = client.get(
         url_for("edit_page", uuid="first"))
     assert bytes(notification_url.encode('utf-8')) in res.data
+
+    # Re #242 - wasnt saving?
     assert bytes("New ChangeDetection.io Notification".encode('utf-8')) in res.data
 
 
 
-    ## Now recheck, and it should have sent the notification
+    # Because we hit 'send test notification on save'
     time.sleep(3)
-    set_modified_response()
 
     notification_submission = None
 
-    # Trigger a check
-    client.get(url_for("api_watch_checknow"), follow_redirects=True)
-    time.sleep(3)
     # Verify what was sent as a notification, this file should exist
     with open("test-datastore/notification.txt", "r") as f:
         notification_submission = f.read()
+        # Did we see the URL that had a change, in the notification?
+
+    assert test_url in notification_submission
+
     os.unlink("test-datastore/notification.txt")
 
-    # Did we see the URL that had a change, in the notification?
-    # Diff was correctly executed
+    set_modified_response()
+
+    # Trigger a check
+    client.get(url_for("api_watch_checknow"), follow_redirects=True)
+
+    # Give the thread time to pick it up
+    time.sleep(3)
+
+    # Did the front end see it?
+    res = client.get(
+        url_for("index"))
+
+    assert bytes("just now".encode('utf-8')) in res.data
+
+    notification_submission=None
+    # Verify what was sent as a notification
+    with open("test-datastore/notification.txt", "r") as f:
+        notification_submission = f.read()
+        # Did we see the URL that had a change, in the notification?
+
     assert test_url in notification_submission
-    assert ':-)' in notification_submission
+
+    # Diff was correctly executed
     assert "Diff Full: Some initial text" in notification_submission
     assert "Diff: (changed) Which is across multiple lines" in notification_submission
-    assert "(into   ) which has this one new line" in notification_submission
-    # Re #342 - check for accidental python byte encoding of non-utf8/string
-    assert "b'" not in notification_submission
-    assert re.search('Watch UUID: [0-9a-f]{8}(-[0-9a-f]{4}){3}-[0-9a-f]{12}', notification_submission, re.IGNORECASE)
-    assert "Watch title: my title" in notification_submission
-    assert "Watch tag: my tag" in notification_submission
-    assert "diff/" in notification_submission
-    assert "preview/" in notification_submission
-    assert ":-)" in notification_submission
-    assert "New ChangeDetection.io Notification - {}".format(test_url) in notification_submission
+    assert "(-> into) which has this one new line" in notification_submission
+
 
     if env_base_url:
         # Re #65 - did we see our BASE_URl ?
@@ -129,17 +135,50 @@ def test_check_notification(client, live_server):
     else:
         logging.debug(">>> Skipping BASE_URL check")
 
+    ##  Now configure something clever, we go into custom config (non-default) mode, this is returned by the endpoint
+    with open("test-datastore/endpoint-content.txt", "w") as f:
+        f.write(";jasdhflkjadshf kjhsdfkjl ahslkjf haslkjd hfaklsj hf\njl;asdhfkasj stuff we will detect\n")
 
+    res = client.post(
+        url_for("settings_page"),
+        data={"notification_title": "New ChangeDetection.io Notification - {watch_url}",
+              "notification_urls": "json://foobar.com", #Re #143 should not see that it sent without [test checkbox]
+              "minutes_between_check": 180,
+              "fetch_backend": "html_requests",
+              },
+        follow_redirects=True
+    )
+    assert b"Settings updated." in res.data
+    # Re #143 - should not see this if we didnt hit the test box
+    assert b"Test notification queued" not in res.data
 
-    # This should insert the {current_snapshot}
-    set_more_modified_response()
+    # Trigger a check
     client.get(url_for("api_watch_checknow"), follow_redirects=True)
+
+    # Give the thread time to pick it up
     time.sleep(3)
-    # Verify what was sent as a notification, this file should exist
+
+    # Did the front end see it?
+    res = client.get(
+        url_for("index"))
+
+    assert bytes("just now".encode('utf-8')) in res.data
+
     with open("test-datastore/notification.txt", "r") as f:
         notification_submission = f.read()
-    assert "Ohh yeah awesome" in notification_submission
+        print ("Notification submission was:", notification_submission)
+        # Re #342 - check for accidental python byte encoding of non-utf8/string
+        assert "b'" not in notification_submission
 
+        assert re.search('Watch UUID: [0-9a-f]{8}(-[0-9a-f]{4}){3}-[0-9a-f]{12}', notification_submission, re.IGNORECASE)
+        assert "Watch title: my title" in notification_submission
+        assert "Watch tag: my tag" in notification_submission
+        assert "diff/" in notification_submission
+        assert "preview/" in notification_submission
+        assert ":-)" in notification_submission
+        assert "New ChangeDetection.io Notification - {}".format(test_url) in notification_submission
+        # This should insert the {current_snapshot}
+        assert "stuff we will detect" in notification_submission
 
     # Prove that "content constantly being marked as Changed with no Updating causes notification" is not a thing
     # https://github.com/dgtlmoon/changedetection.io/discussions/192
@@ -147,58 +186,22 @@ def test_check_notification(client, live_server):
 
     # Trigger a check
     client.get(url_for("api_watch_checknow"), follow_redirects=True)
-    time.sleep(1)
+    time.sleep(3)
     client.get(url_for("api_watch_checknow"), follow_redirects=True)
-    time.sleep(1)
+    time.sleep(3)
     client.get(url_for("api_watch_checknow"), follow_redirects=True)
-    time.sleep(1)
+    time.sleep(3)
     assert os.path.exists("test-datastore/notification.txt") == False
 
-    # cleanup for the next
-    client.get(
-        url_for("api_delete", uuid="all"),
-        follow_redirects=True
-    )
-
-
-def test_notification_validation(client, live_server):
-    #live_server_setup(live_server)
-    time.sleep(3)
-    # re #242 - when you edited an existing new entry, it would not correctly show the notification settings
-    # Add our URL to the import page
-    test_url = url_for('test_endpoint', _external=True)
-    res = client.post(
-        url_for("api_watch_add"),
-        data={"url": test_url, "tag": 'nice one'},
-        follow_redirects=True
-    )
-
-    assert b"Watch added" in res.data
-
-    # Re #360 some validation
-    res = client.post(
-        url_for("edit_page", uuid="first"),
-        data={"notification_urls": 'json://localhost/foobar',
-              "notification_title": "",
-              "notification_body": "",
-              "notification_format": "Text",
-              "url": test_url,
-              "tag": "my tag",
-              "title": "my title",
-              "headers": "",
-              "fetch_backend": "html_requests"},
-        follow_redirects=True
-    )
-    assert b"Notification Body and Title is required when a Notification URL is used" in res.data
 
     # Now adding a wrong token should give us an error
     res = client.post(
         url_for("settings_page"),
-        data={"application-notification_title": "New ChangeDetection.io Notification - {watch_url}",
-              "application-notification_body": "Rubbish: {rubbish}\n",
-              "application-notification_format": "Text",
-              "application-notification_urls": "json://localhost/foobar",
-              "requests-time_between_check-minutes": 180,
+        data={"notification_title": "New ChangeDetection.io Notification - {watch_url}",
+              "notification_body": "Rubbish: {rubbish}\n",
+              "notification_format": "Text",
+              "notification_urls": "json://foobar.com",
+              "minutes_between_check": 180,
               "fetch_backend": "html_requests"
               },
         follow_redirects=True
@@ -206,8 +209,19 @@ def test_notification_validation(client, live_server):
 
     assert bytes("is not a valid token".encode('utf-8')) in res.data
 
-    # cleanup for the next
-    client.get(
-        url_for("api_delete", uuid="all"),
+    # Re #360 some validation
+    res = client.post(
+        url_for("edit_page", uuid="first"),
+        data={"notification_urls": notification_url,
+              "notification_title": "",
+              "notification_body": "",
+              "notification_format": "Text",
+              "url": test_url,
+              "tag": "my tag",
+              "title": "my title",
+              "headers": "",
+              "fetch_backend": "html_requests",
+              "trigger_check": "y"},
         follow_redirects=True
     )
+    assert b"Notification Body and Title is required when a Notification URL is used" in res.data

changedetectionio/tests/test_notification_errors.py

@@ -35,7 +35,7 @@ def test_check_notification_error_handling(client, live_server):
               "tag": "",
               "title": "",
               "headers": "",
-              "time_between_check-minutes": "180",
+              "minutes_between_check": "180",
               "fetch_backend": "html_requests",
               "trigger_check": "y"},
         follow_redirects=True

changedetectionio/tests/test_request.py

@@ -27,7 +27,6 @@ def test_headers_in_request(client, live_server):
     )
     assert b"1 Imported" in res.data
 
-    time.sleep(3)
     cookie_header = '_ga=GA1.2.1022228332; cookie-preferences=analytics:accepted;'
 
 
@@ -78,58 +77,6 @@ def test_body_in_request(client, live_server):
     # Add our URL to the import page
     test_url = url_for('test_body', _external=True)
 
-    res = client.post(
-        url_for("import_page"),
-        data={"urls": test_url},
-        follow_redirects=True
-    )
-    assert b"1 Imported" in res.data
-
-    time.sleep(3)
-
-    # add the first 'version'
-    res = client.post(
-        url_for("edit_page", uuid="first"),
-        data={
-              "url": test_url,
-              "tag": "",
-              "method": "POST",
-              "fetch_backend": "html_requests",
-              "body": "something something"},
-        follow_redirects=True
-    )
-    assert b"Updated watch." in res.data
-
-    time.sleep(3)
-
-    # Now the change which should trigger a change
-    body_value = 'Test Body Value'
-    res = client.post(
-        url_for("edit_page", uuid="first"),
-        data={
-              "url": test_url,
-              "tag": "",
-              "method": "POST",
-              "fetch_backend": "html_requests",
-              "body": body_value},
-        follow_redirects=True
-    )
-    assert b"Updated watch." in res.data
-
-    time.sleep(3)
-
-    # The service should echo back the body
-    res = client.get(
-        url_for("preview_page", uuid="first"),
-        follow_redirects=True
-    )
-
-    # If this gets stuck something is wrong, something should always be there
-    assert b"No history found" not in res.data
-    # We should see what we sent in the reply
-    assert str.encode(body_value) in res.data
-
-    ####### data sanity checks
     # Add the test URL twice, we will check
     res = client.post(
         url_for("import_page"),
@@ -138,15 +85,14 @@ def test_body_in_request(client, live_server):
     )
     assert b"1 Imported" in res.data
 
-    watches_with_body = 0
-    with open('test-datastore/url-watches.json') as f:
-        app_struct = json.load(f)
-        for uuid in app_struct['watching']:
-            if app_struct['watching'][uuid]['body']==body_value:
-                watches_with_body += 1
+    res = client.post(
+        url_for("import_page"),
+        data={"urls": test_url},
+        follow_redirects=True
+    )
+    assert b"1 Imported" in res.data
 
-    # Should be only one with body set
-    assert watches_with_body==1
+    body_value = 'Test Body Value'
 
     # Attempt to add a body with a GET method
     res = client.post(
@@ -161,6 +107,40 @@ def test_body_in_request(client, live_server):
     )
     assert b"Body must be empty when Request Method is set to GET" in res.data
 
+    # Add a properly formatted body with a proper method
+    res = client.post(
+        url_for("edit_page", uuid="first"),
+        data={
+              "url": test_url,
+              "tag": "",
+              "method": "POST",
+              "fetch_backend": "html_requests",
+              "body": body_value},
+        follow_redirects=True
+    )
+    assert b"Updated watch." in res.data
+
+    # Give the thread time to pick up the first version
+    time.sleep(5)
+
+    # The service should echo back the body
+    res = client.get(
+        url_for("preview_page", uuid="first"),
+        follow_redirects=True
+    )
+
+    # Check if body returned contains the specified data
+    assert str.encode(body_value) in res.data
+
+    watches_with_body = 0
+    with open('test-datastore/url-watches.json') as f:
+        app_struct = json.load(f)
+        for uuid in app_struct['watching']:
+            if app_struct['watching'][uuid]['body']==body_value:
+                watches_with_body += 1
+
+    # Should be only one with body set
+    assert watches_with_body==1
 
 def test_method_in_request(client, live_server):
     # Add our URL to the import page

changedetectionio/tests/test_security.py

@@ -1,36 +0,0 @@
-from flask import url_for
-from . util import set_original_response, set_modified_response, live_server_setup
-import time
-
-def test_setup(live_server):
-    live_server_setup(live_server)
-
-def test_file_access(client, live_server):
-
-    res = client.post(
-        url_for("import_page"),
-        data={"urls": 'https://localhost'},
-        follow_redirects=True
-    )
-
-    assert b"1 Imported" in res.data
-
-    # Attempt to add a body with a GET method
-    res = client.post(
-        url_for("edit_page", uuid="first"),
-        data={
-              "url": 'file:///etc/passwd',
-              "tag": "",
-              "method": "GET",
-              "fetch_backend": "html_requests",
-              "body": ""},
-        follow_redirects=True
-    )
-    time.sleep(3)
-
-    res = client.get(
-        url_for("index", uuid="first"),
-        follow_redirects=True
-    )
-
-    assert b'denied for security reasons' in res.data

changedetectionio/tests/test_share_watch.py

@@ -1,76 +0,0 @@
-#!/usr/bin/python3
-
-import time
-from flask import url_for
-from urllib.request import urlopen
-from .util import set_original_response, set_modified_response, live_server_setup
-import re
-
-sleep_time_for_fetch_thread = 3
-
-
-def test_share_watch(client, live_server):
-    set_original_response()
-    live_server_setup(live_server)
-
-    test_url = url_for('test_endpoint', _external=True)
-    css_filter = ".nice-filter"
-
-    # Add our URL to the import page
-    res = client.post(
-        url_for("import_page"),
-        data={"urls": test_url},
-        follow_redirects=True
-    )
-
-    assert b"1 Imported" in res.data
-
-    # Goto the edit page, add our ignore text
-    # Add our URL to the import page
-    res = client.post(
-        url_for("edit_page", uuid="first"),
-        data={"css_filter": css_filter, "url": test_url, "tag": "", "headers": "", 'fetch_backend': "html_requests"},
-        follow_redirects=True
-    )
-    assert b"Updated watch." in res.data
-    # Check it saved
-    res = client.get(
-        url_for("edit_page", uuid="first"),
-    )
-    assert bytes(css_filter.encode('utf-8')) in res.data
-
-    # click share the link
-    res = client.get(
-        url_for("api_share_put_watch", uuid="first"),
-        follow_redirects=True
-    )
-
-    assert b"Share this link:" in res.data
-    assert b"https://changedetection.io/share/" in res.data
-
-    html = res.data.decode()
-    share_link_search = re.search('<span id="share-link">(.*)</span>', html, re.IGNORECASE)
-    assert share_link_search
-
-    # Now delete what we have, we will try to re-import it
-    # Cleanup everything
-    res = client.get(url_for("api_delete", uuid="all"), follow_redirects=True)
-    assert b'Deleted' in res.data
-
-    # Add our URL to the import page
-    res = client.post(
-        url_for("import_page"),
-        data={"urls": share_link_search.group(1)},
-        follow_redirects=True
-    )
-
-    assert b"1 Imported" in res.data
-
-    # Now hit edit, we should see what we expect
-    # that the import fetched the meta-data
-
-    # Check it saved
-    res = client.get(
-        url_for("edit_page", uuid="first"),
-    )
-    assert bytes(css_filter.encode('utf-8')) in res.data

changedetectionio/tests/test_source.py

@@ -1,95 +0,0 @@
-#!/usr/bin/python3
-
-import time
-from flask import url_for
-from urllib.request import urlopen
-from .util import set_original_response, set_modified_response, live_server_setup
-
-sleep_time_for_fetch_thread = 3
-
-def test_setup(live_server):
-    live_server_setup(live_server)
-
-def test_check_basic_change_detection_functionality_source(client, live_server):
-    set_original_response()
-    test_url = 'source:'+url_for('test_endpoint', _external=True)
-    # Add our URL to the import page
-    res = client.post(
-        url_for("import_page"),
-        data={"urls": test_url},
-        follow_redirects=True
-    )
-
-    assert b"1 Imported" in res.data
-
-    time.sleep(sleep_time_for_fetch_thread)
-
-    #####################
-
-    # Check HTML conversion detected and workd
-    res = client.get(
-        url_for("preview_page", uuid="first"),
-        follow_redirects=True
-    )
-
-    # Check this class DOES appear (that we didnt see the actual source)
-    assert b'foobar-detection' in res.data
-
-    # Make a change
-    set_modified_response()
-
-    # Force recheck
-    res = client.get(url_for("api_watch_checknow"), follow_redirects=True)
-    assert b'1 watches are queued for rechecking.' in res.data
-
-    time.sleep(5)
-
-    # Now something should be ready, indicated by having a 'unviewed' class
-    res = client.get(url_for("index"))
-    assert b'unviewed' in res.data
-
-    res = client.get(
-        url_for("diff_history_page", uuid="first"),
-        follow_redirects=True
-    )
-
-    assert b'<title>modified head title' in res.data
-
-
-
-
-def test_check_ignore_elements(client, live_server):
-    set_original_response()
-
-    time.sleep(2)
-    test_url = 'source:'+url_for('test_endpoint', _external=True)
-    # Add our URL to the import page
-    res = client.post(
-        url_for("import_page"),
-        data={"urls": test_url},
-        follow_redirects=True
-    )
-
-    assert b"1 Imported" in res.data
-
-    time.sleep(sleep_time_for_fetch_thread)
-
-    #####################
-    # We want <span> and <p> ONLY, but ignore span with .foobar-detection
-
-    res = client.post(
-        url_for("edit_page", uuid="first"),
-        data={"css_filter": 'span,p', "url": test_url, "tag": "", "subtractive_selectors": ".foobar-detection", 'fetch_backend': "html_requests"},
-        follow_redirects=True
-    )
-
-    time.sleep(sleep_time_for_fetch_thread)
-
-    res = client.get(
-        url_for("preview_page", uuid="first"),
-        follow_redirects=True
-    )
-
-    assert b'foobar-detection' not in res.data
-    assert b'&lt;br' not in res.data
-    assert b'&lt;p' in res.data

changedetectionio/tests/test_watch_fields_storage.py

@@ -20,8 +20,8 @@ def test_check_watch_field_storage(client, live_server):
 
     res = client.post(
         url_for("edit_page", uuid="first"),
-        data={ "notification_urls": "json://127.0.0.1:30000\r\njson://128.0.0.1\r\n",
-               "time_between_check-minutes": 126,
+        data={ "notification_urls": "json://myapi.com",
+               "minutes_between_check": 126,
                "css_filter" : ".fooclass",
                "title" : "My title",
                "ignore_text" : "ignore this",
@@ -38,14 +38,8 @@ def test_check_watch_field_storage(client, live_server):
         url_for("edit_page", uuid="first"),
         follow_redirects=True
     )
-    # checks that we dont get an error when using blank lines in the field value
-    assert not b"json://127.0.0.1\n\njson" in res.data
-    assert not b"json://127.0.0.1\r\n\njson" in res.data
-    assert not b"json://127.0.0.1\r\n\rjson" in res.data
-
-    assert b"json://127.0.0.1" in res.data
-    assert b"json://128.0.0.1" in res.data
 
+    assert b"json://myapi.com" in res.data
     assert b"126" in res.data
     assert b".fooclass" in res.data
     assert b"My title" in res.data
@@ -62,8 +56,8 @@ def test_check_recheck_global_setting(client, live_server):
     res = client.post(
         url_for("settings_page"),
         data={
-               "requests-time_between_check-minutes": 1566,
-               'application-fetch_backend': "html_requests"
+               "minutes_between_check": 1566,
+               'fetch_backend': "html_requests"
                },
         follow_redirects=True
     )
@@ -94,8 +88,8 @@ def test_check_recheck_global_setting(client, live_server):
     res = client.post(
         url_for("settings_page"),
         data={
-               "requests-time_between_check-minutes": 222,
-                'application-fetch_backend': "html_requests"
+               "minutes_between_check": 222,
+                'fetch_backend': "html_requests"
                },
         follow_redirects=True
     )
@@ -114,7 +108,7 @@ def test_check_recheck_global_setting(client, live_server):
     res = client.post(
         url_for("edit_page", uuid="first"),
         data={"url": test_url,
-              "time_between_check-minutes": 55,
+              "minutes_between_check": 55,
               'fetch_backend': "html_requests"
               },
         follow_redirects=True
@@ -130,8 +124,8 @@ def test_check_recheck_global_setting(client, live_server):
     res = client.post(
         url_for("settings_page"),
         data={
-               "requests-time_between_check-minutes": 666,
-                "application-fetch_backend": "html_requests"
+               "minutes_between_check": 666,
+                'fetch_backend': "html_requests"
                },
         follow_redirects=True
     )
@@ -140,7 +134,7 @@ def test_check_recheck_global_setting(client, live_server):
     res = client.post(
         url_for("edit_page", uuid="first"),
         data={"url": test_url,
-              "time_between_check-minutes": "",
+              "minutes_between_check": "",
               'fetch_backend': "html_requests"
               },
         follow_redirects=True
@@ -153,3 +147,4 @@ def test_check_recheck_global_setting(client, live_server):
         follow_redirects=True
     )
     assert b"666" in res.data
+

changedetectionio/tests/test_xpath_selector.py

@@ -116,46 +116,4 @@ def test_xpath_validation(client, live_server):
         data={"css_filter": "/something horrible", "url": test_url, "tag": "", "headers": "", 'fetch_backend': "html_requests"},
         follow_redirects=True
     )
-    assert b"is not a valid XPath expression" in res.data
-
-
-# actually only really used by the distll.io importer, but could be handy too
-def test_check_with_prefix_css_filter(client, live_server):
-    res = client.get(url_for("api_delete", uuid="all"), follow_redirects=True)
-    assert b'Deleted' in res.data
-
-    # Give the endpoint time to spin up
-    time.sleep(1)
-
-    set_original_response()
-
-    # Add our URL to the import page
-    test_url = url_for('test_endpoint', _external=True)
-    res = client.post(
-        url_for("import_page"),
-        data={"urls": test_url},
-        follow_redirects=True
-    )
-    assert b"1 Imported" in res.data
-    time.sleep(3)
-
-    res = client.post(
-        url_for("edit_page", uuid="first"),
-        data={"css_filter":  "xpath://*[contains(@class, 'sametext')]", "url": test_url, "tag": "", "headers": "", 'fetch_backend': "html_requests"},
-        follow_redirects=True
-    )
-
-    assert b"Updated watch." in res.data
-    time.sleep(3)
-
-    res = client.get(
-        url_for("preview_page", uuid="first"),
-        follow_redirects=True
-    )
-
-    with open('/tmp/fuck.html', 'wb') as f:
-        f.write(res.data)
-    assert b"Some text thats the same" in res.data #in selector
-    assert b"Some text that will change" not in res.data #not in selector
-
-    client.get(url_for("api_delete", uuid="all"), follow_redirects=True)
+    assert b"is not a valid XPath expression" in res.data

changedetectionio/tests/unit/test-content/after-2.txt

@@ -1,3 +0,0 @@
-After twenty years, as cursed as I may be
-ok
-and insure that I'm one of those computer nerds.

changedetectionio/tests/unit/test-content/after.txt

@@ -2,6 +2,5 @@ After twenty years, as cursed as I may be
 for having learned computerese,
 I continue to examine bits, bytes and words
 xok
-next-x-ok
 and insure that I'm one of those computer nerds.
 and something new

changedetectionio/tests/unit/test_notification_diff.py

@@ -12,19 +12,12 @@ from changedetectionio import diff
 class TestDiffBuilder(unittest.TestCase):
 
     def test_expected_diff_output(self):
-        base_dir = os.path.dirname(__file__)
-        output = diff.render_diff(previous_file=base_dir + "/test-content/before.txt", newest_file=base_dir + "/test-content/after.txt")
+        base_dir=os.path.dirname(__file__)
+        output = diff.render_diff(base_dir+"/test-content/before.txt", base_dir+"/test-content/after.txt")
         output = output.split("\n")
-        self.assertIn('(changed) ok', output)
-        self.assertIn('(into   ) xok', output)
-        self.assertIn('(into   ) next-x-ok', output)
-        self.assertIn('(added  ) and something new', output)
-
-
-        output = diff.render_diff(previous_file=base_dir + "/test-content/before.txt", newest_file=base_dir + "/test-content/after-2.txt")
-        output = output.split("\n")
-        self.assertIn('(removed) for having learned computerese,', output)
-        self.assertIn('(removed) I continue to examine bits, bytes and words', output)
+        self.assertIn("(changed) ok", output)
+        self.assertIn("(-> into) xok", output)
+        self.assertIn("(added) and something new", output)
 
         # @todo test blocks of changed, blocks of added, blocks of removed
 

changedetectionio/tests/util.py

@@ -10,7 +10,6 @@ def set_original_response():
      <p>Which is across multiple lines</p>
      </br>
      So let's see what happens.  </br>
-     <span class="foobar-detection" style='display:none'></span>
      </body>
      </html>
     """
@@ -36,40 +35,24 @@ def set_modified_response():
 
     return None
 
-def set_more_modified_response():
-    test_return_data = """<html>
-    <head><title>modified head title</title></head>
-    <body>
-     Some initial text</br>
-     <p>which has this one new line</p>
-     </br>
-     So let's see what happens.  </br>
-     Ohh yeah awesome<br/>
-     </body>
-     </html>
-    """
-
-    with open("test-datastore/endpoint-content.txt", "w") as f:
-        f.write(test_return_data)
-
-    return None
-
 
 def live_server_setup(live_server):
 
+
     @live_server.app.route('/test-endpoint')
     def test_endpoint():
         ctype = request.args.get('content_type')
-        status_code = request.args.get('status_code')
 
-        try:
-            # Tried using a global var here but didn't seem to work, so reading from a file instead.
-            with open("test-datastore/endpoint-content.txt", "r") as f:
-                resp = make_response(f.read(), status_code)
-                resp.headers['Content-Type'] = ctype if ctype else 'text/html'
-                return resp
-        except FileNotFoundError:
-            return make_response('', status_code)
+        # Tried using a global var here but didn't seem to work, so reading from a file instead.
+        with open("test-datastore/endpoint-content.txt", "r") as f:
+            resp = make_response(f.read())
+            resp.headers['Content-Type'] = ctype if ctype else 'text/html'
+            return resp
+
+    @live_server.app.route('/test-403')
+    def test_endpoint_403_error():
+        resp = make_response('', 403)
+        return resp
 
     # Just return the headers in the request
     @live_server.app.route('/test-headers')
@@ -85,7 +68,6 @@ def live_server_setup(live_server):
     # Just return the body in the request
     @live_server.app.route('/test-body', methods=['POST', 'GET'])
     def test_body():
-        print ("TEST-BODY GOT", request.data, "returning")
         return request.data
 
     # Just return the verb in the request
@@ -102,7 +84,7 @@ def live_server_setup(live_server):
             if data != None:
                 f.write(data)
 
-        print("\n>> Test notification endpoint was hit.\n", data)
+        print("\n>> Test notification endpoint was hit.\n")
         return "Text was set"
 
 

changedetectionio/update_worker.py

@@ -2,7 +2,6 @@ import threading
 import queue
 import time
 
-from changedetectionio import content_fetcher
 # A single update worker
 #
 # Requests for checking on a single site(watch) from a queue of watches
@@ -33,29 +32,28 @@ class update_worker(threading.Thread):
 
             else:
                 self.current_uuid = uuid
+                from changedetectionio import content_fetcher
 
                 if uuid in list(self.datastore.data['watching'].keys()):
 
                     changed_detected = False
                     contents = ""
-                    screenshot = False
                     update_obj= {}
                     now = time.time()
 
                     try:
-                        changed_detected, update_obj, contents, screenshot = update_handler.run(uuid)
+
+                        changed_detected, update_obj, contents = update_handler.run(uuid)
 
                         # Re #342
                         # In Python 3, all strings are sequences of Unicode characters. There is a bytes type that holds raw bytes.
                         # We then convert/.decode('utf-8') for the notification etc
                         if not isinstance(contents, (bytes, bytearray)):
                             raise Exception("Error - returned data from the fetch handler SHOULD be bytes")
+
+
                     except PermissionError as e:
                         self.app.logger.error("File permission error updating", uuid, str(e))
-                    except content_fetcher.ReplyWithContentButNoText as e:
-                        # Totally fine, it's by choice - just continue on, nothing more to care about
-                        # Page had elements/content but no renderable text
-                        pass
                     except content_fetcher.EmptyReply as e:
                         # Some kind of custom to-str handler in the exception handler that does this?
                         err_text = "EmptyReply: Status Code {}".format(e.status_code)
@@ -145,9 +143,6 @@ class update_worker(threading.Thread):
                         # Always record that we atleast tried
                         self.datastore.update_watch(uuid=uuid, update_obj={'fetch_time': round(time.time() - now, 3),
                                                                            'last_checked': round(time.time())})
-                        # Always save the screenshot if it's available
-                        if screenshot:
-                            self.datastore.save_screenshot(watch_uuid=uuid, screenshot=screenshot)
 
                 self.current_uuid = None  # Done
                 self.q.task_done()

docker-compose.yml

@@ -1,9 +1,9 @@
 version: '2'
 services:
-    changedetection:
+    changedetection.io:
       image: ghcr.io/dgtlmoon/changedetection.io
-      container_name: changedetection
-      hostname: changedetection
+      container_name: changedetection.io
+      hostname: changedetection.io
       volumes:
         - changedetection-data:/datastore
 
@@ -17,19 +17,12 @@ services:
   #       Alternative WebDriver/selenium URL, do not use "'s or 's!
   #      - WEBDRIVER_URL=http://browser-chrome:4444/wd/hub
   #
-  #       WebDriver proxy settings webdriver_proxyType, webdriver_ftpProxy, webdriver_noProxy,
-  #                                webdriver_proxyAutoconfigUrl, webdriver_autodetect,
+  #       WebDriver proxy settings webdriver_proxyType, webdriver_ftpProxy, webdriver_httpProxy, webdriver_noProxy,
+  #                                webdriver_proxyAutoconfigUrl, webdriver_sslProxy, webdriver_autodetect,
   #                                webdriver_socksProxy, webdriver_socksUsername, webdriver_socksVersion, webdriver_socksPassword
   #
   #             https://selenium-python.readthedocs.io/api.html#module-selenium.webdriver.common.proxy
   #
-  #       Alternative Playwright URL, do not use "'s or 's!
-  #      - PLAYWRIGHT_DRIVER_URL=ws://playwright-chrome:3000/
-  #
-  #       Playwright proxy settings playwright_proxy_server, playwright_proxy_bypass, playwright_proxy_username, playwright_proxy_password
-  #
-  #             https://playwright.dev/python/docs/api/class-browsertype#browser-type-launch-option-proxy
-  #
   #        Plain requsts - proxy support example.
   #      - HTTP_PROXY=socks5h://10.10.1.10:1080
   #      - HTTPS_PROXY=socks5h://10.10.1.10:1080
@@ -65,13 +58,6 @@ services:
 #            # Workaround to avoid the browser crashing inside a docker container
 #            # See https://github.com/SeleniumHQ/docker-selenium#quick-start
 #            - /dev/shm:/dev/shm
-#        restart: unless-stopped
-
-     # Used for fetching pages via Playwright+Chrome where you need Javascript support.
-
-#    playwright-chrome:
-#        hostname: playwright-chrome
-#        image: browserless/chrome
 #        restart: unless-stopped
 
 volumes:

requirements.txt

@@ -1,9 +1,9 @@
 flask~= 2.0
-flask_wtf
+
 eventlet>=0.31.0
 validators
 timeago ~=1.0
-inscriptis ~= 2.2
+inscriptis ~= 1.2
 feedgen ~= 0.9
 flask-login ~= 0.5
 pytz
@@ -13,11 +13,11 @@ requests[socks] ~= 2.26
 urllib3 > 1.26
 chardet > 2.3.0
 
-wtforms ~= 3.0
+wtforms ~= 2.3.3
 jsonpath-ng ~= 1.5.3
 
 # Notification library
-apprise ~= 0.9.8.3
+apprise ~= 0.9.6
 
 # apprise mqtt https://github.com/dgtlmoon/changedetection.io/issues/315
 paho-mqtt
@@ -34,10 +34,5 @@ lxml
 
 # 3.141 was missing socksVersion, 3.150 was not in pypi, so we try 4.1.0
 selenium ~= 4.1.0
-
-# https://stackoverflow.com/questions/71652965/importerror-cannot-import-name-safe-str-cmp-from-werkzeug-security/71653849#71653849
-# ImportError: cannot import name 'safe_str_cmp' from 'werkzeug.security'
-# need to revisit flask login versions
-werkzeug ~= 2.0.0
-
-# playwright is installed at Dockerfile build time because it's not available on all platforms
+pytest ~=6.2
+pytest-flask ~=1.2

setup.py

@@ -32,11 +32,11 @@ setup(
     long_description_content_type='text/markdown',
     keywords='website change monitor for changes notification change detection '
              'alerts tracking website tracker change alert website and monitoring',
-    entry_points={"console_scripts": ["changedetection.io=changedetectionio.changedetection:main"]},
-    zip_safe=True,
-    scripts=["changedetection.py"],
+    zip_safe=False,
+    entry_points={"console_scripts": ["changedetection.io=changedetection:main"]},
     author='dgtlmoon',
     url='https://changedetection.io',
+    scripts=['changedetection.py'],
     packages=['changedetectionio'],
     include_package_data=True,
     install_requires=install_requires,