WIP - remove column, use a smart 'data-int' to sort on instead

* Update CONTRIBUTING.md

* Add option for tags on import (#377)

* Add option for tags on import and backup

* .add_watch() can accept empty tag
Use https://changedetection.io/CHANGELOG.txt as a nice default page to watch

* plaintext mime type fix - Don't attempt to extract HTML content from plaintext, this will remove lines and break changedetection (#391)

* #323 Adding note about discord:// 2000 char limit (#392)

* Adding note about discord:// 2000 char limit

* Ability to use a generated salted password in deployments as env var SALTED_PASS (#397)

* Ability to use a generated salted password in deployments as env var SALTED_PASS

* Offer instance on Lemonade
Tidy README

* Update README - Tidy up sections

* Update README - Fix docker section

* Update README.md

* /preview format doesnt need <pre> - fixing too many returnlines in content on diff/preview page

* fixed the reference to wiki for rpi section (#402)

* Add notification note - tgram:// bots cant send messages to other bots, so you should specify chat ID of non-bot user.

* Notification error log handler (#403)

* Add a notifications debug/error log interface (Link available under the notification URLs list)

* Refactor tests for notification error log handler (#404)

* Introduce -h option to allow listening not on 0.0.0.0. (#406)

* Fix typo in the startup create-directory command suggestion (#405)

* Use flask url_for() for webdriver chrome icon instead of relative path

* merging latest upstream changes

Co-authored-by: dgtlmoon <dgtlmoon@gmail.com>
Co-authored-by: Tim Loderhose <timlod@users.noreply.github.com>
Co-authored-by: Radu Ursache <3800336+rursache@users.noreply.github.com>
Co-authored-by: Alexander Aleksandrovič Klimov <al2klimov@gmail.com>

README.md

@@ -15,13 +15,19 @@ Open source web page monitoring, notification and change detection.
 <img src="https://raw.githubusercontent.com/dgtlmoon/changedetection.io/master/screenshot.png" style="max-width:100%;" alt="Self-hosted web page change monitoring"  title="Self-hosted web page change monitoring"  />
 
 
-**Get your own instance now on Lemonade!**
+**Get your own private instance now! Let us host it for you!**
 
 [![Deploy to Lemonade](https://lemonade.changedetection.io/static/images/lemonade.svg)](https://lemonade.changedetection.io/start)
 
+
+[_Let us host your own private instance - We accept PayPal and Bitcoin, Support the further development of changedetection.io!_](https://lemonade.changedetection.io/start)
+
+
+
 - Automatic Updates, Automatic Backups, No Heroku "paused application", don't miss a change!
 - Javascript browser included
-- Pay with Bitcoin
+- Unlimited checks and watches!
+
 
 #### Example use cases
 
@@ -99,6 +105,8 @@ See the wiki for more information https://github.com/dgtlmoon/changedetection.io
 ## Filters
 XPath, JSONPath and CSS support comes baked in! You can be as specific as you need, use XPath exported from various XPath element query creation tools.
 
+(We support LXML re:test, re:math and re:replace.)
+
 ## Notifications
 
 ChangeDetection.io supports a massive amount of notifications (including email, office365, custom APIs, etc) when a web-page has a change detected thanks to the <a href="https://github.com/caronc/apprise">apprise</a> library.

changedetectionio/__init__.py

@@ -35,8 +35,9 @@ from flask import (
     url_for,
 )
 from flask_login import login_required
+from changedetectionio import html_tools
 
-__version__ = '0.39.8'
+__version__ = '0.39.9'
 
 datastore = None
 
@@ -322,6 +323,7 @@ def changedetection_app(config=None, datastore_o=None):
     @app.route("/", methods=['GET'])
     @login_required
     def index():
+
         limit_tag = request.args.get('tag')
         pause_uuid = request.args.get('pause')
 
@@ -331,12 +333,17 @@ def changedetection_app(config=None, datastore_o=None):
 
         if pause_uuid:
             try:
-                datastore.data['watching'][pause_uuid]['paused'] ^= True
+                if pause_uuid == 'pause-all' or pause_uuid == 'resume-all':
+                    action = True if pause_uuid == 'pause-all' else False
+                    for watch_uuid, watch in datastore.data['watching'].items():
+                        if datastore.data['watching'][watch_uuid]['tag'] == limit_tag or limit_tag is None :
+                            datastore.data['watching'][watch_uuid]['paused'] = action
+                else :
+                    datastore.data['watching'][pause_uuid]['paused'] ^= True
                 datastore.needs_write = True
-
-                return redirect(url_for('index', tag = limit_tag))
             except KeyError:
-                pass
+                flash("No watch by that UUID found, or error setting paused state.", 'error');
+            return redirect(url_for('index', tag = limit_tag))
 
         # Sort by last_changed and add the uuid which is usually the key..
         sorted_watches = []
@@ -361,13 +368,24 @@ def changedetection_app(config=None, datastore_o=None):
         from changedetectionio import forms
         form = forms.quickWatchForm(request.form)
 
+        # Extra page <title> (n) unviewed
+        extra_title = ""
+        if datastore.data['unviewed_count'] > 0:
+            extra_title = " ({})".format(str(datastore.data['unviewed_count']))
+
+
         output = render_template("watch-overview.html",
                                  form=form,
                                  watches=sorted_watches,
                                  tags=existing_tags,
                                  active_tag=limit_tag,
                                  app_rss_token=datastore.data['settings']['application']['rss_access_token'],
-                                 has_unviewed=datastore.data['has_unviewed'])
+                                 has_unviewed=datastore.data['has_unviewed'],
+                                 # Don't link to hosting when we're on the hosting environment
+                                 hosted_sticky=os.getenv("SALTED_PASS", False) == False,
+                                 guid=datastore.data['app_guid'],
+                                 extra_title=extra_title
+                                )
 
         return output
 
@@ -441,7 +459,7 @@ def changedetection_app(config=None, datastore_o=None):
                 raw_content = file.read()
 
                 handler = fetch_site_status.perform_site_check(datastore=datastore)
-                stripped_content = handler.strip_ignore_text(raw_content,
+                stripped_content = html_tools.strip_ignore_text(raw_content,
                                                              datastore.data['watching'][uuid]['ignore_text'])
 
                 if datastore.data['settings']['application'].get('ignore_whitespace', False):
@@ -546,10 +564,14 @@ def changedetection_app(config=None, datastore_o=None):
                     flash('No notification URLs set, cannot send test.', 'error')
 
             # Diff page [edit] link should go back to diff page
-            if request.args.get("next") and request.args.get("next") == 'diff':
+            if request.args.get("next") and request.args.get("next") == 'diff' and not form.save_and_preview_button.data:
                 return redirect(url_for('diff_history_page', uuid=uuid))
             else:
-                return redirect(url_for('index'))
+                if form.save_and_preview_button.data:
+                    flash('You may need to reload this page to see the new content.')
+                    return redirect(url_for('preview_page', uuid=uuid))
+                else:
+                    return redirect(url_for('index'))
 
         else:
             if request.method == 'POST' and not form.validate():
@@ -656,8 +678,10 @@ def changedetection_app(config=None, datastore_o=None):
         if request.method == 'POST':
             urls = request.values.get('urls').split("\n")
             for url in urls:
-                url = url.strip()
+              
                 url, *tags = url.split(" ")
+                url = url.strip()
+
                 # Flask wtform validators wont work with basic auth, use validators package
                 if len(url) and validators.url(url):
                     new_uuid = datastore.add_watch(url=url.strip(), tag=" ".join(tags))
@@ -685,20 +709,134 @@ def changedetection_app(config=None, datastore_o=None):
     @login_required
     def mark_all_viewed():
 
-        # Save the current newest history as the most recently viewed
-        for watch_uuid, watch in datastore.data['watching'].items():
-            datastore.set_last_viewed(watch_uuid, watch['newest_history_key'])
+        limit_tag = request.args.get('tag')
 
-        flash("Cleared all statuses.")
-        return redirect(url_for('index'))
+        # Save the current newest history as the most recently viewed
+        try: 
+            for watch_uuid, watch in datastore.data['watching'].items():
+                if datastore.data['watching'][watch_uuid]['tag'] == limit_tag or limit_tag is None :
+                    datastore.set_last_viewed(watch_uuid, watch['newest_history_key'])
+            
+            datastore.needs_write = True
+
+            return redirect(url_for('index', tag = limit_tag))
+        except KeyError:
+            pass
+
+    # process selected
+    @app.route("/api/process-selected", methods=["POST"])
+    @login_required
+    def process_selected():
+
+        func = request.form.get('func')
+        limit_tag = request.form.get('tag')
+        uuids = request.form.get('uuids')
+
+        if uuids == '' :
+            flash("No watches selected.")
+            
+        else :
+
+            if func == 'recheck_selected' :
+
+                i = 0
+
+                running_uuids = []
+                for t in running_update_threads:
+                    running_uuids.append(t.current_uuid)
+
+                try :
+                    for uuid in uuids.split(',') :
+                        if uuid not in running_uuids and not datastore.data['watching'][uuid]['paused']:
+                            update_q.put(uuid)
+                            i += 1
+                    
+                except KeyError :
+                    pass
+
+                flash("{} watch{} {} rechecking.".format(i, "" if (i == 1) else "es", "is" if (i == 1) else "are"), "notice")
+                #flash("{} watches are rechecking.".format(i))
+            
+            # Clear selected statuses, so we do not see the 'unviewed' class
+            elif func == 'mark_selected_viewed' :
+
+                try :
+                    for uuid in uuids.split(',') :
+                        datastore.data['watching'][uuid]['last_viewed'] = datastore.data['watching'][uuid]['newest_history_key']
+               
+                except KeyError :
+                    pass
+
+                datastore.needs_write = True
+
+            # Reset selected statuses, so we see the 'unviewed' class
+            # both funcs will contain the uuid list from the processChecked javascript function
+            elif func == 'mark_selected_notviewed' or func == 'mark_all_notviewed' :
+
+                # count within limit_tag and count successes and capture unchanged
+                tagged = 0
+                marked = 0
+                unchanged = []
+                
+                try :
+                    for uuid in uuids.split(',') :
+                        # increment count with limit_tag
+                        tagged += 1
+                        dates = list(datastore.data['watching'][uuid]['history'].keys())
+                        # Convert to int, sort and back to str again
+                        dates = [int(i) for i in dates]
+                        dates.sort(reverse=True)
+                        dates = [str(i) for i in dates]
+
+                        # must be more than 1 history to mark as not viewed
+                        if len(dates) > 1 :
+                            # Save the next earliest history as the most recently viewed
+                            datastore.set_last_viewed(uuid, dates[1])
+                            # increment successes
+                            marked += 1
+                            
+                        else : 
+                            if datastore.data['watching'][uuid]['title'] :
+                                unchanged.append(datastore.data['watching'][uuid]['title'])
+                            else :
+                                unchanged.append(datastore.data['watching'][uuid]['url'])
+
+                except KeyError :
+                        pass
+
+                datastore.needs_write = True
+
+                if marked < tagged :
+                    flash("The following {} not have enough history to be remarked:".format("watch does" if len(unchanged) == 1 else "watches do"), "notice")
+                    for i in range(len(unchanged)):
+                        flash(unchanged[i], "notice")
+                
+            elif func == 'delete_selected' :
+
+                # reachable only after confirmation in javascript processChecked(func, tag) function
+                try :
+                    i = 0
+                    for uuid in uuids.split(',') :
+                        datastore.delete(uuid)
+                        i += 1
+                        
+                except KeyError :
+                    pass
+
+                datastore.needs_write = True
+                
+                flash("{0} {1} deleted.".format(i, "watch was" if (i) == 1 else "watches were")) 
+            
+            else :
+                
+                flash("Invalid parameter received.")
+
+        render_template(url_for('index'), tag = limit_tag)
+        return index()
 
     @app.route("/diff/<string:uuid>", methods=['GET'])
     @login_required
     def diff_history_page(uuid):
-        from changedetectionio import content_fetcher
-
-        newest_version_file_contents = ""
-        previous_version_file_contents = ""
 
         # More for testing, possible to return the first/only
         if uuid == 'first':
@@ -724,28 +862,28 @@ def changedetection_app(config=None, datastore_o=None):
 
         # Save the current newest history as the most recently viewed
         datastore.set_last_viewed(uuid, dates[0])
+        newest_file = watch['history'][dates[0]]
 
-        previous_version = request.args.get('previous_version')
-        if ('content-type' in watch and content_fetcher.supported_binary_type(watch['content-type'])):
-            template = "diff-image.html"
-        else:
-            newest_file = watch['history'][dates[0]]
+        try:
             with open(newest_file, 'r') as f:
                 newest_version_file_contents = f.read()
+        except Exception as e:
+            newest_version_file_contents = "Unable to read {}.\n".format(newest_file)
 
-            try:
-                previous_file = watch['history'][previous_version]
-            except KeyError:
-                # Not present, use a default value, the second one in the sorted list.
-                previous_file = watch['history'][dates[1]]
+        previous_version = request.args.get('previous_version')
+        try:
+            previous_file = watch['history'][previous_version]
+        except KeyError:
+            # Not present, use a default value, the second one in the sorted list.
+            previous_file = watch['history'][dates[1]]
 
+        try:
             with open(previous_file, 'r') as f:
                 previous_version_file_contents = f.read()
+        except Exception as e:
+            previous_version_file_contents = "Unable to read {}.\n".format(previous_file)
 
-            template = "diff.html"
-
-        output = render_template(template,
-                                 watch_a=watch,
+        output = render_template("diff.html", watch_a=watch,
                                  newest=newest_version_file_contents,
                                  previous=previous_version_file_contents,
                                  extra_stylesheets=extra_stylesheets,
@@ -755,14 +893,16 @@ def changedetection_app(config=None, datastore_o=None):
                                  current_previous_version=str(previous_version),
                                  current_diff_url=watch['url'],
                                  extra_title=" - Diff - {}".format(watch['title'] if watch['title'] else watch['url']),
-                                 left_sticky= True )
+                                 left_sticky=True)
 
         return output
 
     @app.route("/preview/<string:uuid>", methods=['GET'])
     @login_required
     def preview_page(uuid):
-        from changedetectionio import content_fetcher
+        content = []
+        ignored_line_numbers = []
+        trigger_line_numbers = []
 
         # More for testing, possible to return the first/only
         if uuid == 'first':
@@ -776,26 +916,52 @@ def changedetection_app(config=None, datastore_o=None):
             flash("No history found for the specified link, bad link?", "error")
             return redirect(url_for('index'))
 
-        newest = list(watch['history'].keys())[-1]
-        fname = watch['history'][newest]
-
-        if ('content-type' in watch and content_fetcher.supported_binary_type(watch['content-type'])):
-            template = "preview-image.html"
-            content = fname
-        else:
-            template = "preview.html"
+        if len(watch['history']):
+            timestamps = sorted(watch['history'].keys(), key=lambda x: int(x))
+            filename = watch['history'][timestamps[-1]]
             try:
-                with open(fname, 'r') as f:
-                    content = f.read()
-            except:
-                content = "Cant read {}".format(fname)
+                with open(filename, 'r') as f:
+                    tmp = f.readlines()
+
+                    # Get what needs to be highlighted
+                    ignore_rules = watch.get('ignore_text', []) + datastore.data['settings']['application']['global_ignore_text']
+
+                    # .readlines will keep the \n, but we will parse it here again, in the future tidy this up
+                    ignored_line_numbers = html_tools.strip_ignore_text(content="".join(tmp),
+                                                                        wordlist=ignore_rules,
+                                                                        mode='line numbers'
+                                                                        )
+
+                    trigger_line_numbers = html_tools.strip_ignore_text(content="".join(tmp),
+                                                                        wordlist=watch['trigger_text'],
+                                                                        mode='line numbers'
+                                                                        )
+                    # Prepare the classes and lines used in the template
+                    i=0
+                    for l in tmp:
+                        classes=[]
+                        i+=1
+                        if i in ignored_line_numbers:
+                            classes.append('ignored')
+                        if i in trigger_line_numbers:
+                            classes.append('triggered')
+                        content.append({'line': l, 'classes': ' '.join(classes)})
+
+
+            except Exception as e:
+                content.append({'line': "File doesnt exist or unable to read file {}".format(filename), 'classes': ''})
+        else:
+            content.append({'line': "No history found", 'classes': ''})
+
 
         output = render_template("preview.html",
                                  content=content,
                                  extra_stylesheets=extra_stylesheets,
+                                 ignored_line_numbers=ignored_line_numbers,
+                                 triggered_line_numbers=trigger_line_numbers,
                                  current_diff_url=watch['url'],
-                                 uuid=uuid,
-                                 watch=watch)
+                                 watch=watch,
+                                 uuid=uuid)
         return output
 
     @app.route("/settings/notification-logs", methods=['GET'])
@@ -807,49 +973,6 @@ def changedetection_app(config=None, datastore_o=None):
 
         return output
 
-
-    # render an image which contains the diff of two images
-    # We always compare the newest against whatever compare_date we are given
-    @app.route("/diff/show-image/<string:uuid>/<string:datestr>")
-    def show_single_image(uuid, datestr):
-
-        from flask import make_response
-        watch = datastore.data['watching'][uuid]
-
-        if datestr == 'None' or datestr is None:
-            datestr = list(watch['history'].keys())[0]
-
-        fname = watch['history'][datestr]
-        with open(fname, 'rb') as f:
-            resp = make_response(f.read())
-            
-        # @todo assumption here about the type, re-encode? detect?
-        resp.headers['Content-Type'] = 'image/jpeg'
-        return resp
-
-    # render an image which contains the diff of two images
-    # We always compare the newest against whatever compare_date we are given
-    @app.route("/diff/image/<string:uuid>/<string:compare_date>")
-    def render_diff_image(uuid, compare_date):
-        from changedetectionio import image_diff
-
-        from flask import make_response
-        watch = datastore.data['watching'][uuid]
-        newest = list(watch['history'].keys())[-1]
-
-        # @todo this is weird
-        if compare_date == 'None' or compare_date is None:
-            compare_date = list(watch['history'].keys())[0]
-
-        new_img = watch['history'][newest]
-        prev_img = watch['history'][compare_date]
-        img = image_diff.render_diff(new_img, prev_img)
-
-        resp = make_response(img)
-        resp.headers['Content-Type'] = 'image/jpeg'
-        return resp
-
-
     @app.route("/api/<string:uuid>/snapshot/current", methods=['GET'])
     @login_required
     def api_snapshot(uuid):
@@ -965,6 +1088,10 @@ def changedetection_app(config=None, datastore_o=None):
 
         if form.validate():
 
+            # get action parameter (add paused button value is 'add', watch button value is 'watch'
+            #action = request.form.get('action')
+            add_paused = request.form.get('add-paused')
+            
             url = request.form.get('url').strip()
             if datastore.url_exists(url):
                 flash('The URL {} already exists'.format(url), "error")
@@ -972,10 +1099,17 @@ def changedetection_app(config=None, datastore_o=None):
 
             # @todo add_watch should throw a custom Exception for validation etc
             new_uuid = datastore.add_watch(url=url, tag=request.form.get('tag').strip())
-            # Straight into the queue.
-            update_q.put(new_uuid)
+            
+            if add_paused :
+                datastore.data['watching'][new_uuid]['paused'] = True
+                datastore.needs_write = True
+                flash("Watch added in a paused state.")
+                                
+            else : # watch now
+                # Straight into the queue. 
+                update_q.put(new_uuid)
 
-            flash("Watch added.")
+                flash("Watch added.")
             return redirect(url_for('index'))
         else:
             flash("Error")
@@ -1038,7 +1172,9 @@ def changedetection_app(config=None, datastore_o=None):
                 if watch_uuid not in running_uuids and not datastore.data['watching'][watch_uuid]['paused']:
                     update_q.put(watch_uuid)
                     i += 1
+
         flash("{} watches are queued for rechecking.".format(i))
+
         return redirect(url_for('index', tag=tag))
 
     # @todo handle ctrl break
@@ -1133,22 +1269,42 @@ def ticker_thread_check_time_launch_checks():
                 running_uuids.append(t.current_uuid)
 
         # Re #232 - Deepcopy the data incase it changes while we're iterating through it all
-        copied_datastore = deepcopy(datastore)
+        while True:
+            try:
+                copied_datastore = deepcopy(datastore)
+            except RuntimeError as e:
+                # RuntimeError: dictionary changed size during iteration
+                time.sleep(0.1)
+            else:
+                break
+
+        # Re #438 - Don't place more watches in the queue to be checked if the queue is already large
+        while update_q.qsize() >= 2000:
+            time.sleep(1)
 
         # Check for watches outside of the time threshold to put in the thread queue.
+        now = time.time()
+        max_system_wide = int(copied_datastore.data['settings']['requests']['minutes_between_check']) * 60
+
         for uuid, watch in copied_datastore.data['watching'].items():
+
+            # No need todo further processing if it's paused
+            if watch['paused']:
+                continue
+
             # If they supplied an individual entry minutes to threshold.
-            if 'minutes_between_check' in watch and watch['minutes_between_check'] is not None:
+            watch_minutes_between_check = watch.get('minutes_between_check', None)
+            if watch_minutes_between_check is not None:
                 # Cast to int just incase
-                max_time = int(watch['minutes_between_check']) * 60
+                max_time = int(watch_minutes_between_check) * 60
             else:
                 # Default system wide.
-                max_time = int(copied_datastore.data['settings']['requests']['minutes_between_check']) * 60
+                max_time = max_system_wide
 
-            threshold = time.time() - max_time
+            threshold = now - max_time
 
-            # Yeah, put it in the queue, it's more than time.
-            if not watch['paused'] and watch['last_checked'] <= threshold:
+            # Yeah, put it in the queue, it's more than time
+            if watch['last_checked'] <= threshold:
                 if not uuid in running_uuids and uuid not in update_q.queue:
                     update_q.put(uuid)
 

changedetectionio/content_fetcher.py

@@ -5,9 +5,8 @@ from selenium import webdriver
 from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
 from selenium.webdriver.common.proxy import Proxy as SeleniumProxy
 from selenium.common.exceptions import WebDriverException
+import urllib3.exceptions
 
-# image/jpeg etc
-supported_binary_types = ['image']
 
 class EmptyReply(Exception):
     def __init__(self, status_code, url):
@@ -52,15 +51,6 @@ class Fetcher():
 #    def return_diff(self, stream_a, stream_b):
 #        return
 
-# Assume we dont support it as binary if its not in our list
-def supported_binary_type(content_type):
-    # Not a binary thing we support? then use text (also used for JSON/XML etc)
-    # @todo - future - use regex for matching
-    if content_type and content_type.lower().strip().split('/')[0] not in (string.lower() for string in supported_binary_types):
-        return False
-
-    return True
-
 def available_fetchers():
         import inspect
         from changedetectionio import content_fetcher
@@ -166,18 +156,15 @@ class html_requests(Fetcher):
                          verify=False)
 
         # https://stackoverflow.com/questions/44203397/python-requests-get-returns-improperly-decoded-text-instead-of-utf-8
-
-        if not supported_binary_type(r.headers.get('Content-Type', '')):
-            content = r.text
-        else:
-            content = r.content
+        # Return bytes here
+        html = r.text
 
         # @todo test this
         # @todo maybe you really want to test zero-byte return pages?
-        if not r or not content or not len(content):
+        if not r or not html or not len(html):
             raise EmptyReply(url=url, status_code=r.status_code)
 
         self.status_code = r.status_code
-        self.content = content
+        self.content = html
         self.headers = r.headers
 

changedetectionio/fetch_site_status.py

@@ -1,5 +1,6 @@
 import time
 from changedetectionio import content_fetcher
+from changedetectionio import html_tools
 import hashlib
 from inscriptis import get_text
 import urllib3
@@ -16,52 +17,15 @@ class perform_site_check():
         super().__init__(*args, **kwargs)
         self.datastore = datastore
 
-    def strip_ignore_text(self, content, list_ignore_text):
-        import re
-        ignore = []
-        ignore_regex = []
-        for k in list_ignore_text:
-
-            # Is it a regex?
-            if k[0] == '/':
-                ignore_regex.append(k.strip(" /"))
-            else:
-                ignore.append(k)
-
-        output = []
-        for line in content.splitlines():
-
-            # Always ignore blank lines in this mode. (when this function gets called)
-            if len(line.strip()):
-                regex_matches = False
-
-                # if any of these match, skip
-                for regex in ignore_regex:
-                    try:
-                        if re.search(regex, line, re.IGNORECASE):
-                            regex_matches = True
-                    except Exception as e:
-                        continue
-
-                if not regex_matches and not any(skip_text in line for skip_text in ignore):
-                    output.append(line.encode('utf8'))
-
-        return "\n".encode('utf8').join(output)
-
-
-
     def run(self, uuid):
         timestamp = int(time.time())  # used for storage etc too
 
         changed_detected = False
         stripped_text_from_html = ""
-        fetched_md5 = ""
-
-        original_content_before_filters = False
 
         watch = self.datastore.data['watching'][uuid]
-
         # Unset any existing notification error
+
         update_obj = {'last_notification_error': False, 'last_error': False}
 
         extra_headers = self.datastore.get_val(uuid, 'headers')
@@ -95,7 +59,6 @@ class perform_site_check():
 
             fetcher = klass()
             fetcher.run(url, timeout, request_headers, request_body, request_method)
-
             # Fetching complete, now filters
             # @todo move to class / maybe inside of fetcher abstract base?
 
@@ -105,39 +68,31 @@ class perform_site_check():
             #  - Do we convert to JSON?
             # https://stackoverflow.com/questions/41817578/basic-method-chaining ?
             # return content().textfilter().jsonextract().checksumcompare() ?
-            update_obj['content-type'] = fetcher.headers.get('Content-Type', '').lower().strip()
-            
-            # Could be 'application/json; charset=utf-8' etc
-            is_json = 'application/json' in update_obj['content-type']
-            is_text_or_html = 'text/' in update_obj['content-type'] # text/plain , text/html etc
-            is_binary = not is_text_or_html and content_fetcher.supported_binary_type(update_obj['content-type'])
-            css_filter_rule = watch['css_filter']
-            has_filter_rule = css_filter_rule and len(css_filter_rule.strip())
 
-            # Auto-detect application/json, make it reformat the JSON to something nice
+            is_json = 'application/json' in fetcher.headers.get('Content-Type', '')
+            is_html = not is_json
+            css_filter_rule = watch['css_filter']
+
+            has_filter_rule = css_filter_rule and len(css_filter_rule.strip())
             if is_json and not has_filter_rule:
                 css_filter_rule = "json:$"
                 has_filter_rule = True
 
-            ##### CONVERT THE INPUT TO TEXT, EXTRACT THE PARTS THAT NEED TO BE FILTERED
+            if has_filter_rule:
+                if 'json:' in css_filter_rule:
+                    stripped_text_from_html = html_tools.extract_json_as_string(content=fetcher.content, jsonpath_filter=css_filter_rule)
+                    is_html = False
 
-            # Dont depend on the content-type header here, maybe it's not present
-            if 'json:' in css_filter_rule:
-                is_json = True
-                rule = css_filter_rule.replace('json:', '')
-                stripped_text_from_html = html_tools.extract_json_as_string(content=fetcher.content,
-                                                                            jsonpath_filter=rule).encode('utf-8')
-                is_text_or_html = False
-                original_content_before_filters = stripped_text_from_html
-
-            if is_text_or_html:
+            if is_html:
                 # CSS Filter, extract the HTML that matches and feed that into the existing inscriptis::get_text
                 html_content = fetcher.content
-                if 'text/plain' in update_obj['content-type']:
-                    stripped_text_from_html = html_content
 
-                # Assume it's HTML if it's not text/plain
-                if not 'text/plain' in update_obj['content-type']:
+                # If not JSON,  and if it's not text/plain..
+                if 'text/plain' in fetcher.headers.get('Content-Type', '').lower():
+                    # Don't run get_text or xpath/css filters on plaintext
+                    stripped_text_from_html = html_content
+                else:
+                    # Then we assume HTML
                     if has_filter_rule:
                         # For HTML/XML we offer xpath as an option, just start a regular xPath "/.."
                         if css_filter_rule[0] == '/':
@@ -145,52 +100,33 @@ class perform_site_check():
                         else:
                             # CSS Filter, extract the HTML that matches and feed that into the existing inscriptis::get_text
                             html_content = html_tools.css_filter(css_filter=css_filter_rule, html_content=fetcher.content)
+
                     # get_text() via inscriptis
                     stripped_text_from_html = get_text(html_content)
 
-                    # Extract title as title
-                    if self.datastore.data['settings']['application']['extract_title_as_title'] or watch['extract_title_as_title']:
-                        if not watch['title'] or not len(watch['title']):
-                            update_obj['title'] = html_tools.extract_element(find='title', html_content=fetcher.content)
-
-                # Re #340 - return the content before the 'ignore text' was applied
-                original_content_before_filters = stripped_text_from_html.encode('utf-8')
 
+            # Re #340 - return the content before the 'ignore text' was applied
+            text_content_before_ignored_filter = stripped_text_from_html.encode('utf-8')
 
             # We rely on the actual text in the html output.. many sites have random script vars etc,
             # in the future we'll implement other mechanisms.
 
             update_obj["last_check_status"] = fetcher.get_last_status_code()
 
-            ######## AFTER FILTERING, STRIP OUT IGNORE TEXT
-            if is_text_or_html:
-                text_to_ignore = watch.get('ignore_text', []) + self.datastore.data['settings']['application'].get('global_ignore_text', [])
-                if len(text_to_ignore):
-                    stripped_text_from_html = self.strip_ignore_text(stripped_text_from_html, text_to_ignore)
-                else:
-                    stripped_text_from_html = stripped_text_from_html.encode('utf8')
+            # If there's text to skip
+            # @todo we could abstract out the get_text() to handle this cleaner
+            text_to_ignore = watch.get('ignore_text', []) + self.datastore.data['settings']['application'].get('global_ignore_text', [])
+            if len(text_to_ignore):
+                stripped_text_from_html = html_tools.strip_ignore_text(stripped_text_from_html, text_to_ignore)
+            else:
+                stripped_text_from_html = stripped_text_from_html.encode('utf8')
 
-
-            ######## CALCULATE CHECKSUM FOR DIFF DETECTION
             # Re #133 - if we should strip whitespaces from triggering the change detected comparison
-            if is_text_or_html:
-                if self.datastore.data['settings']['application'].get('ignore_whitespace', False):
-                    fetched_md5 = hashlib.md5(stripped_text_from_html.translate(None, b'\r\n\t ')).hexdigest()
-                else:
-                    fetched_md5 = hashlib.md5(stripped_text_from_html).hexdigest()
-
-            if is_json:
+            if self.datastore.data['settings']['application'].get('ignore_whitespace', False):
+                fetched_md5 = hashlib.md5(stripped_text_from_html.translate(None, b'\r\n\t ')).hexdigest()
+            else:
                 fetched_md5 = hashlib.md5(stripped_text_from_html).hexdigest()
 
-            # Goal here in the future is to be able to abstract out different content type checks into their own class
-
-            if is_binary:
-                # @todo - use some actual image hash here where possible, audio hash, etc etc
-                m = hashlib.sha256()
-                m.update(fetcher.content)
-                fetched_md5 = m.hexdigest()
-                original_content_before_filters = fetcher.content
-
             # On the first run of a site, watch['previous_md5'] will be an empty string, set it the current one.
             if not len(watch['previous_md5']):
                 watch['previous_md5'] = fetched_md5
@@ -198,24 +134,16 @@ class perform_site_check():
 
             blocked_by_not_found_trigger_text = False
 
-            # Trigger text can apply to JSON parsed documents too
-            if is_text_or_html or is_json:
-                if len(watch['trigger_text']):
-                    blocked_by_not_found_trigger_text = True
-                    for line in watch['trigger_text']:
-                        # Because JSON wont serialize a re.compile object
-                        if line[0] == '/' and line[-1] == '/':
-                            regex = re.compile(line.strip('/'), re.IGNORECASE)
-                            # Found it? so we don't wait for it anymore
-                            r = re.search(regex, str(stripped_text_from_html))
-                            if r:
-                                blocked_by_not_found_trigger_text = False
-                                break
+            if len(watch['trigger_text']):
+                # Yeah, lets block first until something matches
+                blocked_by_not_found_trigger_text = True
+                # Filter and trigger works the same, so reuse it
+                result = html_tools.strip_ignore_text(content=str(stripped_text_from_html),
+                                                      wordlist=watch['trigger_text'],
+                                                      mode="line numbers")
+                if result:
+                    blocked_by_not_found_trigger_text = False
 
-                        elif line.lower() in str(stripped_text_from_html).lower():
-                            # We found it don't wait for it.
-                            blocked_by_not_found_trigger_text = False
-                            break
 
             if not blocked_by_not_found_trigger_text and watch['previous_md5'] != fetched_md5:
                 changed_detected = True
@@ -223,5 +151,11 @@ class perform_site_check():
                 update_obj["last_changed"] = timestamp
 
 
-        # original_content_before_filters is returned for saving the data to disk
-        return changed_detected, update_obj, original_content_before_filters
+            # Extract title as title
+            if is_html:
+                if self.datastore.data['settings']['application']['extract_title_as_title'] or watch['extract_title_as_title']:
+                    if not watch['title'] or not len(watch['title']):
+                        update_obj['title'] = html_tools.extract_element(find='title', html_content=fetcher.content)
+
+
+        return changed_detected, update_obj, text_content_before_ignored_filter

changedetectionio/forms.py

@@ -1,6 +1,7 @@
 from wtforms import Form, SelectField, RadioField, BooleanField, StringField, PasswordField, validators, IntegerField, fields, TextAreaField, \
     Field
-from wtforms import widgets
+
+from wtforms import widgets, SubmitField
 from wtforms.validators import ValidationError
 from wtforms.fields import html5
 from changedetectionio import content_fetcher
@@ -290,6 +291,9 @@ class watchForm(commonSettingsForm):
     method = SelectField('Request Method', choices=valid_method, default=default_method)
     trigger_text = StringListField('Trigger/wait for text', [validators.Optional(), ValidateListRegex()])
 
+    save_button = SubmitField('Save', render_kw={"class": "pure-button pure-button-primary"})
+    save_and_preview_button = SubmitField('Save & Preview', render_kw={"class": "pure-button pure-button-primary"})
+
     def validate(self, **kwargs):
         if not super().validate():
             return False

changedetectionio/html_tools.py

@@ -1,7 +1,7 @@
 import json
 from bs4 import BeautifulSoup
 from jsonpath_ng.ext import parse
-
+import re
 
 class JSONNotFound(ValueError):
     def __init__(self, msg):
@@ -25,7 +25,7 @@ def xpath_filter(xpath_filter, html_content):
     tree = html.fromstring(html_content)
     html_block = ""
 
-    for item in tree.xpath(xpath_filter.strip()):
+    for item in tree.xpath(xpath_filter.strip(), namespaces={'re':'http://exslt.org/regular-expressions'}):
         html_block+= etree.tostring(item, pretty_print=True).decode('utf-8')+"<br/>"
 
     return html_block
@@ -105,3 +105,50 @@ def extract_json_as_string(content, jsonpath_filter):
         return ''
 
     return stripped_text_from_html
+
+# Mode     - "content" return the content without the matches (default)
+#          - "line numbers" return a list of line numbers that match (int list)
+#
+# wordlist - list of regex's (str) or words (str)
+def strip_ignore_text(content, wordlist, mode="content"):
+    ignore = []
+    ignore_regex = []
+
+    # @todo check this runs case insensitive
+    for k in wordlist:
+
+        # Is it a regex?
+        if k[0] == '/':
+            ignore_regex.append(k.strip(" /"))
+        else:
+            ignore.append(k)
+
+    i = 0
+    output = []
+    ignored_line_numbers = []
+    for line in content.splitlines():
+        i += 1
+        # Always ignore blank lines in this mode. (when this function gets called)
+        if len(line.strip()):
+            regex_matches = False
+
+            # if any of these match, skip
+            for regex in ignore_regex:
+                try:
+                    if re.search(regex, line, re.IGNORECASE):
+                        regex_matches = True
+                except Exception as e:
+                    continue
+
+            if not regex_matches and not any(skip_text.lower() in line.lower() for skip_text in ignore):
+                output.append(line.encode('utf8'))
+            else:
+                ignored_line_numbers.append(i)
+
+
+
+    # Used for finding out what to highlight
+    if mode == "line numbers":
+        return ignored_line_numbers
+
+    return "\n".encode('utf8').join(output)

changedetectionio/image_diff.py

@@ -1,41 +0,0 @@
-# import the necessary packages
-from skimage.metrics import structural_similarity as compare_ssim
-import argparse
-import imutils
-import cv2
-
-# From https://www.pyimagesearch.com/2017/06/19/image-difference-with-opencv-and-python/
-def render_diff(fpath_imageA, fpath_imageB):
-
-	imageA = cv2.imread(fpath_imageA)
-	imageB = cv2.imread(fpath_imageB)
-
-	# convert the images to grayscale
-	grayA = cv2.cvtColor(imageA, cv2.COLOR_BGR2GRAY)
-	grayB = cv2.cvtColor(imageB, cv2.COLOR_BGR2GRAY)
-
-	# compute the Structural Similarity Index (SSIM) between the two
-	# images, ensuring that the difference image is returned
-	(score, diff) = compare_ssim(grayA, grayB, full=True)
-	diff = (diff * 255).astype("uint8")
-	print("SSIM: {}".format(score))
-
-	# threshold the difference image, followed by finding contours to
-	# obtain the regions of the two input images that differ
-	thresh = cv2.threshold(diff, 0, 255,
-		cv2.THRESH_BINARY_INV | cv2.THRESH_OTSU)[1]
-	cnts = cv2.findContours(thresh.copy(), cv2.RETR_EXTERNAL,
-		cv2.CHAIN_APPROX_SIMPLE)
-	cnts = imutils.grab_contours(cnts)
-
-	# loop over the contours
-	for c in cnts:
-		# compute the bounding box of the contour and then draw the
-		# bounding box on both input images to represent where the two
-		# images differ
-		(x, y, w, h) = cv2.boundingRect(c)
-		cv2.rectangle(imageA, (x, y), (x + w, y + h), (0, 0, 255), 2)
-		cv2.rectangle(imageB, (x, y), (x + w, y + h), (0, 0, 255), 2)
-
-	#return cv2.imencode('.jpg', imageB)[1].tobytes()
-	return cv2.imencode('.jpg', imageA)[1].tobytes()

changedetectionio/static/images/notviewed.svg

@@ -0,0 +1 @@
+<svg id="svg2" xmlns="http://www.w3.org/2000/svg" width="9.5" height="15" viewBox="0 0 9.5 15"><path id="path3740" d="M2.2,0A2.41,2.41,0,0,0,0,1.5V2.8H2.2V0Z" transform="translate(0 0)" style="fill:#0078e7"/><path id="rect3728" d="M.3,1.7l-.2,1H2.2V.2A2.76,2.76,0,0,0,.3,1.7Z" transform="translate(0 0)" style="fill:#0078e7"/><path id="path3655" d="M9.5,2.6h0L2.3,0,2,.2,9.2,2.8v.1" transform="translate(0 0)" style="fill:#0078e7"/><path id="path3645" d="M9.2,2.8V13.3l.2-.3V2.5" transform="translate(0 0)" style="fill:#0078e7"/><path id="rect3517" d="M2,.2,9.2,2.8V13.4L2,10.8Z" transform="translate(0 0)" style="fill:#0078e7"/><path id="path3657" d="M2.1.2,9.2,2.8" transform="translate(0 0)" style="fill:#0078e7;stroke:#0078e7;stroke-miterlimit:4.660399913787842;stroke-width:0.10000000149011612px"/><path id="path3684" d="M.5,9.6.6,2.4.4,2.3S1.2,1.1,2,1L8.8,3.4v.1" transform="translate(0 0)" style="fill:#fff;fill-rule:evenodd"/><path id="path3679" d="M8.9,3.4,8.7,13,7.3,14.3.5,11.9V9.5" transform="translate(0 0)" style="fill:#fff;fill-rule:evenodd"/><path id="path3669" d="M7.5,4.2h0L.3,1.6,0,1.9,7.2,4.5v.1" transform="translate(0 0)" style="fill:#0078e7"/><path id="path3671" d="M7.2,4.5V15l.2-.3V4.2" transform="translate(0 0)" style="fill:#0078e7"/><path id="path3673" d="M0,1.9,7.2,4.5V15L0,12.4Z" transform="translate(0 0)" style="fill:#0078e7"/><path id="path3675" d="M.1,1.9,7.2,4.5" transform="translate(0 0)" style="fill:#0078e7;stroke:#0078e7;stroke-miterlimit:4.660399913787842;stroke-width:0.10000000149011612px"/></svg>

changedetectionio/static/images/pause.svg

@@ -1,84 +1 @@
-<?xml version="1.0" encoding="UTF-8" standalone="no"?>
-<svg
-   xmlns:dc="http://purl.org/dc/elements/1.1/"
-   xmlns:cc="http://creativecommons.org/ns#"
-   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
-   xmlns:svg="http://www.w3.org/2000/svg"
-   xmlns="http://www.w3.org/2000/svg"
-   version="1.1"
-   id="Capa_1"
-   x="0px"
-   y="0px"
-   viewBox="0 0 15 14.998326"
-   xml:space="preserve"
-   width="15"
-   height="14.998326"><metadata
-   id="metadata39"><rdf:RDF><cc:Work
-       rdf:about=""><dc:format>image/svg+xml</dc:format><dc:type
-         rdf:resource="http://purl.org/dc/dcmitype/StillImage" /><dc:title></dc:title></cc:Work></rdf:RDF></metadata><defs
-   id="defs37" />
-<path
-   id="path2"
-   style="fill:#1b98f8;fill-opacity:1;stroke-width:0.0292893"
-   d="M 7.4975161,6.5052867e-4 C 4.549072,-0.04028702 1.7055675,1.8548221 0.58868606,4.5801341 -0.57739762,7.2574642 0.02596981,10.583326 2.069916,12.671949 4.0364753,14.788409 7.2763651,15.56067 9.989207,14.57284 12.801145,13.617602 14.87442,10.855325 14.985833,7.8845744 15.172496,4.9966544 13.49856,2.1100704 10.911002,0.8209349 9.8598067,0.28073592 8.6791261,-0.00114855 7.4975161,6.5052867e-4 Z M 6.5602569,10.251923 c -0.00509,0.507593 -0.5693885,0.488472 -0.9352002,0.468629 -0.3399386,0.0018 -0.8402048,0.07132 -0.9297965,-0.374189 -0.015842,-1.8973128 -0.015872,-3.7979649 0,-5.6952784 0.1334405,-0.5224315 0.7416869,-0.3424086 1.1377562,-0.374189 0.3969969,-0.084515 0.8245634,0.1963256 0.7272405,0.6382917 0,1.7789118 0,3.5578239 0,5.3367357 z m 3.7490371,0 c -0.0051,0.507593 -0.5693888,0.488472 -0.9352005,0.468629 -0.3399386,0.0018 -0.8402048,0.07132 -0.9297965,-0.374189 -0.015842,-1.8973128 -0.015872,-3.7979649 0,-5.6952784 0.1334405,-0.5224315 0.7416869,-0.3424086 1.1377562,-0.374189 0.3969969,-0.084515 0.8245638,0.1963256 0.7272408,0.6382917 0,1.7789118 0,3.5578239 0,5.3367357 z" />
-<g
-   id="g4"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g6"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g8"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g10"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g12"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g14"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g16"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g18"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g20"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g22"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g24"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g26"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g28"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g30"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-<g
-   id="g32"
-   transform="translate(-0.01903604,0.02221043)">
-</g>
-</svg>
+<svg id="Capa_1" data-name="Capa 1" xmlns="http://www.w3.org/2000/svg" width="15.03" height="15.03" viewBox="0 0 15.03 15.03"><path id="path2" d="M7.5,0A7.56,7.56,0,0,0,.6,4.6a7.37,7.37,0,0,0,1.5,8.1A7.52,7.52,0,0,0,10,14.6,7.53,7.53,0,0,0,10.9.8,7.73,7.73,0,0,0,7.5,0ZM6.6,10.3c0,.5-.6.5-.9.5s-.8.1-.9-.4V4.7c.1-.5.7-.3,1.1-.4a.53.53,0,0,1,.7.6Zm3.7,0c0,.5-.6.5-.9.5s-.8.1-.9-.4V4.7c.1-.5.7-.3,1.1-.4a.53.53,0,0,1,.7.6Z" transform="translate(0.01 0)" style="fill:#0078e7"/></svg>

changedetectionio/static/images/play.svg

@@ -0,0 +1 @@
+<svg id="Capa_1" data-name="Capa 1" xmlns="http://www.w3.org/2000/svg" width="15" height="15" viewBox="0 0 15 15"><path id="path2" d="M7.5,0A7.62,7.62,0,0,0,0,7.5,7.55,7.55,0,0,0,7.5,15,7.55,7.55,0,0,0,15,7.5,7.6,7.6,0,0,0,10.9.8,9.42,9.42,0,0,0,7.5,0Z" transform="translate(0 0)" style="fill:#0078e7"/><polygon points="11.4 8 5.8 4.8 5.8 11.3 11.4 8" style="fill:#fff"/></svg>

changedetectionio/static/images/search.svg

@@ -0,0 +1 @@
+<svg id="Layer_1" data-name="Layer 1" xmlns="http://www.w3.org/2000/svg" width="32" height="32" viewBox="0 0 32 32"><circle cx="16" cy="16" r="16" style="fill:#0078e7"/><path d="M24,26.85l-4.93-5a8.53,8.53,0,0,1-4.71,1.41,8.63,8.63,0,1,1,7.32-4.12l5,5c.26.26,0,.9-.49,1.44l-.74.74C24.86,26.88,24.23,27.11,24,26.85Zm-3.9-12.23a5.75,5.75,0,1,0-5.74,5.79A5.76,5.76,0,0,0,20.07,14.62Z" style="fill:#fff"/></svg>

changedetectionio/static/images/sortable.svg

@@ -0,0 +1 @@
+<svg id="Layer_1" data-name="Layer 1" xmlns="http://www.w3.org/2000/svg" width="10.93" height="14.99" viewBox="0 0 10.93 14.99"><path d="M5.5,1,9.6,6.1H1.3Zm0,13L1.3,8.9H9.5Z" transform="translate(0.02 -0.01)" style="fill:#0078e7;stroke:#0078e7;stroke-miterlimit:10;stroke-width:1.25px"/></svg>

changedetectionio/static/images/viewed.svg

@@ -0,0 +1 @@
+<svg id="svg2" xmlns="http://www.w3.org/2000/svg" width="16.33" height="11.64" viewBox="0 0 16.33 11.64"><path id="path2416" d="M14.2,6.5l1.4,3.1-6.2.8s0,.2-.4.1a.65.65,0,0,1-.6-.6L1.8,11.1.6,4.2H.9v.1H.8L2,10.9,8.4,9.8V10h.4s0,.4.5.4l.1-.1,6-.8-1.2-3Z" transform="translate(-0.01 -0.04)" style="fill:#0078e7;stroke:#007ec0;fill-rule:evenodd"/><path id="path2400" d="M1,4.3H.9L2,10.9,8.4,9.8s-1.7-.9-6.1,1L1,4.3Z" transform="translate(-0.01 -0.04)" style="fill:#1187c3;fill-rule:evenodd"/><path id="path2402" d="M1,4.1V3.6h.1V3.2l.2-.3h.1V2.5L2.8,8.9l-.1.3v.2l-.1.1-.2.1-.1,1.1L1,4.1Z" transform="translate(-0.01 -0.04)" style="fill:#fff;stroke:#0078e7;stroke-width:0.5px;fill-rule:evenodd"/><path id="path2388" d="M2.3,10.8l.1-1.2.3-.3V9.2l.2-.3s5.5-.2,5.9.8l-.4.1s-1.7-.9-6.1,1Z" transform="translate(-0.01 -0.04)" style="fill:#fff;stroke:#0078e7;stroke-width:0.5px;fill-rule:evenodd"/><path id="path2394" d="M2.2,2.5H1.5L2.9,8.9s5.3-.2,5.9.8c0,0-.1-1.1-5.4-1.6L2.2,2.5Z" transform="translate(-0.01 -0.04)" style="fill:#fff;stroke:#0078e7;stroke-width:0.5px;fill-rule:evenodd"/><path id="path2398" d="M2,1.3,3.4,8s5,.5,5.4,1.7l-2-6.3c0-.1,0-1.1-4.8-2.1Z" transform="translate(-0.01 -0.04)" style="fill:#fff;stroke:#0078e7;stroke-width:0.5px;fill-rule:evenodd"/><path id="path2403" d="M6.9,3.3l5-2.9,2.5,5.9L8.8,9.7,6.9,3.3Z" transform="translate(-0.01 -0.04)" style="fill:#fff;stroke:#0078e7;stroke-width:0.5px;fill-rule:evenodd"/><path id="path2411" d="M8.5,10V9.8l.3-.1s.2.6.6.6v.1a.49.49,0,0,1-.5-.5l-.4.1Z" transform="translate(-0.01 -0.04)" style="fill:#1187c3;stroke:#007ec0;stroke-width:0.5px;fill-rule:evenodd"/><path id="path2415" d="M8.8,9.7s.2.6.6.5l6-.8-.3-.6h-.3c.1,0-5,.2-6,.9Z" transform="translate(-0.01 -0.04)" style="fill:#fff;stroke:#0078e7;stroke-width:0.5px;fill-rule:evenodd"/><path id="path2404" d="M15.1,8.9l-1-2.3L8.8,9.7v.1s.5-.6,6-.9V9Z" transform="translate(-0.01 -0.04)" style="fill:#fff;stroke:#0078e7;stroke-width:0.5px;fill-rule:evenodd"/></svg>

changedetectionio/static/js/tabs.js

@@ -1,10 +1,13 @@
 // Rewrite this is a plugin.. is all this JS really 'worth it?'
 
-
+// display correct label and messages for minutes or seconds 
+document.addEventListener("DOMContentLoaded", function(event) {
+	use_seconds_change();
+});
 window.addEventListener('hashchange', function() {
   var tabs = document.getElementsByClassName('active');
   while (tabs[0]) {
-    tabs[0].classList.remove('active')
+    tabs[0].classList.remove('active');
   }
   set_active_tab();
 }, false);
@@ -37,7 +40,7 @@ function focus_error_tab() {
     var tabs = document.querySelectorAll('.tabs li a'),i;
     for (i = 0; i < tabs.length; ++i) {
       var tab_name=tabs[i].hash.replace('#','');
-      var pane_errors=document.querySelectorAll('#'+tab_name+' .error')
+      var pane_errors=document.querySelectorAll('#'+tab_name+' .error');
       if (pane_errors.length) {
         document.location.hash = '#'+tab_name;
         return true;
@@ -45,7 +48,3 @@ function focus_error_tab() {
     }
     return false;
 }
-
-
-
-

changedetectionio/static/js/tbltools.js

@@ -0,0 +1,406 @@
+// table tools
+
+// must be a var for keyChar and keyCode use
+var CONSTANT_ESCAPE_KEY = 27;
+var CONSTANT_S_KEY = 83;
+var CONSTANT_s_KEY = 115;
+
+// globals
+var loading;
+var sort_column; // new window or tab is always last_changed
+var sort_order;  // new window or tab is always descending
+
+// restore scroll position on submit/reload 
+document.addEventListener("DOMContentLoaded", function(event) {
+	load_functions();
+	var scrollpos = sessionStorage.getItem('scrollpos');
+	if (scrollpos) window.scrollTo(0, scrollpos);
+});
+
+// mobile scroll position retention 
+if (/Android|webOS|iPhone|iPad|iPod|BlackBerry|IEMobile|Opera Mini/i.test(navigator.userAgent)) {
+	document.addEventListener("visibilitychange", function() {
+		storeScrollAndSearch();
+	});
+} else {
+	// non-mobile scroll position retention 
+	window.onbeforeunload = function(e) {
+		storeScrollAndSearch();
+	};
+}
+function storeScrollAndSearch() {
+	sessionStorage.setItem('scrollpos', window.pageYOffset);
+	sessionStorage.setItem('searchtxt', document.getElementById("txtInput").value);
+}
+
+// mobile positioning of checkbox-controls grid popup
+document.addEventListener("touchstart", touchStartHandler, false);
+var touchXY = {};
+function touchStartHandler(event) {
+	var touches = event.changedTouches;
+	touchXY = {
+		clientX : touches[0].clientX,
+		clientY : touches[0].clientY
+	};
+}
+
+// (ctl)-alt-s search hotkey
+document.onkeyup = function(e) {
+	var e = e || window.event; // for IE to cover IEs window event-object
+	if (e.altKey && (e.which == CONSTANT_S_KEY || e.which == CONSTANT_s_KEY)) {
+		document.getElementById("txtInput").focus();
+		return false;
+	}
+}
+
+// new window or tab loading
+function load_functions() {
+	// loading
+	loading = true;
+	// retain checked items
+	checkChange();
+	// retrieve saved sorting
+	getSort();
+	// sort if not default
+	sortTable(sort_column);
+	// search
+	if (isSessionStorageSupported()) {
+		// retrieve search
+		if (sessionStorage.getItem("searchtxt") != null) {
+			document.getElementById("txtInput").value = sessionStorage.getItem("searchtxt");
+			tblSearch(this);
+		}
+	}
+}
+
+// sorting
+function sortTable(n) {
+	var table, rows, switching, i, x, y, shouldSwitch, dir, switchcount = 0,
+		sortimgs, sortableimgs;
+	table = document.getElementById("watch-table");
+	switching = true;
+	//Set the sorting direction, either default 9, 1 or saved
+	if (loading) {
+		getSort();
+		dir = (sort_order == 0) ? "asc" : "desc";
+		loading = false;
+	} else {
+		dir = "asc";
+	}
+	/*Make a loop that will continue until
+	no switching has been done:*/
+	while (switching) {
+		//start by saying: no switching is done:
+		switching = false;
+		rows = table.rows;
+		/*Loop through all table rows (except the
+		first, which contains table headers):*/
+		for (i = 1; i < (rows.length - 1); i++) {
+			//start by saying there should be no switching:
+			shouldSwitch = false;
+			/*Get the two elements you want to compare,
+			one from current row and one from the next:*/
+			x = rows[i].getElementsByTagName("TD")[n];
+			y = rows[i + 1].getElementsByTagName("TD")[n];
+			x = x.innerHTML.toLowerCase();
+			y = y.innerHTML.toLowerCase();
+			if (!isNaN(x)) { // handle numeric columns
+				x = parseFloat(x);
+				y = parseFloat(y);
+			}
+			if (n == 1) { // handle play/pause column
+				x = rows[i].getElementsByTagName("TD")[n].getElementsByTagName("img")[0].src;
+				y = rows[i + 1].getElementsByTagName("TD")[n].getElementsByTagName("img")[0].src;
+			}
+			/*check if the two rows should switch place,
+			based on the direction, asc or desc:*/
+			if (dir == "asc") {
+				if (x > y) {
+					//if so, mark as a switch and break the loop:
+					shouldSwitch = true;
+					break;
+				}
+			} else if (dir == "desc") {
+				if (x < y) {
+					//if so, mark as a switch and break the loop:
+					shouldSwitch = true;
+					break;
+				}
+			}
+		}
+		if (shouldSwitch) {
+			/*If a switch has been marked, make the switch
+			and mark that a switch has been done:*/
+			rows[i].parentNode.insertBefore(rows[i + 1], rows[i]);
+			switching = true;
+			//Each time a switch is done, increase this count by 1:
+			switchcount++;
+		} else {
+			/*If no switching has been done AND the direction is "asc",
+			set the direction to "desc" and run the while loop again.*/
+			if (switchcount == 0 && dir == "asc") {
+				dir = "desc";
+				switching = true;
+			}
+		}
+	}
+	// hide all asc/desc sort arrows
+	sortimgs = document.querySelectorAll('[id^="sort-"]');
+	for (i = 0; i < sortimgs.length; i++) {
+		sortimgs[i].style.display = "none";
+	}
+	// show current asc/desc sort arrow and set sort_order var
+	if (dir == "asc") {
+		document.getElementById("sort-" + n + "a").style.display = "";
+	} else {
+		document.getElementById("sort-" + n + "d").style.display = "";
+	}
+	// show all sortable indicators
+	sortableimgs = document.querySelectorAll('[id^="sortable-"]');
+	for (i = 0; i < sortableimgs.length; i++) {
+		sortableimgs[i].style.display = "";
+	}
+	// hide sortable indicator from current column
+	document.getElementById("sortable-" + n).style.display = "none";
+	// save sorting
+	sessionStorage.setItem("sort_column", n);
+	sessionStorage.setItem("sort_order", (dir == "asc") ? 0 : 1);
+	// restripe rows
+	restripe();
+}
+
+// check/uncheck all checkboxes
+function checkAll(e) {
+	var elemID = event.srcElement.id;
+	if (!elemID) return;
+	var elem = document.getElementById(elemID);
+	var rect = elem.getBoundingClientRect();
+	var offsetLeft = document.documentElement.scrollLeft + rect.left;
+	var offsetTop;
+	if (/Android|webOS|iPhone|iPad|iPod|BlackBerry|IEMobile|Opera Mini/i.test(navigator.userAgent)) {
+		offsetTop = touchXY.clientY; // + rect.top;
+	}
+	else {
+		offsetTop = document.documentElement.scrollTop + rect.top;
+	}
+	var i;
+	var checkboxes = document.getElementsByName('check');
+	var checkboxFunctions = document.getElementById('checkbox-functions');
+	if (e.checked) {
+		for (i = 0; i < checkboxes.length; i++) {
+			checkboxes[i].checked = true;
+		}
+		checkboxFunctions.style.display = "";
+		checkboxFunctions.style.left = offsetLeft + 30 + "px";
+		checkboxFunctions.style.top = offsetTop + "px";
+	} else {
+		for (i = 0; i < checkboxes.length; i++) {
+			checkboxes[i].checked = false;
+		}
+		checkboxFunctions.style.display = "none";
+	}
+}
+
+// show/hide checkbox controls grid popup and check/uncheck checkall checkbox if all other checkboxes are checked/unchecked
+function checkChange(e) {
+	var elemID = event.srcElement.id;
+	if (!elemID) return;
+	var elem = document.getElementById(elemID);
+	var rect = elem.getBoundingClientRect();
+	var offsetLeft = document.documentElement.scrollLeft + rect.left;
+	var offsetTop;
+	if (/Android|webOS|iPhone|iPad|iPod|BlackBerry|IEMobile|Opera Mini/i.test(navigator.userAgent)) {
+		offsetTop = touchXY.clientY; // + rect.top;
+	}
+	else {
+		offsetTop = document.documentElement.scrollTop + rect.top;
+	}
+	var i;
+	var totalCheckbox = document.querySelectorAll('input[name="check"]').length;
+	var totalChecked = document.querySelectorAll('input[name="check"]:checked').length;
+	var checkboxFunctions = document.getElementById('checkbox-functions');
+	if(totalCheckbox == totalChecked) {
+		document.getElementsByName("showhide")[0].checked=true;
+	}
+	else {
+		document.getElementsByName("showhide")[0].checked=false;
+	}
+	if (totalChecked > 0) {
+		checkboxFunctions.style.display = "";
+		checkboxFunctions.style.left = offsetLeft + 30 + "px";
+		if ( offsetTop > ( window.innerHeight - checkboxFunctions.offsetHeight) ) {
+			checkboxFunctions.style.top = (window.innerHeight - checkboxFunctions.offsetHeight) + "px";
+		}
+		else {
+			checkboxFunctions.style.top = offsetTop + "px";
+		}
+	} else {
+		checkboxFunctions.style.display = "none";
+	}
+}
+
+// search watches in Title column
+function tblSearch(evt) {
+	var code = evt.charCode || evt.keyCode;
+	if (code == CONSTANT_ESCAPE_KEY) {
+		document.getElementById("txtInput").value = '';
+	}
+	var input, filter, table, tr, td, i, txtValue;
+	input = document.getElementById("txtInput");
+	filter = input.value.toUpperCase();
+	table = document.getElementById("watch-table");
+	tr = table.getElementsByTagName("tr");
+	for (i = 1; i < tr.length; i++) { // skip header
+		td = tr[i].getElementsByTagName("td")[3]; // col 3 is the hidden title/url column
+		if (td) {
+			txtValue = td.textContent || td.innerText;
+			if (txtValue.toUpperCase().indexOf(filter) > -1) {
+				tr[i].style.display = "";
+			} else {
+				tr[i].style.display = "none";
+			}
+		}
+	}
+	// restripe rows
+	restripe();
+	if (code == CONSTANT_ESCAPE_KEY) {
+		document.getElementById("watch-table-wrapper").focus();
+	}
+}
+
+// restripe after searching or sorting
+function restripe() {
+	var i, visrows = [];
+	var table = document.getElementById("watch-table");
+	var rows = table.getElementsByTagName("tr");
+
+	for (i = 1; i < rows.length; i++) { // skip header
+		if (rows[i].style.display !== "none") {
+			visrows.push(rows[i]);
+		}
+	}
+	for (i = 0; i < visrows.length; i++) {
+		var row = visrows[i];
+		if (i % 2 == 0) {
+			row.classList.remove('pure-table-odd');
+			row.classList.add('pure-table-even');
+		} else {
+			row.classList.remove('pure-table-even');
+			row.classList.add('pure-table-odd');
+		}
+		var cells = row.getElementsByTagName("td");
+		for (var j = 0; j < cells.length; j++) {
+			if (i % 2 == 0) {
+				cells[j].style.background = "#f2f2f2";
+			} else {
+				cells[j].style.background = "#ffffff";
+			}
+		}
+		// uncomment to renumber rows ascending:    var cells = row.getElementsByTagName("td");
+		// uncomment to renumber rows ascending:    cells[0].innerText = i+1;
+	}
+}
+
+// get checked or all uuids
+function getChecked(items) {
+	var i, checkedArr, uuids = '';
+
+	if (items === undefined) {
+		checkedArr = document.querySelectorAll('input[name="check"]:checked');
+	} else {
+		checkedArr = document.querySelectorAll('input[name="check"]');
+	}
+	if (checkedArr.length > 0) {
+		let output = [];
+		for (i = 0; i < checkedArr.length; i++) {
+			output.push(checkedArr[i].parentNode.parentNode.getAttribute("id"));
+		}
+		for (i = 0; i < checkedArr.length; i++) {
+			if (i < checkedArr.length - 1) {
+				uuids += output[i] + ",";
+			} else {
+				uuids += output[i];
+			}
+		}
+	}
+	return uuids;
+}
+
+// process selected watches 
+function processChecked(func, tag) {
+	var uuids, result;
+
+	if (func == 'mark_all_notviewed') {
+		uuids = getChecked('all');
+	} else {
+		uuids = getChecked();
+	}
+	// confirm if deleting
+	if (func == 'delete_selected' && uuids.length > 0) {
+		result = confirm('Deletions cannot be undone.\n\nAre you sure you want to continue?');
+		if (result == false) {
+			return;
+		}
+	}
+	// href locations
+	var currenturl = window.location;
+	var posturl = location.protocol + '//' + location.host + '/api/process-selected';
+	// posting vars
+	const XHR = new XMLHttpRequest(),
+		FD = new FormData();
+	// fill form data
+	FD.append('func', func);
+	FD.append('tag', tag);
+	FD.append('uuids', uuids);
+	// success
+	XHR.addEventListener('load', function(event) {
+		window.location = currenturl;
+	});
+	// error
+	XHR.addEventListener(' error', function(event) {
+		alert('Error posting request.');
+	});
+	// set up request
+	XHR.open('POST', posturl);
+	// send
+	XHR.send(FD);
+}
+
+function clearSearch() {
+	document.getElementById("txtInput").value = '';
+	tblSearch(CONSTANT_ESCAPE_KEY);
+}
+
+function isSessionStorageSupported() {
+	var storage = window.sessionStorage;
+	try {
+		storage.setItem('test', 'test');
+		storage.removeItem('test');
+		return true;
+	} catch (e) {
+		return false;
+	}
+}
+
+function getSort() {
+	if (isSessionStorageSupported()) {
+		// retrieve sort settings if set
+		if (sessionStorage.getItem("sort_column") != null) {
+			sort_column = sessionStorage.getItem("sort_column");
+			sort_order = sessionStorage.getItem("sort_order");
+		} else {
+			sort_column = 7; // last changed
+			sort_order = 1; // desc
+			//alert("Your web browser does not support retaining sorting and page position.");
+		}
+	}
+}
+
+function closeGridDisplay() {
+	document.getElementsByName("showhide")[0].checked = false;
+	var checkboxes = document.getElementsByName('check');
+	for (i = 0; i < checkboxes.length; i++) {
+		checkboxes[i].checked = false;
+	}
+	document.getElementById("checkbox-functions").style.display = "none";
+}

changedetectionio/static/styles/diff.css

@@ -54,3 +54,19 @@ ins {
   body {
     height: 99%;
     /* Hide scroll bar in Firefox */ } }
+
+td#diff-col div {
+  text-align: justify;
+  white-space: pre-wrap; }
+
+.ignored {
+  background-color: #ccc;
+  /*  border: #0d91fa 1px solid; */
+  opacity: 0.7; }
+
+.triggered {
+  background-color: #1b98f8; }
+
+/* ignored and triggered? make it obvious error */
+.ignored.triggered {
+  background-color: #ff0000; }

changedetectionio/static/styles/diff.scss

@@ -66,3 +66,23 @@ ins {
 		height: 99%; /* Hide scroll bar in Firefox */
 	}
 }
+
+td#diff-col div {
+    text-align: justify;
+    white-space: pre-wrap;
+}
+
+.ignored {
+    background-color: #ccc;
+   /*  border: #0d91fa 1px solid; */
+    opacity: 0.7;
+}
+
+.triggered {
+    background-color: #1b98f8;
+}
+
+/* ignored and triggered? make it obvious error */
+.ignored.triggered {
+  background-color: #ff0000;
+}

changedetectionio/static/styles/package.json

@@ -4,8 +4,7 @@
   "description": "",
   "main": "index.js",
   "scripts": {
-    "build": "node-sass styles.scss diff.scss -o .",
-    "watch": "node-sass --watch styles.scss diff.scss -o ."
+    "build": "node-sass styles.scss -o .;node-sass diff.scss -o ."
   },
   "author": "",
   "license": "ISC",

changedetectionio/static/styles/styles.scss

@@ -317,11 +317,9 @@ footer {
     */
 }
 
-
-
 .sticky-tab {
   position: absolute;
-  top: 80px;
+  top: 60px;
   font-size: 8px;
   background: #fff;
   padding: 10px;
@@ -331,6 +329,11 @@ footer {
   &#right-sticky {
     right: 0px;
   }
+  &#hosted-sticky {
+    right: 0px;
+    top: 100px;
+    font-weight: bold;
+  }
 }
 
 #new-version-text a {
@@ -567,3 +570,8 @@ $form-edge-padding: 20px;
   }
 }
 
+ul {
+    padding-left: 1em;
+    padding-top: 0px;
+    margin-top: 4px;
+}

changedetectionio/store.py

@@ -184,10 +184,6 @@ class ChangeDetectionStore:
 
     def update_watch(self, uuid, update_obj):
 
-        # Skip if 'paused' state
-        if self.__data['watching'][uuid]['paused']:
-            return
-
         with self.lock:
 
             # In python 3.9 we have the |= dict operator, but that still will lose data on nested structures...
@@ -205,6 +201,7 @@ class ChangeDetectionStore:
     @property
     def data(self):
         has_unviewed = False
+        unviewed_count = 0
         for uuid, v in self.__data['watching'].items():
             self.__data['watching'][uuid]['newest_history_key'] = self.get_newest_history_key(uuid)
             if int(v['newest_history_key']) <= int(v['last_viewed']):
@@ -213,6 +210,7 @@ class ChangeDetectionStore:
             else:
                 self.__data['watching'][uuid]['viewed'] = False
                 has_unviewed = True
+                unviewed_count += 1
 
             # #106 - Be sure this is None on empty string, False, None, etc
             # Default var for fetch_backend
@@ -225,6 +223,7 @@ class ChangeDetectionStore:
           self.__data['settings']['application']['base_url'] = env_base_url.strip('" ')
 
         self.__data['has_unviewed'] = has_unviewed
+        self.__data['unviewed_count'] = unviewed_count
 
         return self.__data
 
@@ -372,9 +371,7 @@ class ChangeDetectionStore:
         if not os.path.isdir(output_path):
             mkdir(output_path)
 
-        suffix = "stripped.txt"
-
-        fname = "{}/{}.{}".format(output_path, uuid.uuid4(), suffix)
+        fname = "{}/{}.stripped.txt".format(output_path, uuid.uuid4())
         with open(fname, 'wb') as f:
             f.write(contents)
             f.close()
@@ -400,13 +397,10 @@ class ChangeDetectionStore:
                 # system was out of memory, out of RAM etc
                 with open(self.json_store_path+".tmp", 'w') as json_file:
                     json.dump(data, json_file, indent=4)
-
+                os.rename(self.json_store_path+".tmp", self.json_store_path)
             except Exception as e:
                 logging.error("Error writing JSON!! (Main JSON file save was skipped) : %s", str(e))
 
-            else:
-                os.rename(self.json_store_path+".tmp", self.json_store_path)
-
             self.needs_write = False
 
     # Thread runner, this helps with thread/write issues when there are many operations that want to update the JSON

changedetectionio/templates/_helpers.jinja

@@ -25,3 +25,6 @@
 {% endmacro %}
 
 
+{% macro render_button(field) %}
+  {{ field(**kwargs)|safe }}
+{% endmacro %}

changedetectionio/templates/base.html

@@ -34,6 +34,9 @@
         <ul class="pure-menu-list"  id="top-right-menu">
         {% if current_user.is_authenticated or not has_password %}
             {% if not current_diff_url %}
+            <li class="pure-menu-item">
+                <span class="search-box"><input type="text" id="txtInput" onkeyup="tblSearch(event)" onmouseup="clearSearch(event)" placeholder="Title..." /></span>
+            </li>
             <li class="pure-menu-item">
                 <a href="{{ url_for('get_backup')}}" class="pure-menu-link">BACKUP</a>
             </li>
@@ -68,7 +71,7 @@
         </ul>
     </div>
 </div>
-
+{% if hosted_sticky %}<div class="sticky-tab" id="hosted-sticky"><a href="https://lemonade.changedetection.io/start?ref={{guid}}">Let us host your instance!</a></div>{% endif %}
 {% if left_sticky %}<div class="sticky-tab" id="left-sticky"><a href="{{url_for('preview_page', uuid=uuid)}}">Show current snapshot</a></div> {% endif %}
 {% if right_sticky %}<div class="sticky-tab" id="right-sticky">{{ right_sticky }}</div> {% endif %}
 <section class="content">

changedetectionio/templates/diff-image.html

@@ -1,59 +0,0 @@
-{% extends 'base.html' %}
-
-{% block content %}
-
-<div id="settings">
-    <h1>Differences</h1>
-    <form class="pure-form " action="" method="GET">
-        <fieldset>
-            {% if versions|length >= 1 %}
-            <label for="diff-version">Compare newest (<span id="current-v-date"></span>) with</label>
-            <select id="diff-version" name="previous_version">
-                {% for version in versions %}
-                <option value="{{version}}" {% if version== current_previous_version %} selected="" {% endif %}>
-                    {{version}}
-                </option>
-                {% endfor %}
-            </select>
-            <button type="submit" class="pure-button pure-button-primary">Go</button>
-            {% endif %}
-        </fieldset>
-    </form>
-
-</div>
-
-<div id="diff-ui">
-  <img style="max-width: 100%" src="{{ url_for('render_diff_image', uuid=uuid, compare_date=current_previous_version) }}" />
-
-    <div>
-        <span style="width: 50%">
-            <img style="max-width: 100%" src="{{ url_for('show_single_image', uuid=uuid, datestr=newest_version_timestamp) }}" />
-        </span>
-        <span style="width: 50%">
-            <img style="max-width: 100%" src="{{ url_for('show_single_image', uuid=uuid, datestr=current_previous_version) }}" />
-        </span>
-    </div>
-</div>
-
-
-<script type="text/javascript" src="{{url_for('static_content', group='js', filename='diff.js')}}"></script>
-
-<script defer="">
-window.onload = function() {
-    /* Set current version date as local time in the browser also */
-    var current_v = document.getElementById("current-v-date");
-    var dateObject = new Date({{ newest_version_timestamp }}*1000);
-    current_v.innerHTML=dateObject.toLocaleString();
-
-    /* Convert what is options from UTC time.time() to local browser time */
-    var diffList=document.getElementById("diff-version");
-    if (typeof(diffList) != 'undefined' && diffList != null) {
-        for (var option of diffList.options) {
-          var dateObject = new Date(option.value*1000);
-          option.label=dateObject.toLocaleString();
-        }
-    }
-}
-</script>
-
-{% endblock %}

changedetectionio/templates/diff.html

@@ -36,6 +36,7 @@
     <a onclick="next_diff();">Jump</a>
 </div>
 <div id="diff-ui">
+    <div class="tip">Pro-tip: Use <strong>show current snapshot</strong> tab to visualise what will be ignored.</div>
     <table>
         <tbody>
         <tr>

changedetectionio/templates/edit.html

@@ -1,6 +1,7 @@
 {% extends 'base.html' %}
 {% block content %}
 {% from '_helpers.jinja' import render_field %}
+{% from '_helpers.jinja' import render_button %}
 {% from '_common_fields.jinja' import render_common_settings_form %}
 <script type="text/javascript" src="{{url_for('static_content', group='js', filename='tabs.js')}}" defer></script>
 
@@ -88,6 +89,18 @@ User-Agent: wonderbra 1.0") }}
 
             <div class="tab-pane-inner" id="filters-and-triggers">
                 <fieldset>
+                        <div class="pure-control-group">
+                            <strong>Pro-tips:</strong><br/>
+                            <ul>
+                                <li>
+                                    Use the preview page to see your filters and triggers highlighted.
+                                </li>
+                                <li>
+                                    Some sites use JavaScript to create the content, for this you should <a href="https://github.com/dgtlmoon/changedetection.io/wiki/Fetching-pages-with-WebDriver">use the Chrome/WebDriver Fetcher</a>
+                                </li>
+                            </ul>
+                    </div>
+
                     <div class="pure-control-group">
                         {{ render_field(form.css_filter, placeholder=".class-name or #some-id, or other CSS selector rule.",
                         class="m-d") }}
@@ -114,6 +127,7 @@ User-Agent: wonderbra 1.0") }}
                             <li>Each line processed separately, any line matching will be ignored (removed before creating the checksum)</li>
                             <li>Regular Expression support, wrap the line in forward slash <b>/regex/</b></li>
                             <li>Changing this will affect the comparison checksum which may trigger an alert</li>
+                            <li>Use the preview/show current tab to see ignores</li>
                         </ul>
                 </span>
 
@@ -138,7 +152,8 @@ User-Agent: wonderbra 1.0") }}
             <div id="actions">
                 <div class="pure-control-group">
 
-                    <button type="submit" class="pure-button pure-button-primary">Save</button>
+                      {{ render_button(form.save_button) }} {{ render_button(form.save_and_preview_button) }}
+
                     <a href="{{url_for('api_delete', uuid=uuid)}}"
                        class="pure-button button-small button-error ">Delete</a>
                     <a href="{{url_for('api_clone', uuid=uuid)}}"

changedetectionio/templates/preview-image.html

@@ -1,13 +0,0 @@
-{% extends 'base.html' %}
-
-{% block content %}
-
-<div id="settings">
-    <h1>Current</h1>
-</div>
-
-<div id="diff-ui">
-    image goes here
-</div>
-
-{% endblock %}

changedetectionio/templates/preview.html

@@ -3,19 +3,21 @@
 {% block content %}
 
 <div id="settings">
-    <h1>Current</h1>
+    <h1>Current - {{watch.last_checked|format_timestamp_timeago}}</h1>
 </div>
 
 <div id="diff-ui">
+    <span class="ignored">Grey lines are ignored</span> <span class="triggered">Blue lines are triggers</span>
     <table>
         <tbody>
         <tr>
             <td id="diff-col">
-                <span id="result">{{content}}</span>
+                    {% for row in content %}
+                    <div class="{{row.classes}}">{{row.line}}</div>
+                    {% endfor %}
             </td>
         </tr>
         </tbody>
     </table>
 </div>
-
 {% endblock %}

changedetectionio/templates/settings.html

@@ -95,6 +95,7 @@
                             <li>Each line processed separately, any line matching will be ignored (removed before creating the checksum)</li>
                             <li>Regular Expression support, wrap the line in forward slash <b>/regex/</b></li>
                             <li>Changing this will affect the comparison checksum which may trigger an alert</li>
+                            <li>Use the preview/show current tab to see ignores</li>
                         </ul>
                      </span>
                     </fieldset>

changedetectionio/templates/watch-overview.html

@@ -2,103 +2,146 @@
 {% block content %}
 {% from '_helpers.jinja' import render_simple_field %}
 
+<script src="{{url_for('static_content', group='js', filename='tbltools.js')}}"></script>
+
 <div class="box">
+	<form class="pure-form" action="{{ url_for('api_watch_add') }}" method="POST" id="new-watch-form">
+		<fieldset>
+			<legend>Add a new change detection watch</legend>
+				{{ render_simple_field(form.url, placeholder="https://...", required=true) }}
+				{{ render_simple_field(form.tag, value=active_tag if active_tag else '', placeholder="tag") }}
+			<div id="watch-actions"><button type="submit" class="pure-button pure-button-primary" name="action" value="watch">Watch</button>&nbsp;
+			<span id="add-paused"><label><input type="checkbox" name="add-paused">&nbsp;Add Paused</label></span></div>
+	   </fieldset>
+		<!-- add extra stuff, like do a http POST and send headers -->
+		<!-- user/pass r = requests.get('https://api.github.com/user', auth=('user', 'pass')) -->
+	</form>
+    <div id="watch-table-wrapper" tabindex="-1">
+		<div id="categories">
+			<a href="{{url_for('index')}}" class="pure-button button-tag {{'active' if not active_tag }}">All</a>
+			{% for tag in tags %}
+				{% if tag != "" %}
+					<a href="{{url_for('index', tag=tag) }}" class="pure-button button-tag {{'active' if active_tag == tag }}">{{ tag }}</a>
+				{% endif %}
+			{% endfor %}
+		</div>
+			
+		<div id="controls-top">
+			<div id="checkbox-functions" style="display: none;">
+				<ul id="post-list-buttons-top-grid">
+					<li id="grid-item-1">
+						<a href="javascript:processChecked('recheck_selected', '{{ active_tag }}');" class="pure-button button-tag " title="Recheck Selected{%if active_tag%} in &quot;{{active_tag}}&quot;{%endif%}">Recheck&nbsp;</a>
+					</li>
+					<li id="grid-item-2">
+						<a href="javascript:processChecked('mark_selected_notviewed', '{{ active_tag }}');" class="pure-button button-tag " title="Mark Selected as Unviewed{%if active_tag%} in &quot;{{active_tag}}&quot;{%endif%}">Unviewed</a>
+					</li>
+					<li id="grid-item-3">
+						<a href="javascript:processChecked('mark_selected_viewed', '{{ active_tag }}');" class="pure-button button-tag " title="Mark Selected as Viewed{%if active_tag%} in &quot;{{active_tag}}&quot;{%endif%}">Viewed&nbsp;&nbsp;</a>
+					</li>
+					<li id="grid-item-4">
+						<a href="javascript:processChecked('delete_selected', '{{ active_tag }}');" class="pure-button button-tag danger " title="Delete Selected{%if active_tag%} in &quot;{{active_tag}}&quot;{%endif%}">Delete&nbsp;&nbsp;</a>
+					</li>
+					<li id="grid-item-5">
+					   <a href="javascript:closeGridDisplay();" class="pure-button button-tag ">Cancel&nbsp;&nbsp;</a>
+					</li>
+				</ul>
+			</div>
+		</div>
 
-    <form class="pure-form" action="{{ url_for('api_watch_add') }}" method="POST" id="new-watch-form">
-        <fieldset>
-            <legend>Add a new change detection watch</legend>
-                {{ render_simple_field(form.url, placeholder="https://...", required=true) }}
-                {{ render_simple_field(form.tag, value=active_tag if active_tag else '', placeholder="tag") }}
-            <button type="submit" class="pure-button pure-button-primary">Watch</button>
-        </fieldset>
-        <!-- add extra stuff, like do a http POST and send headers -->
-        <!-- user/pass r = requests.get('https://api.github.com/user', auth=('user', 'pass')) -->
-    </form>
-    <div>
-        <a href="{{url_for('index')}}" class="pure-button button-tag {{'active' if not active_tag }}">All</a>
-        {% for tag in tags %}
-            {% if tag != "" %}
-                <a href="{{url_for('index', tag=tag) }}" class="pure-button button-tag {{'active' if active_tag == tag }}">{{ tag }}</a>
-            {% endif %}
-        {% endfor %}
-    </div>
+		<div>
+			<table class="pure-table pure-table-striped watch-table" id="watch-table">
+				<thead>
+				<tr id="header">
+					<th class="inline chkbox-header"><input id="chk-all" type="checkbox" name="showhide" onchange="checkAll(this)" title="Check/Uncheck All">&nbsp;&nbsp;#</th>
+					<th class="pause-resume-header" onclick="sortTable(1)">
+						<span class="clickable"
+																				 title="Sort by Pause/Resume"><a
+							href="{{url_for('index', pause='pause-all', tag=active_tag)}}"><img
+							src="{{url_for('static_content', group='images', filename='pause.svg')}}" alt="Pause"
+							title="Pause All {%if active_tag%}in &quot;{{active_tag}}&quot; {%endif%}"/></a>&nbsp;<a
+							href="{{url_for('index', pause='resume-all', tag=active_tag)}}"><img
+							src="{{url_for('static_content', group='images', filename='play.svg')}}" alt="Resume"
+							title="Resume All {%if active_tag%}in &quot;{{active_tag}}&quot; {%endif%}"/></a>&nbsp;&nbsp;<span
+							id="sortable-1"><img
+							src="{{url_for('static_content', group='images', filename='sortable.svg')}}"
+							alt="sort"/></span><span class="sortarrow"><span id="sort-1a"
+																			 style="display:none;">&#9650;</span><span
+							id="sort-1d" style="display:none;">&#9660;</span></span></span></th>
+					<th onclick="sortTable(3)"><span class="clickable" title="Sort by Title">Title&nbsp;&nbsp;<span id="sortable-3"><img src="{{url_for('static_content', group='images', filename='sortable.svg')}}" alt="sort" /></span><span class="sortarrow"><span id="sort-3a" style="display:none;">&#9650;</span><span id="sort-3d" style="display:none;">&#9660;</span></span></span></th>
+					<th onclick="sortTable(5)"><span class="clickable" title="Sort by Last Checked">Checked&nbsp;&nbsp;<span id="sortable-5"><img src="{{url_for('static_content', group='images', filename='sortable.svg')}}" alt="sort" /></span><span class="sortarrow"><span id="sort-5a" style="display:none;">&#9650;</span><span id="sort-5d" style="display:none;">&#9660;</span></span></span></th>
+					<th onclick="sortTable(7)"><span class="clickable" title="Sort by Last Changed">Changed&nbsp;&nbsp;<span id="sortable-7" style="display:none;"><img src="{{url_for('static_content', group='images', filename='sortable.svg')}}" alt="sort" /></span><span class="sortarrow"><span id="sort-7a" style="display:none;">&#9650;</span><span id="sort-7d">&#9660;</span></span></span></th>
+					<th>Actions</th>
+				</tr>
+				</thead>
+				<tbody>
+					{% for watch in watches %}
+						<tr id="{{ watch.uuid }}"
+							class="{{ loop.cycle('pure-table-odd', 'pure-table-even') }}
+																								  
+							{% if watch.last_error is defined and watch.last_error != False %}error{% endif %}
+							{% if watch.last_notification_error is defined and watch.last_notification_error != False %}error{% endif %}
+							{% if watch.paused is defined and watch.paused != False %}paused{% endif %}
+							{% if watch.newest_history_key| int > watch.last_viewed| int %}unviewed{% endif %}">
+							<td class="inline chkbox"><input id="chk-{{ loop.index }}" type="checkbox" name="check" onchange="checkChange(this);">&nbsp;&nbsp;{{ loop.index }}</td>
+							<td class="inline pause-resume">
+							{% if watch.paused %}
+								<a href="{{url_for('index', pause=watch.uuid, tag=active_tag)}}"><img src="{{url_for('static_content', group='images', filename='play.svg')}}" alt="Resume" title="Resume"/></a>
+							{% else %}
+								<a href="{{url_for('index', pause=watch.uuid, tag=active_tag)}}"><img src="{{url_for('static_content', group='images', filename='pause.svg')}}" alt="Pause" title="Pause"/></a>
+							{% endif %}
+							</td>
+							<td class="title-col inline">{{watch.title if watch.title is not none and watch.title|length > 0 else watch.url}}
+								<a class="external inline-hover-img" target="_blank" rel="noopener" href="{{ watch.url }}"></a>
+								{%if watch.fetch_backend == "html_webdriver" %}<img style="height: 1em; display:inline-block;" src="{{url_for('static_content', group='images', filename='Google-Chrome-icon.png')}}" />{% endif %}
 
-    <div id="watch-table-wrapper">
-        <table class="pure-table pure-table-striped watch-table">
-            <thead>
-            <tr>
-                <th>#</th>
-                <th></th>
-                <th></th>
-                <th>Last Checked</th>
-                <th>Last Changed</th>
-                <th></th>
-            </tr>
-            </thead>
-            <tbody>
-
-
-            {% for watch in watches %}
-            <tr id="{{ watch.uuid }}"
-                class="{{ loop.cycle('pure-table-odd', 'pure-table-even') }}
-                {% if watch.last_error is defined and watch.last_error != False %}error{% endif %}
-                {% if watch.last_notification_error is defined and watch.last_notification_error != False %}error{% endif %}
-                {% if watch.paused is defined and watch.paused != False %}paused{% endif %}
-                {% if watch.newest_history_key| int > watch.last_viewed| int %}unviewed{% endif %}">
-                <td class="inline">{{ loop.index }}</td>
-                <td class="inline paused-state state-{{watch.paused}}"><a href="{{url_for('index', pause=watch.uuid, tag=active_tag)}}"><img src="{{url_for('static_content', group='images', filename='pause.svg')}}" alt="Pause" title="Pause"/></a></td>
-
-                <td class="title-col inline">{{watch.title if watch.title is not none and watch.title|length > 0 else watch.url}}
-                    <a class="external" target="_blank" rel="noopener" href="{{ watch.url }}"></a>
-                    {%if watch.fetch_backend == "html_webdriver" %}<img style="height: 1em; display:inline-block;" src="{{url_for('static_content', group='images', filename='Google-Chrome-icon.png')}}" />{% endif %}
-
-                    {% if watch.last_error is defined and watch.last_error != False %}
-                    <div class="fetch-error">{{ watch.last_error }}</div>
-                    {% endif %}
-                    {% if watch.last_notification_error is defined and watch.last_notification_error != False %}
-                    <div class="fetch-error notification-error">{{ watch.last_notification_error }}</div>
-                    {% endif %}
-                    {% if not active_tag %}
-                    <span class="watch-tag-list">{{ watch.tag}}</span>
-                    {% endif %}
-                </td>
-                <td class="last-checked">{{watch|format_last_checked_time}}</td>
-                <td class="last-changed">{% if watch.history|length >= 2 and watch.last_changed %}
-                    {{watch.last_changed|format_timestamp_timeago}}
-                    {% else %}
-                    Not yet
-                    {% endif %}
-                </td>
-                <td>
-                    <a href="{{ url_for('api_watch_checknow', uuid=watch.uuid, tag=request.args.get('tag')) }}"
-                       class="pure-button button-small pure-button-primary">Recheck</a>
-                    <a href="{{ url_for('edit_page', uuid=watch.uuid)}}" class="pure-button button-small pure-button-primary">Edit</a>
-                    {% if watch.history|length >= 2 %}
-                    <a href="{{ url_for('diff_history_page', uuid=watch.uuid) }}" target="{{watch.uuid}}" class="pure-button button-small pure-button-primary">Diff</a>
-                    {% else %}
-                        {% if watch.history|length == 1 %}
-                            <a href="{{ url_for('preview_page', uuid=watch.uuid)}}" target="{{watch.uuid}}" class="pure-button button-small pure-button-primary">Preview</a>
-                        {% endif %}
-                    {% endif %}
-                </td>
-            </tr>
-            {% endfor %}
-            </tbody>
-        </table>
-        <ul id="post-list-buttons">
-            {% if has_unviewed %}
-            <li>
-                <a href="{{url_for('mark_all_viewed', tag=request.args.get('tag')) }}" class="pure-button button-tag ">Mark all viewed</a>
-            </li>
-            {% endif %}
-            <li>
-               <a href="{{ url_for('api_watch_checknow', tag=active_tag) }}" class="pure-button button-tag ">Recheck
-                all {% if active_tag%}in "{{active_tag}}"{%endif%}</a>
-            </li>
-            <li>
-                <a href="{{ url_for('rss', tag=active_tag , token=app_rss_token)}}"><img alt="RSS Feed" id="feed-icon" src="{{url_for('static_content', group='images', filename='Generic_Feed-icon.svg')}}" height="15"></a>
-            </li>
-        </ul>
-    </div>
+								{% if watch.last_error is defined and watch.last_error != False %}
+								<div class="fetch-error">{{ watch.last_error }}</div>
+								{% endif %}
+								{% if watch.last_notification_error is defined and watch.last_notification_error != False %}
+								<div class="fetch-error notification-error">{{ watch.last_notification_error }}</div>
+								{% endif %}
+								{% if not active_tag %}
+								<span class="watch-tag-list">{{ watch.tag}}</span>
+								{% endif %}
+							</td>
+							<td class="last-checked">{{watch|format_last_checked_time}}</td>
+							<td class="last-changed">{% if watch.history|length >= 2 and watch.last_changed %}
+								{{watch.last_changed|format_timestamp_timeago}}
+								{% else %}
+								Not yet
+								{% endif %}
+							</td>
+							<td>
+								<a href="{{ url_for('api_watch_checknow', uuid=watch.uuid, tag=request.args.get('tag')) }}"
+								   class="pure-button button-small pure-button-primary">Recheck</a>
+								<a href="{{ url_for('edit_page', uuid=watch.uuid)}}" class="pure-button button-small pure-button-primary">Edit</a>
+								{% if watch.history|length >= 2 %}
+								<a href="{{ url_for('diff_history_page', uuid=watch.uuid) }}" target="{{watch.uuid}}" class="pure-button button-small pure-button-primary">Diff</a>
+								{% else %}
+									{% if watch.history|length == 1 %}
+										<a href="{{ url_for('preview_page', uuid=watch.uuid)}}" target="{{watch.uuid}}" class="pure-button button-small pure-button-primary">Preview</a>
+									{% endif %}
+								{% endif %}
+							</td>
+						</tr>
+					{% endfor %}
+				</tbody>
+			</table>
+			<ul id="post-list-buttons">
+				{% if has_unviewed %}
+				<li>
+					<a href="{{url_for('mark_all_viewed', tag=request.args.get('tag')) }}" class="pure-button button-tag ">Mark all viewed</a>
+				</li>
+				{% endif %}
+				<li>
+				   <a href="{{ url_for('api_watch_checknow', tag=active_tag) }}" class="pure-button button-tag ">Recheck
+					all {% if active_tag%}in "{{active_tag}}"{%endif%}</a>
+				</li>
+				<li>
+					<a href="{{ url_for('rss', tag=active_tag , token=app_rss_token)}}"><img alt="RSS Feed" id="feed-icon" src="{{url_for('static_content', group='images', filename='Generic_Feed-icon.svg')}}" height="15"></a>
+				</li>
+			</ul>
+		</div>
+	</div>
 </div>
 {% endblock %}

changedetectionio/tests/test_api.py

@@ -14,7 +14,6 @@ def set_response_data(test_return_data):
 
 
 def test_snapshot_api_detects_change(client, live_server):
-
     test_return_data = "Some initial text"
 
     test_return_data_modified = "Some NEW nice initial text"
@@ -27,7 +26,7 @@ def test_snapshot_api_detects_change(client, live_server):
     time.sleep(1)
 
     # Add our URL to the import page
-    test_url = url_for('test_endpoint', _external=True)
+    test_url = url_for('test_endpoint', content_type="text/plain", _external=True)
     res = client.post(
         url_for("import_page"),
         data={"urls": test_url},

changedetectionio/tests/test_backend.py

@@ -7,6 +7,13 @@ from . util import set_original_response, set_modified_response, live_server_set
 
 sleep_time_for_fetch_thread = 3
 
+# Basic test to check inscriptus is not adding return line chars, basically works etc
+def test_inscriptus():
+    from inscriptis import get_text
+    html_content="<html><body>test!<br/>ok man</body></html>"
+    stripped_text_from_html = get_text(html_content)
+    assert stripped_text_from_html == 'test!\nok man'
+
 
 def test_check_basic_change_detection_functionality(client, live_server):
     set_original_response()
@@ -100,14 +107,6 @@ def test_check_basic_change_detection_functionality(client, live_server):
     # It should have picked up the <title>
     assert b'head title' in res.data
 
-
-    # be sure the HTML converter worked
-    res = client.get(url_for("preview_page", uuid="first"))
-    assert b'<html>' not in res.data
-
-    res = client.get(url_for("preview_page", uuid="first"))
-    assert b'Some initial text' in res.data
-
     #
     # Cleanup everything
     res = client.get(url_for("api_delete", uuid="all"), follow_redirects=True)

changedetectionio/tests/test_binary_fetch.py

@@ -1,56 +0,0 @@
-#!/usr/bin/python3
-
-import time
-import secrets
-from flask import url_for
-from . util import live_server_setup
-
-
-def test_binary_file_change(client, live_server):
-    with open("test-datastore/test.bin", "wb") as f:
-        f.write(secrets.token_bytes())
-
-    live_server_setup(live_server)
-
-    sleep_time_for_fetch_thread = 3
-
-    # Give the endpoint time to spin up
-    time.sleep(1)
-
-    # Add our URL to the import page
-    test_url = url_for('test_binaryfile_endpoint', _external=True)
-    res = client.post(
-        url_for("import_page"),
-        data={"urls": test_url},
-        follow_redirects=True
-    )
-    assert b"1 Imported" in res.data
-
-    # Trigger a check
-    client.get(url_for("api_watch_checknow"), follow_redirects=True)
-
-    # Give the thread time to pick it up
-    time.sleep(sleep_time_for_fetch_thread)
-
-    # Trigger a check
-    client.get(url_for("api_watch_checknow"), follow_redirects=True)
-
-    # It should report nothing found (no new 'unviewed' class)
-    res = client.get(url_for("index"))
-    assert b'unviewed' not in res.data
-    assert b'/test-binary-endpoint' in res.data
-
-    #  Make a change
-    with open("test-datastore/test.bin", "wb") as f:
-        f.write(secrets.token_bytes())
-
-
-    # Trigger a check
-    client.get(url_for("api_watch_checknow"), follow_redirects=True)
-
-    # Give the thread time to pick it up
-    time.sleep(sleep_time_for_fetch_thread)
-
-    # It should report nothing found (no new 'unviewed' class)
-    res = client.get(url_for("index"))
-    assert b'unviewed' in res.data

changedetectionio/tests/test_ignore_regex_text.py

@@ -3,6 +3,7 @@
 import time
 from flask import url_for
 from . util import live_server_setup
+from changedetectionio import html_tools
 
 def test_setup(live_server):
     live_server_setup(live_server)
@@ -23,7 +24,7 @@ def test_strip_regex_text_func():
     ignore_lines = ["sometimes", "/\s\d{2,3}\s/", "/ignore-case text/"]
 
     fetcher = fetch_site_status.perform_site_check(datastore=False)
-    stripped_content = fetcher.strip_ignore_text(test_content, ignore_lines)
+    stripped_content = html_tools.strip_ignore_text(test_content, ignore_lines)
 
     assert b"but 1 lines" in stripped_content
     assert b"igNORe-cAse text" not in stripped_content

changedetectionio/tests/test_ignore_text.py

@@ -3,6 +3,7 @@
 import time
 from flask import url_for
 from . util import live_server_setup
+from changedetectionio import html_tools
 
 def test_setup(live_server):
     live_server_setup(live_server)
@@ -23,7 +24,7 @@ def test_strip_text_func():
     ignore_lines = ["sometimes"]
 
     fetcher = fetch_site_status.perform_site_check(datastore=False)
-    stripped_content = fetcher.strip_ignore_text(test_content, ignore_lines)
+    stripped_content = html_tools.strip_ignore_text(test_content, ignore_lines)
 
     assert b"sometimes" not in stripped_content
     assert b"Some content" in stripped_content
@@ -52,6 +53,8 @@ def set_modified_original_ignore_response():
      <p>Which is across multiple lines</p>
      </br>
      So let's see what happens.  </br>
+     <p>new ignore stuff</p>
+     <p>blah</p>
      </body>
      </html>
 
@@ -67,7 +70,7 @@ def set_modified_ignore_response():
        <body>
      Some initial text</br>
      <p>Which is across multiple lines</p>
-     <P>ZZZZZ</P>
+     <P>ZZZZz</P>
      </br>
      So let's see what happens.  </br>
      </body>
@@ -82,7 +85,8 @@ def set_modified_ignore_response():
 def test_check_ignore_text_functionality(client, live_server):
     sleep_time_for_fetch_thread = 3
 
-    ignore_text = "XXXXX\r\nYYYYY\r\nZZZZZ"
+    # Use a mix of case in ZzZ to prove it works case-insensitive.
+    ignore_text = "XXXXX\r\nYYYYY\r\nzZzZZ\r\nnew ignore stuff"
     set_original_ignore_response()
 
     # Give the endpoint time to spin up
@@ -142,13 +146,25 @@ def test_check_ignore_text_functionality(client, live_server):
     assert b'unviewed' not in res.data
     assert b'/test-endpoint' in res.data
 
+
+
+
+
     # Just to be sure.. set a regular modified change..
     set_modified_original_ignore_response()
     client.get(url_for("api_watch_checknow"), follow_redirects=True)
     time.sleep(sleep_time_for_fetch_thread)
+
     res = client.get(url_for("index"))
     assert b'unviewed' in res.data
 
+    # Check the preview/highlighter, we should be able to see what we ignored, but it should be highlighted
+    # We only introduce the "modified" content that includes what we ignore so we can prove the newest version also displays
+    # at /preview
+    res = client.get(url_for("preview_page", uuid="first"))
+    # We should be able to see what we ignored
+    assert b'<div class="ignored">new ignore stuff' in res.data
+
     res = client.get(url_for("api_delete", uuid="all"), follow_redirects=True)
     assert b'Deleted' in res.data
 

changedetectionio/tests/test_jsonpath_selector.py

@@ -162,7 +162,7 @@ def test_check_json_without_filter(client, live_server):
     time.sleep(1)
 
     # Add our URL to the import page
-    test_url = url_for('test_endpoint_json', _external=True)
+    test_url = url_for('test_endpoint', content_type="application/json", _external=True)
     client.post(
         url_for("import_page"),
         data={"urls": test_url},
@@ -193,7 +193,7 @@ def test_check_json_filter(client, live_server):
     time.sleep(1)
 
     # Add our URL to the import page
-    test_url = url_for('test_endpoint', _external=True)
+    test_url = url_for('test_endpoint', content_type="application/json", _external=True)
     res = client.post(
         url_for("import_page"),
         data={"urls": test_url},
@@ -258,7 +258,7 @@ def test_check_json_filter_bool_val(client, live_server):
     # Give the endpoint time to spin up
     time.sleep(1)
 
-    test_url = url_for('test_endpoint', _external=True)
+    test_url = url_for('test_endpoint', content_type="application/json", _external=True)
 
     res = client.post(
         url_for("import_page"),
@@ -313,7 +313,7 @@ def test_check_json_ext_filter(client, live_server):
     time.sleep(1)
 
     # Add our URL to the import page
-    test_url = url_for('test_endpoint', _external=True)
+    test_url = url_for('test_endpoint', content_type="application/json", _external=True)
     res = client.post(
         url_for("import_page"),
         data={"urls": test_url},

changedetectionio/tests/test_trigger.py

@@ -129,3 +129,8 @@ def test_trigger_functionality(client, live_server):
     time.sleep(sleep_time_for_fetch_thread)
     res = client.get(url_for("index"))
     assert b'unviewed' in res.data
+
+    # Check the preview/highlighter, we should be able to see what we triggered on, but it should be highlighted
+    res = client.get(url_for("preview_page", uuid="first"))
+    # We should be able to see what we ignored
+    assert b'<div class="triggered">foobar' in res.data

changedetectionio/tests/test_xpath_selector.py

@@ -96,6 +96,7 @@ def test_check_markup_xpath_filter_restriction(client, live_server):
     res = client.get(url_for("index"))
     assert b'unviewed' not in res.data
 
+
 def test_xpath_validation(client, live_server):
 
     # Give the endpoint time to spin up

changedetectionio/tests/util.py

@@ -1,5 +1,6 @@
 #!/usr/bin/python3
 
+from flask import make_response, request
 
 def set_original_response():
     test_return_data = """<html>
@@ -37,37 +38,19 @@ def set_modified_response():
 
 def live_server_setup(live_server):
 
-    @live_server.app.route('/test-binary-endpoint')
-    def test_binaryfile_endpoint():
-
-        from flask import make_response
-
-        # Tried using a global var here but didn't seem to work, so reading from a file instead.
-        with open("test-datastore/test.bin", "rb") as f:
-            resp = make_response(f.read())
-            resp.headers['Content-Type'] = 'image/jpeg'
-            return resp
 
     @live_server.app.route('/test-endpoint')
     def test_endpoint():
+        ctype = request.args.get('content_type')
+
         # Tried using a global var here but didn't seem to work, so reading from a file instead.
-        with open("test-datastore/endpoint-content.txt", "r") as f:
-            return f.read()
-
-    @live_server.app.route('/test-endpoint-json')
-    def test_endpoint_json():
-
-        from flask import make_response
-
         with open("test-datastore/endpoint-content.txt", "r") as f:
             resp = make_response(f.read())
-            resp.headers['Content-Type'] = 'application/json'
+            resp.headers['Content-Type'] = ctype if ctype else 'text/html'
             return resp
 
     @live_server.app.route('/test-403')
     def test_endpoint_403_error():
-
-        from flask import make_response
         resp = make_response('', 403)
         return resp
 
@@ -75,7 +58,6 @@ def live_server_setup(live_server):
     @live_server.app.route('/test-headers')
     def test_headers():
 
-        from flask import request
         output= []
 
         for header in request.headers:
@@ -86,24 +68,16 @@ def live_server_setup(live_server):
     # Just return the body in the request
     @live_server.app.route('/test-body', methods=['POST', 'GET'])
     def test_body():
-
-        from flask import request
-
         return request.data
 
     # Just return the verb in the request
     @live_server.app.route('/test-method', methods=['POST', 'GET', 'PATCH'])
     def test_method():
-
-        from flask import request
-
         return request.method
 
     # Where we POST to as a notification
     @live_server.app.route('/test_notification_endpoint', methods=['POST', 'GET'])
     def test_notification_endpoint():
-        from flask import request
-
         with open("test-datastore/notification.txt", "wb") as f:
             # Debug method, dump all POST to file also, used to prove #65
             data = request.stream.read()
@@ -117,8 +91,6 @@ def live_server_setup(live_server):
     # Just return the verb in the request
     @live_server.app.route('/test-basicauth', methods=['GET'])
     def test_basicauth_method():
-
-        from flask import request
         auth = request.authorization
         ret = " ".join([auth.username, auth.password, auth.type])
         return ret

changedetectionio/update_worker.py

@@ -42,6 +42,7 @@ class update_worker(threading.Thread):
                     now = time.time()
 
                     try:
+
                         changed_detected, update_obj, contents = update_handler.run(uuid)
 
                         # Re #342
@@ -134,8 +135,10 @@ class update_worker(threading.Thread):
 
                         except Exception as e:
                             # Catch everything possible here, so that if a worker crashes, we don't lose it until restart!
+                            print("!!!! Exception in update_worker !!!\n", e)
                             self.app.logger.error("Exception reached processing watch UUID: %s - %s", uuid, str(e))
                             self.datastore.update_watch(uuid=uuid, update_obj={'last_error': str(e)})
+
                     finally:
                         # Always record that we atleast tried
                         self.datastore.update_watch(uuid=uuid, update_obj={'fetch_time': round(time.time() - now, 3),
@@ -144,4 +147,7 @@ class update_worker(threading.Thread):
                 self.current_uuid = None  # Done
                 self.q.task_done()
 
+                # Give the CPU time to interrupt
+                time.sleep(0.1)
+
             self.app.config.exit.wait(1)