tag 0.38.2

Re #172 - be sure that we are non-greedy matching the first : when splitting the headers so we dont break "Cookie" header (#175 )
Update heroku.yml
2025-12-09 17:45:34 +00:00 · 2021-08-07 14:18:28 +02:00 · 2021-08-07 14:15:41 +02:00 · 2021-07-28 17:18:10 +02:00 · 2021-07-28 13:59:50 +02:00 · 2021-07-26 07:11:54 +02:00
26 changed files with 406 additions and 150 deletions
--- a/2
+++ b/2
@@ -12,7 +12,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
    libxslt-dev \
    zlib1g-dev \
    g++
-    
+
 RUN mkdir /install
 WORKDIR /install

--- a/README.md
+++ b/README.md
@@ -98,10 +98,24 @@ Detect changes and monitor data in JSON API's by using the built-in JSONPath sel

 ![image](https://user-images.githubusercontent.com/275001/125165842-0ce01980-e1dc-11eb-9e73-d8137dd162dc.png)

-This will re-parse the JSON and apply indent to the text, making it super easy to monitor and detect changes in JSON API results
+This will re-parse the JSON and apply formatting to the text, making it super easy to monitor and detect changes in JSON API results

 ![image](https://user-images.githubusercontent.com/275001/125165995-d9ea5580-e1dc-11eb-8030-f0deced2661a.png)

+#### Parse JSON embedded in HTML!
+
+When you enable a `json:` filter, you can even automatically extract and parse embedded JSON inside a HTML page! Amazingly handy for sites that build content based on JSON, such as many e-commerce websites. 
+
+```
+<html>
+...
+<script type="application/ld+json">
+  {"@context":"http://schema.org","@type":"Product","name":"Nan Optipro Stage 1 Baby Formula  800g","price": 23.50 }
+</script>
+```  
+
+`json:$.price` would give `23.50`, or you can extract the whole structure
+
 ### Proxy

 A proxy for ChangeDetection.io can be configured by setting environment the 
--- a/backend/init.py
+++ b/backend/init.py
@@ -223,7 +223,6 @@ def changedetection_app(config=None, datastore_o=None):
    @login_required
    def index():
        limit_tag = request.args.get('tag')
-
        pause_uuid = request.args.get('pause')

        if pause_uuid:
@@ -279,7 +278,11 @@ def changedetection_app(config=None, datastore_o=None):
            return response

        else:
+            from backend import forms
+            form = forms.quickWatchForm(request.form)
+
            output = render_template("watch-overview.html",
+                                     form=form,
                                     watches=sorted_watches,
                                     tags=existing_tags,
                                     active_tag=limit_tag,
@@ -296,25 +299,28 @@ def changedetection_app(config=None, datastore_o=None):
        if request.method == 'POST':
            confirmtext = request.form.get('confirmtext')
            limit_date = request.form.get('limit_date')
+            limit_timestamp = 0

-            try:
-                limit_date = limit_date.replace('T', ' ')
-                # I noticed chrome will show '/' but actually submit '-'
-                limit_date = limit_date.replace('-', '/')
-                # In the case that :ss seconds are supplied
-                limit_date = re.sub('(\d\d:\d\d)(:\d\d)', '\\1', limit_date)
+            # Re #149 - allow empty/0 timestamp limit
+            if len(limit_date):
+                try:
+                    limit_date = limit_date.replace('T', ' ')
+                    # I noticed chrome will show '/' but actually submit '-'
+                    limit_date = limit_date.replace('-', '/')
+                    # In the case that :ss seconds are supplied
+                    limit_date = re.sub('(\d\d:\d\d)(:\d\d)', '\\1', limit_date)

-                str_to_dt = datetime.datetime.strptime(limit_date, '%Y/%m/%d %H:%M')
-                limit_timestamp = int(str_to_dt.timestamp())
+                    str_to_dt = datetime.datetime.strptime(limit_date, '%Y/%m/%d %H:%M')
+                    limit_timestamp = int(str_to_dt.timestamp())

-                if limit_timestamp > time.time():
-                    flash("Timestamp is in the future, cannot continue.", 'error')
+                    if limit_timestamp > time.time():
+                        flash("Timestamp is in the future, cannot continue.", 'error')
+                        return redirect(url_for('scrub_page'))
+
+                except ValueError:
+                    flash('Incorrect date format, cannot continue.', 'error')
                    return redirect(url_for('scrub_page'))

-            except ValueError:
-                flash('Incorrect date format, cannot continue.', 'error')
-                return redirect(url_for('scrub_page'))
-
            if confirmtext == 'scrub':
                changes_removed = 0
                for uuid, watch in datastore.data['watching'].items():
@@ -483,29 +489,10 @@ def changedetection_app(config=None, datastore_o=None):
            datastore.data['settings']['application']['notification_title'] = form.notification_title.data
            datastore.data['settings']['application']['notification_body'] = form.notification_body.data

-            if len(form.notification_urls.data):
-                import apprise
-                apobj = apprise.Apprise()
-                apobj.debug = True
-
-                # Add each notification
-                for n in datastore.data['settings']['application']['notification_urls']:
-                    apobj.add(n)
-                outcome = apobj.notify(
-                    body='Hello from the worlds best and simplest web page change detection and monitoring service!',
-                    title='Changedetection.io Notification Test',
-                )
-
-                if outcome:
-                    flash("{} Notification URLs reached.".format(len(form.notification_urls.data)), "notice")
-                else:
-                    flash("One or more Notification URLs failed", 'error')
-
-
            datastore.data['settings']['application']['notification_urls'] = form.notification_urls.data
            datastore.needs_write = True

-            if form.trigger_check.data:
+            if form.trigger_check.data and len(form.notification_urls.data):
                n_object = {'watch_url': "Test from changedetection.io!",
                            'notification_urls': form.notification_urls.data}
                notification_q.put(n_object)
@@ -522,7 +509,10 @@ def changedetection_app(config=None, datastore_o=None):
        if request.method == 'POST' and not form.validate():
            flash("An error occurred, please see below.", "error")

-        output = render_template("settings.html", form=form)
+        # Same as notification.py
+        base_url = os.getenv('BASE_URL', '').strip('"')
+        output = render_template("settings.html", form=form, base_url=base_url)
+
        return output

    @app.route("/import", methods=['GET', "POST"])
@@ -729,19 +719,26 @@ def changedetection_app(config=None, datastore_o=None):
    @app.route("/api/add", methods=['POST'])
    @login_required
    def api_watch_add():
+        from backend import forms
+        form = forms.quickWatchForm(request.form)

-        url = request.form.get('url').strip()
-        if datastore.url_exists(url):
-            flash('The URL {} already exists'.format(url), "error")
+        if form.validate():
+
+            url = request.form.get('url').strip()
+            if datastore.url_exists(url):
+                flash('The URL {} already exists'.format(url), "error")
+                return redirect(url_for('index'))
+
+            # @todo add_watch should throw a custom Exception for validation etc
+            new_uuid = datastore.add_watch(url=url, tag=request.form.get('tag').strip())
+            # Straight into the queue.
+            update_q.put(new_uuid)
+
+            flash("Watch added.")
+            return redirect(url_for('index'))
+        else:
+            flash("Error")
            return redirect(url_for('index'))
-
-        # @todo add_watch should throw a custom Exception for validation etc
-        new_uuid = datastore.add_watch(url=url, tag=request.form.get('tag').strip())
-        # Straight into the queue.
-        update_q.put(new_uuid)
-
-        flash("Watch added.")
-        return redirect(url_for('index'))

    @app.route("/api/delete", methods=['GET'])
    @login_required
--- a/backend/fetch_site_status.py
+++ b/backend/fetch_site_status.py
@@ -63,7 +63,7 @@ class perform_site_check():
        extra_headers = self.datastore.get_val(uuid, 'headers')

        # Tweak the base config with the per-watch ones
-        request_headers = self.datastore.data['settings']['headers']
+        request_headers = self.datastore.data['settings']['headers'].copy()
        request_headers.update(extra_headers)

        # https://github.com/psf/requests/issues/4525
@@ -92,27 +92,8 @@ class perform_site_check():
            css_filter_rule = self.datastore.data['watching'][uuid]['css_filter']
            if css_filter_rule and len(css_filter_rule.strip()):
                if 'json:' in css_filter_rule:
-                    # POC hack, @todo rename vars, see how it fits in with the javascript version
-                    import json
-                    from jsonpath_ng import jsonpath, parse
-
-                    json_data = json.loads(html)
-                    jsonpath_expression = parse(css_filter_rule.replace('json:', ''))
-                    match = jsonpath_expression.find(json_data)
-                    s = []
-
-                    # More than one result, we will return it as a JSON list.
-                    if len(match) > 1:
-                        for i in match:
-                            s.append(i.value)
-
-                    # Single value, use just the value, as it could be later used in a token in notifications.
-                    if len(match) == 1:
-                        s = match[0].value
-
-                    stripped_text_from_html = json.dumps(s, indent=4)
+                    stripped_text_from_html = html_tools.extract_json_as_string(html, css_filter_rule)
                    is_html = False
-
                else:
                    # CSS Filter, extract the HTML that matches and feed that into the existing inscriptis::get_text
                    html = html_tools.css_filter(css_filter=css_filter_rule, html_content=r.content)
--- a/backend/forms.py
+++ b/backend/forms.py
@@ -75,7 +75,7 @@ class StringDictKeyValue(StringField):
            # Remove empty strings
            cleaned = list(filter(None, valuelist[0].split("\n")))
            for s in cleaned:
-                parts = s.strip().split(':')
+                parts = s.strip().split(':', 1)
                if len(parts) == 2:
                    self.data.update({parts[0].strip(): parts[1].strip()})

@@ -124,13 +124,15 @@ class ValidateCSSJSONInput(object):
                message = field.gettext('\'%s\' is not a valid JSONPath expression. (%s)')
                raise ValidationError(message % (input, str(e)))

-
-class watchForm(Form):
+class quickWatchForm(Form):
    # https://wtforms.readthedocs.io/en/2.3.x/fields/#module-wtforms.fields.html5
    # `require_tld` = False is needed even for the test harness "http://localhost:5005.." to run

    url = html5.URLField('URL', [validators.URL(require_tld=False)])
    tag = StringField('Tag', [validators.Optional(), validators.Length(max=35)])
+
+class watchForm(quickWatchForm):
+
    minutes_between_check = html5.IntegerField('Maximum time in minutes until recheck',
                                               [validators.Optional(), validators.NumberRange(min=1)])
    css_filter = StringField('CSS/JSON Filter', [ValidateCSSJSONInput()])
--- a/backend/html_tools.py
+++ b/backend/html_tools.py
@@ -1,6 +1,12 @@
+import json
 from bs4 import BeautifulSoup
+from jsonpath_ng import parse


+class JSONNotFound(ValueError):
+    def __init__(self, msg):
+        ValueError.__init__(self, msg)
+
 # Given a CSS Rule, and a blob of HTML, return the blob of HTML that matches
 def css_filter(css_filter, html_content):
    soup = BeautifulSoup(html_content, "html.parser")
@@ -24,3 +30,61 @@ def extract_element(find='title', html_content=''):

    return element_text

+#
+def _parse_json(json_data, jsonpath_filter):
+    s=[]
+    jsonpath_expression = parse(jsonpath_filter.replace('json:', ''))
+    match = jsonpath_expression.find(json_data)
+
+    # More than one result, we will return it as a JSON list.
+    if len(match) > 1:
+        for i in match:
+            s.append(i.value)
+
+    # Single value, use just the value, as it could be later used in a token in notifications.
+    if len(match) == 1:
+        s = match[0].value
+
+    if not s:
+        raise JSONNotFound("No Matching JSON could be found for the rule {}".format(jsonpath_filter.replace('json:', '')))
+
+    stripped_text_from_html = json.dumps(s, indent=4)
+
+    return stripped_text_from_html
+
+def extract_json_as_string(content, jsonpath_filter):
+
+    stripped_text_from_html = False
+
+    # Try to parse/filter out the JSON, if we get some parser error, then maybe it's embedded <script type=ldjson>
+    try:
+        stripped_text_from_html = _parse_json(json.loads(content), jsonpath_filter)
+    except json.JSONDecodeError:
+
+        # Foreach <script json></script> blob.. just return the first that matches jsonpath_filter
+        s = []
+        soup = BeautifulSoup(content, 'html.parser')
+        bs_result = soup.findAll('script')
+
+        if not bs_result:
+            raise JSONNotFound("No parsable JSON found in this document")
+
+        for result in bs_result:
+            # Skip empty tags, and things that dont even look like JSON
+            if not result.string or not '{' in result.string:
+                continue
+                
+            try:
+                json_data = json.loads(result.string)
+            except json.JSONDecodeError:
+                # Just skip it
+                continue
+            else:
+                stripped_text_from_html = _parse_json(json_data, jsonpath_filter)
+                if stripped_text_from_html:
+                    break
+
+    if not stripped_text_from_html:
+        raise JSONNotFound("No JSON matching the rule '%s' found" % jsonpath_filter.replace('json:',''))
+
+    return stripped_text_from_html
--- a/backend/notification.py
+++ b/backend/notification.py
@@ -4,6 +4,7 @@ import apprise
 def process_notification(n_object, datastore):
    apobj = apprise.Apprise()
    for url in n_object['notification_urls']:
+        print (">> Process Notification: AppRise notifying {}".format(url.strip()))
        apobj.add(url.strip())

    # Get the notification body from datastore
@@ -38,11 +39,13 @@ def create_notification_parameters(n_object):
    base_url = os.getenv('BASE_URL', '').strip('"')
    watch_url = n_object['watch_url']

-    if base_url != '':
-        diff_url = "{}/diff/{}".format(base_url, uuid)
-        preview_url = "{}/preview/{}".format(base_url, uuid)
-    else:
-        diff_url = preview_url = ''
+    # Re #148 - Some people have just {base_url} in the body or title, but this may break some notification services
+    #           like 'Join', so it's always best to atleast set something obvious so that they are not broken.
+    if base_url == '':
+        base_url = "<base-url-env-var-not-set>"
+
+    diff_url = "{}/diff/{}".format(base_url, uuid)
+    preview_url = "{}/preview/{}".format(base_url, uuid)

    return {
        'base_url': base_url,
--- a/backend/static/images/favicon.ico
+++ b/backend/static/images/favicon.ico
--- a/backend/static/styles/styles.css
+++ b/backend/static/styles/styles.css
@@ -201,12 +201,13 @@ body:after, body:before {
  padding: 1em;
  border-radius: 10px;
  margin-bottom: 1em; }
-
-#new-watch-form legend {
-  color: #fff; }
-
-#new-watch-form input {
-  width: auto !important; }
+  #new-watch-form input {
+    width: auto !important;
+    display: inline-block; }
+  #new-watch-form .label {
+    display: none; }
+  #new-watch-form legend {
+    color: #fff; }

 #diff-col {
  padding-left: 40px; }
@@ -280,7 +281,7 @@ footer {
  /* The list of errors */ }
  .pure-form .pure-control-group, .pure-form .pure-group, .pure-form .pure-controls {
    padding-bottom: 1em; }
-    .pure-form .pure-control-group dd, .pure-form .pure-group dd, .pure-form .pure-controls dd {
+    .pure-form .pure-control-group div, .pure-form .pure-group div, .pure-form .pure-controls div {
      margin: 0px; }
  .pure-form .error input {
    background-color: #ffebeb; }
@@ -296,8 +297,6 @@ footer {
      color: #dd0000; }
  .pure-form label {
    font-weight: bold; }
-  .pure-form input[type=url] {
-    width: 100%; }
  .pure-form textarea {
    width: 100%; }

@@ -306,7 +305,7 @@ footer {
    max-width: 95%; }
  .edit-form {
    padding: 0.5em;
-    margin: 0.5em; }
+    margin: 0; }
  #nav-menu {
    overflow-x: scroll; } }

@@ -316,6 +315,8 @@ This query will take effect for any screen smaller than 760px
 and also iPads specifically.
 */
@media only screen and (max-width: 760px), (min-device-width: 768px) and (max-device-width: 1024px) {
+  input[type='text'] {
+    width: 100%; }
  .watch-table {
    /* Force table to not be like tables anymore */
    /* Force table to not be like tables anymore */
@@ -353,3 +354,12 @@ and also iPads specifically.
      background-color: #eee; }
    .watch-table.pure-table-striped tr:nth-child(2n-1) td {
      background-color: inherit; } }
+
+/** Desktop vs mobile input field strategy
+- We dont use 'size' with <input> because `size` is too unreliable to override, and will often push-out
+- Rely always on width in CSS
+*/
+@media only screen and (min-width: 761px) {
+  /* m-d is medium-desktop */
+  .m-d {
+    min-width: 80%; } }
--- a/backend/static/styles/styles.scss
+++ b/backend/static/styles/styles.scss
@@ -266,15 +266,20 @@ body:after, body:before {
  padding: 1em;
  border-radius: 10px;
  margin-bottom: 1em;
+  input {
+    width: auto !important;
+    display: inline-block;
+  }
+  .label {
+    display: none;
+  }
+  legend {
+    color: #fff;
+  }
 }

-#new-watch-form legend {
-  color: #fff;
-}

-#new-watch-form input {
-  width: auto !important;
-}
+

 #diff-col {
  padding-left: 40px;
@@ -367,7 +372,7 @@ footer {
 .pure-form {
    .pure-control-group, .pure-group, .pure-controls {
        padding-bottom: 1em;
-        dd {
+        div {
            margin: 0px;
        }
    }
@@ -377,7 +382,8 @@ footer {
        background-color: #ffebeb;
    }
  }
-   /* The list of errors */
+
+  /* The list of errors */
  ul.errors {
    padding: .5em .6em;
    border: 1px solid #dd0000;
@@ -395,10 +401,6 @@ footer {
    font-weight: bold;
  }

-  input[type=url] {
-    width: 100%;
-  }
-
  textarea {
    width: 100%;
  }
@@ -410,7 +412,7 @@ footer {
  }
  .edit-form {
    padding: 0.5em;
-    margin: 0.5em;
+    margin: 0;
  }
  #nav-menu {
    overflow-x: scroll;
@@ -423,9 +425,12 @@ Max width before this PARTICULAR table gets nasty
 This query will take effect for any screen smaller than 760px
 and also iPads specifically.
 */
-@media only screen and (max-width: 760px),
-(min-device-width: 768px) and (max-device-width: 1024px) {
+@media only screen and (max-width: 760px), (min-device-width: 768px) and (max-device-width: 1024px) {

+  input[type='text'] {
+    width: 100%;
+  }
+    
  .watch-table {
    /* Force table to not be like tables anymore */
    thead, tbody, th, td, tr {
@@ -490,3 +495,15 @@ and also iPads specifically.
  }
 }

+
+/** Desktop vs mobile input field strategy
+- We dont use 'size' with <input> because `size` is too unreliable to override, and will often push-out
+- Rely always on width in CSS
+*/
+@media only screen and (min-width: 761px) {
+/* m-d is medium-desktop */
+    .m-d {
+        min-width: 80%;
+    }
+}
+
--- a/backend/store.py
+++ b/backend/store.py
@@ -42,7 +42,7 @@ class ChangeDetectionStore:
                    'notification_urls': [], # Apprise URL list
                    # Custom notification content
                    'notification_title': 'ChangeDetection.io Notification - {watch_url}',
-                    'notification_body': '{base_url}'
+                    'notification_body': '{watch_url} had a change.'
                }
            }
        }
@@ -117,7 +117,7 @@ class ChangeDetectionStore:
                self.add_watch(url='https://www.gov.uk/coronavirus', tag='Covid')
                self.add_watch(url='https://changedetection.io', tag='Tech news')

-        self.__data['version_tag'] = "0.38"
+        self.__data['version_tag'] = "0.38.2"

        # Helper to remove password protection
        password_reset_lockfile = "{}/removepassword.lock".format(self.datastore_path)
--- a/backend/templates/_helpers.jinja
+++ b/backend/templates/_helpers.jinja
@@ -1,6 +1,6 @@
 {% macro render_field(field) %}
-  <dt {% if field.errors %} class="error" {% endif %}>{{ field.label }}
-  <dd {% if field.errors %} class="error" {% endif %}>{{ field(**kwargs)|safe }}
+  <div {% if field.errors %} class="error" {% endif %}>{{ field.label }}</div>
+  <div {% if field.errors %} class="error" {% endif %}>{{ field(**kwargs)|safe }}
  {% if field.errors %}
    <ul class=errors>
    {% for error in field.errors %}
@@ -8,5 +8,18 @@
    {% endfor %}
    </ul>
  {% endif %}
-  </dd>
+  </div>
+{% endmacro %}
+
+{% macro render_simple_field(field) %}
+  <span class="label {% if field.errors %}error{% endif %}">{{ field.label }}</span>
+  <span {% if field.errors %} class="error" {% endif %}>{{ field(**kwargs)|safe }}
+  {% if field.errors %}
+    <ul class=errors>
+    {% for error in field.errors %}
+      <li>{{ error }}</li>
+    {% endfor %}
+    </ul>
+  {% endif %}
+  </span>
 {% endmacro %}
--- a/backend/templates/base.html
+++ b/backend/templates/base.html
@@ -26,7 +26,7 @@
        {% if current_diff_url %}
        <a class=current-diff-url href="{{ current_diff_url }}"><span style="max-width: 30%; overflow: hidden;">{{ current_diff_url }}</span></a>
        {% else %}
-        {% if new_version_available %}
+        {% if new_version_available and not (has_password and not current_user.is_authenticated) %}
        <span id="new-version-text" class="pure-menu-heading"><a href="https://github.com/dgtlmoon/changedetection.io">A new version is available</a></span>
        {% endif %}
        {% endif %}
--- a/backend/templates/edit.html
+++ b/backend/templates/edit.html
@@ -5,16 +5,16 @@
    <form class="pure-form pure-form-stacked" action="{{ url_for('edit_page', uuid=uuid, next = request.args.get('next') ) }}" method="POST">
        <fieldset>
            <div class="pure-control-group">
-                {{ render_field(form.url, placeholder="https://...", size=30, required=true) }}
+                {{ render_field(form.url, placeholder="https://...", required=true, class="m-d") }}
            </div>
            <div class="pure-control-group">
-                {{ render_field(form.title, size=30) }}
+                {{ render_field(form.title, class="m-d") }}
            </div>
            <div class="pure-control-group">
-                {{ render_field(form.tag, size=10) }}
+                {{ render_field(form.tag) }}
            </div>
            <div class="pure-control-group">
-                {{ render_field(form.minutes_between_check, size=5) }}
+                {{ render_field(form.minutes_between_check) }}
                {% if using_default_minutes %}
                    <span class="pure-form-message-inline">Currently using the <a href="{{ url_for('settings_page', uuid=uuid) }}">default global settings</a>, change to another value if you want to be specific.</span>
                {% else %}
@@ -22,7 +22,7 @@
                {% endif %}
            </div>
            <div class="pure-control-group">
-                {{ render_field(form.css_filter, size=25, placeholder=".class-name or #some-id, or other CSS selector rule.") }}
+                {{ render_field(form.css_filter, placeholder=".class-name or #some-id, or other CSS selector rule.", class="m-d") }}
                <span class="pure-form-message-inline">
                    <ul>
                        <li>CSS - Limit text to this CSS rule, only text matching this CSS rule is included.</li>
@@ -57,6 +57,7 @@ AWS SNS - sns://AccessKeyID/AccessSecretKey/RegionName/+PhoneNo
 SMTPS - mailtos://user:pass@mail.domain.com?to=receivingAddress@example.com
 ") }}
                <span class="pure-form-message-inline">Use <a target=_new href="https://github.com/caronc/apprise">AppRise URLs</a> for notification to just about any service!</span>
+                <span class="pure-form-message-inline">Note: This overrides any global settings notification URLs</span>
            </div>

            <div class="pure-controls">
--- a/backend/templates/settings.html
+++ b/backend/templates/settings.html
@@ -7,14 +7,14 @@
    <form class="pure-form pure-form-stacked settings" action="{{url_for('settings_page')}}" method="POST">
        <fieldset>
            <div class="pure-control-group">
-                {{ render_field(form.minutes_between_check, size=5) }}
+                {{ render_field(form.minutes_between_check) }}
                <span class="pure-form-message-inline">Default time for all watches, when the watch does not have a specific time setting.</span>
            </div>
            <div class="pure-control-group">
                {% if current_user.is_authenticated %}
                    <a href="{{url_for('settings_page', removepassword='yes')}}" class="pure-button pure-button-primary">Remove password</a>
                {% else %}
-                {{ render_field(form.password, size=10) }}
+                {{ render_field(form.password) }}
                <span class="pure-form-message-inline">Password protection for your changedetection.io application.</span>
                {% endif %}
            </div>
@@ -39,7 +39,7 @@ SMTPS - mailtos://user:pass@mail.domain.com?to=receivingAddress@example.com") }}
                <div id="notification-customisation" style="display:none;">

                    <div class="pure-control-group">
-                        {{ render_field(form.notification_title, size=80) }}
+                        {{ render_field(form.notification_title, class="m-d") }}
                        <span class="pure-form-message-inline">Title for all notifications</span>
                    </div>
                    <div class="pure-control-group">
@@ -82,15 +82,14 @@ SMTPS - mailtos://user:pass@mail.domain.com?to=receivingAddress@example.com") }}
                            </tbody>
                        </table>
                        <span class="pure-form-message-inline">
-                            URLs generated by changedetection.io (such as <code>{diff_url}</code>) require the <code>BASE_URL</code> environment variable set.
+                            URLs generated by changedetection.io (such as <code>{diff_url}</code>) require the <code>BASE_URL</code> environment variable set.<br/>
+                            Your <code>BASE_URL</code> var is currently "{{base_url}}"
                        </span>
                    </div>
                </div>
-
-                <span class="pure-form-message-inline"><label for="trigger-test-notification" class="pure-checkbox">
-                    <input type="checkbox" id="trigger-test-notification" name="trigger-test-notification"> Send test notification on save.</label>
-                </span>
-
+                <div class="pure-control-group">
+                    {{ render_field(form.trigger_check) }}
+                </div>
            </div>
            <div class="pure-control-group">
                <button type="submit" class="pure-button pure-button-primary">Save</button>
--- a/backend/templates/watch-overview.html
+++ b/backend/templates/watch-overview.html
@@ -1,14 +1,14 @@
 {% extends 'base.html' %}
-
 {% block content %}
+{% from '_helpers.jinja' import render_simple_field %}

 <div class="box">

    <form class="pure-form" action="{{ url_for('api_watch_add') }}" method="POST" id="new-watch-form">
        <fieldset>
            <legend>Add a new change detection watch</legend>
-            <input type="url" placeholder="https://..." name="url"/>
-            <input type="text" placeholder="tag" size="10" name="tag" value="{{active_tag if active_tag}}"/>
+                {{ render_simple_field(form.url, placeholder="https://...", required=true) }}
+                {{ render_simple_field(form.tag, value=active_tag if active_tag else '', placeholder="tag") }}
            <button type="submit" class="pure-button pure-button-primary">Watch</button>
        </fieldset>
        <!-- add extra stuff, like do a http POST and send headers -->
--- a/backend/tests/conftest.py
+++ b/backend/tests/conftest.py
@@ -12,6 +12,21 @@ import os

 global app

+
+def cleanup(datastore_path):
+    # Unlink test output files
+    files = ['output.txt',
+             'url-watches.json',
+             'notification.txt',
+             'count.txt',
+             'endpoint-content.txt']
+    for file in files:
+        try:
+            os.unlink("{}/{}".format(datastore_path, file))
+            x = 1
+        except FileNotFoundError:
+            pass
+
@pytest.fixture(scope='session')
 def app(request):
    """Create application for the tests."""
@@ -25,15 +40,7 @@ def app(request):
    # Enable a BASE_URL for notifications to work (so we can look for diff/ etc URLs)
    os.environ["BASE_URL"] = "http://mysite.com/"

-    # Unlink test output files
-    files = ['test-datastore/output.txt',
-             "{}/url-watches.json".format(datastore_path),
-             'test-datastore/notification.txt']
-    for file in files:
-        try:
-            os.unlink(file)
-        except FileNotFoundError:
-            pass
+    cleanup(datastore_path)

    app_config = {'datastore_path': datastore_path}
    datastore = store.ChangeDetectionStore(datastore_path=app_config['datastore_path'], include_default_watches=False)
@@ -43,13 +50,8 @@ def app(request):
    def teardown():
        datastore.stop_thread = True
        app.config.exit.set()
-        for fname in ["url-watches.json", "count.txt", "output.txt"]:
-            try:
-                os.unlink("{}/{}".format(datastore_path, fname))
-            except FileNotFoundError:
-                # This is fine in the case of a failure.
-                pass
-
+        cleanup(datastore_path)
+        
    request.addfinalizer(teardown)
    yield app

--- a/backend/tests/test_headers.py
+++ b/backend/tests/test_headers.py
@@ -0,0 +1,79 @@
+import json
+import time
+from flask import url_for
+from . util import set_original_response, set_modified_response, live_server_setup
+
+# Hard to just add more live server URLs when one test is already running (I think)
+# So we add our test here (was in a different file)
+def test_headers_in_request(client, live_server):
+    live_server_setup(live_server)
+
+    # Add our URL to the import page
+    test_url = url_for('test_headers', _external=True)
+
+    # Add the test URL twice, we will check
+    res = client.post(
+        url_for("import_page"),
+        data={"urls": test_url},
+        follow_redirects=True
+    )
+    assert b"1 Imported" in res.data
+
+    res = client.post(
+        url_for("import_page"),
+        data={"urls": test_url},
+        follow_redirects=True
+    )
+    assert b"1 Imported" in res.data
+
+    cookie_header = '_ga=GA1.2.1022228332; cookie-preferences=analytics:accepted;'
+
+
+    # Add some headers to a request
+    res = client.post(
+        url_for("edit_page", uuid="first"),
+        data={
+              "url": test_url,
+              "tag": "",
+              "headers": "xxx:ooo\ncool:yeah\r\ncookie:"+cookie_header},
+        follow_redirects=True
+    )
+    assert b"Updated watch." in res.data
+
+
+    # Give the thread time to pick up the first version
+    time.sleep(5)
+
+    # The service should echo back the request headers
+    res = client.get(
+        url_for("preview_page", uuid="first"),
+        follow_redirects=True
+    )
+
+    # Flask will convert the header key to uppercase
+    assert b"Xxx:ooo" in res.data
+    assert b"Cool:yeah" in res.data
+
+    # The test call service will return the headers as the body
+    from html import escape
+    assert escape(cookie_header).encode('utf-8') in res.data
+
+    time.sleep(5)
+
+    # Re #137 -  Examine the JSON index file, it should have only one set of headers entered
+    watches_with_headers = 0
+    with open('test-datastore/url-watches.json') as f:
+        app_struct = json.load(f)
+        for uuid in app_struct['watching']:
+            if (len(app_struct['watching'][uuid]['headers'])):
+                watches_with_headers += 1
+
+    # Should be only one with headers set
+    assert watches_with_headers==1
+
+
+
+
+
+
+
--- a/backend/tests/test_jsonpath_selector.py
+++ b/backend/tests/test_jsonpath_selector.py
@@ -3,6 +3,45 @@
 import time
 from flask import url_for
 from . util import live_server_setup
+import pytest
+
+def test_unittest_inline_html_extract():
+    # So lets pretend that the JSON we want is inside some HTML
+    content="""
+    <html>
+    
+    food and stuff and more
+    <script>
+    alert('nothing really good here');
+    </script>
+    
+    <script type="application/ld+json">
+  xx {"@context":"http://schema.org","@type":"Product","name":"Nan Optipro Stage 1 Baby Formula  800g","description":"During the first year of life, nutrition is critical for your baby. NAN OPTIPRO 1 is tailored to ensure your formula fed infant receives balanced, high quality nutrition.<br />Starter infant formula. The age optimised protein source (whey dominant) is from cow’s milk.<br />Backed by more than 150 years of Nestlé expertise.<br />For hygiene and convenience, it is available in an innovative packaging format with a separate storage area for the scoop, and a semi-transparent window which allows you to see how much powder is left in the can without having to open it.","image":"https://cdn0.woolworths.media/content/wowproductimages/large/155536.jpg","brand":{"@context":"http://schema.org","@type":"Organization","name":"Nan"},"gtin13":"7613287517388","offers":{"@context":"http://schema.org","@type":"Offer","potentialAction":{"@context":"http://schema.org","@type":"BuyAction"},"availability":"http://schema.org/InStock","itemCondition":"http://schema.org/NewCondition","price":23.5,"priceCurrency":"AUD"},"review":[],"sku":"155536"}
+</script>
+<body>
+and it can also be repeated
+<script type="application/ld+json">
+  {"@context":"http://schema.org","@type":"Product","name":"Nan Optipro Stage 1 Baby Formula  800g","description":"During the first year of life, nutrition is critical for your baby. NAN OPTIPRO 1 is tailored to ensure your formula fed infant receives balanced, high quality nutrition.<br />Starter infant formula. The age optimised protein source (whey dominant) is from cow’s milk.<br />Backed by more than 150 years of Nestlé expertise.<br />For hygiene and convenience, it is available in an innovative packaging format with a separate storage area for the scoop, and a semi-transparent window which allows you to see how much powder is left in the can without having to open it.","image":"https://cdn0.woolworths.media/content/wowproductimages/large/155536.jpg","brand":{"@context":"http://schema.org","@type":"Organization","name":"Nan"},"gtin13":"7613287517388","offers":{"@context":"http://schema.org","@type":"Offer","potentialAction":{"@context":"http://schema.org","@type":"BuyAction"},"availability":"http://schema.org/InStock","itemCondition":"http://schema.org/NewCondition","price":23.5,"priceCurrency":"AUD"},"review":[],"sku":"155536"}
+</script>
+<h4>ok</h4>
+</body>
+</html>
+
+    """
+    from .. import html_tools
+
+    # See that we can find the second <script> one, which is not broken, and matches our filter
+    text = html_tools.extract_json_as_string(content, "$.offers.price")
+    assert text == "23.5"
+
+    text = html_tools.extract_json_as_string('{"id":5}', "$.id")
+    assert text == "5"
+
+    # When nothing at all is found, it should throw JSONNOTFound
+    # Which is caught and shown to the user in the watch-overview table
+    with pytest.raises(html_tools.JSONNotFound) as e_info:
+        html_tools.extract_json_as_string('COMPLETE GIBBERISH, NO JSON!', "$.id")
+

 def test_setup(live_server):
    live_server_setup(live_server)
--- a/backend/tests/test_notification.py
+++ b/backend/tests/test_notification.py
@@ -52,11 +52,14 @@ def test_check_notification(client, live_server):
    # Because we hit 'send test notification on save'
    time.sleep(3)

+    notification_submission = None
+
    # Verify what was sent as a notification, this file should exist
    with open("test-datastore/notification.txt", "r") as f:
        notification_submission = f.read()
        # Did we see the URL that had a change, in the notification?
-        assert test_url in notification_submission
+
+    assert test_url in notification_submission

    os.unlink("test-datastore/notification.txt")

@@ -75,11 +78,13 @@ def test_check_notification(client, live_server):

    assert bytes("just now".encode('utf-8')) in res.data

+    notification_submission=None
    # Verify what was sent as a notification
    with open("test-datastore/notification.txt", "r") as f:
        notification_submission = f.read()
        # Did we see the URL that had a change, in the notification?
-        assert test_url in notification_submission
+
+    assert test_url in notification_submission

        # Re #65 - did we see our foobar.com BASE_URL ?
        #assert bytes("https://foobar.com".encode('utf-8')) in notification_submission
@@ -94,10 +99,13 @@ def test_check_notification(client, live_server):
        url_for("settings_page"),
        data={"notification_title": "New ChangeDetection.io Notification - {watch_url}",
              "notification_body": "{base_url}\n{watch_url}\n{preview_url}\n{diff_url}\n{current_snapshot}\n:-)",
+              "notification_urls": "json://foobar.com", #Re #143 should not see that it sent without [test checkbox]
              "minutes_between_check": 180},
        follow_redirects=True
    )
    assert b"Settings updated." in res.data
+    # Re #143 - should not see this if we didnt hit the test box
+    assert b"Notifications queued" not in res.data

    # Trigger a check
    client.get(url_for("api_watch_checknow"), follow_redirects=True)
--- a/backend/tests/util.py
+++ b/backend/tests/util.py
@@ -37,12 +37,26 @@ def set_modified_response():

 def live_server_setup(live_server):

+
    @live_server.app.route('/test-endpoint')
    def test_endpoint():
        # Tried using a global var here but didn't seem to work, so reading from a file instead.
        with open("test-datastore/output.txt", "r") as f:
            return f.read()

+    # Just return the headers in the request
+    @live_server.app.route('/test-headers')
+    def test_headers():
+
+        from flask import request
+        output= []
+
+        for header in request.headers:
+             output.append("{}:{}".format(str(header[0]),str(header[1])   ))
+
+        return "\n".join(output)
+
+
    # Where we POST to as a notification
    @live_server.app.route('/test_notification_endpoint', methods=['POST'])
    def test_notification_endpoint():
--- a/backend/update_worker.py
+++ b/backend/update_worker.py
@@ -46,28 +46,33 @@ class update_worker(threading.Thread):
                                    self.datastore.save_history_text(uuid=uuid, contents=contents, result_obj=result)
                                    watch = self.datastore.data['watching'][uuid]

+                                    print (">> Change detected in UUID {} - {}".format(uuid, watch['url']))
+
+                                    # Get the newest snapshot data to be possibily used in a notification
                                    newest_key = self.datastore.get_newest_history_key(uuid)
                                    if newest_key:
                                        with open(watch['history'][newest_key], 'r') as f:
                                            newest_version_file_contents = f.read().strip()

                                    n_object = {
-                                        'watch_url': self.datastore.data['watching'][uuid]['url'],
+                                        'watch_url': watch['url'],
                                        'uuid': uuid,
                                        'current_snapshot': newest_version_file_contents
                                    }

                                    # Did it have any notification alerts to hit?
                                    if len(watch['notification_urls']):
-                                        print("Processing notifications for UUID: {}".format(uuid))
+                                        print(">>> Notifications queued for UUID from watch {}".format(uuid))
                                        n_object['notification_urls'] = watch['notification_urls']
                                        self.notification_q.put(n_object)

                                    # No? maybe theres a global setting, queue them all
                                    elif len(self.datastore.data['settings']['application']['notification_urls']):
-                                        print("Processing GLOBAL notifications for UUID: {}".format(uuid))
+                                        print(">>> Watch notification URLs were empty, using GLOBAL notifications for UUID: {}".format(uuid))
                                        n_object['notification_urls'] = self.datastore.data['settings']['application']['notification_urls']
                                        self.notification_q.put(n_object)
+                                    else:
+                                        print(">>> NO notifications queued, watch and global notification URLs were empty.")

                            except Exception as e:
                                print("!!!! Exception in update_worker !!!\n", e)
--- a/changedetection.py
+++ b/changedetection.py
@@ -14,7 +14,7 @@ from backend import store

 def main(argv):
    ssl_mode = False
-    port = 5000
+    port = os.environ.get('PORT') or 5000
    do_cleanup = False

    # Must be absolute so that send_from_directory doesnt try to make it relative to backend/
@@ -85,7 +85,7 @@ def main(argv):
                                               server_side=True), app)

    else:
-        eventlet.wsgi.server(eventlet.listen(('', port)), app)
+        eventlet.wsgi.server(eventlet.listen(('', int(port))), app)


 if __name__ == '__main__':
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -7,6 +7,9 @@ services:
      volumes:
        - changedetection-data:/datastore
  #    environment:
+  #        Default listening port, can also be changed with the -p option
+  #      - PORT=5000
+
  #      - PUID=1000
  #      - PGID=1000
  #        Proxy support example.
@@ -14,7 +17,7 @@ services:
  #      - HTTPS_PROXY="socks5h://10.10.1.10:1080"
  #        An exclude list (useful for notification URLs above) can be specified by with
  #      - NO_PROXY="localhost,192.168.0.0/24"
-  #        Base URL of your changedetection.io install (Added to notification alert
+  #        Base URL of your changedetection.io install (Added to the notification alert)
  #      - BASE_URL="https://mysite.com"

  #        Respect proxy_pass type settings, `proxy_set_header Host "localhost";` and `proxy_set_header X-Forwarded-Prefix /app;`
--- a/heroku.yml
+++ b/heroku.yml
@@ -0,0 +1,5 @@
+build:
+  docker:
+    changedetection: Dockerfile
+run:
+  changedetection: python ./changedetection.py -d /datastore -p $PORT
--- a/screenshot.png
+++ b/screenshot.png
Author	SHA1	Message	Date
dgtlmoon	00fe4d4e41	tag 0.38.2	2021-08-07 14:18:28 +02:00
dgtlmoon	f88561e713	Re #172 - be sure that we are non-greedy matching the first : when splitting the headers so we dont break "Cookie" header (#175 )	2021-08-07 14:15:41 +02:00
dgtlmoon	dd193ffcec	Update heroku.yml Re #156 - You can specify the port here too, to be sure	2021-07-28 17:18:10 +02:00
dgtlmoon	1e39a1b745	Re #156 - PORT should always be an Integer	2021-07-28 13:59:50 +02:00
Leigh	1084603375	Merge branch 'master' of github.com:dgtlmoon/changedetection.io	2021-07-26 07:11:54 +02:00
Leigh	3f9d949534	Re #159 - Adding env var example to docker-config.yml	2021-07-26 07:10:57 +02:00
Tim Chepeleff	684deaed35	Add Heroku Deployment Support (#159 ) * add heroku.yml * Use environment supplied port * Update changedetection.py	2021-07-26 06:43:23 +02:00
Leigh	1b931fef20	Re #154 - Handle missing JSON better	2021-07-25 13:55:28 +02:00
Leigh	d1976db149	high res	2021-07-25 09:18:05 +02:00
Leigh	a8fb17df9a	higher res screenshot	2021-07-25 09:17:02 +02:00
Leigh	8f28c80ef5	Update screenshot	2021-07-25 09:16:07 +02:00
Leigh	5a2c534fde	Assert that html_tools.JSONNotFound is correctly raised	2021-07-25 07:22:29 +02:00
dgtlmoon	e2304b2ce0	Re #154 Ldjson extract parse (#158 ) * Use parsable JSON hiding in <script type="application/ld+json"> where possible, if it matches the filter rule, use it. * Update README.md	2021-07-25 07:02:19 +02:00
dgtlmoon	b87236ea20	Responsive fix for input field on mobile	2021-07-22 21:39:41 +10:00
dgtlmoon	dfbc9bfc53	Re #148 - Always set something for {base_url} so we dont send possibly an empty body/title notification which could break some services.	2021-07-22 20:09:42 +10:00
dgtlmoon	f3ba051df4	Add medium-size-desktop class to notification custom title	2021-07-22 20:06:27 +10:00
dgtlmoon	affe39ff98	Notification default: Make sure to use atleast some text here, a blank notification body could be problematic for some services	2021-07-22 20:00:51 +10:00
dgtlmoon	0f5d5e6caf	Re #150 - stop using 'size' across all elements and rely on CSS for a better mobile experience (stops fields from pushing out)	2021-07-22 19:38:10 +10:00
Preston	2a66ac1db0	fix: setting overflow in mobile view (#150 )	2021-07-22 18:54:01 +10:00
dgtlmoon	07308eedbd	Re #121 , #123 - Show the current base_url value	2021-07-22 10:52:29 +10:00
dgtlmoon	750b882546	Re #149 - allow empty timestamp limit for scrub operation	2021-07-22 10:32:42 +10:00
dgtlmoon	1c09407e24	Dont show "new version available" message when password is enabled and user is logged out	2021-07-21 21:47:42 +10:00
dgtlmoon	7e87591ae5	test fix - dont trigger notifications in header test	2021-07-21 20:31:52 +10:00
dgtlmoon	9e6c2bf3e0	Strengthen the notification tests	2021-07-21 20:21:12 +10:00
dgtlmoon	c396cf8176	Re #137 - Adding test to confirm that headers are not repeated	2021-07-21 19:51:12 +10:00
dgtlmoon	b19a037fac	Add debug output to notify loop	2021-07-21 13:13:31 +10:00
dgtlmoon	5cd4a36896	Add note to field	2021-07-21 13:05:30 +10:00
dgtlmoon	aec3531127	Cleanup test helper data before and after running	2021-07-21 12:49:32 +10:00
dgtlmoon	78434114be	Improve debug info	2021-07-21 12:49:22 +10:00
dgtlmoon	f877cbfe8c	0.38.1 tag	2021-07-20 17:57:27 +10:00
dgtlmoon	fe4963ec04	Re #143 - Remove old notification test code, fix form handler (#145 ) * Re #143 - global notification settings box fix - Remove old notification test code, fix form handler, add test	2021-07-20 17:44:01 +10:00
dgtlmoon	32a798128c	Update README.md	2021-07-18 18:15:44 +10:00
dgtlmoon	cf4e294a9c	Re #135 - refactor the quick add widget (#136 ) * Re #135 - refactor the quick add widget * Fix W3C validation issues	2021-07-18 13:26:23 +10:00
Richard Schwab	b008269a70	Partially revert `47e5a7cf09` (#138 ) Copy HTTP headers from the global template instead of updating the global template when fetching a site. fixes #137	2021-07-18 10:12:23 +10:00