WIP

README.md

@@ -13,33 +13,15 @@ _Live your data-life pro-actively, Detect website changes and perform meaningful
 
 - Chrome browser included.
 - Super fast, no registration needed setup.
-- Get started watching and receiving website change notifications straight away.
+- Start watching and receiving change notifications instantly.
 
 
-### Target specific parts of the webpage using the Visual Selector tool.
+Easily see what changed, examine by word, line, or individual character.
 
-Available when connected to a <a href="https://github.com/dgtlmoon/changedetection.io/wiki/Playwright-content-fetcher">playwright content fetcher</a> (included as part of our subscription service)
-
-[<img src="https://raw.githubusercontent.com/dgtlmoon/changedetection.io/master/docs/visualselector-anim.gif" style="max-width:100%;" alt="Self-hosted web page change monitoring context difference "  title="Self-hosted web page change monitoring context difference " />](https://lemonade.changedetection.io/start?src=github)
-
-### Easily see what changed, examine by word, line, or individual character.
-
-[<img src="https://raw.githubusercontent.com/dgtlmoon/changedetection.io/master/docs/screenshot-diff.png" style="max-width:100%;" alt="Self-hosted web page change monitoring context difference "  title="Self-hosted web page change monitoring context difference " />](https://lemonade.changedetection.io/start?src=github)
+<img src="https://raw.githubusercontent.com/dgtlmoon/changedetection.io/master/docs/screenshot-diff.png" style="max-width:100%;" alt="Self-hosted web page change monitoring context difference "  title="Self-hosted web page change monitoring context difference " />
 
 
-### Perform interactive browser steps
-
-Fill in text boxes, click buttons and more, setup your changedetection scenario. 
-
-Using the **Browser Steps** configuration, add basic steps before performing change detection, such as logging into websites, adding a product to a cart, accept cookie logins, entering dates and refining searches.
-
-[<img src="docs/browsersteps-anim.gif" style="max-width:100%;" alt="Self-hosted web page change monitoring context difference "  title="Website change detection with interactive browser steps, login, cookies etc" />](https://lemonade.changedetection.io/start?src=github)
-
-After **Browser Steps** have been run, then visit the **Visual Selector** tab to refine the content you're interested in.
-Requires Playwright to be enabled.
-
-
-### Example use cases
+#### Example use cases
 
 - Products and services have a change in pricing
 - _Out of stock notification_ and _Back In stock notification_
@@ -77,8 +59,27 @@ _Need an actual Chrome runner with Javascript support? We support fetching via W
 
 We [recommend and use Bright Data](https://brightdata.grsm.io/n0r16zf7eivq) global proxy services, Bright Data will match any first deposit up to $100 using our signup link.
 
+## Screenshots
+
 Please :star: star :star: this project and help it grow! https://github.com/dgtlmoon/changedetection.io/
 
+### Target specific parts of the webpage using the Visual Selector tool.
+
+Available when connected to a <a href="https://github.com/dgtlmoon/changedetection.io/wiki/Playwright-content-fetcher">playwright content fetcher</a> (included as part of our subscription service)
+
+<img src="https://raw.githubusercontent.com/dgtlmoon/changedetection.io/master/docs/visualselector-anim.gif" style="max-width:100%;" alt="Self-hosted web page change monitoring context difference "  title="Self-hosted web page change monitoring context difference " />
+
+### Perform interactive browser steps
+
+Fill in text boxes, click buttons and more, setup your changedetection scenario. 
+
+Using the **Browser Steps** configuration, add basic steps before performing change detection, such as logging into websites, adding a product to a cart, accept cookie logins, entering dates and refining searches.
+
+<img src="docs/browsersteps-anim.gif" style="max-width:100%;" alt="Self-hosted web page change monitoring context difference "  title="Website change detection with interactive browser steps, login, cookies etc" />
+
+After **Browser Steps** have been run, then visit the **Visual Selector** tab to refine the content you're interested in.
+Requires Playwright to be enabled.
+
 ## Installation
 
 ### Docker

changedetectionio/__init__.py

@@ -27,7 +27,6 @@ from flask import (
     session,
     url_for,
 )
-from flask_compress import Compress as FlaskCompress
 from flask_login import login_required
 from flask_restful import abort, Api
 from flask_wtf import CSRFProtect
@@ -52,10 +51,6 @@ app = Flask(__name__,
             static_url_path="",
             static_folder="static",
             template_folder="templates")
-from flask_compress import Compress
-
-# Super handy for compressing large BrowserSteps responses and others
-FlaskCompress(app)
 
 # Stop browser caching of assets
 app.config['SEND_FILE_MAX_AGE_DEFAULT'] = 0

changedetectionio/blueprint/browser_steps/__init__.py

@@ -21,6 +21,9 @@
 # OR
 # - use multiprocessing to bump this over to its own process and add some transport layer (queue/pipes)
 
+
+
+
 from distutils.util import strtobool
 from flask import Blueprint, request, make_response
 from flask_login import login_required
@@ -30,35 +33,38 @@ from changedetectionio.store import ChangeDetectionStore
 
 browsersteps_live_ui_o = {}
 browsersteps_playwright_browser_interface = None
-browsersteps_playwright_browser_interface_browser = None
-browsersteps_playwright_browser_interface_context = None
-browsersteps_playwright_browser_interface_end_time = None
 browsersteps_playwright_browser_interface_start_time = None
+browsersteps_playwright_browser_interface_browser = None
+browsersteps_playwright_browser_interface_end_time = None
+
 
 def cleanup_playwright_session():
-
-    global browsersteps_live_ui_o
+    print("Cleaning up old playwright session because time was up")
     global browsersteps_playwright_browser_interface
+    global browsersteps_live_ui_o
     global browsersteps_playwright_browser_interface_browser
-    global browsersteps_playwright_browser_interface_context
-    global browsersteps_playwright_browser_interface_end_time
+    global browsersteps_playwright_browser_interface
     global browsersteps_playwright_browser_interface_start_time
+    global browsersteps_playwright_browser_interface_end_time
+
+    import psutil
+
+    current_process = psutil.Process()
+    children = current_process.children(recursive=True)
+    for child in children:
+        print (child)
+        print('Child pid is {}'.format(child.pid))
+
+    # .stop() hangs sometimes if its called when there are no children to process
+    # but how do we know this is our child? dunno
+    if children:
+        browsersteps_playwright_browser_interface.stop()
 
     browsersteps_live_ui_o = {}
+    browsersteps_playwright_browser_interface = None
+    browsersteps_playwright_browser_interface_start_time = None
     browsersteps_playwright_browser_interface_browser = None
     browsersteps_playwright_browser_interface_end_time = None
-    browsersteps_playwright_browser_interface_start_time = None
-    print("Cleaning up old playwright session because time was up, calling .goodbye()")
-
-    try:
-        browsersteps_playwright_browser_interface_context.goodbye()
-    except Exception as e:
-        print ("Got exception in shutdown, probably OK")
-        print (str(e))
-
-    browsersteps_playwright_browser_interface = None
-    browsersteps_playwright_browser_interface_context = None
-
     print ("Cleaning up old playwright session because time was up - done")
 
 def construct_blueprint(datastore: ChangeDetectionStore):
@@ -93,8 +99,12 @@ def construct_blueprint(datastore: ChangeDetectionStore):
         if browsersteps_playwright_browser_interface_end_time:
             remaining = browsersteps_playwright_browser_interface_end_time-time.time()
             if browsersteps_playwright_browser_interface_end_time and remaining <= 0:
+
+
                 cleanup_playwright_session()
-                return make_response('Browser session expired, please reload the Browser Steps interface', 401)
+
+                return make_response('Browser session expired, please reload the Browser Steps interface', 500)
+
 
         # Actions - step/apply/etc, do the thing and return state
         if request.method == 'POST':
@@ -121,11 +131,21 @@ def construct_blueprint(datastore: ChangeDetectionStore):
                 this_session.call_action(action_name=step_operation,
                                          selector=step_selector,
                                          optional_value=step_optional_value)
+            except playwright._impl._api_types.TimeoutError as e:
+                print("Element wasnt found :-(", step_operation)
+                return make_response("Element was not found on page", 401)
+
+            except playwright._impl._api_types.Error as e:
+                # Browser/playwright level error
+                print("Browser error - got playwright._impl._api_types.Error, try reloading the session/browser")
+                print (str(e))
 
-            except Exception as e:
-                print("Exception when calling step operation", step_operation, str(e))
                 # Try to find something of value to give back to the user
-                return make_response(str(e).splitlines()[0], 401)
+                for l in str(e).splitlines():
+                    if 'DOMException' in l:
+                        return make_response(l, 401)
+
+                return make_response('Browser session ran out of time :( Please reload this page.', 401)
 
             # Get visual selector ready/update its data (also use the current filter info from the page?)
             # When the last 'apply' button was pressed
@@ -142,11 +162,10 @@ def construct_blueprint(datastore: ChangeDetectionStore):
             if not browsersteps_playwright_browser_interface:
                 print("Starting connection with playwright")
                 logging.debug("browser_steps.py connecting")
+                from playwright.sync_api import sync_playwright
+
+                browsersteps_playwright_browser_interface = sync_playwright().start()
 
-                global browsersteps_playwright_browser_interface_context
-                from . import nonContext
-                browsersteps_playwright_browser_interface_context = nonContext.c_sync_playwright()
-                browsersteps_playwright_browser_interface = browsersteps_playwright_browser_interface_context.start()
 
                 time.sleep(1)
                 # At 20 minutes, some other variable is closing it
@@ -186,45 +205,21 @@ def construct_blueprint(datastore: ChangeDetectionStore):
             cleanup_playwright_session()
             return make_response('Browser session ran out of time :( Please reload this page.', 401)
 
-        response = None
+        try:
+            state = this_session.get_current_state()
+        except playwright._impl._api_types.Error as e:
+            return make_response("Browser session ran out of time :( Please reload this page."+str(e), 401)
 
-        if request.method == 'POST':
-            # Screenshots and other info only needed on requesting a step (POST)
-            try:
-                state = this_session.get_current_state()
-            except playwright._impl._api_types.Error as e:
-                return make_response("Browser session ran out of time :( Please reload this page."+str(e), 401)
+        p = {'screenshot': "data:image/png;base64,{}".format(
+            base64.b64encode(state[0]).decode('ascii')),
+            'xpath_data': state[1],
+            'session_age_start': this_session.age_start,
+            'browser_time_remaining': round(remaining)
+        }
 
-            # Use send_file() which is way faster than read/write loop on bytes
-            import json
-            from tempfile import mkstemp
-            from flask import send_file
-            tmp_fd, tmp_file = mkstemp(text=True, suffix=".json", prefix="changedetectionio-")
 
-            output = json.dumps({'screenshot': "data:image/jpeg;base64,{}".format(
-                base64.b64encode(state[0]).decode('ascii')),
-                'xpath_data': state[1],
-                'session_age_start': this_session.age_start,
-                'browser_time_remaining': round(remaining)
-            })
-
-            with os.fdopen(tmp_fd, 'w') as f:
-                f.write(output)
-
-            response = make_response(send_file(path_or_file=tmp_file,
-                                               mimetype='application/json; charset=UTF-8',
-                                               etag=True))
-            # No longer needed
-            os.unlink(tmp_file)
-
-        elif request.method == 'GET':
-            # Just enough to get the session rolling, it will call for goto-site via POST next
-            response = make_response({
-                'session_age_start': this_session.age_start,
-                'browser_time_remaining': round(remaining)
-            })
-
-        return response
+        # @todo BSON/binary JSON, faster xfer, OR pick it off the disk
+        return p
 
     return browser_steps_blueprint
 

changedetectionio/blueprint/browser_steps/browser_steps.py

@@ -22,7 +22,6 @@ browser_step_ui_config = {'Choose one': '0 0',
                           'Click element': '1 0',
                           'Click element containing text': '0 1',
                           'Enter text in field': '1 1',
-                          'Execute JS': '0 1',
 #                          'Extract text and use as filter': '1 0',
                           'Goto site': '0 0',
                           'Press Enter': '0 0',
@@ -90,7 +89,7 @@ class steppable_browser_interface():
             return
         elem = self.page.get_by_text(value)
         if elem.count():
-            elem.first.click(delay=randint(200, 500), timeout=3000)
+            elem.first.click(delay=randint(200, 500))
 
     def action_enter_text_in_field(self, selector, value):
         if not len(selector.strip()):
@@ -98,9 +97,6 @@ class steppable_browser_interface():
 
         self.page.fill(selector, value, timeout=10 * 1000)
 
-    def action_execute_js(self, selector, value):
-        self.page.evaluate(value)
-
     def action_click_element(self, selector, value):
         print("Clicking element")
         if not len(selector.strip()):
@@ -146,10 +142,10 @@ class steppable_browser_interface():
         self.page.keyboard.press("PageDown", delay=randint(200, 500))
 
     def action_check_checkbox(self, selector, value):
-        self.page.locator(selector).check(timeout=1000)
+        self.page.locator(selector).check()
 
     def action_uncheck_checkbox(self, selector, value):
-        self.page.locator(selector, timeout=1000).uncheck(timeout=1000)
+        self.page.locator(selector).uncheck()
 
 
 # Responsible for maintaining a live 'context' with browserless
@@ -211,7 +207,7 @@ class browsersteps_live_ui(steppable_browser_interface):
         # Listen for all console events and handle errors
         self.page.on("console", lambda msg: print(f"Browser steps console - {msg.type}: {msg.text} {msg.args}"))
 
-        print("Time to browser setup", time.time() - now)
+        print("time to browser setup", time.time() - now)
         self.page.wait_for_timeout(1 * 1000)
 
     def mark_as_closed(self):
@@ -236,7 +232,7 @@ class browsersteps_live_ui(steppable_browser_interface):
         self.page.evaluate("var include_filters=''")
         # Go find the interactive elements
         # @todo in the future, something smarter that can scan for elements with .click/focus etc event handlers?
-        elements = 'a,button,input,select,textarea,i,th,td,p,li,h1,h2,h3,h4,div,span'
+        elements = 'a,button,input,select,textarea,i,th,td,p,li,h1,h2,h3,h4'
         xpath_element_js = xpath_element_js.replace('%ELEMENTS%', elements)
         xpath_data = self.page.evaluate("async () => {" + xpath_element_js + "}")
         # So the JS will find the smallest one first

changedetectionio/blueprint/browser_steps/nonContext.py

@@ -1,18 +0,0 @@
-from playwright.sync_api import PlaywrightContextManager
-import asyncio
-
-# So playwright wants to run as a context manager, but we do something horrible and hacky
-# we are holding the session open for as long as possible, then shutting it down, and opening a new one
-# So it means we don't get to use PlaywrightContextManager' __enter__ __exit__
-# To work around this, make goodbye() act the same as the __exit__()
-#
-# But actually I think this is because the context is opened correctly with __enter__() but we timeout the connection
-# then theres some lock condition where we cant destroy it without it hanging
-
-class c_PlaywrightContextManager(PlaywrightContextManager):
-
-    def goodbye(self) -> None:
-        self.__exit__()
-
-def c_sync_playwright() -> PlaywrightContextManager:
-    return c_PlaywrightContextManager()

changedetectionio/blueprint/extract/__init__.py

@@ -0,0 +1,24 @@
+
+from distutils.util import strtobool
+from flask import Blueprint, request, make_response
+from flask_login import login_required
+import os
+import logging
+from changedetectionio.store import ChangeDetectionStore
+
+
+
+def construct_blueprint(datastore: ChangeDetectionStore):
+
+    browser_steps_blueprint = Blueprint('browser_steps', __name__, template_folder="templates")
+
+    @login_required
+    @browser_steps_blueprint.route("/extract-regex", methods=['POST'])
+    def browsersteps_ui_update():
+        import time
+
+        return {123123123: 'yup'}
+
+    return browser_steps_blueprint
+
+

changedetectionio/forms.py

@@ -400,15 +400,6 @@ class watchForm(commonSettingsForm):
             self.body.errors.append('Body must be empty when Request Method is set to GET')
             result = False
 
-        # Attempt to validate jinja2 templates in the URL
-        from jinja2 import Environment
-        # Jinja2 available in URLs along with https://pypi.org/project/jinja2-time/
-        jinja2_env = Environment(extensions=['jinja2_time.TimeExtension'])
-        try:
-            ready_url = str(jinja2_env.from_string(self.url.data).render())
-        except Exception as e:
-            self.url.errors.append('Invalid template syntax')
-            result = False
         return result
 
 

changedetectionio/model/Watch.py

@@ -93,23 +93,12 @@ class model(dict):
     @property
     def link(self):
         url = self.get('url', '')
-        ready_url = url
         if '{%' in url or '{{' in url:
             from jinja2 import Environment
             # Jinja2 available in URLs along with https://pypi.org/project/jinja2-time/
             jinja2_env = Environment(extensions=['jinja2_time.TimeExtension'])
-            try:
-                ready_url = str(jinja2_env.from_string(url).render())
-            except Exception as e:
-                from flask import (
-                    flash, Markup, url_for
-                )
-                message = Markup('<a href="{}#general">The URL {} is invalid and cannot be used, click to edit</a>'.format(
-                    url_for('edit_page', uuid=self.get('uuid')), self.get('url', '')))
-                flash(message, 'error')
-                return ''
-
-        return ready_url
+            return str(jinja2_env.from_string(url).render())
+        return url
 
     @property
     def label(self):

changedetectionio/res/xpath_element_scraper.js

@@ -1,13 +1,3 @@
-// @file Scrape the page looking for elements of concern (%ELEMENTS%)
-// http://matatk.agrip.org.uk/tests/position-and-width/
-// https://stackoverflow.com/questions/26813480/when-is-element-getboundingclientrect-guaranteed-to-be-updated-accurate
-//
-// Some pages like https://www.londonstockexchange.com/stock/NCCL/ncondezi-energy-limited/analysis
-// will automatically force a scroll somewhere, so include the position offset
-// Lets hope the position doesnt change while we iterate the bbox's, but this is better than nothing
-
-var scroll_y=+document.documentElement.scrollTop || document.body.scrollTop
-
 // Include the getXpath script directly, easier than fetching
 function getxpath(e) {
         var n = e;
@@ -81,13 +71,8 @@ var bbox;
 for (var i = 0; i < elements.length; i++) {
     bbox = elements[i].getBoundingClientRect();
 
-    // Forget really small ones
-    if (bbox['width'] < 10 && bbox['height'] < 10) {
-        continue;
-    }
-
-    // Don't include elements that are offset from canvas
-    if (bbox['top']+scroll_y < 0 || bbox['left'] < 0) {
+    // forget really small ones
+    if (bbox['width'] < 15 && bbox['height'] < 15) {
         continue;
     }
 
@@ -124,16 +109,14 @@ for (var i = 0; i < elements.length; i++) {
         continue;
     }
 
-    // @todo Possible to ONLY list where it's clickable to save JSON xfer size
     size_pos.push({
         xpath: xpath_result,
         width: Math.round(bbox['width']),
         height: Math.round(bbox['height']),
         left: Math.floor(bbox['left']),
-        top: Math.floor(bbox['top'])+scroll_y,
+        top: Math.floor(bbox['top']),
         tagName: (elements[i].tagName) ? elements[i].tagName.toLowerCase() : '',
-        tagtype: (elements[i].tagName == 'INPUT' && elements[i].type) ? elements[i].type.toLowerCase() : '',
-        isClickable: (elements[i].onclick) || window.getComputedStyle(elements[i]).cursor == "pointer"
+        tagtype: (elements[i].tagName == 'INPUT' && elements[i].type) ? elements[i].type.toLowerCase() : ''
     });
 
 }
@@ -167,7 +150,6 @@ if (include_filters.length) {
 
         if (q) {
             bbox = q.getBoundingClientRect();
-            console.log("xpath_element_scraper: Got filter element, scroll from top was "+scroll_y)
         } else {
             console.log("xpath_element_scraper: filter element "+f+" was not found");
         }
@@ -175,10 +157,10 @@ if (include_filters.length) {
         if (bbox && bbox['width'] > 0 && bbox['height'] > 0) {
             size_pos.push({
                 xpath: f,
-                width: parseInt(bbox['width']),
-                height: parseInt(bbox['height']),
-                left: parseInt(bbox['left']),
-                top: parseInt(bbox['top'])+scroll_y
+                width: Math.round(bbox['width']),
+                height: Math.round(bbox['height']),
+                left: Math.floor(bbox['left']),
+                top: Math.floor(bbox['top'])
             });
         }
     }

changedetectionio/static/favicons/browserconfig.xml

@@ -1,9 +0,0 @@
-<?xml version="1.0" encoding="utf-8"?>
-<browserconfig>
-    <msapplication>
-        <tile>
-            <square150x150logo src="favicons/mstile-150x150.png"/>
-            <TileColor>#da532c</TileColor>
-        </tile>
-    </msapplication>
-</browserconfig>

changedetectionio/static/favicons/safari-pinned-tab.svg

@@ -1,35 +0,0 @@
-<?xml version="1.0" standalone="no"?>
-<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 20010904//EN"
- "http://www.w3.org/TR/2001/REC-SVG-20010904/DTD/svg10.dtd">
-<svg version="1.0" xmlns="http://www.w3.org/2000/svg"
- width="256.000000pt" height="256.000000pt" viewBox="0 0 256.000000 256.000000"
- preserveAspectRatio="xMidYMid meet">
-<metadata>
-Created by potrace 1.14, written by Peter Selinger 2001-2017
-</metadata>
-<g transform="translate(0.000000,256.000000) scale(0.100000,-0.100000)"
-fill="#000000" stroke="none">
-<path d="M0 1280 l0 -1280 1280 0 1280 0 0 1280 0 1280 -1280 0 -1280 0 0
--1280z m1555 936 c387 -112 675 -426 741 -810 24 -138 15 -352 -20 -470 -106
--353 -360 -606 -713 -712 -75 -22 -113 -27 -253 -31 -144 -5 -176 -2 -252 16
--316 75 -564 271 -707 557 -67 136 -92 237 -98 401 -7 164 5 253 47 378 106
-315 349 556 665 659 114 37 180 45 350 41 125 -2 165 -7 240 -29z"/>
-<path d="M1091 2165 c-364 -82 -629 -328 -738 -682 -24 -80 -27 -103 -27 -258
--1 -146 2 -182 21 -251 74 -271 259 -497 508 -621 477 -238 1061 -35 1294 450
-61 126 83 220 88 379 7 194 -15 307 -93 461 -126 251 -340 428 -614 507 -99
-29 -343 37 -439 15z m829 -473 c55 -54 100 -106 100 -116 0 -21 -184 -213
--212 -222 -24 -7 -48 12 -48 38 0 11 26 47 58 80 l57 60 -151 -3 c-145 -4
--152 -5 -190 -31 -22 -15 -78 -73 -124 -128 l-85 -99 -32 31 -32 31 30 38 c17
-22 70 79 117 128 66 67 97 92 127 100 22 6 106 11 188 11 81 0 147 3 147 8 0
-4 -25 31 -55 61 -55 55 -65 77 -43 99 25 25 50 10 148 -86z m-1002 -101 c46
--24 141 -121 312 -321 203 -236 290 -330 322 -346 22 -11 60 -14 169 -12 l141
-3 -51 58 c-28 32 -51 64 -51 71 0 18 21 36 43 36 24 0 217 -193 217 -217 0
--19 -185 -210 -212 -219 -24 -7 -48 12 -48 38 0 10 23 43 50 72 l50 53 -52 7
-c-29 3 -93 6 -142 6 -104 0 -152 12 -200 52 -19 15 -135 144 -258 286 -274
-316 -305 347 -354 361 -22 6 -94 11 -161 11 -67 0 -128 3 -137 6 -22 9 -21 61
-2 67 9 3 86 5 170 6 133 1 158 -2 190 -18z m227 -468 c23 -34 17 -43 -103
--172 -119 -128 -131 -133 -343 -129 l-154 3 0 35 c0 34 1 35 50 42 28 3 96 7
-153 7 64 1 115 6 136 15 20 8 71 56 127 120 52 58 99 106 105 106 7 0 20 -12
-29 -27z"/>
-</g>
-</svg>

changedetectionio/static/favicons/site.webmanifest

@@ -1,19 +0,0 @@
-{
-    "name": "",
-    "short_name": "",
-    "icons": [
-        {
-            "src": "android-chrome-192x192.png",
-            "sizes": "192x192",
-            "type": "image/png"
-        },
-        {
-            "src": "android-chrome-256x256.png",
-            "sizes": "256x256",
-            "type": "image/png"
-        }
-    ],
-    "theme_color": "#ffffff",
-    "background_color": "#ffffff",
-    "display": "standalone"
-}

changedetectionio/static/js/browser-steps.js

@@ -10,10 +10,10 @@ $(document).ready(function () {
         }
     })
     var browsersteps_session_id;
-    var browserless_seconds_remaining = 0;
+    var browserless_seconds_remaining=0;
     var apply_buttons_disabled = false;
     var include_text_elements = $("#include_text_elements");
-    var xpath_data = false;
+    var xpath_data;
     var current_selected_i;
     var state_clicked = false;
     var c;
@@ -25,42 +25,11 @@ $(document).ready(function () {
     $(window).resize(function () {
         set_scale();
     });
-    // Should always be disabled
-    $('#browser_steps >li:first-child select').val('Goto site').attr('disabled', 'disabled');
-
-    $('#browsersteps-click-start').click(function () {
-        $("#browsersteps-click-start").fadeOut();
-        $("#browsersteps-selector-wrapper .spinner").fadeIn();
-        start();
-    });
 
     $('a#browsersteps-tab').click(function () {
-        reset();
+        start();
     });
 
-    window.addEventListener('hashchange', function () {
-        if (window.location.hash == '#browser-steps') {
-            reset();
-        }
-    });
-
-    function reset() {
-        xpath_data = false;
-        $('#browsersteps-img').removeAttr('src');
-        $("#browsersteps-click-start").show();
-        $("#browsersteps-selector-wrapper .spinner").hide();
-        browserless_seconds_remaining = 0;
-        browsersteps_session_id = false;
-        apply_buttons_disabled = false;
-        ctx.clearRect(0, 0, c.width, c.height);
-        set_first_gotosite_disabled();
-    }
-
-    function set_first_gotosite_disabled() {
-        $('#browser_steps >li:first-child select').val('Goto site').attr('disabled', 'disabled');
-        $('#browser_steps >li:first-child').css('opacity', '0.5');
-    }
-
     // Show seconds remaining until playwright/browserless needs to restart the session
     // (See comment at the top of changedetectionio/blueprint/browser_steps/__init__.py )
     setInterval(() => {
@@ -71,6 +40,21 @@ $(document).ready(function () {
     }, "1000")
 
 
+    if (window.location.hash == '#browser-steps') {
+        start();
+    }
+
+    window.addEventListener('hashchange', function () {
+        if (window.location.hash == '#browser-steps') {
+            start();
+        }
+        // For when the page loads
+        if (!window.location.hash || window.location.hash != '#browser-steps') {
+            $("img#browsersteps-img").attr('src', '');
+            return;
+        }
+    });
+
     function set_scale() {
 
         // some things to check if the scaling doesnt work
@@ -103,6 +87,7 @@ $(document).ready(function () {
         // @todo is click better?
         $('#browsersteps-selector-canvas').off("mousemove mousedown click");
         // Undo disable_browsersteps_ui
+        $("#browser_steps select,input").removeAttr('disabled').css('opacity', '1.0');
         $("#browser-steps-ui").css('opacity', '1.0');
 
         // init
@@ -118,7 +103,7 @@ $(document).ready(function () {
             // https://developer.mozilla.org/en-US/docs/Web/API/MouseEvent
             e.preventDefault()
             console.log(e);
-            console.log("current xpath in index is " + current_selected_i);
+            console.log("current xpath in index is "+current_selected_i);
             last_click_xy = {'x': parseInt((1 / x_scale) * e.offsetX), 'y': parseInt((1 / y_scale) * e.offsetY)}
             process_selected(current_selected_i);
             current_selected_i = false;
@@ -133,10 +118,6 @@ $(document).ready(function () {
         });
 
         $('#browsersteps-selector-canvas').bind('mousemove', function (e) {
-            if (!xpath_data) {
-                return;
-            }
-
             // checkbox if find elements is enabled
             ctx.clearRect(0, 0, c.width, c.height);
             ctx.fillStyle = 'rgba(255,0,0, 0.1)';
@@ -172,8 +153,8 @@ $(document).ready(function () {
                         // does it mean sort the xpath list by size (w*h) i think so!
                     } else {
 
-                        if (include_text_elements[0].checked === true) {
-                            // blue one with background instead?
+                        if ( include_text_elements[0].checked === true) {
+                        // blue one with background instead?
                             ctx.fillStyle = 'rgba(0,0,255, 0.1)';
                             ctx.strokeStyle = 'rgba(0,0,200, 0.7)';
                             $('#browsersteps-selector-canvas').css('cursor', 'grab');
@@ -194,6 +175,7 @@ $(document).ready(function () {
     // });
 
 
+
     // callback for clicking on an xpath on the canvas
     function process_selected(xpath_data_index) {
         found_something = false;
@@ -208,23 +190,25 @@ $(document).ready(function () {
             console.log(x);
             if (x && first_available.length) {
                 // @todo will it let you click shit that has a layer ontop? probably not.
-                if (x['tagtype'] === 'text' || x['tagtype'] === 'email' || x['tagName'] === 'textarea' || x['tagtype'] === 'password' || x['tagtype'] === 'search') {
+                if (x['tagtype'] === 'text' || x['tagtype'] === 'email' || x['tagName'] === 'textarea' || x['tagtype'] === 'password' || x['tagtype'] === 'search' ) {
                     $('select', first_available).val('Enter text in field').change();
                     $('input[type=text]', first_available).first().val(x['xpath']);
                     $('input[placeholder="Value"]', first_available).addClass('ok').click().focus();
                     found_something = true;
                 } else {
-                    if (x['isClickable'] || x['tagName'].startsWith('h') || x['tagName'] === 'a' || x['tagName'] === 'button' || x['tagtype'] === 'submit' || x['tagtype'] === 'checkbox' || x['tagtype'] === 'radio' || x['tagtype'] === 'li') {
+                    // Assume it's just for clicking on
+                    // what are we clicking on?
+                    if (x['tagName'].startsWith('h')|| x['tagName'] === 'a' || x['tagName'] === 'button' || x['tagtype'] === 'submit'|| x['tagtype'] === 'checkbox'|| x['tagtype'] === 'radio'|| x['tagtype'] === 'li') {
                         $('select', first_available).val('Click element').change();
                         $('input[type=text]', first_available).first().val(x['xpath']);
                         found_something = true;
                     }
                 }
 
-                first_available.xpath_data_index = xpath_data_index;
+                first_available.xpath_data_index=xpath_data_index;
 
                 if (!found_something) {
-                    if (include_text_elements[0].checked === true) {
+                    if ( include_text_elements[0].checked === true) {
                         // Suggest that we use as filter?
                         // @todo filters should always be in the last steps, nothing non-filter after it
                         found_something = true;
@@ -236,6 +220,8 @@ $(document).ready(function () {
                     }
                 }
             }
+        } else {
+
         }
     }
 
@@ -248,15 +234,15 @@ $(document).ready(function () {
 
     function start() {
         console.log("Starting browser-steps UI");
-        browsersteps_session_id = Date.now();
+        browsersteps_session_id=Date.now();
         // @todo This setting of the first one should be done at the datalayer but wtforms doesnt wanna play nice
         $('#browser_steps >li:first-child').removeClass('empty');
-        set_first_gotosite_disabled();
-        $('#browser-steps-ui .loader .spinner').show();
+        $('#browser_steps >li:first-child select').val('Goto site').attr('disabled', 'disabled');
+        $('#browser-steps-ui .loader').show();
         $('.clear,.remove', $('#browser_steps >li:first-child')).hide();
         $.ajax({
             type: "GET",
-            url: browser_steps_sync_url + "&browsersteps_session_id=" + browsersteps_session_id,
+            url: browser_steps_sync_url+"&browsersteps_session_id="+browsersteps_session_id,
             statusCode: {
                 400: function () {
                     // More than likely the CSRF token was lost when the server restarted
@@ -265,12 +251,10 @@ $(document).ready(function () {
             }
         }).done(function (data) {
             xpath_data = data.xpath_data;
-            $("#loading-status-text").fadeIn();
+            $('#browsersteps-img').attr('src', data.screenshot);
             // This should trigger 'Goto site'
-            console.log("Got startup response, requesting Goto-Site (first) step fake click");
             $('#browser_steps >li:first-child .apply').click();
             browserless_seconds_remaining = data.browser_time_remaining;
-            set_first_gotosite_disabled();
         }).fail(function (data) {
             console.log(data);
             alert('There was an error communicating with the server.');
@@ -279,7 +263,7 @@ $(document).ready(function () {
     }
 
     function disable_browsersteps_ui() {
-        set_first_gotosite_disabled();
+        $("#browser_steps select,input").attr('disabled', 'disabled').css('opacity', '0.5');
         $("#browser-steps-ui").css('opacity', '0.3');
         $('#browsersteps-selector-canvas').off("mousemove mousedown click");
     }
@@ -326,14 +310,11 @@ $(document).ready(function () {
 
     // Add the extra buttons to the steps
     $('ul#browser_steps li').each(function (i) {
-            var s = '<div class="control">' + '<a data-step-index=' + i + ' class="pure-button button-secondary button-green button-xsmall apply" >Apply</a>&nbsp;';
-            if (i > 0) {
-                // The first step never gets these (Goto-site)
-                s += '<a data-step-index=' + i + ' class="pure-button button-secondary button-xsmall clear" >Clear</a>&nbsp;' +
-                    '<a data-step-index=' + i + ' class="pure-button button-secondary button-red button-xsmall remove" >Remove</a>';
-            }
-            s += '</div>';
-            $(this).append(s)
+            $(this).append('<div class="control">' +
+                '<a data-step-index=' + i + ' class="pure-button button-secondary button-green button-xsmall apply" >Apply</a>&nbsp;' +
+                '<a data-step-index=' + i + ' class="pure-button button-secondary button-xsmall clear" >Clear</a>&nbsp;' +
+                '<a data-step-index=' + i + ' class="pure-button button-secondary button-red button-xsmall remove" >Remove</a>' +
+                '</div>')
         }
     );
 
@@ -370,15 +351,15 @@ $(document).ready(function () {
 
     $('ul#browser_steps li .control .apply').click(function (event) {
         // sequential requests @todo refactor
-        if (apply_buttons_disabled) {
+        if(apply_buttons_disabled) {
             return;
         }
 
         var current_data = $(event.currentTarget).closest('li');
-        $('#browser-steps-ui .loader .spinner').fadeIn();
-        apply_buttons_disabled = true;
-        $('ul#browser_steps li .control .apply').css('opacity', 0.5);
-        $("#browsersteps-img").css('opacity', 0.65);
+        $('#browser-steps-ui .loader').fadeIn();
+        apply_buttons_disabled=true;
+        $('ul#browser_steps li .control .apply').css('opacity',0.5);
+        $("#browsersteps-img").css('opacity',0.65);
 
         var is_last_step = 0;
         var step_n = $(event.currentTarget).data('step-index');
@@ -390,17 +371,17 @@ $(document).ready(function () {
             }
         });
 
-        if (is_last_step == (step_n + 1)) {
+        if (is_last_step == (step_n+1)) {
             is_last_step = true;
         } else {
             is_last_step = false;
         }
 
-        console.log("Requesting step via POST " + $("select[id$='operation']", current_data).first().val());
+
         // POST the currently clicked step form widget back and await response, redraw
         $.ajax({
             method: "POST",
-            url: browser_steps_sync_url + "&browsersteps_session_id=" + browsersteps_session_id,
+            url: browser_steps_sync_url+"&browsersteps_session_id="+browsersteps_session_id,
             data: {
                 'operation': $("select[id$='operation']", current_data).first().val(),
                 'selector': $("input[id$='selector']", current_data).first().val(),
@@ -412,36 +393,26 @@ $(document).ready(function () {
                 400: function () {
                     // More than likely the CSRF token was lost when the server restarted
                     alert("There was a problem processing the request, please reload the page.");
-                    $("#loading-status-text").hide();
-                    $('#browser-steps-ui .loader .spinner').fadeOut();
-                },
-                401: function (data) {
-                    // More than likely the CSRF token was lost when the server restarted
-                    alert(data.responseText);
-                    $("#loading-status-text").hide();
-                    $('#browser-steps-ui .loader .spinner').fadeOut();
                 }
             }
         }).done(function (data) {
             // it should return the new state (selectors available and screenshot)
             xpath_data = data.xpath_data;
             $('#browsersteps-img').attr('src', data.screenshot);
-            $('#browser-steps-ui .loader .spinner').fadeOut();
-            apply_buttons_disabled = false;
-            $("#browsersteps-img").css('opacity', 1);
-            $('ul#browser_steps li .control .apply').css('opacity', 1);
+            $('#browser-steps-ui .loader').fadeOut();
+            apply_buttons_disabled=false;
+            $("#browsersteps-img").css('opacity',1);
+            $('ul#browser_steps li .control .apply').css('opacity',1);
             browserless_seconds_remaining = data.browser_time_remaining;
-            $("#loading-status-text").hide();
-            set_first_gotosite_disabled();
         }).fail(function (data) {
             console.log(data);
             if (data.responseText.includes("Browser session expired")) {
                 disable_browsersteps_ui();
             }
-            apply_buttons_disabled = false;
-            $("#loading-status-text").hide();
-            $('ul#browser_steps li .control .apply').css('opacity', 1);
-            $("#browsersteps-img").css('opacity', 1);
+            apply_buttons_disabled=false;
+            $('ul#browser_steps li .control .apply').css('opacity',1);
+            $("#browsersteps-img").css('opacity',1);
+            //$('#browsersteps-selector-wrapper .loader').fadeOut(2500);
         });
 
     });

changedetectionio/static/js/extract.js

@@ -0,0 +1,40 @@
+$(document).ready(function () {
+
+    $('#extract').click(function (e) {
+        download_csv_file();
+    });
+
+//create CSV file data in an array
+    var csvFileData = [
+        ['Alan Walker', 'Singer'],
+        ['Cristiano Ronaldo', 'Footballer'],
+        ['Saina Nehwal', 'Badminton Player'],
+        ['Arijit Singh', 'Singer'],
+        ['Terence Lewis', 'Dancer']
+    ];
+
+//create a user-defined function to download CSV file
+    function download_csv_file() {
+
+        //define the heading for each row of the data
+        var csv = 'Name,Profession\n';
+
+        //merge the data with CSV
+        csvFileData.forEach(function (row) {
+            csv += row.join(',');
+            csv += "\n";
+        });
+
+        //display the created CSV data on the web browser
+        document.write(csv);
+
+
+        var hiddenElement = document.createElement('a');
+        hiddenElement.href = 'data:text/csv;charset=utf-8,' + encodeURI(csv);
+        hiddenElement.target = '_blank';
+        //provide the name for the CSV file to be downloaded
+        hiddenElement.download = 'Famous Personalities.csv';
+        hiddenElement.click();
+    }
+
+});

changedetectionio/static/styles/parts/browser-steps.scss

@@ -6,11 +6,6 @@
   }
 
   li {
-    &:not(:first-child) {
-      &:hover {
-        opacity: 1.0;
-      }
-    }
     list-style: decimal;
     padding: 5px;
     .control {
@@ -75,8 +70,6 @@
     transform: translate(-50%, -50%);
     margin-left: -40px;
     z-index: 100;
-    max-width: 350px;
-    text-align: center;
   }
 
   /* nice tall skinny one */
@@ -85,10 +78,4 @@
     height: 80px;
     font-size: 3px;
   }
-
-  #browsersteps-click-start {
-    &:hover {
-      cursor: pointer;
-    }
-  }
 }

changedetectionio/static/styles/styles.css

@@ -50,8 +50,6 @@ nvm use v14.18.1 && npm install && npm run build
   #browser_steps li {
     list-style: decimal;
     padding: 5px; }
-    #browser_steps li:not(:first-child):hover {
-      opacity: 1.0; }
     #browser_steps li .control {
       padding-left: 5px;
       padding-right: 5px; }
@@ -98,15 +96,11 @@ nvm use v14.18.1 && npm install && npm run build
     top: 50%;
     transform: translate(-50%, -50%);
     margin-left: -40px;
-    z-index: 100;
-    max-width: 350px;
-    text-align: center; }
+    z-index: 100; }
   #browsersteps-selector-wrapper .spinner, #browsersteps-selector-wrapper .spinner:after {
     width: 80px;
     height: 80px;
     font-size: 3px; }
-  #browsersteps-selector-wrapper #browsersteps-click-start:hover {
-    cursor: pointer; }
 
 .arrow {
   border: solid #1b98f8;

changedetectionio/templates/base.html

@@ -13,17 +13,6 @@
         <link rel="stylesheet" href="{{ m }}?ver=1000">
         {% endfor %}
     {% endif %}
-
-    <link rel="apple-touch-icon" sizes="180x180" href="{{url_for('static_content', group='favicons', filename='apple-touch-icon.png')}}">
-    <link rel="icon" type="image/png" sizes="32x32" href="{{url_for('static_content', group='favicons', filename='favicon-32x32.png')}}">
-    <link rel="icon" type="image/png" sizes="16x16" href="{{url_for('static_content', group='favicons', filename='favicon-16x16.png')}}">
-    <link rel="manifest" href="{{url_for('static_content', group='favicons', filename='site.webmanifest')}}">
-    <link rel="mask-icon" href="{{url_for('static_content', group='favicons', filename='safari-pinned-tab.svg')}}" color="#5bbad5">
-    <link rel="shortcut icon" href="{{url_for('static_content', group='favicons', filename='favicon.ico')}}">
-    <meta name="msapplication-TileColor" content="#da532c">
-    <meta name="msapplication-config" content="favicons/browserconfig.xml">
-    <meta name="theme-color" content="#ffffff">
-
     <style>
     body::before {
         background-image: url({{url_for('static_content', group='images', filename='gradient-border.png')}});

changedetectionio/templates/edit.html

@@ -23,6 +23,7 @@
 <script type="text/javascript" src="{{url_for('static_content', group='js', filename='visual-selector.js')}}" defer></script>
 {% if playwright_enabled %}
 <script type="text/javascript" src="{{url_for('static_content', group='js', filename='browser-steps.js')}}" defer></script>
+<script type="text/javascript" src="{{url_for('static_content', group='js', filename='extract.js')}}" defer></script>
 {% endif %}
 
 <div class="edit-form monospaced-textarea">
@@ -37,6 +38,7 @@
             <li class="tab"><a id="visualselector-tab" href="#visualselector">Visual Filter Selector</a></li>
             <li class="tab"><a href="#filters-and-triggers">Filters &amp; Triggers</a></li>
             <li class="tab"><a href="#notifications">Notifications</a></li>
+            <li class="tab"><a href="#export">Export</a></li>
         </ul>
     </div>
 
@@ -160,18 +162,14 @@ User-Agent: wonderbra 1.0") }}
                             <input type=checkbox id="include_text_elements" > <label for="include_text_elements">Turn on text finder</label>
                         </div>
 
-                        <div id="loading-status-text" style="display: none;">Please wait, first browser step can take a little time to load..<div class="spinner"></div></div>
+
                         <div class="flex-wrapper" >
 
                             <div id="browser-steps-ui" class="noselect"  style="width: 100%; background-color: #eee; border-radius: 5px;">
 
                                 <div class="noselect"  id="browsersteps-selector-wrapper" style="width: 100%">
-                                    <span class="loader" >
-                                        <span id="browsersteps-click-start">
-                                            <h2 >Click here to Start</h2>
-                                            Please allow 10-15 seconds for the browser to connect.
-                                        </span>
-                                        <div class="spinner"  style="display: none;"></div>
+                                    <span class="loader">
+                                        <div class="spinner"></div>
                                     </span>
                                     <img  class="noselect" id="browsersteps-img" src="" style="max-width: 100%; width: 100%;" />
                                     <canvas  class="noselect" id="browsersteps-selector-canvas" style="max-width: 100%; width: 100%;"></canvas>
@@ -378,6 +376,30 @@ Unavailable") }}
                 </fieldset>
             </div>
 
+            <div class="tab-pane-inner" id="export">
+                    <div class="pure-control-group">
+                    </div>
+
+                <fieldset>
+                    <div class="pure-control-group">
+                        <div>
+                            Enter a regular-expression, all history snapshots will be scanned and where the regex
+                            matches, it will exported along with the times-stamp.<br>
+                            For a complete backup, use the <strong>Backup</strong> button<br>
+                        </div>
+                        <p>
+                            <label>Scan and Extract</label>
+                            <input name=regex type="text" placeholder="Example: Temperature (\d+)">
+                        </p>
+                        <span class="pure-form-message-inline">
+                                The regex you enter will be stored for next time
+                            </span>
+                        <div class="pure-control-group">
+                            <input class="pure-button pure-button-primary" id="extract" name="extract" type="button" value="Scan and Extract">
+                        </div>
+                    </div>
+                </fieldset>
+            </div>
             <div id="actions">
                 <div class="pure-control-group">
                     {{ render_button(form.save_button) }}

requirements.txt

@@ -1,6 +1,5 @@
 flask~=2.0
 flask_wtf
-flask-compress
 eventlet>=0.31.0
 validators
 timeago~=1.0
@@ -58,3 +57,6 @@ jq~=1.3 ;python_version >= "3.8" and sys_platform == "linux"
 # Any current modern version, required so far for screenshot PNG->JPEG conversion but will be used more in the future
 pillow
 # playwright is installed at Dockerfile build time because it's not available on all platforms
+
+# For shutting down playwright BrowserSteps nicely
+psutil