Compare commits

...
Author SHA1 Message Date
dgtlmoonandClaude Opus 5 bbeea9fe38 Dockerfile - actually generate locales, price formatting was falling back to C
The `locales` package was installed "For presenting price amounts correctly in
the restock/price detection overview", but nothing ever ran locale-gen. The
package only ships /etc/locale.gen; it does not build any locale. So published
images had only C, C.utf8 and POSIX available, which made the existing
`ENV LC_ALL=en_US.UTF-8` unsatisfiable:

    $ docker run --rm --entrypoint bash ghcr.io/dgtlmoon/changedetection.io:latest \
        -c 'locale -a'
    C
    C.utf8
    POSIX

locale.setlocale() in flask_app.py therefore raised, was caught by the existing
`except locale.Error` and logged "Unable to set locale ... is not installed
maybe?", and the process stayed on the C locale. The knock-on effect is that the
format_number_locale / format_int_locale Jinja filters lose their thousands
separators, contradicting format_number_locale's own docstring:

    before:  format_number_locale(1234567.89) -> '1234567.89'
    after:   format_number_locale(1234567.89) -> '1,234,567.89'

Those filters render prices in the watchlist overview
(blueprint/watchlist/templates/watch-overview-single-row.html), so every
published release has been showing unseparated price amounts.

How it got this way: LC_ALL arrived on master in d1b1dd70f (#3340) without a
locale-gen to satisfy it. The matching `RUN locale-gen en_US.UTF-8` was written
back in 3f73695e7 (2024-07-22), but that commit only ever existed on the
unmerged 2486-charset-encoding branch, so master has never had it.

Generate one glibc locale per UI translation in changedetectionio/translations
rather than only en_US, so that operators can override LC_ALL / LANG to any
language the UI actually offers and get correct local formatting (de_DE gives
1.234.567,89, fr_FR gives 1 234 567,89). Territories for bare language codes
come from CLDR likely-subtags, not from uppercasing the code, which would have
been wrong for cs, ja, ko, uk and zh.

Costs ~21MB and ~16s of build time.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-13 11:48:01 +02:00
dgtlmoonandClaude Opus 5 48723a09dd UI - Search modal - fix Enter dismissing the form instead of searching (#4428)
Build and push containers / metadata (push) Canceled after 0s
Build and push containers / build-push-containers (push) Canceled after 0s
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Build distribution 📦 (push) Canceled after 0s
ChangeDetection.io App Test / lint-code (push) Canceled after 0s
ChangeDetection.io App Test / lint-translations (push) Canceled after 0s
ChangeDetection.io App Test / lint-template-i18n (push) Canceled after 0s
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Test the built package works basically. (push) Canceled after 0s
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Publish Python 🐍 distribution 📦 to PyPI (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-11 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-12 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-13 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-14 (push) Canceled after 0s
Pressing Enter in the search box closed the modal without running the search;
you had to click Search with the mouse.

Implicit form submission fires a click at the submit button, and a
keyboard-synthesised click carries detail 0 and coordinates of 0,0. The
backdrop-click handler only tested the coordinates against the dialog's
bounding box, so 0,0 read as "outside" - it closed the dialog and blanked the
input while the click was still bubbling. By the time the submit ran, `q` was
empty and `required` rejected it, so nothing was searched.

Ignore clicks with detail 0 - only a real pointer can hit the backdrop.

Verified with Chromium against a local instance: Enter now lands on
/?q=<term>, and mouse submit, backdrop click, Escape and Enter-on-empty all
still behave.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-13 08:25:40 +02:00
dgtlmoonandClaude Opus 5 718295f30b UI - Search modal - keep the search scoped to the tag you were viewing (#4425)
Build and push containers / metadata (push) Canceled after 0s
Build and push containers / build-push-containers (push) Canceled after 0s
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Build distribution 📦 (push) Canceled after 0s
ChangeDetection.io Container Build Test / Build linux/amd64 (alpine) (push) Canceled after 0s
ChangeDetection.io Container Build Test / Build linux/arm64 (alpine) (push) Canceled after 0s
ChangeDetection.io Container Build Test / Build linux/amd64 (main) (push) Canceled after 0s
ChangeDetection.io Container Build Test / Build linux/arm/v7 (main) (push) Canceled after 0s
ChangeDetection.io Container Build Test / Build linux/arm/v8 (main) (push) Canceled after 0s
ChangeDetection.io Container Build Test / Build linux/arm64 (main) (push) Canceled after 0s
ChangeDetection.io App Test / lint-code (push) Canceled after 0s
ChangeDetection.io App Test / lint-translations (push) Canceled after 0s
ChangeDetection.io App Test / lint-template-i18n (push) Canceled after 0s
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Test the built package works basically. (push) Canceled after 0s
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Publish Python 🐍 distribution 📦 to PyPI (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-11 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-12 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-13 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-14 (push) Canceled after 0s
The modal submitted the active tag as `tags=<uuid>`, but the watchlist filters on
`tag` - nothing reads `tags`. So searching from inside a tag view silently
searched every watch, despite the label promising "URL or Title in '<tag>'".

The test pulls the hidden field straight out of the rendered modal and feeds it
back to the watchlist, so the field name can't drift from the arg again.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-12 17:48:48 +02:00
dgtlmoon 7cc62f8321 Merge branch 'master' of github.com:dgtlmoon/changedetection.io 2026-09-12 17:09:59 +02:00
cd66f9ed54 Search modal native submit (#4427)
* UI - Search - Fix search modal navigating to the host root on sub-path deployments

base_path was referenced by search-modal.js but never defined, so searching
always jumped to the host root instead of the X-Forwarded-Prefix sub-path.

* UI - Search modal - submit natively instead of rebuilding the URL in JS

Alternative to defining a `base_path` JS global: give the search form a
server-rendered `action`, so url_for() supplies the reverse-proxy sub-path the
same way every other link on the page already does, and let the browser submit.

Drops the submit handler and the Enter handler from search-modal.js - Enter in
the input reaches the footer's submit button via implicit submission, which also
runs the `required` validation the synthetic `new Event('submit')` skipped.

The hidden tag field is only rendered when a tag is active, so a plain search no
longer carries an empty value.

Also drops the nginx-job grep for the rendered markup - test_search.py already
covers the sub-path case, and asserting on an exact HTML attribute string from a
shell grep breaks on any unrelated edit to that tag.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: ponstream24 <87808547+ponstream24@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-12 17:09:10 +02:00
dgtlmoonandClaude Opus 5 8d938b5966 Browser fetchers - Follow re-navigation, and one content-ready deadline (re-targets #4421 + #4422 at master) (#4426)
* Browser fetchers - Judge a fetch on the document we end up extracting, not the first navigation

The goal is to compare the text of the page the browser lands on, even when the site navigates
again after the first response. Both fetchers were bound to the first navigation, which shows up
as two different bugs:

1. pyppeteer hangs until the hard processing timeout. Its navigation watcher is bound to the
   loaderId of the navigation it started, so when the site replaces that document the 'load' it
   waits for never arrives for that loaderId. With timeout=0 and setDefaultNavigationTimeout(0)
   there is nothing to break the wait, so goto() blocks until
   PUPPETEER_MAX_PROCESSING_TIMEOUT_SECONDS (180s) kills the fetch and the watch records an empty
   xpath_data - while the browser is sitting on a fully loaded page. Traced on slated.com:

     0.24s goto start
     0.74s main frame networkIdle    loaderId=A0E0E0B2   <- never gets 'load'
     2.72s main frame init           loaderId=56D2B5EF   <- re-navigated to get.slated.com
     3.49s main frame load           loaderId=56D2B5EF   <- fires for the new document
    25.2s  goto still hanging, frame._loaderId is now 56D2B5EF

   Now the navigation races goto() against the main frame firing 'load', bounded by
   BROWSER_NAVIGATION_TIMEOUT_SECONDS (default 30), and falls back to the document we can see.
   slated.com / getastra.com / addupsolutions.com went from a 180s timeout with no content to
   200 with full content in 5-35s.

2. Both fetchers reported the status of the interstitial. A site that gates unseen visitors with
   an error status plus a client-side redirect (reported against fotokoch.de: 503 + meta refresh,
   then a 200 with the real page) failed the watch even though the content was present, and the
   only workaround was ignore_status_codes, which also hides genuine 404s and 500s forever.
   The fetchers now keep the latest main-frame document response and judge on that - the refresh
   lands during the existing extra_wait, so the 200 wins.

Playwright also waits for a settled load state before extracting, which is what produced
"Execution context was destroyed, most likely because of a navigation" when the refresh collided
with extraction.

The navigation-response tracker is installed once per page and shared between the fetcher and
action_goto_url() rather than each navigation adding its own listener - 'response' fires once per
HTTP response, hundreds of times on a heavy page, so the callbacks are worth not duplicating.
Verified one listener remains after an install plus four navigations.

Selenium is unaffected either way - it hardcodes status_code = 200 because WebDriver cannot see
the HTTP status.

Tested: new test_renavigation.py covers the interstitial case end to end and was checked to fail
without the fix and pass with it, on both fetchers. The test endpoint gates on last-seen time
rather than a hit count, because a counter lets the second check see a clean 200 and the test
then passes without the fix. Full browser suite 11 passed on playwright and on pyppeteer, 494
unit tests pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Puppeteer fetcher - One content-ready deadline instead of a stopLoading watchdog per frame event

Page.stopLoading is what stops a page that would otherwise load forever waiting on a subresource
that never answers, so that we can still screenshot and scrape what rendered. That intent was
right, but it was implemented as a fire-and-forget task armed by every frame event, which measured
on a single fetch of an iframe-heavy page came to:

  14 watchdog tasks spawned
  11 page-wide Page.stopLoading calls
  3 tasks outliving the fetch and firing against a closed page

Page.stopLoading takes no frame or loader argument - it is the Stop button, and it stops the whole
page. Verified directly: one call stopped a pending main frame and a pending iframe in the same
instant. So the other 10 calls were redundant, and because they landed at arbitrary later times
they could stop a *subsequent* navigation we actually wanted - which is the likeliest reason the
same URL fetched in 8s on one run and 35s on the next.

Replaced with a single deadline, awaited inline so nothing can outlive the fetch (there is no
create_task left in this file at all):

    navigate (bounded)  ->  wait the configured delay  ->  Page.stopLoading  ->  extract

The delay is measured from when navigation finished, not from when it started. Anchoring it to the
start would quietly rob a slow-loading page of its settle time, and letting JS-rendered content
appear after load is the whole point of the setting. Verified with a server that takes 5s to answer
and renders via JS 2s after load: total 9.6s for a 4s delay, and the late content is captured.

Because a page is never reliably "finished" - many sites navigate as part of their normal design -
the delay restarts when the MAIN frame replaces its document, so a redirect or interstitial gets
the same settle time the first document got. Iframes do not restart it, and it is capped by
BROWSER_CONTENT_READY_MAX_RESETS (default 2).

Only the existing "wait n seconds before extracting text" stays user-facing;
BROWSER_NAVIGATION_TIMEOUT_SECONDS is a safety net with a sane default rather than a second knob
for users to reason about. This matches what other scrapers do: bound the navigation, do not fail
when it times out, settle, then extract.

Timings are also more predictable now - the four reported URLs went from 5-35s of variance to
4.6-7.2s at a 3s delay, all with full content and a 200.

Tested: 11 passed pyppeteer browser suite, 7 passed playwright, 494 unit + llm.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-12 17:08:28 +02:00
dgtlmoon 0d6b254fdb UI - Really fix #4413 #4419 - Mobile menu fix 2026-09-12 16:24:23 +02:00
dgtlmoon 900a77e828 Worker id/available fetcher (#4415)
* Worker ID # should be available to the content fetchers

* Actually init with the worker ID
2026-09-12 16:04:39 +02:00
dgtlmoon 35f0294274 UI - Really fix #4413 #4419 Counter bubble in minimal sidebar 2026-09-12 16:03:22 +02:00
dgtlmoon de38c6cd4c Rebuild SCSS/CSS 2026-09-12 15:40:49 +02:00
Samar Nathani b5d768c1d6 feat(ui): show count badges in minimal sidebar (#4419)
In minimal sidebar mode, count badges (unread changes, queue size)
were hidden until hover. This makes them always visible so users can
see at a glance how many items need attention without expanding the
rail.

Closes #4413
2026-09-12 15:40:23 +02:00
dgtlmoonandClaude Opus 5 6318fc70f4 Browser fetchers - Report the real status code when Chrome aborts a bodiless error response (#4420)
* Browser fetchers - Report the real status code when Chrome aborts a bodiless error response

Chrome 153+ refuses to commit a navigation when a 4xx/5xx arrives with a zero-length body:
page.goto() raises net::ERR_HTTP_RESPONSE_CODE_FAILURE instead of returning the response. The
response is received fine, we just never get it as a return value, so the raw net:: string
landed in last_error instead of "Error - 404".

Verified against two browser images, same HTTP server:

  Chrome 153    empty-body 404 -> raises ERR_HTTP_RESPONSE_CODE_FAILURE
  Chrome 153    404 with body  -> status=404
  Chromium 119  empty-body 404 -> status=404
  Chromium 119  404 with body  -> status=404

The fix keeps the main-frame response from the 'response' event and hands that back when goto
raises, so .status / .all_headers() and the existing non-200 branch (which also captures the
screenshot) work unchanged. The latest matching response wins, so a redirect chain still
reports its final hop. Any other error re-raises as before, and if no response was captured we
re-raise too - the status is never invented, which keeps older browsers on exactly their old
path.

Two independent navigation sites needed it:

 - browser_steps.py action_goto_url - covers the playwright fetcher, the live Browser Steps UI,
   the Goto URL / Goto site steps, and the CloakBrowser plugin which imports it. This is the one
   that broke CI: content_fetchers/__init__.py forces playwright when a watch has browser steps,
   so test_non_200_errors_report_browsersteps ran the playwright path in the pyppeteer jobs too.

 - puppeteer.py - its own goto retry loop, used when FAST_PUPPETEER_CHROME_FETCHER is set and the
   watch has no browser steps. No test covers that path; verified by driving the fetcher directly.

Note pyppeteer exposes isNavigationRequest / frame / mainFrame as properties where playwright
uses is_navigation_request() as a method. Mixing them up raises 'bool' object is not callable,
which gets swallowed as a renderer page error rather than failing loudly. Checked against the
pinned pyppeteer-ng==2.0.0rc16.

Selenium is unaffected - it hardcodes status_code = 200 because WebDriver cannot see the HTTP
status, so it never reaches the non-200 branch.

Tested with the full CI browser set (test_content, test_errorhandling, test_fetch_data,
test_custom_js_before_content): 10 passed on each of Chrome 153 + playwright, Chrome 153 +
pyppeteer, Chromium 119 + playwright, Chromium 119 + pyppeteer, the last two against a canonical
Dockerfile.chromium119 build so Chromium is the only variable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* CI - Make a fail-fast test abort say what was skipped rather than looking like a total failure

Fail-fast is kept deliberately - the first failure is nearly always the real problem and it keeps
the run short - but nothing said so, which made a single failing assertion read as "every browser
test is broken".

The playwright and pyppeteer jobs each ran four pytest files as four commands in one `run:` block,
which GitHub executes under `bash -e`. tests/visualselector/test_fetch_data.py is the third, so
when one 404 assertion failed there, test_custom_js_before_content.py never ran, and the later
"Headers and requests" and "Restock detection" steps were skipped as a consequence - three test
files silently dropped, reported only as dashes in the job list. run_basic_tests.sh has the same
shape: 8 independent pytest groups under `set -e`, so a failure in the first parallel group hides
the 7 after it.

No behaviour change to when we stop - only to what gets reported:

 - Each browser test file now runs inside its own ::group:: so the log is navigable, and the
   failing file is named in a ::error:: annotation that states plainly that the remaining files
   and steps were SKIPPED, not failed.
 - run_basic_tests.sh gets an ERR trap saying the same thing, with the line number of the group
   that aborted.

Verified the loop stops on the third file, names it and exits 1, that the all-pass path still
exits 0, and that the trap reports the failing line while preserving the exit code. YAML parses
and run_basic_tests.sh passes bash -n.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Fix unit test failure - install the navigation-response tracker only on a page that supports events

action_goto_url() registered its 'response' listener unconditionally, which broke
test_fetch_url_gate.py::TestBrowserStepGotoUrlGate::test_permitted_url_still_navigates:

    self.page.on("response", _keep_navigation_response)
    E  AttributeError: '_RecordingPage' object has no attribute 'on'

The three refusal tests in that class still passed because validate_fetch_url_async() raises before
reaching the listener, so only the permitted-URL case (the one that actually navigates) hit it.

The listener now lives in track_latest_navigation_response(), which returns None for a page that
has no event support instead of raising. That also removes a real inefficiency: registering per
navigation meant a page accumulated a listener per goto(), and 'response' fires for every
subresource - measured 133 events on getastra.com (54 script, 37 image, 19 fetch, 11 xhr, ...) of
which only 2 were navigations. The tracker is installed once per page and shared, verified as one
listener remaining after an install plus four navigations.

Tested: 494 unit + llm tests pass (was 1 failed / 480 passed), and the full browser set still
passes 10/10 on playwright and 10/10 on pyppeteer, so the Chrome 153 "Error - 404" recovery still
works through the shared tracker.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-12 13:15:44 +02:00
dgtlmoon 5842e7a158 UI - 'Paused' status icon now has red dot instead of yellow #4418
Build and push containers / metadata (push) Canceled after 0s
Build and push containers / build-push-containers (push) Canceled after 0s
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Build distribution 📦 (push) Canceled after 0s
ChangeDetection.io App Test / lint-code (push) Canceled after 0s
ChangeDetection.io App Test / lint-translations (push) Canceled after 0s
ChangeDetection.io App Test / lint-template-i18n (push) Canceled after 0s
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Test the built package works basically. (push) Canceled after 0s
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Publish Python 🐍 distribution 📦 to PyPI (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-11 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-12 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-13 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-14 (push) Canceled after 0s
2026-09-11 13:40:09 +02:00
dgtlmoon 07d00d0811 0.60.4
Build and push containers / metadata (push) Canceled after 0s
Build and push containers / build-push-containers (push) Canceled after 0s
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Build distribution 📦 (push) Canceled after 0s
ChangeDetection.io App Test / lint-code (push) Canceled after 0s
ChangeDetection.io App Test / lint-translations (push) Canceled after 0s
ChangeDetection.io App Test / lint-template-i18n (push) Canceled after 0s
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Test the built package works basically. (push) Canceled after 0s
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Publish Python 🐍 distribution 📦 to PyPI (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-11 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-12 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-13 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-14 (push) Canceled after 0s
2026-09-10 07:04:49 +02:00
dgtlmoon e990d9909f UI - Watch list - Minor tweak to cell padding
Build and push containers / metadata (push) Canceled after 0s
Build and push containers / build-push-containers (push) Canceled after 0s
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Build distribution 📦 (push) Canceled after 0s
ChangeDetection.io App Test / lint-code (push) Canceled after 0s
ChangeDetection.io App Test / lint-translations (push) Canceled after 0s
ChangeDetection.io App Test / lint-template-i18n (push) Canceled after 0s
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Test the built package works basically. (push) Canceled after 0s
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Publish Python 🐍 distribution 📦 to PyPI (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-11 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-12 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-13 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-14 (push) Canceled after 0s
2026-09-09 18:54:57 +02:00
dgtlmoon 1e60bafc52 Rebuild translations 2026-09-09 18:51:00 +02:00
dgtlmoon 8354242121 UI - Watchlist - bring back Restock & Price column to watchlist overview (#4410) 2026-09-09 18:50:46 +02:00
滅üanddgtlmoon f71daee2c4 i18n: Update zh_Hant_TW translations (#4264)
Co-authored-by: dgtlmoon <dgtlmoon@gmail.com>
2026-09-09 18:49:09 +02:00
dgtlmoon d57043be8b Merge branch 'master' of github.com:dgtlmoon/changedetection.io 2026-09-09 18:39:45 +02:00
dgtlmoon 689bbe412c UI - Watch list - Making list status icons larger, fixing icon text 2026-09-09 18:31:50 +02:00
dgtlmoon f3c0efac95 UI - Groups - Fixing button alignment and using better form handling (#4409) 2026-09-09 18:26:10 +02:00
dgtlmoon 4cf1bbe7cd Rebuild translations 2026-09-09 17:14:56 +02:00
dgtlmoon 7d45bf104d UI - Watch stats - Adding a little extra debug to 'Stats' tab 2026-09-09 17:13:05 +02:00
dgtlmoon 6b75954061 UI - Edit watch - Moving notification error alert to 'error' tab, hiding less used Page Title and Link options under foldout (#4406)
Build and push containers / metadata (push) Canceled after 0s
Build and push containers / build-push-containers (push) Canceled after 0s
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Build distribution 📦 (push) Canceled after 0s
ChangeDetection.io Container Build Test / Build linux/amd64 (alpine) (push) Canceled after 0s
ChangeDetection.io Container Build Test / Build linux/arm64 (alpine) (push) Canceled after 0s
ChangeDetection.io Container Build Test / Build linux/amd64 (main) (push) Canceled after 0s
ChangeDetection.io Container Build Test / Build linux/arm/v7 (main) (push) Canceled after 0s
ChangeDetection.io Container Build Test / Build linux/arm/v8 (main) (push) Canceled after 0s
ChangeDetection.io Container Build Test / Build linux/arm64 (main) (push) Canceled after 0s
ChangeDetection.io App Test / lint-code (push) Canceled after 0s
ChangeDetection.io App Test / lint-translations (push) Canceled after 0s
ChangeDetection.io App Test / lint-template-i18n (push) Canceled after 0s
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Test the built package works basically. (push) Canceled after 0s
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Publish Python 🐍 distribution 📦 to PyPI (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-11 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-12 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-13 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-14 (push) Canceled after 0s
2026-09-09 17:01:07 +02:00
dgtlmoon 62fd5eed8f UI - Mobile - Recheck/edit/etc buttons - Centered horizontally 2026-09-09 17:00:05 +02:00
dgtlmoon 30fc5d318a UI - Mobile - Tidy up row padding 2026-09-09 16:57:40 +02:00
dgtlmoon 9657ff4bf9 UI - Watch list - Dont hide buttons on mobile, set light opacity on desktop 2026-09-09 16:47:29 +02:00
dgtlmoonandClaude Opus 5 821b538ae6 Performance - (watched page lister) open link single eval (#4405)
* Watchlist/Notifications - Resolve 'Link to Open' once per row, don't leak 'DISABLED' into notification tokens

Follow-up to #4290.

`watch.open_link` was read three times per watch list row (two hrefs plus the
new title attribute). Each read re-runs `_resolve_link()`, which for a Jinja2
templated URL builds a fresh ImmutableSandboxedEnvironment - the cost the
comment in validate_url.py already warns about. Hoisted to a single
`{% set open_link = watch.open_link %}` alongside the other per-row lookups.

`watch_open_url` was set from `watch.open_link`, which returns the string
'DISABLED' when the URL fails validation, so the default RSS body template
(`RSS_TEMPLATE_HTML_DEFAULT`) rendered `<a href="DISABLED">` for those watches.
Now falls back to the raw URL, matching the neighbouring `watch_url` token.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Rebuild template

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-09 16:19:34 +02:00
dgtlmoon c104179ab5 Rebuild translations 2026-09-09 16:13:13 +02:00
Jeff Hedlund 2f60135cd0 feature: Implemented optional override link (#4290)
* Implemented optional link to override feature

* Added translation strings (untranslated)
2026-09-09 16:06:24 +02:00
dgtlmoon f6d710ca85 Apprise notifications - Updating to 1.13.1 (#4404) 2026-09-09 15:50:58 +02:00
dgtlmoon 7f3645f4a8 API - DELETE for Watch history #4397 (#4403) 2026-09-09 15:46:00 +02:00
455e0228ca Tests - Fix TestHistoryPathTraversal on macOS, and cover the containment check (#4390)
* Compare against a resolved data_dir in the history path-traversal test

Watch.history resolves entries with os.path.realpath, so
test_normal_snapshot_entry_is_accepted compared a resolved path against an
unresolved data_dir. On macOS the datastore lives under /tmp, which is a
symlink to /private/tmp, so the assertion fails for a path that is in fact
inside the directory. The guard is correct; the test was not.

Resolve both sides, matching what the production code does.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Add a test that actually exercises the history containment check

Disabling the containment check in Watch.history left every test in
TestHistoryPathTraversal passing. os.path.basename() reduces both traversal
fixtures ('/etc/passwd', '../../etc/passwd') to 'passwd', so neither reaches
the check — they stop at the os.path.exists() test below it.

A bare '..' survives basename() and resolves to the parent of data_dir, which
exists, so the containment check is what rejects it. With that check disabled
this new test is the only one in the class that fails.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: GG5533 <285285461+GG5533@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-09 15:43:54 +02:00
dgtlmoon cd15016465 UI - Language selector menu-bar icon was cut off by a couple of pixels 2026-09-09 15:16:47 +02:00
dgtlmoon d6996b7a7e Translations - Rebuild catalog 2026-09-09 15:13:59 +02:00
dgtlmoon f1c681185f UI - 'Toast' notifications of actions was at the same height as menu tabs, moved down slightly. 2026-09-09 15:12:28 +02:00
dgtlmoon d901efc8e7 UI - Left menu folds out ONTOP of content, Improving VisualSelector & BrowserSteps width, Darkmode transitions off, Extra sidebar modes #4380 (#4401) 2026-09-09 15:09:27 +02:00
Hans Valerie 1dc7850d6c i18n: add Indonesian translation (#4402) 2026-09-09 15:08:40 +02:00
dgtlmoon 0f4b556af0 0.60.3
Build and push containers / metadata (push) Canceled after 0s
Build and push containers / build-push-containers (push) Canceled after 0s
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Build distribution 📦 (push) Canceled after 0s
ChangeDetection.io App Test / lint-code (push) Canceled after 0s
ChangeDetection.io App Test / lint-translations (push) Canceled after 0s
ChangeDetection.io App Test / lint-template-i18n (push) Canceled after 0s
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Test the built package works basically. (push) Canceled after 0s
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Publish Python 🐍 distribution 📦 to PyPI (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-11 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-12 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-13 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-14 (push) Canceled after 0s
2026-09-04 14:12:00 +02:00
dgtlmoon 7a3bc2ab9e GHSA-56fq-63vj-9992 Add-watch-UI should be POST/CSRF protected @AmerMrkaljevic (#4375) 2026-09-04 12:39:36 +02:00
dgtlmoon 3a71777499 UI - Fixing lots of actions that should be a POST style action which led to a 404
Also solves some of GHSA-56fq-63vj-9992 Add-watch-UI should be POST/CSRF protected
2026-09-04 12:38:20 +02:00
dgtlmoon e0fb224d41 UI - Fixing lots of actions that should be a POST style action which led to a 404 (#4370) 2026-09-04 10:56:23 +02:00
dgtlmoon 050172129e Translations - support for translation overlays/local translations of existing strings (#4365)
Build and push containers / metadata (push) Canceled after 0s
Build and push containers / build-push-containers (push) Canceled after 0s
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Build distribution 📦 (push) Canceled after 0s
ChangeDetection.io Container Build Test / Build linux/amd64 (alpine) (push) Canceled after 0s
ChangeDetection.io Container Build Test / Build linux/arm64 (alpine) (push) Canceled after 0s
ChangeDetection.io Container Build Test / Build linux/amd64 (main) (push) Canceled after 0s
ChangeDetection.io Container Build Test / Build linux/arm/v7 (main) (push) Canceled after 0s
ChangeDetection.io Container Build Test / Build linux/arm/v8 (main) (push) Canceled after 0s
ChangeDetection.io Container Build Test / Build linux/arm64 (main) (push) Canceled after 0s
ChangeDetection.io App Test / lint-code (push) Canceled after 0s
ChangeDetection.io App Test / lint-translations (push) Canceled after 0s
ChangeDetection.io App Test / lint-template-i18n (push) Canceled after 0s
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Test the built package works basically. (push) Canceled after 0s
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Publish Python 🐍 distribution 📦 to PyPI (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-11 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-12 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-13 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-14 (push) Canceled after 0s
* Translations - support for translation overlays/local translations of existing strings

* Dont forget translations_overlay for pip
2026-09-03 11:15:46 +02:00
dgtlmoon e573837789 0.60.2 2026-09-03 09:15:35 +02:00
Andrew PeabodyandAndrew Peabody 3d6de42924 fix(conditions): support zero values in condition filtering and json logic conversion (#4362)
* fix(conditions): support zero values in condition filtering and json logic conversion

- Fix filter_complete_rules dropping rules where value is 0/0.0 due to 0 == False in Python
- Fix convert_to_jsonlogic raising EmptyConditionRuleRowNotUsable on value=0 due to truthiness check
- Fix str != 'None' type comparison typo in convert_to_jsonlogic
- Add comprehensive unit tests covering zero value condition filtering, conversion, and execution

* chore: re-trigger CI checks

---------

Co-authored-by: Andrew Peabody <apeabody@users.noreply.github.com>
2026-09-03 08:57:34 +02:00
dgtlmoonandClaude Opus 5 fbd9472218 fix(api): accept an existing tag UUID in the watch tag field, and deprecate it (#4361)
Build and push containers / metadata (push) Canceled after 0s
Build and push containers / build-push-containers (push) Canceled after 0s
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Build distribution 📦 (push) Canceled after 0s
ChangeDetection.io App Test / lint-code (push) Canceled after 0s
ChangeDetection.io App Test / lint-translations (push) Canceled after 0s
ChangeDetection.io App Test / lint-template-i18n (push) Canceled after 0s
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Test the built package works basically. (push) Canceled after 0s
Publish Python 🐍distribution 📦 to PyPI and TestPyPI / Publish Python 🐍 distribution 📦 to PyPI (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-11 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-12 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-13 (push) Canceled after 0s
ChangeDetection.io App Test / test-application-3-14 (push) Canceled after 0s
`tag` on POST /watch was documented as taking a tag UUID, but the value went to
add_tag(title): a UUID silently created a junk tag *titled* with that UUID and never
applied the tag the caller asked for. `tags` (UUIDs) was the only thing that worked.

- `tag=` now resolves an existing tag UUID to that tag, still falling back to title
  matching/creation for names. A UUID-shaped value matching nothing is skipped with a
  warning rather than becoming a group named after a UUID.
- Blank tokens ("One,,Two,") no longer store False in watch['tags'] - add_tag() returns
  False for an empty title and it was appended unguarded. Consumers tolerate it
  (get_all_tags_for_watch() dictfilt()s over known tags) but it is not valid data.
- add_tag()'s title search is extracted to tag_uuid_for_title(), so existence can be
  tested without creating as a side effect. add_tag()'s contract is unchanged.
- api-spec: `tag` is marked `deprecated: true` (so Redoc renders the badge) and states
  plainly that it takes names, not UUIDs. `tags` now says what it really does - applied
  verbatim, never creates, unknown UUIDs stored as dangling refs. Rendered docs rebuilt.

Every claim in the new field docs is asserted in test_api_tags.py against the real API.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 16:46:27 +02:00
dgtlmoonandClaude Opus 5 2669a5cee6 Feature/Watch limit - Adding test (#4360)
* Env var - PAGE_WATCH_LIMIT enhancements

* Rebuild API docs

* Bump APi doc version

* test: cover PAGE_WATCH_LIMIT across every add path

- API create returns 429; API import returns 429 and refuses the batch whole
- quick-add and the UI importer flash the limit (importer once per file, not per row)
  and hand unimported URLs back
- clone at the limit no longer KeyErrors
- an instance already over the limit still loads from disk and stays editable, only
  new watches are refused
- the Info tab shows the limit only when one is set
- add_watch() with no request context returns None instead of raising from flash()
- an absent, empty or unparseable env var all mean unlimited

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 15:43:59 +02:00
141 changed files with 9891 additions and 1808 deletions
+13
View File
@@ -52,6 +52,19 @@ jobs:
git diff --stat changedetectionio/translations
exit 1
fi
- name: Check translation overlay
# Deliberately after extract_messages above, so overrides are validated against a freshly
# extracted messages.pot. An overlay entry is keyed on the exact upstream msgid: when a
# string is reworded upstream the override stops matching and silently reverts to upstream
# wording. This is the only thing that makes that visible.
# See changedetectionio/translations_overlay/README.md
if: hashFiles('changedetectionio/translations_overlay/**/*.po') != ''
run: |
find changedetectionio/translations_overlay -name "*.po" | while read f; do
echo "Checking $f"
msgfmt --check-format -o /dev/null "$f"
done
python changedetectionio/translations_overlay/manage.py check
lint-template-i18n:
runs-on: ubuntu-latest
@@ -201,10 +201,20 @@ jobs:
- name: Playwright - Specific tests in built container
run: |
docker run --rm -e "FLASK_SERVER_NAME=cdio" -e "PLAYWRIGHT_DRIVER_URL=ws://sockpuppetbrowser:3000" --network changedet-network --hostname=cdio test-changedetectionio bash -c 'cd changedetectionio;pytest -vv --capture=tee-sys --showlocals --tb=long --live-server-host=0.0.0.0 --live-server-port=5004 tests/fetchers/test_content.py'
docker run --rm -e "FLASK_SERVER_NAME=cdio" -e "PLAYWRIGHT_DRIVER_URL=ws://sockpuppetbrowser:3000" --network changedet-network --hostname=cdio test-changedetectionio bash -c 'cd changedetectionio;pytest -vv --capture=tee-sys --showlocals --tb=long --live-server-host=0.0.0.0 --live-server-port=5004 tests/test_errorhandling.py'
docker run --rm -e "FLASK_SERVER_NAME=cdio" -e "PLAYWRIGHT_DRIVER_URL=ws://sockpuppetbrowser:3000" --network changedet-network --hostname=cdio test-changedetectionio bash -c 'cd changedetectionio;pytest -vv --capture=tee-sys --showlocals --tb=long --live-server-host=0.0.0.0 --live-server-port=5004 tests/visualselector/test_fetch_data.py'
docker run --rm -e "FLASK_SERVER_NAME=cdio" -e "PLAYWRIGHT_DRIVER_URL=ws://sockpuppetbrowser:3000" --network changedet-network --hostname=cdio test-changedetectionio bash -c 'cd changedetectionio;pytest -vv --capture=tee-sys --showlocals --tb=long --live-server-host=0.0.0.0 --live-server-port=5004 tests/fetchers/test_custom_js_before_content.py'
# Deliberately fail fast - the first failure is nearly always the real problem and it
# keeps the run short. Each file is wrapped in its own log group and the failing one is
# named explicitly, because everything after it is SKIPPED rather than run, and that is
# otherwise easy to misread as "the whole browser suite broke".
for t in tests/fetchers/test_content.py tests/test_errorhandling.py tests/visualselector/test_fetch_data.py tests/fetchers/test_custom_js_before_content.py tests/fetchers/test_renavigation.py; do
echo "::group::pytest $t"
if ! docker run --rm -e "FLASK_SERVER_NAME=cdio" -e "PLAYWRIGHT_DRIVER_URL=ws://sockpuppetbrowser:3000" --network changedet-network --hostname=cdio test-changedetectionio \
bash -c "cd changedetectionio;pytest -vv --capture=tee-sys --showlocals --tb=long --live-server-host=0.0.0.0 --live-server-port=5004 $t"; then
echo "::endgroup::"
echo "::error::$t FAILED - stopping here. Any later test file and the remaining steps of this job were SKIPPED, not failed."
exit 1
fi
echo "::endgroup::"
done
- name: Playwright - Headers and requests
run: |
@@ -242,10 +252,17 @@ jobs:
- name: Pyppeteer - Specific tests in built container
run: |
docker run --rm -e "FLASK_SERVER_NAME=cdio" -e "FAST_PUPPETEER_CHROME_FETCHER=True" -e "PLAYWRIGHT_DRIVER_URL=ws://sockpuppetbrowser:3000" --network changedet-network --hostname=cdio test-changedetectionio bash -c 'cd changedetectionio;pytest --live-server-host=0.0.0.0 --live-server-port=5004 tests/fetchers/test_content.py'
docker run --rm -e "FLASK_SERVER_NAME=cdio" -e "FAST_PUPPETEER_CHROME_FETCHER=True" -e "PLAYWRIGHT_DRIVER_URL=ws://sockpuppetbrowser:3000" --network changedet-network --hostname=cdio test-changedetectionio bash -c 'cd changedetectionio;pytest --live-server-host=0.0.0.0 --live-server-port=5004 tests/test_errorhandling.py'
docker run --rm -e "FLASK_SERVER_NAME=cdio" -e "FAST_PUPPETEER_CHROME_FETCHER=True" -e "PLAYWRIGHT_DRIVER_URL=ws://sockpuppetbrowser:3000" --network changedet-network --hostname=cdio test-changedetectionio bash -c 'cd changedetectionio;pytest --live-server-host=0.0.0.0 --live-server-port=5004 tests/visualselector/test_fetch_data.py'
docker run --rm -e "FLASK_SERVER_NAME=cdio" -e "FAST_PUPPETEER_CHROME_FETCHER=True" -e "PLAYWRIGHT_DRIVER_URL=ws://sockpuppetbrowser:3000" --network changedet-network --hostname=cdio test-changedetectionio bash -c 'cd changedetectionio;pytest --live-server-host=0.0.0.0 --live-server-port=5004 tests/fetchers/test_custom_js_before_content.py'
# Fail fast, but name the file that failed - see the note in the playwright job above
for t in tests/fetchers/test_content.py tests/test_errorhandling.py tests/visualselector/test_fetch_data.py tests/fetchers/test_custom_js_before_content.py tests/fetchers/test_renavigation.py; do
echo "::group::pytest $t"
if ! docker run --rm -e "FLASK_SERVER_NAME=cdio" -e "FAST_PUPPETEER_CHROME_FETCHER=True" -e "PLAYWRIGHT_DRIVER_URL=ws://sockpuppetbrowser:3000" --network changedet-network --hostname=cdio test-changedetectionio \
bash -c "cd changedetectionio;pytest --live-server-host=0.0.0.0 --live-server-port=5004 $t"; then
echo "::endgroup::"
echo "::error::$t FAILED - stopping here. Any later test file and the remaining steps of this job were SKIPPED, not failed."
exit 1
fi
echo "::endgroup::"
done
- name: Pyppeteer - Headers and requests checks
run: |
+25
View File
@@ -101,6 +101,31 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
libxrender-dev \
&& apt-get clean && rm -rf /var/lib/apt/lists/*
# Actually generate the locales. Installing the `locales` package above only
# ships /etc/locale.gen - it does not build any locale, so the image had just
# C, C.utf8 and POSIX. That made the `ENV LC_ALL=en_US.UTF-8` below unsatisfiable:
# locale.setlocale() in flask_app.py failed, fell back to C, and the
# format_number_locale / format_int_locale Jinja filters silently lost their
# thousands separators - 1234567.89 rendered as "1234567.89" rather than
# "1,234,567.89" in the restock/price overview, which is the very thing the
# `locales` package was added for.
#
# More than en_US is generated so that operators can override LC_ALL / LANG and
# get formatting for their own region (de_DE gives 1.234.567,89, fr_FR gives
# 1 234 567,89). Costs ~21MB and ~16s of build time.
#
# This list mirrors the UI translations in changedetectionio/translations - one
# glibc locale per language we ship a translation for, so any language a user
# can pick in the UI also has a working locale. Keep the two in sync when adding
# a translation. The territory for each bare language code comes from CLDR's
# likely-subtags (cs -> cs_CZ, ja -> ja_JP, ko -> ko_KR, uk -> uk_UA, zh ->
# zh_CN, zh_Hant_TW -> zh_TW), NOT from uppercasing the language code.
RUN for l in cs_CZ de_DE en_GB en_US es_ES fr_FR id_ID it_IT ja_JP ko_KR \
pl_PL pt_BR ru_RU tr_TR uk_UA zh_CN zh_TW; do \
sed -i "s/^# *${l}.UTF-8 UTF-8/${l}.UTF-8 UTF-8/" /etc/locale.gen; \
done \
&& locale-gen
# https://stackoverflow.com/questions/58701233/docker-logs-erroneously-appears-empty-until-container-stops
ENV PYTHONUNBUFFERED=1
+1
View File
@@ -13,6 +13,7 @@ recursive-include changedetectionio/store *
recursive-include changedetectionio/templates *
recursive-include changedetectionio/tests *
recursive-include changedetectionio/translations *
recursive-include changedetectionio/translations_overlay *
recursive-include changedetectionio/widgets *
prune changedetectionio/static/package-lock.json
prune changedetectionio/static/styles/node_modules
+1 -1
View File
@@ -2,7 +2,7 @@
# Read more https://github.com/dgtlmoon/changedetection.io/wiki
# Semver means never use .01, or 00. Should be .1.
__version__ = '0.55.8'
__version__ = '0.60.4'
from changedetectionio.strtobool import strtobool
from json.decoder import JSONDecodeError
+16
View File
@@ -141,6 +141,7 @@ class Watch(Resource):
watch['last_changed'] = watch_obj.last_changed
watch['viewed'] = watch_obj.viewed
watch['link'] = watch_obj.link
watch['open_link'] = watch_obj.open_link
# Resolved processor config: tag override wins over watch-level config (mirrors restock processor logic)
import json
@@ -293,6 +294,20 @@ class WatchHistory(Resource):
abort(404, message='No watch exists with the UUID of {}'.format(uuid))
return watch.history, 200
# Delete all history/snapshots for a watch, but keep the watch itself
# curl -X DELETE http://localhost:5000/api/v1/watch/<uuid_str:uuid>/history
@auth.check_token
@validate_openapi_request('deleteWatchHistory')
def delete(self, uuid):
"""Clear all snapshot history for a watch (the watch itself is kept)."""
if not self.datastore.data['watching'].get(uuid):
abort(404, message='No watch exists with the UUID of {}'.format(uuid))
# Same call as the UI "Clear history" button - wipes snapshots/screenshots and
# resets last_checked etc, while preserving the watch and its processor config
self.datastore.clear_watch_history(uuid)
return 'OK', 204
class WatchSingleHistory(Resource):
def __init__(self, **kwargs):
@@ -596,6 +611,7 @@ class CreateWatch(Resource):
'last_checked': watch['last_checked'],
'last_error': watch['last_error'],
'link': watch.link,
'open_link': watch.open_link,
'page_title': watch['page_title'],
'tags': [*tags], # Unpack dict keys to list (can't use list() since variable named 'list')
'title': watch['title'],
@@ -42,7 +42,7 @@ def construct_blueprint(datastore: ChangeDetectionStore):
system_default_browser=browser_config.system_default_description(datastore),
)
@add_watch_ui_blueprint.route("/snapshot", methods=['GET'])
@add_watch_ui_blueprint.route("/snapshot", methods=['POST'])
@login_optionally_required
def add_watch_ui_snapshot():
"""One-shot live fetch of an arbitrary URL for the Add Watch visual selector.
@@ -52,6 +52,10 @@ def construct_blueprint(datastore: ChangeDetectionStore):
connect, "Goto site", grab the screenshot + xpath element data, then tear
the browser down again. Element selection then happens client-side on the
returned data, exactly like the watch Edit page's visual selector.
POST-only and CSRF protected on purpose: this drives a real browser fetch and
writes a temporary watch dir, so as a GET it could be triggered cross-origin
(or by any tag/link that issues a GET) without the operator's consent.
"""
import base64
from changedetectionio.blueprint.browser_steps import (
@@ -71,7 +75,7 @@ def construct_blueprint(datastore: ChangeDetectionStore):
# backslash/parser-differential rejection of GHSA-rph4-96w6-q594 (GHSA-56fq-63vj-9992).
# Note this fetch never reaches difference_detection_processor.call_browser(), so it gets
# no gating from there - it has to validate for itself.
url = (request.args.get('url') or '').strip()
url = (request.form.get('url') or '').strip()
ok, reason = is_fetch_url_allowed(url)
if not ok:
logger.warning(f"Add-watch snapshot: refused '{url}' - {reason}")
@@ -82,7 +86,7 @@ def construct_blueprint(datastore: ChangeDetectionStore):
# Either way it has to be able to render a preview - the plain HTTP client
# produces no screenshot and no element data, so previewing with it is pointless
# (and it used to be the silent default here, see the system-default bug).
fetcher_name = (request.args.get('fetch_backend') or '').strip() or browser_config.default_visual_browser(datastore)
fetcher_name = (request.form.get('fetch_backend') or '').strip() or browser_config.default_visual_browser(datastore)
if not fetcher_name or not browser_config.is_visual_capable(fetcher_name, datastore):
logger.warning(f"Add-watch snapshot: refused browser '{fetcher_name}' for '{url}'")
return make_response('No interactive browser available that can render a live preview '
@@ -59,9 +59,18 @@ $(document).ready(() => {
$.ajax({
url: add_watch_snapshot_url,
// POST, never GET - this makes the server-side browser fetch a URL of our
// choosing, so it must not be triggerable cross-origin. csrf.js adds the
// X-CSRFToken header to every non-GET ajax call; the CSRF field on the form
// is sent too so it works even if that handler hasn't run yet.
method: 'POST',
// Preview with the browser picked in the list - that same browser is what
// gets saved on the watch, so what you see here is what it will check with.
data: {url: url, fetch_backend: $('input[name="fetch_backend"]:checked').val() || ''},
data: {
url: url,
fetch_backend: $('input[name="fetch_backend"]:checked').val() || '',
csrf_token: $('#new-watch-form input[name="csrf_token"]').val() || '',
},
dataType: 'json',
}).done((data) => {
showState('ready');
@@ -98,7 +98,7 @@ def construct_blueprint(datastore: ChangeDetectionStore):
backups_blueprint.register_blueprint(construct_restore_blueprint(datastore))
backup_threads = []
@backups_blueprint.route("/request-backup", methods=['GET'])
@backups_blueprint.route("/request-backup", methods=['POST'])
@login_optionally_required
def request_backup():
if any(thread.is_alive() for thread in backup_threads):
@@ -35,8 +35,10 @@
</p>
{% endif %}
<a class="pure-button pure-button-primary"
href="{{ url_for('backups.request_backup') }}">{{ _('Create backup') }}</a>
<form method="POST" action="{{ url_for('backups.request_backup') }}" style="display: inline;">
<input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
<button type="submit" class="pure-button pure-button-primary">{{ _('Create backup') }}</button>
</form>
{% if available_backups %}
{# POST + CSRF token: this permanently deletes every backup archive, so it must
not be reachable from a bare GET (an <img src=...> on any page the operator
@@ -296,7 +296,7 @@ def construct_blueprint(datastore: ChangeDetectionStore):
return browsersteps_start_session
@browser_steps_blueprint.route("/browsersteps_start_session", methods=['GET'])
@browser_steps_blueprint.route("/browsersteps_start_session", methods=['POST'])
@login_optionally_required
def browsersteps_start_session():
# A new session was requested, return sessionID
@@ -100,7 +100,7 @@ def construct_blueprint(datastore: ChangeDetectionStore):
results = _recalc_check_status(uuid=uuid)
return results
@check_proxies_blueprint.route("/<uuid_str:uuid>/start", methods=['GET'])
@check_proxies_blueprint.route("/<uuid_str:uuid>/start", methods=['POST'])
@login_optionally_required
def start_check(uuid):
+16
View File
@@ -0,0 +1,16 @@
"""Left-rail (action sidebar) display modes.
A leaf module on purpose: forms.py, flask_app.py and model/App.py all need these, and
model/App.py seeding its default from forms.py would drag the whole form stack (~550
modules) into the model layer.
"""
from flask_babel import lazy_gettext as _l
# The complete set of left-rail modes - flask_app.get_sidebar_mode_class() maps these
# (and only these) onto body classes, so a new mode here needs a new mapping there.
MENU_SIDEBAR_ACTIONMODES = [
('expandable', _l('Expand on hover')), # Slim icon rail that expands on hover/focus
('pinned-expanded', _l('Always expanded')), # Always expanded, never collapses
('minimal', _l('Stays minimal')), # Always small, never expands
]
MENU_SIDEBAR_ACTIONMODES_DEFAULT = 'expandable'
@@ -14,7 +14,7 @@ def construct_blueprint(datastore: ChangeDetectionStore, update_q: PriorityQueue
price_data_follower_blueprint = Blueprint('price_data_follower', __name__)
@price_data_follower_blueprint.route("/<uuid_str:uuid>/accept", methods=['GET'])
@price_data_follower_blueprint.route("/<uuid_str:uuid>/accept", methods=['POST'])
@login_optionally_required
def accept(uuid):
datastore.data['watching'][uuid]['track_ldjson_price_data'] = PRICE_DATA_TRACK_ACCEPT
@@ -24,7 +24,7 @@ def construct_blueprint(datastore: ChangeDetectionStore, update_q: PriorityQueue
worker_pool.queue_item_async_safe(update_q, queuedWatchMetaData.PrioritizedItem(priority=1, item={'uuid': uuid}))
return redirect(url_for("watchlist.index"))
@price_data_follower_blueprint.route("/<uuid_str:uuid>/reject", methods=['GET'])
@price_data_follower_blueprint.route("/<uuid_str:uuid>/reject", methods=['POST'])
@login_optionally_required
def reject(uuid):
datastore.data['watching'][uuid]['track_ldjson_price_data'] = PRICE_DATA_TRACK_REJECT
+1 -1
View File
@@ -24,4 +24,4 @@ RSS_TEMPLATE_PLAINTEXT_DEFAULT = "<pre>{{watch_label}} had a change.\n\n{{diff}}
# @todo add some [edit]/[history]/[goto] etc links
# @todo need {{watch_edit_link}} + delete + history link token
RSS_TEMPLATE_HTML_DEFAULT = "<html><body>\n<h4><a href=\"{{watch_url}}\">{{watch_label}}</a></h4>\n<p>{{diff}}</p>\n</body></html>\n"
RSS_TEMPLATE_HTML_DEFAULT = "<html><body>\n<h4><a href=\"{{watch_open_url}}\">{{watch_label}}</a></h4>\n<p>{{diff}}</p>\n</body></html>\n"
@@ -105,7 +105,7 @@ def construct_single_watch_routes(rss_blueprint, datastore):
fe = fg.add_entry()
title_suffix = f"Change @ {res['original_context']['change_datetime']}"
populate_feed_entry(fe, watch, res.get('body', ''), guid, timestamp_to,
link={'href': watch.get('url')}, title_suffix=title_suffix)
link={'href': watch.open_link_override or watch.get('url')}, title_suffix=title_suffix)
add_watch_categories(fe, watch, datastore)
response = make_response(fg.rss_str())
@@ -278,7 +278,7 @@ def construct_blueprint(datastore: ChangeDetectionStore):
return output
@settings_blueprint.route("/reset-api-key", methods=['GET'])
@settings_blueprint.route("/reset-api-key", methods=['POST'])
@login_optionally_required
def settings_reset_api_key():
secret = secrets.token_hex(16)
@@ -295,7 +295,7 @@ def construct_blueprint(datastore: ChangeDetectionStore):
logs=notification_debug_log if len(notification_debug_log) else ["Notification logs are empty - no notifications sent yet."])
return output
@settings_blueprint.route("/toggle-all-paused", methods=['GET'])
@settings_blueprint.route("/toggle-all-paused", methods=['POST'])
@login_optionally_required
def toggle_all_paused():
current_state = datastore.data['settings']['application'].get('all_paused', False)
@@ -309,7 +309,7 @@ def construct_blueprint(datastore: ChangeDetectionStore):
return redirect(url_for('watchlist.index'))
@settings_blueprint.route("/toggle-all-muted", methods=['GET'])
@settings_blueprint.route("/toggle-all-muted", methods=['POST'])
@login_optionally_required
def toggle_all_muted():
current_state = datastore.data['settings']['application'].get('all_muted', False)
+1 -1
View File
@@ -130,7 +130,7 @@ def construct_llm_blueprint(datastore: ChangeDetectionStore):
logger.exception("LLM model list full traceback:")
return jsonify({'models': [], 'error': str(e)}), 400
@llm_blueprint.route("/test", methods=['GET'])
@llm_blueprint.route("/test", methods=['POST'])
@login_optionally_required
def llm_test():
from flask import request
@@ -208,7 +208,7 @@ nav
</div>
</div>
<div class="pure-control-group">
<a href="{{url_for('settings.settings_reset_api_key')}}" class="pure-button button-small button-cancel">{{ _('Regenerate API key') }}</a>
<button type="submit" formmethod="post" formnovalidate formaction="{{url_for('settings.settings_reset_api_key')}}" class="pure-button button-small button-cancel">{{ _('Regenerate API key') }}</button>
</div>
<div class="pure-control-group">
<h4>{{ _('Chrome Extension') }}</h4>
@@ -577,7 +577,10 @@
if (mult.trim()) params.set('local_token_multiplier', mult.trim());
try {
const resp = await fetch('{{ url_for("settings.llm.llm_test") }}?' + params);
const resp = await fetch('{{ url_for("settings.llm.llm_test") }}?' + params, {
method: 'POST',
headers: {'X-CSRFToken': csrftoken}
});
const data = await resp.json();
if (data.ok) {
result.style.cssText = 'display:block; background:rgba(39,174,96,0.08); border:1px solid rgba(39,174,96,0.3); border-radius:5px; padding:0.6em 0.85em; font-size:0.88em; line-height:1.45;';
+1 -1
View File
@@ -62,7 +62,7 @@ def construct_blueprint(datastore: ChangeDetectionStore):
return redirect(url_for('tags.tags_overview_page'))
@tags_blueprint.route("/mute/<uuid_str:uuid>", methods=['GET'])
@tags_blueprint.route("/mute/<uuid_str:uuid>", methods=['POST'])
@login_optionally_required
def mute(uuid):
tag = datastore.data['settings']['application']['tags'].get(uuid)
@@ -64,20 +64,18 @@ html[data-darkmode="true"] .watch-tag-list.tag-{{ class_name }} {
<td colspan="3">{{ _('No website organisational tags/groups configured') }}</td>
</tr>
{% endif %}
<form method="POST" style="display: inline;">
<input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
{% for uuid, tag in available_tags %}
{#-{{ loop.cycle('pure-table-odd', 'pure-table-even') }}-#}
<tr id="{{ uuid }}" class="">
<td class="watch-controls">
<a class="link-mute state-{{'on' if tag.notification_muted else 'off'}}" href="{{url_for('tags.mute', uuid=tag.uuid)}}" aria-label="{{ _('Mute notifications') }}" title="{{ _('Mute notifications') }}"><i data-feather="{{ 'bell-off' if tag.notification_muted else 'bell' }}" class="icon icon-mute"></i></a>
<button formaction="{{url_for('tags.mute', uuid=tag.uuid)}}" type="submit" class="bare-btn link-mute state-{{'on' if tag.notification_muted else 'off'}}" aria-label="{{ _('Mute notifications') }}" title="{{ _('Mute notifications') }}"><i data-feather="{{ 'bell-off' if tag.notification_muted else 'bell' }}" class="icon icon-mute"></i></button>
</td>
<td class="watch-count">{{ "{:,}".format(tag_count[uuid]) if uuid in tag_count else 0 }}</td>
<td class="title-col inline"> <a href="{{url_for('watchlist.index', tag=uuid) }}" class="watch-tag-list tag-{{ tag.title|sanitize_tag_class }}">{{ tag.title }}</a></td>
<td>
<button formaction="{{ url_for('ui.form_watch_checknow', tag=uuid) }}" type="submit" class="cdio-btn"><i data-feather="refresh-cw"></i>{{ _('Recheck') }}</button>
<a class="cdio-btn" href="{{ url_for('tags.form_tag_edit', uuid=uuid) }}">{{ _('Edit') }}</a>
<form method="POST" action="{{ url_for('ui.form_watch_checknow', tag=uuid) }}" style="display: inline;">
<input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
<button type="submit" class="cdio-btn"><i data-feather="refresh-cw"></i>{{ _('Recheck') }}</button>
</form>
<a class="cdio-btn cdio-btn--danger"
href="{{ url_for('tags.delete', uuid=uuid) }}"
data-method="POST"
@@ -100,6 +98,7 @@ html[data-darkmode="true"] .watch-tag-list.tag-{{ class_name }} {
</td>
</tr>
{% endfor %}
</form>
</tbody>
</table>
</div>
+2 -2
View File
@@ -407,7 +407,7 @@ def construct_blueprint(datastore: ChangeDetectionStore, update_q, worker_pool,
return redirect(url_for('watchlist.index'))
@ui_blueprint.route("/share-url/<uuid_str:uuid>", methods=['GET'])
@ui_blueprint.route("/share-url/<uuid_str:uuid>", methods=['POST'])
@login_optionally_required
def form_share_put_watch(uuid):
"""Given a watch UUID, upload the info and return a share-link
@@ -455,7 +455,7 @@ def construct_blueprint(datastore: ChangeDetectionStore, update_q, worker_pool,
return redirect(url_for('watchlist.index'))
@ui_blueprint.route("/language/auto-detect", methods=['GET'])
@ui_blueprint.route("/language/auto-detect", methods=['POST'])
def delete_locale_language_session_var_if_it_exists():
"""Clear the session locale preference to auto-detect from browser Accept-Language header"""
if 'locale' in session:
@@ -70,6 +70,8 @@ def construct_blueprint(datastore: ChangeDetectionStore):
# use the same as when it is triggered, but then override it with the form test values
n_object = NotificationContextData({
'watch_url': request.form.get('window_url', "https://changedetection.io"),
# Falls back to watch_url when the watch has no 'Link to Open' set
'watch_open_url': watch.open_link_override or request.form.get('window_url', "https://changedetection.io"),
'notification_urls': notification_urls
})
+1 -1
View File
@@ -104,7 +104,7 @@ def construct_blueprint(datastore: ChangeDetectionStore):
output = render_template("preview.html",
capabilities=capabilities,
content=content,
current_diff_url=watch['url'],
current_diff_url=watch.open_link,
current_version=timestamp,
extra_stylesheets=extra_stylesheets,
extra_title=f" - {gettext('Diff')} - {watch.label} @ {timestamp}",
@@ -8,7 +8,7 @@
{% endif %}
const highlight_submit_ignore_url="{{url_for('ui.ui_edit.highlight_submit_ignore_url', uuid=uuid)}}";
const watch_url= {{watch_a.link|tojson}};
const watch_url= {{watch_a.open_link|tojson}};
// Initial scroll position: if set, scroll to this line number in #difference on page load
const initialScrollToLineNumber = {{ initial_scroll_line_number|default('null') }};
@@ -100,6 +100,23 @@
{{ render_field(form.title, class="m-d", placeholder=watch.label) }}
<span class="pure-form-message-inline">{{ _('Automatically uses the page title if found, you can also use your own title/description here') }}</span>
</div>
<div class="pure-control-group">
<details class="form-wrapper">
<summary>{{ _('Extra Page Title and Link options') }}</summary>
<br>
<div class=" border-fieldset">
<div class="pure-control-group">
{{ render_field(form.link_to_open, placeholder="https://...", class="m-d") }}
<span class="pure-form-message-inline">{{ _('Optional - links in the list, history and notifications open this URL instead. Does not change the URL being checked.') }}</span>
</div>
<div class="pure-control-group">
{{ render_ternary_field(form.use_page_title_in_list) }}
</div>
</div>
</details>
</div>
<div class="pure-control-group time-between-check border-fieldset">
{{ render_checkbox_field(form.time_between_check_use_default, class="use-default-timecheck") }}
@@ -119,13 +136,6 @@
</div>
<br>
</div>
<div class="pure-control-group">
{{ render_checkbox_field(form.filter_failure_notification_send) }}
<span class="pure-form-message-inline">
{{ _('Sends a notification when the filter can no longer be seen on the page, good for knowing when the page changed and your filter will not work anymore.') }}
</span>
</div>
<div class="pure-control-group">
{{ render_field(form.history_snapshot_max_length, class="history_snapshot_max_length") }}
<span class="pure-form-message-inline">{{ _('Limit collection of history snapshots for each watch to this number of history items.') }}
@@ -133,9 +143,7 @@
{{ _('Set to empty to use system settings default') }}
</span>
</div>
<div class="pure-control-group">
{{ render_ternary_field(form.use_page_title_in_list) }}
</div>
</fieldset>
</div>
@@ -250,7 +258,7 @@ Math: {{ 1 + 1 }}") }}
<div class="flex-wrapper" >
<div id="browser-steps-ui" class="noselect">
<div class="noselect" id="browsersteps-selector-wrapper" style="width: 100%">
<div class="noselect" id="browsersteps-selector-wrapper">
<span class="loader" >
<span id="browsersteps-click-start">
<h2 >{{ _('Click here to Start') }}</h2>
@@ -295,6 +303,12 @@ Math: {{ 1 + 1 }}") }}
</span>
</div>
{% endif %}
<div class="pure-control-group">
{{ render_checkbox_field(form.filter_failure_notification_send) }}
<span class="pure-form-message-inline">
{{ _('Sends a notification when the filter can no longer be seen on the page, good for knowing when the page changed and your filter will not work anymore.') }}
</span>
</div>
<div class="field-group" id="notification-field-group">
{% if has_default_notification_urls %}
<div class="inline-warning">
@@ -507,6 +521,26 @@ Math: {{ 1 + 1 }}") }}
<td>{{ _('Server type reply') }}</td>
<td>{{ watch.get('remote_server_reply') }}</td>
</tr>
<tr>
<td><code>capabilities.supports_xpath_element_data</code></td>
<td>{{ _('Yes') if capabilities.supports_xpath_element_data else _('No') }}</td>
</tr>
<tr>
<td><code>capabilities.supports_screenshots</code></td>
<td>{{ _('Yes') if capabilities.supports_screenshots else _('No') }}</td>
</tr>
<tr>
<td><code>capabilities.supports_visual_selector</code></td>
<td>{{ _('Yes') if capabilities.supports_visual_selector else _('No') }}</td>
</tr>
<tr>
<td><code>supports_browser_steps</code></td>
<td>{{ _('Yes') if capabilities.supports_browser_steps else _('No') }}</td>
</tr>
<tr>
<td><code>visual_selector_data_ready</code></td>
<td>{{ _('Yes') if visual_selector_data_ready else _('No') }}</td>
</tr>
{% if not llm_features_disabled and settings_application.get('llm', {}).get('model') %}
<tr>
<td>{{ _('AI tokens (last check)') }}</td>
@@ -18,6 +18,24 @@ from changedetectionio.blueprint.watchlist.row_context import watch_row_context
def construct_blueprint(datastore: ChangeDetectionStore, update_q, queuedWatchMetaData):
watchlist_blueprint = Blueprint('watchlist', __name__, template_folder="templates")
@watchlist_blueprint.route("/toggle", methods=['POST'])
@login_optionally_required
def toggle():
op = request.args.get('op')
uuid = request.args.get('uuid')
watch = datastore.data['watching'].get(uuid)
if not watch:
flash(_('Watch not found'), 'error')
else:
if op == 'pause':
watch.toggle_pause()
elif op == 'mute':
watch.toggle_mute()
watch.commit()
return redirect(url_for('watchlist.index', tag=request.args.get('tag')))
@watchlist_blueprint.route("/", methods=['GET'])
@login_optionally_required
def index():
@@ -36,17 +54,6 @@ def construct_blueprint(datastore: ChangeDetectionStore, update_q, queuedWatchMe
if request.args.get('rss'):
return redirect(url_for('rss.feed', tag=active_tag_uuid))
op = request.args.get('op')
if op:
uuid = request.args.get('uuid')
if op == 'pause':
datastore.data['watching'][uuid].toggle_pause()
elif op == 'mute':
datastore.data['watching'][uuid].toggle_mute()
datastore.data['watching'][uuid].commit()
return redirect(url_for('watchlist.index', tag = active_tag_uuid))
# Sort by last_changed and add the uuid which is usually the key..
sorted_watches = []
active_processor = request.args.get('processor', '').strip()
@@ -30,10 +30,10 @@ def watch_row_context(datastore, active_tag_uuid=None, queued_uuids=None):
return {
'active_tag_uuid': active_tag_uuid,
# Kept 0 (rather than any_watches_have_processor_by_name) while the price column is
# disabled — it also decides cols_required on the page, so page and row must agree or
# a pushed row ends up with a different <td> count than the table header.
'any_has_restock_price_processor': 0,
# Also decides cols_required on the page, so page and row must agree or a pushed row
# ends up with a different <td> count than the table header — which is why it is
# computed here and not passed separately by the page.
'any_has_restock_price_processor': datastore.any_watches_have_processor_by_name("restock_diff"),
'datastore': datastore,
'has_proxies': datastore.proxy_list,
'processor_descriptions': processors.get_processor_descriptions(),
@@ -11,6 +11,9 @@
{%- set checking_now = is_checking_now(watch) -%}
{%- set history_n = watch.history_n -%}
{%- set favicon = watch.get_favicon_filename() -%}
{# Resolved once per row: each read re-validates the URL and, for a Jinja2 templated
URL, builds a fresh sandboxed environment - see validate_url.py #}
{%- set open_link = watch.open_link -%}
{%- set error_texts = watch.compile_error_texts(has_proxies=has_proxies) -%}
{%- set system_use_url_watchlist = datastore.data['settings']['application']['ui'].get('use_page_title_in_list') -%}
{# Class settings mirrored in changedetectionio/static/js/realtime.js for the frontend #}
@@ -35,10 +38,12 @@
<td class="inline checkbox-uuid" ><div><input name="uuids" type="checkbox" value="{{ watch.uuid}} " >{# <span class="counter-i">{{ loop.index+pagination.skip }}</span>#}</div></td>
<td class="inline watch-controls">
<div>
<a class="ajax-op state-off pause-toggle" data-op="pause" aria-label="{{ _('Pause checks') }}" title="{{ _('Pause checks') }}" href="{{url_for('watchlist.index', op='pause', uuid=watch.uuid, tag=active_tag_uuid)}}"><i data-feather="pause" class="icon icon-pause"></i></a>
<a class="ajax-op state-on pause-toggle" data-op="pause" style="display: none" aria-label="{{ _('UnPause checks') }}" title="{{ _('UnPause checks') }}" href="{{url_for('watchlist.index', op='pause', uuid=watch.uuid, tag=active_tag_uuid)}}"><i data-feather="play" class="icon icon-unpause"></i></a>
<a class="ajax-op state-off mute-toggle" data-op="mute" aria-label="{{ _('Mute notification') }}" title="{{ _('Mute notification') }}" href="{{url_for('watchlist.index', op='mute', uuid=watch.uuid, tag=active_tag_uuid)}}"><i data-feather="bell" class="icon icon-mute"></i></a>
<a class="ajax-op state-on mute-toggle" data-op="mute" style="display: none" aria-label="{{ _('UnMute notification') }}" title="{{ _('UnMute notification') }}" href="{{url_for('watchlist.index', op='mute', uuid=watch.uuid, tag=active_tag_uuid)}}"><i data-feather="bell-off" class="icon icon-mute"></i></a>
{%- set pause_action = url_for('watchlist.toggle', op='pause', uuid=watch.uuid, tag=active_tag_uuid) -%}
{%- set mute_action = url_for('watchlist.toggle', op='mute', uuid=watch.uuid, tag=active_tag_uuid) -%}
<button type="submit" formmethod="post" formaction="{{ pause_action }}" class="bare-btn ajax-op state-off pause-toggle" data-op="pause" aria-label="{{ _('Pause checks') }}" title="{{ _('Pause checks') }}"><i data-feather="pause" class="icon icon-pause"></i></button>
<button type="submit" formmethod="post" formaction="{{ pause_action }}" class="bare-btn ajax-op state-on pause-toggle" data-op="pause" style="display: none" aria-label="{{ _('UnPause checks') }}" title="{{ _('UnPause checks') }}"><i data-feather="play" class="icon icon-unpause"></i></button>
<button type="submit" formmethod="post" formaction="{{ mute_action }}" class="bare-btn ajax-op state-off mute-toggle" data-op="mute" aria-label="{{ _('Mute notification') }}" title="{{ _('Mute notification') }}"><i data-feather="bell" class="icon icon-mute"></i></button>
<button type="submit" formmethod="post" formaction="{{ mute_action }}" class="bare-btn ajax-op state-on mute-toggle" data-op="mute" style="display: none" aria-label="{{ _('UnMute notification') }}" title="{{ _('UnMute notification') }}"><i data-feather="bell-off" class="icon icon-mute"></i></button>
</div>
</td>
@@ -46,7 +51,7 @@
<div class="grid-wrapper">
{% if 'favicons_enabled' not in ui_settings or ui_settings['favicons_enabled'] %}
<div class="favicon">
<a target="_blank" rel="noopener" href="{{ watch.link.replace('source:','') }}">
<a target="_blank" rel="noopener" href="{{ open_link }}">
{# Intersection Observer lazy loading: store real URL in data-src, load only when visible in viewport #}
<img alt="Favicon thumbnail"
class="favicon lazy-favicon"
@@ -72,7 +77,7 @@
{% else %}
{{ watch.get('title') or watch.link }}
{% endif %}
<a class="external" target="_blank" rel="noopener" href="{{ watch.link.replace('source:','') }}">&nbsp;</a>
<a class="external" target="_blank" rel="noopener" href="{{ open_link }}" title="{{ open_link }}">&nbsp;</a>
</span>
{%- for watch_tag_uuid, watch_tag in datastore.get_all_tags_for_watch(watch['uuid']).items() -%}
@@ -81,20 +86,27 @@
<div class="error-text" style="display:none;">{{ error_texts|safe }}</div>
{%- if watch['processor'] == 'text_json_diff' -%}
{%- if watch['has_ldjson_price_data'] and not watch['track_ldjson_price_data'] -%}
<div class="ldjson-price-track-offer">Switch to Restock & Price watch mode? <a href="{{url_for('price_data_follower.accept', uuid=watch.uuid)}}" class="pure-button button-xsmall">Yes</a> <a href="{{url_for('price_data_follower.reject', uuid=watch.uuid)}}" class="">No</a></div>
<div class="ldjson-price-track-offer">Switch to Restock & Price watch mode? <button type="submit" formmethod="post" formaction="{{url_for('price_data_follower.accept', uuid=watch.uuid)}}" class="pure-button button-xsmall">Yes</button> <button type="submit" formmethod="post" formaction="{{url_for('price_data_follower.reject', uuid=watch.uuid)}}" class="bare-btn bare-btn--link">No</button></div>
{%- endif -%}
{%- endif -%}
</div>
<div class="status-icons">
<a class="link-spread" href="{{url_for('ui.form_share_put_watch', uuid=watch.uuid)}}"><img src="{{url_for('static_content', group='images', filename='spread.svg')}}" class="status-icon icon icon-spread" title="{{ _('Create a link to share watch config with others') }}" ></a>
<button type="submit" formmethod="post" formaction="{{url_for('ui.form_share_put_watch', uuid=watch.uuid)}}" class="bare-btn link-spread"><img src="{{url_for('static_content', group='images', filename='spread.svg')}}" class="status-icon icon icon-spread" title="{{ _('Create a link to share watch config with others') }}" ></button>
{%- set effective_fetcher = watch.get_fetch_backend if watch.get_fetch_backend != "system" else system_default_fetcher -%}
{%- if effective_fetcher and ("html_webdriver" in effective_fetcher or "html_" in effective_fetcher or "extra_browser_" in effective_fetcher) -%}
{{ effective_fetcher|fetcher_status_icons }}
{%- endif -%}
{%- if watch.is_pdf -%}<img class="status-icon" src="{{url_for('static_content', group='images', filename='pdf-icon.svg')}}" alt="Converting PDF to text" >{%- endif -%}
{%- if watch.is_pdf -%}<img class="status-icon" src="{{url_for('static_content', group='images', filename='pdf-icon.svg')}}" title="Converting PDF to text" alt="Converting PDF to text" >{%- endif -%}
{%- if watch.has_browser_steps -%}<img class="status-icon status-browsersteps" src="{{url_for('static_content', group='images', filename='steps.svg')}}" alt="Browser Steps is enabled" >{%- endif -%}
</div>
</div>
</td>
{%- if any_has_restock_price_processor -%}
<td class="restock-and-price">
{%- if watch['processor'] == 'restock_diff' -%}
{#- @todo - this could be injected somehow watch.extra_row_info or something -#}
<div class="restock-info-wrap">
@@ -135,15 +147,9 @@
{%- endif -%}
</div>
{%- endif -%}
</div>
</td>
{#
{%- if any_has_restock_price_processor -%}
<td class="restock-and-price">
</td>
{%- endif -%}
#}
{#last_checked becomes fetch-start-time#}
<td class="last-checked" data-timestamp="{{ watch.last_checked }}" data-fetchduration={{ watch.fetch_time }} data-eta_complete="{{ watch.last_checked+watch.fetch_time }}" data-label="{{ _('Checked') }}">
<div class="spinner-wrapper" style="display:none;" >
@@ -163,7 +169,9 @@
<div>
{%- set target_attr = ' target="' ~ watch.uuid ~ '"' if datastore.data['settings']['application']['ui'].get('open_diff_in_new_tab') else '' -%}
<a href="" class="already-in-queue-button recheck cdio-btn cdio-btn--primary cdio-btn--sm" style="display: none;" disabled="disabled"><i data-feather="clock"></i>{{ _('Queued') }}</a>
<a href="{{ url_for('ui.form_watch_checknow', uuid=watch.uuid, tag=request.args.get('tag')) }}" data-op='recheck' class="ajax-op recheck cdio-btn cdio-btn--primary cdio-btn--sm"><i data-feather="refresh-cw"></i>{{ _('Recheck') }}</a>
<button type="submit" formmethod="post"
formaction="{{ url_for('ui.form_watch_checknow', uuid=watch.uuid, tag=request.args.get('tag')) }}"
data-op='recheck' class="ajax-op recheck cdio-btn cdio-btn--primary cdio-btn--sm"><i data-feather="refresh-cw"></i>{{ _('Recheck') }}</button>
<a href="{{ url_for('ui.ui_edit.edit_page', uuid=watch.uuid, tag=active_tag_uuid)}}#general" class="cdio-btn cdio-btn--primary cdio-btn--sm">{{ _('Edit') }}</a>
<a href="{{ url_for('ui.ui_diff.diff_history_page', uuid=watch.uuid)}}" {{target_attr}} class="cdio-btn cdio-btn--primary cdio-btn--sm history-link ai-history-btn" style="display: none;" data-uuid="{{ watch.uuid }}" data-summary-url="{{ url_for('ui.ui_diff.diff_llm_summary', uuid=watch.uuid) }}" data-processor-data-url="{{ url_for('ui.ui_diff.diff_history_page_processor_data', uuid=watch.uuid) }}"><span class="btn-label-history">{{ _('History') }}</span><span class="btn-label-summary">&#x2728; {{ _('Summary') }}</span></a>
<a href="{{ url_for('ui.ui_preview.preview_page', uuid=watch.uuid)}}" {{target_attr}} class="cdio-btn cdio-btn--primary cdio-btn--sm preview-link" style="display: none;">{{ _('Preview') }}</a>
@@ -140,8 +140,10 @@ window.watchOverviewI18n = {
priceLow: {{ _('Currently low')|tojson }},
priceTypical: {{ _('Currently typical')|tojson }},
priceHigh: {{ _('Currently high')|tojson }},
cheaperThan: {{ _('cheaper than %s% of tracked prices')|tojson }},
pricierThan: {{ _('more expensive than %s% of tracked prices')|tojson }},
{# TRANSLATORS: {pct} is a percentage including the % sign, e.g. "80%" #}
cheaperThan: {{ _('cheaper than {pct} of tracked prices')|tojson }},
{# TRANSLATORS: {pct} is a percentage including the % sign, e.g. "80%" #}
pricierThan: {{ _('more expensive than {pct} of tracked prices')|tojson }},
typicalNote: {{ _('around the usual price')|tojson }},
avgLabel: {{ _('avg')|tojson }}
};
@@ -9,6 +9,39 @@ from changedetectionio.content_fetchers.base import get_playwright_bypass_csp, m
from changedetectionio.jinja2_custom import render as jinja_render
from changedetectionio.validate_url import validate_fetch_url_async
def track_latest_navigation_response(page):
"""Record the latest main-frame document response seen on this page, and return the holder.
Idempotent on purpose - every navigation would otherwise add another 'response' listener, and
that event fires once per HTTP response (hundreds on a heavy page), so the callbacks are worth
not duplicating. One tracker per page is installed and then shared by the fetcher and by every
action_goto_url() call on it.
Returns a dict that holds {'response': <latest main-frame document response>}, or None if the
page does not support event listeners (the unit test stubs, mainly).
"""
if not hasattr(page, 'on'):
return None
existing = getattr(page, '_cdio_latest_navigation_response', None)
if existing is not None:
return existing
latest = {}
def _keep(response):
try:
if response.frame == page.main_frame and response.request.is_navigation_request():
latest['response'] = response
except Exception as e:
# Never let a bookkeeping listener break a fetch
logger.debug(f"Could not record navigation response: {e}")
page.on("response", _keep)
page._cdio_latest_navigation_response = latest
return latest
def browser_steps_get_valid_steps(browser_steps: list):
if browser_steps is not None and len(browser_steps):
valid_steps = list(filter(
@@ -145,8 +178,23 @@ class steppable_browser_interface():
# and private-IP SSRF possible via a browser step (GHSA-hm22-wg2m-35v4).
await validate_fetch_url_async(value)
# Chrome 153+ refuses to commit a navigation when an error status arrives with a
# zero-length body, so page.goto() raises net::ERR_HTTP_RESPONSE_CODE_FAILURE instead of
# handing back the response. The response was received fine, we just never get it as a
# return value, so fall back to the page's navigation-response tracker and hand that back -
# callers then report a real "Error - 404" instead of a raw net:: string.
navigation_response = track_latest_navigation_response(self.page)
now = time.time()
response = await self.page.goto(value, timeout=0, wait_until='load')
try:
response = await self.page.goto(value, timeout=0, wait_until='load')
except Exception as e:
if 'ERR_HTTP_RESPONSE_CODE_FAILURE' not in str(e) or not navigation_response:
raise
response = navigation_response['response']
logger.debug(f"Navigation was aborted by the browser (empty body on an error status), "
f"recovered status {response.status} from the response event")
logger.debug(f"Time to goto URL {time.time()-now:.2f}s")
return response
+17 -3
View File
@@ -34,7 +34,13 @@ CUSTOM_OPERATIONS = {
def filter_complete_rules(ruleset):
rules = [
rule for rule in ruleset
if all(value not in ("", False, "None", None) for value in [rule["operator"], rule["field"], rule["value"]])
if all(
rule.get(k) is not None
and rule.get(k) is not False
and rule.get(k) != ""
and rule.get(k) != "None"
for k in ("operator", "field", "value")
)
]
return rules
@@ -54,12 +60,20 @@ def convert_to_jsonlogic(logic_operator: str, rule_dict: list):
field = condition["field"]
value = condition["value"]
if not operator or operator == 'None' or not value or not field:
if (
not operator
or operator == 'None'
or not field
or value is None
or value is False
or value == ""
or value == "None"
):
raise EmptyConditionRuleRowNotUsable()
# Convert value to int/float if possible
try:
if isinstance(value, str) and "." in value and str != "None":
if isinstance(value, str) and "." in value and value != "None":
value = float(value)
else:
value = int(value)
@@ -90,6 +90,7 @@ class Fetcher():
screenshot_format = None
status_code = None
webdriver_js_execute_code = None
worker_id = None
xpath_data = None
xpath_element_js = ""
@@ -120,6 +121,11 @@ class Fetcher():
if kwargs and 'lock_viewport_elements' in kwargs:
self.lock_viewport_elements = kwargs.get('lock_viewport_elements')
# Which async worker is driving this fetch, subclasses use it to keep per-worker browser
# state (profile dirs etc) apart, stays None when we're not called from a worker
if kwargs and 'worker_id' in kwargs:
self.worker_id = kwargs.get('worker_id')
@classmethod
def get_status_icon_data(cls):
@@ -296,6 +296,20 @@ class fetcher(Fetcher):
self.page = await context.new_page()
# Track the LATEST main-frame document response for the whole fetch, not just the one
# goto() returns. This app compares the text of the page the browser ends up on, and a
# site that gates with an interstitial (503/429 + meta-refresh) navigates to the real
# page *during* the extra_wait below - judging the fetch on the first response fails a
# watch whose content is present and fine. Same for plain client-side redirects.
# Shared with action_goto_url() so only one 'response' listener exists on the page.
from changedetectionio.browser_steps.browser_steps import track_latest_navigation_response
# Must be an identity check - the tracker hands back the same (initially empty, so
# falsy) dict the listener writes into, and `or {}` would quietly swap in a different
# one that never gets updated.
latest_navigation_response = track_latest_navigation_response(self.page)
if latest_navigation_response is None:
latest_navigation_response = {}
# Listen for all console events and handle errors
self.page.on("console", lambda msg: logger.debug(f"Playwright console: Watch URL: {url} {msg.type}: {msg.text} {msg.args}"))
@@ -336,6 +350,25 @@ class fetcher(Fetcher):
extra_wait = int(os.getenv("WEBDRIVER_DELAY_BEFORE_CONTENT_READY", 5)) + self.render_extract_delay
await self.page.wait_for_timeout(extra_wait * 1000)
# A meta-refresh or client-side redirect usually lands during that wait, so judge the
# fetch on the document we are actually about to extract rather than the first one.
latest = latest_navigation_response.get('response')
if latest is not None and latest is not response:
logger.debug(f"Page navigated again while waiting, judging the fetch on {latest.url} "
f"(status {latest.status}) instead of the first response for {url}")
response = latest
try:
self.headers = await response.all_headers()
except Exception as e:
logger.debug(f"Could not refresh headers from the final document: {e}")
# Don't extract while a navigation is mid-flight, that is what produces
# "Execution context was destroyed, most likely because of a navigation"
try:
await self.page.wait_for_load_state('load', timeout=extra_wait * 1000)
except Exception as e:
logger.debug(f"Page did not reach a settled load state, continuing anyway: {e}")
try:
self.status_code = response.status
except Exception as e:
+143 -42
View File
@@ -399,56 +399,157 @@ class fetcher(Fetcher):
# Enable Network domain to detect when first bytes arrive
await self.page._client.send('Network.enable')
# Now set up the frame navigation handlers
async def handle_frame_navigation(event=None):
# Wait n seconds after the frameStartedLoading, not from any frameStartedLoading/frameStartedNavigating
logger.debug(f"Frame navigated: {event}")
w = extra_wait - 2 if extra_wait > 4 else 2
logger.debug(f"Waiting {w} seconds before calling Page.stopLoading...")
await asyncio.sleep(w)
# Navigate (bounded), then wait the configured "wait n seconds before extracting text"
# delay, then stop whatever is still loading, then extract. The delay is measured from
# when navigation finished, not from when it started, because the point of it is to let
# JS-rendered content appear *after* load - anchoring it to the start would quietly give a
# slow-loading page almost no settle time.
#
# Only that delay is a user-facing setting. The navigation bound above is a safety net with
# a sane default, not a tuning knob, so there is still one number for users to think about.
#
# There is no way to know a page is "finished" - plenty of sites navigate as part of their
# normal design, and some sit forever on a subresource that never answers. So the delay
# restarts if the MAIN frame replaces its document (a redirect or interstitial gets the
# same settle time the first document got), iframes do not restart it, and it is capped so
# a page that re-navigates in a loop cannot extend it indefinitely.
max_content_ready_resets = int(os.getenv("BROWSER_CONTENT_READY_MAX_RESETS", 2))
# Check if page still exists (might have been closed due to error during sleep)
if not self.page or not hasattr(self.page, '_client'):
logger.debug("Page already closed, skipping stopLoading")
return
async def wait_for_content_ready_then_stop_loading():
main_frame_id = self.page.mainFrame._id
renavigated = asyncio.Event()
logger.debug("Issuing stopLoading command...")
await self.page._client.send('Page.stopLoading')
logger.debug("stopLoading command sent!")
def _on_main_frame_navigation(event):
if event.get('frameId') == main_frame_id:
renavigated.set()
async def setup_frame_handlers_on_first_response(event):
# Only trigger for the main document response
if event.get('type') == 'Document':
logger.debug("First response received, setting up frame handlers for forced page stop load.")
self.page._client.on('Page.frameStartedNavigating', lambda e: asyncio.create_task(handle_frame_navigation(e)))
self.page._client.on('Page.frameStartedLoading', lambda e: asyncio.create_task(handle_frame_navigation(e)))
self.page._client.on('Page.frameStoppedLoading', lambda e: logger.debug(f"Frame stopped loading: {e}"))
logger.debug("First response received, setting up frame handlers for forced page stop load DONE SETUP")
# De-register this listener - we only need it once
self.page._client.remove_listener('Network.responseReceived', setup_frame_handlers_on_first_response)
self.page._client.on('Page.frameStartedLoading', _on_main_frame_navigation)
self.page._client.on('Page.frameStoppedLoading', lambda e: logger.debug(f"Frame stopped loading: {e}"))
try:
resets = 0
while True:
renavigated.clear()
try:
await asyncio.wait_for(renavigated.wait(), timeout=extra_wait)
# Main frame started a new document
resets += 1
if resets > max_content_ready_resets:
logger.debug(f"Main frame keeps re-navigating, not restarting the content-ready wait again")
break
logger.debug(f"Main frame started a new document, restarting the {extra_wait}s "
f"content-ready wait ({resets}/{max_content_ready_resets})")
except asyncio.TimeoutError:
# Quiet for the whole delay - the page is as ready as it is going to get
break
finally:
self.page._client.remove_listener('Page.frameStartedLoading', _on_main_frame_navigation)
# Listen for first response to trigger frame handler setup
self.page._client.on('Network.responseReceived', setup_frame_handlers_on_first_response)
# Stop whatever is still in flight so the DOM and screenshot come from what rendered,
# rather than waiting on a subresource that may never answer
try:
logger.debug(f"Content-ready wait of {extra_wait}s elapsed, issuing Page.stopLoading before extracting")
await self.page._client.send('Page.stopLoading')
logger.debug("stopLoading command sent!")
except Exception as e:
logger.debug(f"Page.stopLoading skipped, page is most likely already gone: {e}")
# Track the LATEST main-frame document response for the whole fetch, not just the one that
# goto() happens to return. This app compares the text of the page the browser ends up on,
# and plenty of sites navigate again after the first response:
# - an interstitial answering 503/429 with a meta-refresh into the real 200 page, where
# judging the first response fails a watch whose content is sitting right there
# - a plain client-side redirect to another host (slated.com -> get.slated.com)
# It also covers Chrome 153+, which refuses to commit a navigation when an error status
# arrives with a zero-length body: goto() raises net::ERR_HTTP_RESPONSE_CODE_FAILURE rather
# than returning the response, but the response itself still arrives on this event.
navigation_response = {}
def _keep_navigation_response(response):
# Note pyppeteer exposes these as properties, unlike playwright where they are methods
if response.frame == self.page.mainFrame and response.request.isNavigationRequest:
navigation_response['response'] = response
self.page.on('response', _keep_navigation_response)
# pyppeteer's navigation watcher is bound to the loaderId of the navigation it started. If
# the page replaces that document (redirect/interstitial) the 'load' it waits for never
# arrives for that loaderId, so goto() never returns - and with timeout=0 it would block
# until the hard PUPPETEER_MAX_PROCESSING_TIMEOUT_SECONDS kill, burning a worker slot for
# minutes on a page that is fully loaded. Bound it, then fall back to the document we can
# see. Verified against slated.com and getastra.com, which hang indefinitely otherwise.
nav_timeout = int(os.getenv("BROWSER_NAVIGATION_TIMEOUT_SECONDS", 30))
response = None
attempt=0
while not response:
logger.debug(f"Attempting page fetch {url} attempt {attempt}")
asyncio.create_task(handle_frame_navigation())
response = await self.page.goto(url, timeout=0)
await asyncio.sleep(1 + extra_wait)
# Check if page still exists before sending command
if self.page and hasattr(self.page, '_client'):
await self.page._client.send('Page.stopLoading')
try:
while not response:
logger.debug(f"Attempting page fetch {url} attempt {attempt}")
# Race goto() against the main frame actually firing 'load'. In the re-navigation
# case goto() can never resolve, but the replacement document does fire 'load' -
# usually within a few seconds - so this returns then instead of sitting out the
# whole nav_timeout. Whichever arrives first means "the document is loaded".
main_frame_loaded = asyncio.Event()
main_frame_id = self.page.mainFrame._id
if response:
break
if not response:
logger.warning("Page did not fetch! trying again!")
if response is None and attempt>=2:
logger.warning(f"Content Fetcher > Response object was none (as in, the response from the browser was empty, not just the content) exiting attempt {attempt}")
raise EmptyReply(url=url, status_code=None)
attempt+=1
def _on_lifecycle(event):
if event.get('name') == 'load' and event.get('frameId') == main_frame_id:
main_frame_loaded.set()
self.page._client.on('Page.lifecycleEvent', _on_lifecycle)
goto_task = asyncio.ensure_future(self.page.goto(url, timeout=0))
load_task = asyncio.ensure_future(main_frame_loaded.wait())
try:
done, _pending = await asyncio.wait({goto_task, load_task},
timeout=nav_timeout,
return_when=asyncio.FIRST_COMPLETED)
if goto_task in done:
try:
response = goto_task.result()
except Exception as e:
if 'ERR_HTTP_RESPONSE_CODE_FAILURE' not in str(e) or not navigation_response:
raise
response = navigation_response['response']
logger.debug(f"Navigation was aborted by the browser (empty body on an error status), "
f"recovered status {response.status} from the response event")
else:
# Either the replacement document loaded, or we ran out of patience
response = navigation_response.get('response')
if not response:
raise BrowserFetchTimedOut(msg=f"Browser did not finish navigating to {url} within "
f"{nav_timeout}s and no main-frame response was seen.")
why = ("the page replaced the document it started on" if load_task in done
else f"navigation did not settle within {nav_timeout}s")
logger.warning(f"Continuing with the document actually loaded ({why}) - "
f"status {response.status} for {response.url}")
finally:
self.page._client.remove_listener('Page.lifecycleEvent', _on_lifecycle)
for t in (goto_task, load_task):
if not t.done():
t.cancel()
if response:
break
if not response:
logger.warning("Page did not fetch! trying again!")
if response is None and attempt>=2:
logger.warning(f"Content Fetcher > Response object was none (as in, the response from the browser was empty, not just the content) exiting attempt {attempt}")
raise EmptyReply(url=url, status_code=None)
attempt+=1
# Navigation is done; now honour "wait n seconds before extracting text" and then
# force-stop whatever is still loading, so extraction always gets what rendered.
# Awaited inline rather than fired off as a task, so nothing can outlive the fetch.
await wait_for_content_ready_then_stop_loading()
# That wait is where a meta-refresh interstitial typically swaps in the real page, so
# re-check which document we are actually on before judging the status code.
latest = navigation_response.get('response')
if latest is not None and latest is not response:
logger.debug(f"Page navigated again while waiting, judging the fetch on {latest.url} "
f"(status {latest.status}) instead of {response.url} (status {response.status})")
response = latest
finally:
self.page.remove_listener('response', _keep_navigation_response)
self.headers = response.headers
File diff suppressed because it is too large Load Diff
+17 -3
View File
@@ -4,6 +4,7 @@ from loguru import logger
from wtforms.widgets.core import TimeInput
from flask_babel import lazy_gettext as _l, gettext
from changedetectionio.blueprint.menu_modes import MENU_SIDEBAR_ACTIONMODES, MENU_SIDEBAR_ACTIONMODES_DEFAULT
from changedetectionio.blueprint.rss import RSS_FORMAT_TYPES, RSS_TEMPLATE_TYPE_OPTIONS, RSS_TEMPLATE_HTML_DEFAULT
from changedetectionio.llm.ui_strings import LLM_INTENT_WATCH_PLACEHOLDER
from changedetectionio.llm.evaluator import (
@@ -939,6 +940,7 @@ class SingleBrowserStep(Form):
class processor_text_json_diff_form(commonSettingsForm):
url = StringField(_l('Web Page URL'), validators=[validateURL()])
link_to_open = StringField(_l('Open Link Override'), validators=[validators.Optional(), validateURL()], default='')
tags = StringTagUUID(_l('Group Tag'), [validators.Optional()], default='')
time_between_check = EnhancedFormField(
@@ -1046,6 +1048,19 @@ class processor_text_json_diff_form(commonSettingsForm):
self.url.errors.append(gettext('Invalid template syntax: %(error)s') % {'error': e})
result = False
# Attempt to validate jinja2 templates in the optional "Link to Open"
if self.link_to_open.data and self.link_to_open.data.strip():
try:
jinja_render(template_str=self.link_to_open.data)
except ModuleNotFoundError as e:
logger.error(e)
self.link_to_open.errors.append(gettext('Invalid template syntax configuration: %(error)s') % {'error': e})
result = False
except Exception as e:
logger.error(e)
self.link_to_open.errors.append(gettext('Invalid template syntax: %(error)s') % {'error': e})
result = False
# Attempt to validate jinja2 templates in the body
if self.body.data and self.body.data.strip():
try:
@@ -1164,9 +1179,8 @@ class globalSettingsApplicationUIForm(Form):
choices=[('long', _l('Long (1 minute ago)')), ('short', _l('Short (1m ago)'))],
default='long', validators=[validators.Optional()])
sidebar_mode = SelectField(_l('Navigation sidebar'),
choices=[('collapsed', _l('Collapsed icon rail (expands on hover)')),
('pinned', _l('Always expanded'))],
default='collapsed', validators=[validators.Optional()])
choices=MENU_SIDEBAR_ACTIONMODES,
default=MENU_SIDEBAR_ACTIONMODES_DEFAULT, validators=[validators.Optional()])
# datastore.data['settings']['application']..
class globalSettingsApplicationForm(commonSettingsForm):
+3
View File
@@ -19,6 +19,7 @@ def get_timeago_locale(flask_locale, short=False):
- Swedish: Flask uses 'sv', timeago uses 'sv_SE'
- Norwegian: Flask uses 'no', timeago uses 'nb_NO' or 'nn_NO'
- Hindi: Flask uses 'hi', timeago uses 'in_HI'
- Indonesian: Flask uses 'id', timeago uses 'in_ID'
- Czech: Flask uses 'cs', but timeago doesn't support Czech - fallback to English
Args:
@@ -46,6 +47,7 @@ def get_timeago_locale(flask_locale, short=False):
'sv': 'sv_SE', # Swedish
'no': 'nb_NO', # Norwegian Bokmål
'hi': 'in_HI', # Hindi
'id': 'in_ID', # Indonesian
'cs': 'en', # Czech not supported by timeago, fallback to English
'ja': 'ja', # Japanese
'uk': 'uk', # Ukrainian
@@ -156,6 +158,7 @@ LANGUAGE_DATA = {
'tr': {'flag': 'fi fi-tr fis', 'name': 'Türkçe'},
'ar': {'flag': 'fi fi-sa fis', 'name': 'العربية'},
'hi': {'flag': 'fi fi-in fis', 'name': 'हिन्दी'},
'id': {'flag': 'fi fi-id fis', 'name': 'Bahasa Indonesia'},
'uk': {'flag': 'fi fi-ua fis', 'name': 'Українська'},
}
+2 -1
View File
@@ -1,6 +1,7 @@
from os import getenv
from copy import deepcopy
from changedetectionio.blueprint.menu_modes import MENU_SIDEBAR_ACTIONMODES_DEFAULT
from changedetectionio.blueprint.rss import RSS_FORMAT_TYPES, RSS_CONTENT_FORMAT_DEFAULT
from changedetectionio.model.Tags import TagsDict
@@ -80,7 +81,7 @@ class model(dict):
'socket_io_enabled': True,
'favicons_enabled': True,
'timeago_format': 'long', # 'long' = "1 minute ago", 'short' = "1m ago"
'sidebar_mode': 'collapsed', # 'collapsed' = slim icon rail, expands on hover; 'pinned' = always expanded
'sidebar_mode': MENU_SIDEBAR_ACTIONMODES_DEFAULT, # one of blueprint.menu_modes.MENU_SIDEBAR_ACTIONMODES
},
}
}
+42 -8
View File
@@ -272,10 +272,12 @@ class model(EntityPersistenceMixin, watch_base):
def has_unviewed(self):
return int(self.newest_history_key) > int(self['last_viewed']) and self.__history_n >= 2
@property
def link(self):
def _resolve_link(self, url, flash_on_template_error=True):
"""Validate, Jinja2-render and de-'source:' a URL so it is safe to put in an href.
url = self.get('url', '')
Returns 'DISABLED' when the URL is missing/unsafe, or '' when its Jinja2 template
could not be rendered.
"""
if not is_safe_valid_url(url):
return 'DISABLED'
@@ -286,11 +288,15 @@ class model(EntityPersistenceMixin, watch_base):
ready_url = jinja_render(template_str=url)
except Exception as e:
logger.critical(f"Invalid URL template for: '{url}' - {str(e)}")
from flask import flash, url_for
from markupsafe import Markup
message = Markup('<a href="{}#general">The URL {} is invalid and cannot be used, click to edit</a>'.format(
url_for('ui.ui_edit.edit_page', uuid=self.get('uuid')), self.get('url', '')))
flash(message, 'error')
# has_request_context() guard: this is also reached from worker threads and the
# notification path, where flash() would raise "Working outside of request context".
from flask import has_request_context
if flash_on_template_error and has_request_context():
from flask import flash, url_for
from markupsafe import Markup
message = Markup('<a href="{}#general">The URL {} is invalid and cannot be used, click to edit</a>'.format(
url_for('ui.ui_edit.edit_page', uuid=self.get('uuid')), url))
flash(message, 'error')
return ''
if ready_url.startswith('source:'):
@@ -301,6 +307,34 @@ class model(EntityPersistenceMixin, watch_base):
return 'DISABLED'
return ready_url
@property
def link(self):
"""The URL that actually gets fetched/checked."""
return self._resolve_link(self.get('url', ''))
@property
def open_link_override(self):
"""The resolved optional 'Link to Open', or '' when unset/unusable."""
override = str(self.get('link_to_open') or '').strip()
if not override:
return ''
# Don't flash on a broken template here - `link` already surfaces that for the watched URL,
# and a bad "Link to Open" should never stop the page rendering.
ready = self._resolve_link(override, flash_on_template_error=False)
if not ready or ready == 'DISABLED':
return ''
return ready
@property
def open_link(self):
"""The URL a human should be sent to when they click through to "the site".
Some watches point at an API endpoint or RSS feed that is useless in a browser -
'Link to Open' lets the user store the real page for those. Falls back to `link`.
"""
return self.open_link_override or self.link
@property
def domain_only_from_link(self):
from urllib.parse import urlparse
+1
View File
@@ -207,6 +207,7 @@ class watch_base(dict):
'last_error': False,
'last_notification_error': None,
'last_viewed': 0, # history key value of the last viewed via the [diff] link
'link_to_open': '', # Optional human-facing URL to open instead of 'url' (eg. watch an API/RSS endpoint, open the real page)
'llm_backend_profile': True, # @note - now its just a bool but in the near future we can select a LLM profile or 'off'/false
'llm_change_summary': '', # Prompt for AI Change Summary — replaces {{ diff }} in notifications
'llm_change_summary_mode': 'replace', # 'replace' the inherited prompt, or 'append' to it
@@ -241,6 +241,7 @@ class NotificationContextData(dict):
'watch_tag': None,
'watch_title': None,
'watch_url': 'https://WATCH-PLACE-HOLDER/',
'watch_open_url': 'https://WATCH-PLACE-HOLDER/', # watch['link_to_open'] when set, otherwise the same as watch_url
'watch_uuid': 'XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX', # Converted to 'watch_uuid' in create_notification_parameters
})
@@ -350,6 +351,7 @@ def set_basic_notification_vars(current_snapshot, prev_snapshot, watch, triggere
'triggered_text': triggered_text,
'uuid': watch.get('uuid') if watch else None,
'watch_url': watch.get('url') if watch else None,
'watch_open_url': (watch.open_link_override or watch.get('url')) if watch else None,
'watch_uuid': watch.get('uuid') if watch else None,
'watch_mime_type': watch.get('content-type')
}
@@ -511,6 +513,7 @@ Thanks - Your omniscient changedetection.io installation.
if 'notification_urls' in n_object:
n_object.update({
'watch_url': watch['url'],
'watch_open_url': watch.open_link_override or watch['url'],
'uuid': watch_uuid,
'screenshot': None
})
@@ -560,6 +563,7 @@ Thanks - Your omniscient changedetection.io installation.
if 'notification_urls' in n_object:
n_object.update({
'watch_url': watch['url'],
'watch_open_url': watch.open_link_override or watch['url'],
'uuid': watch_uuid
})
self.notification_q.put(n_object)
+8 -2
View File
@@ -22,11 +22,16 @@ class difference_detection_processor():
preferred_proxy = None
screenshot_format = SCREENSHOT_FORMAT_JPEG
last_raw_content_checksum = None
worker_id = None
def __init__(self, datastore, watch_uuid):
def __init__(self, datastore, watch_uuid, worker_id=None):
self.datastore = datastore
self.watch_uuid = watch_uuid
# Which async worker is driving this check, passed down to the fetcher in call_browser()
# so it can keep per-worker browser state apart, None when we're not called from a worker
self.worker_id = worker_id
# Create a stable snapshot of the watch for processing
# Why deepcopy?
# 1. Prevents "dict changed during iteration" errors if watch is modified during processing
@@ -201,7 +206,8 @@ class difference_detection_processor():
# When browser_connection_url is None, it method should default to working out whats the best defaults (os env vars etc)
self.fetcher = fetcher_obj(proxy_override=proxy_url,
custom_browser_connection_url=custom_browser_connection_url,
screenshot_format=self.screenshot_format
screenshot_format=self.screenshot_format,
worker_id=self.worker_id
)
# Stamp the resolved backend name so downstream consumers (processors, plugins)
@@ -430,7 +430,7 @@ def render(watch, datastore, request, url_for, render_template, flash, redirect)
change_percentage=change_percentage,
comparison_data=comparison_data, # Full history for charts/visualization
comparison_method=method_display,
current_diff_url=watch['url'],
current_diff_url=watch.open_link,
from_version=from_version,
percentage_different=change_percentage,
threshold=pixel_difference_threshold_sensitivity,
@@ -106,5 +106,5 @@ def render(watch, datastore, request, url_for, render_template, flash, redirect)
uuid=watch.get('uuid'),
versions=versions,
timestamp=timestamp,
current_diff_url=watch['url']
current_diff_url=watch.open_link
)
@@ -191,7 +191,7 @@ def render(watch, datastore, request, url_for, render_template, flash, redirect,
'restock_diff/difference.html',
uuid=uuid,
watch=watch,
current_diff_url=watch['url'],
current_diff_url=watch.open_link,
extra_title=f" - {watch.label} - {gettext('Price history')}",
last_error=watch['last_error'],
screenshot=watch.get_screenshot(),
@@ -86,8 +86,10 @@
price_low: {{ _('Currently low')|tojson }},
price_typical: {{ _('Currently typical')|tojson }},
price_high: {{ _('Currently high')|tojson }},
cheaper_than: {{ _('cheaper than %s% of tracked prices')|tojson }},
pricier_than: {{ _('more expensive than %s% of tracked prices')|tojson }},
{# TRANSLATORS: {pct} is a percentage including the % sign, e.g. "80%" #}
cheaper_than: {{ _('cheaper than {pct} of tracked prices')|tojson }},
{# TRANSLATORS: {pct} is a percentage including the % sign, e.g. "80%" #}
pricier_than: {{ _('more expensive than {pct} of tracked prices')|tojson }},
typical_note: {{ _('around the usual price')|tojson }},
avg_label: {{ _('avg')|tojson }}
};
@@ -216,7 +216,7 @@ def render(watch, datastore, request, url_for, render_template, flash, redirect,
#initial_scroll_line_number=100,
bottom_horizontal_offscreen_contents=offscreen_content,
content=content,
current_diff_url=watch['url'],
current_diff_url=watch.open_link,
diff_cell_grid=diff_cell_grid,
diff_prefs=diff_prefs,
extra_classes=' '.join(filter(None, ['difference-page', 'llm-configured' if llm_configured else ''])),
+6
View File
@@ -9,6 +9,12 @@
# exit when any command fails
set -e
# Failing fast is deliberate here (the first failure is usually the real problem, and it keeps
# the run short) - but this script runs 8 independent pytest groups, so make it obvious that the
# groups after the failure were SKIPPED rather than passed. Otherwise one failing test reads as
# "the whole basic suite is broken".
trap 'rc=$?; echo "::error::run_basic_tests.sh aborted at line $LINENO (exit $rc) - the test groups after this point were SKIPPED, not run"' ERR
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
rm tests/logs/* -f
+4 -2
View File
@@ -78,7 +78,9 @@ $(document).ready(function () {
// bootstrap it, this will trigger everything else
$('#browsersteps-img').bind('load', function () {
$('body').addClass('full-width');
console.log("Loaded background...");
console.log(`Loaded background ${this.naturalWidth}px` );
// For the UI width of the whole edit area
document.documentElement.style.setProperty('--browser-steps-max-width', `${this.naturalWidth+200}px` );
document.getElementById("browsersteps-selector-canvas");
c = document.getElementById("browsersteps-selector-canvas");
@@ -284,7 +286,7 @@ $(document).ready(function () {
$('#browser-steps-ui .loader .spinner').show();
// Request a new session
$.ajax({
type: "GET",
type: "POST",
url: browser_steps_start_url,
statusCode: {
400: function () {
+1 -1
View File
@@ -73,7 +73,7 @@ $(function () {
// Request start, needs CSRF?
$.ajax({
type: "GET",
type: "POST",
url: recheck_proxy_start_url,
}).done(function (data) {
$.each(data, function (proxy_key, state) {
@@ -23,8 +23,11 @@
no_data: 'No price data available to graph yet.', load_error: 'Could not load price history.',
changes: 'Changes', avg_price: 'Average price',
price_low: 'Currently low', price_typical: 'Currently typical', price_high: 'Currently high',
cheaper_than: 'cheaper than %s% of tracked prices',
pricier_than: 'more expensive than %s% of tracked prices',
// {pct} is substituted with an already-formatted percentage, e.g. "80%". The percent sign is
// NOT part of the msgid on purpose: a bare "%" followed by a letter (" of") parses as a
// python-format conversion ("% o"), which makes pybabel reject any translation of it.
cheaper_than: 'cheaper than {pct} of tracked prices',
pricier_than: 'more expensive than {pct} of tracked prices',
typical_note: 'around the usual price', avg_label: 'avg' };
const OVERLAY_MIN_POINTS = 5; // need enough history for low/typical/high to be meaningful
@@ -260,9 +263,9 @@
$pill.append($('<span class="rg-status-label"></span>').text(i18n['price_' + summary.status] || ''));
let sub;
if (summary.status === 'low') {
sub = (i18n.cheaper_than || '').replace('%s', summary.cheaper_than_pct);
sub = (i18n.cheaper_than || '').replace('{pct}', summary.cheaper_than_pct + '%');
} else if (summary.status === 'high') {
sub = (i18n.pricier_than || '').replace('%s', summary.pricier_than_pct);
sub = (i18n.pricier_than || '').replace('{pct}', summary.pricier_than_pct + '%');
} else {
sub = i18n.typical_note || '';
}
+12 -39
View File
@@ -7,7 +7,6 @@
// The Search button is rendered in the left rail and the mobile drawer.
const openSearchButtons = document.querySelectorAll('.js-open-search-modal');
const closeSearchButton = document.getElementById('close-search-modal');
const searchForm = document.getElementById('search-form');
const searchInput = document.getElementById('search-modal-input');
if (!searchModal || openSearchButtons.length === 0) {
@@ -63,6 +62,15 @@
// Close modal when clicking the backdrop
searchModal.addEventListener('click', function(e) {
// Only real pointer clicks can land on the backdrop. Keyboard-synthesised clicks
// report detail 0 and coordinates of 0,0, which the geometry test below reads as
// "outside the dialog" - and implicit form submission (Enter in the input) fires
// exactly such a click at the Search button. That closed the modal and blanked
// the input mid-dispatch, so the submit that followed hit an empty `required`
// field and was rejected: Enter appeared to just dismiss the form.
if (e.detail === 0) {
return;
}
const rect = searchModal.getBoundingClientRect();
const isInDialog = (
rect.top <= e.clientY &&
@@ -93,43 +101,8 @@
}
});
// Handle Enter key in search input
if (searchInput) {
searchInput.addEventListener('keydown', function(e) {
if (e.key === 'Enter') {
e.preventDefault();
if (searchForm) {
// Trigger form submission programmatically
searchForm.dispatchEvent(new Event('submit'));
}
}
});
}
// Handle form submission
if (searchForm) {
searchForm.addEventListener('submit', function(e) {
e.preventDefault();
// Get form data
const formData = new FormData(searchForm);
const searchQuery = formData.get('q');
const tags = formData.get('tags');
// Build URL
const params = new URLSearchParams();
if (searchQuery) {
params.append('q', searchQuery);
}
if (tags) {
params.append('tags', tags);
}
// Navigate to search results (always redirect to watchlist home)
// Use base_path if available (for sub-path deployments like /enlighten-richerx)
const basePath = typeof base_path !== 'undefined' ? base_path : '';
window.location.href = basePath + '/?' + params.toString();
});
}
// Submission is left to the browser: the form carries a server-rendered action
// (correct under a reverse-proxy sub-path) and Enter in the input triggers implicit
// submission via the footer's submit button, which also runs `required` validation.
});
})();
+9 -7
View File
@@ -1,13 +1,15 @@
// Left-rail expand/collapse state.
// Adds `action-side-bar-expanded` to <body> whenever the rail is showing its
// labels. In pinned mode (body.actionside-bar-on) the class is already present
// from page load; in collapsed mode (body.actionsidebar-minimal) the rail only
// expands on hover/focus, so we toggle the class to match.
// Adds `actionsidebar-expanded` to <body> whenever the rail is showing its
// labels. In 'pinned-expanded' mode (body.actionside-bar-on) the class is already
// present from page load; in 'expandable' mode (body.actionsidebar-minimal) the rail
// only expands on hover/focus, so we toggle the class to match. 'minimal' mode is
// also the collapsed rail but carries `actionsidebar-no-expand` and never rolls out.
(function() {
'use strict';
document.addEventListener('DOMContentLoaded', function() {
if (!document.body.classList.contains('actionsidebar-minimal')) {
if (!document.body.classList.contains('actionsidebar-minimal') ||
document.body.classList.contains('actionsidebar-no-expand')) {
return;
}
@@ -16,8 +18,8 @@
return;
}
const expand = () => document.body.classList.add('action-side-bar-expanded');
const collapse = () => document.body.classList.remove('action-side-bar-expanded');
const expand = () => document.body.classList.add('actionsidebar-expanded');
const collapse = () => document.body.classList.remove('actionsidebar-expanded');
inner.addEventListener('mouseenter', expand);
inner.addEventListener('mouseleave', collapse);
@@ -115,7 +115,9 @@ window.initVisualSelector = function (opts) {
$('#selector-current-xpath, #clear-selector').hide();
})
.on('load', () => {
console.log("Loaded background...");
console.log(`Loaded background ${$selectorBackgroundElem[0].naturalWidth}px`);
// For the UI width of the whole edit area
document.documentElement.style.setProperty('--visualselector-max-width', `${$selectorBackgroundElem[0].naturalWidth}px` );
c = $selectorCanvasElem[0];
xctx = c.getContext("2d");
ctx = c.getContext("2d");
@@ -32,21 +32,27 @@ $action-sidebar-content-slot: 1100px;
// Hidden on mobile: the mobile drawer (hamburger) carries these items
display: none;
}
}
body.actionside-bar-on {
.action-sidebar {
/* width: $action-sidebar-width-expanded; // reserve enough to host the expanded inner block*/
// MINIMAL MODE: the rail's footprint in the flex row is frozen at the
// collapsed width (+ the inner block's horizontal padding, which is
// content-box). The inner block is taken out of flow (absolute, below) so
// its hover expansion rolls out ON TOP of the page content instead of
// widening this flex item and reflowing `.app-main`.
body.actionsidebar-minimal & {
flex: 0 0 auto;
width: calc(#{$action-sidebar-width-collapsed} + #{$common-gap * 2});
overflow: visible; // let the expanded inner block escape the frame
}
}
// The actual interactive item block — wraps only the items, not the whole height.
//
// Two body-level modes drive the rail's display state:
// body.actionside-bar-on → always expanded (icons + labels visible all the time)
// body.actionsidebar-minimal → icon-only collapsed rail; expands on hover/focus
// `actionside-bar-on` is the default applied in templates/base.html. Both the
// hover-to-expand width animation and the label fade-in are gated on the
// Body-level classes drive the rail's display state (emitted by
// flask_app.get_sidebar_mode_class() from the `sidebar_mode` setting):
// body.actionside-bar-on → always expanded (icons + labels visible all the time)
// body.actionsidebar-minimal → icon-only collapsed rail; expands on hover/focus
// body.actionsidebar-no-expand → alongside the above: stays collapsed, no hover-out
// Both the hover-to-expand width animation and the label fade-in are gated on the
// minimal class so that the always-expanded mode (or no mode) doesn't trigger
// a needless layout jump when the user mouses across the rail.
.action-sidebar-inner {
@@ -64,12 +70,40 @@ body.actionside-bar-on {
padding-left: $common-gap;
padding-right: $common-gap;
// Hover-to-expand only fires in minimal mode.
body.actionsidebar-minimal &:hover,
body.actionsidebar-minimal &:focus-within {
// MINIMAL MODE: lift the block out of the flex flow so growing it can't
// resize the rail (and therefore can't shift the content column). The rail
// is `position: sticky`, i.e. a positioned element, so top/left/bottom here
// anchor to it; top+bottom keep the column full-height, which the footer
// list's `margin-top: auto` still needs.
//
// Once it floats above the page it needs its own backdrop, or the content
// it rolls over would read straight through the rail's translucent 5% white.
// It can't just be a flat colour: the page background is a fixed, viewport-
// sized gradient (`body:after`), so the rail repaints that same gradient with
// `background-attachment: fixed` — which makes the viewport the positioning
// area — and lays the rail's own 5% white on top. Result: opaque, but
// pixel-identical to the translucent rail it replaces, at rest and rolled out.
body.actionsidebar-minimal & {
position: absolute;
top: 0;
left: 0;
bottom: 0;
z-index: 1;
background-color: var(--color-background-page); // fallback under the gradient
background-image:
linear-gradient(rgba(255, 255, 255, 0.05), rgba(255, 255, 255, 0.05)),
var(--page-background-gradient);
background-attachment: fixed, fixed;
}
// Hover-to-expand only fires in minimal mode, and not for 'minimal' (no-expand).
body.actionsidebar-minimal:not(.actionsidebar-no-expand) &:hover,
body.actionsidebar-minimal:not(.actionsidebar-no-expand) &:focus-within {
width: $action-sidebar-width-expanded;
// Enter is smoother.
transition: width 0.22s cubic-bezier(0.2, 0.7, 0.2, 1);
// Lift it off the page while it's rolled out over the content.
box-shadow: var(--color-sidebar-shadow);
}
// Always-expanded mode: no hover gate; the rail just sits open.
@@ -228,7 +262,7 @@ ul.action-sidebar-list {
letter-spacing: 0.06em;
font-weight: 700;
pointer-events: none;
opacity: 0;
opacity: 0.9;
transition: opacity 0.05s ease-out;
// Unread-count variant: a small accent bubble pinned to the right of the row
@@ -247,11 +281,17 @@ ul.action-sidebar-list {
}
}
// Count badges (unread, queue) are always visible in minimal mode so users
// can see at a glance how many items need attention without expanding the rail.
body.actionsidebar-minimal .action-badge--count {
opacity: 1;
}
// When the inner block expands in MINIMAL mode, reveal every label/badge
// together with a smooth enter. Always-expanded mode handles labels via the
// `body.actionside-bar-on` block below — no hover gate needed there.
body.actionsidebar-minimal .action-sidebar-inner:hover,
body.actionsidebar-minimal .action-sidebar-inner:focus-within {
body.actionsidebar-minimal:not(.actionsidebar-no-expand) .action-sidebar-inner:hover,
body.actionsidebar-minimal:not(.actionsidebar-no-expand) .action-sidebar-inner:focus-within {
.action-sidebar-item {
.action-label {
opacity: 1;
@@ -441,7 +481,7 @@ body.actionside-bar-on {
}
}
&.action-side-bar-expanded {
&.actionsidebar-expanded {
#checking-now-stats-sidebar{
display: block;
}
@@ -449,15 +489,29 @@ body.actionside-bar-on {
#logo-expanded {
display: inline-block;
}
#logo-short {
display: none;
}
}
}
}
.action-side-bar-expanded {
#cdio-logo {
#logo-short {
&:not(.actionsidebar-expanded) {
.action-label {
display: none;
}
.action-badge--count {
position: relative;
left: -8px;
top: -8px;
}
.action-sidebar-item.is-disabled {
.action-badge {
display: none;
}
}
}
}
@@ -79,7 +79,6 @@
width: 100%;
overflow-y: scroll;
position: relative;
height: 80vh;
> img {
position: absolute;
@@ -3,9 +3,13 @@
// Clean outline button — surface background, hairline border, muted label that
// darkens on hover. The app's neutral/secondary action button (toolbars, list
// actions), a calmer alternative to the solid pure-button. Pairs with .seg.
.cdio-btn {
display: inline-flex;
align-items: center;
vertical-align: middle;
box-sizing: border-box;
gap: 6px;
height: 32px;
padding: 0 12px;
@@ -91,3 +95,26 @@
&:hover { color: #d68a00; border-color: #d68a00; }
}
}
// State-mutating controls have to POST, and only a <button> can submit - these strip
// the browser's button chrome so they render exactly like the <a> they replaced.
.bare-btn {
background: none;
border: 0;
padding: 0;
margin: 0;
font: inherit;
// Matches `a { color: var(--color-link) }` - these replace anchors, and the row
// icons stroke with currentColor, so `inherit` would pick up .watch-controls red.
color: var(--color-link);
cursor: pointer;
&:focus-visible {
outline: 2px solid var(--color-link);
outline-offset: 2px;
}
&--link {
text-decoration: underline;
}
}
@@ -95,7 +95,9 @@
overflow-y: auto;
padding-top: 60px;
.action-label {
display: block !important;
}
#cdio-logo {
color: var(--color-text);
@@ -122,7 +124,7 @@
li {
border-bottom: 1px solid var(--color-border-table-cell);
>* {
>*, >form>button {
display: block;
padding: 1rem 1.5rem;
color: var(--color-text);
@@ -134,6 +136,14 @@
background: var(--color-background-menu-link-hover);
}
}
// Buttons shrink-wrap and centre their label; anchors don't. No global
// border-box reset here, so this must not be folded into the rule above.
>form>button {
box-sizing: border-box;
width: 100%;
text-align: left;
}
&#menu-pause, &#menu-mute {
display: none;
}
@@ -1,7 +1,7 @@
#language-selector-flag {
display: inline-block;
width: 1.2em;
height: 1.2em;
width: 22px;
height: 22px;
vertical-align: middle;
border-radius: 50%;
overflow: hidden;
@@ -18,10 +18,20 @@
flex-direction: column;
gap: 0.5rem;
padding: 0.5rem 0;
// The csrf mini-form around each option is layout-transparent, so the buttons
// stay the flex items.
> form {
display: contents;
}
}
.language-option {
display: flex;
background: none;
font: inherit;
text-align: left;
cursor: pointer;
align-items: center;
gap: 1rem;
padding: 0.25rem;
@@ -15,6 +15,13 @@
.pure-menu-item {
height: initial;
// Mini POST forms (pause/mute/log out need a csrf_token) are layout-transparent,
// so the button inside sits where the plain <a> used to.
> form {
display: contents;
}
svg {
height: 1.2rem;
}
@@ -16,7 +16,7 @@
}
&.toast-top-center {
top: 100px;
top: 120px;
left: 50%;
transform: translateX(-50%);
}
@@ -128,6 +128,8 @@ ul#top-right-menu {
font-weight: 600;
white-space: nowrap;
text-decoration: none;
font-family: inherit;
cursor: pointer;
&:hover {
background: rgba(255, 255, 255, 0.18);
@@ -144,9 +146,11 @@ ul#top-right-menu {
// Paused = steady amber dot (no pulse).
&.paused .live-dot {
background: #e8a33d;
box-shadow: 0 0 0 3px rgba(232, 163, 61, 0.3);
background: #e83d3d;
box-shadow: 0 0 0 3px rgb(232 61 61 / 0.3);
animation: none;
width: 12px;
height: 12px;
}
// Icon-led pills (e.g. the mute toggle) keep the feather icon compact.
@@ -27,6 +27,12 @@
--color-background-gradient-first: #5ad8f7;
--color-background-gradient-second: #2f50af;
--color-background-gradient-third: #9150bf;
// The page's atmosphere gradient, shared so anything that needs to sit ON the
// page background (e.g. the left rail rolling out over content) can repaint
// an identical copy instead of approximating it with a flat colour. The
// referenced gradient stops are re-declared per theme below, so this single
// definition follows light/dark automatically.
--page-background-gradient: linear-gradient(130deg, var(--color-background-gradient-first), var(--color-background-gradient-second) 41.07%, var(--color-background-gradient-third) 84.05%);
--color-background: var(--color-white);
--color-text: var(--color-grey-200);
--color-link: #1b98f8;
@@ -1,3 +1,12 @@
body:has(#browser-steps:target) .edit-form {
max-width: min(var(--browser-steps-max-width, 1280px), 95vw);
width: 95%;
}
body:has(#visualselector:target) .edit-form {
width: 95%;
max-width: min(var(--visualselector-max-width, 1280px), 95vw);
}
#selector-wrapper {
height: 100%;
@@ -93,8 +93,11 @@ $watch-table-mobile-max: 767px;
.watch-table {
tbody {
tr {
padding-bottom: 10px;
padding-top: 10px;
padding-inline: 12px;
padding-top: 12px;
padding-bottom: 12px;
display: grid;
grid-template-columns: $grid-col-checkbox 1fr $grid-col-watch;
grid-template-rows: auto auto auto auto;
@@ -148,7 +151,7 @@ $watch-table-mobile-max: 767px;
grid-row: 4;
display: flex;
align-items: center;
justify-content: flex-start;
justify-content: center;
}
> td.watch-controls {
@@ -112,6 +112,10 @@ body.watch-selection-active #checkbox-operations {
}
.pure-table.watch-table td {
/* override of pure-css */
padding: .7em 0.7em;
}
// Watch-list-specific styling layered on top of the shared .cdio-table base.
.watch-table {
@@ -205,7 +209,7 @@ body.watch-selection-active #checkbox-operations {
}
&.queued {
a.recheck {
.recheck {
display: none !important;
}
@@ -216,7 +220,7 @@ body.watch-selection-active #checkbox-operations {
}
&.paused {
a.pause-toggle {
.pause-toggle {
&.state-on {
display: inline !important;
}
@@ -228,7 +232,7 @@ body.watch-selection-active #checkbox-operations {
}
&.notification_muted {
a.mute-toggle {
.mute-toggle {
&.state-on {
display: inline !important;
}
@@ -279,9 +283,11 @@ body.watch-selection-active #checkbox-operations {
display: inline-flex;
align-items: center;
gap: 6px;
> * {
opacity: 0;
transition: opacity 0.12s ease;
@media (min-width: 767px) {
> * {
opacity: 0.2;
transition: opacity 0.12s ease;
}
}
}
}
@@ -359,6 +365,7 @@ body.watch-selection-active #checkbox-operations {
gap: 4px; /* Space between image and text */
> * {
vertical-align: middle;
height: 1.4rem;
}
}
@@ -42,7 +42,8 @@
@use "parts/sub_tabs";
// Smooth transitions for theme switching
body,
// Disabled - people complained
/*body,
.pure-table,
.pure-table thead,
.pure-table td,
@@ -65,7 +66,7 @@ code,
a,
.watch-controls img {
transition: color 0.4s ease, background-color 0.4s ease, background 0.4s ease, border-color 0.4s ease, box-shadow 0.4s ease;
}
}*/
body {
color: var(--color-text);
@@ -305,7 +306,7 @@ code {
body:after {
content: "";
background: linear-gradient(130deg, var(--color-background-gradient-first), var(--color-background-gradient-second) 41.07%, var(--color-background-gradient-third) 84.05%);
background: var(--page-background-gradient);
}
body:after,
@@ -697,12 +698,12 @@ footer {
}
}
@media only screen and (max-width: 760px),
(min-device-width: 768px) and (max-device-width: $desktop-wide-breakpoint) {
.edit-form {
padding: 0.5em;
margin: 0;
width: 100%;
}
#nav-menu {
File diff suppressed because one or more lines are too long
+44 -9
View File
@@ -49,6 +49,9 @@ dictfilt = lambda x, y: dict([(i, x[i]) for i in x if i in set(y)])
# Is there an existing library to ensure some data store (JSON etc) is in sync with CRUD methods?
# Open a github issue if you know something :)
# https://stackoverflow.com/questions/6190468/how-to-trigger-function-on-value-change
_TAG_UUID_RE = re.compile(r'^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$', re.IGNORECASE)
class ChangeDetectionStore(DatastoreUpdatesMixin, FileSavingDataStore):
__version_check = True
@@ -791,11 +794,31 @@ class ChangeDetectionStore(DatastoreUpdatesMixin, FileSavingDataStore):
return None
if tag and type(tag) == str:
# Then it's probably a string of the actual tag by name, split and add it
for t in tag.split(','):
# for each stripped tag, add tag as UUID
for a_t in t.split(','):
tag_uuid = self.add_tag(a_t)
# A comma separated string of tag *titles*, created when they don't exist yet.
# An existing tag's UUID is accepted here too: the API documented this field as taking
# a UUID for years, and honouring that beats creating a tag *titled* with the UUID.
existing_tag_uuids = self.__data['settings']['application'].get('tags', {})
for tag_name in tag.split(','):
tag_name = tag_name.strip()
if not tag_name:
continue
if _TAG_UUID_RE.match(tag_name):
if tag_name in existing_tag_uuids:
apply_extras['tags'].append(tag_name)
continue
# UUID-shaped but no such tag, and no tag literally titled that either -
# a stale or foreign ID. Skip it rather than leave behind a group named
# after a UUID, which is never what the caller wanted.
if not self.tag_uuid_for_title(tag_name):
logger.warning(f"Tag '{tag_name}' looks like a UUID but no such tag exists, skipping")
continue
tag_uuid = self.add_tag(tag_name)
# add_tag() returns False for a title it won't create - never let that into the list,
# a falsy entry blows up every lookup of watch['tags']
if tag_uuid:
apply_extras['tags'].append(tag_uuid)
# Or if UUIDs given directly
@@ -1069,6 +1092,18 @@ class ChangeDetectionStore(DatastoreUpdatesMixin, FileSavingDataStore):
return ret
def tag_uuid_for_title(self, title):
"""UUID of the tag with this title (case/space insensitive), or None. Creates nothing."""
n = title.strip().lower()
if not n:
return None
for uuid, tag in self.__data['settings']['application'].get('tags', {}).items():
if n == tag.get('title', '').lower().strip():
return uuid
return None
def add_tag(self, title):
# If name exists, return that
n = title.strip().lower()
@@ -1076,10 +1111,10 @@ class ChangeDetectionStore(DatastoreUpdatesMixin, FileSavingDataStore):
if not n:
return False
for uuid, tag in self.__data['settings']['application'].get('tags', {}).items():
if n == tag.get('title', '').lower().strip():
logger.warning(f"Tag '{title}' already exists, skipping creation.")
return uuid
existing_uuid = self.tag_uuid_for_title(title)
if existing_uuid:
logger.warning(f"Tag '{title}' already exists, skipping creation.")
return existing_uuid
# Eventually almost everything todo with a watch will apply as a Tag
# So we use the same model as a Watch
@@ -25,6 +25,10 @@
<td><code>{{ '{{watch_url}}' }}</code></td>
<td>{{ _('The URL being watched.') }}</td>
</tr>
<tr>
<td><code>{{ '{{watch_open_url}}' }}</code></td>
<td>{{ _('The Open Link Override set on the watch, otherwise the same as watch_url. Use this when the watched URL is an API endpoint or RSS feed but you want to link to the real page.') }}</td>
</tr>
<tr>
<td><code>{{ '{{watch_uuid}}' }}</code></td>
<td>{{ _('The UUID of the watch.') }}</td>
+20 -7
View File
@@ -271,13 +271,19 @@
<div class="modal-body">
<div class="language-list">
{% for locale, lang_data in available_languages.items()|sort %}
<a href="{{ url_for('set_language', locale=locale, redirect=request.path) }}" class="language-option" data-locale="{{ locale }}">
<span class="lang-option {{ lang_data.flag }}"></span> <span class="language-name">{{ lang_data.name }}</span>
</a>
<form method="POST" action="{{ url_for('set_language', locale=locale, redirect=request.path) }}">
<input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
<button type="submit" class="language-option" data-locale="{{ locale }}">
<span class="lang-option {{ lang_data.flag }}"></span> <span class="language-name">{{ lang_data.name }}</span>
</button>
</form>
{% endfor %}
</div>
<div>
<a href="{{ url_for('ui.delete_locale_language_session_var_if_it_exists', redirect=request.path) }}" >{{ _('Auto-detect from browser') }}</a>
<form method="POST" action="{{ url_for('ui.delete_locale_language_session_var_if_it_exists', redirect=request.path) }}" style="display: inline;">
<input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
<button type="submit" class="bare-btn">{{ _('Auto-detect from browser') }}</button>
</form>
</div>
<div>
{{ _('Language support is in beta, please help us improve by opening a PR on GitHub with any updates.') }}
@@ -311,11 +317,18 @@
<h2 class="modal-title" id="search-modal-title">{{ _('Search') }}</h2>
</div>
<div class="modal-body">
<form id="search-form" method="GET">
{# Plain GET submit to the watchlist - url_for() carries the reverse-proxy sub-path
(SCRIPT_NAME), so no client-side URL building is needed. #}
<form id="search-form" method="GET" action="{{ url_for('watchlist.index') }}">
<div class="pure-control-group">
<label for="search-modal-input">{% if active_tag_uuid %}{{ _("URL or Title in '%(title)s'", title=active_tag.title) }}{% else %}{{ _('URL or Title') }}{% endif %}</label>
{# Matches watch_passes_search() in blueprint/watchlist/filters.py - title, URL and last error text #}
<label for="search-modal-input">{{ _('URL, title or error text') }}</label>
<input id="search-modal-input" class="m-d" name="q" placeholder="{{ _('Enter search term...') }}" required type="text" value="" autofocus>
<input name="tags" type="hidden" value="{% if active_tag_uuid %}{{active_tag_uuid}}{% endif %}">
{# 'tag' (not 'tags') - that's the arg the watchlist filters on #}
{% if active_tag_uuid %}
<input name="tag" type="hidden" value="{{ active_tag_uuid }}">
<span class="pure-form-message-inline">{{ _("Searching in current group '%(title)s' only", title=active_tag.title) }}</span>
{% endif %}
</div>
</form>
</div>
+12 -3
View File
@@ -5,14 +5,23 @@
{% if current_user.is_authenticated or not has_password %}
{% if not current_diff_url %}
<li class="pure-menu-item" id="menu-pause">
<a class="status-pill {{ 'paused' if all_paused }}" href="{{ url_for('settings.toggle_all_paused') }}" aria-label="{% if all_paused %}{{ _('Resume automatic scheduling') }}{% else %}{{ _('Pause auto-queue scheduling of watches') }}{% endif %}" title="{% if all_paused %}{{ _('Scheduling paused — click to resume') }}{% else %}{{ _('Scheduling active — click to pause all') }}{% endif %}"><span class="live-dot"></span>{% if all_paused %}{{ _('Paused') }}{% else %}{{ _('Running') }}{% endif %}</a>
<form method="POST" action="{{ url_for('settings.toggle_all_paused') }}">
<input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
<button type="submit" class="status-pill {{ 'paused' if all_paused }}" aria-label="{% if all_paused %}{{ _('Resume automatic scheduling') }}{% else %}{{ _('Pause auto-queue scheduling of watches') }}{% endif %}" title="{% if all_paused %}{{ _('Scheduling paused — click to resume') }}{% else %}{{ _('Scheduling active — click to pause all') }}{% endif %}"><span class="live-dot"></span>{% if all_paused %}{{ _('Paused') }}{% else %}{{ _('Running') }}{% endif %}</button>
</form>
</li>
<li class="pure-menu-item " id="menu-mute">
<a class="status-pill {{ 'muted' if all_muted }}" href="{{ url_for('settings.toggle_all_muted') }}" aria-label="{% if all_muted %}{{ _('Unmute notifications') }}{% else %}{{ _('Mute notifications') }}{% endif %}" title="{% if all_muted %}{{ _('Notifications are muted - click to unmute') }}{% else %}{{ _('Mute notifications') }}{% endif %}"><i data-feather="{{ 'bell-off' if all_muted else 'bell' }}" class="action-icon"></i>{% if all_muted %}{{ _('Muted') }}{% else %}{{ _('Alerts on') }}{% endif %}</a>
<form method="POST" action="{{ url_for('settings.toggle_all_muted') }}">
<input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
<button type="submit" class="status-pill {{ 'muted' if all_muted }}" aria-label="{% if all_muted %}{{ _('Unmute notifications') }}{% else %}{{ _('Mute notifications') }}{% endif %}" title="{% if all_muted %}{{ _('Notifications are muted - click to unmute') }}{% else %}{{ _('Mute notifications') }}{% endif %}"><i data-feather="{{ 'bell-off' if all_muted else 'bell' }}" class="action-icon"></i>{% if all_muted %}{{ _('Muted') }}{% else %}{{ _('Alerts on') }}{% endif %}</button>
</form>
</li>
{%- if current_user.is_authenticated -%}
<li class="pure-menu-item menu-collapsible">
<a href="{{ url_for('logout', redirect=request.path) }}" ><i data-feather="log-out" class="action-icon"></i>&nbsp;{{ _('Log out') }}</a>
<form method="POST" action="{{ url_for('logout', redirect=request.path) }}">
<input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
<button type="submit" class="bare-btn"><i data-feather="log-out" class="action-icon"></i>&nbsp;{{ _('Log out') }}</button>
</form>
</li>
{%- endif -%}
+2 -2
View File
@@ -7,12 +7,12 @@
<li class="action-sidebar-li" id="action-sidebar-logo">
{%- if has_password and not current_user.is_authenticated -%}
<a id="cdio-logo" href="https://changedetection.io" rel="noopener">
<a id="cdio-logo" href="https://changedetection.io" rel="noopener" title="ChangeDetection.io intelligent web page change detection.">
<span id="logo-expanded"><strong>Change</strong>Detection.io</span>
<span id="logo-short"><strong>CD</strong>IO</span>
</a>
{%- else -%}
<a id="cdio-logo" href="{{url_for('watchlist.index')}}">
<a id="cdio-logo" href="{{url_for('watchlist.index')}}" title="ChangeDetection.io intelligent web page change detection.">
<span id="logo-expanded"><strong>Change</strong>Detection.io</span>
<span id="logo-short"><strong>CD</strong>IO</span>
</a>
@@ -0,0 +1,60 @@
#!/usr/bin/env python3
"""The browser fetchers must judge a fetch on the document they end up extracting.
Plenty of sites answer the first request with an interstitial carrying an error status and a
client-side redirect, then serve the real page. Judging the fetch on the first navigation fails a
watch whose content is present and fine (reported against fotokoch.de: 503 + meta refresh -> 200),
and on the pyppeteer fetcher a replaced document used to hang goto() until the hard processing
timeout because its navigation watcher is bound to the loaderId it started on.
"""
import os
from flask import url_for
from ..util import wait_for_all_checks
def _cdio(url):
# The browser runs in another container in CI and reaches the test server as 'cdio'
return url.replace('localhost.localdomain', 'cdio').replace('localhost', 'cdio')
def test_interstitial_redirect_is_followed(client, live_server, measure_memory_usage, datastore_path):
assert os.getenv('PLAYWRIGHT_DRIVER_URL'), "Needs PLAYWRIGHT_DRIVER_URL set for this test"
res = client.post(
url_for("settings.settings_page"),
data={
"application-empty_pages_are_a_change": "",
"requests-time_between_check-minutes": 180,
'application-fetch_backend': "html_webdriver",
},
follow_redirects=True
)
assert b"Settings updated." in res.data
test_url = _cdio(url_for('test_interstitial', key='renav', _external=True))
res = client.post(
url_for("imports.import_page"),
data={"urls": test_url},
follow_redirects=True
)
assert b"1 Imported" in res.data
wait_for_all_checks(client)
# The interstitial answered 503, so judging the first navigation would have failed the watch
uuid = next(iter(live_server.app.config['DATASTORE'].data['watching']))
watch = live_server.app.config['DATASTORE'].data['watching'][uuid]
assert not watch.get('last_error'), \
f"Watch was judged on the interstitial instead of the page it landed on: {watch.get('last_error')}"
res = client.get(url_for("watchlist.index"))
assert b'Error - 503' not in res.data
assert watch.history_n >= 1, "Fetch succeeded but no snapshot was stored"
snapshot = watch.get_history_snapshot(list(watch.history.keys())[-1])
assert 'The real page content is here' in snapshot
assert 'Browser check in progress' not in snapshot
client.post(url_for("ui.form_delete", uuid="all"), follow_redirects=True)
@@ -84,10 +84,12 @@ def test_socks5(client, live_server, measure_memory_usage, datastore_path):
# PROXY CHECKER WIDGET CHECK - this needs more checking
uuid = next(iter(live_server.app.config['DATASTORE'].data['watching']))
res = client.get(
# POST only - it kicks off real fetches through every configured proxy
res = client.post(
url_for("check_proxies.start_check", uuid=uuid),
follow_redirects=True
)
assert res.status_code == 200
# It's probably already finished super fast :(
#assert b"RUNNING" in res.data
@@ -125,7 +125,7 @@ def test_check_access_control(app, client, live_server, measure_memory_usage, da
follow_redirects=True
)
res = c.get(url_for("logout"),
res = c.post(url_for("logout"),
follow_redirects=True)
assert b"Login" in res.data
@@ -79,26 +79,41 @@ def test_snapshot_refuses_browser_that_cannot_preview(client, live_server, measu
from changedetectionio.blueprint.add_watch_ui import browser_config
monkeypatch.setattr(browser_config, 'is_visual_capable', lambda name, datastore: False)
snapshot_url = url_for('add_watch_ui.add_watch_ui_snapshot')
# Nothing capable, and no explicit browser asked for -> nothing to preview with
res = client.get(url_for('add_watch_ui.add_watch_ui_snapshot', url='https://example.com'))
res = client.post(snapshot_url, data={'url': 'https://example.com'})
assert res.status_code == 400
assert b'No interactive browser' in res.data
# Explicitly asking for a browser that can't preview is refused just the same
res = client.get(url_for('add_watch_ui.add_watch_ui_snapshot', url='https://example.com',
fetch_backend='html_requests'))
res = client.post(snapshot_url, data={'url': 'https://example.com',
'fetch_backend': 'html_requests'})
assert res.status_code == 400
# A made-up name never resolves to a capable fetcher either (real capability lookup here)
monkeypatch.undo()
res = client.get(url_for('add_watch_ui.add_watch_ui_snapshot', url='https://example.com',
fetch_backend='../../etc/passwd'))
res = client.post(snapshot_url, data={'url': 'https://example.com',
'fetch_backend': '../../etc/passwd'})
assert res.status_code == 400
res = client.get(url_for('add_watch_ui.add_watch_ui_snapshot', url='https://example.com',
fetch_backend='os'))
res = client.post(snapshot_url, data={'url': 'https://example.com',
'fetch_backend': 'os'})
assert res.status_code == 400
def test_snapshot_is_post_only(client, live_server, measure_memory_usage, datastore_path):
"""A GET must not reach the endpoint at all.
/snapshot drives a real server-side browser fetch and hands the rendered result back in
the response (GHSA-56fq-63vj-9992). As a GET that is reachable by anything that can make
the operator's browser issue a request - an <img>/<iframe>/link from another site - with
no CSRF token in play. POST-only + CSRFProtect means only our own page can trigger it.
"""
# Method mismatch surfaces as 404 here rather than 405
res = client.get(url_for('add_watch_ui.add_watch_ui_snapshot') + '?url=https://example.com')
assert res.status_code in (404, 405)
def test_submit_rejects_unknown_fetcher(client, live_server, measure_memory_usage, datastore_path):
"""A posted browser is checked server side, so a doctored form can't pin a junk fetcher."""
datastore = _datastore(client)
+78
View File
@@ -338,6 +338,84 @@ def test_api_simple(client, live_server, measure_memory_usage, datastore_path):
)
assert len(res.json) == 0, "Watch list should be empty"
def test_api_delete_watch_history(client, live_server, measure_memory_usage, datastore_path):
"""DELETE /api/v1/watch/<uuid>/history should wipe the snapshots but keep the watch (#4397)"""
api_key = live_server.app.config['DATASTORE'].data['settings']['application'].get('api_access_token')
set_original_response(datastore_path=datastore_path)
test_url = url_for('test_endpoint', _external=True)
res = client.post(
url_for("createwatch"),
data=json.dumps({"url": test_url}),
headers={'content-type': 'application/json', 'x-api-key': api_key},
follow_redirects=True
)
assert res.status_code == 201
watch_uuid = res.json.get('uuid')
wait_for_all_checks(client)
# A second snapshot so we know we're clearing more than one
set_modified_response(datastore_path=datastore_path)
client.get(url_for("watch", uuid=watch_uuid, recheck='1'), headers={'x-api-key': api_key})
wait_for_all_checks(client)
res = client.get(
url_for("watchhistory", uuid=watch_uuid),
headers={'x-api-key': api_key},
)
assert len(res.json) == 2, "Should have two history entries before clearing"
# Unknown watch UUID should 404 and not blow up
res = client.delete(
url_for("watchhistory", uuid='4d8b5b4a-8e0b-4d4a-9f57-3f2b1c0d9e11'),
headers={'x-api-key': api_key},
)
assert res.status_code == 404
# Requires the API key
res = client.delete(url_for("watchhistory", uuid=watch_uuid))
assert res.status_code == 403
# Pause it first - clearing resets last_checked to 0 which otherwise makes the ticker
# queue an instant recheck, and that would race with the assertions below
client.get(url_for("watch", uuid=watch_uuid, paused='paused'), headers={'x-api-key': api_key})
# Now really clear it
res = client.delete(
url_for("watchhistory", uuid=watch_uuid),
headers={'x-api-key': api_key},
)
assert res.status_code == 204
res = client.get(
url_for("watchhistory", uuid=watch_uuid),
headers={'x-api-key': api_key},
)
assert res.json == {}, "History should be empty after DELETE"
# The watch itself must survive, with its state reset
res = client.get(
url_for("watch", uuid=watch_uuid),
headers={'x-api-key': api_key}
)
assert res.status_code == 200
assert res.json.get('url') == test_url
assert res.json.get('history_n') == 0
assert res.json.get('last_checked') == 0
assert res.json.get('previous_md5') == False
# And a snapshot fetch now has nothing to give
res = client.get(
url_for("watchsinglehistory", uuid=watch_uuid, timestamp='latest'),
headers={'x-api-key': api_key},
)
assert res.status_code == 404
delete_all_watches(client)
def test_roundtrip_API(client, live_server, measure_memory_usage, datastore_path):
"""
Test the full round trip, this way we test the default Model fits back into OpenAPI spec
+77
View File
@@ -321,3 +321,80 @@ def test_roundtrip_API(client, live_server, measure_memory_usage, datastore_path
date_created = res.json.get('date_created')
assert date_created != 454444444444, "ReadOnly date_created should not be updateable"
assert date_created != "454444444444", "ReadOnly date_created should not be updateable"
def test_api_watch_tag_field_accepts_names_and_uuids(client, live_server, measure_memory_usage, datastore_path):
"""The `tag` field on a watch takes tag *names*, `tags` takes UUIDs.
`tag` was documented as taking a UUID for years while the code fed it to add_tag(title),
so a UUID silently created a junk group *titled* with that UUID and never applied the tag
the caller asked for. Both spellings now resolve to the same tag.
"""
api_key = live_server.app.config['DATASTORE'].data['settings']['application'].get('api_access_token')
datastore = live_server.app.config['DATASTORE']
test_url = url_for('test_endpoint', _external=True)
hdr = {'x-api-key': api_key, 'content-type': 'application/json'}
def titles_of(watch_uuid):
tags = datastore.data['settings']['application']['tags']
return sorted(tags[t].get('title') for t in datastore.data['watching'][watch_uuid].get('tags'))
# A name creates the group
res = client.post(url_for("createwatch"), data=json.dumps({"url": test_url, "tag": "helloworld"}), headers=hdr)
assert res.status_code == 201
assert titles_of(res.json['uuid']) == ['helloworld']
# An existing tag's UUID links to that tag rather than making a group named after the UUID
res = client.post(url_for("tag"), data=json.dumps({"title": "RealTag"}), headers=hdr)
assert res.status_code == 201
real_tag_uuid = res.json['uuid']
tag_count_before = len(datastore.data['settings']['application']['tags'])
res = client.post(url_for("createwatch"), data=json.dumps({"url": f"{test_url}?p=2", "tag": real_tag_uuid}), headers=hdr)
assert res.status_code == 201
assert real_tag_uuid in datastore.data['watching'][res.json['uuid']].get('tags')
assert titles_of(res.json['uuid']) == ['RealTag']
assert len(datastore.data['settings']['application']['tags']) == tag_count_before, "No junk tag titled with a UUID"
# `tags` with UUIDs keeps working
res = client.post(url_for("createwatch"), data=json.dumps({"url": f"{test_url}?p=3", "tags": [real_tag_uuid]}), headers=hdr)
assert res.status_code == 201
assert titles_of(res.json['uuid']) == ['RealTag']
# Names and UUIDs can be mixed, and blank entries from a trailing comma are dropped -
# add_tag() returns False for those and a falsy entry breaks every watch['tags'] lookup
res = client.post(url_for("createwatch"),
data=json.dumps({"url": f"{test_url}?p=4", "tag": f"Mixed,,{real_tag_uuid},"}), headers=hdr)
assert res.status_code == 201
assert titles_of(res.json['uuid']) == ['Mixed', 'RealTag']
assert all(datastore.data['watching'][res.json['uuid']].get('tags')), "No falsy entries in tags"
# A UUID that matches no tag is skipped rather than becoming a group named after it
unknown_uuid = '0be0272a-19dc-4c97-8aae-5a68df319489'
tag_count_before = len(datastore.data['settings']['application']['tags'])
res = client.post(url_for("createwatch"),
data=json.dumps({"url": f"{test_url}?p=5", "tag": unknown_uuid}), headers=hdr)
assert res.status_code == 201
assert datastore.data['watching'][res.json['uuid']].get('tags') == []
assert len(datastore.data['settings']['application']['tags']) == tag_count_before
# Names are matched case-insensitively against existing tags, as the spec claims
tag_count_before = len(datastore.data['settings']['application']['tags'])
res = client.post(url_for("createwatch"),
data=json.dumps({"url": f"{test_url}?p=6", "tag": "rEaLtAg"}), headers=hdr)
assert res.status_code == 201
assert datastore.data['watching'][res.json['uuid']].get('tags') == [real_tag_uuid]
assert len(datastore.data['settings']['application']['tags']) == tag_count_before, "Casing must not fork a second tag"
# `tags` is applied verbatim and never creates: an unknown UUID is stored as a dangling
# reference that simply resolves to no group. Documented, and harmless because the lookup
# is a dictfilt() over known tags - pinned here so changing it has to be deliberate.
bogus = 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee'
tag_count_before = len(datastore.data['settings']['application']['tags'])
res = client.post(url_for("createwatch"),
data=json.dumps({"url": f"{test_url}?p=7", "tags": [bogus]}), headers=hdr)
assert res.status_code == 201
assert datastore.data['watching'][res.json['uuid']].get('tags') == [bogus]
assert len(datastore.data['settings']['application']['tags']) == tag_count_before
assert datastore.get_all_tags_for_watch(res.json['uuid']) == {}
assert client.get(url_for("watchlist.index")).status_code == 200, "A dangling tag ref must not break the list"
@@ -96,7 +96,7 @@ def test_check_ldjson_price_autodetect(client, live_server, measure_memory_usage
assert b'ldjson-price-track-offer' in res.data
# Accept it
client.get(url_for('price_data_follower.accept', uuid=uuid, follow_redirects=True))
client.post(url_for('price_data_follower.accept', uuid=uuid), follow_redirects=True)
client.post(url_for("ui.form_watch_checknow"), follow_redirects=True)
wait_for_all_checks(client)
# Offer should be gone
+2 -2
View File
@@ -24,7 +24,7 @@ def test_backup(client, live_server, measure_memory_usage, datastore_path):
wait_for_all_checks(client)
# Launch the thread in the background to create the backup
res = client.get(
res = client.post(
url_for("backups.request_backup"),
follow_redirects=True
)
@@ -136,7 +136,7 @@ def test_backup_restore(client, live_server, measure_memory_usage, datastore_pat
wait_for_all_checks(client)
# Create a full backup
client.get(url_for("backups.request_backup"), follow_redirects=True)
client.post(url_for("backups.request_backup"), follow_redirects=True)
time.sleep(4)
# Download the latest backup zip
@@ -493,7 +493,7 @@ def test_tag_mute_persists(client, live_server):
tag_uuid = datastore.add_tag('Test Tag')
# Mute the tag
response = client.get(url_for("tags.mute", uuid=tag_uuid))
response = client.post(url_for("tags.mute", uuid=tag_uuid))
assert response.status_code == 302 # Redirect
# Verify muted in memory
+21 -21
View File
@@ -11,7 +11,7 @@ def test_zh_TW(client, live_server, measure_memory_usage, datastore_path):
# Be sure we got a session cookie
res = client.get(url_for("watchlist.index"), follow_redirects=True)
res = client.get(
res = client.post(
url_for("set_language", locale="zh_Hant_TW"), # Traditional
follow_redirects=True
)
@@ -21,7 +21,7 @@ def test_zh_TW(client, live_server, measure_memory_usage, datastore_path):
assert '選擇語言'.encode() in res.data
# Check second set works
res = client.get(
res = client.post(
url_for("set_language", locale="en_GB"),
follow_redirects=True
)
@@ -30,7 +30,7 @@ def test_zh_TW(client, live_server, measure_memory_usage, datastore_path):
assert b"Select Language" in res.data, "Second set of language worked"
# Check arbitration between zh_Hant_TW<->zh
res = client.get(
res = client.post(
url_for("set_language", locale="zh"), # Simplified chinese
follow_redirects=True
)
@@ -89,7 +89,7 @@ def test_language_switching(client, live_server, measure_memory_usage, datastore
client.get(url_for("add_watch_ui.add_watch_ui_index"), follow_redirects=True)
# Step 1: Set the language to Italian using the /set-language endpoint
res = client.get(
res = client.post(
url_for("set_language", locale="it"),
follow_redirects=True
)
@@ -119,7 +119,7 @@ def test_language_switching(client, live_server, measure_memory_usage, datastore
# NB: use 'en_GB' not 'en' — only the variants are in language_codes; the
# plain 'en' code is silently rejected by set_language and the locale would
# remain at 'it', defeating the round-trip assertion below.
res = client.get(
res = client.post(
url_for("set_language", locale="en_GB"),
follow_redirects=True
)
@@ -152,7 +152,7 @@ def test_invalid_locale(client, live_server, measure_memory_usage, datastore_pat
# bare 'en' is NOT in language_codes and is silently rejected by
# set_language, so passing it here would leave the session locale unset
# and let the (unrelated) Accept-Language fallback decide what renders.
res = client.get(
res = client.post(
url_for("set_language", locale="en_GB"),
follow_redirects=True
)
@@ -160,7 +160,7 @@ def test_invalid_locale(client, live_server, measure_memory_usage, datastore_pat
assert res.status_code == 200
# Try to set an invalid locale
res = client.get(
res = client.post(
url_for("set_language", locale="invalid_locale_xyz"),
follow_redirects=True
)
@@ -190,7 +190,7 @@ def test_language_persistence_in_session(client, live_server, measure_memory_usa
client.get(url_for("watchlist.index"), follow_redirects=True)
# Set language to Italian
res = client.get(
res = client.post(
url_for("set_language", locale="it"),
follow_redirects=True
)
@@ -215,7 +215,7 @@ def test_language_persistence_in_session(client, live_server, measure_memory_usa
assert sess.get('locale') == 'it', "Locale should be set in session"
# Call auto-detect to clear the locale
res = client.get(
res = client.post(
url_for("ui.delete_locale_language_session_var_if_it_exists"),
follow_redirects=True
)
@@ -254,7 +254,7 @@ def test_set_language_with_redirect(client, live_server, measure_memory_usage, d
client.get(url_for("watchlist.index"), follow_redirects=True)
# Set language with a redirect parameter (simulating language change from /settings)
res = client.get(
res = client.post(
url_for("set_language", locale="de", redirect="/settings"),
follow_redirects=False
)
@@ -268,7 +268,7 @@ def test_set_language_with_redirect(client, live_server, measure_memory_usage, d
assert sess.get('locale') == 'de'
# Test with invalid locale (should still redirect safely)
res = client.get(
res = client.post(
url_for("set_language", locale="invalid_locale", redirect="/settings"),
follow_redirects=False
)
@@ -276,7 +276,7 @@ def test_set_language_with_redirect(client, live_server, measure_memory_usage, d
assert '/settings' in res.location
# Test with malicious redirect (should default to watchlist)
res = client.get(
res = client.post(
url_for("set_language", locale="en", redirect="https://evil.com"),
follow_redirects=False
)
@@ -296,7 +296,7 @@ def test_time_unit_translations(client, live_server, measure_memory_usage, datas
client.get(url_for("watchlist.index"), follow_redirects=True)
# Test Italian translations
res = client.get(url_for("set_language", locale="it"), follow_redirects=True)
res = client.post(url_for("set_language", locale="it"), follow_redirects=True)
assert res.status_code == 200
res = client.get(url_for("settings.settings_page"), follow_redirects=True)
@@ -312,7 +312,7 @@ def test_time_unit_translations(client, live_server, measure_memory_usage, datas
assert b"Time Between Check" not in res.data, "Should not have English 'Time Between Check'"
# Test Korean translations
res = client.get(url_for("set_language", locale="ko"), follow_redirects=True)
res = client.post(url_for("set_language", locale="ko"), follow_redirects=True)
assert res.status_code == 200
res = client.get(url_for("settings.settings_page"), follow_redirects=True)
@@ -332,7 +332,7 @@ def test_time_unit_translations(client, live_server, measure_memory_usage, datas
assert b"Time Between Check" not in res.data, "Should not have English 'Time Between Check'"
# Test Chinese Simplified translations
res = client.get(url_for("set_language", locale="zh"), follow_redirects=True)
res = client.post(url_for("set_language", locale="zh"), follow_redirects=True)
assert res.status_code == 200
res = client.get(url_for("settings.settings_page"), follow_redirects=True)
@@ -348,7 +348,7 @@ def test_time_unit_translations(client, live_server, measure_memory_usage, datas
assert b"Time Between Check" not in res.data, "Should not have English 'Time Between Check'"
# Test German translations
res = client.get(url_for("set_language", locale="de"), follow_redirects=True)
res = client.post(url_for("set_language", locale="de"), follow_redirects=True)
assert res.status_code == 200
res = client.get(url_for("settings.settings_page"), follow_redirects=True)
@@ -363,7 +363,7 @@ def test_time_unit_translations(client, live_server, measure_memory_usage, datas
assert b"Time Between Check" not in res.data, "Should not have English 'Time Between Check'"
# Test Russian translations
res = client.get(url_for("set_language", locale="ru"), follow_redirects=True)
res = client.post(url_for("set_language", locale="ru"), follow_redirects=True)
assert res.status_code == 200
res = client.get(url_for("settings.settings_page"), follow_redirects=True)
@@ -378,7 +378,7 @@ def test_time_unit_translations(client, live_server, measure_memory_usage, datas
assert b"Time Between Check" not in res.data, "Should not have English 'Time Between Check'"
# Test Traditional Chinese (zh_Hant_TW) translations
res = client.get(url_for("set_language", locale="zh_Hant_TW"), follow_redirects=True)
res = client.post(url_for("set_language", locale="zh_Hant_TW"), follow_redirects=True)
assert res.status_code == 200
res = client.get(url_for("settings.settings_page"), follow_redirects=True)
@@ -627,7 +627,7 @@ def test_session_locale_overrides_accept_language(client, live_server, measure_m
"Expected Taiwan flag 'fi fi-tw' from auto-detect"
# Step 2: User explicitly selects Korean language
res = client.get(
res = client.post(
url_for("set_language", locale="ko"),
headers={'Accept-Language': 'zh-TW,zh;q=0.9,en;q=0.8'}, # Browser still sends zh-TW
follow_redirects=True
@@ -700,7 +700,7 @@ def test_clear_history_translated_confirmation(client, live_server, measure_memo
wait_for_all_checks(client)
# Set language to German
res = client.get(
res = client.post(
url_for("set_language", locale="de"),
follow_redirects=True
)
@@ -726,7 +726,7 @@ def test_clear_history_translated_confirmation(client, live_server, measure_memo
"German confirmation word 'loschen' should be accepted (issue #3865)"
# Switch back to English and verify English word still works
res = client.get(
res = client.post(
url_for("set_language", locale="en_US"),
follow_redirects=True
)
@@ -25,7 +25,7 @@ def test_language_endpoints_work_for_anonymous_users(client, live_server, measur
follow_redirects=True)
assert res.status_code == 200
client.get(url_for("logout"), follow_redirects=True)
client.post(url_for("logout"), follow_redirects=True)
# Both language links are rendered on the login page, so both must be reachable
res = client.get(url_for("login"))
@@ -33,13 +33,13 @@ def test_language_endpoints_work_for_anonymous_users(client, live_server, measur
assert b'language-selector' in res.data, "Language modal trigger should render for anonymous users"
# Picking a specific language must not redirect to the login page
res = client.get(url_for("set_language", locale="de"), follow_redirects=False)
res = client.post(url_for("set_language", locale="de"), follow_redirects=False)
assert res.status_code == 302
assert '/login' not in res.headers.get("Location", ""), \
"set_language must not bounce anonymous users to /login"
# ...and neither must clearing it back to auto-detect
res = client.get(url_for("ui.delete_locale_language_session_var_if_it_exists"), follow_redirects=False)
res = client.post(url_for("ui.delete_locale_language_session_var_if_it_exists"), follow_redirects=False)
assert res.status_code == 302
assert '/login' not in res.headers.get("Location", ""), \
"Auto-detect must not bounce anonymous users to /login (it renders on the login page)"
@@ -0,0 +1,171 @@
"""
Tests for the optional per-watch "Link to Open" (`link_to_open`).
A watch may point at something that is useless in a browser - an API endpoint or an RSS
feed - while the human-readable page lives somewhere else. `link_to_open` stores that page,
and every "go to the site" affordance (watch list, history/preview header, the
{{watch_open_url}} notification token) uses it in preference to the watched URL.
"""
import os
from flask import url_for
from .util import set_original_response, set_modified_response, wait_for_all_checks, \
wait_for_notification_endpoint_output
OPEN_URL = "https://example.com/the-real-human-page"
def _add_watch(client, test_url):
res = client.post(
url_for("ui.ui_views.form_quick_watch_add"),
data={"url": test_url, "tags": ''},
follow_redirects=True
)
assert b"Watch added" in res.data
wait_for_all_checks(client)
def _edit_watch(client, test_url, **extra):
data = {
"url": test_url,
"tags": "",
"headers": "",
"fetch_backend": "html_requests",
"time_between_check_use_default": "y",
}
data.update(extra)
return client.post(url_for("ui.ui_edit.edit_page", uuid="first"), data=data, follow_redirects=True)
def test_link_to_open_is_saved_and_used_in_the_watch_list(client, live_server, measure_memory_usage, datastore_path):
set_original_response(datastore_path=datastore_path)
test_url = url_for('test_endpoint', _external=True)
_add_watch(client, test_url)
# Without an override, the watch list links to the watched URL
res = client.get(url_for("watchlist.index"))
assert f'href="{test_url}"'.encode('utf-8') in res.data
assert OPEN_URL.encode('utf-8') not in res.data
res = _edit_watch(client, test_url, link_to_open=OPEN_URL)
assert b"Updated watch." in res.data
# It round-trips back into the edit form
res = client.get(url_for("ui.ui_edit.edit_page", uuid="first"))
assert OPEN_URL.encode('utf-8') in res.data
# ...and the watch list now points at it instead of the watched URL
res = client.get(url_for("watchlist.index"))
assert f'href="{OPEN_URL}"'.encode('utf-8') in res.data
assert f'href="{test_url}"'.encode('utf-8') not in res.data
def test_link_to_open_used_in_history_header(client, live_server, measure_memory_usage, datastore_path):
set_original_response(datastore_path=datastore_path)
test_url = url_for('test_endpoint', _external=True)
_add_watch(client, test_url)
set_modified_response(datastore_path=datastore_path)
client.post(url_for("ui.form_watch_checknow"), follow_redirects=True)
wait_for_all_checks(client)
assert b"Updated watch." in _edit_watch(client, test_url, link_to_open=OPEN_URL).data
uuid = next(iter(client.application.config.get('DATASTORE').data['watching'].keys()))
# Top-of-page link on the history/diff page
res = client.get(url_for("ui.ui_diff.diff_history_page", uuid=uuid))
assert f'class="current-diff-url" href="{OPEN_URL}"'.encode('utf-8') in res.data
# ...and on the preview page
res = client.get(url_for("ui.ui_preview.preview_page", uuid=uuid))
assert f'class="current-diff-url" href="{OPEN_URL}"'.encode('utf-8') in res.data
def test_invalid_link_to_open_is_rejected(client, live_server, measure_memory_usage, datastore_path):
set_original_response(datastore_path=datastore_path)
test_url = url_for('test_endpoint', _external=True)
_add_watch(client, test_url)
res = _edit_watch(client, test_url, link_to_open="javascript:alert(1)")
assert b"Updated watch." not in res.data
assert b"Watch protocol is not permitted or invalid URL format" in res.data
# Blank is fine - it means "use the watched URL"
res = _edit_watch(client, test_url, link_to_open="")
assert b"Updated watch." in res.data
watch = list(client.application.config.get('DATASTORE').data['watching'].values())[0]
assert watch.open_link == test_url
def test_watch_open_url_notification_token(client, live_server, measure_memory_usage, datastore_path):
set_original_response(datastore_path=datastore_path)
notification_file = os.path.join(datastore_path, "notification.txt")
if os.path.isfile(notification_file):
os.unlink(notification_file)
test_url = url_for('test_endpoint', _external=True)
_add_watch(client, test_url)
notification_url = url_for('test_notification_endpoint', _external=True).replace('http://', 'post://')
res = _edit_watch(
client,
test_url,
link_to_open=OPEN_URL,
notification_urls=notification_url,
notification_title="Test",
notification_body="watched={{watch_url}}\nopen={{watch_open_url}}",
notification_format="text",
)
assert b"Updated watch." in res.data
wait_for_all_checks(client)
set_modified_response(datastore_path=datastore_path)
client.post(url_for("ui.form_watch_checknow"), follow_redirects=True)
wait_for_all_checks(client)
assert wait_for_notification_endpoint_output(datastore_path=datastore_path)
with open(notification_file, 'r') as f:
body = f.read()
assert f"watched={test_url}" in body
assert f"open={OPEN_URL}" in body
os.unlink(notification_file)
def test_watch_open_url_token_falls_back_to_watch_url(client, live_server, measure_memory_usage, datastore_path):
"""With no 'Link to Open' set, {{watch_open_url}} must still render the watched URL."""
set_original_response(datastore_path=datastore_path)
notification_file = os.path.join(datastore_path, "notification.txt")
if os.path.isfile(notification_file):
os.unlink(notification_file)
test_url = url_for('test_endpoint', _external=True)
_add_watch(client, test_url)
notification_url = url_for('test_notification_endpoint', _external=True).replace('http://', 'post://')
res = _edit_watch(
client,
test_url,
notification_urls=notification_url,
notification_title="Test",
notification_body="open={{watch_open_url}}",
notification_format="text",
)
assert b"Updated watch." in res.data
wait_for_all_checks(client)
set_modified_response(datastore_path=datastore_path)
client.post(url_for("ui.form_watch_checknow"), follow_redirects=True)
wait_for_all_checks(client)
assert wait_for_notification_endpoint_output(datastore_path=datastore_path)
with open(notification_file, 'r') as f:
body = f.read()
assert f"open={test_url}" in body
os.unlink(notification_file)
@@ -393,12 +393,12 @@ def test_llm_models_endpoint_blocks_private_api_base(
def test_llm_test_endpoint_blocks_private_api_base(
client, live_server, measure_memory_usage, datastore_path, monkeypatch):
"""GET /settings/llm/test must refuse api_base pointing at private/loopback
"""POST /settings/llm/test must refuse api_base pointing at private/loopback
hosts and must never reach litellm.completion()."""
monkeypatch.delenv('ALLOW_IANA_RESTRICTED_ADDRESSES', raising=False)
for bad in _SSRF_PRIVATE_HOSTS:
res = client.get(
res = client.post(
url_for('settings.llm.llm_test'),
query_string={'model': 'openai/gpt-4', 'api_base': bad},
)
@@ -530,7 +530,7 @@ def test_llm_test_refuses_to_leak_stored_key_to_different_api_base(
monkeypatch.setattr(llm_client, 'completion',
lambda **kw: calls.append(kw) or ('', 0, 0, 0))
res = client.get(
res = client.post(
url_for('settings.llm.llm_test'),
query_string={
'model': 'gpt-4o-mini',
@@ -38,7 +38,7 @@ def test_rss_tag_feed_ignores_security_token(client, live_server, datastore_path
wait_for_all_checks(client)
# Logout
client.get(url_for("logout"), follow_redirects=True)
client.post(url_for("logout"), follow_redirects=True)
# Request the tag RSS feed WITH the token
res = client.get(
+41
View File
@@ -1,5 +1,6 @@
from flask import url_for
from .util import set_original_response, set_modified_response, live_server_setup
import re
import time
@@ -71,3 +72,43 @@ def test_search_in_tag_limit(client, live_server, measure_memory_usage, datastor
assert urls[0].split(' ')[0].encode('utf-8') in res.data, urls[0].encode('utf-8')
assert urls[1].split(' ')[0].encode('utf-8') not in res.data, urls[0].encode('utf-8')
def test_search_modal_form_action(client, live_server, measure_memory_usage, datastore_path):
# The search modal submits as a plain GET form, so its action has to carry the
# reverse-proxy sub-path (SCRIPT_NAME), otherwise search jumps to the host root.
res = client.get(url_for("watchlist.index"))
assert b'<form id="search-form" method="GET" action="/">' in res.data
res = client.get("/", base_url="http://localhost/sub-path")
assert b'<form id="search-form" method="GET" action="/sub-path/">' in res.data
def test_search_modal_tag_field_is_filterable(client, live_server, measure_memory_usage, datastore_path):
# The modal carries the active tag as a hidden field so a search stays scoped to the
# tag you were viewing. The field name has to be the one the watchlist filters on.
urls = ['https://localhost:12300?first-result=1 tag-one',
'https://localhost:5000?second-result=1 tag-two'
]
res = client.post(
url_for("imports.import_page"),
data={"urls": "\r\n".join(urls)},
follow_redirects=True
)
assert b"2 Imported" in res.data
res = client.get(url_for("watchlist.index") + "?tag=tag-one")
form = re.search(rb'<form id="search-form".*?</form>', res.data, re.DOTALL)
assert form, "search modal form not rendered"
field = re.search(rb'<input name="([^"]+)" type="hidden" value="([^"]+)"', form.group(0))
assert field, f"no populated hidden tag field in {form.group(0)}"
name, value = field.group(1).decode(), field.group(2).decode()
# The scoping is spelled out in the modal, so narrowed results aren't a surprise
assert b'Searching in current group' in form.group(0)
assert b'tag-one' in form.group(0)
# 'localhost' matches both watches, so only the tag field can narrow it down
res = client.get(url_for("watchlist.index") + f"?q=localhost&{name}={value}")
assert urls[0].split(' ')[0].encode('utf-8') in res.data
assert urls[1].split(' ')[0].encode('utf-8') not in res.data, f"'{name}' is not filtered on by the watchlist"
+5 -5
View File
@@ -440,7 +440,7 @@ def test_login_redirect_with_password(client, live_server, measure_memory_usage,
assert b"evil.com" not in res.data
# Logout for cleanup
client.get(url_for("logout"))
client.post(url_for("logout"))
# Test 5: Incorrect password with redirect should stay on login page
res = client.post(
@@ -483,7 +483,7 @@ def test_login_redirect_from_protected_page(client, live_server, measure_memory_
client.application.config['DATASTORE'].data['settings']['application']['password'] = salted_pass
# Logout to ensure we're not authenticated
client.get(url_for("logout"))
client.post(url_for("logout"))
# Try to access a protected page (edit page for first watch)
res = client.get(
@@ -524,7 +524,7 @@ def test_login_redirect_from_protected_page(client, live_server, measure_memory_
assert b'Edit' in res.data or b'Watching' in res.data
# Cleanup
client.get(url_for("logout"))
client.post(url_for("logout"))
del client.application.config['DATASTORE'].data['settings']['application']['password']
@@ -554,7 +554,7 @@ def test_logout_with_redirect(client, live_server, measure_memory_usage, datasto
assert res.status_code == 200
# Now logout with a redirect parameter (simulating logout from /settings)
res = client.get(
res = client.post(
url_for("logout", redirect="/settings"),
follow_redirects=False
)
@@ -961,7 +961,7 @@ def test_ghsa_8757_69j2_hx56_backup_restore_history_path_traversal(client, live_
wait_for_all_checks(client)
# Download a legitimate backup to use as a template
client.get(url_for("backups.request_backup"), follow_redirects=True)
client.post(url_for("backups.request_backup"), follow_redirects=True)
time.sleep(4)
res = client.get(url_for("backups.download_backup", filename="latest"), follow_redirects=True)
assert res.content_type == "application/zip"
+1 -1
View File
@@ -33,7 +33,7 @@ def test_share_watch(client, live_server, measure_memory_usage, datastore_path):
assert bytes(include_filters.encode('utf-8')) in res.data
# click share the link
res = client.get(
res = client.post(
url_for("ui.form_share_put_watch", uuid=uuid),
follow_redirects=True
)
@@ -0,0 +1,181 @@
"""Guards for the translation overlay layer (changedetectionio/translations_overlay).
The overlay is a second gettext tree merged on top of ``changedetectionio/translations``, letting a
deployment reword individual strings without editing the ``_()`` call site. See that directory's
README.md. Three separate things can break it, so there is a test for each.
1. Overlay entries key on the exact upstream msgid. When a string is reworded upstream the override
stops matching and silently reverts to upstream wording - no error, no log entry.
``test_overlay_catalogs_are_valid`` makes that a build failure.
2. The layering relies on Flask-Babel merging catalogs with ``dict.update`` semantics (later
directory wins per-message). Were a Flask-Babel upgrade to change that to an ``add_fallback``
chain, overrides would stop applying while everything still looked fine.
``test_overlay_overrides_a_string_in_a_rendered_page`` pins it against a real rendered page.
3. The directory has to actually reach ``BABEL_TRANSLATION_DIRECTORIES``.
``test_overlay_dir_*`` cover the wiring in flask_app.py, including the env var.
"""
import os
import subprocess
import sys
import pytest
from babel.messages.catalog import Catalog
from babel.messages.mofile import write_mo
from flask import url_for
PKG_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
REPO_ROOT = os.path.dirname(PKG_DIR)
OVERLAY_DIR = os.path.join(PKG_DIR, 'translations_overlay')
BASE_DIR = os.path.join(PKG_DIR, 'translations')
MANAGE = os.path.join(OVERLAY_DIR, 'manage.py')
# A msgid that renders as a settings-page tab label. The test asserts it is present *before*
# overriding it, so a rename upstream fails loudly rather than making the test silently vacuous.
OVERRIDDEN_MSGID = 'Global Filters'
SENTINEL = 'zzOverlaySentinelFiltersZZ'
def _write_mo(root, locale, entries):
"""Write a compiled catalog at <root>/<locale>/LC_MESSAGES/messages.mo."""
catalog = Catalog(locale=locale, domain='messages')
for msgid, msgstr in entries.items():
catalog.add(msgid, msgstr)
mo_dir = os.path.join(root, locale, 'LC_MESSAGES')
os.makedirs(mo_dir, exist_ok=True)
with open(os.path.join(mo_dir, 'messages.mo'), 'wb') as fp:
write_mo(fp, catalog)
def _import_app_with(env_overlay_dir):
"""Import flask_app in a clean subprocess and report its BABEL_TRANSLATION_DIRECTORIES.
Has to be a subprocess: the config is built at module import time, so it cannot be re-evaluated
under a different environment once flask_app is already in sys.modules.
"""
env = dict(os.environ, TRANSLATION_OVERLAY_DIR=env_overlay_dir)
result = subprocess.run(
[
sys.executable,
'-c',
'from changedetectionio import flask_app;'
'print("DIRS=" + flask_app.app.config["BABEL_TRANSLATION_DIRECTORIES"])',
],
capture_output=True,
text=True,
env=env,
cwd=REPO_ROOT,
)
assert result.returncode == 0, f"importing flask_app failed:\n{result.stdout}\n{result.stderr}"
line = [l for l in result.stdout.splitlines() if l.startswith('DIRS=')]
assert line, f"no config line in output:\n{result.stdout}\n{result.stderr}"
return line[0][len('DIRS=') :].split(';')
# ---------------------------------------------------------------------------
# 1. The overlay catalogs shipped in this repo are internally consistent
# ---------------------------------------------------------------------------
@pytest.mark.skipif(
not os.path.isdir(OVERLAY_DIR), reason='no translation overlay in this deployment'
)
def test_overlay_catalogs_are_valid():
"""Every override must still match an upstream msgid, be non-empty, and be compiled.
A failure here usually means upstream edited a string the overlay overrides. Re-copy the new
msgid verbatim from translations/messages.pot into the overlay catalog, then recompile with
`python changedetectionio/translations_overlay/manage.py compile`.
"""
result = subprocess.run([sys.executable, MANAGE, 'check'], capture_output=True, text=True)
assert result.returncode == 0, (
f"translation overlay is invalid:\n{result.stdout}{result.stderr}"
)
@pytest.mark.skipif(
not os.path.isdir(OVERLAY_DIR), reason='no translation overlay in this deployment'
)
def test_overlay_locales_have_a_base_catalog():
"""An overlay for a locale the app does not ship never loads, so it is silently dead."""
for locale in sorted(os.listdir(OVERLAY_DIR)):
if not os.path.isfile(os.path.join(OVERLAY_DIR, locale, 'LC_MESSAGES', 'messages.po')):
continue
assert os.path.isdir(os.path.join(BASE_DIR, locale)), (
f"overlay locale {locale!r} has no base catalog in translations/"
)
# ---------------------------------------------------------------------------
# 2. The merge actually happens, end to end, on a real page
# ---------------------------------------------------------------------------
def test_overlay_overrides_a_string_in_a_rendered_page(client, live_server, tmp_path):
"""A real overlay catalog changes real rendered output, and only the string it names."""
app = client.application
# The rest of this test injects its own directory, which would still pass if flask_app.py had
# stopped configuring the real one. Tie the two together so that regression fails here too.
if os.path.isdir(OVERLAY_DIR):
configured = app.config['BABEL_TRANSLATION_DIRECTORIES'].split(';')
assert OVERLAY_DIR in configured, (
f"{OVERLAY_DIR} exists but is not in BABEL_TRANSLATION_DIRECTORIES ({configured})"
)
baseline = client.get(url_for('settings.settings_page'))
assert baseline.status_code == 200
assert OVERRIDDEN_MSGID.encode() in baseline.data, (
f"{OVERRIDDEN_MSGID!r} no longer renders on the settings page - this test needs a new msgid"
)
assert SENTINEL.encode() not in baseline.data
overlay = tmp_path / 'overlay'
# en_GB is BABEL_DEFAULT_LOCALE, and the test client sends no Accept-Language header
_write_mo(str(overlay), 'en_GB', {OVERRIDDEN_MSGID: SENTINEL})
# The default Domain delegates to the app-level directory list, and caches per (locale, domain),
# so both have to be touched for a new catalog to be picked up mid-process.
dirs = app.extensions['babel'].translation_directories
domain_cache = app.extensions['babel'].instance.domain_instance.cache
dirs.append(str(overlay))
domain_cache.clear()
try:
overridden = client.get(url_for('settings.settings_page'))
assert overridden.status_code == 200
assert SENTINEL.encode() in overridden.data, (
'overlay catalog did not override the base catalog'
)
assert OVERRIDDEN_MSGID.encode() not in overridden.data, 'base wording is still rendering'
# Neighbouring tab label, deliberately not in the overlay - merging must not drop it
assert b'UI Options' in overridden.data, (
'overlay replaced the catalog instead of merging into it'
)
finally:
dirs.remove(str(overlay))
domain_cache.clear()
restored = client.get(url_for('settings.settings_page'))
assert SENTINEL.encode() not in restored.data
assert OVERRIDDEN_MSGID.encode() in restored.data, 'base wording did not come back'
# ---------------------------------------------------------------------------
# 3. flask_app.py wires the directory up, and stays a no-op when there isn't one
# ---------------------------------------------------------------------------
def test_overlay_dir_from_env_var_is_used(tmp_path):
overlay = tmp_path / 'my-overlay'
overlay.mkdir()
dirs = _import_app_with(str(overlay))
assert dirs[-1] == str(overlay), f"TRANSLATION_OVERLAY_DIR not appended, got {dirs}"
assert dirs[0] == BASE_DIR, 'base catalog must stay first so the overlay wins on conflicts'
def test_missing_overlay_dir_is_a_noop(tmp_path):
"""No overlay directory means the config is exactly what it was before the feature existed."""
dirs = _import_app_with(str(tmp_path / 'does-not-exist'))
assert dirs == [BASE_DIR], f"expected only the base catalog, got {dirs}"
+222
View File
@@ -0,0 +1,222 @@
#!/usr/bin/env python3
"""PAGE_WATCH_LIMIT - the optional cap on how many watches one instance will hold.
The limit is enforced in datastore.add_watch(), which every add path funnels through, but each
surface has to report it in its own terms: a flash for the UI, a 429 for the API, one flash for
a whole file in the importers, and nothing at all (just a None) where there's no request context.
"""
import json
from flask import url_for
from .util import delete_all_watches
def test_watch_limit_absent_or_junk_means_unlimited(client, live_server, measure_memory_usage, datastore_path, monkeypatch):
"""No env var, an empty one, or an unparseable one all leave the limit switched off."""
datastore = live_server.app.config['DATASTORE']
monkeypatch.delenv('PAGE_WATCH_LIMIT', raising=False)
assert datastore.watch_limit is None
assert datastore.watch_limit_reached() is False
# Junk must not block every add, and must not raise
monkeypatch.setenv('PAGE_WATCH_LIMIT', 'not-a-number')
assert datastore.watch_limit is None
assert datastore.watch_limit_reached() is False
# Set-but-empty is the same as unset
monkeypatch.setenv('PAGE_WATCH_LIMIT', '')
assert datastore.watch_limit is None
assert datastore.watch_limit_reached() is False
def test_api_create_watch_refused_at_limit(client, live_server, measure_memory_usage, datastore_path, monkeypatch):
api_key = live_server.app.config['DATASTORE'].data['settings']['application'].get('api_access_token')
datastore = live_server.app.config['DATASTORE']
test_url = url_for('test_endpoint', _external=True)
monkeypatch.setenv('PAGE_WATCH_LIMIT', '1')
res = client.post(
url_for("createwatch"),
data=json.dumps({"url": test_url}),
headers={'content-type': 'application/json', 'x-api-key': api_key},
)
assert res.status_code == 201
res = client.post(
url_for("createwatch"),
data=json.dumps({"url": f"{test_url}?second=1"}),
headers={'content-type': 'application/json', 'x-api-key': api_key},
)
assert res.status_code == 429
assert b'Watch limit reached (1/1 watches)' in res.data
assert len(datastore.data['watching']) == 1
delete_all_watches(client)
def test_api_import_refuses_the_whole_batch(client, live_server, measure_memory_usage, datastore_path, monkeypatch):
"""A 429 from import always means nothing was created, so the same request can be retried."""
api_key = live_server.app.config['DATASTORE'].data['settings']['application'].get('api_access_token')
datastore = live_server.app.config['DATASTORE']
test_url = url_for('test_endpoint', _external=True)
headers = {'x-api-key': api_key, 'content-type': 'text/plain'}
monkeypatch.setenv('PAGE_WATCH_LIMIT', '3')
res = client.post(url_for("import"), data=f"{test_url}?a=1\n{test_url}?a=2", headers=headers)
assert res.status_code == 200
assert len(res.json) == 2
# Two more would make four - refused whole rather than importing the one that fits
res = client.post(url_for("import"), data=f"{test_url}?a=3\n{test_url}?a=4", headers=headers)
assert res.status_code == 429
assert b'would exceed it' in res.data
assert len(datastore.data['watching']) == 2
# The one that does fit still goes in
res = client.post(url_for("import"), data=f"{test_url}?a=3", headers=headers)
assert res.status_code == 200
assert len(datastore.data['watching']) == 3
delete_all_watches(client)
def test_quick_watch_add_refused_at_limit(client, live_server, measure_memory_usage, datastore_path, monkeypatch):
datastore = live_server.app.config['DATASTORE']
test_url = url_for('test_endpoint', _external=True)
monkeypatch.setenv('PAGE_WATCH_LIMIT', '1')
res = client.post(
url_for("ui.ui_views.form_quick_watch_add"),
data={"url": test_url, 'tags': ''},
follow_redirects=True
)
assert b'Watch added' in res.data
res = client.post(
url_for("ui.ui_views.form_quick_watch_add"),
data={"url": f"{test_url}?second=1", 'tags': ''},
follow_redirects=True
)
assert b'Watch limit reached (1/1 watches)' in res.data
assert b'Watch added' not in res.data
assert len(datastore.data['watching']) == 1
delete_all_watches(client)
def test_ui_import_reports_limit_once_and_hands_back_the_rest(client, live_server, measure_memory_usage, datastore_path, monkeypatch):
"""The importer stops at the limit instead of letting add_watch() flash per remaining row."""
datastore = live_server.app.config['DATASTORE']
test_url = url_for('test_endpoint', _external=True)
monkeypatch.setenv('PAGE_WATCH_LIMIT', '2')
urls = "\n".join(f"{test_url}?i={i}" for i in range(5))
res = client.post(url_for("imports.import_page"), data={"urls": urls}, follow_redirects=True)
assert res.data.count(b'Watch limit reached') == 1, "The limit should be reported once for the file, not once per row"
assert len(datastore.data['watching']) == 2
# 3 unprocessed URLs come back in the textarea to retry once there's room
assert b'3 Skipped' in res.data
assert b'i=4' in res.data
delete_all_watches(client)
def test_clone_refused_at_limit(client, live_server, measure_memory_usage, datastore_path, monkeypatch):
"""Clone used to raise KeyError(None) here, then redirect to an edit page for uuid=None."""
datastore = live_server.app.config['DATASTORE']
test_url = url_for('test_endpoint', _external=True)
uuid = datastore.add_watch(url=test_url)
monkeypatch.setenv('PAGE_WATCH_LIMIT', '1')
res = client.post(url_for("ui.form_clone", uuid=uuid), follow_redirects=True)
assert res.status_code == 200
assert b'Watch limit reached (1/1 watches)' in res.data
assert b'Cloned' not in res.data
assert len(datastore.data['watching']) == 1
delete_all_watches(client)
def test_over_limit_instance_still_loads_and_edits(client, live_server, measure_memory_usage, datastore_path, monkeypatch):
"""A limit set below what an install already holds must only block *new* watches.
Everything already there keeps loading from disk and stays editable - the limit is not
retroactive and never hides or drops a watch.
"""
api_key = live_server.app.config['DATASTORE'].data['settings']['application'].get('api_access_token')
datastore = live_server.app.config['DATASTORE']
test_url = url_for('test_endpoint', _external=True)
monkeypatch.delenv('PAGE_WATCH_LIMIT', raising=False)
uuids = [datastore.add_watch(url=f"{test_url}?i={i}") for i in range(3)]
assert all(uuids)
# Now cap it well below what's already stored
monkeypatch.setenv('PAGE_WATCH_LIMIT', '1')
assert datastore.watch_limit_reached() is True
# Re-reading from disk is not gated by the limit
datastore._load_watches()
assert len(datastore.data['watching']) == 3
# Still all listed
assert client.get(url_for("watchlist.index")).status_code == 200
res = client.get(url_for("createwatch"), headers={'x-api-key': api_key})
assert len(res.json) == 3
# And still editable
res = client.put(
url_for("watch", uuid=uuids[0]),
data=json.dumps({"title": "Still editable"}),
headers={'content-type': 'application/json', 'x-api-key': api_key},
)
assert res.status_code == 200
assert datastore.data['watching'][uuids[0]].get('title') == "Still editable"
# Only adding is refused
res = client.post(
url_for("createwatch"),
data=json.dumps({"url": f"{test_url}?new=1"}),
headers={'content-type': 'application/json', 'x-api-key': api_key},
)
assert res.status_code == 429
assert len(datastore.data['watching']) == 3
delete_all_watches(client)
def test_limit_shown_in_settings_info_tab(client, live_server, measure_memory_usage, datastore_path, monkeypatch):
"""The Info tab names the limit only when one is configured."""
monkeypatch.delenv('PAGE_WATCH_LIMIT', raising=False)
res = client.get(url_for("settings.settings_page"))
assert b'Maximum number of page watches' not in res.data
monkeypatch.setenv('PAGE_WATCH_LIMIT', '42')
res = client.get(url_for("settings.settings_page"))
assert b'Maximum number of page watches' in res.data
assert b'42' in res.data
def test_limit_reported_without_a_request_context(client, live_server, measure_memory_usage, datastore_path, monkeypatch, mocker):
"""The CLI (-u) and the API's background import thread call add_watch() with no request
context, where flash() raises RuntimeError instead of reporting anything."""
datastore = live_server.app.config['DATASTORE']
test_url = url_for('test_endpoint', _external=True)
monkeypatch.setenv('PAGE_WATCH_LIMIT', '1')
assert datastore.add_watch(url=test_url)
# pytest-flask pushes a request context around every test, so take it away
mocker.patch('changedetectionio.store.has_request_context', return_value=False)
assert datastore.add_watch(url=f"{test_url}?second=1") is None
assert len(datastore.data['watching']) == 1
delete_all_watches(client)
@@ -79,5 +79,69 @@ class TestTriggerConditions(unittest.TestCase):
self.assertTrue(result.get('result'))
def test_conditions_filter_complete_rules_with_zero_values(self):
from changedetectionio.conditions import filter_complete_rules
rules = [
{"operator": "==", "field": "word_count", "value": 0},
{"operator": "<=", "field": "levenshtein_distance", "value": 0.0},
{"operator": "==", "field": "price", "value": "0"},
{"operator": "==", "field": "empty_val", "value": ""},
{"operator": "==", "field": "none_val", "value": None},
{"operator": "==", "field": "str_none", "value": "None"},
{"operator": "", "field": "missing_op", "value": 0},
{"operator": "None", "field": "str_none_op", "value": 0},
{"operator": "==", "field": "", "value": 0},
]
complete = filter_complete_rules(rules)
self.assertEqual(len(complete), 3)
self.assertEqual(complete[0]["value"], 0)
self.assertEqual(complete[1]["value"], 0.0)
self.assertEqual(complete[2]["value"], "0")
def test_conditions_convert_to_jsonlogic_with_zero_values(self):
from json_logic import jsonLogic
from changedetectionio.conditions import convert_to_jsonlogic
rule_int_zero = [{"operator": "==", "field": "word_count", "value": 0}]
jl_int = convert_to_jsonlogic("and", rule_int_zero)
self.assertEqual(jl_int, {"==": [{"var": "word_count"}, 0]})
self.assertTrue(jsonLogic(jl_int, {"word_count": 0}))
self.assertFalse(jsonLogic(jl_int, {"word_count": 5}))
rule_float_zero = [{"operator": "<=", "field": "levenshtein_distance", "value": 0.0}]
jl_float = convert_to_jsonlogic("and", rule_float_zero)
self.assertEqual(jl_float, {"<=": [{"var": "levenshtein_distance"}, 0.0]})
self.assertTrue(jsonLogic(jl_float, {"levenshtein_distance": 0.0}))
self.assertFalse(jsonLogic(jl_float, {"levenshtein_distance": 2.5}))
def test_conditions_execution_zero_word_count(self):
# Test condition checking for empty page (word_count == 0)
self.store.data['watching'][self.watch_uuid].update(
{
"conditions_match_logic": "ALL",
"conditions": [
{"operator": "==", "field": "word_count", "value": 0},
],
}
)
# Empty text has word_count 0 -> condition should match (True)
res_empty = execute_ruleset_against_all_plugins(
current_watch_uuid=self.watch_uuid,
application_datastruct=self.store.data,
ephemeral_data={'text': ""},
)
self.assertTrue(res_empty.get('result'))
# Non-empty text has word_count > 0 -> condition should not match (False)
res_nonempty = execute_ruleset_against_all_plugins(
current_watch_uuid=self.watch_uuid,
application_datastruct=self.store.data,
ephemeral_data={'text': "Hello world"},
)
self.assertFalse(res_nonempty.get('result'))
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,242 @@
#!/usr/bin/env python3
"""Tests for the shared "may the server fetch this URL?" gate.
# run from dir above changedetectionio/ dir
# python3 -m unittest changedetectionio.tests.unit.test_fetch_url_gate
Every server-side fetch entry point routes through validate_url.is_fetch_url_allowed(). Before it
existed, the file:// and private-IP rules were enforced inline in call_browser() only, so any fetch
path that did not go through call_browser() was unprotected:
* a "Goto URL" browser step could read file:///etc/passwd (GHSA-hm22-wg2m-35v4)
* /add-watch-ui/snapshot url= could fetch internal hosts (GHSA-56fq-63vj-9992)
These tests pin the gate's rules AND the browser-step choke point, so a future fetch path that
forgets to call the gate is the only way to regress it.
"""
import asyncio
import unittest
from unittest.mock import patch
from changedetectionio.browser_steps.browser_steps import steppable_browser_interface
from changedetectionio.validate_url import (
is_fetch_url_allowed,
is_special_purpose_ip,
validate_fetch_url,
validate_fetch_url_async,
)
# tests/conftest.py sets ALLOW_IANA_RESTRICTED_ADDRESSES=true for the functional suite, so the
# locked-down default has to be re-asserted explicitly rather than assumed.
LOCKED_DOWN = {'ALLOW_IANA_RESTRICTED_ADDRESSES': 'false', 'ALLOW_FILE_URI': 'false'}
OPTED_IN = {'ALLOW_IANA_RESTRICTED_ADDRESSES': 'true', 'ALLOW_FILE_URI': 'true'}
class TestFetchUrlGate(unittest.TestCase):
def assertBlocked(self, url):
ok, reason = is_fetch_url_allowed(url)
self.assertFalse(ok, f"URL '{url}' should have been blocked")
self.assertTrue(reason, f"URL '{url}' was blocked without a reason to show the user")
def assertAllowed(self, url):
ok, reason = is_fetch_url_allowed(url)
self.assertTrue(ok, f"URL '{url}' should have been allowed, got: {reason}")
def test_file_uri_blocked_by_default(self):
with patch.dict('os.environ', LOCKED_DOWN):
# All the spellings that reach the same local file
for url in ('file:///etc/passwd', 'FILE:///etc/passwd', 'file:/etc/passwd', 'file://etc/passwd'):
with self.subTest(url=url):
self.assertBlocked(url)
def test_file_uri_allowed_when_operator_opts_in(self):
with patch.dict('os.environ', OPTED_IN):
self.assertAllowed('file:///etc/passwd')
def test_file_uri_blocked_even_if_safe_protocol_regex_was_loosened(self):
"""An operator who widens SAFE_PROTOCOL_REGEX for some other scheme must not get local
file reads thrown in for free - hence the explicit file: check ahead of is_safe_valid_url()."""
env = dict(LOCKED_DOWN, SAFE_PROTOCOL_REGEX='^(http|https|ftp|file):')
with patch.dict('os.environ', env):
self.assertBlocked('file:///etc/passwd')
def test_private_and_reserved_addresses_blocked_by_default(self):
with patch.dict('os.environ', LOCKED_DOWN):
for url in ('http://127.0.0.1:5000/',
'http://localhost/',
'http://169.254.169.254/latest/meta-data/', # cloud metadata
'http://192.168.1.1/',
'http://10.0.0.1/',
'http://[::1]/'):
with self.subTest(url=url):
self.assertBlocked(url)
def test_cgnat_and_other_non_global_addresses_blocked_by_default(self):
"""GHSA-gwph-fp79-379w - the 0.54.1 predicate only tested is_private/is_loopback/
is_link_local/is_reserved, none of which are True for RFC 6598 CGNAT space, so
100.64.0.0/10 (an ISP's other subscribers, CPE admin panels, CGNAT gateways) stayed
fetchable. These are IP literals, so no DNS is involved and CI cannot flake."""
with patch.dict('os.environ', LOCKED_DOWN):
for url in ('http://100.64.0.1/', # RFC 6598 CGNAT, first usable
'http://100.127.255.254/', # RFC 6598 CGNAT, last usable
'http://100.100.100.100/', # inside CGNAT (Alibaba Cloud metadata)
'http://192.88.99.1/', # RFC 7526 deprecated 6to4 relay anycast
'http://224.0.0.1/', # IPv4 multicast all-hosts
'http://[ff02::1]/'): # IPv6 multicast all-nodes
with self.subTest(url=url):
self.assertBlocked(url)
def test_cgnat_allowed_when_operator_opts_in(self):
"""CGNAT is legitimate for operators monitoring their own carrier network, so the
opt-in has to release it the same way it releases 127.0.0.1."""
with patch.dict('os.environ', OPTED_IN):
self.assertAllowed('http://100.64.0.1/')
def test_special_purpose_ip_classification(self):
"""The predicate itself, without DNS - one place to pin what is and is not fetchable."""
for ip in ('100.64.0.1', '100.127.255.254', '192.88.99.1', '224.0.0.1', 'ff02::1',
'127.0.0.1', '10.0.0.1', '169.254.169.254', '192.168.1.1', '::1',
'0.0.0.0', '255.255.255.255', '198.18.0.1', 'fc00::1', 'fe80::1',
'::ffff:100.64.0.1', # CGNAT wrapped as an IPv4-mapped IPv6 address
'2002:6440:1::'): # CGNAT wrapped as a 6to4 address
with self.subTest(ip=ip):
blocked, why = is_special_purpose_ip(ip)
self.assertTrue(blocked, f"{ip} should be refused")
self.assertTrue(why, f"{ip} was refused without a stated reason")
for ip in ('1.1.1.1', '8.8.8.8', '93.184.216.34', '2606:4700:4700::1111'):
with self.subTest(ip=ip):
blocked, why = is_special_purpose_ip(ip)
self.assertFalse(blocked, f"public address {ip} was refused as '{why}'")
def test_cgnat_boundaries_are_exact(self):
"""100.64.0.0/10 ends at 100.127.255.255 - 100.63.x and 100.128.x are ordinary public
space and must not be collateral damage from a /8-sized over-block."""
for ip in ('100.63.255.255', '100.128.0.0'):
with self.subTest(ip=ip):
blocked, why = is_special_purpose_ip(ip)
self.assertFalse(blocked, f"public address {ip} was refused as '{why}'")
def test_private_addresses_allowed_when_operator_opts_in(self):
with patch.dict('os.environ', OPTED_IN):
self.assertAllowed('http://127.0.0.1:5000/')
def test_source_prefix_is_stripped_before_the_hostname_check(self):
"""Load-bearing, not cosmetic: urlparse('source:http://127.0.0.1/') reports NO hostname,
so leaving the prefix on would hand the private-IP check nothing to look at and let it pass."""
with patch.dict('os.environ', LOCKED_DOWN):
self.assertBlocked('source:http://127.0.0.1/')
self.assertBlocked('SOURCE:http://169.254.169.254/')
self.assertBlocked('source:file:///etc/passwd')
def test_jinja2_is_rendered_before_the_hostname_check(self):
"""The fetch uses the rendered URL, so the rendered URL is what must be judged - otherwise
a template expression hides the real target from the check."""
with patch.dict('os.environ', LOCKED_DOWN):
self.assertBlocked("http://{{ '127.0.0.1' }}/")
self.assertBlocked("http://{% if 1 %}127.0.0.1{% endif %}/")
def test_parser_differential_payload_always_rejected(self):
"""GHSA-rph4-96w6-q594: urlparse sees PUBLIC, urllib3 connects to INTERNAL. A backslash has
no legitimate use in a URL, so this is refused even with both opt-ins enabled."""
for env in (LOCKED_DOWN, OPTED_IN):
with self.subTest(env=env), patch.dict('os.environ', env):
self.assertBlocked('http://127.0.0.1:8888\\@example.com/')
def test_unsupported_schemes_rejected(self):
with patch.dict('os.environ', LOCKED_DOWN):
for url in ('javascript:alert(1)', 'data:text/html,<h1>x', 'chrome://version'):
with self.subTest(url=url):
self.assertBlocked(url)
def test_empty_input_rejected(self):
with patch.dict('os.environ', LOCKED_DOWN):
for url in ('', ' ', None):
with self.subTest(url=url):
self.assertBlocked(url)
def test_ordinary_public_urls_still_allowed(self):
# Unresolvable hostnames are allowed by design (DNS may be down, domain not yet live), so
# these pass with or without working DNS in CI.
with patch.dict('os.environ', LOCKED_DOWN):
for url in ('https://example.com/',
'source:https://example.com/',
'https://example.com/path?a=b&c=d#frag'):
with self.subTest(url=url):
self.assertAllowed(url)
def test_validate_fetch_url_raises_with_the_reason(self):
with patch.dict('os.environ', LOCKED_DOWN):
with self.assertRaises(ValueError):
validate_fetch_url('file:///etc/passwd')
validate_fetch_url('https://example.com/') # must not raise
def test_validate_fetch_url_async_raises_with_the_reason(self):
with patch.dict('os.environ', LOCKED_DOWN):
with self.assertRaises(ValueError):
asyncio.run(validate_fetch_url_async('http://127.0.0.1/'))
asyncio.run(validate_fetch_url_async('https://example.com/')) # must not raise
class _RecordingPage:
"""Stands in for the Playwright page so we can assert navigation never happened."""
def __init__(self):
self.goto_calls = []
async def goto(self, url, **kwargs):
self.goto_calls.append(url)
return None
async def wait_for_timeout(self, ms):
return None
class TestBrowserStepGotoUrlGate(unittest.TestCase):
"""GHSA-hm22-wg2m-35v4 - browser step values are raw user input and were never validated.
action_goto_url() is the single choke point for every navigation we initiate (the "Goto URL"
step, "Goto site", the live Browser Steps UI and the Add Watch preview all land here), so the
assertion that matters is that page.goto() is never reached for a refused URL.
"""
def _interface(self, start_url='https://example.com/'):
interface = steppable_browser_interface(start_url=start_url)
interface.page = _RecordingPage()
return interface
def test_goto_url_step_cannot_read_local_files(self):
with patch.dict('os.environ', LOCKED_DOWN):
interface = self._interface()
with self.assertRaises(ValueError):
asyncio.run(interface.action_goto_url(value='file:///etc/passwd'))
self.assertEqual(interface.page.goto_calls, [], "Chromium was navigated to a refused URL")
def test_goto_url_step_cannot_reach_private_addresses(self):
with patch.dict('os.environ', LOCKED_DOWN):
for url in ('http://127.0.0.1:5000/', 'http://169.254.169.254/latest/meta-data/'):
with self.subTest(url=url):
interface = self._interface()
with self.assertRaises(ValueError):
asyncio.run(interface.action_goto_url(value=url))
self.assertEqual(interface.page.goto_calls, [])
def test_goto_site_step_validates_the_start_url_too(self):
with patch.dict('os.environ', LOCKED_DOWN):
interface = self._interface(start_url='source:http://127.0.0.1/')
with self.assertRaises(ValueError):
asyncio.run(interface.action_goto_site())
self.assertEqual(interface.page.goto_calls, [])
def test_permitted_url_still_navigates(self):
with patch.dict('os.environ', LOCKED_DOWN):
interface = self._interface()
asyncio.run(interface.action_goto_url(value='https://example.com/'))
self.assertEqual(interface.page.goto_calls, ['https://example.com/'])
if __name__ == '__main__':
unittest.main()
@@ -370,6 +370,23 @@ class TestHistoryPathTraversal(unittest.TestCase):
history = watch.history
self.assertEqual(history, {}, "Path traversal entry must be rejected")
def test_parent_dir_entry_is_rejected_by_the_containment_check(self):
"""A bare '..' is the shortest entry that reaches and fails the containment check.
os.path.basename() reduces '/etc/passwd' and '../../etc/passwd' to
'passwd', so those two stop at the os.path.exists() check below the
guard rather than at the guard itself. '..' survives basename() intact
and resolves to the parent of data_dir, which does exist, so the
containment check is what rejects it. Not the only such input —
'../..' and 'foo/..' collapse to the same thing — and not the only
reason the check exists: it also blocks a filename inside data_dir
that is itself a symlink pointing outside.
"""
watch = self._make_watch()
self._write_history_txt(watch, ['1000000000,..\n'])
history = watch.history
self.assertEqual(history, {}, "Parent-directory entry must be rejected")
def test_normal_snapshot_entry_is_accepted(self):
"""A bare filename written by save_history_blob must still load correctly."""
import uuid as uuid_builder
@@ -377,8 +394,11 @@ class TestHistoryPathTraversal(unittest.TestCase):
watch.save_history_blob(contents="hello world", timestamp=1000000000, snapshot_id=str(uuid_builder.uuid4()))
history = watch.history
self.assertEqual(len(history), 1, "Normal snapshot entry must be accepted")
# Watch.history resolves entries with os.path.realpath, so compare against a
# resolved data_dir. On macOS the datastore lives under /tmp, which is a symlink
# to /private/tmp, and an unresolved comparison fails there for a correct path.
self.assertTrue(
list(history.values())[0].startswith(watch.data_dir),
list(history.values())[0].startswith(os.path.realpath(watch.data_dir)),
"Resolved path must be inside the watch data directory"
)
@@ -408,5 +428,47 @@ class TestHistoryPathTraversal(unittest.TestCase):
)
class TestLinkToOpen(unittest.TestCase):
"""`link_to_open` - the optional human-facing URL used instead of the watched URL."""
def _make_watch(self, **fields):
mock_datastore = {'settings': {'application': {}}, 'watching': {}}
return Watch.model(datastore_path='/tmp', __datastore=mock_datastore, default=fields)
def test_defaults_to_the_watched_url(self):
watch = self._make_watch(url='https://example.com/feed.xml')
assert watch.open_link_override == ''
assert watch.open_link == 'https://example.com/feed.xml'
assert watch.open_link == watch.link
def test_override_wins_when_set(self):
watch = self._make_watch(url='https://example.com/api/v1/items.json',
link_to_open='https://example.com/shop/items')
assert watch.link == 'https://example.com/api/v1/items.json'
assert watch.open_link == 'https://example.com/shop/items'
assert watch.open_link_override == 'https://example.com/shop/items'
def test_whitespace_only_override_is_ignored(self):
watch = self._make_watch(url='https://example.com/feed.xml', link_to_open=' ')
assert watch.open_link_override == ''
assert watch.open_link == 'https://example.com/feed.xml'
def test_unsafe_override_falls_back_to_the_watched_url(self):
watch = self._make_watch(url='https://example.com/feed.xml',
link_to_open='javascript:alert(1)')
assert watch.open_link_override == ''
assert watch.open_link == 'https://example.com/feed.xml'
def test_source_prefix_is_stripped_from_the_override(self):
watch = self._make_watch(url='source:https://example.com/feed.xml',
link_to_open='source:https://example.com/page')
assert watch.open_link == 'https://example.com/page'
def test_jinja2_in_the_override_is_rendered(self):
watch = self._make_watch(url='https://example.com/feed.xml',
link_to_open='https://example.com/page?id={{ 1+1 }}')
assert watch.open_link == 'https://example.com/page?id=2'
if __name__ == '__main__':
unittest.main()
+25
View File
@@ -249,6 +249,31 @@ def new_live_server_setup(live_server):
import secrets
return "Random content - {}\n".format(secrets.token_hex(64))
# Re-navigation gate: the first hit answers with an error status AND a client-side meta
# refresh, the second serves the real page. This mirrors sites that gate visitors they have
# not seen recently (reported against fotokoch.de, which answers 503 + meta refresh and then
# serves a 200). The browser follows the refresh, so the fetch has to be judged on the
# document we actually end up extracting rather than on the interstitial.
# Keyed on last-seen time rather than a hit count, so EVERY fresh check starts out gated -
# a counter would serve a clean 200 to the second check and let the test pass without the fix.
_interstitial_last_seen = {}
@live_server.app.route('/test-interstitial')
def test_interstitial():
key = request.args.get('key', 'default')
now = time.time()
seen_recently = (now - _interstitial_last_seen.get(key, 0)) < 10
_interstitial_last_seen[key] = now
if not seen_recently:
resp = make_response(
'<html><head><meta http-equiv="refresh" content="1"></head>'
'<body>Browser check in progress, you will be redirected</body></html>', 503)
else:
resp = make_response(
'<html><body><h1>The real page content is here</h1></body></html>', 200)
resp.headers['Content-Type'] = 'text/html'
return resp
@live_server.app.route('/test-endpoint2')
def test_endpoint2():
return "<html><body>some basic content</body></html>"
@@ -205,8 +205,8 @@ def test_browsersteps_edit_UI_startsession(client, live_server, measure_memory_u
uuid = client.application.config.get('DATASTORE').add_watch(url=test_url, extras={'fetch_backend': 'html_webdriver', 'paused': True})
# Test starting a browsersteps session
res = client.get(
# Test starting a browsersteps session (POST only - it spins up a real browser)
res = client.post(
url_for("browser_steps.browsersteps_start_session", uuid=uuid),
follow_redirects=True
)
+1
View File
@@ -198,6 +198,7 @@ Never fix one language and move on.
| `en_US` | English (US) |
| `es` | Spanish (Español) |
| `fr` | French (Français) |
| `id` | Indonesian (Bahasa Indonesia) |
| `it` | Italian (Italiano) |
| `ja` | Japanese (日本語) |
| `ko` | Korean (한국어) |

Some files were not shown because too many files have changed in this diff Show More