Four independent ways the signal misled:
Adding a label to a PR skips every gate job, yet the report still ran, found no
results file and overwrote a correct green comment with a red "no results". It
now returns early when the run carries no completed mock job.
A run aborted by the global timeout still uploads a results file whose failure
count is zero, so the header could go green over a red job; the verdict now
requires the mock job's own conclusion too.
The report's concurrency key was the branch name alone, so a fork PR opened from
a branch named `main` lost its comment to upstream `main` activity. The gate's
key split `opened` from `synchronize`, leaving both runs live so the older one
could finish last.
Stand failure screenshots carried the stand URL and the login into a public
artifact — screenshots are now tier-aware like traces and videos already were.
Also guard the Tier-2 teardown: the trap is the script's last statement, so a
teardown hiccup turned a passing run red.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The sticky PR comment took its header from the whole-workflow conclusion, so a
visual-snapshot diff — advisory, never a required check — rendered a red ❌ over a
"62 passed, 0 failed" mock-tier stats line, reading as a gate failure it isn't.
Track the header on the mock gate instead (✅ pass / ❌ real failure) and surface a
differing visual job as a separate ⚠️ line pointing at the diff artifact.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Real-tier (Tier 2/3) fixes, the blocking set:
- stand-smoke asserts a per-route anchor (breadcrumb title span) instead
of a shared sidebar link that 0-matches on /settings/* and double-matches
on /templates — the smoke was structurally red on 3 of 5 routes.
- stand-tier trace/video forced off (they embedded the session cookie and
the stand password in a public-repo artifact) and the stand project
selects @stand via config grep (the CI step selected zero tests).
- run-local-tier's sandbox filter is anchored to the seeded 9xxxx range
(`^/?pentagi-terminal-9[0-9]{4,}$`) so it can't force-remove a dev's
flow-9 sandbox; sandbox cleanup moved behind the keep-stack gate; compose
logs are captured before teardown.
- mock-LLM wraps the request handler (a non-JSON body crashed the process);
its transcript now issues a real terminal exec via a delegate chain and
flow-run asserts the terminal output; OpenAI-shaped error envelope.
Mock-gate rigor + coverage:
- order-insensitive deep-subset variable matching; unmatched WS subscribes
are now accounted at teardown; REST/GraphQL entries can pin a body/vars
subset; favorite-star asserts the persisted (not optimistic) value.
- new flows/create smoke (the primary journey had no mock coverage), a
graphql-ws protocol unit test against the real client, and a
route-manifest completeness test.
- affected-routes no longer under-scopes shared component dirs.
Hygiene: .mjs tools now linted/prettier'd; dead gitignore line; JSDoc/
sheet aria-label/schema-compat/trend/serve-dist/report-comment fixes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Lays the Phase-3 substrate on top of the three tiers.
- schema-compat pre-flight (e2e/tools/schema-compat.mjs): introspects a
target backend's live GraphQL schema and validates every frontend
operation against it, so deploy skew (a renamed/removed field) fails
once, readably, instead of as dozens of red specs. Verified against the
live Tier-2 backend both ways: 105 operations pass, an injected bogus
field is caught with the exact location.
- trend aggregation (trend.mjs): turns a run's results.json into one JSONL
record (p50/p95 spec duration, slowest three, pass/flaky/fail) so slow
regressions are visible, not just green/red; CI appends it to a
90-day-retained artifact.
- diff-scoping (affected-routes.ts + affected.ts): maps a diff to the
manifest routes it touches via each route's owning sources — the
substrate for selective runs and for scoping the exploratory agent.
Pure mapping fn, unit-tested (backend-only → none; shared infra → all;
owned → that route).
- CI: an e2e-stand job (label-gated + a protected Environment whose
reviewers approve before secrets are exposed; schema-compat runs first),
a trend step, and an LLM-independent @stand smoke.
- docs: the stand tier, trend/affected tools, and the LLM advisory recipe
(playwright-mcp + init-agents + guardrails) — the recipe the
deterministic tiers plug into, not a bespoke bot.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
18 baselines (9 manifest routes × light/dark) compared only inside
mcr.microsoft.com/playwright:v<version>-noble, so pixels are identical on
every machine and CI; the wrapper derives the tag from the installed
@playwright/test version, making the pin drift-proof, and a CI guard
fails loudly if the workflow's container tag falls behind.
macOS hosts cannot run the visual project directly (parallel darwin
baselines) nor mount their node_modules into the container (native vite
binaries): the wrapper builds dist on the host and the container serves
it with a dependency-free static server — route mocks intercept API
calls before the network, so no proxy is needed. The xterm canvas is
masked (SwiftShader pixels are driver-dependent). Determinism proven by
back-to-back container runs. The e2e-visual CI job is advisory, never a
required check.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
specs/real/** now exercise the actual agent loop end to end: create flow
-> image/language/title -> tool-call-ID sampling -> subtask_list plan ->
done barrier -> subtask_patch refine -> report_result — with messages
streamed over the real GraphQL websocket and the flow settling in
Waiting. The mock LLM is ~150 lines of Node driven by a deterministic
first-match transcript; the custom provider env seam means zero backend
changes.
The stack is fully isolated from a developer machine:
- own compose project/network/ports (8444/5433), coexists with a dev stack
- --env-file /dev/null so the developer's .env (live keys, DOCKER_HOST,
config paths) never leaks into the e2e backend
- flow ids seeded from 90001: sandbox containers are named
pentagi-terminal-<flowId> on the shared docker daemon, so the range
keeps them clear of dev flows and makes runner cleanup unambiguous
(down -v cannot remove them — the backend spawns them outside compose)
- pentagi healthcheck via busybox wget (the alpine image has no curl),
since the base service has none and up --wait returns too early
Auth for live tiers moved to the canonical setup-project + storageState;
the localStorage seed is now mock-tier-only (a forged client session has
no cookie behind it and the first 401 wipes it). CI gains a nightly/
dispatch-only e2e-local job.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
setup-go's cache:true already caches the build and module caches keyed
on go.sum; the extra actions/cache step restored the same read-only
~/go/pkg/mod on top and failed with "Cannot open: File exists" whenever
both caches hit. Also sets the workflow's GITHUB_TOKEN to contents:read.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The default tier runs the production bundle (vite build + preview)
against a cassette-driven mock of the whole API surface — GraphQL over
HTTP, graphql-transport-ws subscriptions, and REST — so the suite needs
no backend, no keys, and no secrets, and fork PRs can run it.
- mock engine: (operationName, variables) matching with sequenced
entries and world flags (login flips /info guest->user without
call-order coupling); unmatched calls answer 501 and fail the test,
so nothing leaks through the vite preview proxy to a live backend
- ws mock follows the graphql-transport-ws contract the app's client
needs: immediate ack, nothing before ack, streams stay open, delta
cursors survive reconnects, drops use retryable close codes
- clock and timezone pinned on the mock tier: formatDate branches on
isToday/isThisYear, so unpinned cassette dates rot within a day
- cassettes are TS modules typed against the generated GraphQL types,
so schema/operation drift fails the existing tsc gate at compile time
- CI: fork-safe e2e.yml (read-only token, no secrets, no write steps)
plus e2e-report.yml posting a sticky PR comment via workflow_run,
resolving fork PRs by head SHA
- smoke specs (login redirect, form login, authenticated /flows render)
green on the mock tier; e2e/ wired into tsc, eslint, and prettier
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The "Frontend - Install dependencies" step ran pnpm install --frozen-lockfile
with continue-on-error: true, swallowing the ERR_PNPM_OUTDATED_LOCKFILE that the
flag exists to raise. The rest of the frontend pipeline (prettier, lint,
type-check, test) already gates strictly, so a stale pnpm-lock.yaml would only
surface later as a confusing downstream "command not found". Drop the flag so
lockfile drift fails loudly at the install step. Backend steps left as-is.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- add frontend/.nvmrc (24.17.0) as the single source of truth for Node;
GitHub CI reads it via node-version-file, Dockerfile uses node:24.17.0-slim
- bump packageManager to pnpm@11.8.0; drop "corepack prepare pnpm@latest"
so the pnpm version derives from packageManager everywhere
- migrate pnpm onlyBuiltDependencies -> allowBuilds in pnpm-workspace.yaml
(the package.json "pnpm" field is no longer read by pnpm 11)
- add a CI step that fails if the Dockerfile Node tag drifts from .nvmrc
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Prettier / Lint / Test were `continue-on-error: true` (advisory — failures did not fail CI), the same gap that let type errors pile up. Now Prettier, Lint, Type check and Test all block the lint-and-test job (which runs on every branch push).
Prerequisite: `prettier --write` on 5 pre-existing non-conformant files (pages/login.tsx, lib/report/report-pdf.tsx, 3 *.test.tsx) so the now-blocking Prettier check passes — pure formatting, no logic change. Install stays advisory (setup step); backend checks unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
vite build strips types and never type-checks, and the old build`s bare tsc (solution config, files:[]) checked nothing — so type errors reached main unnoticed (which is how 76 had accumulated). Now they fail fast everywhere.
- build: `tsc -b && vite build` — `pnpm build` (and the Docker image build, which runs `pnpm run build`) fails on any type error before bundling.
- add `typescript` script (`tsc -b`, checks both app + node project configs).
- ci.yml: blocking "Frontend - Type check" step in lint-and-test (runs on every branch push; docker-build only runs on main/tags).
Verified: a type error makes both `pnpm run typescript` and `pnpm build` exit non-zero (vite never runs); removing it restores a clean build.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
