lee/pentagi
1
0
Fork 0
mirror of https://github.com/vxcontrol/pentagi.git synced 2026-08-25 20:46:31 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(ci): don't post a green e2e comment when the run's result is unconfirmed

Browse Source 
The sticky comment computed `mockPassed` by falling back to results.json alone
whenever the jobs API was unreachable. A run killed by the global timeout writes
its results.json before the abort is observed — the un-run tests land in
`skipped`, so `unexpected` stays 0 — so that fallback posted a ✅ on a run that
never finished.

Model three states instead of a boolean: pass (zero failures AND the job
confirmed success), fail, and unknown (jobs API unreachable, so the job cannot
be confirmed). Unknown renders ⚠️ with a note to check the run, never ✅.
Verified by driving the extracted script over a timeout-shaped results.json with
a rejecting jobs API: it now yields ⚠️ unknown where it used to yield ✅.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Sergey Kozyrenko
2026-07-25 03:19:04 +07:00
co-authored by Claude Opus 4.8
parent 81f3d06ebd
commit fab990d9b9
1 changed files with 14 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/e2e-report.yml
+14 -6
View File
@@ -112,19 +112,27 @@ jobs:
core.info(`results.json unavailable: ${error}`);
}
// Both halves are required: results.json is written before an abort can be
// observed, so a run killed by the global timeout still reports zero failures.
// When the jobs API was unreachable, fall back to results.json alone.
const mockPassed = noFailures && (jobsFetched ? mockJob?.conclusion === 'success' : true);
// The job conclusion is load-bearing, not a nicety: results.json is written before a
// global-timeout abort can be observed (its un-run tests land in `skipped`, so
// `unexpected` stays 0), and only the job's own conclusion reflects the abort. So a
// green needs BOTH zero failures AND a successful job. When the jobs API was
// unreachable we cannot confirm the job, so the verdict is unknown — never a green.
const mockStatus = !noFailures || (jobsFetched && mockJob?.conclusion !== 'success')
? 'fail'
: jobsFetched
? 'pass'
: 'unknown';
// Key the advisory on the e2e-visual-diffs artifact (uploaded only when the snapshot
// step itself failed), not on the visual job's conclusion — a tag-guard, install or
// build failure would otherwise read as "snapshots differ".
const visualDiffers = '${{ steps.visualdiffs.outcome }}' === 'success';
const conclusion = !mockPassed ? '❌' : visualDiffers ? '⚠️' : '✅';
const conclusion = mockStatus === 'fail' ? '❌' : mockStatus === 'unknown' || visualDiffers ? '⚠️' : '✅';
const marker = '<!-- e2e-report -->';
const advisoryLine = visualDiffers
const advisoryLine = mockStatus === 'unknown'
? '⚠️ **Result unconfirmed** — the workflow jobs API was unreachable, so this reflects `results.json` alone, which cannot distinguish a clean pass from a run killed mid-flight. Check the run before merging.'
: visualDiffers
? '⚠️ **Visual snapshots differ** — advisory only, does not block merge. Download the `e2e-visual-diffs` artifact to compare expected / actual / diff.'
: null;
const body = [