fix(tests): keep test-results across CI suites; loud pipeline alert

The screenshot report went silent on green CI runs. CI runs four sequential
`playwright test` invocations (wx, asyncify, kicad, perf) that all default their
outputDir to `test-results`; Playwright clears the whole outputDir at the start
of every run, so each suite wiped the previous suite's screenshots. By the time
`compare` ran, only perf-*.json remained -> all 354 baselines read as "removed"
-> the no-render guard stayed silent.

- Gate Playwright's outputDir off test-results/ in CI only
  (process.env.CI ? 'pw-artifacts/<suite>' : 'test-results') in the wx / asyncify
  / kicad configs, so the committed-baseline screenshots (page.screenshot to
  'test-results/...') accumulate in test-results/ across the suites for compare.
  Local single-suite behavior is unchanged.
- post-discord: replace the silent no-render guard with a loud, image-less
  pipeline alert ("No screenshots produced this run") when 0 render; genuine
  partial removals still post as the normal capped REMOVED list.
- gitignore pw-artifacts/; upload tests/pw-artifacts/** as a CI artifact.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Viktor Vaczi 2026-07-02 09:20:35 +02:00
commit b270507505
6 changed files with 31 additions and 4 deletions

View file

@ -217,11 +217,25 @@ async function main(): Promise<void> {
const sha = process.env.GITHUB_SHA;
const report = readReport(root);
// Defensive: if the run produced no screenshots at all (renders missing), don't
// post a bogus "everything removed" report. (The workflow already gates this step
// to successful runs, where renders exist.)
// Loud pipeline alert: if the run produced ZERO screenshots (every baseline shows as
// "removed"), that's almost always a build/pipeline failure — not a real mass removal.
// Post a distinct, image-less alert instead of dumping a giant REMOVED list.
if (report && report.changed.length + report.added.length + report.unchangedCount === 0) {
console.log('[discord] no screenshots rendered — nothing to report');
const shortSha = sha ? sha.slice(0, 7) : 'local';
const runUrl =
process.env.GITHUB_SERVER_URL && process.env.GITHUB_REPOSITORY && process.env.GITHUB_RUN_ID
? `${process.env.GITHUB_SERVER_URL}/${process.env.GITHUB_REPOSITORY}/actions/runs/${process.env.GITHUB_RUN_ID}`
: '';
const alert =
`⚠️ **No screenshots produced this run** · \`${shortSha}\` — compare saw 0 rendered ` +
`vs ${report.removed.length} baselines. Likely a build/pipeline failure, not a mass removal.` +
(runUrl ? `\n${runUrl}` : '');
if (args.dryRun) {
console.log('--- message 1/1 (0 files) ---\n' + alert);
} else {
await postMessage(webhook!, { content: alert, files: [] });
console.log('[discord] posted pipeline alert (0 screenshots rendered)');
}
return;
}
const prevDir = fetchPreviousPerf((args.repo as string) || process.env.GITHUB_REPOSITORY, sha);