fix(tests): enforce screenshot gate; drop shadow dir, flaky retinascale, 12 stale

Finalize the screenshot review system after the first Linux re-baseline
(that run showed 355/356 stable — the placeholder floor is fine as-is):

- ENFORCE the gate: `screenshots:check --fail-on-change` now fails the build on any
  changed/added/removed vs baselines (it still posts the drift report first, so the
  failure is actionable; a real render change → re-promote). The generic "CI failed"
  notice is suppressed for gate failures so we don't double-post on drift.
- Exclude retinascale-01-loaded: a fullPage HiDPI test whose captured height + DPR
  scaling vary run-to-run (~60% inter-run diff) — a flaky test, not render noise.
  IGNORE_SCREENSHOTS in config.ts; compare/promote/gen-manifest skip it; baseline removed.
- Kill the baseline-dir shadowing: drop e2e/baseline-screenshots/ from BASELINE_DIRS +
  delete its 3 files (grid-tab-final, wxgrid-controls, wxgrid-dedicated-page) that
  duplicated names in baseline-screenshots/ with different bytes.
- Prune 12 stale baselines (renamed/removed specs: wizard-01..04, gerbview-wizard-01..04,
  zoom-pl_editor-*, popup-03-palette).
- Delete the dead compare-screenshots.sh / update-baseline-screenshots.sh.
- Regenerate screenshot-manifest.json (355 entries).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Viktor Vaczi 2026-07-02 11:30:33 +02:00
commit 3efcdcf91c
24 changed files with 98 additions and 251 deletions

View file

@ -9,8 +9,12 @@
* the npm scripts and CI steps run from).
*/
/** Committed baseline directories, scanned in order. Filenames are the keys. */
export const BASELINE_DIRS = ['baseline-screenshots', 'e2e/baseline-screenshots'] as const;
/**
* Committed baseline directory. Single dir on purpose: `e2e/baseline-screenshots/`
* used to also hold a few names (grid-tab-final, wxgrid-*) that ALSO live here with
* different bytes — so they got silently shadowed. Consolidated to one dir.
*/
export const BASELINE_DIRS = ['baseline-screenshots'] as const;
/** Where Playwright writes the current run's screenshots (gitignored). */
export const RESULTS_DIR = 'test-results';
@ -74,6 +78,20 @@ export const FLOORS: Record<string, EngineFloor> = {
*/
export const IGNORE_REGIONS: Record<string, Array<{ x: number; y: number; width: number; height: number }>> = {};
/**
* Screenshots excluded from comparison entirely (not compared, not counted as
* changed/added/removed, not put in the manifest). For nondeterministic captures
* that can't be a stable baseline — e.g. `retinascale-01-loaded` is a `fullPage`
* HiDPI test whose captured height + DPR scaling vary run-to-run (~60% inter-run
* diff observed), a flaky test rather than render noise.
*/
export const IGNORE_SCREENSHOTS = new Set<string>(['retinascale-01-loaded.png']);
/** True if a screenshot is excluded from comparison. */
export function isIgnored(name: string): boolean {
return IGNORE_SCREENSHOTS.has(name);
}
export type ManifestEntry = { name: string; engine: string };
export type Manifest = { screenshots: ManifestEntry[] };