AI_Assistant/pcbjam
Watch
1
0
Fork
You've already forked pcbjam
 0
Code Issues Pull requests Projects Releases 4 Packages Wiki Activity Actions 2
pcbjam/tests/tools/screenshots
History
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Viktor Vaczi 3efcdcf91c fix(tests): enforce screenshot gate; drop shadow dir, flaky retinascale, 12 stale 
Finalize the screenshot review system after the first Linux re-baseline
(that run showed 355/356 stable — the placeholder floor is fine as-is):

- ENFORCE the gate: `screenshots:check --fail-on-change` now fails the build on any
  changed/added/removed vs baselines (it still posts the drift report first, so the
  failure is actionable; a real render change → re-promote). The generic "CI failed"
  notice is suppressed for gate failures so we don't double-post on drift.
- Exclude retinascale-01-loaded: a fullPage HiDPI test whose captured height + DPR
  scaling vary run-to-run (~60% inter-run diff) — a flaky test, not render noise.
  IGNORE_SCREENSHOTS in config.ts; compare/promote/gen-manifest skip it; baseline removed.
- Kill the baseline-dir shadowing: drop e2e/baseline-screenshots/ from BASELINE_DIRS +
  delete its 3 files (grid-tab-final, wxgrid-controls, wxgrid-dedicated-page) that
  duplicated names in baseline-screenshots/ with different bytes.
- Prune 12 stale baselines (renamed/removed specs: wizard-01..04, gerbview-wizard-01..04,
  zoom-pl_editor-*, popup-03-palette).
- Delete the dead compare-screenshots.sh / update-baseline-screenshots.sh.
- Regenerate screenshot-manifest.json (355 entries).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-02 11:30:33 +02:00
..
changelog.ts feat(tests): screenshot regression + Discord review, perf-tracked 2026-07-01 18:17:20 +02:00
compare.ts fix(tests): enforce screenshot gate; drop shadow dir, flaky retinascale, 12 stale 2026-07-02 11:30:33 +02:00
config.ts fix(tests): enforce screenshot gate; drop shadow dir, flaky retinascale, 12 stale 2026-07-02 11:30:33 +02:00
gen-manifest.ts fix(tests): enforce screenshot gate; drop shadow dir, flaky retinascale, 12 stale 2026-07-02 11:30:33 +02:00
image-ops.ts feat(tests): screenshot regression + Discord review, perf-tracked 2026-07-01 18:17:20 +02:00
noise.ts feat(tests): screenshot regression + Discord review, perf-tracked 2026-07-01 18:17:20 +02:00
perf-report.ts feat(tests): screenshot regression + Discord review, perf-tracked 2026-07-01 18:17:20 +02:00
post-discord.ts fix(tests): keep test-results across CI suites; loud pipeline alert 2026-07-02 09:20:35 +02:00
promote.ts fix(tests): enforce screenshot gate; drop shadow dir, flaky retinascale, 12 stale 2026-07-02 11:30:33 +02:00
README.md feat(tests): screenshot regression + Discord review, perf-tracked 2026-07-01 18:17:20 +02:00

README.md

Screenshot regression + Discord review tooling

One comparison engine + a churn-free updater + a Discord reporter for the e2e screenshots. Design and rationale: ~/.claude/plans/…snowglobe.md (or ask).

Source of truth = CI's Linux render. The dev never authors baselines on the Mac (Mac fonts/GL ≠ CI). Instead, CI renders on every push; when a render change is intentional you promote CI's artifact into the committed baselines. The environment isn't pinned — if the host's Mesa/fonts drift, the gate lights up in Discord and you just re-promote (broad + low-intensity change ⇒ likely drift).

Files

  • config.ts — thresholds, baseline dirs, per-engine floors (calibrate!), clustering knobs.
  • image-ops.ts — PNG load/save, pixelmatch diff (AA-excluded), connected-component boxes, triptych compositing, size-cap resize.
  • compare.ts — the engine: classify baselines vs test-results/ → test-results/screenshot-diff/report.json + triptych/heatmap PNGs. --pair diffs two files.
  • promote.ts — churn-free updater: pull a CI run's shots (gh run download) or --from DIR; overwrite a baseline only when pixels differ beyond the floor (verbatim bytes, no re-encode) → no git churn.
  • perf-report.ts — renders the track-only runtime-perf table (loadMs/openMs/FPS) with Δ vs the previous main run (fetched via gh).
  • post-discord.ts — the always-on CI-on-main report: SHA + e2e status + perf table, then screenshot triptychs (batched, size-capped, flood-collapsed).
  • changelog.ts — Discord trigger B: git-history diff of committed baseline PNGs (no build/GPU).
  • noise.ts — calibration: diff two identical-input renders → per-engine noise floor.
  • gen-manifest.ts — regenerate screenshot-manifest.json (canonical name list + best-effort engine tag) by scanning the committed baselines + specs.

npm scripts (run from tests/)

npm run screenshots:check      # gate: baselines vs test-results → report.json (exit 0; add --fail-on-change to gate)
npm run screenshots:promote -- --run <ci-run-id>   # churn-free re-baseline from a CI run (or --from DIR)
npm run screenshots:report -- --e2e pass           # post the CI report to Discord (main+push only; needs DISCORD_WEBHOOK_URL)
npm run screenshots:changelog                       # post the baseline changelog (main+push only)
npm run screenshots:noise -- run1/ run2/            # calibrate floors
npm run screenshots:manifest                        # regenerate the manifest (--check to verify it's fresh)

Activation checklist

  • screenshot-manifest.json generated (name list authoritative; engine tags best-effort until calibration).
  • scale:'device'→'css' normalized (no-op at CI's DSF=1).
  1. Add the DISCORD_WEBHOOK_URL repo secret — until then everything is inert.
  2. Calibrate: run the suite twice in CI, screenshots:noise the two dirs, set FLOORS in config.ts.
  3. First re-baseline: promote a clean CI run's render, commit (expect a big, one-time chrome-font diff vs the Mac baselines).
  4. Delete the old scripts/{compare,update-baseline}-screenshots.sh.
  5. Once floors are proven stable, flip the gate to --fail-on-change.