13551f4f22
New tooling in tests/tools/screenshots/ (TypeScript via tsx): - compare.ts: one pixelmatch engine (AA-excluded), connected-component "where to look" boxes, old|new+boxes|heatmap triptych, per-engine floors. - promote.ts: churn-free updater — overwrite a baseline only when decoded pixels differ beyond the floor, copying CI bytes verbatim (no re-encode churn); pulls a CI run via `gh run download` or a local --from dir. - post-discord.ts: always-on CI-on-main report (SHA + e2e status + the track-only runtime-perf table), then screenshot triptychs, batched + size-capped + flood-collapsed + 429-aware. - perf-report.ts: perf table with Δ vs the previous main run (via gh). - changelog.ts: no-build git-history baseline differ (Discord trigger B). - noise.ts / gen-manifest.ts: calibration + manifest generation. CI wiring: - wasm-build.yml: post-test step runs the gate + report on the already- produced test-results (no extra build); report-only (continue-on-error), posts only on push to main, inert without DISCORD_WEBHOOK_URL. - ci-ubicloud.yml: secrets: inherit (pass the webhook through). - screenshot-changelog.yml: ~30s no-build changelog on baseline changes. screenshot-manifest.json: canonical 354-name set + best-effort engine tags (313 chromium-swiftshader / 41 firefox-llvmpipe). Normalize scale:'device'->'css' across 18 spec files (no-op at CI DSF=1) so committed baselines are uniformly css-scaled. Design: CI's Linux render is the single source of truth; no pinned container (accept rare env drift -> re-promote); dev commits via promote. Replaces the byte-cmp compare-screenshots.sh + file-size-proxy update-baseline-screenshots.sh (kept for now until the first re-baseline). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
40 lines
3.1 KiB
Markdown
40 lines
3.1 KiB
Markdown
# Screenshot regression + Discord review tooling
|
|
|
|
One comparison engine + a churn-free updater + a Discord reporter for the e2e
|
|
screenshots. Design and rationale: `~/.claude/plans/…snowglobe.md` (or ask).
|
|
|
|
**Source of truth = CI's Linux render.** The dev never authors baselines on the
|
|
Mac (Mac fonts/GL ≠ CI). Instead, CI renders on every push; when a render change
|
|
is intentional you *promote* CI's artifact into the committed baselines. The
|
|
environment isn't pinned — if the host's Mesa/fonts drift, the gate lights up in
|
|
Discord and you just re-promote (broad + low-intensity change ⇒ likely drift).
|
|
|
|
## Files
|
|
- `config.ts` — thresholds, baseline dirs, per-engine floors (calibrate!), clustering knobs.
|
|
- `image-ops.ts` — PNG load/save, pixelmatch diff (AA-excluded), connected-component boxes, triptych compositing, size-cap resize.
|
|
- `compare.ts` — the engine: classify baselines vs `test-results/` → `test-results/screenshot-diff/report.json` + triptych/heatmap PNGs. `--pair` diffs two files.
|
|
- `promote.ts` — churn-free updater: pull a CI run's shots (`gh run download`) or `--from DIR`; overwrite a baseline only when pixels differ beyond the floor (verbatim bytes, no re-encode) → no git churn.
|
|
- `perf-report.ts` — renders the track-only runtime-perf table (loadMs/openMs/FPS) with Δ vs the previous main run (fetched via `gh`).
|
|
- `post-discord.ts` — the always-on CI-on-main report: SHA + e2e status + perf table, then screenshot triptychs (batched, size-capped, flood-collapsed).
|
|
- `changelog.ts` — Discord trigger B: git-history diff of committed baseline PNGs (no build/GPU).
|
|
- `noise.ts` — calibration: diff two identical-input renders → per-engine noise floor.
|
|
- `gen-manifest.ts` — regenerate `screenshot-manifest.json` (canonical name list + best-effort engine tag) by scanning the committed baselines + specs.
|
|
|
|
## npm scripts (run from `tests/`)
|
|
```
|
|
npm run screenshots:check # gate: baselines vs test-results → report.json (exit 0; add --fail-on-change to gate)
|
|
npm run screenshots:promote -- --run <ci-run-id> # churn-free re-baseline from a CI run (or --from DIR)
|
|
npm run screenshots:report -- --e2e pass # post the CI report to Discord (main+push only; needs DISCORD_WEBHOOK_URL)
|
|
npm run screenshots:changelog # post the baseline changelog (main+push only)
|
|
npm run screenshots:noise -- run1/ run2/ # calibrate floors
|
|
npm run screenshots:manifest # regenerate the manifest (--check to verify it's fresh)
|
|
```
|
|
|
|
## Activation checklist
|
|
- [x] `screenshot-manifest.json` generated (name list authoritative; engine tags best-effort until calibration).
|
|
- [x] `scale:'device'`→`'css'` normalized (no-op at CI's DSF=1).
|
|
1. Add the `DISCORD_WEBHOOK_URL` repo secret — until then everything is inert.
|
|
2. Calibrate: run the suite twice in CI, `screenshots:noise` the two dirs, set `FLOORS` in `config.ts`.
|
|
3. First re-baseline: `promote` a clean CI run's render, commit (expect a big, one-time chrome-font diff vs the Mac baselines).
|
|
4. Delete the old `scripts/{compare,update-baseline}-screenshots.sh`.
|
|
5. Once floors are proven stable, flip the gate to `--fail-on-change`.
|