feat(tests): screenshot regression + Discord review, perf-tracked

New tooling in tests/tools/screenshots/ (TypeScript via tsx):
- compare.ts: one pixelmatch engine (AA-excluded), connected-component
  "where to look" boxes, old|new+boxes|heatmap triptych, per-engine floors.
- promote.ts: churn-free updater — overwrite a baseline only when decoded
  pixels differ beyond the floor, copying CI bytes verbatim (no re-encode
  churn); pulls a CI run via `gh run download` or a local --from dir.
- post-discord.ts: always-on CI-on-main report (SHA + e2e status + the
  track-only runtime-perf table), then screenshot triptychs, batched +
  size-capped + flood-collapsed + 429-aware.
- perf-report.ts: perf table with Δ vs the previous main run (via gh).
- changelog.ts: no-build git-history baseline differ (Discord trigger B).
- noise.ts / gen-manifest.ts: calibration + manifest generation.

CI wiring:
- wasm-build.yml: post-test step runs the gate + report on the already-
  produced test-results (no extra build); report-only (continue-on-error),
  posts only on push to main, inert without DISCORD_WEBHOOK_URL.
- ci-ubicloud.yml: secrets: inherit (pass the webhook through).
- screenshot-changelog.yml: ~30s no-build changelog on baseline changes.

screenshot-manifest.json: canonical 354-name set + best-effort engine tags
(313 chromium-swiftshader / 41 firefox-llvmpipe).

Normalize scale:'device'->'css' across 18 spec files (no-op at CI DSF=1)
so committed baselines are uniformly css-scaled.

Design: CI's Linux render is the single source of truth; no pinned
container (accept rare env drift -> re-promote); dev commits via promote.
Replaces the byte-cmp compare-screenshots.sh + file-size-proxy
update-baseline-screenshots.sh (kept for now until the first re-baseline).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Viktor Vaczi 2026-07-01 18:01:15 +02:00
commit 13551f4f22
36 changed files with 3186 additions and 63 deletions

View file

@ -35,6 +35,9 @@ concurrency:
jobs:
build:
uses: ./.github/workflows/wasm-build.yml
# Pass repo secrets (DISCORD_WEBHOOK_URL) through to the reusable build so the
# post-test step can post the screenshot + perf report on push to main.
secrets: inherit
with:
# -O1 asyncify shrink — the level we ship (release.yml uses the same -O1).
# 3D viewer ON so 3d-viewer.spec.ts has a viewer.

View file

@ -0,0 +1,41 @@
name: screenshot-changelog
# Discord trigger B: when a push to main changes committed baseline PNGs, post an
# old | new+boxes | heatmap triptych per changed baseline (ADDED image / REMOVED
# title too). No build, no GPU — it only diffs two git revisions of PNGs that are
# already CI-rendered, so it runs in ~30s. Complements the re-render drift gate in
# wasm-build.yml (which catches un-blessed renders); this is the human-facing feed
# of intentional baseline updates as they land.
on:
push:
branches: [main]
paths:
- 'tests/baseline-screenshots/**.png'
- 'tests/e2e/baseline-screenshots/**.png'
concurrency:
group: screenshot-changelog-${{ github.ref }}
cancel-in-progress: false
jobs:
changelog:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
# HEAD^ is needed to diff the tip commit's baseline changes. (A push of
# multiple commits reports only the tip commit's baseline diff — fine for
# a changelog; baseline refreshes are single commits in practice.)
fetch-depth: 2
- uses: actions/setup-node@v4
with:
node-version: 20
- name: Install screenshot tooling deps
working-directory: tests
run: npm ci
- name: Post baseline changelog to Discord
working-directory: tests
env:
DISCORD_WEBHOOK_URL: ${{ secrets.DISCORD_WEBHOOK_URL }}
run: npm run screenshots:changelog

View file

@ -42,6 +42,13 @@ on:
description: "Upload the publishable output/ subset as the 'wasm-output' artifact"
type: boolean
default: false
secrets:
# Declared so this reusable workflow may reference ${{ secrets.DISCORD_WEBHOOK_URL }}
# (an undeclared secret reference is a workflow startup failure). ci-ubicloud.yml
# passes it via `secrets: inherit`; release.yml doesn't (required: false) → the
# screenshot/perf report step just no-ops there.
DISCORD_WEBHOOK_URL:
required: false
jobs:
build-and-test:
@ -295,11 +302,13 @@ jobs:
run: npm run setup:kicad
- name: wxWidgets e2e (npm run test)
id: wx_e2e
if: inputs.run_tests
working-directory: tests
run: npm run test
- name: KiCad e2e (npm run test:kicad:ci)
id: kicad_e2e
if: inputs.run_tests
working-directory: tests
run: xvfb-run -a npm run test:kicad:ci
@ -314,6 +323,24 @@ jobs:
working-directory: tests
run: xvfb-run -a npm run test:perf
# Screenshot drift gate + always-on Discord report (screenshots + perf).
# Report-only during rollout: compare.ts exits 0 without --fail-on-change and
# the step is continue-on-error, so it never blocks the build — flip to gating
# once the per-engine floors are calibrated (tests/tools/screenshots/config.ts,
# seeded by `npm run screenshots:noise`). post-discord posts ONLY on push to
# main and no-ops without DISCORD_WEBHOOK_URL (inert on PRs/forks). It reads the
# already-produced test-results (screenshots + perf-*.json) — no extra build.
- name: Screenshot compare + Discord report (screenshots + perf)
if: always() && inputs.run_tests
continue-on-error: true
working-directory: tests
env:
DISCORD_WEBHOOK_URL: ${{ secrets.DISCORD_WEBHOOK_URL }}
GH_TOKEN: ${{ github.token }}
run: |
npm run screenshots:check
npm run screenshots:report -- --e2e ${{ (steps.wx_e2e.outcome == 'success' && steps.kicad_e2e.outcome == 'success') && 'pass' || 'fail' }}
- name: Upload test logs & screenshots
if: always() && inputs.run_tests
uses: actions/upload-artifact@v4