feat(tests): screenshot regression + Discord review, perf-tracked

New tooling in tests/tools/screenshots/ (TypeScript via tsx):
- compare.ts: one pixelmatch engine (AA-excluded), connected-component
  "where to look" boxes, old|new+boxes|heatmap triptych, per-engine floors.
- promote.ts: churn-free updater — overwrite a baseline only when decoded
  pixels differ beyond the floor, copying CI bytes verbatim (no re-encode
  churn); pulls a CI run via `gh run download` or a local --from dir.
- post-discord.ts: always-on CI-on-main report (SHA + e2e status + the
  track-only runtime-perf table), then screenshot triptychs, batched +
  size-capped + flood-collapsed + 429-aware.
- perf-report.ts: perf table with Δ vs the previous main run (via gh).
- changelog.ts: no-build git-history baseline differ (Discord trigger B).
- noise.ts / gen-manifest.ts: calibration + manifest generation.

CI wiring:
- wasm-build.yml: post-test step runs the gate + report on the already-
  produced test-results (no extra build); report-only (continue-on-error),
  posts only on push to main, inert without DISCORD_WEBHOOK_URL.
- ci-ubicloud.yml: secrets: inherit (pass the webhook through).
- screenshot-changelog.yml: ~30s no-build changelog on baseline changes.

screenshot-manifest.json: canonical 354-name set + best-effort engine tags
(313 chromium-swiftshader / 41 firefox-llvmpipe).

Normalize scale:'device'->'css' across 18 spec files (no-op at CI DSF=1)
so committed baselines are uniformly css-scaled.

Design: CI's Linux render is the single source of truth; no pinned
container (accept rare env drift -> re-promote); dev commits via promote.
Replaces the byte-cmp compare-screenshots.sh + file-size-proxy
update-baseline-screenshots.sh (kept for now until the first re-baseline).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Viktor Vaczi 2026-07-01 18:01:15 +02:00
commit 13551f4f22
36 changed files with 3186 additions and 63 deletions

View file

@ -229,7 +229,7 @@ async function completeWizard(page: Page): Promise<void> {
}, null, { timeout: 150000 });
await page.waitForTimeout(2000);
await page.screenshot({ path: 'test-results/eeschema-wizard-00-initial.png', scale: 'device' });
await page.screenshot({ path: 'test-results/eeschema-wizard-00-initial.png', scale: 'css' });
for (let i = 1; i <= 10; i++) {
let clicked = await clickByLabel(page, 'Next >');
@ -241,7 +241,7 @@ async function completeWizard(page: Page): Promise<void> {
await page.waitForTimeout(500);
await page.screenshot({
path: `test-results/eeschema-wizard-${String(i).padStart(2, '0')}-finish.png`,
scale: 'device'
scale: 'css'
});
}
@ -251,7 +251,7 @@ async function completeWizard(page: Page): Promise<void> {
await page.waitForTimeout(500);
await page.screenshot({
path: `test-results/eeschema-wizard-${String(i).padStart(2, '0')}.png`,
scale: 'device'
scale: 'css'
});
}
@ -313,7 +313,7 @@ test.describe('Eeschema WASM', () => {
path: 'test-results/eeschema-loaded-css.png',
scale: 'css'
});
await page.screenshot({ path: 'test-results/eeschema-loaded.png', scale: 'device' });
await page.screenshot({ path: 'test-results/eeschema-loaded.png', scale: 'css' });
const canvasCount = await page.locator('canvas').count();
expect(canvasCount).toBeGreaterThan(0);
@ -408,7 +408,7 @@ test.describe('Eeschema WASM', () => {
await page.screenshot({
path: 'test-results/eeschema-draw-wires-00-before-tool-click.png',
scale: 'device'
scale: 'css'
});
expect(await clickByTooltip(page, 'Draw Wires', { elementType: 'tool' })).toBe(true);
@ -429,7 +429,7 @@ test.describe('Eeschema WASM', () => {
const afterToolClick = await page.screenshot({
path: 'test-results/eeschema-draw-wires-01-after-click.png',
scale: 'device'
scale: 'css'
});
const glCanvasId = await page.evaluate(() => {
@ -475,7 +475,7 @@ test.describe('Eeschema WASM', () => {
const afterDrawing = await page.screenshot({
path: 'test-results/eeschema-draw-wires-02-after-drawing.png',
scale: 'device'
scale: 'css'
});
const diffRegion: DiffRegion = {