pcbjam/tests/tools/screenshots/config.ts

125 lines
5.6 KiB
TypeScript
Raw Normal View History

feat(tests): screenshot regression + Discord review, perf-tracked New tooling in tests/tools/screenshots/ (TypeScript via tsx): - compare.ts: one pixelmatch engine (AA-excluded), connected-component "where to look" boxes, old|new+boxes|heatmap triptych, per-engine floors. - promote.ts: churn-free updater — overwrite a baseline only when decoded pixels differ beyond the floor, copying CI bytes verbatim (no re-encode churn); pulls a CI run via `gh run download` or a local --from dir. - post-discord.ts: always-on CI-on-main report (SHA + e2e status + the track-only runtime-perf table), then screenshot triptychs, batched + size-capped + flood-collapsed + 429-aware. - perf-report.ts: perf table with Δ vs the previous main run (via gh). - changelog.ts: no-build git-history baseline differ (Discord trigger B). - noise.ts / gen-manifest.ts: calibration + manifest generation. CI wiring: - wasm-build.yml: post-test step runs the gate + report on the already- produced test-results (no extra build); report-only (continue-on-error), posts only on push to main, inert without DISCORD_WEBHOOK_URL. - ci-ubicloud.yml: secrets: inherit (pass the webhook through). - screenshot-changelog.yml: ~30s no-build changelog on baseline changes. screenshot-manifest.json: canonical 354-name set + best-effort engine tags (313 chromium-swiftshader / 41 firefox-llvmpipe). Normalize scale:'device'->'css' across 18 spec files (no-op at CI DSF=1) so committed baselines are uniformly css-scaled. Design: CI's Linux render is the single source of truth; no pinned container (accept rare env drift -> re-promote); dev commits via promote. Replaces the byte-cmp compare-screenshots.sh + file-size-proxy update-baseline-screenshots.sh (kept for now until the first re-baseline). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 18:01:15 +02:00
/**
* Central config for the screenshot regression + review tooling.
*
* The comparison engine (compare.ts), the churn-free updater (promote.ts) and
* the Discord reporter (post-discord.ts) all read their knobs from here so
* there is exactly one place to tune thresholds and paths.
*
* Paths are relative to the `tests/` directory (that is the working directory
* the npm scripts and CI steps run from).
*/
/**
* Committed baseline directory. Single dir on purpose: `e2e/baseline-screenshots/`
* used to also hold a few names (grid-tab-final, wxgrid-*) that ALSO live here with
* different bytes — so they got silently shadowed. Consolidated to one dir.
*/
export const BASELINE_DIRS = ['baseline-screenshots'] as const;
feat(tests): screenshot regression + Discord review, perf-tracked New tooling in tests/tools/screenshots/ (TypeScript via tsx): - compare.ts: one pixelmatch engine (AA-excluded), connected-component "where to look" boxes, old|new+boxes|heatmap triptych, per-engine floors. - promote.ts: churn-free updater — overwrite a baseline only when decoded pixels differ beyond the floor, copying CI bytes verbatim (no re-encode churn); pulls a CI run via `gh run download` or a local --from dir. - post-discord.ts: always-on CI-on-main report (SHA + e2e status + the track-only runtime-perf table), then screenshot triptychs, batched + size-capped + flood-collapsed + 429-aware. - perf-report.ts: perf table with Δ vs the previous main run (via gh). - changelog.ts: no-build git-history baseline differ (Discord trigger B). - noise.ts / gen-manifest.ts: calibration + manifest generation. CI wiring: - wasm-build.yml: post-test step runs the gate + report on the already- produced test-results (no extra build); report-only (continue-on-error), posts only on push to main, inert without DISCORD_WEBHOOK_URL. - ci-ubicloud.yml: secrets: inherit (pass the webhook through). - screenshot-changelog.yml: ~30s no-build changelog on baseline changes. screenshot-manifest.json: canonical 354-name set + best-effort engine tags (313 chromium-swiftshader / 41 firefox-llvmpipe). Normalize scale:'device'->'css' across 18 spec files (no-op at CI DSF=1) so committed baselines are uniformly css-scaled. Design: CI's Linux render is the single source of truth; no pinned container (accept rare env drift -> re-promote); dev commits via promote. Replaces the byte-cmp compare-screenshots.sh + file-size-proxy update-baseline-screenshots.sh (kept for now until the first re-baseline). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 18:01:15 +02:00
/** Where Playwright writes the current run's screenshots (gitignored). */
export const RESULTS_DIR = 'test-results';
/** Where compare.ts writes diff/heatmap/triptych artifacts (gitignored). */
export const DIFF_OUT_DIR = 'test-results/screenshot-diff';
/** The manifest that records every expected screenshot + which engine renders it. */
export const MANIFEST_PATH = 'screenshot-manifest.json';
/**
* pixelmatch per-pixel settings.
* - `threshold` is the YIQ perceptual distance (0..1) below which two pixels
* are considered equal. 0.1 tolerates gamma/AA jitter but catches real colour
* change.
* - `includeAA: false` (the pixelmatch default) means anti-aliased edge pixels
* are DETECTED AND IGNORED — exactly the sub-pixel/AA noise the old
* `maxDiff>16` counter was dominated by (see screenshot-compare.ts:36-43).
*/
export const PIXELMATCH = { threshold: 0.1, includeAA: false } as const;
/** Colour pixelmatch paints a real (non-AA) diff pixel with — the mask reads this back. */
export const DIFF_COLOR: [number, number, number] = [255, 0, 0];
/** Connected-component clustering ("where to look") parameters. */
export const CLUSTER = {
dilate: 2, // grow the mask so fragmented glyph pixels merge into one box
minBoxArea: 16, // drop specks smaller than this (px²)
maxBoxes: 6, // draw at most this many (largest-first) red boxes
boxColor: [255, 0, 0] as [number, number, number],
} as const;
/** Horizontal montage layout for the old | new+boxes | heatmap triptych. */
export const TRIPTYCH = {
gap: 8,
bg: [24, 24, 24, 255] as [number, number, number, number],
padFill: [40, 0, 40, 255] as [number, number, number, number], // magenta pad on dim-mismatch
} as const;
/**
* Per-engine verdict floors. A screenshot is CHANGED when its AA-excluded
* changed-pixel ratio exceeds `changedRatio`. `meanChannelGuard` is recorded
* for the drift-vs-regression heuristic (broad + low-intensity ⇒ environment
* drift, not a localized regression), not for the primary verdict.
*
* Set to 0.5% — the re-baseline run showed 355/356 images intra-CI-stable well
* under this, but a couple of runs since had sub-1% inter-run flakiness, so 0.5%
* gives headroom while still catching real localized changes. (`npm run
* screenshots:noise` on two CI renders can refine per-engine numbers later.)
feat(tests): screenshot regression + Discord review, perf-tracked New tooling in tests/tools/screenshots/ (TypeScript via tsx): - compare.ts: one pixelmatch engine (AA-excluded), connected-component "where to look" boxes, old|new+boxes|heatmap triptych, per-engine floors. - promote.ts: churn-free updater — overwrite a baseline only when decoded pixels differ beyond the floor, copying CI bytes verbatim (no re-encode churn); pulls a CI run via `gh run download` or a local --from dir. - post-discord.ts: always-on CI-on-main report (SHA + e2e status + the track-only runtime-perf table), then screenshot triptychs, batched + size-capped + flood-collapsed + 429-aware. - perf-report.ts: perf table with Δ vs the previous main run (via gh). - changelog.ts: no-build git-history baseline differ (Discord trigger B). - noise.ts / gen-manifest.ts: calibration + manifest generation. CI wiring: - wasm-build.yml: post-test step runs the gate + report on the already- produced test-results (no extra build); report-only (continue-on-error), posts only on push to main, inert without DISCORD_WEBHOOK_URL. - ci-ubicloud.yml: secrets: inherit (pass the webhook through). - screenshot-changelog.yml: ~30s no-build changelog on baseline changes. screenshot-manifest.json: canonical 354-name set + best-effort engine tags (313 chromium-swiftshader / 41 firefox-llvmpipe). Normalize scale:'device'->'css' across 18 spec files (no-op at CI DSF=1) so committed baselines are uniformly css-scaled. Design: CI's Linux render is the single source of truth; no pinned container (accept rare env drift -> re-promote); dev commits via promote. Replaces the byte-cmp compare-screenshots.sh + file-size-proxy update-baseline-screenshots.sh (kept for now until the first re-baseline). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 18:01:15 +02:00
*/
export type EngineFloor = { changedRatio: number; meanChannelGuard: number };
export const FLOORS: Record<string, EngineFloor> = {
'firefox-llvmpipe': { changedRatio: 0.005, meanChannelGuard: 2.0 },
'chromium-swiftshader': { changedRatio: 0.005, meanChannelGuard: 2.0 },
default: { changedRatio: 0.005, meanChannelGuard: 2.0 },
feat(tests): screenshot regression + Discord review, perf-tracked New tooling in tests/tools/screenshots/ (TypeScript via tsx): - compare.ts: one pixelmatch engine (AA-excluded), connected-component "where to look" boxes, old|new+boxes|heatmap triptych, per-engine floors. - promote.ts: churn-free updater — overwrite a baseline only when decoded pixels differ beyond the floor, copying CI bytes verbatim (no re-encode churn); pulls a CI run via `gh run download` or a local --from dir. - post-discord.ts: always-on CI-on-main report (SHA + e2e status + the track-only runtime-perf table), then screenshot triptychs, batched + size-capped + flood-collapsed + 429-aware. - perf-report.ts: perf table with Δ vs the previous main run (via gh). - changelog.ts: no-build git-history baseline differ (Discord trigger B). - noise.ts / gen-manifest.ts: calibration + manifest generation. CI wiring: - wasm-build.yml: post-test step runs the gate + report on the already- produced test-results (no extra build); report-only (continue-on-error), posts only on push to main, inert without DISCORD_WEBHOOK_URL. - ci-ubicloud.yml: secrets: inherit (pass the webhook through). - screenshot-changelog.yml: ~30s no-build changelog on baseline changes. screenshot-manifest.json: canonical 354-name set + best-effort engine tags (313 chromium-swiftshader / 41 firefox-llvmpipe). Normalize scale:'device'->'css' across 18 spec files (no-op at CI DSF=1) so committed baselines are uniformly css-scaled. Design: CI's Linux render is the single source of truth; no pinned container (accept rare env drift -> re-promote); dev commits via promote. Replaces the byte-cmp compare-screenshots.sh + file-size-proxy update-baseline-screenshots.sh (kept for now until the first re-baseline). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 18:01:15 +02:00
};
/**
* Optional per-file rectangles to ignore before diffing (e.g. a live clock).
* Keyed by screenshot filename. Empty for now.
*/
export const IGNORE_REGIONS: Record<string, Array<{ x: number; y: number; width: number; height: number }>> = {};
/**
* Screenshots excluded from comparison entirely (not compared, not counted as
* changed/added/removed, not put in the manifest). For nondeterministic captures
* that can't be a stable baseline — e.g. `retinascale-01-loaded` is a `fullPage`
* HiDPI test whose captured height + DPR scaling vary run-to-run (~60% inter-run
* diff observed), a flaky test rather than render noise.
*/
export const IGNORE_SCREENSHOTS = new Set<string>(['retinascale-01-loaded.png']);
/** True if a screenshot is excluded from comparison. */
export function isIgnored(name: string): boolean {
return IGNORE_SCREENSHOTS.has(name);
}
/** Bottom caption strip baked onto each posted composite (status + name + spec). */
export const LABEL = {
maxScale: 3, // bitmap-font scale; auto-fit picks the largest that fits the width
vpad: 5,
hpad: 8,
text: [255, 255, 255] as [number, number, number], // white
colors: {
added: [46, 125, 50] as [number, number, number], // green
removed: [198, 40, 40] as [number, number, number], // red
changed: [239, 108, 0] as [number, number, number], // orange
unchanged: [69, 90, 100] as [number, number, number], // blue-grey (review-only artifacts)
},
};
export type LabelStatus = 'added' | 'removed' | 'changed' | 'unchanged';
/** Caption text: `CHANGED name.png · kicad/pcbnew.spec.ts` (spec omitted if unknown). */
export function labelText(status: LabelStatus, name: string, spec: string | null): string {
const s = status.toUpperCase();
return spec ? `${s} ${name} · ${spec}` : `${s} ${name}`;
}
feat(tests): screenshot regression + Discord review, perf-tracked New tooling in tests/tools/screenshots/ (TypeScript via tsx): - compare.ts: one pixelmatch engine (AA-excluded), connected-component "where to look" boxes, old|new+boxes|heatmap triptych, per-engine floors. - promote.ts: churn-free updater — overwrite a baseline only when decoded pixels differ beyond the floor, copying CI bytes verbatim (no re-encode churn); pulls a CI run via `gh run download` or a local --from dir. - post-discord.ts: always-on CI-on-main report (SHA + e2e status + the track-only runtime-perf table), then screenshot triptychs, batched + size-capped + flood-collapsed + 429-aware. - perf-report.ts: perf table with Δ vs the previous main run (via gh). - changelog.ts: no-build git-history baseline differ (Discord trigger B). - noise.ts / gen-manifest.ts: calibration + manifest generation. CI wiring: - wasm-build.yml: post-test step runs the gate + report on the already- produced test-results (no extra build); report-only (continue-on-error), posts only on push to main, inert without DISCORD_WEBHOOK_URL. - ci-ubicloud.yml: secrets: inherit (pass the webhook through). - screenshot-changelog.yml: ~30s no-build changelog on baseline changes. screenshot-manifest.json: canonical 354-name set + best-effort engine tags (313 chromium-swiftshader / 41 firefox-llvmpipe). Normalize scale:'device'->'css' across 18 spec files (no-op at CI DSF=1) so committed baselines are uniformly css-scaled. Design: CI's Linux render is the single source of truth; no pinned container (accept rare env drift -> re-promote); dev commits via promote. Replaces the byte-cmp compare-screenshots.sh + file-size-proxy update-baseline-screenshots.sh (kept for now until the first re-baseline). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 18:01:15 +02:00
export type ManifestEntry = { name: string; engine: string };
export type Manifest = { screenshots: ManifestEntry[] };
/** Resolve the verdict floor for a screenshot via the manifest's engine tag. */
export function floorFor(name: string, manifest?: Manifest): EngineFloor {
const engine = manifest?.screenshots.find((e) => e.name === name)?.engine;
return (engine && FLOORS[engine]) || FLOORS.default;
}