feat(tests): screenshot regression + Discord review, perf-tracked
New tooling in tests/tools/screenshots/ (TypeScript via tsx): - compare.ts: one pixelmatch engine (AA-excluded), connected-component "where to look" boxes, old|new+boxes|heatmap triptych, per-engine floors. - promote.ts: churn-free updater — overwrite a baseline only when decoded pixels differ beyond the floor, copying CI bytes verbatim (no re-encode churn); pulls a CI run via `gh run download` or a local --from dir. - post-discord.ts: always-on CI-on-main report (SHA + e2e status + the track-only runtime-perf table), then screenshot triptychs, batched + size-capped + flood-collapsed + 429-aware. - perf-report.ts: perf table with Δ vs the previous main run (via gh). - changelog.ts: no-build git-history baseline differ (Discord trigger B). - noise.ts / gen-manifest.ts: calibration + manifest generation. CI wiring: - wasm-build.yml: post-test step runs the gate + report on the already- produced test-results (no extra build); report-only (continue-on-error), posts only on push to main, inert without DISCORD_WEBHOOK_URL. - ci-ubicloud.yml: secrets: inherit (pass the webhook through). - screenshot-changelog.yml: ~30s no-build changelog on baseline changes. screenshot-manifest.json: canonical 354-name set + best-effort engine tags (313 chromium-swiftshader / 41 firefox-llvmpipe). Normalize scale:'device'->'css' across 18 spec files (no-op at CI DSF=1) so committed baselines are uniformly css-scaled. Design: CI's Linux render is the single source of truth; no pinned container (accept rare env drift -> re-promote); dev commits via promote. Replaces the byte-cmp compare-screenshots.sh + file-size-proxy update-baseline-screenshots.sh (kept for now until the first re-baseline). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Viktor Vaczi
parent
05127af431
commit
13551f4f22
36 changed files with 3186 additions and 63 deletions
40
tests/tools/screenshots/README.md
Normal file
40
tests/tools/screenshots/README.md
Normal file
|
|
@ -0,0 +1,40 @@
|
|||
# Screenshot regression + Discord review tooling
|
||||
|
||||
One comparison engine + a churn-free updater + a Discord reporter for the e2e
|
||||
screenshots. Design and rationale: `~/.claude/plans/…snowglobe.md` (or ask).
|
||||
|
||||
**Source of truth = CI's Linux render.** The dev never authors baselines on the
|
||||
Mac (Mac fonts/GL ≠ CI). Instead, CI renders on every push; when a render change
|
||||
is intentional you *promote* CI's artifact into the committed baselines. The
|
||||
environment isn't pinned — if the host's Mesa/fonts drift, the gate lights up in
|
||||
Discord and you just re-promote (broad + low-intensity change ⇒ likely drift).
|
||||
|
||||
## Files
|
||||
- `config.ts` — thresholds, baseline dirs, per-engine floors (calibrate!), clustering knobs.
|
||||
- `image-ops.ts` — PNG load/save, pixelmatch diff (AA-excluded), connected-component boxes, triptych compositing, size-cap resize.
|
||||
- `compare.ts` — the engine: classify baselines vs `test-results/` → `test-results/screenshot-diff/report.json` + triptych/heatmap PNGs. `--pair` diffs two files.
|
||||
- `promote.ts` — churn-free updater: pull a CI run's shots (`gh run download`) or `--from DIR`; overwrite a baseline only when pixels differ beyond the floor (verbatim bytes, no re-encode) → no git churn.
|
||||
- `perf-report.ts` — renders the track-only runtime-perf table (loadMs/openMs/FPS) with Δ vs the previous main run (fetched via `gh`).
|
||||
- `post-discord.ts` — the always-on CI-on-main report: SHA + e2e status + perf table, then screenshot triptychs (batched, size-capped, flood-collapsed).
|
||||
- `changelog.ts` — Discord trigger B: git-history diff of committed baseline PNGs (no build/GPU).
|
||||
- `noise.ts` — calibration: diff two identical-input renders → per-engine noise floor.
|
||||
- `gen-manifest.ts` — regenerate `screenshot-manifest.json` (canonical name list + best-effort engine tag) by scanning the committed baselines + specs.
|
||||
|
||||
## npm scripts (run from `tests/`)
|
||||
```
|
||||
npm run screenshots:check # gate: baselines vs test-results → report.json (exit 0; add --fail-on-change to gate)
|
||||
npm run screenshots:promote -- --run <ci-run-id> # churn-free re-baseline from a CI run (or --from DIR)
|
||||
npm run screenshots:report -- --e2e pass # post the CI report to Discord (main+push only; needs DISCORD_WEBHOOK_URL)
|
||||
npm run screenshots:changelog # post the baseline changelog (main+push only)
|
||||
npm run screenshots:noise -- run1/ run2/ # calibrate floors
|
||||
npm run screenshots:manifest # regenerate the manifest (--check to verify it's fresh)
|
||||
```
|
||||
|
||||
## Activation checklist
|
||||
- [x] `screenshot-manifest.json` generated (name list authoritative; engine tags best-effort until calibration).
|
||||
- [x] `scale:'device'`→`'css'` normalized (no-op at CI's DSF=1).
|
||||
1. Add the `DISCORD_WEBHOOK_URL` repo secret — until then everything is inert.
|
||||
2. Calibrate: run the suite twice in CI, `screenshots:noise` the two dirs, set `FLOORS` in `config.ts`.
|
||||
3. First re-baseline: `promote` a clean CI run's render, commit (expect a big, one-time chrome-font diff vs the Mac baselines).
|
||||
4. Delete the old `scripts/{compare,update-baseline}-screenshots.sh`.
|
||||
5. Once floors are proven stable, flip the gate to `--fail-on-change`.
|
||||
155
tests/tools/screenshots/changelog.ts
Normal file
155
tests/tools/screenshots/changelog.ts
Normal file
|
|
@ -0,0 +1,155 @@
|
|||
/**
|
||||
* Baseline changelog (Discord trigger B) — no build, no GPU.
|
||||
*
|
||||
* On push to main, diff the committed baseline PNGs between two commits and post
|
||||
* a "Baseline changelog" to Discord: a triptych (old | new+boxes | heatmap) for
|
||||
* each CHANGED file, the image for each ADDED, and a titled line for each REMOVED.
|
||||
* "added / removed" only mean anything as a git-history diff, which is why this is
|
||||
* separate from the re-render drift gate.
|
||||
*
|
||||
* CLI (from tests/):
|
||||
* tsx tools/screenshots/changelog.ts [--base REV] [--head REV] [--dry-run] [--force]
|
||||
*/
|
||||
import * as fs from 'fs';
|
||||
import * as os from 'os';
|
||||
import * as path from 'path';
|
||||
import { execFileSync } from 'child_process';
|
||||
import { BASELINE_DIRS, DIFF_OUT_DIR, floorFor } from './config';
|
||||
import { comparePair, type Report } from './compare';
|
||||
import { loadPng, savePng } from './image-ops';
|
||||
import { buildAttachments, paginate, postMessage } from './post-discord';
|
||||
|
||||
function git(root: string, args: string[]): string {
|
||||
return execFileSync('git', ['-C', root, ...args], { encoding: 'utf8' }).trim();
|
||||
}
|
||||
|
||||
function gitBlob(root: string, rev: string, repoPath: string): Buffer | null {
|
||||
try {
|
||||
return execFileSync('git', ['-C', root, 'show', `${rev}:${repoPath}`], { maxBuffer: 1 << 28 });
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function parseArgs(argv: string[]): Record<string, string | boolean> {
|
||||
const out: Record<string, string | boolean> = {};
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
const a = argv[i];
|
||||
if (a === '--dry-run') out.dryRun = true;
|
||||
else if (a === '--force') out.force = true;
|
||||
else if (a === '--base') out.base = argv[++i];
|
||||
else if (a === '--head') out.head = argv[++i];
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
async function main(): Promise<void> {
|
||||
const args = parseArgs(process.argv.slice(2));
|
||||
const cwd = process.cwd();
|
||||
const repoRoot = git(cwd, ['rev-parse', '--show-toplevel']);
|
||||
const head = (args.head as string) || 'HEAD';
|
||||
let base = (args.base as string) || '';
|
||||
if (!base) {
|
||||
try {
|
||||
base = git(cwd, ['rev-parse', `${head}^`]);
|
||||
} catch {
|
||||
console.log('[changelog] no parent commit — nothing to diff');
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
// Repo-relative baseline dir prefixes (e.g. tests/baseline-screenshots).
|
||||
const prefixes = BASELINE_DIRS.map((d) => path.relative(repoRoot, path.join(cwd, d)));
|
||||
const diff = git(repoRoot, ['diff', '--name-status', base, head, '--', ...prefixes]);
|
||||
if (!diff) {
|
||||
console.log('[changelog] no baseline changes between', base.slice(0, 7), 'and', head);
|
||||
return;
|
||||
}
|
||||
|
||||
const outDir = path.join(cwd, DIFF_OUT_DIR);
|
||||
fs.mkdirSync(outDir, { recursive: true });
|
||||
const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'changelog-'));
|
||||
const report: Report = {
|
||||
generatedFor: process.env.GITHUB_SHA || head,
|
||||
changed: [],
|
||||
added: [],
|
||||
removed: [],
|
||||
unchangedCount: 0,
|
||||
driftLikely: false,
|
||||
};
|
||||
|
||||
for (const line of diff.split('\n')) {
|
||||
const parts = line.split('\t');
|
||||
const status = parts[0][0]; // M/A/D/R
|
||||
const repoPath = parts[status === 'R' ? 2 : 1];
|
||||
const oldPath = status === 'R' ? parts[1] : repoPath;
|
||||
const name = path.basename(repoPath);
|
||||
|
||||
if (status === 'A' || status === 'R') {
|
||||
const buf = gitBlob(repoRoot, head, repoPath);
|
||||
if (buf) {
|
||||
const rel = path.join(DIFF_OUT_DIR, `${name}.added.png`);
|
||||
fs.writeFileSync(path.join(cwd, rel), buf);
|
||||
report.added.push({ name, image: rel });
|
||||
}
|
||||
if (status === 'R') report.removed.push({ name: path.basename(oldPath) });
|
||||
} else if (status === 'D') {
|
||||
report.removed.push({ name });
|
||||
} else {
|
||||
// Modified: triptych old vs new.
|
||||
const oldBuf = gitBlob(repoRoot, base, repoPath);
|
||||
const newBuf = gitBlob(repoRoot, head, repoPath);
|
||||
if (!oldBuf || !newBuf) continue;
|
||||
const oldFile = path.join(tmp, `old-${name}`);
|
||||
const newFile = path.join(tmp, `new-${name}`);
|
||||
fs.writeFileSync(oldFile, oldBuf);
|
||||
fs.writeFileSync(newFile, newBuf);
|
||||
const { result, heatmap, triptych } = comparePair(loadPng(oldFile), loadPng(newFile), name, floorFor(name));
|
||||
const triptychRel = path.join(DIFF_OUT_DIR, `${name}.triptych.png`);
|
||||
const heatmapRel = path.join(DIFF_OUT_DIR, `${name}.heatmap.png`);
|
||||
savePng(path.join(cwd, triptychRel), triptych);
|
||||
savePng(path.join(cwd, heatmapRel), heatmap);
|
||||
report.changed.push({ ...result, triptych: triptychRel, heatmap: heatmapRel });
|
||||
}
|
||||
}
|
||||
report.changed.sort((a, b) => b.changedRatio - a.changedRatio);
|
||||
|
||||
const sha7 = (process.env.GITHUB_SHA || head).slice(0, 7);
|
||||
let subject = '';
|
||||
try {
|
||||
subject = git(cwd, ['log', '-1', '--pretty=%s', head]);
|
||||
} catch { /* ignore */ }
|
||||
const header =
|
||||
`🗂️ **Baseline changelog** · \`${sha7}\`` +
|
||||
(subject ? `\n> ${subject}` : '') +
|
||||
`\n${report.changed.length} changed, ${report.added.length} added, ${report.removed.length} removed` +
|
||||
(report.removed.length ? '\n➖ REMOVED: ' + report.removed.map((r) => `\`${r.name}\``).join(', ') : '');
|
||||
|
||||
const { files, notes } = buildAttachments(cwd, report);
|
||||
const messages = paginate(header + (notes.length ? '\n' + notes.join('\n') : ''), files);
|
||||
|
||||
const isMainPush = process.env.GITHUB_REF === 'refs/heads/main' && process.env.GITHUB_EVENT_NAME === 'push';
|
||||
if (args.dryRun || (!args.force && !isMainPush)) {
|
||||
for (const [i, m] of messages.entries()) {
|
||||
console.log(`--- message ${i + 1}/${messages.length} (${m.files.length} files) ---`);
|
||||
if (m.content) console.log(m.content);
|
||||
for (const f of m.files) console.log(` [attach] ${f.name} (${f.buffer.length} bytes)`);
|
||||
}
|
||||
if (!args.dryRun) console.log('[changelog] not a push to main — not posting');
|
||||
return;
|
||||
}
|
||||
const webhook = process.env.DISCORD_WEBHOOK_URL;
|
||||
if (!webhook) {
|
||||
console.log('[changelog] DISCORD_WEBHOOK_URL unset — skipping');
|
||||
return;
|
||||
}
|
||||
for (const m of messages) await postMessage(webhook, m);
|
||||
console.log(`[changelog] posted ${messages.length} message(s)`);
|
||||
}
|
||||
|
||||
if (require.main === module) {
|
||||
main().catch((e) => {
|
||||
console.error(`[changelog] ${e.message}`);
|
||||
process.exitCode = 1;
|
||||
});
|
||||
}
|
||||
238
tests/tools/screenshots/compare.ts
Normal file
238
tests/tools/screenshots/compare.ts
Normal file
|
|
@ -0,0 +1,238 @@
|
|||
/**
|
||||
* The one screenshot comparison engine + its CLI.
|
||||
*
|
||||
* Run modes (from the `tests/` directory):
|
||||
* tsx tools/screenshots/compare.ts # gate: baselines vs test-results
|
||||
* tsx tools/screenshots/compare.ts --fail-on-change # same, but exit 1 on any change
|
||||
* tsx tools/screenshots/compare.ts --pair OLD NEW --name N --out DIR # diff two files
|
||||
*
|
||||
* The gate mode classifies every screenshot into changed / added / removed /
|
||||
* unchanged, writes per-change triptych + heatmap PNGs and a machine-readable
|
||||
* report.json into DIFF_OUT_DIR, and (unless --fail-on-change) exits 0 so it can
|
||||
* run report-only first. post-discord.ts and the changelog workflow import the
|
||||
* exported helpers rather than re-deriving the diff.
|
||||
*/
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
import { PNG } from 'pngjs';
|
||||
import {
|
||||
BASELINE_DIRS,
|
||||
RESULTS_DIR,
|
||||
DIFF_OUT_DIR,
|
||||
MANIFEST_PATH,
|
||||
type EngineFloor,
|
||||
type Manifest,
|
||||
floorFor,
|
||||
} from './config';
|
||||
import { diffImages, cluster, drawBoxes, composite, loadPng, savePng, type Box } from './image-ops';
|
||||
|
||||
export type PairVerdict = 'unchanged' | 'changed';
|
||||
|
||||
export type PairResult = {
|
||||
name: string;
|
||||
verdict: PairVerdict;
|
||||
dimsMatch: boolean;
|
||||
diffPixels: number;
|
||||
changedRatio: number;
|
||||
meanChannelDiff: number;
|
||||
boxes: Box[];
|
||||
/** heuristic: a broad, low-intensity change smells like host env drift, not a code regression */
|
||||
driftHint: 'regression-like' | 'drift-like' | null;
|
||||
};
|
||||
|
||||
/** Broad + low-intensity ⇒ likely environment drift rather than a localized regression. */
|
||||
function driftHint(changed: boolean, changedRatio: number, meanChannelDiff: number): PairResult['driftHint'] {
|
||||
if (!changed) return null;
|
||||
return changedRatio > 0.05 && meanChannelDiff < 4 ? 'drift-like' : 'regression-like';
|
||||
}
|
||||
|
||||
/** Compare two decoded PNGs; returns the verdict/metrics plus the heatmap and old|new+boxes|heatmap triptych. */
|
||||
export function comparePair(
|
||||
oldPng: PNG,
|
||||
newPng: PNG,
|
||||
name: string,
|
||||
floor: EngineFloor
|
||||
): { result: PairResult; heatmap: PNG; triptych: PNG } {
|
||||
const d = diffImages(oldPng, newPng);
|
||||
const changed = !d.dimsMatch || d.changedRatio > floor.changedRatio;
|
||||
const boxes = changed ? cluster(d.mask, d.width, d.height) : [];
|
||||
const triptych = composite([oldPng, drawBoxes(newPng, boxes), d.heatmap]);
|
||||
return {
|
||||
result: {
|
||||
name,
|
||||
verdict: changed ? 'changed' : 'unchanged',
|
||||
dimsMatch: d.dimsMatch,
|
||||
diffPixels: d.diffPixels,
|
||||
changedRatio: d.changedRatio,
|
||||
meanChannelDiff: d.meanChannelDiff,
|
||||
boxes,
|
||||
driftHint: driftHint(changed, d.changedRatio, d.meanChannelDiff),
|
||||
},
|
||||
heatmap: d.heatmap,
|
||||
triptych,
|
||||
};
|
||||
}
|
||||
|
||||
export type ChangedEntry = PairResult & { triptych: string; heatmap: string };
|
||||
export type Report = {
|
||||
generatedFor: string | null;
|
||||
changed: ChangedEntry[];
|
||||
added: Array<{ name: string; image: string }>;
|
||||
removed: Array<{ name: string }>;
|
||||
unchangedCount: number;
|
||||
/** many changes, mostly drift-like ⇒ probably a host Mesa/font refresh; re-promote rather than debug */
|
||||
driftLikely: boolean;
|
||||
};
|
||||
|
||||
const DRIFT_BULK = 20;
|
||||
|
||||
function listPngs(dir: string): string[] {
|
||||
if (!fs.existsSync(dir)) return [];
|
||||
return fs.readdirSync(dir).filter((f) => f.toLowerCase().endsWith('.png'));
|
||||
}
|
||||
|
||||
/** basename → absolute baseline path (first BASELINE_DIRS entry wins; collisions warned). */
|
||||
function baselineIndex(root: string): Map<string, string> {
|
||||
const index = new Map<string, string>();
|
||||
for (const dir of BASELINE_DIRS) {
|
||||
const abs = path.join(root, dir);
|
||||
for (const name of listPngs(abs)) {
|
||||
if (index.has(name)) {
|
||||
console.warn(`[compare] duplicate baseline name ${name} (${dir} shadowed by earlier dir)`);
|
||||
continue;
|
||||
}
|
||||
index.set(name, path.join(abs, name));
|
||||
}
|
||||
}
|
||||
return index;
|
||||
}
|
||||
|
||||
function loadManifest(root: string): Manifest | undefined {
|
||||
const p = path.join(root, MANIFEST_PATH);
|
||||
if (!fs.existsSync(p)) return undefined;
|
||||
try {
|
||||
return JSON.parse(fs.readFileSync(p, 'utf8')) as Manifest;
|
||||
} catch (e) {
|
||||
console.warn(`[compare] could not parse ${MANIFEST_PATH}: ${(e as Error).message}`);
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
/** Full gate run: classify baselines vs the current run's screenshots. */
|
||||
export function classify(root: string, sha: string | null): Report {
|
||||
const baselines = baselineIndex(root);
|
||||
const resultsDir = path.join(root, RESULTS_DIR);
|
||||
const actuals = new Set(listPngs(resultsDir));
|
||||
const manifest = loadManifest(root);
|
||||
const outDir = path.join(root, DIFF_OUT_DIR);
|
||||
fs.mkdirSync(outDir, { recursive: true });
|
||||
|
||||
const report: Report = {
|
||||
generatedFor: sha,
|
||||
changed: [],
|
||||
added: [],
|
||||
removed: [],
|
||||
unchangedCount: 0,
|
||||
driftLikely: false,
|
||||
};
|
||||
|
||||
// Changed / unchanged / removed: iterate the committed baselines.
|
||||
for (const [name, baselinePath] of baselines) {
|
||||
if (!actuals.has(name)) {
|
||||
// Missing output. Only call it REMOVED when the manifest expects it — otherwise
|
||||
// a flaky/OOM'd/skipped spec that simply didn't write a PNG would masquerade as
|
||||
// an intentional removal. (The stronger "did the spec actually run" cross-check
|
||||
// against the Playwright JSON report lands with the manifest work.)
|
||||
if (manifest?.screenshots.some((e) => e.name === name)) {
|
||||
report.removed.push({ name });
|
||||
}
|
||||
continue;
|
||||
}
|
||||
const { result, heatmap, triptych } = comparePair(
|
||||
loadPng(baselinePath),
|
||||
loadPng(path.join(resultsDir, name)),
|
||||
name,
|
||||
floorFor(name, manifest)
|
||||
);
|
||||
if (result.verdict === 'unchanged') {
|
||||
report.unchangedCount++;
|
||||
continue;
|
||||
}
|
||||
const triptychRel = path.join(DIFF_OUT_DIR, `${name}.triptych.png`);
|
||||
const heatmapRel = path.join(DIFF_OUT_DIR, `${name}.heatmap.png`);
|
||||
savePng(path.join(root, triptychRel), triptych);
|
||||
savePng(path.join(root, heatmapRel), heatmap);
|
||||
report.changed.push({ ...result, triptych: triptychRel, heatmap: heatmapRel });
|
||||
}
|
||||
|
||||
// Added: an actual with no committed baseline.
|
||||
for (const name of actuals) {
|
||||
if (baselines.has(name)) continue;
|
||||
const imageRel = path.join(DIFF_OUT_DIR, `${name}.added.png`);
|
||||
fs.copyFileSync(path.join(resultsDir, name), path.join(root, imageRel));
|
||||
report.added.push({ name, image: imageRel });
|
||||
}
|
||||
|
||||
const driftLike = report.changed.filter((c) => c.driftHint === 'drift-like').length;
|
||||
report.driftLikely = report.changed.length >= DRIFT_BULK && driftLike * 2 >= report.changed.length;
|
||||
|
||||
report.changed.sort((a, b) => b.changedRatio - a.changedRatio);
|
||||
report.added.sort((a, b) => a.name.localeCompare(b.name));
|
||||
report.removed.sort((a, b) => a.name.localeCompare(b.name));
|
||||
return report;
|
||||
}
|
||||
|
||||
function parseArgs(argv: string[]): Record<string, string | boolean> {
|
||||
const out: Record<string, string | boolean> = {};
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
const a = argv[i];
|
||||
if (a === '--fail-on-change') out.failOnChange = true;
|
||||
else if (a === '--pair') {
|
||||
out.oldPath = argv[++i];
|
||||
out.newPath = argv[++i];
|
||||
} else if (a === '--name') out.name = argv[++i];
|
||||
else if (a === '--out') out.out = argv[++i];
|
||||
else if (a === '--sha') out.sha = argv[++i];
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function runPair(args: Record<string, string | boolean>): void {
|
||||
const name = (args.name as string) || 'pair';
|
||||
const outDir = (args.out as string) || path.join(RESULTS_DIR, 'screenshot-diff');
|
||||
fs.mkdirSync(outDir, { recursive: true });
|
||||
const oldPng = fs.existsSync(args.oldPath as string)
|
||||
? loadPng(args.oldPath as string)
|
||||
: new PNG({ width: 1, height: 1 });
|
||||
const newPng = fs.existsSync(args.newPath as string)
|
||||
? loadPng(args.newPath as string)
|
||||
: new PNG({ width: 1, height: 1 });
|
||||
const { result, heatmap, triptych } = comparePair(oldPng, newPng, name, floorFor(name));
|
||||
savePng(path.join(outDir, `${name}.triptych.png`), triptych);
|
||||
savePng(path.join(outDir, `${name}.heatmap.png`), heatmap);
|
||||
process.stdout.write(JSON.stringify(result, null, 2) + '\n');
|
||||
}
|
||||
|
||||
function main(): void {
|
||||
const args = parseArgs(process.argv.slice(2));
|
||||
if (args.oldPath && args.newPath) {
|
||||
runPair(args);
|
||||
return;
|
||||
}
|
||||
const root = process.cwd();
|
||||
const report = classify(root, (args.sha as string) || process.env.GITHUB_SHA || null);
|
||||
fs.writeFileSync(path.join(root, DIFF_OUT_DIR, 'report.json'), JSON.stringify(report, null, 2));
|
||||
const { changed, added, removed, unchangedCount, driftLikely } = report;
|
||||
console.log(
|
||||
`[compare] changed=${changed.length} added=${added.length} removed=${removed.length} ` +
|
||||
`unchanged=${unchangedCount}${driftLikely ? ' (looks like environment drift → re-promote)' : ''}`
|
||||
);
|
||||
for (const c of changed.slice(0, 10)) {
|
||||
console.log(` CHANGED ${c.name} ratio=${(c.changedRatio * 100).toFixed(3)}% ${c.driftHint}`);
|
||||
}
|
||||
if (args.failOnChange && (changed.length || added.length || removed.length)) {
|
||||
process.exitCode = 1;
|
||||
}
|
||||
}
|
||||
|
||||
if (require.main === module) main();
|
||||
84
tests/tools/screenshots/config.ts
Normal file
84
tests/tools/screenshots/config.ts
Normal file
|
|
@ -0,0 +1,84 @@
|
|||
/**
|
||||
* Central config for the screenshot regression + review tooling.
|
||||
*
|
||||
* The comparison engine (compare.ts), the churn-free updater (promote.ts) and
|
||||
* the Discord reporter (post-discord.ts) all read their knobs from here so
|
||||
* there is exactly one place to tune thresholds and paths.
|
||||
*
|
||||
* Paths are relative to the `tests/` directory (that is the working directory
|
||||
* the npm scripts and CI steps run from).
|
||||
*/
|
||||
|
||||
/** Committed baseline directories, scanned in order. Filenames are the keys. */
|
||||
export const BASELINE_DIRS = ['baseline-screenshots', 'e2e/baseline-screenshots'] as const;
|
||||
|
||||
/** Where Playwright writes the current run's screenshots (gitignored). */
|
||||
export const RESULTS_DIR = 'test-results';
|
||||
|
||||
/** Where compare.ts writes diff/heatmap/triptych artifacts (gitignored). */
|
||||
export const DIFF_OUT_DIR = 'test-results/screenshot-diff';
|
||||
|
||||
/** The manifest that records every expected screenshot + which engine renders it. */
|
||||
export const MANIFEST_PATH = 'screenshot-manifest.json';
|
||||
|
||||
/**
|
||||
* pixelmatch per-pixel settings.
|
||||
* - `threshold` is the YIQ perceptual distance (0..1) below which two pixels
|
||||
* are considered equal. 0.1 tolerates gamma/AA jitter but catches real colour
|
||||
* change.
|
||||
* - `includeAA: false` (the pixelmatch default) means anti-aliased edge pixels
|
||||
* are DETECTED AND IGNORED — exactly the sub-pixel/AA noise the old
|
||||
* `maxDiff>16` counter was dominated by (see screenshot-compare.ts:36-43).
|
||||
*/
|
||||
export const PIXELMATCH = { threshold: 0.1, includeAA: false } as const;
|
||||
|
||||
/** Colour pixelmatch paints a real (non-AA) diff pixel with — the mask reads this back. */
|
||||
export const DIFF_COLOR: [number, number, number] = [255, 0, 0];
|
||||
|
||||
/** Connected-component clustering ("where to look") parameters. */
|
||||
export const CLUSTER = {
|
||||
dilate: 2, // grow the mask so fragmented glyph pixels merge into one box
|
||||
minBoxArea: 16, // drop specks smaller than this (px²)
|
||||
maxBoxes: 6, // draw at most this many (largest-first) red boxes
|
||||
boxColor: [255, 0, 0] as [number, number, number],
|
||||
} as const;
|
||||
|
||||
/** Horizontal montage layout for the old | new+boxes | heatmap triptych. */
|
||||
export const TRIPTYCH = {
|
||||
gap: 8,
|
||||
bg: [24, 24, 24, 255] as [number, number, number, number],
|
||||
padFill: [40, 0, 40, 255] as [number, number, number, number], // magenta pad on dim-mismatch
|
||||
} as const;
|
||||
|
||||
/**
|
||||
* Per-engine verdict floors. A screenshot is CHANGED when its AA-excluded
|
||||
* changed-pixel ratio exceeds `changedRatio`. `meanChannelGuard` is recorded
|
||||
* for the drift-vs-regression heuristic (broad + low-intensity ⇒ environment
|
||||
* drift, not a localized regression), not for the primary verdict.
|
||||
*
|
||||
* NOTE: these are PLACEHOLDERS. `npm run screenshots:noise` renders the suite
|
||||
* twice on the CI host and prints the real intra-CI floor per engine; set
|
||||
* `changedRatio ≈ measured × 3` from that and commit the numbers here.
|
||||
*/
|
||||
export type EngineFloor = { changedRatio: number; meanChannelGuard: number };
|
||||
|
||||
export const FLOORS: Record<string, EngineFloor> = {
|
||||
'firefox-llvmpipe': { changedRatio: 0.002, meanChannelGuard: 2.0 },
|
||||
'chromium-swiftshader': { changedRatio: 0.002, meanChannelGuard: 2.0 },
|
||||
default: { changedRatio: 0.002, meanChannelGuard: 2.0 },
|
||||
};
|
||||
|
||||
/**
|
||||
* Optional per-file rectangles to ignore before diffing (e.g. a live clock).
|
||||
* Keyed by screenshot filename. Empty for now.
|
||||
*/
|
||||
export const IGNORE_REGIONS: Record<string, Array<{ x: number; y: number; width: number; height: number }>> = {};
|
||||
|
||||
export type ManifestEntry = { name: string; engine: string };
|
||||
export type Manifest = { screenshots: ManifestEntry[] };
|
||||
|
||||
/** Resolve the verdict floor for a screenshot via the manifest's engine tag. */
|
||||
export function floorFor(name: string, manifest?: Manifest): EngineFloor {
|
||||
const engine = manifest?.screenshots.find((e) => e.name === name)?.engine;
|
||||
return (engine && FLOORS[engine]) || FLOORS.default;
|
||||
}
|
||||
124
tests/tools/screenshots/gen-manifest.ts
Normal file
124
tests/tools/screenshots/gen-manifest.ts
Normal file
|
|
@ -0,0 +1,124 @@
|
|||
/**
|
||||
* Generate tests/screenshot-manifest.json — the canonical list of expected
|
||||
* screenshots + the engine that renders each.
|
||||
*
|
||||
* The NAME list is authoritative (it's the committed baseline set) and is what
|
||||
* lets compare/promote tell an intentional REMOVAL from a flaky/absent render.
|
||||
* The ENGINE tag is best-effort (attributed by scanning which spec writes each
|
||||
* `test-results/<prefix>` and which project runs that spec) and only feeds the
|
||||
* per-engine floors — refine after calibration.
|
||||
*
|
||||
* Engine routing (from the two playwright configs):
|
||||
* e2e/*.spec.ts → chromium-swiftshader (npm run test — wx suite)
|
||||
* kicad/*.spec.ts → firefox-llvmpipe, or chromium-swiftshader if the
|
||||
* spec is in PCBNEW_FAMILY_SPECS (chromium-ci on CI)
|
||||
* web/*.spec.ts → firefox-llvmpipe (web config, --project=firefox)
|
||||
*
|
||||
* CLI (from tests/): tsx tools/screenshots/gen-manifest.ts [--check]
|
||||
* --check exits 1 if the committed manifest is stale (for CI hygiene).
|
||||
*/
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
import { BASELINE_DIRS, MANIFEST_PATH, type Manifest } from './config';
|
||||
|
||||
const CHROMIUM = 'chromium-swiftshader';
|
||||
const FIREFOX = 'firefox-llvmpipe';
|
||||
const DEFAULT_ENGINE = CHROMIUM; // baseline-screenshots is dominated by the wx suite
|
||||
|
||||
function listSpecs(dir: string): string[] {
|
||||
const out: string[] = [];
|
||||
if (!fs.existsSync(dir)) return out;
|
||||
for (const e of fs.readdirSync(dir, { withFileTypes: true })) {
|
||||
const p = path.join(dir, e.name);
|
||||
if (e.isDirectory()) out.push(...listSpecs(p));
|
||||
// Only spec files — a screenshot literal in a util (e.g. completeWizard's
|
||||
// wizard-*) would be attributed to the util's dir, not its real caller.
|
||||
// Leaving those unmatched lets them fall to the correct chromium default.
|
||||
else if (e.name.endsWith('.spec.ts')) out.push(p);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/** pcbnew-family spec basenames (routed to chromium-ci on CI), read from the config. */
|
||||
function pcbnewFamily(root: string): Set<string> {
|
||||
const cfg = fs.readFileSync(path.join(root, 'playwright-kicad.config.ts'), 'utf8');
|
||||
const block = cfg.match(/PCBNEW_FAMILY_SPECS\s*=\s*\[([\s\S]*?)\]/)?.[1] ?? '';
|
||||
return new Set([...block.matchAll(/'[^']*?([\w.-]+\.spec\.ts)'/g)].map((m) => m[1]));
|
||||
}
|
||||
|
||||
function engineForSpec(root: string, specPath: string, family: Set<string>): string {
|
||||
const rel = path.relative(root, specPath);
|
||||
const base = path.basename(specPath);
|
||||
if (rel.startsWith('e2e/')) return CHROMIUM;
|
||||
if (rel.startsWith('web/')) return FIREFOX;
|
||||
if (rel.startsWith('kicad/')) return family.has(base) ? CHROMIUM : FIREFOX;
|
||||
return DEFAULT_ENGINE;
|
||||
}
|
||||
|
||||
/** Build prefix → engine from every `test-results/<prefix>` literal in the specs. */
|
||||
function prefixEngineMap(root: string, family: Set<string>): Array<{ prefix: string; engine: string }> {
|
||||
const map = new Map<string, string>();
|
||||
for (const dir of ['e2e', 'kicad', 'web']) {
|
||||
for (const spec of listSpecs(path.join(root, dir))) {
|
||||
const engine = engineForSpec(root, spec, family);
|
||||
const content = fs.readFileSync(spec, 'utf8');
|
||||
for (const m of content.matchAll(/test-results\/([A-Za-z0-9_-]+)/g)) {
|
||||
const prefix = m[1];
|
||||
// First writer wins; a chromium spec shouldn't be overridden by a later firefox one for the same literal.
|
||||
if (!map.has(prefix)) map.set(prefix, engine);
|
||||
}
|
||||
}
|
||||
}
|
||||
// Longest prefix first so the most specific match wins.
|
||||
return [...map.entries()].map(([prefix, engine]) => ({ prefix, engine })).sort((a, b) => b.prefix.length - a.prefix.length);
|
||||
}
|
||||
|
||||
function listBaselines(root: string): string[] {
|
||||
const names = new Set<string>();
|
||||
for (const dir of BASELINE_DIRS) {
|
||||
const abs = path.join(root, dir);
|
||||
if (!fs.existsSync(abs)) continue;
|
||||
for (const f of fs.readdirSync(abs)) if (f.toLowerCase().endsWith('.png')) names.add(f);
|
||||
}
|
||||
return [...names].sort();
|
||||
}
|
||||
|
||||
function main(): void {
|
||||
const check = process.argv.includes('--check');
|
||||
const root = process.cwd();
|
||||
const family = pcbnewFamily(root);
|
||||
const prefixes = prefixEngineMap(root, family);
|
||||
|
||||
let unmatched = 0;
|
||||
const screenshots = listBaselines(root).map((name) => {
|
||||
const stem = name.replace(/\.png$/i, '');
|
||||
const hit = prefixes.find((p) => stem === p.prefix || stem.startsWith(p.prefix));
|
||||
if (!hit) unmatched++;
|
||||
return { name, engine: hit?.engine ?? DEFAULT_ENGINE };
|
||||
});
|
||||
|
||||
const manifest: Manifest & { _note: string } = {
|
||||
_note: 'engine tags are best-effort (gen-manifest.ts); the name list is authoritative. Refine engines after calibration.',
|
||||
screenshots,
|
||||
};
|
||||
const json = JSON.stringify(manifest, null, 2) + '\n';
|
||||
const outPath = path.join(root, MANIFEST_PATH);
|
||||
|
||||
const dist = screenshots.reduce<Record<string, number>>((d, s) => ((d[s.engine] = (d[s.engine] ?? 0) + 1), d), {});
|
||||
console.log(`[manifest] ${screenshots.length} screenshots; engines=${JSON.stringify(dist)}; default-assigned=${unmatched}`);
|
||||
|
||||
if (check) {
|
||||
const current = fs.existsSync(outPath) ? fs.readFileSync(outPath, 'utf8') : '';
|
||||
if (current !== json) {
|
||||
console.error('[manifest] STALE — run `npm run screenshots:manifest` and commit');
|
||||
process.exitCode = 1;
|
||||
} else {
|
||||
console.log('[manifest] up to date');
|
||||
}
|
||||
return;
|
||||
}
|
||||
fs.writeFileSync(outPath, json);
|
||||
console.log(`[manifest] wrote ${MANIFEST_PATH}`);
|
||||
}
|
||||
|
||||
if (require.main === module) main();
|
||||
256
tests/tools/screenshots/image-ops.ts
Normal file
256
tests/tools/screenshots/image-ops.ts
Normal file
|
|
@ -0,0 +1,256 @@
|
|||
/**
|
||||
* Reusable pixel operations for the screenshot tooling: PNG load/save, padding,
|
||||
* the pixelmatch-backed diff, connected-component clustering ("where to look"),
|
||||
* box drawing, and horizontal compositing for the triptych.
|
||||
*
|
||||
* All images are handled as pngjs PNGs whose `.data` is a length `w*h*4` RGBA
|
||||
* Buffer, regardless of the source PNG colour type (pngjs normalizes to RGBA).
|
||||
*/
|
||||
import * as fs from 'fs';
|
||||
import { PNG } from 'pngjs';
|
||||
import pixelmatch from 'pixelmatch';
|
||||
import { PIXELMATCH, DIFF_COLOR, CLUSTER, TRIPTYCH } from './config';
|
||||
|
||||
export type Box = { x: number; y: number; width: number; height: number; area: number };
|
||||
|
||||
export type DiffResult = {
|
||||
width: number;
|
||||
height: number;
|
||||
dimsMatch: boolean;
|
||||
/** AA-excluded changed-pixel count (from pixelmatch). */
|
||||
diffPixels: number;
|
||||
/** diffPixels / (width*height). */
|
||||
changedRatio: number;
|
||||
/** mean |Δ| over every RGBA channel sample of the whole frame (matches the legacy metric). */
|
||||
meanChannelDiff: number;
|
||||
/** pixelmatch heatmap: dimmed base + red diffs / yellow AA. */
|
||||
heatmap: PNG;
|
||||
/** boolean mask (1 = real, non-AA changed pixel) for clustering. */
|
||||
mask: Uint8Array;
|
||||
};
|
||||
|
||||
export function loadPng(file: string): PNG {
|
||||
return PNG.sync.read(fs.readFileSync(file));
|
||||
}
|
||||
|
||||
export function savePng(file: string, png: PNG): void {
|
||||
fs.writeFileSync(file, PNG.sync.write(png));
|
||||
}
|
||||
|
||||
/** New PNG of `w`×`h` filled with `fill` (RGBA), with `src` blitted at top-left. */
|
||||
export function padTo(src: PNG, w: number, h: number, fill: [number, number, number, number]): PNG {
|
||||
const out = new PNG({ width: w, height: h });
|
||||
for (let i = 0; i < out.data.length; i += 4) {
|
||||
out.data[i] = fill[0];
|
||||
out.data[i + 1] = fill[1];
|
||||
out.data[i + 2] = fill[2];
|
||||
out.data[i + 3] = fill[3];
|
||||
}
|
||||
for (let y = 0; y < Math.min(h, src.height); y++) {
|
||||
const srcRow = y * src.width * 4;
|
||||
const dstRow = y * w * 4;
|
||||
const rowBytes = Math.min(w, src.width) * 4;
|
||||
src.data.copy(out.data, dstRow, srcRow, srcRow + rowBytes);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Diff two images. On a dimension mismatch both are padded (magenta) to the
|
||||
* union size and `dimsMatch` is false (the caller treats that as CHANGED).
|
||||
* The changed-pixel mask is read back from the heatmap's red diff pixels, so it
|
||||
* inherits pixelmatch's anti-aliasing exclusion.
|
||||
*/
|
||||
export function diffImages(a: PNG, b: PNG): DiffResult {
|
||||
const dimsMatch = a.width === b.width && a.height === b.height;
|
||||
const width = Math.max(a.width, b.width);
|
||||
const height = Math.max(a.height, b.height);
|
||||
const pa = dimsMatch ? a : padTo(a, width, height, TRIPTYCH.padFill);
|
||||
const pb = dimsMatch ? b : padTo(b, width, height, TRIPTYCH.padFill);
|
||||
|
||||
const heatmap = new PNG({ width, height });
|
||||
const diffPixels = pixelmatch(pa.data, pb.data, heatmap.data, width, height, {
|
||||
threshold: PIXELMATCH.threshold,
|
||||
includeAA: PIXELMATCH.includeAA,
|
||||
diffColor: DIFF_COLOR,
|
||||
});
|
||||
|
||||
// Whole-frame mean channel delta (drift-vs-regression heuristic input).
|
||||
let totalChannelDiff = 0;
|
||||
for (let i = 0; i < pa.data.length; i++) {
|
||||
totalChannelDiff += Math.abs(pa.data[i] - pb.data[i]);
|
||||
}
|
||||
const meanChannelDiff = totalChannelDiff / pa.data.length;
|
||||
|
||||
// Mask = heatmap pixels painted with DIFF_COLOR (red). AA pixels are yellow, so excluded.
|
||||
const mask = new Uint8Array(width * height);
|
||||
for (let p = 0; p < width * height; p++) {
|
||||
const o = p * 4;
|
||||
if (heatmap.data[o] > 200 && heatmap.data[o + 1] < 80 && heatmap.data[o + 2] < 80) {
|
||||
mask[p] = 1;
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
width,
|
||||
height,
|
||||
dimsMatch,
|
||||
diffPixels,
|
||||
changedRatio: diffPixels / (width * height),
|
||||
meanChannelDiff,
|
||||
heatmap,
|
||||
mask,
|
||||
};
|
||||
}
|
||||
|
||||
/** Dilate a boolean mask by `r` (square structuring element), out of place. */
|
||||
function dilate(mask: Uint8Array, w: number, h: number, r: number): Uint8Array {
|
||||
if (r <= 0) return mask;
|
||||
const out = new Uint8Array(w * h);
|
||||
for (let y = 0; y < h; y++) {
|
||||
for (let x = 0; x < w; x++) {
|
||||
if (!mask[y * w + x]) continue;
|
||||
const y0 = Math.max(0, y - r);
|
||||
const y1 = Math.min(h - 1, y + r);
|
||||
const x0 = Math.max(0, x - r);
|
||||
const x1 = Math.min(w - 1, x + r);
|
||||
for (let yy = y0; yy <= y1; yy++) {
|
||||
for (let xx = x0; xx <= x1; xx++) out[yy * w + xx] = 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* 8-connected connected-components over the (dilated) mask → bounding boxes,
|
||||
* largest-area first, capped at `maxBoxes`, specks below `minBoxArea` dropped.
|
||||
*/
|
||||
export function cluster(mask: Uint8Array, w: number, h: number): Box[] {
|
||||
const grown = dilate(mask, w, h, CLUSTER.dilate);
|
||||
const seen = new Uint8Array(w * h);
|
||||
const boxes: Box[] = [];
|
||||
const stack: number[] = [];
|
||||
|
||||
for (let start = 0; start < grown.length; start++) {
|
||||
if (!grown[start] || seen[start]) continue;
|
||||
let minX = w, minY = h, maxX = 0, maxY = 0, count = 0;
|
||||
stack.push(start);
|
||||
seen[start] = 1;
|
||||
while (stack.length) {
|
||||
const p = stack.pop()!;
|
||||
const px = p % w;
|
||||
const py = (p - px) / w;
|
||||
count++;
|
||||
if (px < minX) minX = px;
|
||||
if (px > maxX) maxX = px;
|
||||
if (py < minY) minY = py;
|
||||
if (py > maxY) maxY = py;
|
||||
for (let dy = -1; dy <= 1; dy++) {
|
||||
for (let dx = -1; dx <= 1; dx++) {
|
||||
if (!dx && !dy) continue;
|
||||
const nx = px + dx;
|
||||
const ny = py + dy;
|
||||
if (nx < 0 || ny < 0 || nx >= w || ny >= h) continue;
|
||||
const np = ny * w + nx;
|
||||
if (grown[np] && !seen[np]) {
|
||||
seen[np] = 1;
|
||||
stack.push(np);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
const bw = maxX - minX + 1;
|
||||
const bh = maxY - minY + 1;
|
||||
const area = bw * bh;
|
||||
if (area >= CLUSTER.minBoxArea) {
|
||||
boxes.push({ x: minX, y: minY, width: bw, height: bh, area });
|
||||
}
|
||||
}
|
||||
boxes.sort((p, q) => q.area - p.area);
|
||||
return boxes.slice(0, CLUSTER.maxBoxes);
|
||||
}
|
||||
|
||||
/** Draw 2px rectangle outlines for each box onto a copy of `png`. */
|
||||
export function drawBoxes(png: PNG, boxes: Box[]): PNG {
|
||||
const out = new PNG({ width: png.width, height: png.height });
|
||||
png.data.copy(out.data);
|
||||
const [r, g, b] = CLUSTER.boxColor;
|
||||
const set = (x: number, y: number) => {
|
||||
if (x < 0 || y < 0 || x >= out.width || y >= out.height) return;
|
||||
const o = (y * out.width + x) * 4;
|
||||
out.data[o] = r;
|
||||
out.data[o + 1] = g;
|
||||
out.data[o + 2] = b;
|
||||
out.data[o + 3] = 255;
|
||||
};
|
||||
for (const box of boxes) {
|
||||
for (let t = 0; t < 2; t++) {
|
||||
for (let x = box.x; x < box.x + box.width; x++) {
|
||||
set(x, box.y + t);
|
||||
set(x, box.y + box.height - 1 - t);
|
||||
}
|
||||
for (let y = box.y; y < box.y + box.height; y++) {
|
||||
set(box.x + t, y);
|
||||
set(box.x + box.width - 1 - t, y);
|
||||
}
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/** Nearest-neighbour downscale by `scale` (0<scale<1). Fast, quality secondary — it only exists to fit Discord's size caps. */
|
||||
export function resizeNearest(png: PNG, scale: number): PNG {
|
||||
const w = Math.max(1, Math.round(png.width * scale));
|
||||
const h = Math.max(1, Math.round(png.height * scale));
|
||||
const out = new PNG({ width: w, height: h });
|
||||
for (let y = 0; y < h; y++) {
|
||||
const sy = Math.min(png.height - 1, Math.floor(y / scale));
|
||||
for (let x = 0; x < w; x++) {
|
||||
const sx = Math.min(png.width - 1, Math.floor(x / scale));
|
||||
const s = (sy * png.width + sx) * 4;
|
||||
const d = (y * w + x) * 4;
|
||||
out.data[d] = png.data[s];
|
||||
out.data[d + 1] = png.data[s + 1];
|
||||
out.data[d + 2] = png.data[s + 2];
|
||||
out.data[d + 3] = png.data[s + 3];
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/** Encode `png`, halving resolution until the PNG is <= maxBytes (or it can't shrink further). */
|
||||
export function encodeWithinCap(png: PNG, maxBytes: number): Buffer {
|
||||
let current = png;
|
||||
let buf = PNG.sync.write(current);
|
||||
while (buf.length > maxBytes && current.width > 320) {
|
||||
current = resizeNearest(current, 0.5);
|
||||
buf = PNG.sync.write(current);
|
||||
}
|
||||
return buf;
|
||||
}
|
||||
|
||||
/** Horizontally montage images (heights normalized to the tallest) with a gap + bg. */
|
||||
export function composite(panels: PNG[]): PNG {
|
||||
const gap = TRIPTYCH.gap;
|
||||
const bg = TRIPTYCH.bg;
|
||||
const height = Math.max(...panels.map((p) => p.height));
|
||||
const width = panels.reduce((s, p) => s + p.width, 0) + gap * (panels.length - 1);
|
||||
const out = new PNG({ width, height });
|
||||
for (let i = 0; i < out.data.length; i += 4) {
|
||||
out.data[i] = bg[0];
|
||||
out.data[i + 1] = bg[1];
|
||||
out.data[i + 2] = bg[2];
|
||||
out.data[i + 3] = bg[3];
|
||||
}
|
||||
let xOffset = 0;
|
||||
for (const panel of panels) {
|
||||
for (let y = 0; y < panel.height; y++) {
|
||||
const srcRow = y * panel.width * 4;
|
||||
const dstRow = (y * width + xOffset) * 4;
|
||||
panel.data.copy(out.data, dstRow, srcRow, srcRow + panel.width * 4);
|
||||
}
|
||||
xOffset += panel.width + gap;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
76
tests/tools/screenshots/noise.ts
Normal file
76
tests/tools/screenshots/noise.ts
Normal file
|
|
@ -0,0 +1,76 @@
|
|||
/**
|
||||
* Calibration: measure the intra-CI noise floor.
|
||||
*
|
||||
* Render the suite twice on the same host (into two dirs), then:
|
||||
* tsx tools/screenshots/noise.ts <run1-dir> <run2-dir>
|
||||
* prints, per engine (via the manifest) and globally, the max / p95 / mean
|
||||
* changed-pixel ratio between the two identical-input renders. Set
|
||||
* config.ts FLOORS.<engine>.changedRatio ≈ (max × 3) from this.
|
||||
*
|
||||
* A near-zero floor means the render is deterministic enough to catch real
|
||||
* changes tightly; a large floor exposes nondeterminism to chase before the
|
||||
* gate can be trusted.
|
||||
*/
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
import { MANIFEST_PATH, type Manifest } from './config';
|
||||
import { diffImages, loadPng } from './image-ops';
|
||||
|
||||
function listPngs(dir: string): string[] {
|
||||
return fs.existsSync(dir) ? fs.readdirSync(dir).filter((f) => f.toLowerCase().endsWith('.png')) : [];
|
||||
}
|
||||
|
||||
function engineOf(name: string, manifest?: Manifest): string {
|
||||
return manifest?.screenshots.find((e) => e.name === name)?.engine ?? 'default';
|
||||
}
|
||||
|
||||
function stats(values: number[]): { n: number; max: number; p95: number; mean: number } {
|
||||
if (!values.length) return { n: 0, max: 0, p95: 0, mean: 0 };
|
||||
const sorted = [...values].sort((a, b) => a - b);
|
||||
const p95 = sorted[Math.min(sorted.length - 1, Math.floor(sorted.length * 0.95))];
|
||||
return {
|
||||
n: values.length,
|
||||
max: sorted[sorted.length - 1],
|
||||
p95,
|
||||
mean: values.reduce((s, v) => s + v, 0) / values.length,
|
||||
};
|
||||
}
|
||||
|
||||
function main(): void {
|
||||
const [dir1, dir2] = process.argv.slice(2);
|
||||
if (!dir1 || !dir2) {
|
||||
console.error('usage: noise.ts <run1-dir> <run2-dir>');
|
||||
process.exitCode = 2;
|
||||
return;
|
||||
}
|
||||
const root = process.cwd();
|
||||
const manifestPath = path.join(root, MANIFEST_PATH);
|
||||
const manifest = fs.existsSync(manifestPath)
|
||||
? (JSON.parse(fs.readFileSync(manifestPath, 'utf8')) as Manifest)
|
||||
: undefined;
|
||||
|
||||
const common = listPngs(dir1).filter((n) => fs.existsSync(path.join(dir2, n)));
|
||||
const byEngine = new Map<string, number[]>();
|
||||
const all: number[] = [];
|
||||
for (const name of common) {
|
||||
const d = diffImages(loadPng(path.join(dir1, name)), loadPng(path.join(dir2, name)));
|
||||
const ratio = d.dimsMatch ? d.changedRatio : 1;
|
||||
all.push(ratio);
|
||||
const eng = engineOf(name, manifest);
|
||||
(byEngine.get(eng) ?? byEngine.set(eng, []).get(eng)!).push(ratio);
|
||||
}
|
||||
|
||||
const report = (label: string, values: number[]) => {
|
||||
const s = stats(values);
|
||||
console.log(
|
||||
`${label.padEnd(24)} n=${String(s.n).padStart(4)} max=${(s.max * 100).toFixed(4)}% ` +
|
||||
`p95=${(s.p95 * 100).toFixed(4)}% mean=${(s.mean * 100).toFixed(4)}% ` +
|
||||
`→ suggest changedRatio=${(s.max * 3).toExponential(2)}`
|
||||
);
|
||||
};
|
||||
console.log(`[noise] compared ${common.length} screenshots rendered twice`);
|
||||
for (const [eng, values] of byEngine) report(eng, values);
|
||||
report('ALL', all);
|
||||
}
|
||||
|
||||
if (require.main === module) main();
|
||||
164
tests/tools/screenshots/perf-report.ts
Normal file
164
tests/tools/screenshots/perf-report.ts
Normal file
|
|
@ -0,0 +1,164 @@
|
|||
/**
|
||||
* Renders the track-only runtime-perf block for the CI-on-main Discord comment.
|
||||
*
|
||||
* The perf e2e (tests/kicad/{eeschema,pcbnew}-perf.spec.ts) already writes
|
||||
* test-results/perf-{app}.json — schema { app, when, loadMs, openMs, fps:[{throttle,fps}] }.
|
||||
* We read those, fetch the PREVIOUS successful main run's perf via `gh run
|
||||
* download` (so we can show a Δ without committing a baseline — stays
|
||||
* no-write-back), and format an aligned monospace table (Discord doesn't render
|
||||
* markdown tables, so it goes in a ``` code block).
|
||||
*
|
||||
* Track-only: nothing here gates the build. A regression past REGRESSION_PCT on
|
||||
* the stable metrics (loadMs/openMs) is only flagged (a `*`), never failed. FPS
|
||||
* is CPU-bound/noisy on CI's headless SwiftShader path, so it's shown but marked
|
||||
* indicative.
|
||||
*
|
||||
* CLI (from tests/):
|
||||
* tsx tools/screenshots/perf-report.ts [--results DIR] [--prev DIR] [--repo owner/repo]
|
||||
*/
|
||||
import * as fs from 'fs';
|
||||
import * as os from 'os';
|
||||
import * as path from 'path';
|
||||
import { execFileSync } from 'child_process';
|
||||
import { RESULTS_DIR } from './config';
|
||||
|
||||
export const PERF_APPS = ['eeschema', 'pcbnew'] as const;
|
||||
const REGRESSION_PCT = 10; // stable-metric regression past this is flagged with `*`
|
||||
const CI_WORKFLOW = 'ci-ubicloud.yml';
|
||||
|
||||
export type Fps = { throttle: number; fps: number };
|
||||
export type PerfData = { app: string; when?: string; loadMs: number; openMs: number; fps: Fps[] };
|
||||
|
||||
export function readPerf(dir: string): Map<string, PerfData> {
|
||||
const out = new Map<string, PerfData>();
|
||||
for (const app of PERF_APPS) {
|
||||
const p = path.join(dir, `perf-${app}.json`);
|
||||
if (!fs.existsSync(p)) continue;
|
||||
try {
|
||||
out.set(app, JSON.parse(fs.readFileSync(p, 'utf8')) as PerfData);
|
||||
} catch (e) {
|
||||
console.warn(`[perf] could not parse ${p}: ${(e as Error).message}`);
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Best-effort fetch of the previous successful main CI run's perf JSONs into a
|
||||
* temp dir. Returns the dir, or null if gh is unavailable / no prior run.
|
||||
* `currentSha` is skipped so a re-run doesn't diff against itself.
|
||||
*/
|
||||
export function fetchPreviousPerf(repo: string | undefined, currentSha: string | undefined): string | null {
|
||||
try {
|
||||
const gh = (args: string[]) =>
|
||||
execFileSync('gh', args, { encoding: 'utf8', stdio: ['ignore', 'pipe', 'ignore'] });
|
||||
const repoArgs = repo ? ['--repo', repo] : [];
|
||||
const runs = JSON.parse(
|
||||
gh([
|
||||
'run', 'list', '--workflow', CI_WORKFLOW, '--branch', 'main', '--status', 'success',
|
||||
'--limit', '15', '--json', 'databaseId,headSha', ...repoArgs,
|
||||
])
|
||||
) as Array<{ databaseId: number; headSha: string }>;
|
||||
const prev = runs.find((r) => r.headSha !== currentSha);
|
||||
if (!prev) return null;
|
||||
const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'perf-prev-'));
|
||||
// Artifact is named ubicloud-e2e-<run_id>; -D extracts it under tmp/<artifact>/...
|
||||
gh(['run', 'download', String(prev.databaseId), '-D', tmp, ...repoArgs]);
|
||||
// Find the dir that actually holds the perf-*.json (artifact nests test-results/).
|
||||
const hit = findPerfDir(tmp);
|
||||
return hit;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function findPerfDir(root: string): string | null {
|
||||
const stack = [root];
|
||||
while (stack.length) {
|
||||
const dir = stack.pop()!;
|
||||
let entries: fs.Dirent[];
|
||||
try {
|
||||
entries = fs.readdirSync(dir, { withFileTypes: true });
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
if (entries.some((e) => e.isFile() && /^perf-\w+\.json$/.test(e.name))) return dir;
|
||||
for (const e of entries) if (e.isDirectory()) stack.push(path.join(dir, e.name));
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function pct(cur: number, prev: number): number {
|
||||
return prev === 0 ? 0 : ((cur - prev) / prev) * 100;
|
||||
}
|
||||
|
||||
/** Format a stable-metric value with a Δ vs previous (lower is better). */
|
||||
function fmtMetric(cur: number, prev?: number): string {
|
||||
if (prev === undefined) return `${cur}`;
|
||||
const p = pct(cur, prev);
|
||||
const arrow = p < 0 ? '▼' : p > 0 ? '▲' : '·';
|
||||
const flag = p > REGRESSION_PCT ? '*' : ''; // regression (slower) beyond threshold
|
||||
return `${cur} ${arrow}${Math.abs(p).toFixed(0)}%${flag}`;
|
||||
}
|
||||
|
||||
function fmtFps(fps: Fps[]): string {
|
||||
return [1, 4, 6].map((t) => {
|
||||
const hit = fps.find((f) => f.throttle === t);
|
||||
return hit ? Math.round(hit.fps) : '–';
|
||||
}).join('/');
|
||||
}
|
||||
|
||||
function pad(s: string, n: number): string {
|
||||
return s.length >= n ? s : s + ' '.repeat(n - s.length);
|
||||
}
|
||||
|
||||
export type PerfReport = { block: string; regressed: boolean };
|
||||
|
||||
/** Build the fenced code-block perf table for the Discord comment. */
|
||||
export function buildPerfReport(opts: { resultsDir?: string; prevDir?: string | null } = {}): PerfReport {
|
||||
const cur = readPerf(opts.resultsDir ?? RESULTS_DIR);
|
||||
if (cur.size === 0) return { block: '', regressed: false };
|
||||
const prev = opts.prevDir ? readPerf(opts.prevDir) : new Map<string, PerfData>();
|
||||
|
||||
const headers = ['app', 'loadMs (Δ)', 'openMs (Δ)', 'FPS 1/4/6'];
|
||||
const rows: string[][] = [];
|
||||
let regressed = false;
|
||||
for (const app of PERF_APPS) {
|
||||
const c = cur.get(app);
|
||||
if (!c) continue;
|
||||
const p = prev.get(app);
|
||||
const loadCell = fmtMetric(c.loadMs, p?.loadMs);
|
||||
const openCell = fmtMetric(c.openMs, p?.openMs);
|
||||
if (loadCell.endsWith('*') || openCell.endsWith('*')) regressed = true;
|
||||
rows.push([app, loadCell, openCell, fmtFps(c.fps)]);
|
||||
}
|
||||
if (rows.length === 0) return { block: '', regressed: false };
|
||||
|
||||
const widths = headers.map((h, i) => Math.max(h.length, ...rows.map((r) => r[i].length)));
|
||||
const line = (cells: string[]) => cells.map((c, i) => pad(c, widths[i])).join(' ');
|
||||
const body = [line(headers), rows.map((r) => line(r)).join('\n')].join('\n');
|
||||
const footnote = `${prev.size ? 'Δ vs previous main run. ' : 'no prior main run for Δ. '}` +
|
||||
`* = >${REGRESSION_PCT}% slower (track-only, non-gating). FPS is CI-headless — indicative only.`;
|
||||
return { block: '**Runtime perf** (eeschema + pcbnew)\n```\n' + body + '\n```\n' + footnote, regressed };
|
||||
}
|
||||
|
||||
function parseArgs(argv: string[]): Record<string, string> {
|
||||
const out: Record<string, string> = {};
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
const a = argv[i];
|
||||
if (a === '--results') out.results = argv[++i];
|
||||
else if (a === '--prev') out.prev = argv[++i];
|
||||
else if (a === '--repo') out.repo = argv[++i];
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function main(): void {
|
||||
const args = parseArgs(process.argv.slice(2));
|
||||
const prevDir = args.prev ?? fetchPreviousPerf(args.repo || process.env.GITHUB_REPOSITORY, process.env.GITHUB_SHA);
|
||||
const { block, regressed } = buildPerfReport({ resultsDir: args.results, prevDir });
|
||||
process.stdout.write((block || '(no perf-*.json found)') + '\n');
|
||||
if (regressed) console.error('[perf] a stable metric regressed >10% vs previous main (track-only, not failing)');
|
||||
}
|
||||
|
||||
if (require.main === module) main();
|
||||
226
tests/tools/screenshots/post-discord.ts
Normal file
226
tests/tools/screenshots/post-discord.ts
Normal file
|
|
@ -0,0 +1,226 @@
|
|||
/**
|
||||
* The always-on CI-on-main Discord report.
|
||||
*
|
||||
* One message leads with the commit SHA + e2e pass/fail + the runtime-perf table
|
||||
* (perf-report.ts), then screenshot changes are attached as triptychs
|
||||
* (old | new+boxes | heatmap), ADDED images, and a REMOVED list. Because perf is
|
||||
* always present, the comment fires on every main commit even with no screenshot
|
||||
* change — "see it in hindsight on every commit."
|
||||
*
|
||||
* Safety / robustness:
|
||||
* - Posts only on push to main (unless --force); a missing DISCORD_WEBHOOK_URL is
|
||||
* a silent no-op, so it's inert on PRs / forks (which can't see the secret).
|
||||
* - Attachments batched ≤10/message and downscaled to fit Discord's size caps;
|
||||
* 429 Retry-After honoured; first-run flood (hundreds of ADDED) collapsed.
|
||||
* - --dry-run composes everything and prints it without POSTing (how it's tested).
|
||||
*
|
||||
* CLI (from tests/):
|
||||
* tsx tools/screenshots/post-discord.ts [--dry-run] [--force] [--e2e pass|fail] [--subject S]
|
||||
*/
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
import { execFileSync } from 'child_process';
|
||||
import { DIFF_OUT_DIR } from './config';
|
||||
import { loadPng, encodeWithinCap } from './image-ops';
|
||||
import { buildPerfReport, fetchPreviousPerf } from './perf-report';
|
||||
import type { Report } from './compare';
|
||||
|
||||
const MAX_FILE_BYTES = 8 * 1024 * 1024;
|
||||
const MAX_MSG_BYTES = 24 * 1024 * 1024;
|
||||
const MAX_FILES_PER_MSG = 10;
|
||||
const MAX_TOTAL_FILES = 30; // ~3 messages of images; excess is summarized, not posted
|
||||
const FLOOD_N = 12; // more ADDED than this ⇒ show a few exemplars, not all
|
||||
|
||||
type Attachment = { name: string; buffer: Buffer };
|
||||
type Message = { content: string; files: Attachment[] };
|
||||
|
||||
function sanitize(name: string): string {
|
||||
return name.replace(/[^\w.-]+/g, '_');
|
||||
}
|
||||
|
||||
function e2eBadge(status: string | undefined): string {
|
||||
if (status === 'pass') return '✅ e2e passed';
|
||||
if (status === 'fail') return '❌ e2e failed';
|
||||
return 'ℹ️ e2e status unknown';
|
||||
}
|
||||
|
||||
function commitSubject(sha: string | undefined): string {
|
||||
if (!sha) return '';
|
||||
try {
|
||||
return execFileSync('git', ['log', '-1', '--pretty=%s', sha], { encoding: 'utf8' }).trim();
|
||||
} catch {
|
||||
return '';
|
||||
}
|
||||
}
|
||||
|
||||
function readReport(root: string): Report | null {
|
||||
const p = path.join(root, DIFF_OUT_DIR, 'report.json');
|
||||
if (!fs.existsSync(p)) return null;
|
||||
try {
|
||||
return JSON.parse(fs.readFileSync(p, 'utf8')) as Report;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/** Encode a PNG path to a within-cap attachment, or null if missing/unreadable. */
|
||||
function attach(root: string, rel: string, name: string): Attachment | null {
|
||||
const abs = path.isAbsolute(rel) ? rel : path.join(root, rel);
|
||||
if (!fs.existsSync(abs)) return null;
|
||||
try {
|
||||
return { name: sanitize(name), buffer: encodeWithinCap(loadPng(abs), MAX_FILE_BYTES) };
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/** Build the header text: SHA, subject, e2e, perf table, screenshot summary, removed list. */
|
||||
function buildHeader(report: Report | null, perfBlock: string, meta: { sha?: string; subject: string; e2e?: string }): string {
|
||||
const shortSha = meta.sha ? meta.sha.slice(0, 7) : 'local';
|
||||
const lines = [`📸 **CI screenshot + perf report** · \`${shortSha}\``];
|
||||
if (meta.subject) lines.push(`> ${meta.subject}`);
|
||||
lines.push(e2eBadge(meta.e2e));
|
||||
if (perfBlock) lines.push('', perfBlock);
|
||||
|
||||
lines.push('');
|
||||
if (!report || (!report.changed.length && !report.added.length && !report.removed.length)) {
|
||||
lines.push('✅ no screenshot drift');
|
||||
} else if (report.driftLikely) {
|
||||
lines.push(`⚠️ **${report.changed.length} screenshots changed broadly** — looks like host env drift → re-promote (\`npm run screenshots:promote\`)`);
|
||||
} else {
|
||||
lines.push(`⚠️ **screenshot drift**: ${report.changed.length} changed, ${report.added.length} added, ${report.removed.length} removed`);
|
||||
}
|
||||
if (report?.removed.length) {
|
||||
lines.push('➖ REMOVED: ' + report.removed.map((r) => `\`${r.name}\``).join(', '));
|
||||
}
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
/** Collect the image attachments (changed triptychs + added images) with flood-collapse + total cap. */
|
||||
export function buildAttachments(root: string, report: Report | null): { files: Attachment[]; notes: string[] } {
|
||||
const files: Attachment[] = [];
|
||||
const notes: string[] = [];
|
||||
if (!report) return { files, notes };
|
||||
|
||||
for (const c of report.changed) {
|
||||
if (files.length >= MAX_TOTAL_FILES) break;
|
||||
const a = attach(root, c.triptych, `CHANGED_${c.name}`);
|
||||
if (a) files.push(a);
|
||||
}
|
||||
const addedToShow = report.added.length > FLOOD_N ? report.added.slice(0, 3) : report.added;
|
||||
if (report.added.length > FLOOD_N) {
|
||||
notes.push(`➕ ${report.added.length} added (showing ${addedToShow.length})`);
|
||||
}
|
||||
for (const ad of addedToShow) {
|
||||
if (files.length >= MAX_TOTAL_FILES) break;
|
||||
const a = attach(root, ad.image, `ADDED_${ad.name}`);
|
||||
if (a) files.push(a);
|
||||
}
|
||||
const shown = files.length;
|
||||
const wanted = report.changed.length + addedToShow.length;
|
||||
if (wanted > shown) notes.push(`(${wanted - shown} more images omitted — see the CI artifact)`);
|
||||
return { files, notes };
|
||||
}
|
||||
|
||||
/** Split attachments into messages of ≤10 files and ≤MAX_MSG_BYTES; content only on the first. */
|
||||
export function paginate(header: string, files: Attachment[]): Message[] {
|
||||
if (!files.length) return [{ content: header, files: [] }];
|
||||
const messages: Message[] = [];
|
||||
let batch: Attachment[] = [];
|
||||
let bytes = 0;
|
||||
const flush = () => {
|
||||
messages.push({ content: messages.length === 0 ? header : '', files: batch });
|
||||
batch = [];
|
||||
bytes = 0;
|
||||
};
|
||||
for (const f of files) {
|
||||
if (batch.length >= MAX_FILES_PER_MSG || bytes + f.buffer.length > MAX_MSG_BYTES) flush();
|
||||
batch.push(f);
|
||||
bytes += f.buffer.length;
|
||||
}
|
||||
if (batch.length || messages.length === 0) flush();
|
||||
return messages;
|
||||
}
|
||||
|
||||
async function sleep(ms: number): Promise<void> {
|
||||
return new Promise((r) => setTimeout(r, ms));
|
||||
}
|
||||
|
||||
export async function postMessage(webhook: string, msg: Message): Promise<void> {
|
||||
for (let attemptNo = 0; attemptNo < 6; attemptNo++) {
|
||||
const form = new FormData();
|
||||
form.append('payload_json', JSON.stringify({ content: msg.content || '', allowed_mentions: { parse: [] } }));
|
||||
msg.files.forEach((f, i) => form.append(`files[${i}]`, new Blob([new Uint8Array(f.buffer)], { type: 'image/png' }), f.name));
|
||||
const res = await fetch(webhook, { method: 'POST', body: form });
|
||||
if (res.ok) return;
|
||||
if (res.status === 429) {
|
||||
const retryAfter = Number(res.headers.get('retry-after')) || 2;
|
||||
await sleep((retryAfter + 0.5) * 1000);
|
||||
continue;
|
||||
}
|
||||
throw new Error(`Discord POST failed ${res.status}: ${(await res.text()).slice(0, 300)}`);
|
||||
}
|
||||
throw new Error('Discord POST failed after retries (429)');
|
||||
}
|
||||
|
||||
function parseArgs(argv: string[]): Record<string, string | boolean> {
|
||||
const out: Record<string, string | boolean> = {};
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
const a = argv[i];
|
||||
if (a === '--dry-run') out.dryRun = true;
|
||||
else if (a === '--force') out.force = true;
|
||||
else if (a === '--e2e') out.e2e = argv[++i];
|
||||
else if (a === '--subject') out.subject = argv[++i];
|
||||
else if (a === '--repo') out.repo = argv[++i];
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
async function main(): Promise<void> {
|
||||
const args = parseArgs(process.argv.slice(2));
|
||||
const root = process.cwd();
|
||||
const isMainPush = process.env.GITHUB_REF === 'refs/heads/main' && process.env.GITHUB_EVENT_NAME === 'push';
|
||||
if (!args.force && !args.dryRun && !isMainPush) {
|
||||
console.log('[discord] not a push to main — skipping');
|
||||
return;
|
||||
}
|
||||
const webhook = process.env.DISCORD_WEBHOOK_URL;
|
||||
if (!webhook && !args.dryRun) {
|
||||
console.log('[discord] DISCORD_WEBHOOK_URL unset — skipping (inert on PRs/forks)');
|
||||
return;
|
||||
}
|
||||
|
||||
const sha = process.env.GITHUB_SHA;
|
||||
const report = readReport(root);
|
||||
const prevDir = fetchPreviousPerf((args.repo as string) || process.env.GITHUB_REPOSITORY, sha);
|
||||
const { block: perfBlock } = buildPerfReport({ prevDir });
|
||||
|
||||
let header = buildHeader(report, perfBlock, {
|
||||
sha,
|
||||
subject: (args.subject as string) ?? commitSubject(sha),
|
||||
e2e: args.e2e as string,
|
||||
});
|
||||
const { files, notes } = buildAttachments(root, report);
|
||||
if (notes.length) header += '\n' + notes.join('\n');
|
||||
|
||||
const messages = paginate(header, files);
|
||||
|
||||
if (args.dryRun) {
|
||||
for (const [i, m] of messages.entries()) {
|
||||
console.log(`--- message ${i + 1}/${messages.length} (${m.files.length} files, ${m.files.reduce((s, f) => s + f.buffer.length, 0)} bytes) ---`);
|
||||
if (m.content) console.log(m.content);
|
||||
for (const f of m.files) console.log(` [attach] ${f.name} (${f.buffer.length} bytes)`);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
for (const m of messages) await postMessage(webhook!, m);
|
||||
console.log(`[discord] posted ${messages.length} message(s)`);
|
||||
}
|
||||
|
||||
if (require.main === module) {
|
||||
main().catch((e) => {
|
||||
console.error(`[discord] ${e.message}`);
|
||||
process.exitCode = 1;
|
||||
});
|
||||
}
|
||||
149
tests/tools/screenshots/promote.ts
Normal file
149
tests/tools/screenshots/promote.ts
Normal file
|
|
@ -0,0 +1,149 @@
|
|||
/**
|
||||
* Churn-free baseline updater — "promote CI's render".
|
||||
*
|
||||
* CI's x86 render is the source of truth. This pulls a CI run's screenshots
|
||||
* (`gh run download`) — or a local dir via --from — and, for each one, overwrites
|
||||
* the committed baseline ONLY when the decoded pixels differ beyond the per-engine
|
||||
* floor. Unchanged baselines are left byte-identical (never re-encoded), so git
|
||||
* sees no churn. New shots are added; baselines with no render are reported as
|
||||
* removal candidates and only deleted with --prune.
|
||||
*
|
||||
* CLI (from tests/):
|
||||
* tsx tools/screenshots/promote.ts --run <ci-run-id> [--repo owner/repo] [--prune] [--dry-run]
|
||||
* tsx tools/screenshots/promote.ts --from <dir> [--prune] [--dry-run]
|
||||
*/
|
||||
import * as fs from 'fs';
|
||||
import * as os from 'os';
|
||||
import * as path from 'path';
|
||||
import { execFileSync } from 'child_process';
|
||||
import { BASELINE_DIRS, MANIFEST_PATH, floorFor, type Manifest } from './config';
|
||||
import { diffImages, loadPng } from './image-ops';
|
||||
|
||||
function listPngs(dir: string): string[] {
|
||||
if (!fs.existsSync(dir)) return [];
|
||||
return fs.readdirSync(dir).filter((f) => f.toLowerCase().endsWith('.png'));
|
||||
}
|
||||
|
||||
/** basename → absolute committed baseline path (first BASELINE_DIRS entry wins). */
|
||||
function baselineIndex(root: string): Map<string, string> {
|
||||
const index = new Map<string, string>();
|
||||
for (const dir of BASELINE_DIRS) {
|
||||
for (const name of listPngs(path.join(root, dir))) {
|
||||
if (!index.has(name)) index.set(name, path.join(root, dir, name));
|
||||
}
|
||||
}
|
||||
return index;
|
||||
}
|
||||
|
||||
function loadManifest(root: string): Manifest | undefined {
|
||||
const p = path.join(root, MANIFEST_PATH);
|
||||
if (!fs.existsSync(p)) return undefined;
|
||||
try {
|
||||
return JSON.parse(fs.readFileSync(p, 'utf8')) as Manifest;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
/** Download a CI run's artifact and return the top-level test-results dir holding the shots. */
|
||||
function downloadRun(runId: string, repo?: string): string {
|
||||
const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'promote-'));
|
||||
const repoArgs = repo ? ['--repo', repo] : [];
|
||||
execFileSync('gh', ['run', 'download', runId, '-D', tmp, ...repoArgs], { stdio: 'inherit' });
|
||||
// The artifact stores test-results/** — find that dir; its immediate *.png are the shots
|
||||
// (exclude the nested screenshot-diff/ triptychs).
|
||||
const stack = [tmp];
|
||||
while (stack.length) {
|
||||
const dir = stack.pop()!;
|
||||
const entries = fs.readdirSync(dir, { withFileTypes: true });
|
||||
if (path.basename(dir) === 'test-results') return dir;
|
||||
for (const e of entries) if (e.isDirectory()) stack.push(path.join(dir, e.name));
|
||||
}
|
||||
throw new Error(`no test-results/ dir found in the downloaded artifact under ${tmp}`);
|
||||
}
|
||||
|
||||
type Plan = { updated: string[]; added: string[]; unchanged: string[]; removedCandidates: string[] };
|
||||
|
||||
function buildPlan(root: string, renderDir: string, manifest?: Manifest): { plan: Plan; apply: () => void } {
|
||||
const baselines = baselineIndex(root);
|
||||
const rendered = new Set(listPngs(renderDir));
|
||||
const plan: Plan = { updated: [], added: [], unchanged: [], removedCandidates: [] };
|
||||
const actions: Array<() => void> = [];
|
||||
|
||||
for (const name of rendered) {
|
||||
const src = path.join(renderDir, name);
|
||||
const existing = baselines.get(name);
|
||||
if (!existing) {
|
||||
const dest = path.join(root, BASELINE_DIRS[0], name);
|
||||
plan.added.push(name);
|
||||
actions.push(() => fs.copyFileSync(src, dest)); // verbatim bytes
|
||||
continue;
|
||||
}
|
||||
const d = diffImages(loadPng(existing), loadPng(src));
|
||||
const floor = floorFor(name, manifest);
|
||||
if (!d.dimsMatch || d.changedRatio > floor.changedRatio) {
|
||||
plan.updated.push(name);
|
||||
actions.push(() => fs.copyFileSync(src, existing)); // verbatim bytes, no re-encode → no churn
|
||||
} else {
|
||||
plan.unchanged.push(name); // leave the committed file untouched
|
||||
}
|
||||
}
|
||||
|
||||
// Removal candidates: a committed baseline the manifest expects but this render didn't produce.
|
||||
for (const [name, abs] of baselines) {
|
||||
if (rendered.has(name)) continue;
|
||||
if (manifest && !manifest.screenshots.some((e) => e.name === name)) continue;
|
||||
plan.removedCandidates.push(name);
|
||||
actions.push(() => {}); // pruning is opt-in (see main)
|
||||
void abs;
|
||||
}
|
||||
|
||||
return { plan, apply: () => actions.forEach((a) => a()) };
|
||||
}
|
||||
|
||||
function parseArgs(argv: string[]): Record<string, string | boolean> {
|
||||
const out: Record<string, string | boolean> = {};
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
const a = argv[i];
|
||||
if (a === '--run') out.run = argv[++i];
|
||||
else if (a === '--from') out.from = argv[++i];
|
||||
else if (a === '--repo') out.repo = argv[++i];
|
||||
else if (a === '--prune') out.prune = true;
|
||||
else if (a === '--dry-run') out.dryRun = true;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function main(): void {
|
||||
const args = parseArgs(process.argv.slice(2));
|
||||
const root = process.cwd();
|
||||
if (!args.run && !args.from) {
|
||||
console.error('usage: promote.ts --run <ci-run-id> [--repo owner/repo] | --from <dir> [--prune] [--dry-run]');
|
||||
process.exitCode = 2;
|
||||
return;
|
||||
}
|
||||
const renderDir = args.from ? (args.from as string) : downloadRun(args.run as string, args.repo as string);
|
||||
const manifest = loadManifest(root);
|
||||
const { plan, apply } = buildPlan(root, renderDir, manifest);
|
||||
|
||||
console.log(
|
||||
`[promote] updated=${plan.updated.length} added=${plan.added.length} ` +
|
||||
`unchanged=${plan.unchanged.length} removal-candidates=${plan.removedCandidates.length}`
|
||||
);
|
||||
for (const n of plan.updated) console.log(` UPDATE ${n}`);
|
||||
for (const n of plan.added) console.log(` ADD ${n}`);
|
||||
for (const n of plan.removedCandidates) console.log(` REMOVE? ${n}${args.prune ? ' (pruning)' : ' (use --prune to delete)'}`);
|
||||
|
||||
if (args.dryRun) {
|
||||
console.log('[promote] dry-run — no files written');
|
||||
return;
|
||||
}
|
||||
apply();
|
||||
if (args.prune) {
|
||||
const baselines = baselineIndex(root);
|
||||
for (const n of plan.removedCandidates) fs.rmSync(baselines.get(n)!, { force: true });
|
||||
}
|
||||
console.log('[promote] done — review `git status` and commit the changed baselines');
|
||||
}
|
||||
|
||||
if (require.main === module) main();
|
||||
Loading…
Reference in a new issue