e2e/CI: dual-engine suites, per-engine screenshots, SwiftShader retired, prod web suite, CI-coverage gate
Squash of experiment/ff-big-modules vs main.
Big-module routing removed: native-EH shrank kicad_editor below
SpiderMonkey's x86-64 code budget (runs 29355049705/29356152413 green on
stock Firefox), so BIG_MODULE_SPECS routing and the baseline-only-JIT
crutch are gone — kicad-firefox and kicad-chromium both run the full
suite, with the module compiled the way real users' browsers compile it.
Per-engine screenshots end to end: stableShot/shotPath write
test-results/<engine>/<name>.png; baselines move to
baseline-screenshots/{chromium,firefox}/ and the whole tools/screenshots
pipeline (compare/promote/manifest/spec-map/changelog/Discord) keys on
<engine>/<name>. Previously Firefox and Chromium renders of one spec
overwrote each other and Firefox renders were never actually gated.
Seeded from CI run 29421380806 (92 new firefox baselines, +24 chromium
web-suite shots); manifest generated from the baseline tree.
One merged playwright.config.ts (kicad/asyncify/coroutine/perf as
projects); ~25 dead npm scripts dropped. The web suite is gated in CI for
the first time ever (4 rotted specs fixed, 5 broken lib-bridge specs
triaged as fixme in docs/features/web-e2e-rot/); cheap lint step after
npm ci; last 26 blind-sleep violations fixed.
SwiftShader retired: CI Chromium renders WebGL on ANGLE → Mesa llvmpipe
(--use-gl=angle --use-angle=gl --ignore-gpu-blocklist; the blocklist flag
is mandatory — llvmpipe is blocklisted and WebGL is silently unavailable
without it) in BOTH configs. Under WORKERS=4 congestion SwiftShader
transiently failed the first post-board-load draw and the recovery
cascade ended in a silent permanent Cairo fallback — that engine flip was
the "~1.2% changedRatio both directions" occ-export baseline flake.
Validated 160/160 across two 80-repeat rigs; full analysis in
docs/features/wx-parity-bugs/occ-export-context-eviction.md. Chromium
baselines shift slightly on llvmpipe — promote once from the first green
run. Deflakes the new coverage exposed: presence baselines settle before
capture; presence fixtures declare current file formats; perf gets its
own outputDir so CI evidence survives; occ-export settles the board paint
before the export dialog; menu-item waits (waitForRenderedByLabel before
clickMenuItem) in 4 specs + the TESTING.md rule.
Web suite runs the PROD build, in parallel: webServer becomes backend
`start` + the standalone's e2e:preview (build-preview.mjs: link-wasm →
stash the public/wasm symlink aside during vite build, build-demo.mjs's
move — then vite preview as the persistent server). The wasm middleware
serves /wasm/* in preview and emits COOP/COEP/CORP itself (a pthread
worker script's own response must carry COEP or Chrome kills it with
ERR_BLOCKED_BY_RESPONSE). VITE_* flags bake at build time;
VITE_ALLOW_USER_OVERRIDE joins turbo globalEnv. fullyParallel + default
workers: 5.2m → 1.4m. Determinism fixes the parallel run exposed:
shared-page specs become serial groups; locks.spec grabs alice's exact
item via the new kicadCollabTestSelectByUuid hook (cross-tab "first
footprint" order is not a ysync invariant); quit specs poll page.url()
(quit supersedes its own navigation — NS_BINDING_ABORTED on Firefox).
Suite: 51 passed / 12 skipped / 0 failed in 1.6m.
CI-coverage gate (lint:ci-coverage): every tests/**/*.spec.ts must be
reachable from the npm scripts the workflows invoke — scraped from
.github/workflows/, resolved through package.json, coverage asked from
playwright --list itself. Rules: uncovered-spec + orphan-project (with a
documented LOCAL_ONLY_PROJECTS allowlist). Gating next to
lint:determinism; 138 spec files / 13 projects accounted for.
Product fixes kept from the investigations (reachable on real GPUs too):
wx 7799fd1be5 — paint flags clear before dispatch + Invalidate always
propagates; kicad 3dcfea5e45 — SwiftShader pass-boundary flush +
per-instance font texture + first-frame GL-error drain (GAL recovery
recovers instead of falling back to Cairo) + the user-facing eeschema
switch navigates again under __EMSCRIPTEN__ (project-sync's
FaceRegistered gate had rerouted it into the hidden sync player; caught
by the newly-gated web suite).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018eUxiPApHgGiu9NFyQfhAq
This commit is contained in:
Viktor Vaczi
parent
8a20f69c5e
commit
63ed1f3c1f
556 changed files with 2470 additions and 1389 deletions
188
tests/tools/lint-ci-coverage.ts
Normal file
188
tests/tools/lint-ci-coverage.ts
Normal file
|
|
@ -0,0 +1,188 @@
|
|||
/**
|
||||
* CI-coverage guard: every spec file on disk must be RUN BY CI. Fails (exit 1)
|
||||
* if a test can be added without CI ever executing it — the failure mode that
|
||||
* let the web suite rot unrun for months. Run locally or in CI:
|
||||
*
|
||||
* npx tsx tools/lint-ci-coverage.ts # gate (exit 1 on any violation)
|
||||
* npm run lint:ci-coverage
|
||||
*
|
||||
* Both sides come from ground truth, no hand-maintained file lists:
|
||||
* - "what CI runs": `npm run test:…` invocations scraped from
|
||||
* .github/workflows/*.yml, resolved through package.json to their
|
||||
* `playwright test --config/--project` flags;
|
||||
* - "what that covers": `playwright test --list --reporter=json` with those
|
||||
* exact flags (CI=1), so testDir/testMatch/testIgnore/project semantics are
|
||||
* Playwright's own, never re-implemented here.
|
||||
*
|
||||
* Rules:
|
||||
* - uncovered-spec: a *.spec.ts under tests/ that no CI invocation lists.
|
||||
* - orphan-project: a project defined in a config that no CI script selects
|
||||
* and that is not explicitly allowlisted as local-only below.
|
||||
*/
|
||||
import { execFileSync } from 'child_process';
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
|
||||
const TESTS_ROOT = path.resolve(__dirname, '..');
|
||||
const REPO_ROOT = path.resolve(TESTS_ROOT, '..');
|
||||
const WORKFLOWS_DIR = path.join(REPO_ROOT, '.github', 'workflows');
|
||||
|
||||
// Deliberately-local projects (system browsers CI does not install, or manual
|
||||
// policy runs). A NEW project must either be selected by a CI-invoked script
|
||||
// or be added here on purpose — silence is exactly how the web suite rotted.
|
||||
const LOCAL_ONLY_PROJECTS = new Set([
|
||||
'kicad-chrome', // system Chrome, headed KiCad debugging
|
||||
'asyncify-chrome', // system Chrome
|
||||
'asyncify-webkit', // Safari-engine policy suite, run manually (test:asyncify:safari)
|
||||
'coroutine-chrome', // system Chrome (real V8/GPU)
|
||||
]);
|
||||
|
||||
// Dirs under tests/ that never contain source specs.
|
||||
const EXCLUDED_DIRS = new Set([
|
||||
'node_modules',
|
||||
'apps',
|
||||
'fixtures',
|
||||
'test-results',
|
||||
'pw-artifacts',
|
||||
'playwright-report',
|
||||
'logs',
|
||||
'baseline-screenshots',
|
||||
'3d-regression',
|
||||
'tools',
|
||||
'collab',
|
||||
'scripts',
|
||||
]);
|
||||
|
||||
// ── 1. what CI invokes ────────────────────────────────────────────────────────
|
||||
function ciTestScripts(): string[] {
|
||||
const names = new Set<string>();
|
||||
for (const f of fs.readdirSync(WORKFLOWS_DIR)) {
|
||||
if (!/\.ya?ml$/.test(f)) continue;
|
||||
const body = fs.readFileSync(path.join(WORKFLOWS_DIR, f), 'utf8');
|
||||
for (const m of body.matchAll(/npm run (test:[\w:.-]+)/g)) names.add(m[1]);
|
||||
}
|
||||
if (!names.size) {
|
||||
throw new Error(
|
||||
`no "npm run test:…" invocations found under ${WORKFLOWS_DIR} — ` +
|
||||
'either CI stopped running tests or this lint\'s scrape regex rotted'
|
||||
);
|
||||
}
|
||||
return [...names].sort();
|
||||
}
|
||||
|
||||
// ── 2. resolve scripts to playwright invocations ─────────────────────────────
|
||||
type Invocation = { script: string; config: string; projects: string[] };
|
||||
|
||||
function resolveScript(name: string): Invocation {
|
||||
const pkg = JSON.parse(fs.readFileSync(path.join(TESTS_ROOT, 'package.json'), 'utf8'));
|
||||
const body: string | undefined = pkg.scripts?.[name];
|
||||
if (!body) throw new Error(`CI invokes "npm run ${name}" but tests/package.json has no such script`);
|
||||
|
||||
const segment = body
|
||||
.split('&&')
|
||||
.map((s) => s.trim())
|
||||
.find((s) => /(^|\s)playwright test(\s|$)/.test(s));
|
||||
if (!segment) {
|
||||
throw new Error(
|
||||
`CI script "${name}" ("${body}") contains no "playwright test" segment — ` +
|
||||
'unknown runner; extend lint-ci-coverage.ts to understand it'
|
||||
);
|
||||
}
|
||||
|
||||
const config = segment.match(/--config=(\S+)/)?.[1] ?? 'playwright.config.ts';
|
||||
const projects = [...segment.matchAll(/--project=(\S+)/g)].map((m) => m[1]);
|
||||
return { script: name, config, projects };
|
||||
}
|
||||
|
||||
// ── 3. what those invocations cover ───────────────────────────────────────────
|
||||
type ListResult = { files: Set<string>; definedProjects: Set<string> };
|
||||
|
||||
function listInvocation(inv: Invocation): ListResult {
|
||||
const args = ['playwright', 'test', '--list', '--reporter=json', `--config=${inv.config}`];
|
||||
for (const p of inv.projects) args.push(`--project=${p}`);
|
||||
const out = execFileSync('npx', args, {
|
||||
cwd: TESTS_ROOT,
|
||||
env: { ...process.env, CI: '1' },
|
||||
encoding: 'utf8',
|
||||
maxBuffer: 64 * 1024 * 1024,
|
||||
});
|
||||
const report = JSON.parse(out);
|
||||
|
||||
// Reported file paths are relative to the config's rootDir (tests/ for the
|
||||
// merged config, tests/web for the web one) — normalize to tests/-relative.
|
||||
const rootDir: string = report.config?.rootDir ?? TESTS_ROOT;
|
||||
const normalize = (f: string) => path.relative(TESTS_ROOT, path.resolve(rootDir, f));
|
||||
|
||||
const files = new Set<string>();
|
||||
const walk = (suite: { file?: string; suites?: unknown[]; specs?: { file?: string }[] }) => {
|
||||
if (suite.file) files.add(normalize(suite.file));
|
||||
for (const spec of suite.specs ?? []) if (spec.file) files.add(normalize(spec.file));
|
||||
for (const sub of (suite.suites ?? []) as typeof suite[]) walk(sub);
|
||||
};
|
||||
for (const s of report.suites ?? []) walk(s);
|
||||
|
||||
const definedProjects = new Set<string>(
|
||||
(report.config?.projects ?? []).map((p: { name: string }) => p.name)
|
||||
);
|
||||
return { files, definedProjects };
|
||||
}
|
||||
|
||||
// ── 4. the universe of spec files on disk ─────────────────────────────────────
|
||||
function specUniverse(dir = TESTS_ROOT, rel = ''): string[] {
|
||||
const out: string[] = [];
|
||||
for (const entry of fs.readdirSync(dir, { withFileTypes: true })) {
|
||||
if (entry.isDirectory()) {
|
||||
if (EXCLUDED_DIRS.has(entry.name) || entry.name.startsWith('.')) continue;
|
||||
out.push(...specUniverse(path.join(dir, entry.name), path.join(rel, entry.name)));
|
||||
} else if (entry.name.endsWith('.spec.ts')) {
|
||||
out.push(path.join(rel, entry.name));
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
// ── run ───────────────────────────────────────────────────────────────────────
|
||||
const invocations = ciTestScripts().map(resolveScript);
|
||||
|
||||
const covered = new Set<string>();
|
||||
const defined = new Map<string, string>(); // project -> config that defines it
|
||||
const selected = new Set<string>();
|
||||
for (const inv of invocations) {
|
||||
const { files, definedProjects } = listInvocation(inv);
|
||||
for (const f of files) covered.add(f);
|
||||
for (const p of definedProjects) if (!defined.has(p)) defined.set(p, inv.config);
|
||||
for (const p of inv.projects) selected.add(p);
|
||||
}
|
||||
|
||||
const universe = specUniverse().sort();
|
||||
const violations: string[] = [];
|
||||
|
||||
for (const file of universe) {
|
||||
if (!covered.has(file)) {
|
||||
violations.push(
|
||||
`uncovered-spec: tests/${file} is not listed by any CI-invoked playwright run ` +
|
||||
`(scripts: ${invocations.map((i) => i.script).join(', ')})`
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
for (const [project, config] of [...defined.entries()].sort()) {
|
||||
if (!selected.has(project) && !LOCAL_ONLY_PROJECTS.has(project)) {
|
||||
violations.push(
|
||||
`orphan-project: "${project}" (${config}) is selected by no CI script and not in ` +
|
||||
'LOCAL_ONLY_PROJECTS — wire it into a CI npm script or allowlist it on purpose'
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
if (violations.length) {
|
||||
for (const v of violations) console.error(`✗ ${v}`);
|
||||
console.error(`\n${violations.length} CI-coverage violation(s)`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
console.log(
|
||||
`✓ CI coverage: ${universe.length} spec files all reachable via ` +
|
||||
`${invocations.map((i) => i.script).join(' + ')}; ` +
|
||||
`${defined.size} projects accounted for (${selected.size} on CI, ${LOCAL_ONLY_PROJECTS.size} local-only)`
|
||||
);
|
||||
|
|
@ -9,16 +9,23 @@ is intentional you *promote* CI's artifact into the committed baselines. The
|
|||
environment isn't pinned — if the host's Mesa/fonts drift, the gate lights up in
|
||||
Discord and you just re-promote (broad + low-intensity change ⇒ likely drift).
|
||||
|
||||
**Everything is per-engine.** Specs write `test-results/<engine>/<name>.png`
|
||||
(engine derived from the running browser via `stableShot`/`shotPath`); baselines
|
||||
live in `baseline-screenshots/<engine>/`; the canonical key everywhere in this
|
||||
tooling is `<engine>/<name>` and captions/attachments carry the engine label.
|
||||
The same spec on chromium + firefox is two independent gated screenshots.
|
||||
|
||||
## Files
|
||||
- `config.ts` — thresholds, baseline dirs, per-engine floors (calibrate!), clustering knobs.
|
||||
- `image-ops.ts` — PNG load/save, pixelmatch diff (AA-excluded), connected-component boxes, triptych compositing, size-cap resize.
|
||||
- `compare.ts` — the engine: classify baselines vs `test-results/` → `test-results/screenshot-diff/report.json` + triptych/heatmap PNGs. `--pair` diffs two files.
|
||||
- `compare.ts` — the comparison engine: classify per-engine baselines vs `test-results/<engine>/` → `test-results/screenshot-diff/report.json` + triptych/heatmap PNGs. `--pair` diffs two files.
|
||||
- `promote.ts` — churn-free updater: pull a CI run's shots (`gh run download`) or `--from DIR`; overwrite a baseline only when pixels differ beyond the floor (verbatim bytes, no re-encode) → no git churn.
|
||||
- `perf-report.ts` — renders the track-only runtime-perf table (loadMs/openMs/FPS) with Δ vs the previous main run (fetched via `gh`).
|
||||
- `post-discord.ts` — the always-on CI-on-main report: SHA + e2e status + perf table, then screenshot triptychs (batched, size-capped, flood-collapsed).
|
||||
- `changelog.ts` — Discord trigger B: git-history diff of committed baseline PNGs (no build/GPU).
|
||||
- `noise.ts` — calibration: diff two identical-input renders → per-engine noise floor.
|
||||
- `gen-manifest.ts` — regenerate `screenshot-manifest.json` (canonical name list + best-effort engine tag) by scanning the committed baselines + specs.
|
||||
- `gen-manifest.ts` — regenerate `screenshot-manifest.json` (canonical {name, engine} list) from the committed per-engine baseline tree; `--check` (gating in CI) fails if it drifts.
|
||||
- `spec-map.ts` — best-effort screenshot-name → spec-file attribution for captions (scans `stableShot`/`shotPath` literals).
|
||||
|
||||
## npm scripts (run from `tests/`)
|
||||
```
|
||||
|
|
@ -31,7 +38,7 @@ npm run screenshots:manifest # regenerate the manifest (-
|
|||
```
|
||||
|
||||
## Activation checklist
|
||||
- [x] `screenshot-manifest.json` generated (name list authoritative; engine tags best-effort until calibration).
|
||||
- [x] `screenshot-manifest.json` generated ({name, engine} authoritative — derived from the per-engine baseline tree).
|
||||
- [x] `scale:'device'`→`'css'` normalized (no-op at CI's DSF=1).
|
||||
1. Add the `DISCORD_WEBHOOK_URL` repo secret — until then everything is inert.
|
||||
2. Calibrate: run the suite twice in CI, `screenshots:noise` the two dirs, set `FLOORS` in `config.ts`.
|
||||
|
|
|
|||
|
|
@ -15,7 +15,7 @@ import * as os from 'os';
|
|||
import * as path from 'path';
|
||||
import { execFileSync } from 'child_process';
|
||||
import { PNG } from 'pngjs';
|
||||
import { BASELINE_DIRS, DIFF_OUT_DIR, floorFor, labelText, LABEL } from './config';
|
||||
import { BASELINE_ROOT, DIFF_OUT_DIR, floorFor, labelText, splitKey, LABEL } from './config';
|
||||
import { comparePair, type Report } from './compare';
|
||||
import { loadPng, savePng, withBottomLabel } from './image-ops';
|
||||
import { buildSpecResolver } from './spec-map';
|
||||
|
|
@ -60,9 +60,9 @@ async function main(): Promise<void> {
|
|||
}
|
||||
}
|
||||
|
||||
// Repo-relative baseline dir prefixes (e.g. tests/baseline-screenshots).
|
||||
const prefixes = BASELINE_DIRS.map((d) => path.relative(repoRoot, path.join(cwd, d)));
|
||||
const diff = git(repoRoot, ['diff', '--name-status', base, head, '--', ...prefixes]);
|
||||
// Repo-relative baseline root prefix (e.g. tests/baseline-screenshots).
|
||||
const prefix = path.relative(repoRoot, path.join(cwd, BASELINE_ROOT));
|
||||
const diff = git(repoRoot, ['diff', '--name-status', base, head, '--', prefix]);
|
||||
if (!diff) {
|
||||
console.log('[changelog] no baseline changes between', base.slice(0, 7), 'and', head);
|
||||
return;
|
||||
|
|
@ -89,12 +89,14 @@ async function main(): Promise<void> {
|
|||
fs.writeFileSync(f, b);
|
||||
return loadPng(f);
|
||||
};
|
||||
// Engine-qualified key (`<engine>/<name>.png`) from a repo path under the baseline root.
|
||||
const keyOf = (repoPath: string): string => path.relative(prefix, repoPath).split(path.sep).join('/');
|
||||
// Save a single captioned image (added/removed) and record it in the report.
|
||||
const saveSingle = (img: PNG, name: string, status: 'added' | 'removed'): void => {
|
||||
const saveSingle = (img: PNG, key: string, status: 'added' | 'removed'): void => {
|
||||
const suffix = status === 'added' ? 'added' : 'removed';
|
||||
const rel = path.join(DIFF_OUT_DIR, `${name}.${suffix}.png`);
|
||||
savePng(path.join(cwd, rel), withBottomLabel(img, labelText(status, name, specFor(name)), LABEL.colors[status]));
|
||||
report[status].push({ name, image: rel });
|
||||
const rel = path.join(DIFF_OUT_DIR, `${key.replace('/', '_')}.${suffix}.png`);
|
||||
savePng(path.join(cwd, rel), withBottomLabel(img, labelText(status, key, specFor(splitKey(key).name)), LABEL.colors[status]));
|
||||
report[status].push({ name: key, image: rel });
|
||||
};
|
||||
|
||||
for (const line of diff.split('\n')) {
|
||||
|
|
@ -102,28 +104,27 @@ async function main(): Promise<void> {
|
|||
const status = parts[0][0]; // M/A/D/R
|
||||
const repoPath = parts[status === 'R' ? 2 : 1];
|
||||
const oldPath = status === 'R' ? parts[1] : repoPath;
|
||||
const name = path.basename(repoPath);
|
||||
const key = keyOf(repoPath);
|
||||
|
||||
if (status === 'A' || status === 'R') {
|
||||
const img = blobPng(head, repoPath);
|
||||
if (img) saveSingle(img, name, 'added');
|
||||
if (img) saveSingle(img, key, 'added');
|
||||
if (status === 'R') {
|
||||
const oldName = path.basename(oldPath);
|
||||
const oldImg = blobPng(base, oldPath);
|
||||
if (oldImg) saveSingle(oldImg, oldName, 'removed');
|
||||
if (oldImg) saveSingle(oldImg, keyOf(oldPath), 'removed');
|
||||
}
|
||||
} else if (status === 'D') {
|
||||
const img = blobPng(base, repoPath);
|
||||
if (img) saveSingle(img, name, 'removed');
|
||||
if (img) saveSingle(img, key, 'removed');
|
||||
} else {
|
||||
// Modified: captioned triptych old vs new.
|
||||
const oldImg = blobPng(base, repoPath);
|
||||
const newImg = blobPng(head, repoPath);
|
||||
if (!oldImg || !newImg) continue;
|
||||
const { result, heatmap, triptych } = comparePair(oldImg, newImg, name, floorFor(name));
|
||||
const triptychRel = path.join(DIFF_OUT_DIR, `${name}.triptych.png`);
|
||||
const heatmapRel = path.join(DIFF_OUT_DIR, `${name}.heatmap.png`);
|
||||
savePng(path.join(cwd, triptychRel), withBottomLabel(triptych, labelText('changed', name, specFor(name)), LABEL.colors.changed));
|
||||
const { result, heatmap, triptych } = comparePair(oldImg, newImg, key, floorFor(key));
|
||||
const triptychRel = path.join(DIFF_OUT_DIR, `${key.replace('/', '_')}.triptych.png`);
|
||||
const heatmapRel = path.join(DIFF_OUT_DIR, `${key.replace('/', '_')}.heatmap.png`);
|
||||
savePng(path.join(cwd, triptychRel), withBottomLabel(triptych, labelText('changed', key, specFor(splitKey(key).name)), LABEL.colors.changed));
|
||||
savePng(path.join(cwd, heatmapRel), heatmap);
|
||||
report.changed.push({ ...result, triptych: triptychRel, heatmap: heatmapRel });
|
||||
}
|
||||
|
|
|
|||
|
|
@ -16,7 +16,7 @@ import * as fs from 'fs';
|
|||
import * as path from 'path';
|
||||
import { PNG } from 'pngjs';
|
||||
import {
|
||||
BASELINE_DIRS,
|
||||
BASELINE_ROOT,
|
||||
RESULTS_DIR,
|
||||
DIFF_OUT_DIR,
|
||||
MANIFEST_PATH,
|
||||
|
|
@ -25,6 +25,8 @@ import {
|
|||
floorFor,
|
||||
isIgnored,
|
||||
labelText,
|
||||
listEngineKeys,
|
||||
splitKey,
|
||||
LABEL,
|
||||
} from './config';
|
||||
import { diffImages, cluster, drawBoxes, composite, loadPng, savePng, withBottomLabel, type Box } from './image-ops';
|
||||
|
|
@ -90,24 +92,11 @@ export type Report = {
|
|||
|
||||
const DRIFT_BULK = 20;
|
||||
|
||||
function listPngs(dir: string): string[] {
|
||||
if (!fs.existsSync(dir)) return [];
|
||||
return fs.readdirSync(dir).filter((f) => f.toLowerCase().endsWith('.png'));
|
||||
}
|
||||
|
||||
/** basename → absolute baseline path (first BASELINE_DIRS entry wins; collisions warned). */
|
||||
/** key (`<engine>/<name>`) → absolute baseline path. */
|
||||
function baselineIndex(root: string): Map<string, string> {
|
||||
const abs = path.join(root, BASELINE_ROOT);
|
||||
const index = new Map<string, string>();
|
||||
for (const dir of BASELINE_DIRS) {
|
||||
const abs = path.join(root, dir);
|
||||
for (const name of listPngs(abs)) {
|
||||
if (index.has(name)) {
|
||||
console.warn(`[compare] duplicate baseline name ${name} (${dir} shadowed by earlier dir)`);
|
||||
continue;
|
||||
}
|
||||
index.set(name, path.join(abs, name));
|
||||
}
|
||||
}
|
||||
for (const key of listEngineKeys(abs)) index.set(key, path.join(abs, key));
|
||||
return index;
|
||||
}
|
||||
|
||||
|
|
@ -122,18 +111,23 @@ function loadManifest(root: string): Manifest | undefined {
|
|||
}
|
||||
}
|
||||
|
||||
/** Full gate run: classify baselines vs the current run's screenshots. */
|
||||
/** Flat, collision-free artifact filename for an engine-qualified key. */
|
||||
function artifactName(key: string, suffix: string): string {
|
||||
return `${key.replace('/', '_')}.${suffix}.png`;
|
||||
}
|
||||
|
||||
/** Full gate run: classify baselines vs the current run's screenshots, per engine. */
|
||||
export function classify(root: string, sha: string | null): Report {
|
||||
const baselines = baselineIndex(root);
|
||||
const resultsDir = path.join(root, RESULTS_DIR);
|
||||
const actuals = new Set(listPngs(resultsDir));
|
||||
const actuals = new Set(listEngineKeys(resultsDir));
|
||||
// Drop excluded screenshots from both sides so they're never compared or counted.
|
||||
for (const name of [...baselines.keys()]) if (isIgnored(name)) baselines.delete(name);
|
||||
for (const name of [...actuals]) if (isIgnored(name)) actuals.delete(name);
|
||||
for (const key of [...baselines.keys()]) if (isIgnored(key)) baselines.delete(key);
|
||||
for (const key of [...actuals]) if (isIgnored(key)) actuals.delete(key);
|
||||
const manifest = loadManifest(root);
|
||||
const outDir = path.join(root, DIFF_OUT_DIR);
|
||||
fs.mkdirSync(outDir, { recursive: true });
|
||||
const { specFor } = buildSpecResolver(root); // name → spec, for the caption strip
|
||||
const { specFor } = buildSpecResolver(root); // bare name → spec, for the caption strip
|
||||
|
||||
const report: Report = {
|
||||
generatedFor: sha,
|
||||
|
|
@ -145,45 +139,46 @@ export function classify(root: string, sha: string | null): Report {
|
|||
};
|
||||
|
||||
// Changed / unchanged / removed: iterate the committed baselines.
|
||||
for (const [name, baselinePath] of baselines) {
|
||||
if (!actuals.has(name)) {
|
||||
for (const [key, baselinePath] of baselines) {
|
||||
const { engine, name } = splitKey(key);
|
||||
if (!actuals.has(key)) {
|
||||
// Missing output. Only call it REMOVED when the manifest expects it — otherwise
|
||||
// a flaky/OOM'd/skipped spec that simply didn't write a PNG would masquerade as
|
||||
// an intentional removal. (The stronger "did the spec actually run" cross-check
|
||||
// against the Playwright JSON report lands with the manifest work.)
|
||||
if (manifest?.screenshots.some((e) => e.name === name)) {
|
||||
if (manifest?.screenshots.some((e) => e.name === name && e.engine === engine)) {
|
||||
// Removed now gets a captioned image (the old baseline) so it's visible in Discord.
|
||||
const imageRel = path.join(DIFF_OUT_DIR, `${name}.removed.png`);
|
||||
const labeled = withBottomLabel(loadPng(baselinePath), labelText('removed', name, specFor(name)), LABEL.colors.removed);
|
||||
const imageRel = path.join(DIFF_OUT_DIR, artifactName(key, 'removed'));
|
||||
const labeled = withBottomLabel(loadPng(baselinePath), labelText('removed', key, specFor(name)), LABEL.colors.removed);
|
||||
savePng(path.join(root, imageRel), labeled);
|
||||
report.removed.push({ name, image: imageRel });
|
||||
report.removed.push({ name: key, image: imageRel });
|
||||
}
|
||||
continue;
|
||||
}
|
||||
const { result, heatmap, triptych } = comparePair(
|
||||
loadPng(baselinePath),
|
||||
loadPng(path.join(resultsDir, name)),
|
||||
name,
|
||||
floorFor(name, manifest)
|
||||
loadPng(path.join(resultsDir, key)),
|
||||
key,
|
||||
floorFor(key)
|
||||
);
|
||||
if (result.verdict === 'unchanged') {
|
||||
report.unchangedCount++;
|
||||
continue;
|
||||
}
|
||||
const triptychRel = path.join(DIFF_OUT_DIR, `${name}.triptych.png`);
|
||||
const heatmapRel = path.join(DIFF_OUT_DIR, `${name}.heatmap.png`);
|
||||
savePng(path.join(root, triptychRel), withBottomLabel(triptych, labelText('changed', name, specFor(name)), LABEL.colors.changed));
|
||||
const triptychRel = path.join(DIFF_OUT_DIR, artifactName(key, 'triptych'));
|
||||
const heatmapRel = path.join(DIFF_OUT_DIR, artifactName(key, 'heatmap'));
|
||||
savePng(path.join(root, triptychRel), withBottomLabel(triptych, labelText('changed', key, specFor(name)), LABEL.colors.changed));
|
||||
savePng(path.join(root, heatmapRel), heatmap);
|
||||
report.changed.push({ ...result, triptych: triptychRel, heatmap: heatmapRel });
|
||||
}
|
||||
|
||||
// Added: an actual with no committed baseline.
|
||||
for (const name of actuals) {
|
||||
if (baselines.has(name)) continue;
|
||||
const imageRel = path.join(DIFF_OUT_DIR, `${name}.added.png`);
|
||||
const labeled = withBottomLabel(loadPng(path.join(resultsDir, name)), labelText('added', name, specFor(name)), LABEL.colors.added);
|
||||
for (const key of actuals) {
|
||||
if (baselines.has(key)) continue;
|
||||
const imageRel = path.join(DIFF_OUT_DIR, artifactName(key, 'added'));
|
||||
const labeled = withBottomLabel(loadPng(path.join(resultsDir, key)), labelText('added', key, specFor(splitKey(key).name)), LABEL.colors.added);
|
||||
savePng(path.join(root, imageRel), labeled);
|
||||
report.added.push({ name, image: imageRel });
|
||||
report.added.push({ name: key, image: imageRel });
|
||||
}
|
||||
|
||||
const driftLike = report.changed.filter((c) => c.driftHint === 'drift-like').length;
|
||||
|
|
|
|||
|
|
@ -8,13 +8,44 @@
|
|||
* Paths are relative to the `tests/` directory (that is the working directory
|
||||
* the npm scripts and CI steps run from).
|
||||
*/
|
||||
import * as nodeFs from 'fs';
|
||||
import * as nodePath from 'path';
|
||||
|
||||
/**
|
||||
* Committed baseline directory. Single dir on purpose: `e2e/baseline-screenshots/`
|
||||
* used to also hold a few names (grid-tab-final, wxgrid-*) that ALSO live here with
|
||||
* different bytes — so they got silently shadowed. Consolidated to one dir.
|
||||
* Committed baseline root. One PER-ENGINE subdirectory per browser family:
|
||||
* `baseline-screenshots/{chromium,firefox}/<name>.png`. Renders mirror the
|
||||
* layout (`test-results/<engine>/<name>.png`, written by stableShot/shotPath),
|
||||
* so each engine is gated against its OWN baselines — the old flat namespace
|
||||
* let two engines running the same spec overwrite each other's PNG, and the
|
||||
* surviving file was whichever parallel worker wrote last.
|
||||
*/
|
||||
export const BASELINE_DIRS = ['baseline-screenshots'] as const;
|
||||
export const BASELINE_ROOT = 'baseline-screenshots';
|
||||
|
||||
/** Engine subdirectories the tooling recognizes (a browser family per dir). */
|
||||
export const ENGINES = ['chromium', 'firefox', 'webkit'] as const;
|
||||
|
||||
/**
|
||||
* The canonical screenshot key is `<engine>/<name>.png` — the path of both the
|
||||
* render (under test-results/) and the baseline (under baseline-screenshots/)
|
||||
* relative to their root.
|
||||
*/
|
||||
export function splitKey(key: string): { engine: string; name: string } {
|
||||
const i = key.indexOf('/');
|
||||
return i < 0 ? { engine: 'unknown', name: key } : { engine: key.slice(0, i), name: key.slice(i + 1) };
|
||||
}
|
||||
|
||||
/** Engine-qualified keys (`<engine>/<name>.png`) found under a root's engine subdirs. */
|
||||
export function listEngineKeys(base: string): string[] {
|
||||
const keys: string[] = [];
|
||||
for (const engine of ENGINES) {
|
||||
const dir = nodePath.join(base, engine);
|
||||
if (!nodeFs.existsSync(dir)) continue;
|
||||
for (const f of nodeFs.readdirSync(dir)) {
|
||||
if (f.toLowerCase().endsWith('.png')) keys.push(`${engine}/${f}`);
|
||||
}
|
||||
}
|
||||
return keys;
|
||||
}
|
||||
|
||||
/** Where Playwright writes the current run's screenshots (gitignored). */
|
||||
export const RESULTS_DIR = 'test-results';
|
||||
|
|
@ -67,9 +98,10 @@ export const TRIPTYCH = {
|
|||
*/
|
||||
export type EngineFloor = { changedRatio: number; meanChannelGuard: number };
|
||||
|
||||
// Keyed by engine dir name (CI renderers: firefox = Mesa llvmpipe, chromium = SwiftShader).
|
||||
export const FLOORS: Record<string, EngineFloor> = {
|
||||
'firefox-llvmpipe': { changedRatio: 0.005, meanChannelGuard: 2.0 },
|
||||
'chromium-swiftshader': { changedRatio: 0.005, meanChannelGuard: 2.0 },
|
||||
firefox: { changedRatio: 0.005, meanChannelGuard: 2.0 },
|
||||
chromium: { changedRatio: 0.005, meanChannelGuard: 2.0 },
|
||||
default: { changedRatio: 0.005, meanChannelGuard: 2.0 },
|
||||
};
|
||||
|
||||
|
|
@ -88,9 +120,10 @@ export const IGNORE_REGIONS: Record<string, Array<{ x: number; y: number; width:
|
|||
*/
|
||||
export const IGNORE_SCREENSHOTS = new Set<string>(['retinascale-01-loaded.png']);
|
||||
|
||||
/** True if a screenshot is excluded from comparison. */
|
||||
export function isIgnored(name: string): boolean {
|
||||
return IGNORE_SCREENSHOTS.has(name);
|
||||
/** True if a screenshot is excluded from comparison. Matches the bare name, so an
|
||||
* ignored screenshot is ignored in every engine. */
|
||||
export function isIgnored(key: string): boolean {
|
||||
return IGNORE_SCREENSHOTS.has(splitKey(key).name);
|
||||
}
|
||||
|
||||
/** Bottom caption strip baked onto each posted composite (status + name + spec). */
|
||||
|
|
@ -109,17 +142,21 @@ export const LABEL = {
|
|||
|
||||
export type LabelStatus = 'added' | 'removed' | 'changed' | 'unchanged';
|
||||
|
||||
/** Caption text: `CHANGED name.png · kicad/pcbnew.spec.ts` (spec omitted if unknown). */
|
||||
export function labelText(status: LabelStatus, name: string, spec: string | null): string {
|
||||
/** Caption text: `CHANGED [chromium] name.png · kicad/pcbnew.spec.ts` (spec omitted if
|
||||
* unknown). The engine is PREPENDED — the auto-fit strip truncates the tail, so a
|
||||
* trailing tag could be cut off on long names. Accepts an engine-qualified key. */
|
||||
export function labelText(status: LabelStatus, key: string, spec: string | null): string {
|
||||
const s = status.toUpperCase();
|
||||
return spec ? `${s} ${name} · ${spec}` : `${s} ${name}`;
|
||||
const { engine, name } = splitKey(key);
|
||||
const head = `${s} [${engine}] ${name}`;
|
||||
return spec ? `${head} · ${spec}` : head;
|
||||
}
|
||||
|
||||
export type ManifestEntry = { name: string; engine: string };
|
||||
export type Manifest = { screenshots: ManifestEntry[] };
|
||||
|
||||
/** Resolve the verdict floor for a screenshot via the manifest's engine tag. */
|
||||
export function floorFor(name: string, manifest?: Manifest): EngineFloor {
|
||||
const engine = manifest?.screenshots.find((e) => e.name === name)?.engine;
|
||||
return (engine && FLOORS[engine]) || FLOORS.default;
|
||||
/** Verdict floor for an engine-qualified key — the engine IS the key prefix now,
|
||||
* no manifest lookup needed. */
|
||||
export function floorFor(key: string): EngineFloor {
|
||||
return FLOORS[splitKey(key).engine] || FLOORS.default;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,50 +1,48 @@
|
|||
/**
|
||||
* Generate tests/screenshot-manifest.json — the canonical list of expected
|
||||
* screenshots + the engine that renders each.
|
||||
* screenshots. Each entry is {name, engine}, read STRAIGHT from the per-engine
|
||||
* baseline tree (baseline-screenshots/<engine>/<name>.png) — the engine is the
|
||||
* directory, no spec-scanning guesswork.
|
||||
*
|
||||
* The NAME list is authoritative (it's the committed baseline set) and is what
|
||||
* lets compare/promote tell an intentional REMOVAL from a flaky/absent render.
|
||||
* The ENGINE tag is best-effort (spec-map.ts attributes each name to the spec that
|
||||
* writes it, and that spec's project/engine) and only feeds the per-engine floors.
|
||||
* The list is what lets compare/promote tell an intentional REMOVAL from a
|
||||
* flaky/absent render. promote.ts regenerates it automatically after applying;
|
||||
* --check keeps CI honest if baselines are ever edited by hand.
|
||||
*
|
||||
* CLI (from tests/): tsx tools/screenshots/gen-manifest.ts [--check]
|
||||
* --check exits 1 if the committed manifest is stale (for CI hygiene).
|
||||
*/
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
import { BASELINE_DIRS, MANIFEST_PATH, isIgnored, type Manifest } from './config';
|
||||
import { buildSpecResolver } from './spec-map';
|
||||
import { BASELINE_ROOT, MANIFEST_PATH, isIgnored, listEngineKeys, splitKey, type Manifest } from './config';
|
||||
|
||||
function listBaselines(root: string): string[] {
|
||||
const names = new Set<string>();
|
||||
for (const dir of BASELINE_DIRS) {
|
||||
const abs = path.join(root, dir);
|
||||
if (!fs.existsSync(abs)) continue;
|
||||
for (const f of fs.readdirSync(abs)) if (f.toLowerCase().endsWith('.png') && !isIgnored(f)) names.add(f);
|
||||
}
|
||||
return [...names].sort();
|
||||
/** Render the manifest JSON for the current baseline tree. */
|
||||
export function manifestJson(root: string): string {
|
||||
const screenshots = listEngineKeys(path.join(root, BASELINE_ROOT))
|
||||
.filter((key) => !isIgnored(key))
|
||||
.map((key) => splitKey(key))
|
||||
.map(({ engine, name }) => ({ name, engine }))
|
||||
.sort((a, b) => a.engine.localeCompare(b.engine) || a.name.localeCompare(b.name));
|
||||
|
||||
const manifest: Manifest & { _note: string } = {
|
||||
_note: 'Generated from baseline-screenshots/<engine>/ — run `npm run screenshots:manifest`. The name list is authoritative for removed-screenshot detection.',
|
||||
screenshots,
|
||||
};
|
||||
return JSON.stringify(manifest, null, 2) + '\n';
|
||||
}
|
||||
|
||||
/** Regenerate the manifest on disk (used by the CLI and by promote.ts after apply). */
|
||||
export function writeManifest(root: string): void {
|
||||
fs.writeFileSync(path.join(root, MANIFEST_PATH), manifestJson(root));
|
||||
}
|
||||
|
||||
function main(): void {
|
||||
const check = process.argv.includes('--check');
|
||||
const root = process.cwd();
|
||||
const resolver = buildSpecResolver(root);
|
||||
|
||||
let unmatched = 0;
|
||||
const screenshots = listBaselines(root).map((name) => {
|
||||
if (resolver.specFor(name) === null) unmatched++;
|
||||
return { name, engine: resolver.engineFor(name) };
|
||||
});
|
||||
|
||||
const manifest: Manifest & { _note: string } = {
|
||||
_note: 'engine tags are best-effort (spec-map.ts); the name list is authoritative. Refine engines after calibration.',
|
||||
screenshots,
|
||||
};
|
||||
const json = JSON.stringify(manifest, null, 2) + '\n';
|
||||
const json = manifestJson(root);
|
||||
const outPath = path.join(root, MANIFEST_PATH);
|
||||
const count = (JSON.parse(json) as Manifest).screenshots.length;
|
||||
|
||||
const dist = screenshots.reduce<Record<string, number>>((d, s) => ((d[s.engine] = (d[s.engine] ?? 0) + 1), d), {});
|
||||
console.log(`[manifest] ${screenshots.length} screenshots; engines=${JSON.stringify(dist)}; default-assigned=${unmatched}`);
|
||||
console.log(`[manifest] ${count} screenshots across engines`);
|
||||
|
||||
if (check) {
|
||||
const current = fs.existsSync(outPath) ? fs.readFileSync(outPath, 'utf8') : '';
|
||||
|
|
|
|||
|
|
@ -1,9 +1,10 @@
|
|||
/**
|
||||
* Calibration: measure the intra-CI noise floor.
|
||||
*
|
||||
* Render the suite twice on the same host (into two dirs), then:
|
||||
* Render the suite twice on the same host (into two dirs, each with per-engine
|
||||
* subdirs like test-results/), then:
|
||||
* tsx tools/screenshots/noise.ts <run1-dir> <run2-dir>
|
||||
* prints, per engine (via the manifest) and globally, the max / p95 / mean
|
||||
* prints, per engine (from the subdir) and globally, the max / p95 / mean
|
||||
* changed-pixel ratio between the two identical-input renders. Set
|
||||
* config.ts FLOORS.<engine>.changedRatio ≈ (max × 3) from this.
|
||||
*
|
||||
|
|
@ -13,17 +14,9 @@
|
|||
*/
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
import { MANIFEST_PATH, type Manifest } from './config';
|
||||
import { listEngineKeys, splitKey } from './config';
|
||||
import { diffImages, loadPng } from './image-ops';
|
||||
|
||||
function listPngs(dir: string): string[] {
|
||||
return fs.existsSync(dir) ? fs.readdirSync(dir).filter((f) => f.toLowerCase().endsWith('.png')) : [];
|
||||
}
|
||||
|
||||
function engineOf(name: string, manifest?: Manifest): string {
|
||||
return manifest?.screenshots.find((e) => e.name === name)?.engine ?? 'default';
|
||||
}
|
||||
|
||||
function stats(values: number[]): { n: number; max: number; p95: number; mean: number } {
|
||||
if (!values.length) return { n: 0, max: 0, p95: 0, mean: 0 };
|
||||
const sorted = [...values].sort((a, b) => a - b);
|
||||
|
|
@ -43,20 +36,14 @@ function main(): void {
|
|||
process.exitCode = 2;
|
||||
return;
|
||||
}
|
||||
const root = process.cwd();
|
||||
const manifestPath = path.join(root, MANIFEST_PATH);
|
||||
const manifest = fs.existsSync(manifestPath)
|
||||
? (JSON.parse(fs.readFileSync(manifestPath, 'utf8')) as Manifest)
|
||||
: undefined;
|
||||
|
||||
const common = listPngs(dir1).filter((n) => fs.existsSync(path.join(dir2, n)));
|
||||
const common = listEngineKeys(dir1).filter((k) => fs.existsSync(path.join(dir2, k)));
|
||||
const byEngine = new Map<string, number[]>();
|
||||
const all: number[] = [];
|
||||
for (const name of common) {
|
||||
const d = diffImages(loadPng(path.join(dir1, name)), loadPng(path.join(dir2, name)));
|
||||
for (const key of common) {
|
||||
const d = diffImages(loadPng(path.join(dir1, key)), loadPng(path.join(dir2, key)));
|
||||
const ratio = d.dimsMatch ? d.changedRatio : 1;
|
||||
all.push(ratio);
|
||||
const eng = engineOf(name, manifest);
|
||||
const eng = splitKey(key).engine;
|
||||
(byEngine.get(eng) ?? byEngine.set(eng, []).get(eng)!).push(ratio);
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -20,7 +20,7 @@
|
|||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
import { execFileSync } from 'child_process';
|
||||
import { DIFF_OUT_DIR } from './config';
|
||||
import { DIFF_OUT_DIR, splitKey } from './config';
|
||||
import { loadPng, encodeWithinCap } from './image-ops';
|
||||
import { buildPerfReport, fetchPreviousPerf } from './perf-report';
|
||||
import type { Report } from './compare';
|
||||
|
|
@ -120,13 +120,30 @@ function buildHeader(report: Report | null, perfBlock: string, meta: { sha?: str
|
|||
lines.push(e2eBadge(meta.e2e));
|
||||
if (perfBlock) lines.push('', perfBlock);
|
||||
|
||||
// Per-engine breakdown suffix, e.g. " (2 chromium / 1 firefox)"; single-engine
|
||||
// sets show just the engine name so the origin is always visible.
|
||||
const byEngine = (entries: Array<{ name: string }>): string => {
|
||||
if (!entries.length) return '';
|
||||
const counts = new Map<string, number>();
|
||||
for (const e of entries) {
|
||||
const eng = splitKey(e.name).engine;
|
||||
counts.set(eng, (counts.get(eng) ?? 0) + 1);
|
||||
}
|
||||
if (counts.size < 2) return ` (${[...counts.keys()][0]})`;
|
||||
return ` (${[...counts.entries()].map(([e, n]) => `${n} ${e}`).join(' / ')})`;
|
||||
};
|
||||
|
||||
lines.push('');
|
||||
if (!report || (!report.changed.length && !report.added.length && !report.removed.length)) {
|
||||
lines.push('✅ no screenshot drift');
|
||||
} else if (report.driftLikely) {
|
||||
lines.push(`⚠️ **${report.changed.length} screenshots changed broadly** — looks like host env drift → re-promote (\`npm run screenshots:promote\`)`);
|
||||
} else {
|
||||
lines.push(`⚠️ **screenshot drift**: ${report.changed.length} changed, ${report.added.length} added, ${report.removed.length} removed`);
|
||||
lines.push(
|
||||
`⚠️ **screenshot drift**: ${report.changed.length} changed${byEngine(report.changed)}, ` +
|
||||
`${report.added.length} added${byEngine(report.added)}, ` +
|
||||
`${report.removed.length} removed${byEngine(report.removed)}`
|
||||
);
|
||||
}
|
||||
if (report?.removed.length) {
|
||||
const shown = report.removed.slice(0, REMOVED_CAP).map((r) => `\`${r.name}\``).join(', ');
|
||||
|
|
|
|||
|
|
@ -16,22 +16,15 @@ import * as fs from 'fs';
|
|||
import * as os from 'os';
|
||||
import * as path from 'path';
|
||||
import { execFileSync } from 'child_process';
|
||||
import { BASELINE_DIRS, MANIFEST_PATH, floorFor, isIgnored, type Manifest } from './config';
|
||||
import { BASELINE_ROOT, MANIFEST_PATH, floorFor, isIgnored, listEngineKeys, splitKey, type Manifest } from './config';
|
||||
import { writeManifest } from './gen-manifest';
|
||||
import { diffImages, loadPng } from './image-ops';
|
||||
|
||||
function listPngs(dir: string): string[] {
|
||||
if (!fs.existsSync(dir)) return [];
|
||||
return fs.readdirSync(dir).filter((f) => f.toLowerCase().endsWith('.png'));
|
||||
}
|
||||
|
||||
/** basename → absolute committed baseline path (first BASELINE_DIRS entry wins). */
|
||||
/** key (`<engine>/<name>`) → absolute committed baseline path. */
|
||||
function baselineIndex(root: string): Map<string, string> {
|
||||
const abs = path.join(root, BASELINE_ROOT);
|
||||
const index = new Map<string, string>();
|
||||
for (const dir of BASELINE_DIRS) {
|
||||
for (const name of listPngs(path.join(root, dir))) {
|
||||
if (!index.has(name)) index.set(name, path.join(root, dir, name));
|
||||
}
|
||||
}
|
||||
for (const key of listEngineKeys(abs)) index.set(key, path.join(abs, key));
|
||||
return index;
|
||||
}
|
||||
|
||||
|
|
@ -66,37 +59,41 @@ type Plan = { updated: string[]; added: string[]; unchanged: string[]; removedCa
|
|||
|
||||
function buildPlan(root: string, renderDir: string, manifest?: Manifest): { plan: Plan; apply: () => void } {
|
||||
const baselines = baselineIndex(root);
|
||||
const rendered = new Set(listPngs(renderDir));
|
||||
const rendered = new Set(listEngineKeys(renderDir));
|
||||
// Never promote excluded screenshots (e.g. the flaky retinascale fullPage shot).
|
||||
for (const name of [...rendered]) if (isIgnored(name)) rendered.delete(name);
|
||||
for (const name of [...baselines.keys()]) if (isIgnored(name)) baselines.delete(name);
|
||||
for (const key of [...rendered]) if (isIgnored(key)) rendered.delete(key);
|
||||
for (const key of [...baselines.keys()]) if (isIgnored(key)) baselines.delete(key);
|
||||
const plan: Plan = { updated: [], added: [], unchanged: [], removedCandidates: [] };
|
||||
const actions: Array<() => void> = [];
|
||||
|
||||
for (const name of rendered) {
|
||||
const src = path.join(renderDir, name);
|
||||
const existing = baselines.get(name);
|
||||
for (const key of rendered) {
|
||||
const src = path.join(renderDir, key);
|
||||
const existing = baselines.get(key);
|
||||
if (!existing) {
|
||||
const dest = path.join(root, BASELINE_DIRS[0], name);
|
||||
plan.added.push(name);
|
||||
actions.push(() => fs.copyFileSync(src, dest)); // verbatim bytes
|
||||
const dest = path.join(root, BASELINE_ROOT, key);
|
||||
plan.added.push(key);
|
||||
actions.push(() => {
|
||||
fs.mkdirSync(path.dirname(dest), { recursive: true });
|
||||
fs.copyFileSync(src, dest); // verbatim bytes
|
||||
});
|
||||
continue;
|
||||
}
|
||||
const d = diffImages(loadPng(existing), loadPng(src));
|
||||
const floor = floorFor(name, manifest);
|
||||
const floor = floorFor(key);
|
||||
if (!d.dimsMatch || d.changedRatio > floor.changedRatio) {
|
||||
plan.updated.push(name);
|
||||
plan.updated.push(key);
|
||||
actions.push(() => fs.copyFileSync(src, existing)); // verbatim bytes, no re-encode → no churn
|
||||
} else {
|
||||
plan.unchanged.push(name); // leave the committed file untouched
|
||||
plan.unchanged.push(key); // leave the committed file untouched
|
||||
}
|
||||
}
|
||||
|
||||
// Removal candidates: a committed baseline the manifest expects but this render didn't produce.
|
||||
for (const [name, abs] of baselines) {
|
||||
if (rendered.has(name)) continue;
|
||||
if (manifest && !manifest.screenshots.some((e) => e.name === name)) continue;
|
||||
plan.removedCandidates.push(name);
|
||||
for (const [key, abs] of baselines) {
|
||||
if (rendered.has(key)) continue;
|
||||
const { engine, name } = splitKey(key);
|
||||
if (manifest && !manifest.screenshots.some((e) => e.name === name && e.engine === engine)) continue;
|
||||
plan.removedCandidates.push(key);
|
||||
actions.push(() => {}); // pruning is opt-in (see main)
|
||||
void abs;
|
||||
}
|
||||
|
|
@ -146,7 +143,10 @@ function main(): void {
|
|||
const baselines = baselineIndex(root);
|
||||
for (const n of plan.removedCandidates) fs.rmSync(baselines.get(n)!, { force: true });
|
||||
}
|
||||
console.log('[promote] done — review `git status` and commit the changed baselines');
|
||||
// Keep the manifest in lockstep with the baseline tree — a stale manifest silently
|
||||
// disables removed-screenshot detection for anything added after the last regen.
|
||||
writeManifest(root);
|
||||
console.log('[promote] done (manifest regenerated) — review `git status` and commit');
|
||||
}
|
||||
|
||||
if (require.main === module) main();
|
||||
|
|
|
|||
|
|
@ -1,8 +1,12 @@
|
|||
/**
|
||||
* Resolve a screenshot name → the spec that writes it (and the engine that renders
|
||||
* it), by scanning the specs for `page.screenshot({ path: 'test-results/<prefix>…' })`
|
||||
* literals. Shared by gen-manifest.ts (engine tags) and compare/changelog (caption
|
||||
* labels).
|
||||
* Resolve a screenshot name → the spec that writes it, by scanning the specs for
|
||||
* screenshot-name literals. Used by compare/changelog for the caption strip only —
|
||||
* the ENGINE of a screenshot is the directory it lives in (test-results/<engine>/,
|
||||
* baseline-screenshots/<engine>/), never guessed from specs.
|
||||
*
|
||||
* Two literal shapes are scanned:
|
||||
* - stableShot(page, 'name.png' …) / shotPath(page, 'name.png') — the standard writers
|
||||
* - 'test-results/<prefix>' — legacy direct paths
|
||||
*
|
||||
* Attribution is longest-prefix (`pcbnew-loaded` → the spec that writes `pcbnew-…`).
|
||||
* Names written only by a helper (e.g. `wizard-*` via completeWizard, which lives in
|
||||
|
|
@ -12,10 +16,6 @@
|
|||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
|
||||
export const CHROMIUM = 'chromium-swiftshader';
|
||||
export const FIREFOX = 'firefox-llvmpipe';
|
||||
export const DEFAULT_ENGINE = CHROMIUM; // baseline-screenshots is dominated by the wx suite
|
||||
|
||||
function listSpecs(dir: string): string[] {
|
||||
const out: string[] = [];
|
||||
if (!fs.existsSync(dir)) return out;
|
||||
|
|
@ -28,54 +28,38 @@ function listSpecs(dir: string): string[] {
|
|||
return out;
|
||||
}
|
||||
|
||||
/** pcbnew-family spec basenames (routed to chromium-ci on CI), read from the config. */
|
||||
function pcbnewFamily(root: string): Set<string> {
|
||||
const cfg = fs.readFileSync(path.join(root, 'playwright-kicad.config.ts'), 'utf8');
|
||||
const block = cfg.match(/PCBNEW_FAMILY_SPECS\s*=\s*\[([\s\S]*?)\]/)?.[1] ?? '';
|
||||
return new Set([...block.matchAll(/'[^']*?([\w.-]+\.spec\.ts)'/g)].map((m) => m[1]));
|
||||
}
|
||||
type PrefixEntry = { prefix: string; spec: string };
|
||||
|
||||
function engineForSpec(root: string, specPath: string, family: Set<string>): string {
|
||||
const rel = path.relative(root, specPath);
|
||||
const base = path.basename(specPath);
|
||||
if (rel.startsWith('e2e/')) return CHROMIUM;
|
||||
if (rel.startsWith('web/')) return FIREFOX;
|
||||
if (rel.startsWith('kicad/')) return family.has(base) ? CHROMIUM : FIREFOX;
|
||||
return DEFAULT_ENGINE;
|
||||
}
|
||||
|
||||
type PrefixEntry = { prefix: string; engine: string; spec: string };
|
||||
|
||||
/** prefix → {engine, spec-rel-to-tests}, longest-prefix first. */
|
||||
function prefixMap(root: string, family: Set<string>): PrefixEntry[] {
|
||||
const map = new Map<string, { engine: string; spec: string }>();
|
||||
/** prefix → spec-rel-to-tests, longest-prefix first. */
|
||||
function prefixMap(root: string): PrefixEntry[] {
|
||||
const map = new Map<string, string>();
|
||||
for (const dir of ['e2e', 'kicad', 'web']) {
|
||||
for (const spec of listSpecs(path.join(root, dir))) {
|
||||
const engine = engineForSpec(root, spec, family);
|
||||
const rel = path.relative(root, spec); // e.g. 'kicad/pcbnew.spec.ts'
|
||||
const content = fs.readFileSync(spec, 'utf8');
|
||||
for (const m of content.matchAll(/test-results\/([A-Za-z0-9_-]+)/g)) {
|
||||
const prefix = m[1];
|
||||
if (!map.has(prefix)) map.set(prefix, { engine, spec: rel }); // first writer wins
|
||||
}
|
||||
const add = (prefix: string) => {
|
||||
if (prefix && !map.has(prefix)) map.set(prefix, rel); // first writer wins
|
||||
};
|
||||
// stableShot(page, 'name.png') / shotPath(page, `name-${x}.png`)
|
||||
for (const m of content.matchAll(/(?:stableShot|shotPath)\(\s*\w+,\s*[`']([A-Za-z0-9_-]+)/g)) add(m[1]);
|
||||
// legacy direct 'test-results/<prefix>' literals
|
||||
for (const m of content.matchAll(/test-results\/([A-Za-z0-9_-]+)/g)) add(m[1]);
|
||||
}
|
||||
}
|
||||
return [...map.entries()]
|
||||
.map(([prefix, v]) => ({ prefix, ...v }))
|
||||
.map(([prefix, spec]) => ({ prefix, spec }))
|
||||
.sort((a, b) => b.prefix.length - a.prefix.length);
|
||||
}
|
||||
|
||||
export type SpecResolver = { specFor(name: string): string | null; engineFor(name: string): string };
|
||||
export type SpecResolver = { specFor(name: string): string | null };
|
||||
|
||||
/** Build a name→spec/engine resolver by scanning the specs under `root` once. */
|
||||
/** Build a name→spec resolver by scanning the specs under `root` once. */
|
||||
export function buildSpecResolver(root: string): SpecResolver {
|
||||
const prefixes = prefixMap(root, pcbnewFamily(root));
|
||||
const match = (name: string): PrefixEntry | undefined => {
|
||||
const stem = name.replace(/\.png$/i, '');
|
||||
return prefixes.find((p) => stem === p.prefix || stem.startsWith(p.prefix));
|
||||
};
|
||||
const prefixes = prefixMap(root);
|
||||
return {
|
||||
specFor: (name) => match(name)?.spec ?? null,
|
||||
engineFor: (name) => match(name)?.engine ?? DEFAULT_ENGINE,
|
||||
specFor: (name) => {
|
||||
const stem = name.replace(/\.png$/i, '');
|
||||
return prefixes.find((p) => stem === p.prefix || stem.startsWith(p.prefix))?.spec ?? null;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in a new issue