pcbjam/tests/tools/screenshots/gen-manifest.ts

63 lines
2.5 KiB
TypeScript
Raw Normal View History

feat(tests): screenshot regression + Discord review, perf-tracked New tooling in tests/tools/screenshots/ (TypeScript via tsx): - compare.ts: one pixelmatch engine (AA-excluded), connected-component "where to look" boxes, old|new+boxes|heatmap triptych, per-engine floors. - promote.ts: churn-free updater — overwrite a baseline only when decoded pixels differ beyond the floor, copying CI bytes verbatim (no re-encode churn); pulls a CI run via `gh run download` or a local --from dir. - post-discord.ts: always-on CI-on-main report (SHA + e2e status + the track-only runtime-perf table), then screenshot triptychs, batched + size-capped + flood-collapsed + 429-aware. - perf-report.ts: perf table with Δ vs the previous main run (via gh). - changelog.ts: no-build git-history baseline differ (Discord trigger B). - noise.ts / gen-manifest.ts: calibration + manifest generation. CI wiring: - wasm-build.yml: post-test step runs the gate + report on the already- produced test-results (no extra build); report-only (continue-on-error), posts only on push to main, inert without DISCORD_WEBHOOK_URL. - ci-ubicloud.yml: secrets: inherit (pass the webhook through). - screenshot-changelog.yml: ~30s no-build changelog on baseline changes. screenshot-manifest.json: canonical 354-name set + best-effort engine tags (313 chromium-swiftshader / 41 firefox-llvmpipe). Normalize scale:'device'->'css' across 18 spec files (no-op at CI DSF=1) so committed baselines are uniformly css-scaled. Design: CI's Linux render is the single source of truth; no pinned container (accept rare env drift -> re-promote); dev commits via promote. Replaces the byte-cmp compare-screenshots.sh + file-size-proxy update-baseline-screenshots.sh (kept for now until the first re-baseline). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 18:01:15 +02:00
/**
* Generate tests/screenshot-manifest.json — the canonical list of expected
* screenshots + the engine that renders each.
*
* The NAME list is authoritative (it's the committed baseline set) and is what
* lets compare/promote tell an intentional REMOVAL from a flaky/absent render.
* The ENGINE tag is best-effort (spec-map.ts attributes each name to the spec that
* writes it, and that spec's project/engine) and only feeds the per-engine floors.
feat(tests): screenshot regression + Discord review, perf-tracked New tooling in tests/tools/screenshots/ (TypeScript via tsx): - compare.ts: one pixelmatch engine (AA-excluded), connected-component "where to look" boxes, old|new+boxes|heatmap triptych, per-engine floors. - promote.ts: churn-free updater — overwrite a baseline only when decoded pixels differ beyond the floor, copying CI bytes verbatim (no re-encode churn); pulls a CI run via `gh run download` or a local --from dir. - post-discord.ts: always-on CI-on-main report (SHA + e2e status + the track-only runtime-perf table), then screenshot triptychs, batched + size-capped + flood-collapsed + 429-aware. - perf-report.ts: perf table with Δ vs the previous main run (via gh). - changelog.ts: no-build git-history baseline differ (Discord trigger B). - noise.ts / gen-manifest.ts: calibration + manifest generation. CI wiring: - wasm-build.yml: post-test step runs the gate + report on the already- produced test-results (no extra build); report-only (continue-on-error), posts only on push to main, inert without DISCORD_WEBHOOK_URL. - ci-ubicloud.yml: secrets: inherit (pass the webhook through). - screenshot-changelog.yml: ~30s no-build changelog on baseline changes. screenshot-manifest.json: canonical 354-name set + best-effort engine tags (313 chromium-swiftshader / 41 firefox-llvmpipe). Normalize scale:'device'->'css' across 18 spec files (no-op at CI DSF=1) so committed baselines are uniformly css-scaled. Design: CI's Linux render is the single source of truth; no pinned container (accept rare env drift -> re-promote); dev commits via promote. Replaces the byte-cmp compare-screenshots.sh + file-size-proxy update-baseline-screenshots.sh (kept for now until the first re-baseline). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 18:01:15 +02:00
*
* CLI (from tests/): tsx tools/screenshots/gen-manifest.ts [--check]
* --check exits 1 if the committed manifest is stale (for CI hygiene).
*/
import * as fs from 'fs';
import * as path from 'path';
import { BASELINE_DIRS, MANIFEST_PATH, isIgnored, type Manifest } from './config';
import { buildSpecResolver } from './spec-map';
feat(tests): screenshot regression + Discord review, perf-tracked New tooling in tests/tools/screenshots/ (TypeScript via tsx): - compare.ts: one pixelmatch engine (AA-excluded), connected-component "where to look" boxes, old|new+boxes|heatmap triptych, per-engine floors. - promote.ts: churn-free updater — overwrite a baseline only when decoded pixels differ beyond the floor, copying CI bytes verbatim (no re-encode churn); pulls a CI run via `gh run download` or a local --from dir. - post-discord.ts: always-on CI-on-main report (SHA + e2e status + the track-only runtime-perf table), then screenshot triptychs, batched + size-capped + flood-collapsed + 429-aware. - perf-report.ts: perf table with Δ vs the previous main run (via gh). - changelog.ts: no-build git-history baseline differ (Discord trigger B). - noise.ts / gen-manifest.ts: calibration + manifest generation. CI wiring: - wasm-build.yml: post-test step runs the gate + report on the already- produced test-results (no extra build); report-only (continue-on-error), posts only on push to main, inert without DISCORD_WEBHOOK_URL. - ci-ubicloud.yml: secrets: inherit (pass the webhook through). - screenshot-changelog.yml: ~30s no-build changelog on baseline changes. screenshot-manifest.json: canonical 354-name set + best-effort engine tags (313 chromium-swiftshader / 41 firefox-llvmpipe). Normalize scale:'device'->'css' across 18 spec files (no-op at CI DSF=1) so committed baselines are uniformly css-scaled. Design: CI's Linux render is the single source of truth; no pinned container (accept rare env drift -> re-promote); dev commits via promote. Replaces the byte-cmp compare-screenshots.sh + file-size-proxy update-baseline-screenshots.sh (kept for now until the first re-baseline). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 18:01:15 +02:00
function listBaselines(root: string): string[] {
const names = new Set<string>();
for (const dir of BASELINE_DIRS) {
const abs = path.join(root, dir);
if (!fs.existsSync(abs)) continue;
for (const f of fs.readdirSync(abs)) if (f.toLowerCase().endsWith('.png') && !isIgnored(f)) names.add(f);
feat(tests): screenshot regression + Discord review, perf-tracked New tooling in tests/tools/screenshots/ (TypeScript via tsx): - compare.ts: one pixelmatch engine (AA-excluded), connected-component "where to look" boxes, old|new+boxes|heatmap triptych, per-engine floors. - promote.ts: churn-free updater — overwrite a baseline only when decoded pixels differ beyond the floor, copying CI bytes verbatim (no re-encode churn); pulls a CI run via `gh run download` or a local --from dir. - post-discord.ts: always-on CI-on-main report (SHA + e2e status + the track-only runtime-perf table), then screenshot triptychs, batched + size-capped + flood-collapsed + 429-aware. - perf-report.ts: perf table with Δ vs the previous main run (via gh). - changelog.ts: no-build git-history baseline differ (Discord trigger B). - noise.ts / gen-manifest.ts: calibration + manifest generation. CI wiring: - wasm-build.yml: post-test step runs the gate + report on the already- produced test-results (no extra build); report-only (continue-on-error), posts only on push to main, inert without DISCORD_WEBHOOK_URL. - ci-ubicloud.yml: secrets: inherit (pass the webhook through). - screenshot-changelog.yml: ~30s no-build changelog on baseline changes. screenshot-manifest.json: canonical 354-name set + best-effort engine tags (313 chromium-swiftshader / 41 firefox-llvmpipe). Normalize scale:'device'->'css' across 18 spec files (no-op at CI DSF=1) so committed baselines are uniformly css-scaled. Design: CI's Linux render is the single source of truth; no pinned container (accept rare env drift -> re-promote); dev commits via promote. Replaces the byte-cmp compare-screenshots.sh + file-size-proxy update-baseline-screenshots.sh (kept for now until the first re-baseline). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 18:01:15 +02:00
}
return [...names].sort();
}
function main(): void {
const check = process.argv.includes('--check');
const root = process.cwd();
const resolver = buildSpecResolver(root);
feat(tests): screenshot regression + Discord review, perf-tracked New tooling in tests/tools/screenshots/ (TypeScript via tsx): - compare.ts: one pixelmatch engine (AA-excluded), connected-component "where to look" boxes, old|new+boxes|heatmap triptych, per-engine floors. - promote.ts: churn-free updater — overwrite a baseline only when decoded pixels differ beyond the floor, copying CI bytes verbatim (no re-encode churn); pulls a CI run via `gh run download` or a local --from dir. - post-discord.ts: always-on CI-on-main report (SHA + e2e status + the track-only runtime-perf table), then screenshot triptychs, batched + size-capped + flood-collapsed + 429-aware. - perf-report.ts: perf table with Δ vs the previous main run (via gh). - changelog.ts: no-build git-history baseline differ (Discord trigger B). - noise.ts / gen-manifest.ts: calibration + manifest generation. CI wiring: - wasm-build.yml: post-test step runs the gate + report on the already- produced test-results (no extra build); report-only (continue-on-error), posts only on push to main, inert without DISCORD_WEBHOOK_URL. - ci-ubicloud.yml: secrets: inherit (pass the webhook through). - screenshot-changelog.yml: ~30s no-build changelog on baseline changes. screenshot-manifest.json: canonical 354-name set + best-effort engine tags (313 chromium-swiftshader / 41 firefox-llvmpipe). Normalize scale:'device'->'css' across 18 spec files (no-op at CI DSF=1) so committed baselines are uniformly css-scaled. Design: CI's Linux render is the single source of truth; no pinned container (accept rare env drift -> re-promote); dev commits via promote. Replaces the byte-cmp compare-screenshots.sh + file-size-proxy update-baseline-screenshots.sh (kept for now until the first re-baseline). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 18:01:15 +02:00
let unmatched = 0;
const screenshots = listBaselines(root).map((name) => {
if (resolver.specFor(name) === null) unmatched++;
return { name, engine: resolver.engineFor(name) };
feat(tests): screenshot regression + Discord review, perf-tracked New tooling in tests/tools/screenshots/ (TypeScript via tsx): - compare.ts: one pixelmatch engine (AA-excluded), connected-component "where to look" boxes, old|new+boxes|heatmap triptych, per-engine floors. - promote.ts: churn-free updater — overwrite a baseline only when decoded pixels differ beyond the floor, copying CI bytes verbatim (no re-encode churn); pulls a CI run via `gh run download` or a local --from dir. - post-discord.ts: always-on CI-on-main report (SHA + e2e status + the track-only runtime-perf table), then screenshot triptychs, batched + size-capped + flood-collapsed + 429-aware. - perf-report.ts: perf table with Δ vs the previous main run (via gh). - changelog.ts: no-build git-history baseline differ (Discord trigger B). - noise.ts / gen-manifest.ts: calibration + manifest generation. CI wiring: - wasm-build.yml: post-test step runs the gate + report on the already- produced test-results (no extra build); report-only (continue-on-error), posts only on push to main, inert without DISCORD_WEBHOOK_URL. - ci-ubicloud.yml: secrets: inherit (pass the webhook through). - screenshot-changelog.yml: ~30s no-build changelog on baseline changes. screenshot-manifest.json: canonical 354-name set + best-effort engine tags (313 chromium-swiftshader / 41 firefox-llvmpipe). Normalize scale:'device'->'css' across 18 spec files (no-op at CI DSF=1) so committed baselines are uniformly css-scaled. Design: CI's Linux render is the single source of truth; no pinned container (accept rare env drift -> re-promote); dev commits via promote. Replaces the byte-cmp compare-screenshots.sh + file-size-proxy update-baseline-screenshots.sh (kept for now until the first re-baseline). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 18:01:15 +02:00
});
const manifest: Manifest & { _note: string } = {
_note: 'engine tags are best-effort (spec-map.ts); the name list is authoritative. Refine engines after calibration.',
feat(tests): screenshot regression + Discord review, perf-tracked New tooling in tests/tools/screenshots/ (TypeScript via tsx): - compare.ts: one pixelmatch engine (AA-excluded), connected-component "where to look" boxes, old|new+boxes|heatmap triptych, per-engine floors. - promote.ts: churn-free updater — overwrite a baseline only when decoded pixels differ beyond the floor, copying CI bytes verbatim (no re-encode churn); pulls a CI run via `gh run download` or a local --from dir. - post-discord.ts: always-on CI-on-main report (SHA + e2e status + the track-only runtime-perf table), then screenshot triptychs, batched + size-capped + flood-collapsed + 429-aware. - perf-report.ts: perf table with Δ vs the previous main run (via gh). - changelog.ts: no-build git-history baseline differ (Discord trigger B). - noise.ts / gen-manifest.ts: calibration + manifest generation. CI wiring: - wasm-build.yml: post-test step runs the gate + report on the already- produced test-results (no extra build); report-only (continue-on-error), posts only on push to main, inert without DISCORD_WEBHOOK_URL. - ci-ubicloud.yml: secrets: inherit (pass the webhook through). - screenshot-changelog.yml: ~30s no-build changelog on baseline changes. screenshot-manifest.json: canonical 354-name set + best-effort engine tags (313 chromium-swiftshader / 41 firefox-llvmpipe). Normalize scale:'device'->'css' across 18 spec files (no-op at CI DSF=1) so committed baselines are uniformly css-scaled. Design: CI's Linux render is the single source of truth; no pinned container (accept rare env drift -> re-promote); dev commits via promote. Replaces the byte-cmp compare-screenshots.sh + file-size-proxy update-baseline-screenshots.sh (kept for now until the first re-baseline). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 18:01:15 +02:00
screenshots,
};
const json = JSON.stringify(manifest, null, 2) + '\n';
const outPath = path.join(root, MANIFEST_PATH);
const dist = screenshots.reduce<Record<string, number>>((d, s) => ((d[s.engine] = (d[s.engine] ?? 0) + 1), d), {});
console.log(`[manifest] ${screenshots.length} screenshots; engines=${JSON.stringify(dist)}; default-assigned=${unmatched}`);
if (check) {
const current = fs.existsSync(outPath) ? fs.readFileSync(outPath, 'utf8') : '';
if (current !== json) {
console.error('[manifest] STALE — run `npm run screenshots:manifest` and commit');
process.exitCode = 1;
} else {
console.log('[manifest] up to date');
}
return;
}
fs.writeFileSync(outPath, json);
console.log(`[manifest] wrote ${MANIFEST_PATH}`);
}
if (require.main === module) main();