AI_Assistant/pcbjam
Watch
1
0
Fork
You've already forked pcbjam
 0
Code Issues Pull requests Projects Releases 7 Packages Wiki Activity Actions

tests: move screenshot baselines from git to R2 (content-addressed manifest v2)

Browse source 
Baseline PNGs (492, 35 MB) leave git: they now live in the private R2 bucket
pcbjam-ci-screenshots as immutable sha256/<hex>.png objects, pinned by the
committed screenshot-manifest.json (v2: name, engine, sha256, bytes, dims).
tests/baseline-screenshots/ becomes a gitignored cache materialized by the new
`npm run screenshots:fetch` (r2-sync.ts + aws4fetch r2-store.ts).

- promote.ts: requires the RW keypair (shell env or gitignored tests/.env),
  syncs the cache, uploads new hashes BEFORE rewriting the manifest; the
  manifest diff is the only git-visible output. --prune only edits the
  manifest — R2 objects are never deleted, old commits still resolve.
- compare.ts: skips the gate (exit 0, no report.json) when the manifest
  expects baselines but the cache is empty (secretless callers).
- changelog.ts: diffs the manifest between revs and fetches bytes from R2;
  guards against the migration commit (base manifest not v2 → skip).
- gen-manifest --check: v2 schema + resurrection guard (fails if baseline
  PNGs are ever re-committed); credential-free so every caller can gate.
- wasm-build.yml: declares optional read-only S3 secrets, caches + fetches
  baselines before the lint gate; release.yml/deploy-staging.yml pass them.
- screenshot-changelog.yml: triggers on the manifest path instead of PNGs.

All 492 objects are seeded and hash-verified in the bucket; fetch/compare
degrade to a warn-and-skip without credentials.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Istvan Matejcsok 2026-08-18 16:01:46 +02:00
commit 6dfa6ebd15
511 changed files with 3214 additions and 617 deletions
Download patch file Download diff file Expand all files Collapse all files

41
tests/tools/screenshots/README.md
View file

@ -5,9 +5,35 @@ screenshots. Design and rationale: `~/.claude/plans/…snowglobe.md` (or ask).
**Source of truth = CI's Linux render.** The dev never authors baselines on the
Mac (Mac fonts/GL ≠ CI). Instead, CI renders on every push; when a render change
is intentional you *promote* CI's artifact into the committed baselines. The
environment isn't pinned — if the host's Mesa/fonts drift, the gate lights up in
Discord and you just re-promote (broad + low-intensity change ⇒ likely drift).
is intentional you *promote* CI's artifact into the baselines. The environment
isn't pinned — if the host's Mesa/fonts drift, the gate lights up in Discord and
you just re-promote (broad + low-intensity change ⇒ likely drift).
**Baselines live in R2, not git.** The PNGs sit in a private Cloudflare R2
bucket (`pcbjam-ci-screenshots`), content-addressed as `sha256/<hex>.png` and
immutable; the committed `screenshot-manifest.json` pins each `<engine>/<name>`
to a hash, so every git commit resolves its exact baselines. The local
`baseline-screenshots/` tree is a gitignored cache — `npm run screenshots:fetch`
materializes it; `promote` uploads new hashes and rewrites the manifest, and the
manifest diff is the only thing you commit.
**Credentials** (S3 API, bucket-scoped, region `auto`):
```
CI_SCREENSHOTS_S3_ENDPOINT # https://<account-id>.r2.cloudflarestorage.com
CI_SCREENSHOTS_S3_BUCKET # optional, default pcbjam-ci-screenshots
CI_SCREENSHOTS_S3_ACCESS_KEY_ID # read-only pair in CI; read-write pair for promote
CI_SCREENSHOTS_S3_SECRET_ACCESS_KEY
```
CI holds the read-only pair as repo secrets; devs get the read-write pair from
the team vault (ask) and put it in `tests/.env` (gitignored, auto-loaded by
r2-store.ts; shell env vars take precedence):
```
CI_SCREENSHOTS_S3_ENDPOINT=https://<account-id>.r2.cloudflarestorage.com
CI_SCREENSHOTS_S3_ACCESS_KEY_ID=<rw-access-key-id>
CI_SCREENSHOTS_S3_SECRET_ACCESS_KEY=<rw-secret>
```
Without credentials, fetch and the compare gate skip with a warning (secretless
CI callers stay green); promote refuses to run.
**Everything is per-engine.** Specs write `test-results/<engine>/<name>.png`
(engine derived from the running browser via `stableShot`/`shotPath`); baselines
@ -22,15 +48,18 @@ The same spec on chromium + firefox is two independent gated screenshots.
- `promote.ts` — churn-free updater: pull a CI run's shots (`gh run download`) or `--from DIR`; overwrite a baseline only when pixels differ beyond the floor (verbatim bytes, no re-encode) → no git churn.
- `perf-report.ts` — renders the track-only runtime-perf table (loadMs/openMs/FPS) with Δ vs the previous main run (fetched via `gh`).
- `post-discord.ts` — the always-on CI-on-main report: SHA + e2e status + perf table, then screenshot triptychs (batched, size-capped, flood-collapsed).
- `changelog.ts` — Discord trigger B: git-history diff of committed baseline PNGs (no build/GPU).
- `changelog.ts` — Discord trigger B: git-history diff of the manifest between two revs, PNG bytes fetched from R2 (no build/GPU).
- `noise.ts` — calibration: diff two identical-input renders → per-engine noise floor.
- `gen-manifest.ts` — regenerate `screenshot-manifest.json` (canonical {name, engine} list) from the committed per-engine baseline tree; `--check` (gating in CI) fails if it drifts.
- `gen-manifest.ts` — regenerate `screenshot-manifest.json` ({name, engine, sha256, bytes, width, height}) from the local baseline cache; `--check` (gating in CI, credential-free) validates the schema and fails if baseline PNGs are ever re-committed to git.
- `r2-store.ts` — minimal aws4fetch S3 client for the CAS bucket (get/put/exists by hash, downloads integrity-checked).
- `r2-sync.ts` — cache sync: `--pull` materializes `baseline-screenshots/` from the manifest (idempotent, deletes unlisted files), `--push` seeds/uploads, `--verify` HEADs every hash.
- `spec-map.ts` — best-effort screenshot-name → spec-file attribution for captions (scans `stableShot`/`shotPath` literals).
## npm scripts (run from `tests/`)
```
npm run screenshots:fetch # materialize the baseline cache from R2 (run before check; needs read creds)
npm run screenshots:check # gate: baselines vs test-results → report.json (exit 0; add --fail-on-change to gate)
npm run screenshots:promote -- --run <ci-run-id> # churn-free re-baseline from a CI run (or --from DIR)
npm run screenshots:promote -- --run <ci-run-id> # churn-free re-baseline from a CI run (or --from DIR; needs RW creds)
npm run screenshots:report -- --e2e pass # post the CI report to Discord (main+push only; needs DISCORD_WEBHOOK_URL)
npm run screenshots:changelog # post the baseline changelog (main+push only)
npm run screenshots:noise -- run1/ run2/ # calibrate floors

115
tests/tools/screenshots/changelog.ts
View file

@ -1,33 +1,40 @@
/**
* Baseline changelog (Discord trigger B) — no build, no GPU.
*
* On push to main, diff the committed baseline PNGs between two commits and post
* a "Baseline changelog" to Discord: a triptych (old | new+boxes | heatmap) for
* each CHANGED file, the image for each ADDED, and a titled line for each REMOVED.
* "added / removed" only mean anything as a git-history diff, which is why this is
* separate from the re-render drift gate.
* On push to main, diff the committed screenshot MANIFEST between two commits
* and post a "Baseline changelog" to Discord: a triptych (old | new+boxes |
* heatmap) for each CHANGED baseline, the image for each ADDED, and a titled
* line for each REMOVED. "added / removed" only mean anything as a git-history
* diff, which is why this is separate from the re-render drift gate.
*
* The PNGs themselves are not in git — each manifest entry pins a sha256 and
* the bytes are fetched from the R2 CAS bucket (objects are immutable, so the
* base rev's hashes are always still resolvable). Needs the read credential
* pair; a base rev whose manifest predates the R2 migration (v1) is skipped so
* the migration commit itself doesn't post hundreds of bogus entries.
*
* CLI (from tests/):
* tsx tools/screenshots/changelog.ts [--base REV] [--head REV] [--dry-run] [--force]
*/
import * as fs from 'fs';
import * as os from 'os';
import * as path from 'path';
import { execFileSync } from 'child_process';
import { PNG } from 'pngjs';
import { BASELINE_ROOT, DIFF_OUT_DIR, floorFor, labelText, splitKey, LABEL } from './config';
import { DIFF_OUT_DIR, MANIFEST_PATH, MANIFEST_VERSION, floorFor, labelText, splitKey, LABEL, type Manifest } from './config';
import { comparePair, type Report } from './compare';
import { loadPng, savePng, withBottomLabel } from './image-ops';
import { savePng, withBottomLabel } from './image-ops';
import { buildSpecResolver } from './spec-map';
import { buildAttachments, paginate, postMessage } from './post-discord';
import { missingEnv, storeFromEnv, type R2Store } from './r2-store';
function git(root: string, args: string[]): string {
return execFileSync('git', ['-C', root, ...args], { encoding: 'utf8' }).trim();
}
function gitBlob(root: string, rev: string, repoPath: string): Buffer | null {
/** The screenshot manifest as committed at `rev`, or null if absent/unparsable. */
function manifestAt(repoRoot: string, rev: string, repoPath: string): Manifest | null {
try {
return execFileSync('git', ['-C', root, 'show', `${rev}:${repoPath}`], { maxBuffer: 1 << 28 });
return JSON.parse(execFileSync('git', ['-C', repoRoot, 'show', `${rev}:${repoPath}`], { encoding: 'utf8' })) as Manifest;
} catch {
return null;
}
@ -60,17 +67,41 @@ async function main(): Promise<void> {
}
}
// Repo-relative baseline root prefix (e.g. tests/baseline-screenshots).
const prefix = path.relative(repoRoot, path.join(cwd, BASELINE_ROOT));
const diff = git(repoRoot, ['diff', '--name-status', base, head, '--', prefix]);
if (!diff) {
// Repo-relative manifest path (e.g. tests/screenshot-manifest.json).
const manifestPath = path.relative(repoRoot, path.join(cwd, MANIFEST_PATH)).split(path.sep).join('/');
const baseManifest = manifestAt(repoRoot, base, manifestPath);
const headManifest = manifestAt(repoRoot, head, manifestPath);
if (headManifest?.version !== MANIFEST_VERSION) {
console.log('[changelog] head manifest is not v2 — nothing to diff');
return;
}
if (baseManifest?.version !== MANIFEST_VERSION) {
console.log('[changelog] base manifest predates the R2 migration — skipping (migration commit)');
return;
}
// Engine-qualified key (`<engine>/<name>`) → sha256, per rev.
const hashesOf = (m: Manifest): Map<string, string> =>
new Map(m.screenshots.map((e) => [`${e.engine}/${e.name}`, e.sha256]));
const baseHashes = hashesOf(baseManifest);
const headHashes = hashesOf(headManifest);
const added = [...headHashes.keys()].filter((k) => !baseHashes.has(k));
const removed = [...baseHashes.keys()].filter((k) => !headHashes.has(k));
const changed = [...headHashes.keys()].filter((k) => baseHashes.has(k) && baseHashes.get(k) !== headHashes.get(k));
if (!added.length && !removed.length && !changed.length) {
console.log('[changelog] no baseline changes between', base.slice(0, 7), 'and', head);
return;
}
const store = storeFromEnv();
if (!store) {
console.error(`[changelog] R2 credentials required to fetch baseline bytes: set ${missingEnv().join(', ')}`);
process.exitCode = 1;
return;
}
const outDir = path.join(cwd, DIFF_OUT_DIR);
fs.mkdirSync(outDir, { recursive: true });
const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'changelog-'));
const report: Report = {
generatedFor: process.env.GITHUB_SHA || head,
changed: [],
@ -81,53 +112,25 @@ async function main(): Promise<void> {
};
const { specFor } = buildSpecResolver(cwd);
// Load a git blob (a committed PNG at `rev`) as a decoded PNG, or null if absent.
const blobPng = (rev: string, p: string): PNG | null => {
const b = gitBlob(repoRoot, rev, p);
if (!b) return null;
const f = path.join(tmp, `${rev.slice(0, 7)}-${path.basename(p)}`);
fs.writeFileSync(f, b);
return loadPng(f);
};
// Engine-qualified key (`<engine>/<name>.png`) from a repo path under the baseline root.
const keyOf = (repoPath: string): string => path.relative(prefix, repoPath).split(path.sep).join('/');
const hashPng = async (s: R2Store, hash: string): Promise<PNG> => PNG.sync.read(await s.get(hash));
// Save a single captioned image (added/removed) and record it in the report.
const saveSingle = (img: PNG, key: string, status: 'added' | 'removed'): void => {
const suffix = status === 'added' ? 'added' : 'removed';
const rel = path.join(DIFF_OUT_DIR, `${key.replace('/', '_')}.${suffix}.png`);
const rel = path.join(DIFF_OUT_DIR, `${key.replace('/', '_')}.${status}.png`);
savePng(path.join(cwd, rel), withBottomLabel(img, labelText(status, key, specFor(splitKey(key).name)), LABEL.colors[status]));
report[status].push({ name: key, image: rel });
};
for (const line of diff.split('\n')) {
const parts = line.split('\t');
const status = parts[0][0]; // M/A/D/R
const repoPath = parts[status === 'R' ? 2 : 1];
const oldPath = status === 'R' ? parts[1] : repoPath;
const key = keyOf(repoPath);
if (status === 'A' || status === 'R') {
const img = blobPng(head, repoPath);
if (img) saveSingle(img, key, 'added');
if (status === 'R') {
const oldImg = blobPng(base, oldPath);
if (oldImg) saveSingle(oldImg, keyOf(oldPath), 'removed');
}
} else if (status === 'D') {
const img = blobPng(base, repoPath);
if (img) saveSingle(img, key, 'removed');
} else {
// Modified: captioned triptych old vs new.
const oldImg = blobPng(base, repoPath);
const newImg = blobPng(head, repoPath);
if (!oldImg || !newImg) continue;
const { result, heatmap, triptych } = comparePair(oldImg, newImg, key, floorFor(key));
const triptychRel = path.join(DIFF_OUT_DIR, `${key.replace('/', '_')}.triptych.png`);
const heatmapRel = path.join(DIFF_OUT_DIR, `${key.replace('/', '_')}.heatmap.png`);
savePng(path.join(cwd, triptychRel), withBottomLabel(triptych, labelText('changed', key, specFor(splitKey(key).name)), LABEL.colors.changed));
savePng(path.join(cwd, heatmapRel), heatmap);
report.changed.push({ ...result, triptych: triptychRel, heatmap: heatmapRel });
}
for (const key of added) saveSingle(await hashPng(store, headHashes.get(key)!), key, 'added');
for (const key of removed) saveSingle(await hashPng(store, baseHashes.get(key)!), key, 'removed');
for (const key of changed) {
const oldImg = await hashPng(store, baseHashes.get(key)!);
const newImg = await hashPng(store, headHashes.get(key)!);
const { result, heatmap, triptych } = comparePair(oldImg, newImg, key, floorFor(key));
const triptychRel = path.join(DIFF_OUT_DIR, `${key.replace('/', '_')}.triptych.png`);
const heatmapRel = path.join(DIFF_OUT_DIR, `${key.replace('/', '_')}.heatmap.png`);
savePng(path.join(cwd, triptychRel), withBottomLabel(triptych, labelText('changed', key, specFor(splitKey(key).name)), LABEL.colors.changed));
savePng(path.join(cwd, heatmapRel), heatmap);
report.changed.push({ ...result, triptych: triptychRel, heatmap: heatmapRel });
}
report.changed.sort((a, b) => b.changedRatio - a.changedRatio);

10
tests/tools/screenshots/compare.ts
View file

@ -228,6 +228,16 @@ function main(): void {
return;
}
const root = process.cwd();
// Baselines are a local cache fetched from R2 (`npm run screenshots:fetch`).
// When the manifest expects screenshots but the cache is empty — a secretless
// CI caller where the fetch step skipped, or a dev who hasn't fetched — skip
// the gate instead of misreporting every baseline as removed. No report.json
// is written; post-discord tolerates its absence.
const expected = loadManifest(root)?.screenshots.length ?? 0;
if (expected > 0 && listEngineKeys(path.join(root, BASELINE_ROOT)).length === 0) {
console.log('[compare] baseline cache is empty — run `npm run screenshots:fetch` (needs R2 credentials); skipping');
return;
}
const report = classify(root, (args.sha as string) || process.env.GITHUB_SHA || null);
fs.writeFileSync(path.join(root, DIFF_OUT_DIR, 'report.json'), JSON.stringify(report, null, 2));
const { changed, added, removed, unchangedCount, driftLikely } = report;

53
tests/tools/screenshots/config.ts
View file

@ -12,15 +12,42 @@ import * as nodeFs from 'fs';
import * as nodePath from 'path';
/**
* Committed baseline root. One PER-ENGINE subdirectory per browser family:
* `baseline-screenshots/{chromium,firefox}/<name>.png`. Renders mirror the
* layout (`test-results/<engine>/<name>.png`, written by stableShot/shotPath),
* so each engine is gated against its OWN baselines — the old flat namespace
* let two engines running the same spec overwrite each other's PNG, and the
* surviving file was whichever parallel worker wrote last.
* Baseline root — a GITIGNORED local cache, one PER-ENGINE subdirectory per
* browser family: `baseline-screenshots/{chromium,firefox}/<name>.png`.
* Renders mirror the layout (`test-results/<engine>/<name>.png`, written by
* stableShot/shotPath), so each engine is gated against its OWN baselines —
* the old flat namespace let two engines running the same spec overwrite each
* other's PNG, and the surviving file was whichever parallel worker wrote last.
*
* The baseline PNGs are NOT committed: they live in a private R2 bucket,
* content-addressed by sha256, and the committed manifest
* (screenshot-manifest.json) is what pins each `<engine>/<name>` to a hash.
* `npm run screenshots:fetch` (tools/screenshots/r2-sync.ts) materializes this
* tree from the manifest; promote.ts uploads new hashes and rewrites the
* manifest — the manifest diff is the only thing that lands in git.
*/
export const BASELINE_ROOT = 'baseline-screenshots';
/** Manifest format version — bumped when baselines moved from git to R2. */
export const MANIFEST_VERSION = 2;
/** Default private R2 bucket holding the content-addressed baselines. */
export const R2_DEFAULT_BUCKET = 'pcbjam-ci-screenshots';
/** Key prefix for content-addressed objects: `sha256/<64-hex>.png`. */
export const R2_KEY_PREFIX = 'sha256/';
/**
* Env vars for the bucket's S3 API (read-only keypair in CI, read-write for
* devs running promote). Endpoint form: https://<account-id>.r2.cloudflarestorage.com
*/
export const R2_ENV = {
endpoint: 'CI_SCREENSHOTS_S3_ENDPOINT',
bucket: 'CI_SCREENSHOTS_S3_BUCKET',
accessKeyId: 'CI_SCREENSHOTS_S3_ACCESS_KEY_ID',
secretAccessKey: 'CI_SCREENSHOTS_S3_SECRET_ACCESS_KEY',
} as const;
/** Engine subdirectories the tooling recognizes (a browser family per dir). */
export const ENGINES = ['chromium', 'firefox', 'webkit'] as const;
@ -152,8 +179,18 @@ export function labelText(status: LabelStatus, key: string, spec: string | null)
return spec ? `${head} · ${spec}` : head;
}
export type ManifestEntry = { name: string; engine: string };
export type Manifest = { screenshots: ManifestEntry[] };
/**
* One expected screenshot. `sha256` is the load-bearing field — it resolves the
* R2 object (`sha256/<hex>.png`) holding the baseline bytes; `bytes`/`width`/
* `height` are sanity metadata (cheap pre-hash check on fetch, dimension info
* in reviews).
*/
export type ManifestEntry = { name: string; engine: string; sha256: string; bytes: number; width: number; height: number };
export type Manifest = {
version: number;
storage: { bucket: string; keyPrefix: string };
screenshots: ManifestEntry[];
};
/** Verdict floor for an engine-qualified key — the engine IS the key prefix now,
* no manifest lookup needed. */

179
tests/tools/screenshots/gen-manifest.ts
View file

@ -1,30 +1,73 @@
/**
* Generate tests/screenshot-manifest.json — the canonical list of expected
* screenshots. Each entry is {name, engine}, read STRAIGHT from the per-engine
* baseline tree (baseline-screenshots/<engine>/<name>.png) — the engine is the
* directory, no spec-scanning guesswork.
* screenshots AND the pointer to their bytes. Each entry is
* {name, engine, sha256, bytes, width, height}: the engine comes from the
* baseline cache tree (baseline-screenshots/<engine>/<name>.png), the sha256
* resolves the PNG in the private R2 bucket (`sha256/<hex>.png`). The manifest
* is the SOURCE OF TRUTH — the local tree is a cache materialized from it by
* r2-sync (`npm run screenshots:fetch`).
*
* The list is what lets compare/promote tell an intentional REMOVAL from a
* flaky/absent render. promote.ts regenerates it automatically after applying;
* --check keeps CI honest if baselines are ever edited by hand.
* promote.ts is the single writer: it regenerates this file after applying and
* uploading a CI run's render. Regeneration therefore needs the local tree
* present; --check deliberately does NOT (it must stay credential- and
* network-free so it can gate every CI caller, secretless ones included).
*
* CLI (from tests/): tsx tools/screenshots/gen-manifest.ts [--check]
* --check exits 1 if the committed manifest is stale (for CI hygiene).
* CLI (from tests/): tsx tools/screenshots/gen-manifest.ts [--check [--remote]]
* --check exit 1 unless the committed manifest is well-formed (v2 schema,
* sorted+unique entries, valid hashes) and no baseline PNG has been
* re-committed to git (resurrection guard).
* --remote with --check: also HEAD every hash in R2 (needs credentials;
* manual use only, not wired into CI).
*/
import * as fs from 'fs';
import * as path from 'path';
import { BASELINE_ROOT, MANIFEST_PATH, isIgnored, listEngineKeys, splitKey, type Manifest } from './config';
import { execFileSync } from 'child_process';
import { PNG } from 'pngjs';
import {
BASELINE_ROOT,
ENGINES,
IGNORE_SCREENSHOTS,
MANIFEST_PATH,
MANIFEST_VERSION,
R2_DEFAULT_BUCKET,
R2_KEY_PREFIX,
isIgnored,
listEngineKeys,
splitKey,
type Manifest,
type ManifestEntry,
} from './config';
import { hashFile, missingEnv, storeFromEnv } from './r2-store';
/** Render the manifest JSON for the current baseline tree. */
const NOTE =
'Baselines live in R2, content-addressed by sha256 — the local baseline-screenshots/ tree is a cache ' +
'(`npm run screenshots:fetch`). Regenerated by `npm run screenshots:promote`; never edit hashes by hand. ' +
'The name list is authoritative for removed-screenshot detection.';
/** Render the manifest JSON for the current baseline cache tree (tree must be present). */
export function manifestJson(root: string): string {
const screenshots = listEngineKeys(path.join(root, BASELINE_ROOT))
const base = path.join(root, BASELINE_ROOT);
const screenshots: ManifestEntry[] = listEngineKeys(base)
.filter((key) => !isIgnored(key))
.map((key) => splitKey(key))
.map(({ engine, name }) => ({ name, engine }))
.map((key) => {
const { engine, name } = splitKey(key);
const file = path.join(base, key);
const png = PNG.sync.read(fs.readFileSync(file));
return {
name,
engine,
sha256: hashFile(file),
bytes: fs.statSync(file).size,
width: png.width,
height: png.height,
};
})
.sort((a, b) => a.engine.localeCompare(b.engine) || a.name.localeCompare(b.name));
const manifest: Manifest & { _note: string } = {
_note: 'Generated from baseline-screenshots/<engine>/ — run `npm run screenshots:manifest`. The name list is authoritative for removed-screenshot detection.',
_note: NOTE,
version: MANIFEST_VERSION,
storage: { bucket: R2_DEFAULT_BUCKET, keyPrefix: R2_KEY_PREFIX },
screenshots,
};
return JSON.stringify(manifest, null, 2) + '\n';
@ -35,27 +78,107 @@ export function writeManifest(root: string): void {
fs.writeFileSync(path.join(root, MANIFEST_PATH), manifestJson(root));
}
function main(): void {
const check = process.argv.includes('--check');
const root = process.cwd();
const json = manifestJson(root);
const outPath = path.join(root, MANIFEST_PATH);
const count = (JSON.parse(json) as Manifest).screenshots.length;
/** Schema + hygiene problems with the committed manifest (empty = healthy). Credential-free. */
export function checkManifest(root: string): string[] {
const problems: string[] = [];
const p = path.join(root, MANIFEST_PATH);
if (!fs.existsSync(p)) return [`${MANIFEST_PATH} is missing`];
let manifest: Manifest;
try {
manifest = JSON.parse(fs.readFileSync(p, 'utf8')) as Manifest;
} catch (e) {
return [`${MANIFEST_PATH} is not valid JSON: ${(e as Error).message}`];
}
if (manifest.version !== MANIFEST_VERSION) problems.push(`version is ${manifest.version}, expected ${MANIFEST_VERSION}`);
if (!manifest.storage?.bucket || !manifest.storage?.keyPrefix) problems.push('storage.bucket/keyPrefix missing');
if (!Array.isArray(manifest.screenshots) || manifest.screenshots.length === 0) {
problems.push('screenshots list is missing or empty');
return problems;
}
console.log(`[manifest] ${count} screenshots across engines`);
const engines = new Set<string>(ENGINES);
let prev: ManifestEntry | null = null;
const seen = new Set<string>();
for (const e of manifest.screenshots) {
const id = `${e.engine}/${e.name}`;
if (!e.name?.toLowerCase().endsWith('.png')) problems.push(`${id}: name is not a .png`);
if (!engines.has(e.engine)) problems.push(`${id}: unknown engine`);
if (!/^[0-9a-f]{64}$/.test(e.sha256 ?? '')) problems.push(`${id}: sha256 is not 64 lowercase hex chars`);
if (!(e.bytes > 0) || !(e.width > 0) || !(e.height > 0)) problems.push(`${id}: bytes/width/height must be positive`);
if (IGNORE_SCREENSHOTS.has(e.name)) problems.push(`${id}: is in IGNORE_SCREENSHOTS and must not be listed`);
if (seen.has(id)) problems.push(`${id}: duplicate entry`);
seen.add(id);
if (prev && (prev.engine.localeCompare(e.engine) || prev.name.localeCompare(e.name)) > 0) {
problems.push(`${id}: not sorted (engine, then name)`);
}
prev = e;
}
// Resurrection guard: baselines must never come back as committed files.
try {
const tracked = execFileSync('git', ['ls-files', '--', BASELINE_ROOT], { cwd: root, encoding: 'utf8' }).trim();
if (tracked) {
problems.push(
`git-tracked PNGs under ${BASELINE_ROOT}/ — baselines live in R2, do not commit them:\n ` +
tracked.split('\n').slice(0, 5).join('\n ')
);
}
} catch {
console.warn('[manifest] git unavailable — skipping the resurrection guard');
}
return problems;
}
async function main(): Promise<void> {
const check = process.argv.includes('--check');
const remote = process.argv.includes('--remote');
const root = process.cwd();
const outPath = path.join(root, MANIFEST_PATH);
if (check) {
const current = fs.existsSync(outPath) ? fs.readFileSync(outPath, 'utf8') : '';
if (current !== json) {
console.error('[manifest] STALE — run `npm run screenshots:manifest` and commit');
const problems = checkManifest(root);
if (problems.length) {
console.error(`[manifest] INVALID:\n - ${problems.join('\n - ')}`);
process.exitCode = 1;
} else {
console.log('[manifest] up to date');
return;
}
const manifest = JSON.parse(fs.readFileSync(outPath, 'utf8')) as Manifest;
console.log(`[manifest] ok — ${manifest.screenshots.length} screenshots across engines`);
if (remote) {
const store = storeFromEnv();
if (!store) {
console.error(`[manifest] --remote needs credentials: set ${missingEnv().join(', ')}`);
process.exitCode = 2;
return;
}
const { verifyBaselines } = await import('./r2-sync');
const missing = await verifyBaselines(root, store);
if (missing.length) {
console.error(`[manifest] ${missing.length} hash(es) missing in R2:\n ${missing.join('\n ')}`);
process.exitCode = 1;
} else {
console.log('[manifest] every hash present in R2');
}
}
return;
}
// Regeneration path: requires the cache tree (an absent tree would silently
// produce an empty manifest and disable removed-screenshot detection).
if (listEngineKeys(path.join(root, BASELINE_ROOT)).length === 0) {
console.error('[manifest] baseline cache is empty — run `npm run screenshots:fetch` first (or promote a run)');
process.exitCode = 1;
return;
}
const json = manifestJson(root);
fs.writeFileSync(outPath, json);
console.log(`[manifest] wrote ${MANIFEST_PATH}`);
const count = (JSON.parse(json) as Manifest).screenshots.length;
console.log(`[manifest] wrote ${MANIFEST_PATH} (${count} screenshots)`);
}
if (require.main === module) main();
if (require.main === module) {
main().catch((e) => {
console.error(`[manifest] ${(e as Error).message}`);
process.exitCode = 1;
});
}

70
tests/tools/screenshots/promote.ts
View file

@ -3,10 +3,17 @@
*
* CI's x86 render is the source of truth. This pulls a CI run's screenshots
* (`gh run download`) — or a local dir via --from — and, for each one, overwrites
* the committed baseline ONLY when the decoded pixels differ beyond the per-engine
* floor. Unchanged baselines are left byte-identical (never re-encoded), so git
* sees no churn. New shots are added; baselines with no render are reported as
* removal candidates and only deleted with --prune.
* the cached baseline ONLY when the decoded pixels differ beyond the per-engine
* floor. Unchanged baselines keep their bytes (never re-encoded), so their hash —
* and therefore the manifest — sees no churn. New shots are added; baselines with
* no render are reported as removal candidates and only dropped with --prune.
*
* Baselines live in a private R2 bucket (content-addressed; see r2-store.ts).
* The flow is: sync the local cache from the committed manifest → build/apply
* the plan locally → upload the updated/added bytes to R2 → regenerate the
* manifest. Only the manifest diff is committed; R2 objects are immutable and
* never deleted (--prune removes the manifest entry, old commits still resolve).
* Needs the READ-WRITE credential pair (see tools/screenshots/README.md).
*
* CLI (from tests/):
* tsx tools/screenshots/promote.ts --run <ci-run-id> [--repo owner/repo] [--prune] [--dry-run]
@ -19,8 +26,10 @@ import { execFileSync } from 'child_process';
import { BASELINE_ROOT, MANIFEST_PATH, floorFor, isIgnored, listEngineKeys, splitKey, type Manifest } from './config';
import { writeManifest } from './gen-manifest';
import { diffImages, loadPng } from './image-ops';
import { hashBytes, missingEnv, storeFromEnv, type R2Store } from './r2-store';
import { loadManifestV2, pool, pullBaselines } from './r2-sync';
/** key (`<engine>/<name>`) → absolute committed baseline path. */
/** key (`<engine>/<name>`) → absolute cached baseline path. */
function baselineIndex(root: string): Map<string, string> {
const abs = path.join(root, BASELINE_ROOT);
const index = new Map<string, string>();
@ -84,11 +93,11 @@ function buildPlan(root: string, renderDir: string, manifest?: Manifest): { plan
plan.updated.push(key);
actions.push(() => fs.copyFileSync(src, existing)); // verbatim bytes, no re-encode → no churn
} else {
plan.unchanged.push(key); // leave the committed file untouched
plan.unchanged.push(key); // leave the cached file untouched
}
}
// Removal candidates: a committed baseline the manifest expects but this render didn't produce.
// Removal candidates: a manifest-listed baseline this render didn't produce.
for (const [key, abs] of baselines) {
if (rendered.has(key)) continue;
const { engine, name } = splitKey(key);
@ -101,6 +110,16 @@ function buildPlan(root: string, renderDir: string, manifest?: Manifest): { plan
return { plan, apply: () => actions.forEach((a) => a()) };
}
/** Upload the applied updated/added baselines to R2 (skip-if-exists per hash). */
async function uploadApplied(root: string, store: R2Store, keys: string[]): Promise<number> {
let uploaded = 0;
await pool(keys, 8, async (key) => {
const bytes = fs.readFileSync(path.join(root, BASELINE_ROOT, key));
if ((await store.put(hashBytes(bytes), bytes)) === 'uploaded') uploaded++;
});
return uploaded;
}
function parseArgs(argv: string[]): Record<string, string | boolean> {
const out: Record<string, string | boolean> = {};
for (let i = 0; i < argv.length; i++) {
@ -114,7 +133,7 @@ function parseArgs(argv: string[]): Record<string, string | boolean> {
return out;
}
function main(): void {
async function main(): Promise<void> {
const args = parseArgs(process.argv.slice(2));
const root = process.cwd();
if (!args.run && !args.from) {
@ -122,6 +141,24 @@ function main(): void {
process.exitCode = 2;
return;
}
// Fail fast BEFORE downloading anything: a promote that can't upload would
// otherwise leave the manifest referencing hashes that don't exist in R2.
const store = storeFromEnv();
if (!store) {
console.error(`[promote] R2 read-write credentials required: set ${missingEnv().join(', ')}`);
process.exitCode = 2;
return;
}
// The local tree is a cache — sync it to the committed manifest so the plan
// diffs against exactly what the manifest pins (a stale/absent cache would
// otherwise misreport adds/updates).
if (loadManifestV2(root)) {
const { downloaded, cached, deleted } = await pullBaselines(root, store);
console.log(`[promote] cache synced: downloaded=${downloaded} cached=${cached} deleted=${deleted}`);
} else {
console.warn(`[promote] ${MANIFEST_PATH} is not v2 — skipping cache sync (pre-migration tree?)`);
}
const renderDir = args.from ? (args.from as string) : downloadRun(args.run as string, args.repo as string);
const manifest = loadManifest(root);
const { plan, apply } = buildPlan(root, renderDir, manifest);
@ -135,18 +172,29 @@ function main(): void {
for (const n of plan.removedCandidates) console.log(` REMOVE? ${n}${args.prune ? ' (pruning)' : ' (use --prune to delete)'}`);
if (args.dryRun) {
console.log('[promote] dry-run — no files written');
console.log('[promote] dry-run — no files written, nothing uploaded');
return;
}
apply();
if (args.prune) {
const baselines = baselineIndex(root);
// Prune drops the local file (and, below, the manifest entry). The R2
// object is deliberately kept — old commits must still resolve it.
for (const n of plan.removedCandidates) fs.rmSync(baselines.get(n)!, { force: true });
}
// Upload BEFORE regenerating the manifest: a failure here leaves at worst an
// orphaned CAS object, never a committed manifest pointing at a missing hash.
const uploaded = await uploadApplied(root, store, [...plan.updated, ...plan.added]);
console.log(`[promote] uploaded ${uploaded} object(s) to R2`);
// Keep the manifest in lockstep with the baseline tree — a stale manifest silently
// disables removed-screenshot detection for anything added after the last regen.
writeManifest(root);
console.log('[promote] done (manifest regenerated) — review `git status` and commit');
console.log(`[promote] done — commit the ${MANIFEST_PATH} diff (the only git-visible output)`);
}
if (require.main === module) main();
if (require.main === module) {
main().catch((e) => {
console.error(`[promote] ${(e as Error).message}`);
process.exitCode = 1;
});
}

131
tests/tools/screenshots/r2-store.ts Normal file
View file

@ -0,0 +1,131 @@
/**
* Minimal S3 client for the content-addressed screenshot-baseline bucket.
*
* Baselines live in a PRIVATE R2 bucket as immutable objects keyed by content
* hash (`sha256/<hex>.png`); the committed manifest (screenshot-manifest.json)
* maps `<engine>/<name>` to a hash, so any git commit resolves its exact
* baselines. Auth is a bucket-scoped S3 keypair — read-only in CI, read-write
* for devs (promote) — via the env vars in config.ts R2_ENV; see this
* directory's README for where to get credentials.
*
* aws4fetch does the SigV4 signing (dependency-free; region is always "auto"
* on R2). Objects are never deleted here — a baseline prune only edits the
* manifest, old hashes stay resolvable for old commits.
*/
import * as crypto from 'crypto';
import * as fs from 'fs';
import { AwsClient } from 'aws4fetch';
import { R2_DEFAULT_BUCKET, R2_ENV, R2_KEY_PREFIX } from './config';
export function hashBytes(bytes: Buffer): string {
return crypto.createHash('sha256').update(bytes).digest('hex');
}
export function hashFile(file: string): string {
return hashBytes(fs.readFileSync(file));
}
const RETRIES = 3;
const BACKOFF_MS = 2000;
export class R2Store {
private client: AwsClient;
private base: string;
constructor(opts: { endpoint: string; bucket: string; accessKeyId: string; secretAccessKey: string }) {
// aws4fetch has its own 5xx retry loop; disable it (retries: 0) so the
// backoff policy lives in exactly one place (fetchWithRetry below).
this.client = new AwsClient({
accessKeyId: opts.accessKeyId,
secretAccessKey: opts.secretAccessKey,
region: 'auto',
service: 's3',
retries: 0,
});
this.base = `${opts.endpoint.replace(/\/+$/, '')}/${opts.bucket}`;
}
private url(hash: string): string {
return `${this.base}/${R2_KEY_PREFIX}${hash}.png`;
}
/** Signed fetch, retrying network errors and 5xx — an R2 blip shouldn't fail a whole sync. */
private async fetchWithRetry(url: string, init: RequestInit): Promise<Response> {
let lastErr: unknown;
for (let attempt = 1; attempt <= RETRIES; attempt++) {
try {
const res = await this.client.fetch(url, init);
if (res.status < 500 && res.status !== 429) return res;
lastErr = new Error(`HTTP ${res.status}`);
await res.arrayBuffer().catch(() => undefined); // drain before retrying
} catch (e) {
lastErr = e;
}
if (attempt < RETRIES) await new Promise((r) => setTimeout(r, BACKOFF_MS * attempt));
}
throw new Error(`request failed after ${RETRIES} attempts: ${(lastErr as Error).message}`);
}
async exists(hash: string): Promise<boolean> {
const res = await this.fetchWithRetry(this.url(hash), { method: 'HEAD' });
if (res.status === 200) return true;
if (res.status === 404) return false;
throw new Error(`HEAD ${R2_KEY_PREFIX}${hash}.png → HTTP ${res.status}`);
}
/** Download one object; the key IS the sha256, so every download is integrity-checked. */
async get(hash: string): Promise<Buffer> {
const res = await this.fetchWithRetry(this.url(hash), { method: 'GET' });
if (res.status === 404) throw new Error(`missing object ${R2_KEY_PREFIX}${hash}.png`);
if (res.status !== 200) throw new Error(`GET ${R2_KEY_PREFIX}${hash}.png → HTTP ${res.status}`);
const bytes = Buffer.from(await res.arrayBuffer());
const actual = hashBytes(bytes);
if (actual !== hash) throw new Error(`corrupt download for ${hash} (received sha256 ${actual})`);
return bytes;
}
/** Upload one object unless already present (same-hash PUTs are idempotent regardless). */
async put(hash: string, bytes: Buffer): Promise<'uploaded' | 'exists'> {
if (await this.exists(hash)) return 'exists';
const res = await this.fetchWithRetry(this.url(hash), {
method: 'PUT',
// Buffer is a Uint8Array and a valid fetch body; the cast bridges
// @types/node's Buffer<ArrayBufferLike> vs DOM BufferSource generics.
body: bytes as unknown as BodyInit,
headers: { 'content-type': 'image/png' },
});
if (res.status !== 200) throw new Error(`PUT ${R2_KEY_PREFIX}${hash}.png → HTTP ${res.status}`);
await res.arrayBuffer().catch(() => undefined);
return 'uploaded';
}
}
let envFileLoaded = false;
/** Best-effort load of ./.env (gitignored) so a dev's keypair can live in a file
* instead of the shell. Node ≥20.12 built-in; CI is unaffected (no .env there). */
function ensureEnvFileLoaded(): void {
if (envFileLoaded) return;
envFileLoaded = true;
try {
if (fs.existsSync('.env')) process.loadEnvFile('.env');
} catch {
/* unreadable/malformed .env → shell env only */
}
}
/** The required R2 env vars that are currently unset (bucket has a default, so it's not required). */
export function missingEnv(): string[] {
ensureEnvFileLoaded();
return [R2_ENV.endpoint, R2_ENV.accessKeyId, R2_ENV.secretAccessKey].filter((v) => !process.env[v]);
}
/** Build a store from the environment (shell vars, falling back to ./.env), or null when credentials are absent. */
export function storeFromEnv(): R2Store | null {
if (missingEnv().length) return null;
return new R2Store({
endpoint: process.env[R2_ENV.endpoint]!,
bucket: process.env[R2_ENV.bucket] || R2_DEFAULT_BUCKET,
accessKeyId: process.env[R2_ENV.accessKeyId]!,
secretAccessKey: process.env[R2_ENV.secretAccessKey]!,
});
}

177
tests/tools/screenshots/r2-sync.ts Normal file
View file

@ -0,0 +1,177 @@
/**
* Sync the local baseline-screenshots/ cache with the R2 CAS bucket.
*
* The committed manifest (screenshot-manifest.json) is the source of truth;
* baseline-screenshots/ is a gitignored local cache materialized from it.
*
* CLI (from tests/):
* tsx tools/screenshots/r2-sync.ts --pull # manifest → local tree (CI fetch step + local gate)
* tsx tools/screenshots/r2-sync.ts --push # upload manifest entries missing in R2 (seeding)
* tsx tools/screenshots/r2-sync.ts --verify # HEAD every manifest hash, exit 1 on any miss
*
* --pull without credentials (or with a pre-migration manifest) warns and
* exits 0, so secretless callers of the reusable CI workflow (release.yml,
* pcbjam deploy-staging.yml, fork PRs) stay green — compare.ts then skips its
* gate for the same reason. With credentials, any 404/corrupt object is
* collected and the run exits 1.
*/
import * as fs from 'fs';
import * as path from 'path';
import { BASELINE_ROOT, MANIFEST_PATH, MANIFEST_VERSION, listEngineKeys, type Manifest, type ManifestEntry } from './config';
import { R2Store, hashFile, missingEnv, storeFromEnv } from './r2-store';
const CONCURRENCY = 16;
/** Parse the committed manifest if it is the R2-backed v2 format, else null. */
export function loadManifestV2(root: string): Manifest | null {
const p = path.join(root, MANIFEST_PATH);
if (!fs.existsSync(p)) return null;
try {
const m = JSON.parse(fs.readFileSync(p, 'utf8')) as Manifest;
return m.version === MANIFEST_VERSION ? m : null;
} catch {
return null;
}
}
/** Run `fn` over `items` with at most `limit` in flight. */
export async function pool<T>(items: T[], limit: number, fn: (item: T) => Promise<void>): Promise<void> {
let i = 0;
const workers = Array.from({ length: Math.min(limit, items.length) }, async () => {
while (i < items.length) await fn(items[i++]);
});
await Promise.all(workers);
}
/**
* Materialize the manifest's baselines into the local cache tree. Files already
* hashing correctly are skipped (idempotent — a warm cache or restored
* actions/cache costs nothing), writes are atomic (tmp + rename), and local
* PNGs the manifest no longer lists are deleted so a stale branch switch can't
* leave ghost baselines for compare.ts to gate against.
*/
export async function pullBaselines(
root: string,
store: R2Store
): Promise<{ downloaded: number; cached: number; deleted: number }> {
const manifest = loadManifestV2(root);
if (!manifest) throw new Error(`no v2 ${MANIFEST_PATH} — nothing to pull`);
const base = path.join(root, BASELINE_ROOT);
const wanted = new Map<string, ManifestEntry>();
for (const e of manifest.screenshots) wanted.set(`${e.engine}/${e.name}`, e);
let downloaded = 0;
let cached = 0;
const errors: string[] = [];
await pool([...wanted.values()], CONCURRENCY, async (e) => {
const dest = path.join(base, e.engine, e.name);
if (fs.existsSync(dest) && hashFile(dest) === e.sha256) {
cached++;
return;
}
try {
const bytes = await store.get(e.sha256);
fs.mkdirSync(path.dirname(dest), { recursive: true });
const tmp = `${dest}.tmp-${process.pid}`;
fs.writeFileSync(tmp, bytes);
fs.renameSync(tmp, dest);
downloaded++;
} catch (err) {
errors.push(`${e.engine}/${e.name}: ${(err as Error).message}`);
}
});
let deleted = 0;
for (const key of listEngineKeys(base)) {
if (wanted.has(key)) continue;
fs.rmSync(path.join(base, key), { force: true });
deleted++;
}
if (errors.length) throw new Error(`${errors.length} object(s) failed:\n ${errors.join('\n ')}`);
return { downloaded, cached, deleted };
}
/** Upload every manifest entry's local file to R2 (skipping hashes already present). */
export async function pushBaselines(root: string, store: R2Store): Promise<{ uploaded: number; existing: number }> {
const manifest = loadManifestV2(root);
if (!manifest) throw new Error(`no v2 ${MANIFEST_PATH} — run \`npm run screenshots:manifest\` first`);
const base = path.join(root, BASELINE_ROOT);
let uploaded = 0;
let existing = 0;
const errors: string[] = [];
await pool(manifest.screenshots, CONCURRENCY, async (e) => {
const file = path.join(base, e.engine, e.name);
try {
if (!fs.existsSync(file)) throw new Error('local file missing');
const bytes = fs.readFileSync(file);
const hash = hashFile(file);
if (hash !== e.sha256) throw new Error(`local sha256 ${hash} ≠ manifest — regenerate the manifest`);
if ((await store.put(hash, bytes)) === 'uploaded') uploaded++;
else existing++;
} catch (err) {
errors.push(`${e.engine}/${e.name}: ${(err as Error).message}`);
}
});
if (errors.length) throw new Error(`${errors.length} upload(s) failed:\n ${errors.join('\n ')}`);
return { uploaded, existing };
}
/** HEAD every manifest hash; returns the missing keys. */
export async function verifyBaselines(root: string, store: R2Store): Promise<string[]> {
const manifest = loadManifestV2(root);
if (!manifest) throw new Error(`no v2 ${MANIFEST_PATH}`);
const missing: string[] = [];
await pool(manifest.screenshots, CONCURRENCY, async (e) => {
if (!(await store.exists(e.sha256))) missing.push(`${e.engine}/${e.name} (${e.sha256})`);
});
return missing;
}
async function main(): Promise<void> {
const mode = process.argv.find((a) => a === '--pull' || a === '--push' || a === '--verify');
if (!mode) {
console.error('usage: r2-sync.ts --pull | --push | --verify');
process.exitCode = 2;
return;
}
const root = process.cwd();
if (mode === '--pull' && !loadManifestV2(root)) {
console.log(`[r2-sync] ${MANIFEST_PATH} is not the R2-backed v2 format — nothing to fetch`);
return;
}
const store = storeFromEnv();
if (!store) {
if (mode === '--pull') {
console.log(`[r2-sync] R2 credentials unset (${missingEnv().join(', ')}) — skipping baseline fetch`);
return; // exit 0: secretless CI callers stay green
}
console.error(`[r2-sync] R2 credentials required for ${mode}: set ${missingEnv().join(', ')}`);
process.exitCode = 2;
return;
}
if (mode === '--pull') {
const { downloaded, cached, deleted } = await pullBaselines(root, store);
console.log(`[r2-sync] pull done: downloaded=${downloaded} cached=${cached} deleted=${deleted}`);
} else if (mode === '--push') {
const { uploaded, existing } = await pushBaselines(root, store);
console.log(`[r2-sync] push done: uploaded=${uploaded} already-present=${existing}`);
} else {
const missing = await verifyBaselines(root, store);
if (missing.length) {
console.error(`[r2-sync] ${missing.length} manifest hash(es) MISSING in R2:\n ${missing.join('\n ')}`);
process.exitCode = 1;
} else {
console.log('[r2-sync] verify ok — every manifest hash is present in R2');
}
}
}
if (require.main === module) {
main().catch((e) => {
console.error(`[r2-sync] ${(e as Error).message}`);
process.exitCode = 1;
});
}