AI_Assistant/pcbjam
Watch
1
0
Fork
You've already forked pcbjam
 0
Code Issues Pull requests Projects Releases 4 Packages Wiki Activity Actions 2

screenshots: R2-hosted manifest becomes the baseline source of truth (morelli cutover)

Browse source 
The committed tests/screenshot-manifest.json is retired. CI now downloads
baselines/pcbjam/manifest.json (manifest v3, written only by the morelli
review app + its seed script) to the gitignored .baseline-manifest.json,
and everything downstream (pull, verify, compare) reads that copy:

- config.ts: MANIFEST_VERSION 3, MANIFEST_PATH .baseline-manifest.json,
  R2_BASELINES_MANIFEST_KEY; ManifestEntry grows opaque provenance
- r2-sync.ts: new --manifest mode (atomic fetch; no-creds skip DELETES a
  stale copy so the gate skips rather than using old baselines); --push
  gone (bytes enter the CAS only via morelli's promote)
- compare.ts: hard-skips when no manifest was fetched — a stale warm
  cache can never gate
- wasm-build.yml: fetch-manifest step before the baselines cache; cache
  key now hashes the fetched manifest; the gen-manifest --check lint gate
  goes with the committed manifest
- deleted: screenshot-manifest.json, promote.ts, changelog.ts,
  gen-manifest.ts, screenshot-changelog.yml, promote-screenshots skill
- docs (CLAUDE/README/TESTING/WHATWORKS/tools README): promote flow is
  now https://pcbjam-morelli-staging.pcbjam-staging.workers.dev

Validated locally against the real bucket: fetch-manifest (492), cold
pull 492 / warm pull cached=492, no-creds skip chain, compare gate skip.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Istvan Matejcsok 2026-08-19 11:54:20 +02:00
commit ec7a1b0787
20 changed files with 239 additions and 4793 deletions
Download patch file Download diff file Expand all files Collapse all files

72
tests/tools/screenshots/README.md
View file

@ -1,39 +1,45 @@
# Screenshot regression + Discord review tooling
One comparison engine + a churn-free updater + a Discord reporter for the e2e
screenshots. Design and rationale: `~/.claude/plans/…snowglobe.md` (or ask).
One comparison engine + a Discord reporter for the e2e screenshots; baseline
updates happen in the morelli review app
(https://pcbjam-morelli-staging.pcbjam-staging.workers.dev, repo
github.com/PCBJam/morelli).
**Source of truth = CI's Linux render.** The dev never authors baselines on the
Mac (Mac fonts/GL ≠ CI). Instead, CI renders on every push; when a render change
is intentional you *promote* CI's artifact into the baselines. The environment
isn't pinned — if the host's Mesa/fonts drift, the gate lights up in Discord and
you just re-promote (broad + low-intensity change ⇒ likely drift).
Mac (Mac fonts/GL ≠ CI). CI renders on every push and uploads each run's
screenshots to R2 (`runs/pcbjam/<run-id>/`, 30-day retention, upload-run.ts);
when a render change is intentional you *promote* that run's screenshots in
morelli. The environment isn't pinned — if the host's Mesa/fonts drift, the
gate lights up in Discord and you just re-promote (broad + low-intensity
change ⇒ likely drift).
**Baselines live in R2, not git.** The PNGs sit in a private Cloudflare R2
bucket (`pcbjam-ci-screenshots`), content-addressed as `sha256/<hex>.png` and
immutable; the committed `screenshot-manifest.json` pins each `<engine>/<name>`
to a hash, so every git commit resolves its exact baselines. The local
`baseline-screenshots/` tree is a gitignored cache — `npm run screenshots:fetch`
materializes it; `promote` uploads new hashes and rewrites the manifest, and the
manifest diff is the only thing you commit.
**Baselines live entirely in R2, not git.** The PNGs sit in a private
Cloudflare R2 bucket (`pcbjam-ci-screenshots`), content-addressed as
`sha256/<hex>.png` and immutable; the R2-HOSTED manifest
`baselines/pcbjam/manifest.json` (written only by morelli + its seed script)
pins each `<engine>/<name>` to a hash. The local `baseline-screenshots/` tree
and `.baseline-manifest.json` are gitignored caches —
`npm run screenshots:fetch-manifest && npm run screenshots:fetch` materializes
them. Nothing screenshot-related is committed.
**Credentials** (S3 API, bucket-scoped, region `auto`):
```
CI_SCREENSHOTS_S3_ENDPOINT # https://<account-id>.r2.cloudflarestorage.com
CI_SCREENSHOTS_S3_BUCKET # optional, default pcbjam-ci-screenshots
CI_SCREENSHOTS_S3_ACCESS_KEY_ID # read-only pair in CI; read-write pair for promote
CI_SCREENSHOTS_S3_ACCESS_KEY_ID # read-only pair for fetch; CI's upload step maps in a write pair
CI_SCREENSHOTS_S3_SECRET_ACCESS_KEY
```
CI holds the read-only pair as repo secrets; devs get the read-write pair from
the team vault (ask) and put it in `tests/.env` (gitignored, auto-loaded by
r2-store.ts; shell env vars take precedence):
CI holds the read-only pair as repo secrets (plus `CI_SCREENSHOTS_S3_WRITE_*`
for the run-upload step); devs get a read pair from the team vault (ask) and
put it in `tests/.env` (gitignored, auto-loaded by r2-store.ts; shell env vars
take precedence):
```
CI_SCREENSHOTS_S3_ENDPOINT=https://<account-id>.r2.cloudflarestorage.com
CI_SCREENSHOTS_S3_ACCESS_KEY_ID=<rw-access-key-id>
CI_SCREENSHOTS_S3_SECRET_ACCESS_KEY=<rw-secret>
CI_SCREENSHOTS_S3_ACCESS_KEY_ID=<access-key-id>
CI_SCREENSHOTS_S3_SECRET_ACCESS_KEY=<secret>
```
Without credentials, fetch and the compare gate skip with a warning (secretless
CI callers stay green); promote refuses to run.
Without credentials, the manifest/baseline fetch and the compare gate skip with
a warning (secretless CI callers stay green).
**Everything is per-engine.** Specs write `test-results/<engine>/<name>.png`
(engine derived from the running browser via `stableShot`/`shotPath`); baselines
@ -45,27 +51,29 @@ The same spec on chromium + firefox is two independent gated screenshots.
- `config.ts` — thresholds, baseline dirs, per-engine floors (calibrate!), clustering knobs.
- `image-ops.ts` — PNG load/save, pixelmatch diff (AA-excluded), connected-component boxes, triptych compositing, size-cap resize.
- `compare.ts` — the comparison engine: classify per-engine baselines vs `test-results/<engine>/` → `test-results/screenshot-diff/report.json` + triptych/heatmap PNGs. `--pair` diffs two files.
- `promote.ts` — churn-free updater: pull a CI run's shots (`gh run download`) or `--from DIR`; overwrite a baseline only when pixels differ beyond the floor (verbatim bytes, no re-encode) → no git churn.
- `perf-report.ts` — renders the track-only runtime-perf table (loadMs/openMs/FPS) with Δ vs the previous main run (fetched via `gh`).
- `post-discord.ts` — the always-on CI-on-main report: SHA + e2e status + perf table, then screenshot triptychs (batched, size-capped, flood-collapsed).
- `changelog.ts` — Discord trigger B: git-history diff of the manifest between two revs, PNG bytes fetched from R2 (no build/GPU).
- `noise.ts` — calibration: diff two identical-input renders → per-engine noise floor.
- `gen-manifest.ts` — regenerate `screenshot-manifest.json` ({name, engine, sha256, bytes, width, height}) from the local baseline cache; `--check` (gating in CI, credential-free) validates the schema and fails if baseline PNGs are ever re-committed to git.
- `r2-store.ts` — minimal aws4fetch S3 client for the CAS bucket (get/put/exists by hash, downloads integrity-checked).
- `r2-sync.ts` — cache sync: `--pull` materializes `baseline-screenshots/` from the manifest (idempotent, deletes unlisted files), `--push` seeds/uploads, `--verify` HEADs every hash.
- `r2-sync.ts` — cache sync: `--manifest` downloads the R2-hosted baseline manifest, `--pull` materializes `baseline-screenshots/` from it (idempotent, deletes unlisted files), `--verify` HEADs every hash.
- `upload-run.ts` — CI-only: upload the run's renders + meta.json to `runs/pcbjam/<run-id>/` for morelli (needs the write pair; no-ops without credentials).
- `spec-map.ts` — best-effort screenshot-name → spec-file attribution for captions (scans `stableShot`/`shotPath` literals).
## npm scripts (run from `tests/`)
```
npm run screenshots:fetch # materialize the baseline cache from R2 (run before check; needs read creds)
npm run screenshots:check # gate: baselines vs test-results → report.json (exit 0; add --fail-on-change to gate)
npm run screenshots:promote -- --run <ci-run-id> # churn-free re-baseline from a CI run (or --from DIR; needs RW creds)
npm run screenshots:report -- --e2e pass # post the CI report to Discord (main+push only; needs DISCORD_WEBHOOK_URL)
npm run screenshots:changelog # post the baseline changelog (main+push only)
npm run screenshots:noise -- run1/ run2/ # calibrate floors
npm run screenshots:manifest # regenerate the manifest (--check to verify it's fresh)
npm run screenshots:fetch-manifest # download the R2-hosted baseline manifest (needs read creds)
npm run screenshots:fetch # materialize the baseline cache from R2 (run before check)
npm run screenshots:check # gate: baselines vs test-results → report.json (exit 0; add --fail-on-change to gate)
npm run screenshots:report -- --e2e pass # post the CI report to Discord (main+push only; needs DISCORD_WEBHOOK_URL)
npm run screenshots:noise -- run1/ run2/ # calibrate floors
npm run screenshots:upload-run -- --e2e pass # CI-only: upload the run's renders for morelli
```
Baseline promotion (single or bulk) happens in morelli — pick the run, review
the diffs, Promote. It copies verbatim bytes into the CAS, updates the R2
manifest atomically (with provenance: which run/branch/user), and snapshots the
previous manifest for revert.
## Activation checklist
- [x] `screenshot-manifest.json` generated ({name, engine} authoritative — derived from the per-engine baseline tree).
- [x] `scale:'device'`→`'css'` normalized (no-op at CI's DSF=1).

212
tests/tools/screenshots/changelog.ts
View file

@ -1,212 +0,0 @@
/**
* Baseline changelog (Discord trigger B) — no build, no GPU.
*
* On push to main, diff the committed screenshot MANIFEST between two commits
* and post a "Baseline changelog" to Discord: a triptych (old | new+boxes |
* heatmap) for each CHANGED baseline, the image for each ADDED, and a titled
* line for each REMOVED. "added / removed" only mean anything as a git-history
* diff, which is why this is separate from the re-render drift gate.
*
* The PNGs themselves are not in git — each manifest entry pins a sha256 and
* the bytes are fetched from the R2 CAS bucket (objects are immutable, so the
* base rev's hashes are always still resolvable). Needs the read credential
* pair; a base rev whose manifest predates the R2 migration (v1) is skipped so
* the migration commit itself doesn't post hundreds of bogus entries.
*
* CLI (from tests/):
* tsx tools/screenshots/changelog.ts [--base REV] [--head REV] [--dry-run] [--force]
*/
import * as fs from 'fs';
import * as path from 'path';
import { execFileSync } from 'child_process';
import { PNG } from 'pngjs';
import { DIFF_OUT_DIR, MANIFEST_PATH, MANIFEST_VERSION, floorFor, labelText, splitKey, LABEL, type Manifest } from './config';
import { comparePair, type Report } from './compare';
import { savePng, withBottomLabel } from './image-ops';
import { buildSpecResolver } from './spec-map';
import { buildAttachments, paginate, postMessage } from './post-discord';
import { missingEnv, storeFromEnv, type R2Store } from './r2-store';
function git(root: string, args: string[]): string {
return execFileSync('git', ['-C', root, ...args], { encoding: 'utf8' }).trim();
}
/** The screenshot manifest as committed at `rev`. Returns null ONLY when the
* file doesn't exist at that rev (pre-manifest era); any other git failure —
* shallow clone, bad revision, moved path — throws loudly instead of being
* mistaken for "predates the migration". Unparsable committed JSON also throws. */
function manifestAt(repoRoot: string, rev: string, repoPath: string): Manifest | null {
let raw: string;
try {
raw = execFileSync('git', ['-C', repoRoot, 'show', `${rev}:${repoPath}`], { encoding: 'utf8' });
} catch (e) {
const msg = ((e as { stderr?: Buffer }).stderr?.toString() ?? (e as Error).message).trim();
if (/does not exist in|exists on disk, but not in/i.test(msg)) return null;
throw new Error(`git show ${rev}:${repoPath} failed: ${msg}`);
}
return JSON.parse(raw) as Manifest;
}
function parseArgs(argv: string[]): Record<string, string | boolean> {
const out: Record<string, string | boolean> = {};
for (let i = 0; i < argv.length; i++) {
const a = argv[i];
if (a === '--dry-run') out.dryRun = true;
else if (a === '--force') out.force = true;
else if (a === '--base') out.base = argv[++i];
else if (a === '--head') out.head = argv[++i];
}
return out;
}
async function main(): Promise<void> {
const args = parseArgs(process.argv.slice(2));
const cwd = process.cwd();
const repoRoot = git(cwd, ['rev-parse', '--show-toplevel']);
const head = (args.head as string) || 'HEAD';
let base = (args.base as string) || '';
if (!base) {
try {
base = git(cwd, ['rev-parse', `${head}^`]);
} catch {
console.log('[changelog] no parent commit — nothing to diff');
return;
}
}
// Repo-relative manifest path (e.g. tests/screenshot-manifest.json).
const manifestPath = path.relative(repoRoot, path.join(cwd, MANIFEST_PATH)).split(path.sep).join('/');
const baseManifest = manifestAt(repoRoot, base, manifestPath);
const headManifest = manifestAt(repoRoot, head, manifestPath);
// A NEWER manifest than this tooling understands is a loud error, never a skip.
for (const [label, m] of [['head', headManifest], ['base', baseManifest]] as const) {
if (m && m.version > MANIFEST_VERSION) {
throw new Error(`${label} manifest is version ${m.version}, newer than this tooling (expects ${MANIFEST_VERSION}) — update the checkout`);
}
}
if (headManifest?.version !== MANIFEST_VERSION) {
console.log('[changelog] head manifest predates the R2 migration — nothing to diff');
return;
}
if (baseManifest?.version !== MANIFEST_VERSION) {
console.log('[changelog] base manifest predates the R2 migration — skipping (migration commit)');
return;
}
// Engine-qualified key (`<engine>/<name>`) → sha256, per rev.
const hashesOf = (m: Manifest): Map<string, string> =>
new Map(m.screenshots.map((e) => [`${e.engine}/${e.name}`, e.sha256]));
const baseHashes = hashesOf(baseManifest);
const headHashes = hashesOf(headManifest);
const added = [...headHashes.keys()].filter((k) => !baseHashes.has(k));
const removed = [...baseHashes.keys()].filter((k) => !headHashes.has(k));
const changed = [...headHashes.keys()].filter((k) => baseHashes.has(k) && baseHashes.get(k) !== headHashes.get(k));
if (!added.length && !removed.length && !changed.length) {
console.log('[changelog] no baseline changes between', base.slice(0, 7), 'and', head);
return;
}
// Tolerate missing credentials (warn + exit 0): this is a notification-only
// workflow — turning every manifest push red during a key rotation is worse
// than one missed changelog post. The warning names the fix.
const store = storeFromEnv();
if (!store) {
console.warn(`[changelog] R2 credentials unset (${missingEnv().join(', ')}) — cannot fetch baseline bytes; skipping the changelog post`);
return;
}
const outDir = path.join(cwd, DIFF_OUT_DIR);
fs.mkdirSync(outDir, { recursive: true });
const report: Report = {
generatedFor: process.env.GITHUB_SHA || head,
changed: [],
added: [],
removed: [],
unchangedCount: 0,
driftLikely: false,
};
const { specFor } = buildSpecResolver(cwd);
// One unresolvable object must not abort the whole post — skip that image
// (with a loud note) and keep reporting the rest, like the old git-blob
// path null-skipped unresolvable blobs.
const skipped: string[] = [];
const hashPng = async (s: R2Store, hash: string, key: string): Promise<PNG | null> => {
try {
return PNG.sync.read(await s.get(hash));
} catch (e) {
skipped.push(`${key}: ${(e as Error).message}`);
return null;
}
};
// Save a single captioned image (added/removed) and record it in the report.
const saveSingle = (img: PNG, key: string, status: 'added' | 'removed'): void => {
const rel = path.join(DIFF_OUT_DIR, `${key.replace('/', '_')}.${status}.png`);
savePng(path.join(cwd, rel), withBottomLabel(img, labelText(status, key, specFor(splitKey(key).name)), LABEL.colors[status]));
report[status].push({ name: key, image: rel });
};
for (const key of added) {
const img = await hashPng(store, headHashes.get(key)!, key);
if (img) saveSingle(img, key, 'added');
}
for (const key of removed) {
const img = await hashPng(store, baseHashes.get(key)!, key);
if (img) saveSingle(img, key, 'removed');
}
for (const key of changed) {
const oldImg = await hashPng(store, baseHashes.get(key)!, key);
const newImg = await hashPng(store, headHashes.get(key)!, key);
if (!oldImg || !newImg) continue;
const { result, heatmap, triptych } = comparePair(oldImg, newImg, key, floorFor(key));
const triptychRel = path.join(DIFF_OUT_DIR, `${key.replace('/', '_')}.triptych.png`);
const heatmapRel = path.join(DIFF_OUT_DIR, `${key.replace('/', '_')}.heatmap.png`);
savePng(path.join(cwd, triptychRel), withBottomLabel(triptych, labelText('changed', key, specFor(splitKey(key).name)), LABEL.colors.changed));
savePng(path.join(cwd, heatmapRel), heatmap);
report.changed.push({ ...result, triptych: triptychRel, heatmap: heatmapRel });
}
report.changed.sort((a, b) => b.changedRatio - a.changedRatio);
if (skipped.length) {
console.warn(`[changelog] skipped ${skipped.length} unresolvable image(s):\n ${skipped.join('\n ')}`);
}
const sha7 = (process.env.GITHUB_SHA || head).slice(0, 7);
let subject = '';
try {
subject = git(cwd, ['log', '-1', '--pretty=%s', head]);
} catch { /* ignore */ }
const header =
`🗂️ **Baseline changelog** · \`${sha7}\`` +
(subject ? `\n> ${subject}` : '') +
`\n${report.changed.length} changed, ${report.added.length} added, ${report.removed.length} removed` +
(skipped.length ? `\n⚠️ ${skipped.length} image(s) unresolvable in R2 — see the workflow log` : '') +
(report.removed.length ? '\n➖ REMOVED: ' + report.removed.map((r) => `\`${r.name}\``).join(', ') : '');
const { files, notes } = buildAttachments(cwd, report);
const messages = paginate(header + (notes.length ? '\n' + notes.join('\n') : ''), files);
const isMainPush = process.env.GITHUB_REF === 'refs/heads/main' && process.env.GITHUB_EVENT_NAME === 'push';
if (args.dryRun || (!args.force && !isMainPush)) {
for (const [i, m] of messages.entries()) {
console.log(`--- message ${i + 1}/${messages.length} (${m.files.length} files) ---`);
if (m.content) console.log(m.content);
for (const f of m.files) console.log(` [attach] ${f.name} (${f.buffer.length} bytes)`);
}
if (!args.dryRun) console.log('[changelog] not a push to main — not posting');
return;
}
const webhook = process.env.DISCORD_WEBHOOK_URL;
if (!webhook) {
console.log('[changelog] DISCORD_WEBHOOK_URL unset — skipping');
return;
}
for (const m of messages) await postMessage(webhook, m);
console.log(`[changelog] posted ${messages.length} message(s)`);
}
if (require.main === module) {
main().catch((e) => {
console.error(`[changelog] ${e.message}`);
process.exitCode = 1;
});
}

13
tests/tools/screenshots/compare.ts
View file

@ -9,8 +9,8 @@
* The gate mode classifies every screenshot into changed / added / removed /
* unchanged, writes per-change triptych + heatmap PNGs and a machine-readable
* report.json into DIFF_OUT_DIR, and (unless --fail-on-change) exits 0 so it can
* run report-only first. post-discord.ts and the changelog workflow import the
* exported helpers rather than re-deriving the diff.
* run report-only first. post-discord.ts imports the exported helpers rather
* than re-deriving the diff.
*/
import * as fs from 'fs';
import * as path from 'path';
@ -236,7 +236,14 @@ function main(): void {
// dev who hasn't fetched. Either way skip without writing report.json —
// post-discord posts a distinct "gate SKIPPED" line when it's absent.
const gateManifest = loadManifest(root);
const wanted = (gateManifest?.screenshots ?? []).filter((e) => !isIgnored(`${e.engine}/${e.name}`));
// No fetched manifest at all (creds absent, R2 down, or morelli never
// seeded) ⇒ the gate has no source of truth — skip rather than diff
// against whatever stale cache a previous run left behind.
if (!gateManifest) {
console.log(`[compare] no fetched ${MANIFEST_PATH} — run \`npm run screenshots:fetch-manifest\` (needs R2 credentials); skipping`);
return;
}
const wanted = gateManifest.screenshots.filter((e) => !isIgnored(`${e.engine}/${e.name}`));
const missing = wanted.filter((e) => !fs.existsSync(path.join(root, BASELINE_ROOT, e.engine, e.name)));
if (wanted.length && missing.length) {
const what = missing.length === wanted.length ? 'empty' : `INCOMPLETE (${missing.length}/${wanted.length} missing — partial fetch?)`;

43
tests/tools/screenshots/config.ts
View file

@ -20,16 +20,18 @@ import * as nodePath from 'path';
* other's PNG, and the surviving file was whichever parallel worker wrote last.
*
* The baseline PNGs are NOT committed: they live in a private R2 bucket,
* content-addressed by sha256, and the committed manifest
* (screenshot-manifest.json) is what pins each `<engine>/<name>` to a hash.
* `npm run screenshots:fetch` (tools/screenshots/r2-sync.ts) materializes this
* tree from the manifest; promote.ts uploads new hashes and rewrites the
* manifest — the manifest diff is the only thing that lands in git.
* content-addressed by sha256, pinned by the R2-HOSTED manifest
* (baselines/pcbjam/manifest.json — nothing manifest-related is in git).
* `npm run screenshots:fetch-manifest && npm run screenshots:fetch`
* (tools/screenshots/r2-sync.ts) materializes this tree. Baselines are
* promoted from a CI run in the morelli review app
* (https://pcbjam-morelli-staging.pcbjam-staging.workers.dev) — the old
* promote.ts/git-manifest flow is retired.
*/
export const BASELINE_ROOT = 'baseline-screenshots';
/** Manifest format version — bumped when baselines moved from git to R2. */
export const MANIFEST_VERSION = 2;
/** Manifest format version — v3 = the R2-hosted manifest written by morelli (v2 was the committed-in-git era). */
export const MANIFEST_VERSION = 3;
/** Default private R2 bucket holding the content-addressed baselines. */
export const R2_DEFAULT_BUCKET = 'pcbjam-ci-screenshots';
@ -37,6 +39,9 @@ export const R2_DEFAULT_BUCKET = 'pcbjam-ci-screenshots';
/** Key prefix for content-addressed objects: `sha256/<64-hex>.png`. */
export const R2_KEY_PREFIX = 'sha256/';
/** The R2-hosted baseline manifest — THE source of truth. Written only by morelli (promote) and its seed script. */
export const R2_BASELINES_MANIFEST_KEY = 'baselines/pcbjam/manifest.json';
/**
* Env vars for the bucket's S3 API (read-only keypair in CI, read-write for
* devs running promote). Endpoint form: https://<account-id>.r2.cloudflarestorage.com
@ -80,8 +85,14 @@ export const RESULTS_DIR = 'test-results';
/** Where compare.ts writes diff/heatmap/triptych artifacts (gitignored). */
export const DIFF_OUT_DIR = 'test-results/screenshot-diff';
/** The manifest that records every expected screenshot + which engine renders it. */
export const MANIFEST_PATH = 'screenshot-manifest.json';
/**
* Local, GITIGNORED copy of the R2-hosted baseline manifest, downloaded by
* `npm run screenshots:fetch-manifest` (r2-sync --manifest). Everything
* downstream (fetch, compare, verify) reads this file; its absence means "no
* manifest fetched" and the screenshot gate skips. Deliberately NOT inside
* test-results/ — Playwright wipes that dir at the start of every invocation.
*/
export const MANIFEST_PATH = '.baseline-manifest.json';
/**
* pixelmatch per-pixel settings.
@ -185,10 +196,22 @@ export function labelText(status: LabelStatus, key: string, spec: string | null)
* `height` are sanity metadata (cheap pre-hash check on fetch, dimension info
* in reviews).
*/
export type ManifestEntry = { name: string; engine: string; sha256: string; bytes: number; width: number; height: number };
export type ManifestEntry = {
name: string;
engine: string;
sha256: string;
bytes: number;
width: number;
height: number;
/** morelli-side provenance (which run/branch/user promoted this) — opaque to the CI tooling. */
source?: unknown;
};
export type Manifest = {
version: number;
pipeline?: string;
storage: { bucket: string; keyPrefix: string };
updatedAt?: string;
updatedBy?: string;
screenshots: ManifestEntry[];
};

191
tests/tools/screenshots/gen-manifest.ts
View file

@ -1,191 +0,0 @@
/**
* Generate tests/screenshot-manifest.json — the canonical list of expected
* screenshots AND the pointer to their bytes. Each entry is
* {name, engine, sha256, bytes, width, height}: the engine comes from the
* baseline cache tree (baseline-screenshots/<engine>/<name>.png), the sha256
* resolves the PNG in the private R2 bucket (`sha256/<hex>.png`). The manifest
* is the SOURCE OF TRUTH — the local tree is a cache materialized from it by
* r2-sync (`npm run screenshots:fetch`).
*
* promote.ts is the single writer: it regenerates this file after applying and
* uploading a CI run's render. Regeneration therefore needs the local tree
* present; --check deliberately does NOT (it must stay credential- and
* network-free so it can gate every CI caller, secretless ones included).
*
* CLI (from tests/): tsx tools/screenshots/gen-manifest.ts [--check [--remote]]
* --check exit 1 unless the committed manifest is well-formed (v2 schema,
* sorted+unique entries, valid hashes) and no baseline PNG has been
* re-committed to git (resurrection guard).
* --remote with --check: also HEAD every hash in R2 (needs credentials;
* manual use only, not wired into CI).
*/
import * as fs from 'fs';
import * as path from 'path';
import { execFileSync } from 'child_process';
import { PNG } from 'pngjs';
import {
BASELINE_ROOT,
ENGINES,
IGNORE_SCREENSHOTS,
MANIFEST_PATH,
MANIFEST_VERSION,
R2_DEFAULT_BUCKET,
R2_KEY_PREFIX,
isIgnored,
listEngineKeys,
splitKey,
type Manifest,
type ManifestEntry,
} from './config';
import { hashFile, missingEnv, storeFromEnv } from './r2-store';
/** Locale-independent (code-unit) ordering — the writer runs on dev machines,
* the --check gate on CI, and localeCompare's result depends on the host locale. */
function compareEntries(a: { engine: string; name: string }, b: { engine: string; name: string }): number {
if (a.engine !== b.engine) return a.engine < b.engine ? -1 : 1;
return a.name < b.name ? -1 : a.name > b.name ? 1 : 0;
}
const NOTE =
'Baselines live in R2, content-addressed by sha256 — the local baseline-screenshots/ tree is a cache ' +
'(`npm run screenshots:fetch`). Regenerated by `npm run screenshots:promote`; never edit hashes by hand. ' +
'The name list is authoritative for removed-screenshot detection.';
/** Render the manifest JSON for the current baseline cache tree (tree must be present). */
export function manifestJson(root: string): string {
const base = path.join(root, BASELINE_ROOT);
const screenshots: ManifestEntry[] = listEngineKeys(base)
.filter((key) => !isIgnored(key))
.map((key) => {
const { engine, name } = splitKey(key);
const file = path.join(base, key);
const png = PNG.sync.read(fs.readFileSync(file));
return {
name,
engine,
sha256: hashFile(file),
bytes: fs.statSync(file).size,
width: png.width,
height: png.height,
};
})
.sort(compareEntries);
const manifest: Manifest & { _note: string } = {
_note: NOTE,
version: MANIFEST_VERSION,
storage: { bucket: R2_DEFAULT_BUCKET, keyPrefix: R2_KEY_PREFIX },
screenshots,
};
return JSON.stringify(manifest, null, 2) + '\n';
}
/** Regenerate the manifest on disk (used by the CLI and by promote.ts after apply). */
export function writeManifest(root: string): void {
fs.writeFileSync(path.join(root, MANIFEST_PATH), manifestJson(root));
}
/** Schema + hygiene problems with the committed manifest (empty = healthy). Credential-free. */
export function checkManifest(root: string): string[] {
const problems: string[] = [];
const p = path.join(root, MANIFEST_PATH);
if (!fs.existsSync(p)) return [`${MANIFEST_PATH} is missing`];
let manifest: Manifest;
try {
manifest = JSON.parse(fs.readFileSync(p, 'utf8')) as Manifest;
} catch (e) {
return [`${MANIFEST_PATH} is not valid JSON: ${(e as Error).message}`];
}
if (manifest.version !== MANIFEST_VERSION) problems.push(`version is ${manifest.version}, expected ${MANIFEST_VERSION}`);
if (!manifest.storage?.bucket || !manifest.storage?.keyPrefix) problems.push('storage.bucket/keyPrefix missing');
if (!Array.isArray(manifest.screenshots) || manifest.screenshots.length === 0) {
problems.push('screenshots list is missing or empty');
return problems;
}
const engines = new Set<string>(ENGINES);
let prev: ManifestEntry | null = null;
const seen = new Set<string>();
for (const e of manifest.screenshots) {
const id = `${e.engine}/${e.name}`;
if (!e.name?.toLowerCase().endsWith('.png')) problems.push(`${id}: name is not a .png`);
if (!engines.has(e.engine)) problems.push(`${id}: unknown engine`);
if (!/^[0-9a-f]{64}$/.test(e.sha256 ?? '')) problems.push(`${id}: sha256 is not 64 lowercase hex chars`);
if (!(e.bytes > 0) || !(e.width > 0) || !(e.height > 0)) problems.push(`${id}: bytes/width/height must be positive`);
if (IGNORE_SCREENSHOTS.has(e.name)) problems.push(`${id}: is in IGNORE_SCREENSHOTS and must not be listed`);
if (seen.has(id)) problems.push(`${id}: duplicate entry`);
seen.add(id);
if (prev && compareEntries(prev, e) > 0) {
problems.push(`${id}: not sorted (engine, then name)`);
}
prev = e;
}
// Resurrection guard: baselines must never come back as committed files.
try {
const tracked = execFileSync('git', ['ls-files', '--', BASELINE_ROOT], { cwd: root, encoding: 'utf8' }).trim();
if (tracked) {
problems.push(
`git-tracked PNGs under ${BASELINE_ROOT}/ — baselines live in R2, do not commit them:\n ` +
tracked.split('\n').slice(0, 5).join('\n ')
);
}
} catch {
console.warn('[manifest] git unavailable — skipping the resurrection guard');
}
return problems;
}
async function main(): Promise<void> {
const check = process.argv.includes('--check');
const remote = process.argv.includes('--remote');
const root = process.cwd();
const outPath = path.join(root, MANIFEST_PATH);
if (check) {
const problems = checkManifest(root);
if (problems.length) {
console.error(`[manifest] INVALID:\n - ${problems.join('\n - ')}`);
process.exitCode = 1;
return;
}
const manifest = JSON.parse(fs.readFileSync(outPath, 'utf8')) as Manifest;
console.log(`[manifest] ok — ${manifest.screenshots.length} screenshots across engines`);
if (remote) {
const store = storeFromEnv();
if (!store) {
console.error(`[manifest] --remote needs credentials: set ${missingEnv().join(', ')}`);
process.exitCode = 2;
return;
}
const { verifyBaselines } = await import('./r2-sync');
const missing = await verifyBaselines(root, store);
if (missing.length) {
console.error(`[manifest] ${missing.length} hash(es) missing in R2:\n ${missing.join('\n ')}`);
process.exitCode = 1;
} else {
console.log('[manifest] every hash present in R2');
}
}
return;
}
// Regeneration path: requires the cache tree (an absent tree would silently
// produce an empty manifest and disable removed-screenshot detection).
if (listEngineKeys(path.join(root, BASELINE_ROOT)).length === 0) {
console.error('[manifest] baseline cache is empty — run `npm run screenshots:fetch` first (or promote a run)');
process.exitCode = 1;
return;
}
const json = manifestJson(root);
fs.writeFileSync(outPath, json);
const count = (JSON.parse(json) as Manifest).screenshots.length;
console.log(`[manifest] wrote ${MANIFEST_PATH} (${count} screenshots)`);
}
if (require.main === module) {
main().catch((e) => {
console.error(`[manifest] ${(e as Error).message}`);
process.exitCode = 1;
});
}

2
tests/tools/screenshots/post-discord.ts
View file

@ -142,7 +142,7 @@ function buildHeader(report: Report | null, perfBlock: string, meta: { sha?: str
} else if (!report.changed.length && !report.added.length && !report.removed.length) {
lines.push('✅ no screenshot drift');
} else if (report.driftLikely) {
lines.push(`⚠️ **${report.changed.length} screenshots changed broadly** — looks like host env drift → re-promote (\`npm run screenshots:promote\`)`);
lines.push(`⚠️ **${report.changed.length} screenshots changed broadly** — looks like host env drift → re-promote in morelli (https://pcbjam-morelli-staging.pcbjam-staging.workers.dev)`);
} else {
lines.push(
`⚠️ **screenshot drift**: ${report.changed.length} changed${byEngine(report.changed)}, ` +

205
tests/tools/screenshots/promote.ts
View file

@ -1,205 +0,0 @@
/**
* Churn-free baseline updater — "promote CI's render".
*
* CI's x86 render is the source of truth. This pulls a CI run's screenshots
* (`gh run download`) — or a local dir via --from — and, for each one, overwrites
* the cached baseline ONLY when the decoded pixels differ beyond the per-engine
* floor. Unchanged baselines keep their bytes (never re-encoded), so their hash —
* and therefore the manifest — sees no churn. New shots are added; baselines with
* no render are reported as removal candidates and only dropped with --prune.
*
* Baselines live in a private R2 bucket (content-addressed; see r2-store.ts).
* The flow is: sync the local cache from the committed manifest → build/apply
* the plan locally → upload the updated/added bytes to R2 → regenerate the
* manifest. Only the manifest diff is committed; R2 objects are immutable and
* never deleted (--prune removes the manifest entry, old commits still resolve).
* Needs the READ-WRITE credential pair (see tools/screenshots/README.md).
*
* CLI (from tests/):
* tsx tools/screenshots/promote.ts --run <ci-run-id> [--repo owner/repo] [--prune] [--dry-run]
* tsx tools/screenshots/promote.ts --from <dir> [--prune] [--dry-run]
*/
import * as fs from 'fs';
import * as os from 'os';
import * as path from 'path';
import { execFileSync } from 'child_process';
import { BASELINE_ROOT, MANIFEST_PATH, floorFor, isIgnored, listEngineKeys, splitKey, type Manifest } from './config';
import { writeManifest } from './gen-manifest';
import { diffImages, loadPng } from './image-ops';
import { hashBytes, missingEnv, storeFromEnv, type R2Store } from './r2-store';
import { loadManifestV2, pool, pullBaselines } from './r2-sync';
/** key (`<engine>/<name>`) → absolute cached baseline path. */
function baselineIndex(root: string): Map<string, string> {
const abs = path.join(root, BASELINE_ROOT);
const index = new Map<string, string>();
for (const key of listEngineKeys(abs)) index.set(key, path.join(abs, key));
return index;
}
function loadManifest(root: string): Manifest | undefined {
const p = path.join(root, MANIFEST_PATH);
if (!fs.existsSync(p)) return undefined;
try {
return JSON.parse(fs.readFileSync(p, 'utf8')) as Manifest;
} catch {
return undefined;
}
}
/** Download a CI run's artifact and return the top-level test-results dir holding the shots. */
function downloadRun(runId: string, repo?: string): string {
const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'promote-'));
const repoArgs = repo ? ['--repo', repo] : [];
execFileSync('gh', ['run', 'download', runId, '-D', tmp, ...repoArgs], { stdio: 'inherit' });
// The artifact stores test-results/** — find that dir; its immediate *.png are the shots
// (exclude the nested screenshot-diff/ triptychs).
const stack = [tmp];
while (stack.length) {
const dir = stack.pop()!;
const entries = fs.readdirSync(dir, { withFileTypes: true });
if (path.basename(dir) === 'test-results') return dir;
for (const e of entries) if (e.isDirectory()) stack.push(path.join(dir, e.name));
}
throw new Error(`no test-results/ dir found in the downloaded artifact under ${tmp}`);
}
type Plan = { updated: string[]; added: string[]; unchanged: string[]; removedCandidates: string[] };
function buildPlan(root: string, renderDir: string, manifest?: Manifest): { plan: Plan; apply: () => void } {
const baselines = baselineIndex(root);
const rendered = new Set(listEngineKeys(renderDir));
// Never promote excluded screenshots (e.g. the flaky retinascale fullPage shot).
for (const key of [...rendered]) if (isIgnored(key)) rendered.delete(key);
for (const key of [...baselines.keys()]) if (isIgnored(key)) baselines.delete(key);
const plan: Plan = { updated: [], added: [], unchanged: [], removedCandidates: [] };
const actions: Array<() => void> = [];
for (const key of rendered) {
const src = path.join(renderDir, key);
const existing = baselines.get(key);
if (!existing) {
const dest = path.join(root, BASELINE_ROOT, key);
plan.added.push(key);
actions.push(() => {
fs.mkdirSync(path.dirname(dest), { recursive: true });
fs.copyFileSync(src, dest); // verbatim bytes
});
continue;
}
const d = diffImages(loadPng(existing), loadPng(src));
const floor = floorFor(key);
if (!d.dimsMatch || d.changedRatio > floor.changedRatio) {
plan.updated.push(key);
actions.push(() => fs.copyFileSync(src, existing)); // verbatim bytes, no re-encode → no churn
} else {
plan.unchanged.push(key); // leave the cached file untouched
}
}
// Removal candidates: a manifest-listed baseline this render didn't produce.
for (const [key, abs] of baselines) {
if (rendered.has(key)) continue;
const { engine, name } = splitKey(key);
if (manifest && !manifest.screenshots.some((e) => e.name === name && e.engine === engine)) continue;
plan.removedCandidates.push(key);
actions.push(() => {}); // pruning is opt-in (see main)
void abs;
}
return { plan, apply: () => actions.forEach((a) => a()) };
}
/** Upload the applied updated/added baselines to R2 (skip-if-exists per hash). */
async function uploadApplied(root: string, store: R2Store, keys: string[]): Promise<number> {
let uploaded = 0;
await pool(keys, 8, async (key) => {
const bytes = fs.readFileSync(path.join(root, BASELINE_ROOT, key));
if ((await store.put(hashBytes(bytes), bytes)) === 'uploaded') uploaded++;
});
return uploaded;
}
function parseArgs(argv: string[]): Record<string, string | boolean> {
const out: Record<string, string | boolean> = {};
for (let i = 0; i < argv.length; i++) {
const a = argv[i];
if (a === '--run') out.run = argv[++i];
else if (a === '--from') out.from = argv[++i];
else if (a === '--repo') out.repo = argv[++i];
else if (a === '--prune') out.prune = true;
else if (a === '--dry-run') out.dryRun = true;
}
return out;
}
async function main(): Promise<void> {
const args = parseArgs(process.argv.slice(2));
const root = process.cwd();
if (!args.run && !args.from) {
console.error('usage: promote.ts --run <ci-run-id> [--repo owner/repo] | --from <dir> [--prune] [--dry-run]');
process.exitCode = 2;
return;
}
// Fail fast BEFORE downloading anything: a promote that can't upload would
// otherwise leave the manifest referencing hashes that don't exist in R2.
const store = storeFromEnv();
if (!store) {
console.error(`[promote] R2 read-write credentials required: set ${missingEnv().join(', ')}`);
process.exitCode = 2;
return;
}
// Refuse a pre-migration checkout rather than warn: buildPlan would mark
// most renders "unchanged" (so never uploaded) while writeManifest pins
// their hashes anyway — committing a manifest that references objects R2
// doesn't have. Sync the checkout to a post-migration revision first.
if (!loadManifestV2(root)) {
console.error(`[promote] ${MANIFEST_PATH} is not the R2-backed v2 format — refusing to promote on a pre-migration checkout`);
process.exitCode = 2;
return;
}
// The local tree is a cache — sync it to the committed manifest so the plan
// diffs against exactly what the manifest pins (a stale/absent cache would
// otherwise misreport adds/updates).
const { downloaded, cached, deleted } = await pullBaselines(root, store);
console.log(`[promote] cache synced: downloaded=${downloaded} cached=${cached} deleted=${deleted}`);
const renderDir = args.from ? (args.from as string) : downloadRun(args.run as string, args.repo as string);
const manifest = loadManifest(root);
const { plan, apply } = buildPlan(root, renderDir, manifest);
console.log(
`[promote] updated=${plan.updated.length} added=${plan.added.length} ` +
`unchanged=${plan.unchanged.length} removal-candidates=${plan.removedCandidates.length}`
);
for (const n of plan.updated) console.log(` UPDATE ${n}`);
for (const n of plan.added) console.log(` ADD ${n}`);
for (const n of plan.removedCandidates) console.log(` REMOVE? ${n}${args.prune ? ' (pruning)' : ' (use --prune to delete)'}`);
if (args.dryRun) {
console.log('[promote] dry-run — no files written, nothing uploaded');
return;
}
apply();
if (args.prune) {
const baselines = baselineIndex(root);
// Prune drops the local file (and, below, the manifest entry). The R2
// object is deliberately kept — old commits must still resolve it.
for (const n of plan.removedCandidates) fs.rmSync(baselines.get(n)!, { force: true });
}
// Upload BEFORE regenerating the manifest: a failure here leaves at worst an
// orphaned CAS object, never a committed manifest pointing at a missing hash.
const uploaded = await uploadApplied(root, store, [...plan.updated, ...plan.added]);
console.log(`[promote] uploaded ${uploaded} object(s) to R2`);
// Keep the manifest in lockstep with the baseline tree — a stale manifest silently
// disables removed-screenshot detection for anything added after the last regen.
writeManifest(root);
console.log(`[promote] done — commit the ${MANIFEST_PATH} diff (the only git-visible output)`);
}
if (require.main === module) {
main().catch((e) => {
console.error(`[promote] ${(e as Error).message}`);
process.exitCode = 1;
});
}

15
tests/tools/screenshots/r2-store.ts
View file

@ -99,6 +99,21 @@ export class R2Store {
return 'uploaded';
}
/**
* Download an ARBITRARY key (the R2-hosted baseline manifest) — unlike
* get(), not content-addressed, so no hash verification is possible; null
* on 404 so the caller can degrade like the no-credentials path.
*/
async getKey(key: string): Promise<Buffer | null> {
const res = await this.fetchWithRetry(`${this.base}/${key}`, { method: 'GET' });
if (res.status === 404) {
await res.arrayBuffer().catch(() => undefined);
return null;
}
if (res.status !== 200) throw new Error(`GET ${key} → HTTP ${res.status}`);
return Buffer.from(await res.arrayBuffer());
}
/**
* Upload to an ARBITRARY key (the per-run uploads under runs/…, consumed by
* the morelli review app) — unlike put(), not content-addressed and always

114
tests/tools/screenshots/r2-sync.ts
View file

@ -1,31 +1,38 @@
/**
* Sync the local baseline-screenshots/ cache with the R2 CAS bucket.
*
* The committed manifest (screenshot-manifest.json) is the source of truth;
* baseline-screenshots/ is a gitignored local cache materialized from it.
* The R2-HOSTED manifest (baselines/pcbjam/manifest.json, written only by the
* morelli review app + its seed script) is the source of truth. --manifest
* downloads it to the gitignored MANIFEST_PATH; everything downstream (--pull,
* --verify, compare.ts) reads that local copy, so a single fetch pins the
* whole run to one manifest version.
*
* CLI (from tests/):
* tsx tools/screenshots/r2-sync.ts --pull # manifest → local tree (CI fetch step + local gate)
* tsx tools/screenshots/r2-sync.ts --push # upload manifest entries missing in R2 (seeding)
* tsx tools/screenshots/r2-sync.ts --manifest # R2 manifest → local .baseline-manifest.json
* tsx tools/screenshots/r2-sync.ts --pull # fetched manifest → local tree (CI fetch step + local gate)
* tsx tools/screenshots/r2-sync.ts --verify # HEAD every manifest hash, exit 1 on any miss
*
* --pull without credentials (or with a pre-migration manifest) warns and
* exits 0, so secretless callers of the reusable CI workflow (release.yml,
* pcbjam deploy-staging.yml, fork PRs) stay green — compare.ts then skips its
* gate for the same reason. With credentials, any 404/corrupt object is
* collected and the run exits 1.
* --manifest and --pull without credentials warn and exit 0 — and --manifest
* DELETES a stale local manifest so the gate skips rather than comparing
* against outdated baselines. Secretless callers of the reusable CI workflow
* (release.yml, pcbjam deploy-staging.yml, fork PRs) therefore stay green;
* compare.ts skips its gate when no manifest was fetched. With credentials,
* any 404/corrupt object is collected and the run exits 1.
*
* (--push is gone with the git-manifest era: baseline bytes now enter the CAS
* only via morelli's promote, which copies them from the CI run uploads.)
*/
import * as fs from 'fs';
import * as path from 'path';
import { BASELINE_ROOT, MANIFEST_PATH, MANIFEST_VERSION, listEngineKeys, type Manifest, type ManifestEntry } from './config';
import { BASELINE_ROOT, MANIFEST_PATH, MANIFEST_VERSION, R2_BASELINES_MANIFEST_KEY, listEngineKeys, type Manifest, type ManifestEntry } from './config';
import { R2Store, hashFile, missingEnv, storeFromEnv } from './r2-store';
const CONCURRENCY = 16;
/** Parse the committed manifest if it is the R2-backed v2 format; null for a
* pre-migration (v1/absent/unparsable) manifest. A NEWER version throws — old
* tooling silently no-oping on a future format would disable the whole gate. */
export function loadManifestV2(root: string): Manifest | null {
/** Parse the fetched manifest if it is the morelli-era v3 format; null when
* absent/unparsable (→ the gate skips). A NEWER version throws — old tooling
* silently no-oping on a future format would disable the whole gate. */
export function loadManifest(root: string): Manifest | null {
const p = path.join(root, MANIFEST_PATH);
if (!fs.existsSync(p)) return null;
let m: Manifest;
@ -40,6 +47,25 @@ export function loadManifestV2(root: string): Manifest | null {
return m.version === MANIFEST_VERSION ? m : null;
}
/**
* Download the R2-hosted baseline manifest to MANIFEST_PATH (atomic tmp+rename).
* Returns false when it could not be fetched — in which case any stale local
* copy is removed, so downstream steps skip instead of using old baselines.
*/
export async function fetchManifest(root: string, store: R2Store): Promise<boolean> {
const dest = path.join(root, MANIFEST_PATH);
const bytes = await store.getKey(R2_BASELINES_MANIFEST_KEY);
if (!bytes) {
fs.rmSync(dest, { force: true });
console.warn(`[r2-sync] ${R2_BASELINES_MANIFEST_KEY} not found in R2 — no baselines (seed via morelli first)`);
return false;
}
const tmp = `${dest}.tmp-${process.pid}`;
fs.writeFileSync(tmp, bytes);
fs.renameSync(tmp, dest);
return true;
}
/** Run `fn` over `items` with at most `limit` in flight. */
export async function pool<T>(items: T[], limit: number, fn: (item: T) => Promise<void>): Promise<void> {
let i = 0;
@ -60,8 +86,8 @@ export async function pullBaselines(
root: string,
store: R2Store
): Promise<{ downloaded: number; cached: number; deleted: number }> {
const manifest = loadManifestV2(root);
if (!manifest) throw new Error(`no v2 ${MANIFEST_PATH} — nothing to pull`);
const manifest = loadManifest(root);
if (!manifest) throw new Error(`no fetched ${MANIFEST_PATH} — run \`npm run screenshots:fetch-manifest\` first`);
const base = path.join(root, BASELINE_ROOT);
const wanted = new Map<string, ManifestEntry>();
for (const e of manifest.screenshots) wanted.set(`${e.engine}/${e.name}`, e);
@ -99,35 +125,10 @@ export async function pullBaselines(
return { downloaded, cached, deleted };
}
/** Upload every manifest entry's local file to R2 (skipping hashes already present). */
export async function pushBaselines(root: string, store: R2Store): Promise<{ uploaded: number; existing: number }> {
const manifest = loadManifestV2(root);
if (!manifest) throw new Error(`no v2 ${MANIFEST_PATH} — run \`npm run screenshots:manifest\` first`);
const base = path.join(root, BASELINE_ROOT);
let uploaded = 0;
let existing = 0;
const errors: string[] = [];
await pool(manifest.screenshots, CONCURRENCY, async (e) => {
const file = path.join(base, e.engine, e.name);
try {
if (!fs.existsSync(file)) throw new Error('local file missing');
const bytes = fs.readFileSync(file);
const hash = hashFile(file);
if (hash !== e.sha256) throw new Error(`local sha256 ${hash} ≠ manifest — regenerate the manifest`);
if ((await store.put(hash, bytes)) === 'uploaded') uploaded++;
else existing++;
} catch (err) {
errors.push(`${e.engine}/${e.name}: ${(err as Error).message}`);
}
});
if (errors.length) throw new Error(`${errors.length} upload(s) failed:\n ${errors.join('\n ')}`);
return { uploaded, existing };
}
/** HEAD every manifest hash; returns the missing keys. */
export async function verifyBaselines(root: string, store: R2Store): Promise<string[]> {
const manifest = loadManifestV2(root);
if (!manifest) throw new Error(`no v2 ${MANIFEST_PATH}`);
const manifest = loadManifest(root);
if (!manifest) throw new Error(`no fetched ${MANIFEST_PATH} — run \`npm run screenshots:fetch-manifest\` first`);
const missing: string[] = [];
await pool(manifest.screenshots, CONCURRENCY, async (e) => {
if (!(await store.exists(e.sha256))) missing.push(`${e.engine}/${e.name} (${e.sha256})`);
@ -136,20 +137,27 @@ export async function verifyBaselines(root: string, store: R2Store): Promise<str
}
async function main(): Promise<void> {
const mode = process.argv.find((a) => a === '--pull' || a === '--push' || a === '--verify');
const mode = process.argv.find((a) => a === '--manifest' || a === '--pull' || a === '--verify');
if (!mode) {
console.error('usage: r2-sync.ts --pull | --push | --verify');
console.error('usage: r2-sync.ts --manifest | --pull | --verify');
process.exitCode = 2;
return;
}
const root = process.cwd();
if (mode === '--pull' && !loadManifestV2(root)) {
console.log(`[r2-sync] ${MANIFEST_PATH} is not the R2-backed v2 format — nothing to fetch`);
return;
if (mode === '--pull' && !loadManifest(root)) {
console.log(`[r2-sync] no fetched ${MANIFEST_PATH} — skipping pull (run screenshots:fetch-manifest first)`);
return; // exit 0: manifest fetch skipped/failed → the gate skips, never gates on stale data
}
const store = storeFromEnv();
if (!store) {
if (mode === '--manifest') {
// Delete a stale copy so the downstream gate SKIPS instead of
// comparing against whatever manifest a previous run left behind.
fs.rmSync(path.join(root, MANIFEST_PATH), { force: true });
console.log(`[r2-sync] R2 credentials unset (${missingEnv().join(', ')}) — skipping manifest fetch`);
return; // exit 0: secretless CI callers stay green
}
if (mode === '--pull') {
console.log(`[r2-sync] R2 credentials unset (${missingEnv().join(', ')}) — skipping baseline fetch`);
return; // exit 0: secretless CI callers stay green
@ -159,12 +167,14 @@ async function main(): Promise<void> {
return;
}
if (mode === '--pull') {
if (mode === '--manifest') {
if (await fetchManifest(root, store)) {
const manifest = loadManifest(root);
console.log(`[r2-sync] fetched ${MANIFEST_PATH}: ${manifest?.screenshots.length ?? 0} baselines (updated ${manifest?.updatedAt ?? '?'} by ${manifest?.updatedBy ?? '?'})`);
}
} else if (mode === '--pull') {
const { downloaded, cached, deleted } = await pullBaselines(root, store);
console.log(`[r2-sync] pull done: downloaded=${downloaded} cached=${cached} deleted=${deleted}`);
} else if (mode === '--push') {
const { uploaded, existing } = await pushBaselines(root, store);
console.log(`[r2-sync] push done: uploaded=${uploaded} already-present=${existing}`);
} else {
const missing = await verifyBaselines(root, store);
if (missing.length) {