tests: harden the R2 baseline pipeline's failure paths (review findings)

- compare gate: refuse an INCOMPLETE cache, not just an empty one — a partial
  R2 fetch (continue-on-error step) previously misreported un-fetched
  baselines as ADDED and silently disabled their removed-detection.
- post-discord: a missing report.json now posts "gate SKIPPED", never the
  false "✅ no screenshot drift" — a disabled pipeline must look disabled.
- promote: hard-fail on a pre-migration (non-v2) manifest instead of warning;
  the warn path could commit manifest hashes never uploaded to R2.
- changelog: tolerate per-image R2 failures (skip + loud note, like the old
  git-blob null-skip) instead of aborting the whole post; missing creds warn
  + exit 0 (notification-only workflow); git-show failures are loud errors,
  no longer mistaken for "predates the migration".
- version guard: a manifest NEWER than the tooling throws everywhere instead
  of reading as a silent no-op.
- manifest ordering: locale-independent code-unit comparator in writer +
  checker (localeCompare depends on host locale; writer=dev Mac, checker=CI).
- r2-sync pull: byte-size pre-filter before hashing cached files.
- wasm-build.yml: restore-keys on the baseline cache — manifest changes now
  restore the previous tree and download only the delta.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Istvan Matejcsok 2026-08-19 08:51:44 +02:00
commit 453df4fb9c
7 changed files with 105 additions and 34 deletions

View file

@ -229,13 +229,18 @@ function main(): void {
}
const root = process.cwd();
// Baselines are a local cache fetched from R2 (`npm run screenshots:fetch`).
// When the manifest expects screenshots but the cache is empty — a secretless
// CI caller where the fetch step skipped, or a dev who hasn't fetched — skip
// the gate instead of misreporting every baseline as removed. No report.json
// is written; post-discord tolerates its absence.
const expected = loadManifest(root)?.screenshots.length ?? 0;
if (expected > 0 && listEngineKeys(path.join(root, BASELINE_ROOT)).length === 0) {
console.log('[compare] baseline cache is empty — run `npm run screenshots:fetch` (needs R2 credentials); skipping');
// The gate only runs against a COMPLETE cache: a partial one (an R2 blip on
// a few objects — the CI fetch step is continue-on-error) would misreport
// the un-fetched baselines' renders as ADDED and silently disable removed-
// detection for them. Fully missing = fetch skipped (no credentials) or a
// dev who hasn't fetched. Either way skip without writing report.json —
// post-discord posts a distinct "gate SKIPPED" line when it's absent.
const gateManifest = loadManifest(root);
const wanted = (gateManifest?.screenshots ?? []).filter((e) => !isIgnored(`${e.engine}/${e.name}`));
const missing = wanted.filter((e) => !fs.existsSync(path.join(root, BASELINE_ROOT, e.engine, e.name)));
if (wanted.length && missing.length) {
const what = missing.length === wanted.length ? 'empty' : `INCOMPLETE (${missing.length}/${wanted.length} missing — partial fetch?)`;
console.log(`[compare] baseline cache is ${what} — run \`npm run screenshots:fetch\` (needs R2 credentials); skipping`);
return;
}
const report = classify(root, (args.sha as string) || process.env.GITHUB_SHA || null);