screenshots: R2-hosted manifest becomes the baseline source of truth (morelli cutover)

The committed tests/screenshot-manifest.json is retired. CI now downloads
baselines/pcbjam/manifest.json (manifest v3, written only by the morelli
review app + its seed script) to the gitignored .baseline-manifest.json,
and everything downstream (pull, verify, compare) reads that copy:

- config.ts: MANIFEST_VERSION 3, MANIFEST_PATH .baseline-manifest.json,
  R2_BASELINES_MANIFEST_KEY; ManifestEntry grows opaque provenance
- r2-sync.ts: new --manifest mode (atomic fetch; no-creds skip DELETES a
  stale copy so the gate skips rather than using old baselines); --push
  gone (bytes enter the CAS only via morelli's promote)
- compare.ts: hard-skips when no manifest was fetched — a stale warm
  cache can never gate
- wasm-build.yml: fetch-manifest step before the baselines cache; cache
  key now hashes the fetched manifest; the gen-manifest --check lint gate
  goes with the committed manifest
- deleted: screenshot-manifest.json, promote.ts, changelog.ts,
  gen-manifest.ts, screenshot-changelog.yml, promote-screenshots skill
- docs (CLAUDE/README/TESTING/WHATWORKS/tools README): promote flow is
  now https://pcbjam-morelli-staging.pcbjam-staging.workers.dev

Validated locally against the real bucket: fetch-manifest (492), cold
pull 492 / warm pull cached=492, no-creds skip chain, compare gate skip.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Istvan Matejcsok 2026-08-19 11:54:20 +02:00
commit ec7a1b0787
20 changed files with 239 additions and 4793 deletions

View file

@ -1,42 +0,0 @@
---
name: promote-screenshots
description: Promote a CI run's screenshot renders as the new baselines in the R2 bucket (pcbjam-ci-screenshots). Churn-free - only meaningfully-changed images upload; git gets only the tests/screenshot-manifest.json diff, never PNGs. Needs the read-write R2 keypair in tests/.env. Usage - "/promote-screenshots <ci-run-id> [--prune]". (scoped to pcbjam/ - covers the KiCad WASM e2e pipeline in tests/)
---
# promote-screenshots (pcbjam)
Bless a CI run's rendered screenshots as the new baselines. Baselines live in
the private R2 bucket `pcbjam-ci-screenshots` (prod Cloudflare account),
content-addressed as `sha256/<hex>.png`; the committed
`tests/screenshot-manifest.json` pins each `<engine>/<name>` to a hash. **Only
the manifest diff lands in git — never PNGs.**
## Prerequisites
- The READ-WRITE R2 keypair in the gitignored `tests/.env` (auto-loaded by the
tooling, shell env wins; format in `tests/tools/screenshots/README.md`,
values from the team vault). **Never print, echo, or commit these values.**
If `.env` is missing, ask the user to fill it — promote fails fast without it.
- The CI run id: `gh run list --workflow ci-ubicloud.yml` on PCBJam/pcbjam
(mind the active `gh` account — PCBJam repos need `matejcsok-pcb`).
## Steps
1. From `tests/`, dry-run first and show the user the plan:
`npm run screenshots:promote -- --run <ci-run-id> --dry-run`
2. Sanity-check it: a handful of UPDATE/ADD lines for an intentional UI change
is normal; hundreds of UPDATEs means environment drift — stop and confirm
with the user before applying.
3. Apply: same command without `--dry-run`. Add `--prune` only when the user
confirms screenshots were intentionally removed (prune edits the manifest;
R2 objects are never deleted — old commits still resolve).
4. `git status` must show ONLY `tests/screenshot-manifest.json` modified.
Commit that diff; on main it triggers the Discord baseline changelog.
## Never
- Never commit files under `tests/baseline-screenshots/` (the CI manifest
check fails the build if you do) or the `.env`.
- Never promote local (Mac) renders via `--from` — CI's Linux render is the
only source of truth.
- Never hand-edit hashes in the manifest; promote regenerates it.

View file

@ -1,45 +0,0 @@
name: screenshot-changelog
# Discord trigger B: when a push to main changes the screenshot manifest (the
# committed pin of every R2-stored baseline), post an old | new+boxes | heatmap
# triptych per changed baseline (ADDED image / REMOVED title too). No build, no
# GPU — it diffs two git revisions of the manifest and fetches the PNG bytes
# from the R2 CAS bucket (immutable objects, so the old rev's hashes always
# resolve), so it runs in ~30s. Complements the re-render drift gate in
# wasm-build.yml (which catches un-blessed renders); this is the human-facing
# feed of intentional baseline updates as they land.
on:
push:
branches: [main]
paths:
- 'tests/screenshot-manifest.json'
concurrency:
group: screenshot-changelog-${{ github.ref }}
cancel-in-progress: false
jobs:
changelog:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
# HEAD^ is needed to diff the tip commit's manifest changes. (A push of
# multiple commits reports only the tip commit's baseline diff — fine for
# a changelog; baseline refreshes are single commits in practice.)
fetch-depth: 2
- uses: actions/setup-node@v4
with:
node-version: 20
- name: Install screenshot tooling deps
working-directory: tests
run: npm ci
- name: Post baseline changelog to Discord
working-directory: tests
env:
DISCORD_WEBHOOK_URL: ${{ secrets.DISCORD_WEBHOOK_URL }}
CI_SCREENSHOTS_S3_ENDPOINT: ${{ vars.CI_SCREENSHOTS_S3_ENDPOINT }}
CI_SCREENSHOTS_S3_ACCESS_KEY_ID: ${{ secrets.CI_SCREENSHOTS_S3_ACCESS_KEY_ID }}
CI_SCREENSHOTS_S3_SECRET_ACCESS_KEY: ${{ secrets.CI_SCREENSHOTS_S3_SECRET_ACCESS_KEY }}
run: npm run screenshots:changelog

View file

@ -318,19 +318,37 @@ jobs:
working-directory: tests working-directory: tests
run: npm ci run: npm ci
# Baseline screenshots live in a private R2 bucket, pinned by the committed # The baseline manifest lives in R2 (written only by the morelli review
# manifest; this cache keyed on the manifest hash makes most fetches a # app + its seed script) — download it FIRST so this whole run pins to
# no-op. restore-keys makes a manifest change INCREMENTAL (restore the # one manifest version and the cache step below can key on its hash.
# previous tree, download only the changed objects) instead of a full # continue-on-error + the no-creds skip (which also deletes any stale
# ~35MB re-download. A stale or partial restore is safe: r2-sync # local copy) keep secretless callers green — every later screenshot
# hash-verifies every file and deletes unlisted ones, and compare.ts # step then skips rather than gating on outdated baselines.
# refuses to gate against an incomplete cache. - name: Fetch baseline manifest from R2
if: inputs.run_tests
continue-on-error: true
working-directory: tests
env:
CI_SCREENSHOTS_S3_ENDPOINT: ${{ vars.CI_SCREENSHOTS_S3_ENDPOINT }}
CI_SCREENSHOTS_S3_ACCESS_KEY_ID: ${{ secrets.CI_SCREENSHOTS_S3_ACCESS_KEY_ID }}
CI_SCREENSHOTS_S3_SECRET_ACCESS_KEY: ${{ secrets.CI_SCREENSHOTS_S3_SECRET_ACCESS_KEY }}
run: npm run screenshots:fetch-manifest
# Baseline screenshots live in a private R2 bucket, pinned by the manifest
# fetched above; this cache keyed on the manifest hash makes most fetches
# a no-op (hashFiles evaluates at step run time, AFTER the fetch; a
# skipped fetch hashes to empty → restore-keys still warms the tree).
# restore-keys makes a manifest change INCREMENTAL (restore the previous
# tree, download only the changed objects) instead of a full ~35MB
# re-download. A stale or partial restore is safe: r2-sync hash-verifies
# every file and deletes unlisted ones, and compare.ts refuses to gate
# against an incomplete cache.
- name: Cache screenshot baselines - name: Cache screenshot baselines
if: inputs.run_tests if: inputs.run_tests
uses: actions/cache@v4 uses: actions/cache@v4
with: with:
path: tests/baseline-screenshots path: tests/baseline-screenshots
key: baselines-${{ hashFiles('tests/screenshot-manifest.json') }} key: baselines-${{ hashFiles('tests/.baseline-manifest.json') }}
restore-keys: | restore-keys: |
baselines- baselines-
@ -349,19 +367,17 @@ jobs:
run: npm run screenshots:fetch run: npm run screenshots:fetch
# Cheap hygiene gates (no build needed): the determinism lint keeps the # Cheap hygiene gates (no build needed): the determinism lint keeps the
# banned flake patterns out of the specs, the manifest check validates # banned flake patterns out of the specs, and the CI-coverage lint proves
# screenshot-manifest.json (v2 schema, sorted/unique hashes) and fails if # every spec file on disk is reachable from the npm scripts THIS workflow
# baseline PNGs are ever re-committed to git (credential-free — it never # invokes (a spec/project that CI never runs is how the web suite once
# touches R2, so it gates secretless callers identically), and # rotted unnoticed). (The old screenshot-manifest --check gate went with
# the CI-coverage lint proves every spec file on disk is reachable from # the committed manifest — the manifest now lives in R2, written only by
# the npm scripts THIS workflow invokes (a spec/project that CI never # the morelli app, which validates on every promote.)
# runs is how the web suite once rotted unnoticed). - name: Lint test determinism + CI coverage
- name: Lint test determinism + screenshot manifest + CI coverage
if: inputs.run_tests if: inputs.run_tests
working-directory: tests working-directory: tests
run: | run: |
npm run lint:determinism npm run lint:determinism
npm run screenshots:manifest -- --check
npm run lint:ci-coverage npm run lint:ci-coverage
- name: Install web workspace deps (collab bundle) - name: Install web workspace deps (collab bundle)

View file

@ -7,8 +7,8 @@ The e2e tests are in /tests, with a README and WHATWORKS md files
Test determinism rules (no blind sleeps/ifs, `stableShot` screenshots, retries:0) are in tests/TESTING.md, enforced by `npm run lint:determinism`. Test determinism rules (no blind sleeps/ifs, `stableShot` screenshots, retries:0) are in tests/TESTING.md, enforced by `npm run lint:determinism`.
The e2e tests are separated per feature The e2e tests are separated per feature
Wxwidgets wasm port has hooks for finding positions of UI elements, tests use that Wxwidgets wasm port has hooks for finding positions of UI elements, tests use that
The test screenshot baselines live in a private R2 bucket (content-addressed by sha256), pinned per engine by the committed tests/screenshot-manifest.json; tests/baseline-screenshots/{chromium,firefox}/ is a gitignored local cache — `cd tests && npm run screenshots:fetch` materializes it (needs the R2 credentials in tests/tools/screenshots/README.md). CI's Linux render is the source of truth (tooling: tests/tools/screenshots/, see its README). The test screenshot baselines live in a private R2 bucket (content-addressed by sha256), pinned per engine by the R2-hosted manifest baselines/pcbjam/manifest.json — NOTHING manifest-related is in git; tests/baseline-screenshots/{chromium,firefox}/ is a gitignored local cache — `cd tests && npm run screenshots:fetch-manifest && npm run screenshots:fetch` materializes it (needs the R2 credentials in tests/tools/screenshots/README.md). CI's Linux render is the source of truth (tooling: tests/tools/screenshots/, see its README).
To update baselines, promote a CI run's render (churn-free — only meaningfully-changed images re-upload): `cd tests && npm run screenshots:promote -- --run <ci-run-id>` (needs the read-write R2 credentials), then commit the manifest diff — never commit PNGs. `npm run screenshots:check` is the local gate (fetch first); on each main push CI posts a screenshot-diff + runtime-perf report to Discord. To update baselines, promote a CI run's screenshots in the morelli review app (https://pcbjam-morelli-staging.pcbjam-staging.workers.dev — pick the run, review the diffs, bulk-select, Promote). CI uploads every run's renders to R2 (runs/pcbjam/<run-id>/, 30-day retention) for that purpose. No git commit is involved. `npm run screenshots:check` is the local gate (fetch-manifest + fetch first); on each main push CI posts a screenshot-diff + runtime-perf report to Discord.
The tests have log files in tests/logs/{wxwidgets/kicad}/{test-name} after each run where the js console and cpp logs are visible The tests have log files in tests/logs/{wxwidgets/kicad}/{test-name} after each run where the js console and cpp logs are visible
Always check screenshots for validating tests Always check screenshots for validating tests
Run e2e tests from /tests folder: `npm run test:e2e` (full CI project set, one merged playwright.config.ts) or `npm run test:kicad` (firefox shortcut) — not playwright directly. One spec/engine: `npx playwright test --project=kicad-firefox kicad/pcbnew.spec.ts`. Web-app suite: `npm run test:web`. Run e2e tests from /tests folder: `npm run test:e2e` (full CI project set, one merged playwright.config.ts) or `npm run test:kicad` (firefox shortcut) — not playwright directly. One spec/engine: `npx playwright test --project=kicad-firefox kicad/pcbnew.spec.ts`. Web-app suite: `npm run test:web`.

View file

@ -195,15 +195,18 @@ See [tests/README.md](tests/README.md) for test documentation.
### Screenshots ### Screenshots
CI's Linux render is the source of truth for baseline screenshots. On each `main` CI's Linux render is the source of truth for baseline screenshots. On each `main`
push, CI compares its render against the committed baselines and posts the diff push, CI compares its render against the baselines (pinned by the R2-hosted
(plus the runtime-perf numbers) to Discord. To update baselines after an intended manifest — nothing screenshot-related is committed) and posts the diff (plus the
render change, promote a CI run's render — only meaningfully-changed images runtime-perf numbers) to Discord. To update baselines after an intended render
restage, so it stays churn-free: change, promote the CI run's screenshots in the morelli review app
(https://pcbjam-morelli-staging.pcbjam-staging.workers.dev) — pick the run,
review the diffs, bulk-select, Promote. CI uploads every run's renders to R2
for that purpose (30-day retention).
```bash ```bash
cd tests cd tests
npm run screenshots:check # local gate: current vs baselines npm run screenshots:fetch-manifest && npm run screenshots:fetch # materialize baselines
npm run screenshots:promote -- --run <ci-run-id> # adopt a CI run's render, then commit npm run screenshots:check # local gate: current vs baselines
``` ```
See [tests/tools/screenshots/README.md](tests/tools/screenshots/README.md). See [tests/tools/screenshots/README.md](tests/tools/screenshots/README.md).

9
tests/.gitignore vendored
View file

@ -1,8 +1,11 @@
# Screenshot model: specs capture via stableShot() → page.screenshot into test-results/, which is # Screenshot model: specs capture via stableShot() → page.screenshot into test-results/, which is
# transient and gitignored at the repo root (/tests/test-results/). Baselines live in a private R2 # transient and gitignored at the repo root (/tests/test-results/). Baselines live in a private R2
# bucket, content-addressed by sha256 and pinned by the committed screenshot-manifest.json; # bucket, content-addressed by sha256 and pinned by the R2-hosted manifest (fetched to the
# tests/baseline-screenshots/ is a gitignored local cache (`npm run screenshots:fetch`), authored # gitignored .baseline-manifest.json below — NOTHING manifest-related is committed);
# from CI's deterministic Linux render and diffed OFFLINE by tools/screenshots. (3d-regression/ and # tests/baseline-screenshots/ is a gitignored local cache (`npm run screenshots:fetch-manifest &&
# npm run screenshots:fetch`), authored from CI's deterministic Linux render and diffed OFFLINE by
# tools/screenshots. Baselines are promoted from a CI run in the morelli app. (3d-regression/ and
# gal-regression/ baselines are still committed.) There are no Playwright-native *-snapshots/ # gal-regression/ baselines are still committed.) There are no Playwright-native *-snapshots/
# baselines — Playwright does no inline screenshot comparison. # baselines — Playwright does no inline screenshot comparison.
.baseline-manifest.json
logs/ logs/

View file

@ -55,7 +55,7 @@ tests/
├── logs/ # Test logs (auto-generated) ├── logs/ # Test logs (auto-generated)
├── test-results/{chromium,firefox}/ # Screenshots per engine (auto-generated) ├── test-results/{chromium,firefox}/ # Screenshots per engine (auto-generated)
├── baseline-screenshots/{chromium,firefox}/ # Reference screenshots per engine — gitignored cache of the R2 bucket (npm run screenshots:fetch) ├── baseline-screenshots/{chromium,firefox}/ # Reference screenshots per engine — gitignored cache of the R2 bucket (npm run screenshots:fetch)
├── screenshot-manifest.json # Committed pin of every baseline: {name, engine, sha256, …} (regenerated by screenshots:promote) ├── .baseline-manifest.json # Gitignored copy of the R2-hosted baseline manifest (npm run screenshots:fetch-manifest)
├── apps/ # Built WASM test applications ├── apps/ # Built WASM test applications
│ ├── minimal_test.html # Main test app │ ├── minimal_test.html # Main test app
│ └── standalone/ # Individual component test apps │ └── standalone/ # Individual component test apps
@ -83,13 +83,15 @@ Example log format:
## Screenshots ## Screenshots
Tests capture raw PNGs to `test-results/<engine>/` (engine-scoped via Tests capture raw PNGs to `test-results/<engine>/` (engine-scoped via
`stableShot`/`shotPath`). The offline gate compares them against the committed `stableShot`/`shotPath`). The offline gate compares them against the per-engine
per-engine baselines: baselines pinned by the R2-hosted manifest:
```bash ```bash
npm run screenshots:fetch-manifest && npm run screenshots:fetch
npm run screenshots:check npm run screenshots:check
``` ```
CI's Linux render is the source of truth — update baselines by promoting a CI CI's Linux render is the source of truth — update baselines by promoting a CI
run (`npm run screenshots:promote -- --run <ci-run-id>`), never by copying run in the morelli review app
(https://pcbjam-morelli-staging.pcbjam-staging.workers.dev), never by copying
local renders. Rules and details: `TESTING.md`. local renders. Rules and details: `TESTING.md`.
## Viewing the App Directly ## Viewing the App Directly

View file

@ -46,15 +46,18 @@ only) — not playwright directly. One spec on one engine:
`tests/baseline-screenshots/<engine>/` (+ the still-committed `3d-regression/`, `tests/baseline-screenshots/<engine>/` (+ the still-committed `3d-regression/`,
`gal-regression/`). `gal-regression/`).
- **Baselines live in the private R2 bucket `pcbjam-ci-screenshots`, not git**: - **Baselines live in the private R2 bucket `pcbjam-ci-screenshots`, not git**:
the committed `tests/screenshot-manifest.json` pins each `{name, engine}` to a the R2-hosted manifest `baselines/pcbjam/manifest.json` pins each
sha256, and `baseline-screenshots/` is a gitignored cache — materialize it with `{name, engine}` to a sha256, and `baseline-screenshots/` +
`npm run screenshots:fetch` (needs the R2 credentials; see `.baseline-manifest.json` are gitignored caches — materialize them with
`tools/screenshots/README.md`). CI's lint step validates the manifest and fails `npm run screenshots:fetch-manifest && npm run screenshots:fetch` (needs the
if baseline PNGs are ever committed again. R2 credentials; see `tools/screenshots/README.md`). Nothing
- **CI's Linux render is the source of truth**; baselines are promoted from CI screenshot-related is committed to git.
(`npm run screenshots:promote -- --run <ci-run-id>`, needs the read-write - **CI's Linux render is the source of truth**; baselines are promoted from a
credentials) — commit only the regenerated manifest diff, never PNGs. A local CI run in the morelli review app
(Mac) check shows font/render noise and is not the gate. (https://pcbjam-morelli-staging.pcbjam-staging.workers.dev) — CI uploads each
run's renders to R2 (30-day retention), morelli shows the diffs and writes
the manifest on Promote. A local (Mac) check shows font/render noise and is
not the gate.
- A continuously-animating state (timer, mid-slide) can't be a stable baseline — drop the shot. - A continuously-animating state (timer, mid-slide) can't be a stable baseline — drop the shot.
## Retries ## Retries

View file

@ -405,7 +405,7 @@ tests/
├── test-results/ ├── test-results/
│ ├── chromium/ # Latest run's captures per engine │ ├── chromium/ # Latest run's captures per engine
│ └── firefox/ │ └── firefox/
└── screenshot-manifest.json # Committed pin: {name, engine, sha256, …} per baseline └── .baseline-manifest.json # GITIGNORED copy of the R2-hosted manifest (npm run screenshots:fetch-manifest)
``` ```
Specs write via `stableShot(page, 'name.png')` / `shotPath(page, 'name.png')` Specs write via `stableShot(page, 'name.png')` / `shotPath(page, 'name.png')`
@ -416,7 +416,8 @@ browser, so the same spec on two engines produces two independent captures.
```bash ```bash
cd tests cd tests
npm run screenshots:fetch # materialize the baseline cache from R2 (idempotent) npm run screenshots:fetch-manifest # download the R2-hosted baseline manifest
npm run screenshots:fetch # materialize the baseline cache from R2 (idempotent)
npm run screenshots:check npm run screenshots:check
``` ```
@ -428,14 +429,13 @@ detection is driven by the manifest). On each main push, CI posts the same repor
### Updating Baseline Screenshots ### Updating Baseline Screenshots
CI's Linux render is the source of truth — never copy local (Mac) renders into CI's Linux render is the source of truth — never copy local (Mac) renders into
the baselines. Promote from a CI run instead (churn-free: only meaningfully the baselines. Promote from a CI run in the morelli review app instead
changed images restage, and the manifest regenerates automatically): (churn-free: identical images keep their provenance, and the R2-hosted manifest
updates atomically — no git commit involved):
```bash 1. Open https://pcbjam-morelli-staging.pcbjam-staging.workers.dev (GitHub sign-in).
cd tests 2. Pipeline `pcbjam` → pick the run → review the side-by-side/pixel diffs.
npm run screenshots:promote -- --run <ci-run-id> # needs the read-write R2 keypair (tests/.env) 3. Select the intended changes (or "Select all changed + added") → Promote.
git commit # only screenshot-manifest.json changes — PNGs are uploaded to R2, never committed
```
### Running Tests with Screenshots ### Running Tests with Screenshots

View file

@ -18,14 +18,11 @@
"lint:determinism": "tsx tools/lint-determinism.ts", "lint:determinism": "tsx tools/lint-determinism.ts",
"lint:ci-coverage": "tsx tools/lint-ci-coverage.ts", "lint:ci-coverage": "tsx tools/lint-ci-coverage.ts",
"screenshots:check": "tsx tools/screenshots/compare.ts", "screenshots:check": "tsx tools/screenshots/compare.ts",
"screenshots:promote": "tsx tools/screenshots/promote.ts",
"screenshots:noise": "tsx tools/screenshots/noise.ts", "screenshots:noise": "tsx tools/screenshots/noise.ts",
"screenshots:report": "tsx tools/screenshots/post-discord.ts", "screenshots:report": "tsx tools/screenshots/post-discord.ts",
"screenshots:changelog": "tsx tools/screenshots/changelog.ts", "screenshots:fetch-manifest": "tsx tools/screenshots/r2-sync.ts --manifest",
"screenshots:manifest": "tsx tools/screenshots/gen-manifest.ts",
"screenshots:fetch": "tsx tools/screenshots/r2-sync.ts --pull", "screenshots:fetch": "tsx tools/screenshots/r2-sync.ts --pull",
"screenshots:upload-run": "tsx tools/screenshots/upload-run.ts", "screenshots:upload-run": "tsx tools/screenshots/upload-run.ts",
"screenshots:push": "tsx tools/screenshots/r2-sync.ts --push",
"3d:compare": "tsx tools/screenshots/compare-dirs.ts", "3d:compare": "tsx tools/screenshots/compare-dirs.ts",
"3d:check": "tsx tools/screenshots/compare-dirs.ts --old 3d-regression/baseline --new 3d-regression/output/native --out 3d-regression/output/diff/native-self --floors 3d-regression/floors.json --level native-self --label 3d-native --fail-on-change", "3d:check": "tsx tools/screenshots/compare-dirs.ts --old 3d-regression/baseline --new 3d-regression/output/native --out 3d-regression/output/diff/native-self --floors 3d-regression/floors.json --level native-self --label 3d-native --fail-on-change",
"3d:check:webgl": "tsx tools/screenshots/compare-dirs.ts --old 3d-regression/baseline-webgl --new 3d-regression/output/webgl --out 3d-regression/output/diff/webgl-self --floors 3d-regression/floors.json --level webgl-self --label 3d-webgl --fail-on-change", "3d:check:webgl": "tsx tools/screenshots/compare-dirs.ts --old 3d-regression/baseline-webgl --new 3d-regression/output/webgl --out 3d-regression/output/diff/webgl-self --floors 3d-regression/floors.json --level webgl-self --label 3d-webgl --fail-on-change",

File diff suppressed because it is too large Load diff

View file

@ -1,39 +1,45 @@
# Screenshot regression + Discord review tooling # Screenshot regression + Discord review tooling
One comparison engine + a churn-free updater + a Discord reporter for the e2e One comparison engine + a Discord reporter for the e2e screenshots; baseline
screenshots. Design and rationale: `~/.claude/plans/…snowglobe.md` (or ask). updates happen in the morelli review app
(https://pcbjam-morelli-staging.pcbjam-staging.workers.dev, repo
github.com/PCBJam/morelli).
**Source of truth = CI's Linux render.** The dev never authors baselines on the **Source of truth = CI's Linux render.** The dev never authors baselines on the
Mac (Mac fonts/GL ≠ CI). Instead, CI renders on every push; when a render change Mac (Mac fonts/GL ≠ CI). CI renders on every push and uploads each run's
is intentional you *promote* CI's artifact into the baselines. The environment screenshots to R2 (`runs/pcbjam/<run-id>/`, 30-day retention, upload-run.ts);
isn't pinned — if the host's Mesa/fonts drift, the gate lights up in Discord and when a render change is intentional you *promote* that run's screenshots in
you just re-promote (broad + low-intensity change ⇒ likely drift). morelli. The environment isn't pinned — if the host's Mesa/fonts drift, the
gate lights up in Discord and you just re-promote (broad + low-intensity
change ⇒ likely drift).
**Baselines live in R2, not git.** The PNGs sit in a private Cloudflare R2 **Baselines live entirely in R2, not git.** The PNGs sit in a private
bucket (`pcbjam-ci-screenshots`), content-addressed as `sha256/<hex>.png` and Cloudflare R2 bucket (`pcbjam-ci-screenshots`), content-addressed as
immutable; the committed `screenshot-manifest.json` pins each `<engine>/<name>` `sha256/<hex>.png` and immutable; the R2-HOSTED manifest
to a hash, so every git commit resolves its exact baselines. The local `baselines/pcbjam/manifest.json` (written only by morelli + its seed script)
`baseline-screenshots/` tree is a gitignored cache — `npm run screenshots:fetch` pins each `<engine>/<name>` to a hash. The local `baseline-screenshots/` tree
materializes it; `promote` uploads new hashes and rewrites the manifest, and the and `.baseline-manifest.json` are gitignored caches —
manifest diff is the only thing you commit. `npm run screenshots:fetch-manifest && npm run screenshots:fetch` materializes
them. Nothing screenshot-related is committed.
**Credentials** (S3 API, bucket-scoped, region `auto`): **Credentials** (S3 API, bucket-scoped, region `auto`):
``` ```
CI_SCREENSHOTS_S3_ENDPOINT # https://<account-id>.r2.cloudflarestorage.com CI_SCREENSHOTS_S3_ENDPOINT # https://<account-id>.r2.cloudflarestorage.com
CI_SCREENSHOTS_S3_BUCKET # optional, default pcbjam-ci-screenshots CI_SCREENSHOTS_S3_BUCKET # optional, default pcbjam-ci-screenshots
CI_SCREENSHOTS_S3_ACCESS_KEY_ID # read-only pair in CI; read-write pair for promote CI_SCREENSHOTS_S3_ACCESS_KEY_ID # read-only pair for fetch; CI's upload step maps in a write pair
CI_SCREENSHOTS_S3_SECRET_ACCESS_KEY CI_SCREENSHOTS_S3_SECRET_ACCESS_KEY
``` ```
CI holds the read-only pair as repo secrets; devs get the read-write pair from CI holds the read-only pair as repo secrets (plus `CI_SCREENSHOTS_S3_WRITE_*`
the team vault (ask) and put it in `tests/.env` (gitignored, auto-loaded by for the run-upload step); devs get a read pair from the team vault (ask) and
r2-store.ts; shell env vars take precedence): put it in `tests/.env` (gitignored, auto-loaded by r2-store.ts; shell env vars
take precedence):
``` ```
CI_SCREENSHOTS_S3_ENDPOINT=https://<account-id>.r2.cloudflarestorage.com CI_SCREENSHOTS_S3_ENDPOINT=https://<account-id>.r2.cloudflarestorage.com
CI_SCREENSHOTS_S3_ACCESS_KEY_ID=<rw-access-key-id> CI_SCREENSHOTS_S3_ACCESS_KEY_ID=<access-key-id>
CI_SCREENSHOTS_S3_SECRET_ACCESS_KEY=<rw-secret> CI_SCREENSHOTS_S3_SECRET_ACCESS_KEY=<secret>
``` ```
Without credentials, fetch and the compare gate skip with a warning (secretless Without credentials, the manifest/baseline fetch and the compare gate skip with
CI callers stay green); promote refuses to run. a warning (secretless CI callers stay green).
**Everything is per-engine.** Specs write `test-results/<engine>/<name>.png` **Everything is per-engine.** Specs write `test-results/<engine>/<name>.png`
(engine derived from the running browser via `stableShot`/`shotPath`); baselines (engine derived from the running browser via `stableShot`/`shotPath`); baselines
@ -45,27 +51,29 @@ The same spec on chromium + firefox is two independent gated screenshots.
- `config.ts` — thresholds, baseline dirs, per-engine floors (calibrate!), clustering knobs. - `config.ts` — thresholds, baseline dirs, per-engine floors (calibrate!), clustering knobs.
- `image-ops.ts` — PNG load/save, pixelmatch diff (AA-excluded), connected-component boxes, triptych compositing, size-cap resize. - `image-ops.ts` — PNG load/save, pixelmatch diff (AA-excluded), connected-component boxes, triptych compositing, size-cap resize.
- `compare.ts` — the comparison engine: classify per-engine baselines vs `test-results/<engine>/` → `test-results/screenshot-diff/report.json` + triptych/heatmap PNGs. `--pair` diffs two files. - `compare.ts` — the comparison engine: classify per-engine baselines vs `test-results/<engine>/` → `test-results/screenshot-diff/report.json` + triptych/heatmap PNGs. `--pair` diffs two files.
- `promote.ts` — churn-free updater: pull a CI run's shots (`gh run download`) or `--from DIR`; overwrite a baseline only when pixels differ beyond the floor (verbatim bytes, no re-encode) → no git churn.
- `perf-report.ts` — renders the track-only runtime-perf table (loadMs/openMs/FPS) with Δ vs the previous main run (fetched via `gh`). - `perf-report.ts` — renders the track-only runtime-perf table (loadMs/openMs/FPS) with Δ vs the previous main run (fetched via `gh`).
- `post-discord.ts` — the always-on CI-on-main report: SHA + e2e status + perf table, then screenshot triptychs (batched, size-capped, flood-collapsed). - `post-discord.ts` — the always-on CI-on-main report: SHA + e2e status + perf table, then screenshot triptychs (batched, size-capped, flood-collapsed).
- `changelog.ts` — Discord trigger B: git-history diff of the manifest between two revs, PNG bytes fetched from R2 (no build/GPU).
- `noise.ts` — calibration: diff two identical-input renders → per-engine noise floor. - `noise.ts` — calibration: diff two identical-input renders → per-engine noise floor.
- `gen-manifest.ts` — regenerate `screenshot-manifest.json` ({name, engine, sha256, bytes, width, height}) from the local baseline cache; `--check` (gating in CI, credential-free) validates the schema and fails if baseline PNGs are ever re-committed to git.
- `r2-store.ts` — minimal aws4fetch S3 client for the CAS bucket (get/put/exists by hash, downloads integrity-checked). - `r2-store.ts` — minimal aws4fetch S3 client for the CAS bucket (get/put/exists by hash, downloads integrity-checked).
- `r2-sync.ts` — cache sync: `--pull` materializes `baseline-screenshots/` from the manifest (idempotent, deletes unlisted files), `--push` seeds/uploads, `--verify` HEADs every hash. - `r2-sync.ts` — cache sync: `--manifest` downloads the R2-hosted baseline manifest, `--pull` materializes `baseline-screenshots/` from it (idempotent, deletes unlisted files), `--verify` HEADs every hash.
- `upload-run.ts` — CI-only: upload the run's renders + meta.json to `runs/pcbjam/<run-id>/` for morelli (needs the write pair; no-ops without credentials).
- `spec-map.ts` — best-effort screenshot-name → spec-file attribution for captions (scans `stableShot`/`shotPath` literals). - `spec-map.ts` — best-effort screenshot-name → spec-file attribution for captions (scans `stableShot`/`shotPath` literals).
## npm scripts (run from `tests/`) ## npm scripts (run from `tests/`)
``` ```
npm run screenshots:fetch # materialize the baseline cache from R2 (run before check; needs read creds) npm run screenshots:fetch-manifest # download the R2-hosted baseline manifest (needs read creds)
npm run screenshots:check # gate: baselines vs test-results → report.json (exit 0; add --fail-on-change to gate) npm run screenshots:fetch # materialize the baseline cache from R2 (run before check)
npm run screenshots:promote -- --run <ci-run-id> # churn-free re-baseline from a CI run (or --from DIR; needs RW creds) npm run screenshots:check # gate: baselines vs test-results → report.json (exit 0; add --fail-on-change to gate)
npm run screenshots:report -- --e2e pass # post the CI report to Discord (main+push only; needs DISCORD_WEBHOOK_URL) npm run screenshots:report -- --e2e pass # post the CI report to Discord (main+push only; needs DISCORD_WEBHOOK_URL)
npm run screenshots:changelog # post the baseline changelog (main+push only) npm run screenshots:noise -- run1/ run2/ # calibrate floors
npm run screenshots:noise -- run1/ run2/ # calibrate floors npm run screenshots:upload-run -- --e2e pass # CI-only: upload the run's renders for morelli
npm run screenshots:manifest # regenerate the manifest (--check to verify it's fresh)
``` ```
Baseline promotion (single or bulk) happens in morelli — pick the run, review
the diffs, Promote. It copies verbatim bytes into the CAS, updates the R2
manifest atomically (with provenance: which run/branch/user), and snapshots the
previous manifest for revert.
## Activation checklist ## Activation checklist
- [x] `screenshot-manifest.json` generated ({name, engine} authoritative — derived from the per-engine baseline tree). - [x] `screenshot-manifest.json` generated ({name, engine} authoritative — derived from the per-engine baseline tree).
- [x] `scale:'device'`→`'css'` normalized (no-op at CI's DSF=1). - [x] `scale:'device'`→`'css'` normalized (no-op at CI's DSF=1).

View file

@ -1,212 +0,0 @@
/**
* Baseline changelog (Discord trigger B) — no build, no GPU.
*
* On push to main, diff the committed screenshot MANIFEST between two commits
* and post a "Baseline changelog" to Discord: a triptych (old | new+boxes |
* heatmap) for each CHANGED baseline, the image for each ADDED, and a titled
* line for each REMOVED. "added / removed" only mean anything as a git-history
* diff, which is why this is separate from the re-render drift gate.
*
* The PNGs themselves are not in git — each manifest entry pins a sha256 and
* the bytes are fetched from the R2 CAS bucket (objects are immutable, so the
* base rev's hashes are always still resolvable). Needs the read credential
* pair; a base rev whose manifest predates the R2 migration (v1) is skipped so
* the migration commit itself doesn't post hundreds of bogus entries.
*
* CLI (from tests/):
* tsx tools/screenshots/changelog.ts [--base REV] [--head REV] [--dry-run] [--force]
*/
import * as fs from 'fs';
import * as path from 'path';
import { execFileSync } from 'child_process';
import { PNG } from 'pngjs';
import { DIFF_OUT_DIR, MANIFEST_PATH, MANIFEST_VERSION, floorFor, labelText, splitKey, LABEL, type Manifest } from './config';
import { comparePair, type Report } from './compare';
import { savePng, withBottomLabel } from './image-ops';
import { buildSpecResolver } from './spec-map';
import { buildAttachments, paginate, postMessage } from './post-discord';
import { missingEnv, storeFromEnv, type R2Store } from './r2-store';
function git(root: string, args: string[]): string {
return execFileSync('git', ['-C', root, ...args], { encoding: 'utf8' }).trim();
}
/** The screenshot manifest as committed at `rev`. Returns null ONLY when the
* file doesn't exist at that rev (pre-manifest era); any other git failure —
* shallow clone, bad revision, moved path — throws loudly instead of being
* mistaken for "predates the migration". Unparsable committed JSON also throws. */
function manifestAt(repoRoot: string, rev: string, repoPath: string): Manifest | null {
let raw: string;
try {
raw = execFileSync('git', ['-C', repoRoot, 'show', `${rev}:${repoPath}`], { encoding: 'utf8' });
} catch (e) {
const msg = ((e as { stderr?: Buffer }).stderr?.toString() ?? (e as Error).message).trim();
if (/does not exist in|exists on disk, but not in/i.test(msg)) return null;
throw new Error(`git show ${rev}:${repoPath} failed: ${msg}`);
}
return JSON.parse(raw) as Manifest;
}
function parseArgs(argv: string[]): Record<string, string | boolean> {
const out: Record<string, string | boolean> = {};
for (let i = 0; i < argv.length; i++) {
const a = argv[i];
if (a === '--dry-run') out.dryRun = true;
else if (a === '--force') out.force = true;
else if (a === '--base') out.base = argv[++i];
else if (a === '--head') out.head = argv[++i];
}
return out;
}
async function main(): Promise<void> {
const args = parseArgs(process.argv.slice(2));
const cwd = process.cwd();
const repoRoot = git(cwd, ['rev-parse', '--show-toplevel']);
const head = (args.head as string) || 'HEAD';
let base = (args.base as string) || '';
if (!base) {
try {
base = git(cwd, ['rev-parse', `${head}^`]);
} catch {
console.log('[changelog] no parent commit — nothing to diff');
return;
}
}
// Repo-relative manifest path (e.g. tests/screenshot-manifest.json).
const manifestPath = path.relative(repoRoot, path.join(cwd, MANIFEST_PATH)).split(path.sep).join('/');
const baseManifest = manifestAt(repoRoot, base, manifestPath);
const headManifest = manifestAt(repoRoot, head, manifestPath);
// A NEWER manifest than this tooling understands is a loud error, never a skip.
for (const [label, m] of [['head', headManifest], ['base', baseManifest]] as const) {
if (m && m.version > MANIFEST_VERSION) {
throw new Error(`${label} manifest is version ${m.version}, newer than this tooling (expects ${MANIFEST_VERSION}) — update the checkout`);
}
}
if (headManifest?.version !== MANIFEST_VERSION) {
console.log('[changelog] head manifest predates the R2 migration — nothing to diff');
return;
}
if (baseManifest?.version !== MANIFEST_VERSION) {
console.log('[changelog] base manifest predates the R2 migration — skipping (migration commit)');
return;
}
// Engine-qualified key (`<engine>/<name>`) → sha256, per rev.
const hashesOf = (m: Manifest): Map<string, string> =>
new Map(m.screenshots.map((e) => [`${e.engine}/${e.name}`, e.sha256]));
const baseHashes = hashesOf(baseManifest);
const headHashes = hashesOf(headManifest);
const added = [...headHashes.keys()].filter((k) => !baseHashes.has(k));
const removed = [...baseHashes.keys()].filter((k) => !headHashes.has(k));
const changed = [...headHashes.keys()].filter((k) => baseHashes.has(k) && baseHashes.get(k) !== headHashes.get(k));
if (!added.length && !removed.length && !changed.length) {
console.log('[changelog] no baseline changes between', base.slice(0, 7), 'and', head);
return;
}
// Tolerate missing credentials (warn + exit 0): this is a notification-only
// workflow — turning every manifest push red during a key rotation is worse
// than one missed changelog post. The warning names the fix.
const store = storeFromEnv();
if (!store) {
console.warn(`[changelog] R2 credentials unset (${missingEnv().join(', ')}) — cannot fetch baseline bytes; skipping the changelog post`);
return;
}
const outDir = path.join(cwd, DIFF_OUT_DIR);
fs.mkdirSync(outDir, { recursive: true });
const report: Report = {
generatedFor: process.env.GITHUB_SHA || head,
changed: [],
added: [],
removed: [],
unchangedCount: 0,
driftLikely: false,
};
const { specFor } = buildSpecResolver(cwd);
// One unresolvable object must not abort the whole post — skip that image
// (with a loud note) and keep reporting the rest, like the old git-blob
// path null-skipped unresolvable blobs.
const skipped: string[] = [];
const hashPng = async (s: R2Store, hash: string, key: string): Promise<PNG | null> => {
try {
return PNG.sync.read(await s.get(hash));
} catch (e) {
skipped.push(`${key}: ${(e as Error).message}`);
return null;
}
};
// Save a single captioned image (added/removed) and record it in the report.
const saveSingle = (img: PNG, key: string, status: 'added' | 'removed'): void => {
const rel = path.join(DIFF_OUT_DIR, `${key.replace('/', '_')}.${status}.png`);
savePng(path.join(cwd, rel), withBottomLabel(img, labelText(status, key, specFor(splitKey(key).name)), LABEL.colors[status]));
report[status].push({ name: key, image: rel });
};
for (const key of added) {
const img = await hashPng(store, headHashes.get(key)!, key);
if (img) saveSingle(img, key, 'added');
}
for (const key of removed) {
const img = await hashPng(store, baseHashes.get(key)!, key);
if (img) saveSingle(img, key, 'removed');
}
for (const key of changed) {
const oldImg = await hashPng(store, baseHashes.get(key)!, key);
const newImg = await hashPng(store, headHashes.get(key)!, key);
if (!oldImg || !newImg) continue;
const { result, heatmap, triptych } = comparePair(oldImg, newImg, key, floorFor(key));
const triptychRel = path.join(DIFF_OUT_DIR, `${key.replace('/', '_')}.triptych.png`);
const heatmapRel = path.join(DIFF_OUT_DIR, `${key.replace('/', '_')}.heatmap.png`);
savePng(path.join(cwd, triptychRel), withBottomLabel(triptych, labelText('changed', key, specFor(splitKey(key).name)), LABEL.colors.changed));
savePng(path.join(cwd, heatmapRel), heatmap);
report.changed.push({ ...result, triptych: triptychRel, heatmap: heatmapRel });
}
report.changed.sort((a, b) => b.changedRatio - a.changedRatio);
if (skipped.length) {
console.warn(`[changelog] skipped ${skipped.length} unresolvable image(s):\n ${skipped.join('\n ')}`);
}
const sha7 = (process.env.GITHUB_SHA || head).slice(0, 7);
let subject = '';
try {
subject = git(cwd, ['log', '-1', '--pretty=%s', head]);
} catch { /* ignore */ }
const header =
`🗂️ **Baseline changelog** · \`${sha7}\`` +
(subject ? `\n> ${subject}` : '') +
`\n${report.changed.length} changed, ${report.added.length} added, ${report.removed.length} removed` +
(skipped.length ? `\n⚠️ ${skipped.length} image(s) unresolvable in R2 — see the workflow log` : '') +
(report.removed.length ? '\n➖ REMOVED: ' + report.removed.map((r) => `\`${r.name}\``).join(', ') : '');
const { files, notes } = buildAttachments(cwd, report);
const messages = paginate(header + (notes.length ? '\n' + notes.join('\n') : ''), files);
const isMainPush = process.env.GITHUB_REF === 'refs/heads/main' && process.env.GITHUB_EVENT_NAME === 'push';
if (args.dryRun || (!args.force && !isMainPush)) {
for (const [i, m] of messages.entries()) {
console.log(`--- message ${i + 1}/${messages.length} (${m.files.length} files) ---`);
if (m.content) console.log(m.content);
for (const f of m.files) console.log(` [attach] ${f.name} (${f.buffer.length} bytes)`);
}
if (!args.dryRun) console.log('[changelog] not a push to main — not posting');
return;
}
const webhook = process.env.DISCORD_WEBHOOK_URL;
if (!webhook) {
console.log('[changelog] DISCORD_WEBHOOK_URL unset — skipping');
return;
}
for (const m of messages) await postMessage(webhook, m);
console.log(`[changelog] posted ${messages.length} message(s)`);
}
if (require.main === module) {
main().catch((e) => {
console.error(`[changelog] ${e.message}`);
process.exitCode = 1;
});
}

View file

@ -9,8 +9,8 @@
* The gate mode classifies every screenshot into changed / added / removed / * The gate mode classifies every screenshot into changed / added / removed /
* unchanged, writes per-change triptych + heatmap PNGs and a machine-readable * unchanged, writes per-change triptych + heatmap PNGs and a machine-readable
* report.json into DIFF_OUT_DIR, and (unless --fail-on-change) exits 0 so it can * report.json into DIFF_OUT_DIR, and (unless --fail-on-change) exits 0 so it can
* run report-only first. post-discord.ts and the changelog workflow import the * run report-only first. post-discord.ts imports the exported helpers rather
* exported helpers rather than re-deriving the diff. * than re-deriving the diff.
*/ */
import * as fs from 'fs'; import * as fs from 'fs';
import * as path from 'path'; import * as path from 'path';
@ -236,7 +236,14 @@ function main(): void {
// dev who hasn't fetched. Either way skip without writing report.json — // dev who hasn't fetched. Either way skip without writing report.json —
// post-discord posts a distinct "gate SKIPPED" line when it's absent. // post-discord posts a distinct "gate SKIPPED" line when it's absent.
const gateManifest = loadManifest(root); const gateManifest = loadManifest(root);
const wanted = (gateManifest?.screenshots ?? []).filter((e) => !isIgnored(`${e.engine}/${e.name}`)); // No fetched manifest at all (creds absent, R2 down, or morelli never
// seeded) ⇒ the gate has no source of truth — skip rather than diff
// against whatever stale cache a previous run left behind.
if (!gateManifest) {
console.log(`[compare] no fetched ${MANIFEST_PATH} — run \`npm run screenshots:fetch-manifest\` (needs R2 credentials); skipping`);
return;
}
const wanted = gateManifest.screenshots.filter((e) => !isIgnored(`${e.engine}/${e.name}`));
const missing = wanted.filter((e) => !fs.existsSync(path.join(root, BASELINE_ROOT, e.engine, e.name))); const missing = wanted.filter((e) => !fs.existsSync(path.join(root, BASELINE_ROOT, e.engine, e.name)));
if (wanted.length && missing.length) { if (wanted.length && missing.length) {
const what = missing.length === wanted.length ? 'empty' : `INCOMPLETE (${missing.length}/${wanted.length} missing — partial fetch?)`; const what = missing.length === wanted.length ? 'empty' : `INCOMPLETE (${missing.length}/${wanted.length} missing — partial fetch?)`;

View file

@ -20,16 +20,18 @@ import * as nodePath from 'path';
* other's PNG, and the surviving file was whichever parallel worker wrote last. * other's PNG, and the surviving file was whichever parallel worker wrote last.
* *
* The baseline PNGs are NOT committed: they live in a private R2 bucket, * The baseline PNGs are NOT committed: they live in a private R2 bucket,
* content-addressed by sha256, and the committed manifest * content-addressed by sha256, pinned by the R2-HOSTED manifest
* (screenshot-manifest.json) is what pins each `<engine>/<name>` to a hash. * (baselines/pcbjam/manifest.json — nothing manifest-related is in git).
* `npm run screenshots:fetch` (tools/screenshots/r2-sync.ts) materializes this * `npm run screenshots:fetch-manifest && npm run screenshots:fetch`
* tree from the manifest; promote.ts uploads new hashes and rewrites the * (tools/screenshots/r2-sync.ts) materializes this tree. Baselines are
* manifest — the manifest diff is the only thing that lands in git. * promoted from a CI run in the morelli review app
* (https://pcbjam-morelli-staging.pcbjam-staging.workers.dev) — the old
* promote.ts/git-manifest flow is retired.
*/ */
export const BASELINE_ROOT = 'baseline-screenshots'; export const BASELINE_ROOT = 'baseline-screenshots';
/** Manifest format version — bumped when baselines moved from git to R2. */ /** Manifest format version — v3 = the R2-hosted manifest written by morelli (v2 was the committed-in-git era). */
export const MANIFEST_VERSION = 2; export const MANIFEST_VERSION = 3;
/** Default private R2 bucket holding the content-addressed baselines. */ /** Default private R2 bucket holding the content-addressed baselines. */
export const R2_DEFAULT_BUCKET = 'pcbjam-ci-screenshots'; export const R2_DEFAULT_BUCKET = 'pcbjam-ci-screenshots';
@ -37,6 +39,9 @@ export const R2_DEFAULT_BUCKET = 'pcbjam-ci-screenshots';
/** Key prefix for content-addressed objects: `sha256/<64-hex>.png`. */ /** Key prefix for content-addressed objects: `sha256/<64-hex>.png`. */
export const R2_KEY_PREFIX = 'sha256/'; export const R2_KEY_PREFIX = 'sha256/';
/** The R2-hosted baseline manifest — THE source of truth. Written only by morelli (promote) and its seed script. */
export const R2_BASELINES_MANIFEST_KEY = 'baselines/pcbjam/manifest.json';
/** /**
* Env vars for the bucket's S3 API (read-only keypair in CI, read-write for * Env vars for the bucket's S3 API (read-only keypair in CI, read-write for
* devs running promote). Endpoint form: https://<account-id>.r2.cloudflarestorage.com * devs running promote). Endpoint form: https://<account-id>.r2.cloudflarestorage.com
@ -80,8 +85,14 @@ export const RESULTS_DIR = 'test-results';
/** Where compare.ts writes diff/heatmap/triptych artifacts (gitignored). */ /** Where compare.ts writes diff/heatmap/triptych artifacts (gitignored). */
export const DIFF_OUT_DIR = 'test-results/screenshot-diff'; export const DIFF_OUT_DIR = 'test-results/screenshot-diff';
/** The manifest that records every expected screenshot + which engine renders it. */ /**
export const MANIFEST_PATH = 'screenshot-manifest.json'; * Local, GITIGNORED copy of the R2-hosted baseline manifest, downloaded by
* `npm run screenshots:fetch-manifest` (r2-sync --manifest). Everything
* downstream (fetch, compare, verify) reads this file; its absence means "no
* manifest fetched" and the screenshot gate skips. Deliberately NOT inside
* test-results/ — Playwright wipes that dir at the start of every invocation.
*/
export const MANIFEST_PATH = '.baseline-manifest.json';
/** /**
* pixelmatch per-pixel settings. * pixelmatch per-pixel settings.
@ -185,10 +196,22 @@ export function labelText(status: LabelStatus, key: string, spec: string | null)
* `height` are sanity metadata (cheap pre-hash check on fetch, dimension info * `height` are sanity metadata (cheap pre-hash check on fetch, dimension info
* in reviews). * in reviews).
*/ */
export type ManifestEntry = { name: string; engine: string; sha256: string; bytes: number; width: number; height: number }; export type ManifestEntry = {
name: string;
engine: string;
sha256: string;
bytes: number;
width: number;
height: number;
/** morelli-side provenance (which run/branch/user promoted this) — opaque to the CI tooling. */
source?: unknown;
};
export type Manifest = { export type Manifest = {
version: number; version: number;
pipeline?: string;
storage: { bucket: string; keyPrefix: string }; storage: { bucket: string; keyPrefix: string };
updatedAt?: string;
updatedBy?: string;
screenshots: ManifestEntry[]; screenshots: ManifestEntry[];
}; };

View file

@ -1,191 +0,0 @@
/**
* Generate tests/screenshot-manifest.json — the canonical list of expected
* screenshots AND the pointer to their bytes. Each entry is
* {name, engine, sha256, bytes, width, height}: the engine comes from the
* baseline cache tree (baseline-screenshots/<engine>/<name>.png), the sha256
* resolves the PNG in the private R2 bucket (`sha256/<hex>.png`). The manifest
* is the SOURCE OF TRUTH — the local tree is a cache materialized from it by
* r2-sync (`npm run screenshots:fetch`).
*
* promote.ts is the single writer: it regenerates this file after applying and
* uploading a CI run's render. Regeneration therefore needs the local tree
* present; --check deliberately does NOT (it must stay credential- and
* network-free so it can gate every CI caller, secretless ones included).
*
* CLI (from tests/): tsx tools/screenshots/gen-manifest.ts [--check [--remote]]
* --check exit 1 unless the committed manifest is well-formed (v2 schema,
* sorted+unique entries, valid hashes) and no baseline PNG has been
* re-committed to git (resurrection guard).
* --remote with --check: also HEAD every hash in R2 (needs credentials;
* manual use only, not wired into CI).
*/
import * as fs from 'fs';
import * as path from 'path';
import { execFileSync } from 'child_process';
import { PNG } from 'pngjs';
import {
BASELINE_ROOT,
ENGINES,
IGNORE_SCREENSHOTS,
MANIFEST_PATH,
MANIFEST_VERSION,
R2_DEFAULT_BUCKET,
R2_KEY_PREFIX,
isIgnored,
listEngineKeys,
splitKey,
type Manifest,
type ManifestEntry,
} from './config';
import { hashFile, missingEnv, storeFromEnv } from './r2-store';
/** Locale-independent (code-unit) ordering — the writer runs on dev machines,
* the --check gate on CI, and localeCompare's result depends on the host locale. */
function compareEntries(a: { engine: string; name: string }, b: { engine: string; name: string }): number {
if (a.engine !== b.engine) return a.engine < b.engine ? -1 : 1;
return a.name < b.name ? -1 : a.name > b.name ? 1 : 0;
}
const NOTE =
'Baselines live in R2, content-addressed by sha256 — the local baseline-screenshots/ tree is a cache ' +
'(`npm run screenshots:fetch`). Regenerated by `npm run screenshots:promote`; never edit hashes by hand. ' +
'The name list is authoritative for removed-screenshot detection.';
/** Render the manifest JSON for the current baseline cache tree (tree must be present). */
export function manifestJson(root: string): string {
const base = path.join(root, BASELINE_ROOT);
const screenshots: ManifestEntry[] = listEngineKeys(base)
.filter((key) => !isIgnored(key))
.map((key) => {
const { engine, name } = splitKey(key);
const file = path.join(base, key);
const png = PNG.sync.read(fs.readFileSync(file));
return {
name,
engine,
sha256: hashFile(file),
bytes: fs.statSync(file).size,
width: png.width,
height: png.height,
};
})
.sort(compareEntries);
const manifest: Manifest & { _note: string } = {
_note: NOTE,
version: MANIFEST_VERSION,
storage: { bucket: R2_DEFAULT_BUCKET, keyPrefix: R2_KEY_PREFIX },
screenshots,
};
return JSON.stringify(manifest, null, 2) + '\n';
}
/** Regenerate the manifest on disk (used by the CLI and by promote.ts after apply). */
export function writeManifest(root: string): void {
fs.writeFileSync(path.join(root, MANIFEST_PATH), manifestJson(root));
}
/** Schema + hygiene problems with the committed manifest (empty = healthy). Credential-free. */
export function checkManifest(root: string): string[] {
const problems: string[] = [];
const p = path.join(root, MANIFEST_PATH);
if (!fs.existsSync(p)) return [`${MANIFEST_PATH} is missing`];
let manifest: Manifest;
try {
manifest = JSON.parse(fs.readFileSync(p, 'utf8')) as Manifest;
} catch (e) {
return [`${MANIFEST_PATH} is not valid JSON: ${(e as Error).message}`];
}
if (manifest.version !== MANIFEST_VERSION) problems.push(`version is ${manifest.version}, expected ${MANIFEST_VERSION}`);
if (!manifest.storage?.bucket || !manifest.storage?.keyPrefix) problems.push('storage.bucket/keyPrefix missing');
if (!Array.isArray(manifest.screenshots) || manifest.screenshots.length === 0) {
problems.push('screenshots list is missing or empty');
return problems;
}
const engines = new Set<string>(ENGINES);
let prev: ManifestEntry | null = null;
const seen = new Set<string>();
for (const e of manifest.screenshots) {
const id = `${e.engine}/${e.name}`;
if (!e.name?.toLowerCase().endsWith('.png')) problems.push(`${id}: name is not a .png`);
if (!engines.has(e.engine)) problems.push(`${id}: unknown engine`);
if (!/^[0-9a-f]{64}$/.test(e.sha256 ?? '')) problems.push(`${id}: sha256 is not 64 lowercase hex chars`);
if (!(e.bytes > 0) || !(e.width > 0) || !(e.height > 0)) problems.push(`${id}: bytes/width/height must be positive`);
if (IGNORE_SCREENSHOTS.has(e.name)) problems.push(`${id}: is in IGNORE_SCREENSHOTS and must not be listed`);
if (seen.has(id)) problems.push(`${id}: duplicate entry`);
seen.add(id);
if (prev && compareEntries(prev, e) > 0) {
problems.push(`${id}: not sorted (engine, then name)`);
}
prev = e;
}
// Resurrection guard: baselines must never come back as committed files.
try {
const tracked = execFileSync('git', ['ls-files', '--', BASELINE_ROOT], { cwd: root, encoding: 'utf8' }).trim();
if (tracked) {
problems.push(
`git-tracked PNGs under ${BASELINE_ROOT}/ — baselines live in R2, do not commit them:\n ` +
tracked.split('\n').slice(0, 5).join('\n ')
);
}
} catch {
console.warn('[manifest] git unavailable — skipping the resurrection guard');
}
return problems;
}
async function main(): Promise<void> {
const check = process.argv.includes('--check');
const remote = process.argv.includes('--remote');
const root = process.cwd();
const outPath = path.join(root, MANIFEST_PATH);
if (check) {
const problems = checkManifest(root);
if (problems.length) {
console.error(`[manifest] INVALID:\n - ${problems.join('\n - ')}`);
process.exitCode = 1;
return;
}
const manifest = JSON.parse(fs.readFileSync(outPath, 'utf8')) as Manifest;
console.log(`[manifest] ok — ${manifest.screenshots.length} screenshots across engines`);
if (remote) {
const store = storeFromEnv();
if (!store) {
console.error(`[manifest] --remote needs credentials: set ${missingEnv().join(', ')}`);
process.exitCode = 2;
return;
}
const { verifyBaselines } = await import('./r2-sync');
const missing = await verifyBaselines(root, store);
if (missing.length) {
console.error(`[manifest] ${missing.length} hash(es) missing in R2:\n ${missing.join('\n ')}`);
process.exitCode = 1;
} else {
console.log('[manifest] every hash present in R2');
}
}
return;
}
// Regeneration path: requires the cache tree (an absent tree would silently
// produce an empty manifest and disable removed-screenshot detection).
if (listEngineKeys(path.join(root, BASELINE_ROOT)).length === 0) {
console.error('[manifest] baseline cache is empty — run `npm run screenshots:fetch` first (or promote a run)');
process.exitCode = 1;
return;
}
const json = manifestJson(root);
fs.writeFileSync(outPath, json);
const count = (JSON.parse(json) as Manifest).screenshots.length;
console.log(`[manifest] wrote ${MANIFEST_PATH} (${count} screenshots)`);
}
if (require.main === module) {
main().catch((e) => {
console.error(`[manifest] ${(e as Error).message}`);
process.exitCode = 1;
});
}

View file

@ -142,7 +142,7 @@ function buildHeader(report: Report | null, perfBlock: string, meta: { sha?: str
} else if (!report.changed.length && !report.added.length && !report.removed.length) { } else if (!report.changed.length && !report.added.length && !report.removed.length) {
lines.push('✅ no screenshot drift'); lines.push('✅ no screenshot drift');
} else if (report.driftLikely) { } else if (report.driftLikely) {
lines.push(`⚠️ **${report.changed.length} screenshots changed broadly** — looks like host env drift → re-promote (\`npm run screenshots:promote\`)`); lines.push(`⚠️ **${report.changed.length} screenshots changed broadly** — looks like host env drift → re-promote in morelli (https://pcbjam-morelli-staging.pcbjam-staging.workers.dev)`);
} else { } else {
lines.push( lines.push(
`⚠️ **screenshot drift**: ${report.changed.length} changed${byEngine(report.changed)}, ` + `⚠️ **screenshot drift**: ${report.changed.length} changed${byEngine(report.changed)}, ` +

View file

@ -1,205 +0,0 @@
/**
* Churn-free baseline updater — "promote CI's render".
*
* CI's x86 render is the source of truth. This pulls a CI run's screenshots
* (`gh run download`) — or a local dir via --from — and, for each one, overwrites
* the cached baseline ONLY when the decoded pixels differ beyond the per-engine
* floor. Unchanged baselines keep their bytes (never re-encoded), so their hash —
* and therefore the manifest — sees no churn. New shots are added; baselines with
* no render are reported as removal candidates and only dropped with --prune.
*
* Baselines live in a private R2 bucket (content-addressed; see r2-store.ts).
* The flow is: sync the local cache from the committed manifest → build/apply
* the plan locally → upload the updated/added bytes to R2 → regenerate the
* manifest. Only the manifest diff is committed; R2 objects are immutable and
* never deleted (--prune removes the manifest entry, old commits still resolve).
* Needs the READ-WRITE credential pair (see tools/screenshots/README.md).
*
* CLI (from tests/):
* tsx tools/screenshots/promote.ts --run <ci-run-id> [--repo owner/repo] [--prune] [--dry-run]
* tsx tools/screenshots/promote.ts --from <dir> [--prune] [--dry-run]
*/
import * as fs from 'fs';
import * as os from 'os';
import * as path from 'path';
import { execFileSync } from 'child_process';
import { BASELINE_ROOT, MANIFEST_PATH, floorFor, isIgnored, listEngineKeys, splitKey, type Manifest } from './config';
import { writeManifest } from './gen-manifest';
import { diffImages, loadPng } from './image-ops';
import { hashBytes, missingEnv, storeFromEnv, type R2Store } from './r2-store';
import { loadManifestV2, pool, pullBaselines } from './r2-sync';
/** key (`<engine>/<name>`) → absolute cached baseline path. */
function baselineIndex(root: string): Map<string, string> {
const abs = path.join(root, BASELINE_ROOT);
const index = new Map<string, string>();
for (const key of listEngineKeys(abs)) index.set(key, path.join(abs, key));
return index;
}
function loadManifest(root: string): Manifest | undefined {
const p = path.join(root, MANIFEST_PATH);
if (!fs.existsSync(p)) return undefined;
try {
return JSON.parse(fs.readFileSync(p, 'utf8')) as Manifest;
} catch {
return undefined;
}
}
/** Download a CI run's artifact and return the top-level test-results dir holding the shots. */
function downloadRun(runId: string, repo?: string): string {
const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'promote-'));
const repoArgs = repo ? ['--repo', repo] : [];
execFileSync('gh', ['run', 'download', runId, '-D', tmp, ...repoArgs], { stdio: 'inherit' });
// The artifact stores test-results/** — find that dir; its immediate *.png are the shots
// (exclude the nested screenshot-diff/ triptychs).
const stack = [tmp];
while (stack.length) {
const dir = stack.pop()!;
const entries = fs.readdirSync(dir, { withFileTypes: true });
if (path.basename(dir) === 'test-results') return dir;
for (const e of entries) if (e.isDirectory()) stack.push(path.join(dir, e.name));
}
throw new Error(`no test-results/ dir found in the downloaded artifact under ${tmp}`);
}
type Plan = { updated: string[]; added: string[]; unchanged: string[]; removedCandidates: string[] };
function buildPlan(root: string, renderDir: string, manifest?: Manifest): { plan: Plan; apply: () => void } {
const baselines = baselineIndex(root);
const rendered = new Set(listEngineKeys(renderDir));
// Never promote excluded screenshots (e.g. the flaky retinascale fullPage shot).
for (const key of [...rendered]) if (isIgnored(key)) rendered.delete(key);
for (const key of [...baselines.keys()]) if (isIgnored(key)) baselines.delete(key);
const plan: Plan = { updated: [], added: [], unchanged: [], removedCandidates: [] };
const actions: Array<() => void> = [];
for (const key of rendered) {
const src = path.join(renderDir, key);
const existing = baselines.get(key);
if (!existing) {
const dest = path.join(root, BASELINE_ROOT, key);
plan.added.push(key);
actions.push(() => {
fs.mkdirSync(path.dirname(dest), { recursive: true });
fs.copyFileSync(src, dest); // verbatim bytes
});
continue;
}
const d = diffImages(loadPng(existing), loadPng(src));
const floor = floorFor(key);
if (!d.dimsMatch || d.changedRatio > floor.changedRatio) {
plan.updated.push(key);
actions.push(() => fs.copyFileSync(src, existing)); // verbatim bytes, no re-encode → no churn
} else {
plan.unchanged.push(key); // leave the cached file untouched
}
}
// Removal candidates: a manifest-listed baseline this render didn't produce.
for (const [key, abs] of baselines) {
if (rendered.has(key)) continue;
const { engine, name } = splitKey(key);
if (manifest && !manifest.screenshots.some((e) => e.name === name && e.engine === engine)) continue;
plan.removedCandidates.push(key);
actions.push(() => {}); // pruning is opt-in (see main)
void abs;
}
return { plan, apply: () => actions.forEach((a) => a()) };
}
/** Upload the applied updated/added baselines to R2 (skip-if-exists per hash). */
async function uploadApplied(root: string, store: R2Store, keys: string[]): Promise<number> {
let uploaded = 0;
await pool(keys, 8, async (key) => {
const bytes = fs.readFileSync(path.join(root, BASELINE_ROOT, key));
if ((await store.put(hashBytes(bytes), bytes)) === 'uploaded') uploaded++;
});
return uploaded;
}
function parseArgs(argv: string[]): Record<string, string | boolean> {
const out: Record<string, string | boolean> = {};
for (let i = 0; i < argv.length; i++) {
const a = argv[i];
if (a === '--run') out.run = argv[++i];
else if (a === '--from') out.from = argv[++i];
else if (a === '--repo') out.repo = argv[++i];
else if (a === '--prune') out.prune = true;
else if (a === '--dry-run') out.dryRun = true;
}
return out;
}
async function main(): Promise<void> {
const args = parseArgs(process.argv.slice(2));
const root = process.cwd();
if (!args.run && !args.from) {
console.error('usage: promote.ts --run <ci-run-id> [--repo owner/repo] | --from <dir> [--prune] [--dry-run]');
process.exitCode = 2;
return;
}
// Fail fast BEFORE downloading anything: a promote that can't upload would
// otherwise leave the manifest referencing hashes that don't exist in R2.
const store = storeFromEnv();
if (!store) {
console.error(`[promote] R2 read-write credentials required: set ${missingEnv().join(', ')}`);
process.exitCode = 2;
return;
}
// Refuse a pre-migration checkout rather than warn: buildPlan would mark
// most renders "unchanged" (so never uploaded) while writeManifest pins
// their hashes anyway — committing a manifest that references objects R2
// doesn't have. Sync the checkout to a post-migration revision first.
if (!loadManifestV2(root)) {
console.error(`[promote] ${MANIFEST_PATH} is not the R2-backed v2 format — refusing to promote on a pre-migration checkout`);
process.exitCode = 2;
return;
}
// The local tree is a cache — sync it to the committed manifest so the plan
// diffs against exactly what the manifest pins (a stale/absent cache would
// otherwise misreport adds/updates).
const { downloaded, cached, deleted } = await pullBaselines(root, store);
console.log(`[promote] cache synced: downloaded=${downloaded} cached=${cached} deleted=${deleted}`);
const renderDir = args.from ? (args.from as string) : downloadRun(args.run as string, args.repo as string);
const manifest = loadManifest(root);
const { plan, apply } = buildPlan(root, renderDir, manifest);
console.log(
`[promote] updated=${plan.updated.length} added=${plan.added.length} ` +
`unchanged=${plan.unchanged.length} removal-candidates=${plan.removedCandidates.length}`
);
for (const n of plan.updated) console.log(` UPDATE ${n}`);
for (const n of plan.added) console.log(` ADD ${n}`);
for (const n of plan.removedCandidates) console.log(` REMOVE? ${n}${args.prune ? ' (pruning)' : ' (use --prune to delete)'}`);
if (args.dryRun) {
console.log('[promote] dry-run — no files written, nothing uploaded');
return;
}
apply();
if (args.prune) {
const baselines = baselineIndex(root);
// Prune drops the local file (and, below, the manifest entry). The R2
// object is deliberately kept — old commits must still resolve it.
for (const n of plan.removedCandidates) fs.rmSync(baselines.get(n)!, { force: true });
}
// Upload BEFORE regenerating the manifest: a failure here leaves at worst an
// orphaned CAS object, never a committed manifest pointing at a missing hash.
const uploaded = await uploadApplied(root, store, [...plan.updated, ...plan.added]);
console.log(`[promote] uploaded ${uploaded} object(s) to R2`);
// Keep the manifest in lockstep with the baseline tree — a stale manifest silently
// disables removed-screenshot detection for anything added after the last regen.
writeManifest(root);
console.log(`[promote] done — commit the ${MANIFEST_PATH} diff (the only git-visible output)`);
}
if (require.main === module) {
main().catch((e) => {
console.error(`[promote] ${(e as Error).message}`);
process.exitCode = 1;
});
}

View file

@ -99,6 +99,21 @@ export class R2Store {
return 'uploaded'; return 'uploaded';
} }
/**
* Download an ARBITRARY key (the R2-hosted baseline manifest) — unlike
* get(), not content-addressed, so no hash verification is possible; null
* on 404 so the caller can degrade like the no-credentials path.
*/
async getKey(key: string): Promise<Buffer | null> {
const res = await this.fetchWithRetry(`${this.base}/${key}`, { method: 'GET' });
if (res.status === 404) {
await res.arrayBuffer().catch(() => undefined);
return null;
}
if (res.status !== 200) throw new Error(`GET ${key} → HTTP ${res.status}`);
return Buffer.from(await res.arrayBuffer());
}
/** /**
* Upload to an ARBITRARY key (the per-run uploads under runs/…, consumed by * Upload to an ARBITRARY key (the per-run uploads under runs/…, consumed by
* the morelli review app) — unlike put(), not content-addressed and always * the morelli review app) — unlike put(), not content-addressed and always

View file

@ -1,31 +1,38 @@
/** /**
* Sync the local baseline-screenshots/ cache with the R2 CAS bucket. * Sync the local baseline-screenshots/ cache with the R2 CAS bucket.
* *
* The committed manifest (screenshot-manifest.json) is the source of truth; * The R2-HOSTED manifest (baselines/pcbjam/manifest.json, written only by the
* baseline-screenshots/ is a gitignored local cache materialized from it. * morelli review app + its seed script) is the source of truth. --manifest
* downloads it to the gitignored MANIFEST_PATH; everything downstream (--pull,
* --verify, compare.ts) reads that local copy, so a single fetch pins the
* whole run to one manifest version.
* *
* CLI (from tests/): * CLI (from tests/):
* tsx tools/screenshots/r2-sync.ts --pull # manifest → local tree (CI fetch step + local gate) * tsx tools/screenshots/r2-sync.ts --manifest # R2 manifest → local .baseline-manifest.json
* tsx tools/screenshots/r2-sync.ts --push # upload manifest entries missing in R2 (seeding) * tsx tools/screenshots/r2-sync.ts --pull # fetched manifest → local tree (CI fetch step + local gate)
* tsx tools/screenshots/r2-sync.ts --verify # HEAD every manifest hash, exit 1 on any miss * tsx tools/screenshots/r2-sync.ts --verify # HEAD every manifest hash, exit 1 on any miss
* *
* --pull without credentials (or with a pre-migration manifest) warns and * --manifest and --pull without credentials warn and exit 0 — and --manifest
* exits 0, so secretless callers of the reusable CI workflow (release.yml, * DELETES a stale local manifest so the gate skips rather than comparing
* pcbjam deploy-staging.yml, fork PRs) stay green — compare.ts then skips its * against outdated baselines. Secretless callers of the reusable CI workflow
* gate for the same reason. With credentials, any 404/corrupt object is * (release.yml, pcbjam deploy-staging.yml, fork PRs) therefore stay green;
* collected and the run exits 1. * compare.ts skips its gate when no manifest was fetched. With credentials,
* any 404/corrupt object is collected and the run exits 1.
*
* (--push is gone with the git-manifest era: baseline bytes now enter the CAS
* only via morelli's promote, which copies them from the CI run uploads.)
*/ */
import * as fs from 'fs'; import * as fs from 'fs';
import * as path from 'path'; import * as path from 'path';
import { BASELINE_ROOT, MANIFEST_PATH, MANIFEST_VERSION, listEngineKeys, type Manifest, type ManifestEntry } from './config'; import { BASELINE_ROOT, MANIFEST_PATH, MANIFEST_VERSION, R2_BASELINES_MANIFEST_KEY, listEngineKeys, type Manifest, type ManifestEntry } from './config';
import { R2Store, hashFile, missingEnv, storeFromEnv } from './r2-store'; import { R2Store, hashFile, missingEnv, storeFromEnv } from './r2-store';
const CONCURRENCY = 16; const CONCURRENCY = 16;
/** Parse the committed manifest if it is the R2-backed v2 format; null for a /** Parse the fetched manifest if it is the morelli-era v3 format; null when
* pre-migration (v1/absent/unparsable) manifest. A NEWER version throws — old * absent/unparsable (→ the gate skips). A NEWER version throws — old tooling
* tooling silently no-oping on a future format would disable the whole gate. */ * silently no-oping on a future format would disable the whole gate. */
export function loadManifestV2(root: string): Manifest | null { export function loadManifest(root: string): Manifest | null {
const p = path.join(root, MANIFEST_PATH); const p = path.join(root, MANIFEST_PATH);
if (!fs.existsSync(p)) return null; if (!fs.existsSync(p)) return null;
let m: Manifest; let m: Manifest;
@ -40,6 +47,25 @@ export function loadManifestV2(root: string): Manifest | null {
return m.version === MANIFEST_VERSION ? m : null; return m.version === MANIFEST_VERSION ? m : null;
} }
/**
* Download the R2-hosted baseline manifest to MANIFEST_PATH (atomic tmp+rename).
* Returns false when it could not be fetched — in which case any stale local
* copy is removed, so downstream steps skip instead of using old baselines.
*/
export async function fetchManifest(root: string, store: R2Store): Promise<boolean> {
const dest = path.join(root, MANIFEST_PATH);
const bytes = await store.getKey(R2_BASELINES_MANIFEST_KEY);
if (!bytes) {
fs.rmSync(dest, { force: true });
console.warn(`[r2-sync] ${R2_BASELINES_MANIFEST_KEY} not found in R2 — no baselines (seed via morelli first)`);
return false;
}
const tmp = `${dest}.tmp-${process.pid}`;
fs.writeFileSync(tmp, bytes);
fs.renameSync(tmp, dest);
return true;
}
/** Run `fn` over `items` with at most `limit` in flight. */ /** Run `fn` over `items` with at most `limit` in flight. */
export async function pool<T>(items: T[], limit: number, fn: (item: T) => Promise<void>): Promise<void> { export async function pool<T>(items: T[], limit: number, fn: (item: T) => Promise<void>): Promise<void> {
let i = 0; let i = 0;
@ -60,8 +86,8 @@ export async function pullBaselines(
root: string, root: string,
store: R2Store store: R2Store
): Promise<{ downloaded: number; cached: number; deleted: number }> { ): Promise<{ downloaded: number; cached: number; deleted: number }> {
const manifest = loadManifestV2(root); const manifest = loadManifest(root);
if (!manifest) throw new Error(`no v2 ${MANIFEST_PATH} — nothing to pull`); if (!manifest) throw new Error(`no fetched ${MANIFEST_PATH} — run \`npm run screenshots:fetch-manifest\` first`);
const base = path.join(root, BASELINE_ROOT); const base = path.join(root, BASELINE_ROOT);
const wanted = new Map<string, ManifestEntry>(); const wanted = new Map<string, ManifestEntry>();
for (const e of manifest.screenshots) wanted.set(`${e.engine}/${e.name}`, e); for (const e of manifest.screenshots) wanted.set(`${e.engine}/${e.name}`, e);
@ -99,35 +125,10 @@ export async function pullBaselines(
return { downloaded, cached, deleted }; return { downloaded, cached, deleted };
} }
/** Upload every manifest entry's local file to R2 (skipping hashes already present). */
export async function pushBaselines(root: string, store: R2Store): Promise<{ uploaded: number; existing: number }> {
const manifest = loadManifestV2(root);
if (!manifest) throw new Error(`no v2 ${MANIFEST_PATH} — run \`npm run screenshots:manifest\` first`);
const base = path.join(root, BASELINE_ROOT);
let uploaded = 0;
let existing = 0;
const errors: string[] = [];
await pool(manifest.screenshots, CONCURRENCY, async (e) => {
const file = path.join(base, e.engine, e.name);
try {
if (!fs.existsSync(file)) throw new Error('local file missing');
const bytes = fs.readFileSync(file);
const hash = hashFile(file);
if (hash !== e.sha256) throw new Error(`local sha256 ${hash} ≠ manifest — regenerate the manifest`);
if ((await store.put(hash, bytes)) === 'uploaded') uploaded++;
else existing++;
} catch (err) {
errors.push(`${e.engine}/${e.name}: ${(err as Error).message}`);
}
});
if (errors.length) throw new Error(`${errors.length} upload(s) failed:\n ${errors.join('\n ')}`);
return { uploaded, existing };
}
/** HEAD every manifest hash; returns the missing keys. */ /** HEAD every manifest hash; returns the missing keys. */
export async function verifyBaselines(root: string, store: R2Store): Promise<string[]> { export async function verifyBaselines(root: string, store: R2Store): Promise<string[]> {
const manifest = loadManifestV2(root); const manifest = loadManifest(root);
if (!manifest) throw new Error(`no v2 ${MANIFEST_PATH}`); if (!manifest) throw new Error(`no fetched ${MANIFEST_PATH} — run \`npm run screenshots:fetch-manifest\` first`);
const missing: string[] = []; const missing: string[] = [];
await pool(manifest.screenshots, CONCURRENCY, async (e) => { await pool(manifest.screenshots, CONCURRENCY, async (e) => {
if (!(await store.exists(e.sha256))) missing.push(`${e.engine}/${e.name} (${e.sha256})`); if (!(await store.exists(e.sha256))) missing.push(`${e.engine}/${e.name} (${e.sha256})`);
@ -136,20 +137,27 @@ export async function verifyBaselines(root: string, store: R2Store): Promise<str
} }
async function main(): Promise<void> { async function main(): Promise<void> {
const mode = process.argv.find((a) => a === '--pull' || a === '--push' || a === '--verify'); const mode = process.argv.find((a) => a === '--manifest' || a === '--pull' || a === '--verify');
if (!mode) { if (!mode) {
console.error('usage: r2-sync.ts --pull | --push | --verify'); console.error('usage: r2-sync.ts --manifest | --pull | --verify');
process.exitCode = 2; process.exitCode = 2;
return; return;
} }
const root = process.cwd(); const root = process.cwd();
if (mode === '--pull' && !loadManifestV2(root)) { if (mode === '--pull' && !loadManifest(root)) {
console.log(`[r2-sync] ${MANIFEST_PATH} is not the R2-backed v2 format — nothing to fetch`); console.log(`[r2-sync] no fetched ${MANIFEST_PATH} — skipping pull (run screenshots:fetch-manifest first)`);
return; return; // exit 0: manifest fetch skipped/failed → the gate skips, never gates on stale data
} }
const store = storeFromEnv(); const store = storeFromEnv();
if (!store) { if (!store) {
if (mode === '--manifest') {
// Delete a stale copy so the downstream gate SKIPS instead of
// comparing against whatever manifest a previous run left behind.
fs.rmSync(path.join(root, MANIFEST_PATH), { force: true });
console.log(`[r2-sync] R2 credentials unset (${missingEnv().join(', ')}) — skipping manifest fetch`);
return; // exit 0: secretless CI callers stay green
}
if (mode === '--pull') { if (mode === '--pull') {
console.log(`[r2-sync] R2 credentials unset (${missingEnv().join(', ')}) — skipping baseline fetch`); console.log(`[r2-sync] R2 credentials unset (${missingEnv().join(', ')}) — skipping baseline fetch`);
return; // exit 0: secretless CI callers stay green return; // exit 0: secretless CI callers stay green
@ -159,12 +167,14 @@ async function main(): Promise<void> {
return; return;
} }
if (mode === '--pull') { if (mode === '--manifest') {
if (await fetchManifest(root, store)) {
const manifest = loadManifest(root);
console.log(`[r2-sync] fetched ${MANIFEST_PATH}: ${manifest?.screenshots.length ?? 0} baselines (updated ${manifest?.updatedAt ?? '?'} by ${manifest?.updatedBy ?? '?'})`);
}
} else if (mode === '--pull') {
const { downloaded, cached, deleted } = await pullBaselines(root, store); const { downloaded, cached, deleted } = await pullBaselines(root, store);
console.log(`[r2-sync] pull done: downloaded=${downloaded} cached=${cached} deleted=${deleted}`); console.log(`[r2-sync] pull done: downloaded=${downloaded} cached=${cached} deleted=${deleted}`);
} else if (mode === '--push') {
const { uploaded, existing } = await pushBaselines(root, store);
console.log(`[r2-sync] push done: uploaded=${uploaded} already-present=${existing}`);
} else { } else {
const missing = await verifyBaselines(root, store); const missing = await verifyBaselines(root, store);
if (missing.length) { if (missing.length) {