screenshots: R2-hosted manifest becomes the baseline source of truth (morelli cutover)

The committed tests/screenshot-manifest.json is retired. CI now downloads
baselines/pcbjam/manifest.json (manifest v3, written only by the morelli
review app + its seed script) to the gitignored .baseline-manifest.json,
and everything downstream (pull, verify, compare) reads that copy:

- config.ts: MANIFEST_VERSION 3, MANIFEST_PATH .baseline-manifest.json,
  R2_BASELINES_MANIFEST_KEY; ManifestEntry grows opaque provenance
- r2-sync.ts: new --manifest mode (atomic fetch; no-creds skip DELETES a
  stale copy so the gate skips rather than using old baselines); --push
  gone (bytes enter the CAS only via morelli's promote)
- compare.ts: hard-skips when no manifest was fetched — a stale warm
  cache can never gate
- wasm-build.yml: fetch-manifest step before the baselines cache; cache
  key now hashes the fetched manifest; the gen-manifest --check lint gate
  goes with the committed manifest
- deleted: screenshot-manifest.json, promote.ts, changelog.ts,
  gen-manifest.ts, screenshot-changelog.yml, promote-screenshots skill
- docs (CLAUDE/README/TESTING/WHATWORKS/tools README): promote flow is
  now https://pcbjam-morelli-staging.pcbjam-staging.workers.dev

Validated locally against the real bucket: fetch-manifest (492), cold
pull 492 / warm pull cached=492, no-creds skip chain, compare gate skip.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Istvan Matejcsok 2026-08-19 11:54:20 +02:00
commit ec7a1b0787
20 changed files with 239 additions and 4793 deletions

View file

@ -1,42 +0,0 @@
---
name: promote-screenshots
description: Promote a CI run's screenshot renders as the new baselines in the R2 bucket (pcbjam-ci-screenshots). Churn-free - only meaningfully-changed images upload; git gets only the tests/screenshot-manifest.json diff, never PNGs. Needs the read-write R2 keypair in tests/.env. Usage - "/promote-screenshots <ci-run-id> [--prune]". (scoped to pcbjam/ - covers the KiCad WASM e2e pipeline in tests/)
---
# promote-screenshots (pcbjam)
Bless a CI run's rendered screenshots as the new baselines. Baselines live in
the private R2 bucket `pcbjam-ci-screenshots` (prod Cloudflare account),
content-addressed as `sha256/<hex>.png`; the committed
`tests/screenshot-manifest.json` pins each `<engine>/<name>` to a hash. **Only
the manifest diff lands in git — never PNGs.**
## Prerequisites
- The READ-WRITE R2 keypair in the gitignored `tests/.env` (auto-loaded by the
tooling, shell env wins; format in `tests/tools/screenshots/README.md`,
values from the team vault). **Never print, echo, or commit these values.**
If `.env` is missing, ask the user to fill it — promote fails fast without it.
- The CI run id: `gh run list --workflow ci-ubicloud.yml` on PCBJam/pcbjam
(mind the active `gh` account — PCBJam repos need `matejcsok-pcb`).
## Steps
1. From `tests/`, dry-run first and show the user the plan:
`npm run screenshots:promote -- --run <ci-run-id> --dry-run`
2. Sanity-check it: a handful of UPDATE/ADD lines for an intentional UI change
is normal; hundreds of UPDATEs means environment drift — stop and confirm
with the user before applying.
3. Apply: same command without `--dry-run`. Add `--prune` only when the user
confirms screenshots were intentionally removed (prune edits the manifest;
R2 objects are never deleted — old commits still resolve).
4. `git status` must show ONLY `tests/screenshot-manifest.json` modified.
Commit that diff; on main it triggers the Discord baseline changelog.
## Never
- Never commit files under `tests/baseline-screenshots/` (the CI manifest
check fails the build if you do) or the `.env`.
- Never promote local (Mac) renders via `--from` — CI's Linux render is the
only source of truth.
- Never hand-edit hashes in the manifest; promote regenerates it.

View file

@ -1,45 +0,0 @@
name: screenshot-changelog
# Discord trigger B: when a push to main changes the screenshot manifest (the
# committed pin of every R2-stored baseline), post an old | new+boxes | heatmap
# triptych per changed baseline (ADDED image / REMOVED title too). No build, no
# GPU — it diffs two git revisions of the manifest and fetches the PNG bytes
# from the R2 CAS bucket (immutable objects, so the old rev's hashes always
# resolve), so it runs in ~30s. Complements the re-render drift gate in
# wasm-build.yml (which catches un-blessed renders); this is the human-facing
# feed of intentional baseline updates as they land.
on:
push:
branches: [main]
paths:
- 'tests/screenshot-manifest.json'
concurrency:
group: screenshot-changelog-${{ github.ref }}
cancel-in-progress: false
jobs:
changelog:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
# HEAD^ is needed to diff the tip commit's manifest changes. (A push of
# multiple commits reports only the tip commit's baseline diff — fine for
# a changelog; baseline refreshes are single commits in practice.)
fetch-depth: 2
- uses: actions/setup-node@v4
with:
node-version: 20
- name: Install screenshot tooling deps
working-directory: tests
run: npm ci
- name: Post baseline changelog to Discord
working-directory: tests
env:
DISCORD_WEBHOOK_URL: ${{ secrets.DISCORD_WEBHOOK_URL }}
CI_SCREENSHOTS_S3_ENDPOINT: ${{ vars.CI_SCREENSHOTS_S3_ENDPOINT }}
CI_SCREENSHOTS_S3_ACCESS_KEY_ID: ${{ secrets.CI_SCREENSHOTS_S3_ACCESS_KEY_ID }}
CI_SCREENSHOTS_S3_SECRET_ACCESS_KEY: ${{ secrets.CI_SCREENSHOTS_S3_SECRET_ACCESS_KEY }}
run: npm run screenshots:changelog

View file

@ -318,19 +318,37 @@ jobs:
working-directory: tests
run: npm ci
# Baseline screenshots live in a private R2 bucket, pinned by the committed
# manifest; this cache keyed on the manifest hash makes most fetches a
# no-op. restore-keys makes a manifest change INCREMENTAL (restore the
# previous tree, download only the changed objects) instead of a full
# ~35MB re-download. A stale or partial restore is safe: r2-sync
# hash-verifies every file and deletes unlisted ones, and compare.ts
# refuses to gate against an incomplete cache.
# The baseline manifest lives in R2 (written only by the morelli review
# app + its seed script) — download it FIRST so this whole run pins to
# one manifest version and the cache step below can key on its hash.
# continue-on-error + the no-creds skip (which also deletes any stale
# local copy) keep secretless callers green — every later screenshot
# step then skips rather than gating on outdated baselines.
- name: Fetch baseline manifest from R2
if: inputs.run_tests
continue-on-error: true
working-directory: tests
env:
CI_SCREENSHOTS_S3_ENDPOINT: ${{ vars.CI_SCREENSHOTS_S3_ENDPOINT }}
CI_SCREENSHOTS_S3_ACCESS_KEY_ID: ${{ secrets.CI_SCREENSHOTS_S3_ACCESS_KEY_ID }}
CI_SCREENSHOTS_S3_SECRET_ACCESS_KEY: ${{ secrets.CI_SCREENSHOTS_S3_SECRET_ACCESS_KEY }}
run: npm run screenshots:fetch-manifest
# Baseline screenshots live in a private R2 bucket, pinned by the manifest
# fetched above; this cache keyed on the manifest hash makes most fetches
# a no-op (hashFiles evaluates at step run time, AFTER the fetch; a
# skipped fetch hashes to empty → restore-keys still warms the tree).
# restore-keys makes a manifest change INCREMENTAL (restore the previous
# tree, download only the changed objects) instead of a full ~35MB
# re-download. A stale or partial restore is safe: r2-sync hash-verifies
# every file and deletes unlisted ones, and compare.ts refuses to gate
# against an incomplete cache.
- name: Cache screenshot baselines
if: inputs.run_tests
uses: actions/cache@v4
with:
path: tests/baseline-screenshots
key: baselines-${{ hashFiles('tests/screenshot-manifest.json') }}
key: baselines-${{ hashFiles('tests/.baseline-manifest.json') }}
restore-keys: |
baselines-
@ -349,19 +367,17 @@ jobs:
run: npm run screenshots:fetch
# Cheap hygiene gates (no build needed): the determinism lint keeps the
# banned flake patterns out of the specs, the manifest check validates
# screenshot-manifest.json (v2 schema, sorted/unique hashes) and fails if
# baseline PNGs are ever re-committed to git (credential-free — it never
# touches R2, so it gates secretless callers identically), and
# the CI-coverage lint proves every spec file on disk is reachable from
# the npm scripts THIS workflow invokes (a spec/project that CI never
# runs is how the web suite once rotted unnoticed).
- name: Lint test determinism + screenshot manifest + CI coverage
# banned flake patterns out of the specs, and the CI-coverage lint proves
# every spec file on disk is reachable from the npm scripts THIS workflow
# invokes (a spec/project that CI never runs is how the web suite once
# rotted unnoticed). (The old screenshot-manifest --check gate went with
# the committed manifest — the manifest now lives in R2, written only by
# the morelli app, which validates on every promote.)
- name: Lint test determinism + CI coverage
if: inputs.run_tests
working-directory: tests
run: |
npm run lint:determinism
npm run screenshots:manifest -- --check
npm run lint:ci-coverage
- name: Install web workspace deps (collab bundle)

View file

@ -7,8 +7,8 @@ The e2e tests are in /tests, with a README and WHATWORKS md files
Test determinism rules (no blind sleeps/ifs, `stableShot` screenshots, retries:0) are in tests/TESTING.md, enforced by `npm run lint:determinism`.
The e2e tests are separated per feature
Wxwidgets wasm port has hooks for finding positions of UI elements, tests use that
The test screenshot baselines live in a private R2 bucket (content-addressed by sha256), pinned per engine by the committed tests/screenshot-manifest.json; tests/baseline-screenshots/{chromium,firefox}/ is a gitignored local cache — `cd tests && npm run screenshots:fetch` materializes it (needs the R2 credentials in tests/tools/screenshots/README.md). CI's Linux render is the source of truth (tooling: tests/tools/screenshots/, see its README).
To update baselines, promote a CI run's render (churn-free — only meaningfully-changed images re-upload): `cd tests && npm run screenshots:promote -- --run <ci-run-id>` (needs the read-write R2 credentials), then commit the manifest diff — never commit PNGs. `npm run screenshots:check` is the local gate (fetch first); on each main push CI posts a screenshot-diff + runtime-perf report to Discord.
The test screenshot baselines live in a private R2 bucket (content-addressed by sha256), pinned per engine by the R2-hosted manifest baselines/pcbjam/manifest.json — NOTHING manifest-related is in git; tests/baseline-screenshots/{chromium,firefox}/ is a gitignored local cache — `cd tests && npm run screenshots:fetch-manifest && npm run screenshots:fetch` materializes it (needs the R2 credentials in tests/tools/screenshots/README.md). CI's Linux render is the source of truth (tooling: tests/tools/screenshots/, see its README).
To update baselines, promote a CI run's screenshots in the morelli review app (https://pcbjam-morelli-staging.pcbjam-staging.workers.dev — pick the run, review the diffs, bulk-select, Promote). CI uploads every run's renders to R2 (runs/pcbjam/<run-id>/, 30-day retention) for that purpose. No git commit is involved. `npm run screenshots:check` is the local gate (fetch-manifest + fetch first); on each main push CI posts a screenshot-diff + runtime-perf report to Discord.
The tests have log files in tests/logs/{wxwidgets/kicad}/{test-name} after each run where the js console and cpp logs are visible
Always check screenshots for validating tests
Run e2e tests from /tests folder: `npm run test:e2e` (full CI project set, one merged playwright.config.ts) or `npm run test:kicad` (firefox shortcut) — not playwright directly. One spec/engine: `npx playwright test --project=kicad-firefox kicad/pcbnew.spec.ts`. Web-app suite: `npm run test:web`.

View file

@ -195,15 +195,18 @@ See [tests/README.md](tests/README.md) for test documentation.
### Screenshots
CI's Linux render is the source of truth for baseline screenshots. On each `main`
push, CI compares its render against the committed baselines and posts the diff
(plus the runtime-perf numbers) to Discord. To update baselines after an intended
render change, promote a CI run's render — only meaningfully-changed images
restage, so it stays churn-free:
push, CI compares its render against the baselines (pinned by the R2-hosted
manifest — nothing screenshot-related is committed) and posts the diff (plus the
runtime-perf numbers) to Discord. To update baselines after an intended render
change, promote the CI run's screenshots in the morelli review app
(https://pcbjam-morelli-staging.pcbjam-staging.workers.dev) — pick the run,
review the diffs, bulk-select, Promote. CI uploads every run's renders to R2
for that purpose (30-day retention).
```bash
cd tests
npm run screenshots:check # local gate: current vs baselines
npm run screenshots:promote -- --run <ci-run-id> # adopt a CI run's render, then commit
npm run screenshots:fetch-manifest && npm run screenshots:fetch # materialize baselines
npm run screenshots:check # local gate: current vs baselines
```
See [tests/tools/screenshots/README.md](tests/tools/screenshots/README.md).

9
tests/.gitignore vendored
View file

@ -1,8 +1,11 @@
# Screenshot model: specs capture via stableShot() → page.screenshot into test-results/, which is
# transient and gitignored at the repo root (/tests/test-results/). Baselines live in a private R2
# bucket, content-addressed by sha256 and pinned by the committed screenshot-manifest.json;
# tests/baseline-screenshots/ is a gitignored local cache (`npm run screenshots:fetch`), authored
# from CI's deterministic Linux render and diffed OFFLINE by tools/screenshots. (3d-regression/ and
# bucket, content-addressed by sha256 and pinned by the R2-hosted manifest (fetched to the
# gitignored .baseline-manifest.json below — NOTHING manifest-related is committed);
# tests/baseline-screenshots/ is a gitignored local cache (`npm run screenshots:fetch-manifest &&
# npm run screenshots:fetch`), authored from CI's deterministic Linux render and diffed OFFLINE by
# tools/screenshots. Baselines are promoted from a CI run in the morelli app. (3d-regression/ and
# gal-regression/ baselines are still committed.) There are no Playwright-native *-snapshots/
# baselines — Playwright does no inline screenshot comparison.
.baseline-manifest.json
logs/

View file

@ -55,7 +55,7 @@ tests/
├── logs/ # Test logs (auto-generated)
├── test-results/{chromium,firefox}/ # Screenshots per engine (auto-generated)
├── baseline-screenshots/{chromium,firefox}/ # Reference screenshots per engine — gitignored cache of the R2 bucket (npm run screenshots:fetch)
├── screenshot-manifest.json # Committed pin of every baseline: {name, engine, sha256, …} (regenerated by screenshots:promote)
├── .baseline-manifest.json # Gitignored copy of the R2-hosted baseline manifest (npm run screenshots:fetch-manifest)
├── apps/ # Built WASM test applications
│ ├── minimal_test.html # Main test app
│ └── standalone/ # Individual component test apps
@ -83,13 +83,15 @@ Example log format:
## Screenshots
Tests capture raw PNGs to `test-results/<engine>/` (engine-scoped via
`stableShot`/`shotPath`). The offline gate compares them against the committed
per-engine baselines:
`stableShot`/`shotPath`). The offline gate compares them against the per-engine
baselines pinned by the R2-hosted manifest:
```bash
npm run screenshots:fetch-manifest && npm run screenshots:fetch
npm run screenshots:check
```
CI's Linux render is the source of truth — update baselines by promoting a CI
run (`npm run screenshots:promote -- --run <ci-run-id>`), never by copying
run in the morelli review app
(https://pcbjam-morelli-staging.pcbjam-staging.workers.dev), never by copying
local renders. Rules and details: `TESTING.md`.
## Viewing the App Directly

View file

@ -46,15 +46,18 @@ only) — not playwright directly. One spec on one engine:
`tests/baseline-screenshots/<engine>/` (+ the still-committed `3d-regression/`,
`gal-regression/`).
- **Baselines live in the private R2 bucket `pcbjam-ci-screenshots`, not git**:
the committed `tests/screenshot-manifest.json` pins each `{name, engine}` to a
sha256, and `baseline-screenshots/` is a gitignored cache — materialize it with
`npm run screenshots:fetch` (needs the R2 credentials; see
`tools/screenshots/README.md`). CI's lint step validates the manifest and fails
if baseline PNGs are ever committed again.
- **CI's Linux render is the source of truth**; baselines are promoted from CI
(`npm run screenshots:promote -- --run <ci-run-id>`, needs the read-write
credentials) — commit only the regenerated manifest diff, never PNGs. A local
(Mac) check shows font/render noise and is not the gate.
the R2-hosted manifest `baselines/pcbjam/manifest.json` pins each
`{name, engine}` to a sha256, and `baseline-screenshots/` +
`.baseline-manifest.json` are gitignored caches — materialize them with
`npm run screenshots:fetch-manifest && npm run screenshots:fetch` (needs the
R2 credentials; see `tools/screenshots/README.md`). Nothing
screenshot-related is committed to git.
- **CI's Linux render is the source of truth**; baselines are promoted from a
CI run in the morelli review app
(https://pcbjam-morelli-staging.pcbjam-staging.workers.dev) — CI uploads each
run's renders to R2 (30-day retention), morelli shows the diffs and writes
the manifest on Promote. A local (Mac) check shows font/render noise and is
not the gate.
- A continuously-animating state (timer, mid-slide) can't be a stable baseline — drop the shot.
## Retries

View file

@ -405,7 +405,7 @@ tests/
├── test-results/
│ ├── chromium/ # Latest run's captures per engine
│ └── firefox/
└── screenshot-manifest.json # Committed pin: {name, engine, sha256, …} per baseline
└── .baseline-manifest.json # GITIGNORED copy of the R2-hosted manifest (npm run screenshots:fetch-manifest)
```
Specs write via `stableShot(page, 'name.png')` / `shotPath(page, 'name.png')`
@ -416,7 +416,8 @@ browser, so the same spec on two engines produces two independent captures.
```bash
cd tests
npm run screenshots:fetch # materialize the baseline cache from R2 (idempotent)
npm run screenshots:fetch-manifest # download the R2-hosted baseline manifest
npm run screenshots:fetch # materialize the baseline cache from R2 (idempotent)
npm run screenshots:check
```
@ -428,14 +429,13 @@ detection is driven by the manifest). On each main push, CI posts the same repor
### Updating Baseline Screenshots
CI's Linux render is the source of truth — never copy local (Mac) renders into
the baselines. Promote from a CI run instead (churn-free: only meaningfully
changed images restage, and the manifest regenerates automatically):
the baselines. Promote from a CI run in the morelli review app instead
(churn-free: identical images keep their provenance, and the R2-hosted manifest
updates atomically — no git commit involved):
```bash
cd tests
npm run screenshots:promote -- --run <ci-run-id> # needs the read-write R2 keypair (tests/.env)
git commit # only screenshot-manifest.json changes — PNGs are uploaded to R2, never committed
```
1. Open https://pcbjam-morelli-staging.pcbjam-staging.workers.dev (GitHub sign-in).
2. Pipeline `pcbjam` → pick the run → review the side-by-side/pixel diffs.
3. Select the intended changes (or "Select all changed + added") → Promote.
### Running Tests with Screenshots

View file

@ -18,14 +18,11 @@
"lint:determinism": "tsx tools/lint-determinism.ts",
"lint:ci-coverage": "tsx tools/lint-ci-coverage.ts",
"screenshots:check": "tsx tools/screenshots/compare.ts",
"screenshots:promote": "tsx tools/screenshots/promote.ts",
"screenshots:noise": "tsx tools/screenshots/noise.ts",
"screenshots:report": "tsx tools/screenshots/post-discord.ts",
"screenshots:changelog": "tsx tools/screenshots/changelog.ts",
"screenshots:manifest": "tsx tools/screenshots/gen-manifest.ts",
"screenshots:fetch-manifest": "tsx tools/screenshots/r2-sync.ts --manifest",
"screenshots:fetch": "tsx tools/screenshots/r2-sync.ts --pull",
"screenshots:upload-run": "tsx tools/screenshots/upload-run.ts",
"screenshots:push": "tsx tools/screenshots/r2-sync.ts --push",
"3d:compare": "tsx tools/screenshots/compare-dirs.ts",
"3d:check": "tsx tools/screenshots/compare-dirs.ts --old 3d-regression/baseline --new 3d-regression/output/native --out 3d-regression/output/diff/native-self --floors 3d-regression/floors.json --level native-self --label 3d-native --fail-on-change",
"3d:check:webgl": "tsx tools/screenshots/compare-dirs.ts --old 3d-regression/baseline-webgl --new 3d-regression/output/webgl --out 3d-regression/output/diff/webgl-self --floors 3d-regression/floors.json --level webgl-self --label 3d-webgl --fail-on-change",

File diff suppressed because it is too large Load diff

View file

@ -1,39 +1,45 @@
# Screenshot regression + Discord review tooling
One comparison engine + a churn-free updater + a Discord reporter for the e2e
screenshots. Design and rationale: `~/.claude/plans/…snowglobe.md` (or ask).
One comparison engine + a Discord reporter for the e2e screenshots; baseline
updates happen in the morelli review app
(https://pcbjam-morelli-staging.pcbjam-staging.workers.dev, repo
github.com/PCBJam/morelli).
**Source of truth = CI's Linux render.** The dev never authors baselines on the
Mac (Mac fonts/GL ≠ CI). Instead, CI renders on every push; when a render change
is intentional you *promote* CI's artifact into the baselines. The environment
isn't pinned — if the host's Mesa/fonts drift, the gate lights up in Discord and
you just re-promote (broad + low-intensity change ⇒ likely drift).
Mac (Mac fonts/GL ≠ CI). CI renders on every push and uploads each run's
screenshots to R2 (`runs/pcbjam/<run-id>/`, 30-day retention, upload-run.ts);
when a render change is intentional you *promote* that run's screenshots in
morelli. The environment isn't pinned — if the host's Mesa/fonts drift, the
gate lights up in Discord and you just re-promote (broad + low-intensity
change ⇒ likely drift).
**Baselines live in R2, not git.** The PNGs sit in a private Cloudflare R2
bucket (`pcbjam-ci-screenshots`), content-addressed as `sha256/<hex>.png` and
immutable; the committed `screenshot-manifest.json` pins each `<engine>/<name>`
to a hash, so every git commit resolves its exact baselines. The local
`baseline-screenshots/` tree is a gitignored cache — `npm run screenshots:fetch`
materializes it; `promote` uploads new hashes and rewrites the manifest, and the
manifest diff is the only thing you commit.
**Baselines live entirely in R2, not git.** The PNGs sit in a private
Cloudflare R2 bucket (`pcbjam-ci-screenshots`), content-addressed as
`sha256/<hex>.png` and immutable; the R2-HOSTED manifest
`baselines/pcbjam/manifest.json` (written only by morelli + its seed script)
pins each `<engine>/<name>` to a hash. The local `baseline-screenshots/` tree
and `.baseline-manifest.json` are gitignored caches —
`npm run screenshots:fetch-manifest && npm run screenshots:fetch` materializes
them. Nothing screenshot-related is committed.
**Credentials** (S3 API, bucket-scoped, region `auto`):
```
CI_SCREENSHOTS_S3_ENDPOINT # https://<account-id>.r2.cloudflarestorage.com
CI_SCREENSHOTS_S3_BUCKET # optional, default pcbjam-ci-screenshots
CI_SCREENSHOTS_S3_ACCESS_KEY_ID # read-only pair in CI; read-write pair for promote
CI_SCREENSHOTS_S3_ACCESS_KEY_ID # read-only pair for fetch; CI's upload step maps in a write pair
CI_SCREENSHOTS_S3_SECRET_ACCESS_KEY
```
CI holds the read-only pair as repo secrets; devs get the read-write pair from
the team vault (ask) and put it in `tests/.env` (gitignored, auto-loaded by
r2-store.ts; shell env vars take precedence):
CI holds the read-only pair as repo secrets (plus `CI_SCREENSHOTS_S3_WRITE_*`
for the run-upload step); devs get a read pair from the team vault (ask) and
put it in `tests/.env` (gitignored, auto-loaded by r2-store.ts; shell env vars
take precedence):
```
CI_SCREENSHOTS_S3_ENDPOINT=https://<account-id>.r2.cloudflarestorage.com
CI_SCREENSHOTS_S3_ACCESS_KEY_ID=<rw-access-key-id>
CI_SCREENSHOTS_S3_SECRET_ACCESS_KEY=<rw-secret>
CI_SCREENSHOTS_S3_ACCESS_KEY_ID=<access-key-id>
CI_SCREENSHOTS_S3_SECRET_ACCESS_KEY=<secret>
```
Without credentials, fetch and the compare gate skip with a warning (secretless
CI callers stay green); promote refuses to run.
Without credentials, the manifest/baseline fetch and the compare gate skip with
a warning (secretless CI callers stay green).
**Everything is per-engine.** Specs write `test-results/<engine>/<name>.png`
(engine derived from the running browser via `stableShot`/`shotPath`); baselines
@ -45,27 +51,29 @@ The same spec on chromium + firefox is two independent gated screenshots.
- `config.ts` — thresholds, baseline dirs, per-engine floors (calibrate!), clustering knobs.
- `image-ops.ts` — PNG load/save, pixelmatch diff (AA-excluded), connected-component boxes, triptych compositing, size-cap resize.
- `compare.ts` — the comparison engine: classify per-engine baselines vs `test-results/<engine>/``test-results/screenshot-diff/report.json` + triptych/heatmap PNGs. `--pair` diffs two files.
- `promote.ts` — churn-free updater: pull a CI run's shots (`gh run download`) or `--from DIR`; overwrite a baseline only when pixels differ beyond the floor (verbatim bytes, no re-encode) → no git churn.
- `perf-report.ts` — renders the track-only runtime-perf table (loadMs/openMs/FPS) with Δ vs the previous main run (fetched via `gh`).
- `post-discord.ts` — the always-on CI-on-main report: SHA + e2e status + perf table, then screenshot triptychs (batched, size-capped, flood-collapsed).
- `changelog.ts` — Discord trigger B: git-history diff of the manifest between two revs, PNG bytes fetched from R2 (no build/GPU).
- `noise.ts` — calibration: diff two identical-input renders → per-engine noise floor.
- `gen-manifest.ts` — regenerate `screenshot-manifest.json` ({name, engine, sha256, bytes, width, height}) from the local baseline cache; `--check` (gating in CI, credential-free) validates the schema and fails if baseline PNGs are ever re-committed to git.
- `r2-store.ts` — minimal aws4fetch S3 client for the CAS bucket (get/put/exists by hash, downloads integrity-checked).
- `r2-sync.ts` — cache sync: `--pull` materializes `baseline-screenshots/` from the manifest (idempotent, deletes unlisted files), `--push` seeds/uploads, `--verify` HEADs every hash.
- `r2-sync.ts` — cache sync: `--manifest` downloads the R2-hosted baseline manifest, `--pull` materializes `baseline-screenshots/` from it (idempotent, deletes unlisted files), `--verify` HEADs every hash.
- `upload-run.ts` — CI-only: upload the run's renders + meta.json to `runs/pcbjam/<run-id>/` for morelli (needs the write pair; no-ops without credentials).
- `spec-map.ts` — best-effort screenshot-name → spec-file attribution for captions (scans `stableShot`/`shotPath` literals).
## npm scripts (run from `tests/`)
```
npm run screenshots:fetch # materialize the baseline cache from R2 (run before check; needs read creds)
npm run screenshots:check # gate: baselines vs test-results → report.json (exit 0; add --fail-on-change to gate)
npm run screenshots:promote -- --run <ci-run-id> # churn-free re-baseline from a CI run (or --from DIR; needs RW creds)
npm run screenshots:report -- --e2e pass # post the CI report to Discord (main+push only; needs DISCORD_WEBHOOK_URL)
npm run screenshots:changelog # post the baseline changelog (main+push only)
npm run screenshots:noise -- run1/ run2/ # calibrate floors
npm run screenshots:manifest # regenerate the manifest (--check to verify it's fresh)
npm run screenshots:fetch-manifest # download the R2-hosted baseline manifest (needs read creds)
npm run screenshots:fetch # materialize the baseline cache from R2 (run before check)
npm run screenshots:check # gate: baselines vs test-results → report.json (exit 0; add --fail-on-change to gate)
npm run screenshots:report -- --e2e pass # post the CI report to Discord (main+push only; needs DISCORD_WEBHOOK_URL)
npm run screenshots:noise -- run1/ run2/ # calibrate floors
npm run screenshots:upload-run -- --e2e pass # CI-only: upload the run's renders for morelli
```
Baseline promotion (single or bulk) happens in morelli — pick the run, review
the diffs, Promote. It copies verbatim bytes into the CAS, updates the R2
manifest atomically (with provenance: which run/branch/user), and snapshots the
previous manifest for revert.
## Activation checklist
- [x] `screenshot-manifest.json` generated ({name, engine} authoritative — derived from the per-engine baseline tree).
- [x] `scale:'device'``'css'` normalized (no-op at CI's DSF=1).

View file

@ -1,212 +0,0 @@
/**
* Baseline changelog (Discord trigger B) no build, no GPU.
*
* On push to main, diff the committed screenshot MANIFEST between two commits
* and post a "Baseline changelog" to Discord: a triptych (old | new+boxes |
* heatmap) for each CHANGED baseline, the image for each ADDED, and a titled
* line for each REMOVED. "added / removed" only mean anything as a git-history
* diff, which is why this is separate from the re-render drift gate.
*
* The PNGs themselves are not in git each manifest entry pins a sha256 and
* the bytes are fetched from the R2 CAS bucket (objects are immutable, so the
* base rev's hashes are always still resolvable). Needs the read credential
* pair; a base rev whose manifest predates the R2 migration (v1) is skipped so
* the migration commit itself doesn't post hundreds of bogus entries.
*
* CLI (from tests/):
* tsx tools/screenshots/changelog.ts [--base REV] [--head REV] [--dry-run] [--force]
*/
import * as fs from 'fs';
import * as path from 'path';
import { execFileSync } from 'child_process';
import { PNG } from 'pngjs';
import { DIFF_OUT_DIR, MANIFEST_PATH, MANIFEST_VERSION, floorFor, labelText, splitKey, LABEL, type Manifest } from './config';
import { comparePair, type Report } from './compare';
import { savePng, withBottomLabel } from './image-ops';
import { buildSpecResolver } from './spec-map';
import { buildAttachments, paginate, postMessage } from './post-discord';
import { missingEnv, storeFromEnv, type R2Store } from './r2-store';
function git(root: string, args: string[]): string {
return execFileSync('git', ['-C', root, ...args], { encoding: 'utf8' }).trim();
}
/** The screenshot manifest as committed at `rev`. Returns null ONLY when the
* file doesn't exist at that rev (pre-manifest era); any other git failure
* shallow clone, bad revision, moved path throws loudly instead of being
* mistaken for "predates the migration". Unparsable committed JSON also throws. */
function manifestAt(repoRoot: string, rev: string, repoPath: string): Manifest | null {
let raw: string;
try {
raw = execFileSync('git', ['-C', repoRoot, 'show', `${rev}:${repoPath}`], { encoding: 'utf8' });
} catch (e) {
const msg = ((e as { stderr?: Buffer }).stderr?.toString() ?? (e as Error).message).trim();
if (/does not exist in|exists on disk, but not in/i.test(msg)) return null;
throw new Error(`git show ${rev}:${repoPath} failed: ${msg}`);
}
return JSON.parse(raw) as Manifest;
}
function parseArgs(argv: string[]): Record<string, string | boolean> {
const out: Record<string, string | boolean> = {};
for (let i = 0; i < argv.length; i++) {
const a = argv[i];
if (a === '--dry-run') out.dryRun = true;
else if (a === '--force') out.force = true;
else if (a === '--base') out.base = argv[++i];
else if (a === '--head') out.head = argv[++i];
}
return out;
}
async function main(): Promise<void> {
const args = parseArgs(process.argv.slice(2));
const cwd = process.cwd();
const repoRoot = git(cwd, ['rev-parse', '--show-toplevel']);
const head = (args.head as string) || 'HEAD';
let base = (args.base as string) || '';
if (!base) {
try {
base = git(cwd, ['rev-parse', `${head}^`]);
} catch {
console.log('[changelog] no parent commit — nothing to diff');
return;
}
}
// Repo-relative manifest path (e.g. tests/screenshot-manifest.json).
const manifestPath = path.relative(repoRoot, path.join(cwd, MANIFEST_PATH)).split(path.sep).join('/');
const baseManifest = manifestAt(repoRoot, base, manifestPath);
const headManifest = manifestAt(repoRoot, head, manifestPath);
// A NEWER manifest than this tooling understands is a loud error, never a skip.
for (const [label, m] of [['head', headManifest], ['base', baseManifest]] as const) {
if (m && m.version > MANIFEST_VERSION) {
throw new Error(`${label} manifest is version ${m.version}, newer than this tooling (expects ${MANIFEST_VERSION}) — update the checkout`);
}
}
if (headManifest?.version !== MANIFEST_VERSION) {
console.log('[changelog] head manifest predates the R2 migration — nothing to diff');
return;
}
if (baseManifest?.version !== MANIFEST_VERSION) {
console.log('[changelog] base manifest predates the R2 migration — skipping (migration commit)');
return;
}
// Engine-qualified key (`<engine>/<name>`) → sha256, per rev.
const hashesOf = (m: Manifest): Map<string, string> =>
new Map(m.screenshots.map((e) => [`${e.engine}/${e.name}`, e.sha256]));
const baseHashes = hashesOf(baseManifest);
const headHashes = hashesOf(headManifest);
const added = [...headHashes.keys()].filter((k) => !baseHashes.has(k));
const removed = [...baseHashes.keys()].filter((k) => !headHashes.has(k));
const changed = [...headHashes.keys()].filter((k) => baseHashes.has(k) && baseHashes.get(k) !== headHashes.get(k));
if (!added.length && !removed.length && !changed.length) {
console.log('[changelog] no baseline changes between', base.slice(0, 7), 'and', head);
return;
}
// Tolerate missing credentials (warn + exit 0): this is a notification-only
// workflow — turning every manifest push red during a key rotation is worse
// than one missed changelog post. The warning names the fix.
const store = storeFromEnv();
if (!store) {
console.warn(`[changelog] R2 credentials unset (${missingEnv().join(', ')}) — cannot fetch baseline bytes; skipping the changelog post`);
return;
}
const outDir = path.join(cwd, DIFF_OUT_DIR);
fs.mkdirSync(outDir, { recursive: true });
const report: Report = {
generatedFor: process.env.GITHUB_SHA || head,
changed: [],
added: [],
removed: [],
unchangedCount: 0,
driftLikely: false,
};
const { specFor } = buildSpecResolver(cwd);
// One unresolvable object must not abort the whole post — skip that image
// (with a loud note) and keep reporting the rest, like the old git-blob
// path null-skipped unresolvable blobs.
const skipped: string[] = [];
const hashPng = async (s: R2Store, hash: string, key: string): Promise<PNG | null> => {
try {
return PNG.sync.read(await s.get(hash));
} catch (e) {
skipped.push(`${key}: ${(e as Error).message}`);
return null;
}
};
// Save a single captioned image (added/removed) and record it in the report.
const saveSingle = (img: PNG, key: string, status: 'added' | 'removed'): void => {
const rel = path.join(DIFF_OUT_DIR, `${key.replace('/', '_')}.${status}.png`);
savePng(path.join(cwd, rel), withBottomLabel(img, labelText(status, key, specFor(splitKey(key).name)), LABEL.colors[status]));
report[status].push({ name: key, image: rel });
};
for (const key of added) {
const img = await hashPng(store, headHashes.get(key)!, key);
if (img) saveSingle(img, key, 'added');
}
for (const key of removed) {
const img = await hashPng(store, baseHashes.get(key)!, key);
if (img) saveSingle(img, key, 'removed');
}
for (const key of changed) {
const oldImg = await hashPng(store, baseHashes.get(key)!, key);
const newImg = await hashPng(store, headHashes.get(key)!, key);
if (!oldImg || !newImg) continue;
const { result, heatmap, triptych } = comparePair(oldImg, newImg, key, floorFor(key));
const triptychRel = path.join(DIFF_OUT_DIR, `${key.replace('/', '_')}.triptych.png`);
const heatmapRel = path.join(DIFF_OUT_DIR, `${key.replace('/', '_')}.heatmap.png`);
savePng(path.join(cwd, triptychRel), withBottomLabel(triptych, labelText('changed', key, specFor(splitKey(key).name)), LABEL.colors.changed));
savePng(path.join(cwd, heatmapRel), heatmap);
report.changed.push({ ...result, triptych: triptychRel, heatmap: heatmapRel });
}
report.changed.sort((a, b) => b.changedRatio - a.changedRatio);
if (skipped.length) {
console.warn(`[changelog] skipped ${skipped.length} unresolvable image(s):\n ${skipped.join('\n ')}`);
}
const sha7 = (process.env.GITHUB_SHA || head).slice(0, 7);
let subject = '';
try {
subject = git(cwd, ['log', '-1', '--pretty=%s', head]);
} catch { /* ignore */ }
const header =
`🗂️ **Baseline changelog** · \`${sha7}\`` +
(subject ? `\n> ${subject}` : '') +
`\n${report.changed.length} changed, ${report.added.length} added, ${report.removed.length} removed` +
(skipped.length ? `\n⚠ ${skipped.length} image(s) unresolvable in R2 — see the workflow log` : '') +
(report.removed.length ? '\n REMOVED: ' + report.removed.map((r) => `\`${r.name}\``).join(', ') : '');
const { files, notes } = buildAttachments(cwd, report);
const messages = paginate(header + (notes.length ? '\n' + notes.join('\n') : ''), files);
const isMainPush = process.env.GITHUB_REF === 'refs/heads/main' && process.env.GITHUB_EVENT_NAME === 'push';
if (args.dryRun || (!args.force && !isMainPush)) {
for (const [i, m] of messages.entries()) {
console.log(`--- message ${i + 1}/${messages.length} (${m.files.length} files) ---`);
if (m.content) console.log(m.content);
for (const f of m.files) console.log(` [attach] ${f.name} (${f.buffer.length} bytes)`);
}
if (!args.dryRun) console.log('[changelog] not a push to main — not posting');
return;
}
const webhook = process.env.DISCORD_WEBHOOK_URL;
if (!webhook) {
console.log('[changelog] DISCORD_WEBHOOK_URL unset — skipping');
return;
}
for (const m of messages) await postMessage(webhook, m);
console.log(`[changelog] posted ${messages.length} message(s)`);
}
if (require.main === module) {
main().catch((e) => {
console.error(`[changelog] ${e.message}`);
process.exitCode = 1;
});
}

View file

@ -9,8 +9,8 @@
* The gate mode classifies every screenshot into changed / added / removed /
* unchanged, writes per-change triptych + heatmap PNGs and a machine-readable
* report.json into DIFF_OUT_DIR, and (unless --fail-on-change) exits 0 so it can
* run report-only first. post-discord.ts and the changelog workflow import the
* exported helpers rather than re-deriving the diff.
* run report-only first. post-discord.ts imports the exported helpers rather
* than re-deriving the diff.
*/
import * as fs from 'fs';
import * as path from 'path';
@ -236,7 +236,14 @@ function main(): void {
// dev who hasn't fetched. Either way skip without writing report.json —
// post-discord posts a distinct "gate SKIPPED" line when it's absent.
const gateManifest = loadManifest(root);
const wanted = (gateManifest?.screenshots ?? []).filter((e) => !isIgnored(`${e.engine}/${e.name}`));
// No fetched manifest at all (creds absent, R2 down, or morelli never
// seeded) ⇒ the gate has no source of truth — skip rather than diff
// against whatever stale cache a previous run left behind.
if (!gateManifest) {
console.log(`[compare] no fetched ${MANIFEST_PATH} — run \`npm run screenshots:fetch-manifest\` (needs R2 credentials); skipping`);
return;
}
const wanted = gateManifest.screenshots.filter((e) => !isIgnored(`${e.engine}/${e.name}`));
const missing = wanted.filter((e) => !fs.existsSync(path.join(root, BASELINE_ROOT, e.engine, e.name)));
if (wanted.length && missing.length) {
const what = missing.length === wanted.length ? 'empty' : `INCOMPLETE (${missing.length}/${wanted.length} missing — partial fetch?)`;

View file

@ -20,16 +20,18 @@ import * as nodePath from 'path';
* other's PNG, and the surviving file was whichever parallel worker wrote last.
*
* The baseline PNGs are NOT committed: they live in a private R2 bucket,
* content-addressed by sha256, and the committed manifest
* (screenshot-manifest.json) is what pins each `<engine>/<name>` to a hash.
* `npm run screenshots:fetch` (tools/screenshots/r2-sync.ts) materializes this
* tree from the manifest; promote.ts uploads new hashes and rewrites the
* manifest the manifest diff is the only thing that lands in git.
* content-addressed by sha256, pinned by the R2-HOSTED manifest
* (baselines/pcbjam/manifest.json nothing manifest-related is in git).
* `npm run screenshots:fetch-manifest && npm run screenshots:fetch`
* (tools/screenshots/r2-sync.ts) materializes this tree. Baselines are
* promoted from a CI run in the morelli review app
* (https://pcbjam-morelli-staging.pcbjam-staging.workers.dev) — the old
* promote.ts/git-manifest flow is retired.
*/
export const BASELINE_ROOT = 'baseline-screenshots';
/** Manifest format version — bumped when baselines moved from git to R2. */
export const MANIFEST_VERSION = 2;
/** Manifest format version — v3 = the R2-hosted manifest written by morelli (v2 was the committed-in-git era). */
export const MANIFEST_VERSION = 3;
/** Default private R2 bucket holding the content-addressed baselines. */
export const R2_DEFAULT_BUCKET = 'pcbjam-ci-screenshots';
@ -37,6 +39,9 @@ export const R2_DEFAULT_BUCKET = 'pcbjam-ci-screenshots';
/** Key prefix for content-addressed objects: `sha256/<64-hex>.png`. */
export const R2_KEY_PREFIX = 'sha256/';
/** The R2-hosted baseline manifest — THE source of truth. Written only by morelli (promote) and its seed script. */
export const R2_BASELINES_MANIFEST_KEY = 'baselines/pcbjam/manifest.json';
/**
* Env vars for the bucket's S3 API (read-only keypair in CI, read-write for
* devs running promote). Endpoint form: https://<account-id>.r2.cloudflarestorage.com
@ -80,8 +85,14 @@ export const RESULTS_DIR = 'test-results';
/** Where compare.ts writes diff/heatmap/triptych artifacts (gitignored). */
export const DIFF_OUT_DIR = 'test-results/screenshot-diff';
/** The manifest that records every expected screenshot + which engine renders it. */
export const MANIFEST_PATH = 'screenshot-manifest.json';
/**
* Local, GITIGNORED copy of the R2-hosted baseline manifest, downloaded by
* `npm run screenshots:fetch-manifest` (r2-sync --manifest). Everything
* downstream (fetch, compare, verify) reads this file; its absence means "no
* manifest fetched" and the screenshot gate skips. Deliberately NOT inside
* test-results/ Playwright wipes that dir at the start of every invocation.
*/
export const MANIFEST_PATH = '.baseline-manifest.json';
/**
* pixelmatch per-pixel settings.
@ -185,10 +196,22 @@ export function labelText(status: LabelStatus, key: string, spec: string | null)
* `height` are sanity metadata (cheap pre-hash check on fetch, dimension info
* in reviews).
*/
export type ManifestEntry = { name: string; engine: string; sha256: string; bytes: number; width: number; height: number };
export type ManifestEntry = {
name: string;
engine: string;
sha256: string;
bytes: number;
width: number;
height: number;
/** morelli-side provenance (which run/branch/user promoted this) — opaque to the CI tooling. */
source?: unknown;
};
export type Manifest = {
version: number;
pipeline?: string;
storage: { bucket: string; keyPrefix: string };
updatedAt?: string;
updatedBy?: string;
screenshots: ManifestEntry[];
};

View file

@ -1,191 +0,0 @@
/**
* Generate tests/screenshot-manifest.json the canonical list of expected
* screenshots AND the pointer to their bytes. Each entry is
* {name, engine, sha256, bytes, width, height}: the engine comes from the
* baseline cache tree (baseline-screenshots/<engine>/<name>.png), the sha256
* resolves the PNG in the private R2 bucket (`sha256/<hex>.png`). The manifest
* is the SOURCE OF TRUTH the local tree is a cache materialized from it by
* r2-sync (`npm run screenshots:fetch`).
*
* promote.ts is the single writer: it regenerates this file after applying and
* uploading a CI run's render. Regeneration therefore needs the local tree
* present; --check deliberately does NOT (it must stay credential- and
* network-free so it can gate every CI caller, secretless ones included).
*
* CLI (from tests/): tsx tools/screenshots/gen-manifest.ts [--check [--remote]]
* --check exit 1 unless the committed manifest is well-formed (v2 schema,
* sorted+unique entries, valid hashes) and no baseline PNG has been
* re-committed to git (resurrection guard).
* --remote with --check: also HEAD every hash in R2 (needs credentials;
* manual use only, not wired into CI).
*/
import * as fs from 'fs';
import * as path from 'path';
import { execFileSync } from 'child_process';
import { PNG } from 'pngjs';
import {
BASELINE_ROOT,
ENGINES,
IGNORE_SCREENSHOTS,
MANIFEST_PATH,
MANIFEST_VERSION,
R2_DEFAULT_BUCKET,
R2_KEY_PREFIX,
isIgnored,
listEngineKeys,
splitKey,
type Manifest,
type ManifestEntry,
} from './config';
import { hashFile, missingEnv, storeFromEnv } from './r2-store';
/** Locale-independent (code-unit) ordering the writer runs on dev machines,
* the --check gate on CI, and localeCompare's result depends on the host locale. */
function compareEntries(a: { engine: string; name: string }, b: { engine: string; name: string }): number {
if (a.engine !== b.engine) return a.engine < b.engine ? -1 : 1;
return a.name < b.name ? -1 : a.name > b.name ? 1 : 0;
}
const NOTE =
'Baselines live in R2, content-addressed by sha256 — the local baseline-screenshots/ tree is a cache ' +
'(`npm run screenshots:fetch`). Regenerated by `npm run screenshots:promote`; never edit hashes by hand. ' +
'The name list is authoritative for removed-screenshot detection.';
/** Render the manifest JSON for the current baseline cache tree (tree must be present). */
export function manifestJson(root: string): string {
const base = path.join(root, BASELINE_ROOT);
const screenshots: ManifestEntry[] = listEngineKeys(base)
.filter((key) => !isIgnored(key))
.map((key) => {
const { engine, name } = splitKey(key);
const file = path.join(base, key);
const png = PNG.sync.read(fs.readFileSync(file));
return {
name,
engine,
sha256: hashFile(file),
bytes: fs.statSync(file).size,
width: png.width,
height: png.height,
};
})
.sort(compareEntries);
const manifest: Manifest & { _note: string } = {
_note: NOTE,
version: MANIFEST_VERSION,
storage: { bucket: R2_DEFAULT_BUCKET, keyPrefix: R2_KEY_PREFIX },
screenshots,
};
return JSON.stringify(manifest, null, 2) + '\n';
}
/** Regenerate the manifest on disk (used by the CLI and by promote.ts after apply). */
export function writeManifest(root: string): void {
fs.writeFileSync(path.join(root, MANIFEST_PATH), manifestJson(root));
}
/** Schema + hygiene problems with the committed manifest (empty = healthy). Credential-free. */
export function checkManifest(root: string): string[] {
const problems: string[] = [];
const p = path.join(root, MANIFEST_PATH);
if (!fs.existsSync(p)) return [`${MANIFEST_PATH} is missing`];
let manifest: Manifest;
try {
manifest = JSON.parse(fs.readFileSync(p, 'utf8')) as Manifest;
} catch (e) {
return [`${MANIFEST_PATH} is not valid JSON: ${(e as Error).message}`];
}
if (manifest.version !== MANIFEST_VERSION) problems.push(`version is ${manifest.version}, expected ${MANIFEST_VERSION}`);
if (!manifest.storage?.bucket || !manifest.storage?.keyPrefix) problems.push('storage.bucket/keyPrefix missing');
if (!Array.isArray(manifest.screenshots) || manifest.screenshots.length === 0) {
problems.push('screenshots list is missing or empty');
return problems;
}
const engines = new Set<string>(ENGINES);
let prev: ManifestEntry | null = null;
const seen = new Set<string>();
for (const e of manifest.screenshots) {
const id = `${e.engine}/${e.name}`;
if (!e.name?.toLowerCase().endsWith('.png')) problems.push(`${id}: name is not a .png`);
if (!engines.has(e.engine)) problems.push(`${id}: unknown engine`);
if (!/^[0-9a-f]{64}$/.test(e.sha256 ?? '')) problems.push(`${id}: sha256 is not 64 lowercase hex chars`);
if (!(e.bytes > 0) || !(e.width > 0) || !(e.height > 0)) problems.push(`${id}: bytes/width/height must be positive`);
if (IGNORE_SCREENSHOTS.has(e.name)) problems.push(`${id}: is in IGNORE_SCREENSHOTS and must not be listed`);
if (seen.has(id)) problems.push(`${id}: duplicate entry`);
seen.add(id);
if (prev && compareEntries(prev, e) > 0) {
problems.push(`${id}: not sorted (engine, then name)`);
}
prev = e;
}
// Resurrection guard: baselines must never come back as committed files.
try {
const tracked = execFileSync('git', ['ls-files', '--', BASELINE_ROOT], { cwd: root, encoding: 'utf8' }).trim();
if (tracked) {
problems.push(
`git-tracked PNGs under ${BASELINE_ROOT}/ — baselines live in R2, do not commit them:\n ` +
tracked.split('\n').slice(0, 5).join('\n ')
);
}
} catch {
console.warn('[manifest] git unavailable — skipping the resurrection guard');
}
return problems;
}
async function main(): Promise<void> {
const check = process.argv.includes('--check');
const remote = process.argv.includes('--remote');
const root = process.cwd();
const outPath = path.join(root, MANIFEST_PATH);
if (check) {
const problems = checkManifest(root);
if (problems.length) {
console.error(`[manifest] INVALID:\n - ${problems.join('\n - ')}`);
process.exitCode = 1;
return;
}
const manifest = JSON.parse(fs.readFileSync(outPath, 'utf8')) as Manifest;
console.log(`[manifest] ok — ${manifest.screenshots.length} screenshots across engines`);
if (remote) {
const store = storeFromEnv();
if (!store) {
console.error(`[manifest] --remote needs credentials: set ${missingEnv().join(', ')}`);
process.exitCode = 2;
return;
}
const { verifyBaselines } = await import('./r2-sync');
const missing = await verifyBaselines(root, store);
if (missing.length) {
console.error(`[manifest] ${missing.length} hash(es) missing in R2:\n ${missing.join('\n ')}`);
process.exitCode = 1;
} else {
console.log('[manifest] every hash present in R2');
}
}
return;
}
// Regeneration path: requires the cache tree (an absent tree would silently
// produce an empty manifest and disable removed-screenshot detection).
if (listEngineKeys(path.join(root, BASELINE_ROOT)).length === 0) {
console.error('[manifest] baseline cache is empty — run `npm run screenshots:fetch` first (or promote a run)');
process.exitCode = 1;
return;
}
const json = manifestJson(root);
fs.writeFileSync(outPath, json);
const count = (JSON.parse(json) as Manifest).screenshots.length;
console.log(`[manifest] wrote ${MANIFEST_PATH} (${count} screenshots)`);
}
if (require.main === module) {
main().catch((e) => {
console.error(`[manifest] ${(e as Error).message}`);
process.exitCode = 1;
});
}

View file

@ -142,7 +142,7 @@ function buildHeader(report: Report | null, perfBlock: string, meta: { sha?: str
} else if (!report.changed.length && !report.added.length && !report.removed.length) {
lines.push('✅ no screenshot drift');
} else if (report.driftLikely) {
lines.push(`⚠️ **${report.changed.length} screenshots changed broadly** — looks like host env drift → re-promote (\`npm run screenshots:promote\`)`);
lines.push(`⚠️ **${report.changed.length} screenshots changed broadly** — looks like host env drift → re-promote in morelli (https://pcbjam-morelli-staging.pcbjam-staging.workers.dev)`);
} else {
lines.push(
`⚠️ **screenshot drift**: ${report.changed.length} changed${byEngine(report.changed)}, ` +

View file

@ -1,205 +0,0 @@
/**
* Churn-free baseline updater "promote CI's render".
*
* CI's x86 render is the source of truth. This pulls a CI run's screenshots
* (`gh run download`) or a local dir via --from and, for each one, overwrites
* the cached baseline ONLY when the decoded pixels differ beyond the per-engine
* floor. Unchanged baselines keep their bytes (never re-encoded), so their hash
* and therefore the manifest sees no churn. New shots are added; baselines with
* no render are reported as removal candidates and only dropped with --prune.
*
* Baselines live in a private R2 bucket (content-addressed; see r2-store.ts).
* The flow is: sync the local cache from the committed manifest build/apply
* the plan locally upload the updated/added bytes to R2 regenerate the
* manifest. Only the manifest diff is committed; R2 objects are immutable and
* never deleted (--prune removes the manifest entry, old commits still resolve).
* Needs the READ-WRITE credential pair (see tools/screenshots/README.md).
*
* CLI (from tests/):
* tsx tools/screenshots/promote.ts --run <ci-run-id> [--repo owner/repo] [--prune] [--dry-run]
* tsx tools/screenshots/promote.ts --from <dir> [--prune] [--dry-run]
*/
import * as fs from 'fs';
import * as os from 'os';
import * as path from 'path';
import { execFileSync } from 'child_process';
import { BASELINE_ROOT, MANIFEST_PATH, floorFor, isIgnored, listEngineKeys, splitKey, type Manifest } from './config';
import { writeManifest } from './gen-manifest';
import { diffImages, loadPng } from './image-ops';
import { hashBytes, missingEnv, storeFromEnv, type R2Store } from './r2-store';
import { loadManifestV2, pool, pullBaselines } from './r2-sync';
/** key (`<engine>/<name>`) → absolute cached baseline path. */
function baselineIndex(root: string): Map<string, string> {
const abs = path.join(root, BASELINE_ROOT);
const index = new Map<string, string>();
for (const key of listEngineKeys(abs)) index.set(key, path.join(abs, key));
return index;
}
function loadManifest(root: string): Manifest | undefined {
const p = path.join(root, MANIFEST_PATH);
if (!fs.existsSync(p)) return undefined;
try {
return JSON.parse(fs.readFileSync(p, 'utf8')) as Manifest;
} catch {
return undefined;
}
}
/** Download a CI run's artifact and return the top-level test-results dir holding the shots. */
function downloadRun(runId: string, repo?: string): string {
const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'promote-'));
const repoArgs = repo ? ['--repo', repo] : [];
execFileSync('gh', ['run', 'download', runId, '-D', tmp, ...repoArgs], { stdio: 'inherit' });
// The artifact stores test-results/** — find that dir; its immediate *.png are the shots
// (exclude the nested screenshot-diff/ triptychs).
const stack = [tmp];
while (stack.length) {
const dir = stack.pop()!;
const entries = fs.readdirSync(dir, { withFileTypes: true });
if (path.basename(dir) === 'test-results') return dir;
for (const e of entries) if (e.isDirectory()) stack.push(path.join(dir, e.name));
}
throw new Error(`no test-results/ dir found in the downloaded artifact under ${tmp}`);
}
type Plan = { updated: string[]; added: string[]; unchanged: string[]; removedCandidates: string[] };
function buildPlan(root: string, renderDir: string, manifest?: Manifest): { plan: Plan; apply: () => void } {
const baselines = baselineIndex(root);
const rendered = new Set(listEngineKeys(renderDir));
// Never promote excluded screenshots (e.g. the flaky retinascale fullPage shot).
for (const key of [...rendered]) if (isIgnored(key)) rendered.delete(key);
for (const key of [...baselines.keys()]) if (isIgnored(key)) baselines.delete(key);
const plan: Plan = { updated: [], added: [], unchanged: [], removedCandidates: [] };
const actions: Array<() => void> = [];
for (const key of rendered) {
const src = path.join(renderDir, key);
const existing = baselines.get(key);
if (!existing) {
const dest = path.join(root, BASELINE_ROOT, key);
plan.added.push(key);
actions.push(() => {
fs.mkdirSync(path.dirname(dest), { recursive: true });
fs.copyFileSync(src, dest); // verbatim bytes
});
continue;
}
const d = diffImages(loadPng(existing), loadPng(src));
const floor = floorFor(key);
if (!d.dimsMatch || d.changedRatio > floor.changedRatio) {
plan.updated.push(key);
actions.push(() => fs.copyFileSync(src, existing)); // verbatim bytes, no re-encode → no churn
} else {
plan.unchanged.push(key); // leave the cached file untouched
}
}
// Removal candidates: a manifest-listed baseline this render didn't produce.
for (const [key, abs] of baselines) {
if (rendered.has(key)) continue;
const { engine, name } = splitKey(key);
if (manifest && !manifest.screenshots.some((e) => e.name === name && e.engine === engine)) continue;
plan.removedCandidates.push(key);
actions.push(() => {}); // pruning is opt-in (see main)
void abs;
}
return { plan, apply: () => actions.forEach((a) => a()) };
}
/** Upload the applied updated/added baselines to R2 (skip-if-exists per hash). */
async function uploadApplied(root: string, store: R2Store, keys: string[]): Promise<number> {
let uploaded = 0;
await pool(keys, 8, async (key) => {
const bytes = fs.readFileSync(path.join(root, BASELINE_ROOT, key));
if ((await store.put(hashBytes(bytes), bytes)) === 'uploaded') uploaded++;
});
return uploaded;
}
function parseArgs(argv: string[]): Record<string, string | boolean> {
const out: Record<string, string | boolean> = {};
for (let i = 0; i < argv.length; i++) {
const a = argv[i];
if (a === '--run') out.run = argv[++i];
else if (a === '--from') out.from = argv[++i];
else if (a === '--repo') out.repo = argv[++i];
else if (a === '--prune') out.prune = true;
else if (a === '--dry-run') out.dryRun = true;
}
return out;
}
async function main(): Promise<void> {
const args = parseArgs(process.argv.slice(2));
const root = process.cwd();
if (!args.run && !args.from) {
console.error('usage: promote.ts --run <ci-run-id> [--repo owner/repo] | --from <dir> [--prune] [--dry-run]');
process.exitCode = 2;
return;
}
// Fail fast BEFORE downloading anything: a promote that can't upload would
// otherwise leave the manifest referencing hashes that don't exist in R2.
const store = storeFromEnv();
if (!store) {
console.error(`[promote] R2 read-write credentials required: set ${missingEnv().join(', ')}`);
process.exitCode = 2;
return;
}
// Refuse a pre-migration checkout rather than warn: buildPlan would mark
// most renders "unchanged" (so never uploaded) while writeManifest pins
// their hashes anyway — committing a manifest that references objects R2
// doesn't have. Sync the checkout to a post-migration revision first.
if (!loadManifestV2(root)) {
console.error(`[promote] ${MANIFEST_PATH} is not the R2-backed v2 format — refusing to promote on a pre-migration checkout`);
process.exitCode = 2;
return;
}
// The local tree is a cache — sync it to the committed manifest so the plan
// diffs against exactly what the manifest pins (a stale/absent cache would
// otherwise misreport adds/updates).
const { downloaded, cached, deleted } = await pullBaselines(root, store);
console.log(`[promote] cache synced: downloaded=${downloaded} cached=${cached} deleted=${deleted}`);
const renderDir = args.from ? (args.from as string) : downloadRun(args.run as string, args.repo as string);
const manifest = loadManifest(root);
const { plan, apply } = buildPlan(root, renderDir, manifest);
console.log(
`[promote] updated=${plan.updated.length} added=${plan.added.length} ` +
`unchanged=${plan.unchanged.length} removal-candidates=${plan.removedCandidates.length}`
);
for (const n of plan.updated) console.log(` UPDATE ${n}`);
for (const n of plan.added) console.log(` ADD ${n}`);
for (const n of plan.removedCandidates) console.log(` REMOVE? ${n}${args.prune ? ' (pruning)' : ' (use --prune to delete)'}`);
if (args.dryRun) {
console.log('[promote] dry-run — no files written, nothing uploaded');
return;
}
apply();
if (args.prune) {
const baselines = baselineIndex(root);
// Prune drops the local file (and, below, the manifest entry). The R2
// object is deliberately kept — old commits must still resolve it.
for (const n of plan.removedCandidates) fs.rmSync(baselines.get(n)!, { force: true });
}
// Upload BEFORE regenerating the manifest: a failure here leaves at worst an
// orphaned CAS object, never a committed manifest pointing at a missing hash.
const uploaded = await uploadApplied(root, store, [...plan.updated, ...plan.added]);
console.log(`[promote] uploaded ${uploaded} object(s) to R2`);
// Keep the manifest in lockstep with the baseline tree — a stale manifest silently
// disables removed-screenshot detection for anything added after the last regen.
writeManifest(root);
console.log(`[promote] done — commit the ${MANIFEST_PATH} diff (the only git-visible output)`);
}
if (require.main === module) {
main().catch((e) => {
console.error(`[promote] ${(e as Error).message}`);
process.exitCode = 1;
});
}

View file

@ -99,6 +99,21 @@ export class R2Store {
return 'uploaded';
}
/**
* Download an ARBITRARY key (the R2-hosted baseline manifest) unlike
* get(), not content-addressed, so no hash verification is possible; null
* on 404 so the caller can degrade like the no-credentials path.
*/
async getKey(key: string): Promise<Buffer | null> {
const res = await this.fetchWithRetry(`${this.base}/${key}`, { method: 'GET' });
if (res.status === 404) {
await res.arrayBuffer().catch(() => undefined);
return null;
}
if (res.status !== 200) throw new Error(`GET ${key} → HTTP ${res.status}`);
return Buffer.from(await res.arrayBuffer());
}
/**
* Upload to an ARBITRARY key (the per-run uploads under runs/, consumed by
* the morelli review app) unlike put(), not content-addressed and always

View file

@ -1,31 +1,38 @@
/**
* Sync the local baseline-screenshots/ cache with the R2 CAS bucket.
*
* The committed manifest (screenshot-manifest.json) is the source of truth;
* baseline-screenshots/ is a gitignored local cache materialized from it.
* The R2-HOSTED manifest (baselines/pcbjam/manifest.json, written only by the
* morelli review app + its seed script) is the source of truth. --manifest
* downloads it to the gitignored MANIFEST_PATH; everything downstream (--pull,
* --verify, compare.ts) reads that local copy, so a single fetch pins the
* whole run to one manifest version.
*
* CLI (from tests/):
* tsx tools/screenshots/r2-sync.ts --pull # manifest local tree (CI fetch step + local gate)
* tsx tools/screenshots/r2-sync.ts --push # upload manifest entries missing in R2 (seeding)
* tsx tools/screenshots/r2-sync.ts --manifest # R2 manifest local .baseline-manifest.json
* tsx tools/screenshots/r2-sync.ts --pull # fetched manifest local tree (CI fetch step + local gate)
* tsx tools/screenshots/r2-sync.ts --verify # HEAD every manifest hash, exit 1 on any miss
*
* --pull without credentials (or with a pre-migration manifest) warns and
* exits 0, so secretless callers of the reusable CI workflow (release.yml,
* pcbjam deploy-staging.yml, fork PRs) stay green compare.ts then skips its
* gate for the same reason. With credentials, any 404/corrupt object is
* collected and the run exits 1.
* --manifest and --pull without credentials warn and exit 0 and --manifest
* DELETES a stale local manifest so the gate skips rather than comparing
* against outdated baselines. Secretless callers of the reusable CI workflow
* (release.yml, pcbjam deploy-staging.yml, fork PRs) therefore stay green;
* compare.ts skips its gate when no manifest was fetched. With credentials,
* any 404/corrupt object is collected and the run exits 1.
*
* (--push is gone with the git-manifest era: baseline bytes now enter the CAS
* only via morelli's promote, which copies them from the CI run uploads.)
*/
import * as fs from 'fs';
import * as path from 'path';
import { BASELINE_ROOT, MANIFEST_PATH, MANIFEST_VERSION, listEngineKeys, type Manifest, type ManifestEntry } from './config';
import { BASELINE_ROOT, MANIFEST_PATH, MANIFEST_VERSION, R2_BASELINES_MANIFEST_KEY, listEngineKeys, type Manifest, type ManifestEntry } from './config';
import { R2Store, hashFile, missingEnv, storeFromEnv } from './r2-store';
const CONCURRENCY = 16;
/** Parse the committed manifest if it is the R2-backed v2 format; null for a
* pre-migration (v1/absent/unparsable) manifest. A NEWER version throws old
* tooling silently no-oping on a future format would disable the whole gate. */
export function loadManifestV2(root: string): Manifest | null {
/** Parse the fetched manifest if it is the morelli-era v3 format; null when
* absent/unparsable ( the gate skips). A NEWER version throws old tooling
* silently no-oping on a future format would disable the whole gate. */
export function loadManifest(root: string): Manifest | null {
const p = path.join(root, MANIFEST_PATH);
if (!fs.existsSync(p)) return null;
let m: Manifest;
@ -40,6 +47,25 @@ export function loadManifestV2(root: string): Manifest | null {
return m.version === MANIFEST_VERSION ? m : null;
}
/**
* Download the R2-hosted baseline manifest to MANIFEST_PATH (atomic tmp+rename).
* Returns false when it could not be fetched in which case any stale local
* copy is removed, so downstream steps skip instead of using old baselines.
*/
export async function fetchManifest(root: string, store: R2Store): Promise<boolean> {
const dest = path.join(root, MANIFEST_PATH);
const bytes = await store.getKey(R2_BASELINES_MANIFEST_KEY);
if (!bytes) {
fs.rmSync(dest, { force: true });
console.warn(`[r2-sync] ${R2_BASELINES_MANIFEST_KEY} not found in R2 — no baselines (seed via morelli first)`);
return false;
}
const tmp = `${dest}.tmp-${process.pid}`;
fs.writeFileSync(tmp, bytes);
fs.renameSync(tmp, dest);
return true;
}
/** Run `fn` over `items` with at most `limit` in flight. */
export async function pool<T>(items: T[], limit: number, fn: (item: T) => Promise<void>): Promise<void> {
let i = 0;
@ -60,8 +86,8 @@ export async function pullBaselines(
root: string,
store: R2Store
): Promise<{ downloaded: number; cached: number; deleted: number }> {
const manifest = loadManifestV2(root);
if (!manifest) throw new Error(`no v2 ${MANIFEST_PATH} — nothing to pull`);
const manifest = loadManifest(root);
if (!manifest) throw new Error(`no fetched ${MANIFEST_PATH} — run \`npm run screenshots:fetch-manifest\` first`);
const base = path.join(root, BASELINE_ROOT);
const wanted = new Map<string, ManifestEntry>();
for (const e of manifest.screenshots) wanted.set(`${e.engine}/${e.name}`, e);
@ -99,35 +125,10 @@ export async function pullBaselines(
return { downloaded, cached, deleted };
}
/** Upload every manifest entry's local file to R2 (skipping hashes already present). */
export async function pushBaselines(root: string, store: R2Store): Promise<{ uploaded: number; existing: number }> {
const manifest = loadManifestV2(root);
if (!manifest) throw new Error(`no v2 ${MANIFEST_PATH} — run \`npm run screenshots:manifest\` first`);
const base = path.join(root, BASELINE_ROOT);
let uploaded = 0;
let existing = 0;
const errors: string[] = [];
await pool(manifest.screenshots, CONCURRENCY, async (e) => {
const file = path.join(base, e.engine, e.name);
try {
if (!fs.existsSync(file)) throw new Error('local file missing');
const bytes = fs.readFileSync(file);
const hash = hashFile(file);
if (hash !== e.sha256) throw new Error(`local sha256 ${hash} ≠ manifest — regenerate the manifest`);
if ((await store.put(hash, bytes)) === 'uploaded') uploaded++;
else existing++;
} catch (err) {
errors.push(`${e.engine}/${e.name}: ${(err as Error).message}`);
}
});
if (errors.length) throw new Error(`${errors.length} upload(s) failed:\n ${errors.join('\n ')}`);
return { uploaded, existing };
}
/** HEAD every manifest hash; returns the missing keys. */
export async function verifyBaselines(root: string, store: R2Store): Promise<string[]> {
const manifest = loadManifestV2(root);
if (!manifest) throw new Error(`no v2 ${MANIFEST_PATH}`);
const manifest = loadManifest(root);
if (!manifest) throw new Error(`no fetched ${MANIFEST_PATH} — run \`npm run screenshots:fetch-manifest\` first`);
const missing: string[] = [];
await pool(manifest.screenshots, CONCURRENCY, async (e) => {
if (!(await store.exists(e.sha256))) missing.push(`${e.engine}/${e.name} (${e.sha256})`);
@ -136,20 +137,27 @@ export async function verifyBaselines(root: string, store: R2Store): Promise<str
}
async function main(): Promise<void> {
const mode = process.argv.find((a) => a === '--pull' || a === '--push' || a === '--verify');
const mode = process.argv.find((a) => a === '--manifest' || a === '--pull' || a === '--verify');
if (!mode) {
console.error('usage: r2-sync.ts --pull | --push | --verify');
console.error('usage: r2-sync.ts --manifest | --pull | --verify');
process.exitCode = 2;
return;
}
const root = process.cwd();
if (mode === '--pull' && !loadManifestV2(root)) {
console.log(`[r2-sync] ${MANIFEST_PATH} is not the R2-backed v2 format — nothing to fetch`);
return;
if (mode === '--pull' && !loadManifest(root)) {
console.log(`[r2-sync] no fetched ${MANIFEST_PATH} — skipping pull (run screenshots:fetch-manifest first)`);
return; // exit 0: manifest fetch skipped/failed → the gate skips, never gates on stale data
}
const store = storeFromEnv();
if (!store) {
if (mode === '--manifest') {
// Delete a stale copy so the downstream gate SKIPS instead of
// comparing against whatever manifest a previous run left behind.
fs.rmSync(path.join(root, MANIFEST_PATH), { force: true });
console.log(`[r2-sync] R2 credentials unset (${missingEnv().join(', ')}) — skipping manifest fetch`);
return; // exit 0: secretless CI callers stay green
}
if (mode === '--pull') {
console.log(`[r2-sync] R2 credentials unset (${missingEnv().join(', ')}) — skipping baseline fetch`);
return; // exit 0: secretless CI callers stay green
@ -159,12 +167,14 @@ async function main(): Promise<void> {
return;
}
if (mode === '--pull') {
if (mode === '--manifest') {
if (await fetchManifest(root, store)) {
const manifest = loadManifest(root);
console.log(`[r2-sync] fetched ${MANIFEST_PATH}: ${manifest?.screenshots.length ?? 0} baselines (updated ${manifest?.updatedAt ?? '?'} by ${manifest?.updatedBy ?? '?'})`);
}
} else if (mode === '--pull') {
const { downloaded, cached, deleted } = await pullBaselines(root, store);
console.log(`[r2-sync] pull done: downloaded=${downloaded} cached=${cached} deleted=${deleted}`);
} else if (mode === '--push') {
const { uploaded, existing } = await pushBaselines(root, store);
console.log(`[r2-sync] push done: uploaded=${uploaded} already-present=${existing}`);
} else {
const missing = await verifyBaselines(root, store);
if (missing.length) {