docs: R2 baseline model in TESTING/README/WHATWORKS + promote-screenshots skill

The three docs still described the committed-PNG model (following them would
trip the new resurrection guard); they now document the fetch-from-R2 cache,
the manifest pin, and the credential requirements. The new /promote-screenshots
skill is the runbook for blessing a CI run's render (dry-run first, RW keypair
from tests/.env, commit only the manifest diff).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Istvan Matejcsok 2026-08-19 08:40:34 +02:00
commit 3257ddc966
4 changed files with 65 additions and 14 deletions

View file

@ -0,0 +1,42 @@
---
name: promote-screenshots
description: Promote a CI run's screenshot renders as the new baselines in the R2 bucket (pcbjam-ci-screenshots). Churn-free - only meaningfully-changed images upload; git gets only the tests/screenshot-manifest.json diff, never PNGs. Needs the read-write R2 keypair in tests/.env. Usage - "/promote-screenshots <ci-run-id> [--prune]". (scoped to pcbjam/ - covers the KiCad WASM e2e pipeline in tests/)
---
# promote-screenshots (pcbjam)
Bless a CI run's rendered screenshots as the new baselines. Baselines live in
the private R2 bucket `pcbjam-ci-screenshots` (prod Cloudflare account),
content-addressed as `sha256/<hex>.png`; the committed
`tests/screenshot-manifest.json` pins each `<engine>/<name>` to a hash. **Only
the manifest diff lands in git — never PNGs.**
## Prerequisites
- The READ-WRITE R2 keypair in the gitignored `tests/.env` (auto-loaded by the
tooling, shell env wins; format in `tests/tools/screenshots/README.md`,
values from the team vault). **Never print, echo, or commit these values.**
If `.env` is missing, ask the user to fill it — promote fails fast without it.
- The CI run id: `gh run list --workflow ci-ubicloud.yml` on PCBJam/pcbjam
(mind the active `gh` account — PCBJam repos need `matejcsok-pcb`).
## Steps
1. From `tests/`, dry-run first and show the user the plan:
`npm run screenshots:promote -- --run <ci-run-id> --dry-run`
2. Sanity-check it: a handful of UPDATE/ADD lines for an intentional UI change
is normal; hundreds of UPDATEs means environment drift — stop and confirm
with the user before applying.
3. Apply: same command without `--dry-run`. Add `--prune` only when the user
confirms screenshots were intentionally removed (prune edits the manifest;
R2 objects are never deleted — old commits still resolve).
4. `git status` must show ONLY `tests/screenshot-manifest.json` modified.
Commit that diff; on main it triggers the Discord baseline changelog.
## Never
- Never commit files under `tests/baseline-screenshots/` (the CI manifest
check fails the build if you do) or the `.env`.
- Never promote local (Mac) renders via `--from` — CI's Linux render is the
only source of truth.
- Never hand-edit hashes in the manifest; promote regenerates it.

View file

@ -54,8 +54,8 @@ tests/
│ └── ... │ └── ...
├── logs/ # Test logs (auto-generated) ├── logs/ # Test logs (auto-generated)
├── test-results/{chromium,firefox}/ # Screenshots per engine (auto-generated) ├── test-results/{chromium,firefox}/ # Screenshots per engine (auto-generated)
├── baseline-screenshots/{chromium,firefox}/ # Committed reference screenshots per engine ├── baseline-screenshots/{chromium,firefox}/ # Reference screenshots per engine — gitignored cache of the R2 bucket (npm run screenshots:fetch)
├── screenshot-manifest.json # Authoritative {name, engine} list (npm run screenshots:manifest) ├── screenshot-manifest.json # Committed pin of every baseline: {name, engine, sha256, …} (regenerated by screenshots:promote)
├── apps/ # Built WASM test applications ├── apps/ # Built WASM test applications
│ ├── minimal_test.html # Main test app │ ├── minimal_test.html # Main test app
│ └── standalone/ # Individual component test apps │ └── standalone/ # Individual component test apps

View file

@ -42,13 +42,19 @@ only) — not playwright directly. One spec on one engine:
`toHaveScreenshot`. Raw `page.screenshot`/fs writers must route through `toHaveScreenshot`. Raw `page.screenshot`/fs writers must route through
`shotPath(page, 'name.png')` for the same engine scoping. `shotPath(page, 'name.png')` for the same engine scoping.
- Comparison is offline and per-engine: `npm run screenshots:check` diffs - Comparison is offline and per-engine: `npm run screenshots:check` diffs
`test-results/<engine>/` against the committed baselines in `test-results/<engine>/` against the baselines in
`tests/baseline-screenshots/<engine>/` (+ `3d-regression/`, `gal-regression/`). `tests/baseline-screenshots/<engine>/` (+ the still-committed `3d-regression/`,
`tests/screenshot-manifest.json` (regen: `npm run screenshots:manifest`) is the authoritative `gal-regression/`).
{name, engine} list — CI fails the lint step if it drifts from the baseline tree. - **Baselines live in the private R2 bucket `pcbjam-ci-screenshots`, not git**:
the committed `tests/screenshot-manifest.json` pins each `{name, engine}` to a
sha256, and `baseline-screenshots/` is a gitignored cache — materialize it with
`npm run screenshots:fetch` (needs the R2 credentials; see
`tools/screenshots/README.md`). CI's lint step validates the manifest and fails
if baseline PNGs are ever committed again.
- **CI's Linux render is the source of truth**; baselines are promoted from CI - **CI's Linux render is the source of truth**; baselines are promoted from CI
(`npm run screenshots:promote -- --run <ci-run-id>`). A local (Mac) check shows font/render (`npm run screenshots:promote -- --run <ci-run-id>`, needs the read-write
noise and is not the gate. credentials) — commit only the regenerated manifest diff, never PNGs. A local
(Mac) check shows font/render noise and is not the gate.
- A continuously-animating state (timer, mid-slide) can't be a stable baseline — drop the shot. - A continuously-animating state (timer, mid-slide) can't be a stable baseline — drop the shot.
## Retries ## Retries

View file

@ -390,20 +390,22 @@ cd tests/apps && ./build-test-apps.sh
## Baseline Screenshots ## Baseline Screenshots
The test suite captures raw PNGs during test runs for visual regression testing, The test suite captures raw PNGs during test runs for visual regression testing,
compared offline against committed per-engine baselines (the calibrated gate in compared offline against per-engine baselines (the calibrated gate in
`tools/screenshots/`). `tools/screenshots/`). Baseline PNGs live in the private R2 bucket
`pcbjam-ci-screenshots` (content-addressed by sha256); git commits only the
manifest that pins them.
### Directory Structure ### Directory Structure
``` ```
tests/ tests/
├── baseline-screenshots/ ├── baseline-screenshots/ # GITIGNORED cache of the R2 bucket (npm run screenshots:fetch)
│ ├── chromium/ # Known-good references, Chromium render │ ├── chromium/ # Known-good references, Chromium render
│ └── firefox/ # Known-good references, Firefox render │ └── firefox/ # Known-good references, Firefox render
├── test-results/ ├── test-results/
│ ├── chromium/ # Latest run's captures per engine │ ├── chromium/ # Latest run's captures per engine
│ └── firefox/ │ └── firefox/
└── screenshot-manifest.json # Authoritative {name, engine} list └── screenshot-manifest.json # Committed pin: {name, engine, sha256, …} per baseline
``` ```
Specs write via `stableShot(page, 'name.png')` / `shotPath(page, 'name.png')` Specs write via `stableShot(page, 'name.png')` / `shotPath(page, 'name.png')`
@ -414,6 +416,7 @@ browser, so the same spec on two engines produces two independent captures.
```bash ```bash
cd tests cd tests
npm run screenshots:fetch # materialize the baseline cache from R2 (idempotent)
npm run screenshots:check npm run screenshots:check
``` ```
@ -430,8 +433,8 @@ changed images restage, and the manifest regenerates automatically):
```bash ```bash
cd tests cd tests
npm run screenshots:promote -- --run <ci-run-id> npm run screenshots:promote -- --run <ci-run-id> # needs the read-write R2 keypair (tests/.env)
git commit git commit # only screenshot-manifest.json changes — PNGs are uploaded to R2, never committed
``` ```
### Running Tests with Screenshots ### Running Tests with Screenshots