diff --git a/.claude/skills/promote-screenshots/SKILL.md b/.claude/skills/promote-screenshots/SKILL.md new file mode 100644 index 0000000..43a16b9 --- /dev/null +++ b/.claude/skills/promote-screenshots/SKILL.md @@ -0,0 +1,42 @@ +--- +name: promote-screenshots +description: Promote a CI run's screenshot renders as the new baselines in the R2 bucket (pcbjam-ci-screenshots). Churn-free - only meaningfully-changed images upload; git gets only the tests/screenshot-manifest.json diff, never PNGs. Needs the read-write R2 keypair in tests/.env. Usage - "/promote-screenshots [--prune]". (scoped to pcbjam/ - covers the KiCad WASM e2e pipeline in tests/) +--- + +# promote-screenshots (pcbjam) + +Bless a CI run's rendered screenshots as the new baselines. Baselines live in +the private R2 bucket `pcbjam-ci-screenshots` (prod Cloudflare account), +content-addressed as `sha256/.png`; the committed +`tests/screenshot-manifest.json` pins each `/` to a hash. **Only +the manifest diff lands in git — never PNGs.** + +## Prerequisites + +- The READ-WRITE R2 keypair in the gitignored `tests/.env` (auto-loaded by the + tooling, shell env wins; format in `tests/tools/screenshots/README.md`, + values from the team vault). **Never print, echo, or commit these values.** + If `.env` is missing, ask the user to fill it — promote fails fast without it. +- The CI run id: `gh run list --workflow ci-ubicloud.yml` on PCBJam/pcbjam + (mind the active `gh` account — PCBJam repos need `matejcsok-pcb`). + +## Steps + +1. From `tests/`, dry-run first and show the user the plan: + `npm run screenshots:promote -- --run --dry-run` +2. Sanity-check it: a handful of UPDATE/ADD lines for an intentional UI change + is normal; hundreds of UPDATEs means environment drift — stop and confirm + with the user before applying. +3. Apply: same command without `--dry-run`. Add `--prune` only when the user + confirms screenshots were intentionally removed (prune edits the manifest; + R2 objects are never deleted — old commits still resolve). +4. `git status` must show ONLY `tests/screenshot-manifest.json` modified. + Commit that diff; on main it triggers the Discord baseline changelog. + +## Never + +- Never commit files under `tests/baseline-screenshots/` (the CI manifest + check fails the build if you do) or the `.env`. +- Never promote local (Mac) renders via `--from` — CI's Linux render is the + only source of truth. +- Never hand-edit hashes in the manifest; promote regenerates it. diff --git a/tests/README.md b/tests/README.md index c57830b..fc1cb0c 100644 --- a/tests/README.md +++ b/tests/README.md @@ -54,8 +54,8 @@ tests/ │ └── ... ├── logs/ # Test logs (auto-generated) ├── test-results/{chromium,firefox}/ # Screenshots per engine (auto-generated) -├── baseline-screenshots/{chromium,firefox}/ # Committed reference screenshots per engine -├── screenshot-manifest.json # Authoritative {name, engine} list (npm run screenshots:manifest) +├── baseline-screenshots/{chromium,firefox}/ # Reference screenshots per engine — gitignored cache of the R2 bucket (npm run screenshots:fetch) +├── screenshot-manifest.json # Committed pin of every baseline: {name, engine, sha256, …} (regenerated by screenshots:promote) ├── apps/ # Built WASM test applications │ ├── minimal_test.html # Main test app │ └── standalone/ # Individual component test apps diff --git a/tests/TESTING.md b/tests/TESTING.md index 3f59bc4..fb504e7 100644 --- a/tests/TESTING.md +++ b/tests/TESTING.md @@ -42,13 +42,19 @@ only) — not playwright directly. One spec on one engine: `toHaveScreenshot`. Raw `page.screenshot`/fs writers must route through `shotPath(page, 'name.png')` for the same engine scoping. - Comparison is offline and per-engine: `npm run screenshots:check` diffs - `test-results//` against the committed baselines in - `tests/baseline-screenshots//` (+ `3d-regression/`, `gal-regression/`). - `tests/screenshot-manifest.json` (regen: `npm run screenshots:manifest`) is the authoritative - {name, engine} list — CI fails the lint step if it drifts from the baseline tree. + `test-results//` against the baselines in + `tests/baseline-screenshots//` (+ the still-committed `3d-regression/`, + `gal-regression/`). +- **Baselines live in the private R2 bucket `pcbjam-ci-screenshots`, not git**: + the committed `tests/screenshot-manifest.json` pins each `{name, engine}` to a + sha256, and `baseline-screenshots/` is a gitignored cache — materialize it with + `npm run screenshots:fetch` (needs the R2 credentials; see + `tools/screenshots/README.md`). CI's lint step validates the manifest and fails + if baseline PNGs are ever committed again. - **CI's Linux render is the source of truth**; baselines are promoted from CI - (`npm run screenshots:promote -- --run `). A local (Mac) check shows font/render - noise and is not the gate. + (`npm run screenshots:promote -- --run `, needs the read-write + credentials) — commit only the regenerated manifest diff, never PNGs. A local + (Mac) check shows font/render noise and is not the gate. - A continuously-animating state (timer, mid-slide) can't be a stable baseline — drop the shot. ## Retries diff --git a/tests/WHATWORKS.md b/tests/WHATWORKS.md index 25bde61..3e71be1 100644 --- a/tests/WHATWORKS.md +++ b/tests/WHATWORKS.md @@ -390,20 +390,22 @@ cd tests/apps && ./build-test-apps.sh ## Baseline Screenshots The test suite captures raw PNGs during test runs for visual regression testing, -compared offline against committed per-engine baselines (the calibrated gate in -`tools/screenshots/`). +compared offline against per-engine baselines (the calibrated gate in +`tools/screenshots/`). Baseline PNGs live in the private R2 bucket +`pcbjam-ci-screenshots` (content-addressed by sha256); git commits only the +manifest that pins them. ### Directory Structure ``` tests/ -├── baseline-screenshots/ +├── baseline-screenshots/ # GITIGNORED cache of the R2 bucket (npm run screenshots:fetch) │ ├── chromium/ # Known-good references, Chromium render │ └── firefox/ # Known-good references, Firefox render ├── test-results/ │ ├── chromium/ # Latest run's captures per engine │ └── firefox/ -└── screenshot-manifest.json # Authoritative {name, engine} list +└── screenshot-manifest.json # Committed pin: {name, engine, sha256, …} per baseline ``` Specs write via `stableShot(page, 'name.png')` / `shotPath(page, 'name.png')` @@ -414,6 +416,7 @@ browser, so the same spec on two engines produces two independent captures. ```bash cd tests +npm run screenshots:fetch # materialize the baseline cache from R2 (idempotent) npm run screenshots:check ``` @@ -430,8 +433,8 @@ changed images restage, and the manifest regenerates automatically): ```bash cd tests -npm run screenshots:promote -- --run -git commit +npm run screenshots:promote -- --run # needs the read-write R2 keypair (tests/.env) +git commit # only screenshot-manifest.json changes — PNGs are uploaded to R2, never committed ``` ### Running Tests with Screenshots