docs: R2 baseline model in TESTING/README/WHATWORKS + promote-screenshots skill
The three docs still described the committed-PNG model (following them would trip the new resurrection guard); they now document the fetch-from-R2 cache, the manifest pin, and the credential requirements. The new /promote-screenshots skill is the runbook for blessing a CI run's render (dry-run first, RW keypair from tests/.env, commit only the manifest diff). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
parent
6dfa6ebd15
commit
3257ddc966
4 changed files with 65 additions and 14 deletions
42
.claude/skills/promote-screenshots/SKILL.md
Normal file
42
.claude/skills/promote-screenshots/SKILL.md
Normal file
|
|
@ -0,0 +1,42 @@
|
||||||
|
---
|
||||||
|
name: promote-screenshots
|
||||||
|
description: Promote a CI run's screenshot renders as the new baselines in the R2 bucket (pcbjam-ci-screenshots). Churn-free - only meaningfully-changed images upload; git gets only the tests/screenshot-manifest.json diff, never PNGs. Needs the read-write R2 keypair in tests/.env. Usage - "/promote-screenshots <ci-run-id> [--prune]". (scoped to pcbjam/ - covers the KiCad WASM e2e pipeline in tests/)
|
||||||
|
---
|
||||||
|
|
||||||
|
# promote-screenshots (pcbjam)
|
||||||
|
|
||||||
|
Bless a CI run's rendered screenshots as the new baselines. Baselines live in
|
||||||
|
the private R2 bucket `pcbjam-ci-screenshots` (prod Cloudflare account),
|
||||||
|
content-addressed as `sha256/<hex>.png`; the committed
|
||||||
|
`tests/screenshot-manifest.json` pins each `<engine>/<name>` to a hash. **Only
|
||||||
|
the manifest diff lands in git — never PNGs.**
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
- The READ-WRITE R2 keypair in the gitignored `tests/.env` (auto-loaded by the
|
||||||
|
tooling, shell env wins; format in `tests/tools/screenshots/README.md`,
|
||||||
|
values from the team vault). **Never print, echo, or commit these values.**
|
||||||
|
If `.env` is missing, ask the user to fill it — promote fails fast without it.
|
||||||
|
- The CI run id: `gh run list --workflow ci-ubicloud.yml` on PCBJam/pcbjam
|
||||||
|
(mind the active `gh` account — PCBJam repos need `matejcsok-pcb`).
|
||||||
|
|
||||||
|
## Steps
|
||||||
|
|
||||||
|
1. From `tests/`, dry-run first and show the user the plan:
|
||||||
|
`npm run screenshots:promote -- --run <ci-run-id> --dry-run`
|
||||||
|
2. Sanity-check it: a handful of UPDATE/ADD lines for an intentional UI change
|
||||||
|
is normal; hundreds of UPDATEs means environment drift — stop and confirm
|
||||||
|
with the user before applying.
|
||||||
|
3. Apply: same command without `--dry-run`. Add `--prune` only when the user
|
||||||
|
confirms screenshots were intentionally removed (prune edits the manifest;
|
||||||
|
R2 objects are never deleted — old commits still resolve).
|
||||||
|
4. `git status` must show ONLY `tests/screenshot-manifest.json` modified.
|
||||||
|
Commit that diff; on main it triggers the Discord baseline changelog.
|
||||||
|
|
||||||
|
## Never
|
||||||
|
|
||||||
|
- Never commit files under `tests/baseline-screenshots/` (the CI manifest
|
||||||
|
check fails the build if you do) or the `.env`.
|
||||||
|
- Never promote local (Mac) renders via `--from` — CI's Linux render is the
|
||||||
|
only source of truth.
|
||||||
|
- Never hand-edit hashes in the manifest; promote regenerates it.
|
||||||
|
|
@ -54,8 +54,8 @@ tests/
|
||||||
│ └── ...
|
│ └── ...
|
||||||
├── logs/ # Test logs (auto-generated)
|
├── logs/ # Test logs (auto-generated)
|
||||||
├── test-results/{chromium,firefox}/ # Screenshots per engine (auto-generated)
|
├── test-results/{chromium,firefox}/ # Screenshots per engine (auto-generated)
|
||||||
├── baseline-screenshots/{chromium,firefox}/ # Committed reference screenshots per engine
|
├── baseline-screenshots/{chromium,firefox}/ # Reference screenshots per engine — gitignored cache of the R2 bucket (npm run screenshots:fetch)
|
||||||
├── screenshot-manifest.json # Authoritative {name, engine} list (npm run screenshots:manifest)
|
├── screenshot-manifest.json # Committed pin of every baseline: {name, engine, sha256, …} (regenerated by screenshots:promote)
|
||||||
├── apps/ # Built WASM test applications
|
├── apps/ # Built WASM test applications
|
||||||
│ ├── minimal_test.html # Main test app
|
│ ├── minimal_test.html # Main test app
|
||||||
│ └── standalone/ # Individual component test apps
|
│ └── standalone/ # Individual component test apps
|
||||||
|
|
|
||||||
|
|
@ -42,13 +42,19 @@ only) — not playwright directly. One spec on one engine:
|
||||||
`toHaveScreenshot`. Raw `page.screenshot`/fs writers must route through
|
`toHaveScreenshot`. Raw `page.screenshot`/fs writers must route through
|
||||||
`shotPath(page, 'name.png')` for the same engine scoping.
|
`shotPath(page, 'name.png')` for the same engine scoping.
|
||||||
- Comparison is offline and per-engine: `npm run screenshots:check` diffs
|
- Comparison is offline and per-engine: `npm run screenshots:check` diffs
|
||||||
`test-results/<engine>/` against the committed baselines in
|
`test-results/<engine>/` against the baselines in
|
||||||
`tests/baseline-screenshots/<engine>/` (+ `3d-regression/`, `gal-regression/`).
|
`tests/baseline-screenshots/<engine>/` (+ the still-committed `3d-regression/`,
|
||||||
`tests/screenshot-manifest.json` (regen: `npm run screenshots:manifest`) is the authoritative
|
`gal-regression/`).
|
||||||
{name, engine} list — CI fails the lint step if it drifts from the baseline tree.
|
- **Baselines live in the private R2 bucket `pcbjam-ci-screenshots`, not git**:
|
||||||
|
the committed `tests/screenshot-manifest.json` pins each `{name, engine}` to a
|
||||||
|
sha256, and `baseline-screenshots/` is a gitignored cache — materialize it with
|
||||||
|
`npm run screenshots:fetch` (needs the R2 credentials; see
|
||||||
|
`tools/screenshots/README.md`). CI's lint step validates the manifest and fails
|
||||||
|
if baseline PNGs are ever committed again.
|
||||||
- **CI's Linux render is the source of truth**; baselines are promoted from CI
|
- **CI's Linux render is the source of truth**; baselines are promoted from CI
|
||||||
(`npm run screenshots:promote -- --run <ci-run-id>`). A local (Mac) check shows font/render
|
(`npm run screenshots:promote -- --run <ci-run-id>`, needs the read-write
|
||||||
noise and is not the gate.
|
credentials) — commit only the regenerated manifest diff, never PNGs. A local
|
||||||
|
(Mac) check shows font/render noise and is not the gate.
|
||||||
- A continuously-animating state (timer, mid-slide) can't be a stable baseline — drop the shot.
|
- A continuously-animating state (timer, mid-slide) can't be a stable baseline — drop the shot.
|
||||||
|
|
||||||
## Retries
|
## Retries
|
||||||
|
|
|
||||||
|
|
@ -390,20 +390,22 @@ cd tests/apps && ./build-test-apps.sh
|
||||||
## Baseline Screenshots
|
## Baseline Screenshots
|
||||||
|
|
||||||
The test suite captures raw PNGs during test runs for visual regression testing,
|
The test suite captures raw PNGs during test runs for visual regression testing,
|
||||||
compared offline against committed per-engine baselines (the calibrated gate in
|
compared offline against per-engine baselines (the calibrated gate in
|
||||||
`tools/screenshots/`).
|
`tools/screenshots/`). Baseline PNGs live in the private R2 bucket
|
||||||
|
`pcbjam-ci-screenshots` (content-addressed by sha256); git commits only the
|
||||||
|
manifest that pins them.
|
||||||
|
|
||||||
### Directory Structure
|
### Directory Structure
|
||||||
|
|
||||||
```
|
```
|
||||||
tests/
|
tests/
|
||||||
├── baseline-screenshots/
|
├── baseline-screenshots/ # GITIGNORED cache of the R2 bucket (npm run screenshots:fetch)
|
||||||
│ ├── chromium/ # Known-good references, Chromium render
|
│ ├── chromium/ # Known-good references, Chromium render
|
||||||
│ └── firefox/ # Known-good references, Firefox render
|
│ └── firefox/ # Known-good references, Firefox render
|
||||||
├── test-results/
|
├── test-results/
|
||||||
│ ├── chromium/ # Latest run's captures per engine
|
│ ├── chromium/ # Latest run's captures per engine
|
||||||
│ └── firefox/
|
│ └── firefox/
|
||||||
└── screenshot-manifest.json # Authoritative {name, engine} list
|
└── screenshot-manifest.json # Committed pin: {name, engine, sha256, …} per baseline
|
||||||
```
|
```
|
||||||
|
|
||||||
Specs write via `stableShot(page, 'name.png')` / `shotPath(page, 'name.png')`
|
Specs write via `stableShot(page, 'name.png')` / `shotPath(page, 'name.png')`
|
||||||
|
|
@ -414,6 +416,7 @@ browser, so the same spec on two engines produces two independent captures.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cd tests
|
cd tests
|
||||||
|
npm run screenshots:fetch # materialize the baseline cache from R2 (idempotent)
|
||||||
npm run screenshots:check
|
npm run screenshots:check
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -430,8 +433,8 @@ changed images restage, and the manifest regenerates automatically):
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cd tests
|
cd tests
|
||||||
npm run screenshots:promote -- --run <ci-run-id>
|
npm run screenshots:promote -- --run <ci-run-id> # needs the read-write R2 keypair (tests/.env)
|
||||||
git commit
|
git commit # only screenshot-manifest.json changes — PNGs are uploaded to R2, never committed
|
||||||
```
|
```
|
||||||
|
|
||||||
### Running Tests with Screenshots
|
### Running Tests with Screenshots
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue