tests: move screenshot baselines from git to R2 (content-addressed manifest v2)

Baseline PNGs (492, 35 MB) leave git: they now live in the private R2 bucket
pcbjam-ci-screenshots as immutable sha256/<hex>.png objects, pinned by the
committed screenshot-manifest.json (v2: name, engine, sha256, bytes, dims).
tests/baseline-screenshots/ becomes a gitignored cache materialized by the new
`npm run screenshots:fetch` (r2-sync.ts + aws4fetch r2-store.ts).

- promote.ts: requires the RW keypair (shell env or gitignored tests/.env),
  syncs the cache, uploads new hashes BEFORE rewriting the manifest; the
  manifest diff is the only git-visible output. --prune only edits the
  manifest — R2 objects are never deleted, old commits still resolve.
- compare.ts: skips the gate (exit 0, no report.json) when the manifest
  expects baselines but the cache is empty (secretless callers).
- changelog.ts: diffs the manifest between revs and fetches bytes from R2;
  guards against the migration commit (base manifest not v2 → skip).
- gen-manifest --check: v2 schema + resurrection guard (fails if baseline
  PNGs are ever re-committed); credential-free so every caller can gate.
- wasm-build.yml: declares optional read-only S3 secrets, caches + fetches
  baselines before the lint gate; release.yml/deploy-staging.yml pass them.
- screenshot-changelog.yml: triggers on the manifest path instead of PNGs.

All 492 objects are seeded and hash-verified in the bucket; fetch/compare
degrade to a warn-and-skip without credentials.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Istvan Matejcsok 2026-08-18 16:01:46 +02:00
commit 6dfa6ebd15
511 changed files with 3214 additions and 617 deletions

View file

@ -7,8 +7,8 @@ The e2e tests are in /tests, with a README and WHATWORKS md files
Test determinism rules (no blind sleeps/ifs, `stableShot` screenshots, retries:0) are in tests/TESTING.md, enforced by `npm run lint:determinism`.
The e2e tests are separated per feature
Wxwidgets wasm port has hooks for finding positions of UI elements, tests use that
The test screenshots are tracked with git, per engine (tests/baseline-screenshots/{chromium,firefox}/); CI's Linux render is the source of truth (tooling: tests/tools/screenshots/, see its README).
To update baselines, promote a CI run's render (churn-free — only meaningfully-changed images restage): `cd tests && npm run screenshots:promote -- --run <ci-run-id>`, then commit. `npm run screenshots:check` is the local gate; on each main push CI posts a screenshot-diff + runtime-perf report to Discord.
The test screenshot baselines live in a private R2 bucket (content-addressed by sha256), pinned per engine by the committed tests/screenshot-manifest.json; tests/baseline-screenshots/{chromium,firefox}/ is a gitignored local cache — `cd tests && npm run screenshots:fetch` materializes it (needs the R2 credentials in tests/tools/screenshots/README.md). CI's Linux render is the source of truth (tooling: tests/tools/screenshots/, see its README).
To update baselines, promote a CI run's render (churn-free — only meaningfully-changed images re-upload): `cd tests && npm run screenshots:promote -- --run <ci-run-id>` (needs the read-write R2 credentials), then commit the manifest diff — never commit PNGs. `npm run screenshots:check` is the local gate (fetch first); on each main push CI posts a screenshot-diff + runtime-perf report to Discord.
The tests have log files in tests/logs/{wxwidgets/kicad}/{test-name} after each run where the js console and cpp logs are visible
Always check screenshots for validating tests
Run e2e tests from /tests folder: `npm run test:e2e` (full CI project set, one merged playwright.config.ts) or `npm run test:kicad` (firefox shortcut) — not playwright directly. One spec/engine: `npx playwright test --project=kicad-firefox kicad/pcbnew.spec.ts`. Web-app suite: `npm run test:web`.