tests: move screenshot baselines from git to R2 (content-addressed manifest v2)
Baseline PNGs (492, 35 MB) leave git: they now live in the private R2 bucket pcbjam-ci-screenshots as immutable sha256/<hex>.png objects, pinned by the committed screenshot-manifest.json (v2: name, engine, sha256, bytes, dims). tests/baseline-screenshots/ becomes a gitignored cache materialized by the new `npm run screenshots:fetch` (r2-sync.ts + aws4fetch r2-store.ts). - promote.ts: requires the RW keypair (shell env or gitignored tests/.env), syncs the cache, uploads new hashes BEFORE rewriting the manifest; the manifest diff is the only git-visible output. --prune only edits the manifest — R2 objects are never deleted, old commits still resolve. - compare.ts: skips the gate (exit 0, no report.json) when the manifest expects baselines but the cache is empty (secretless callers). - changelog.ts: diffs the manifest between revs and fetches bytes from R2; guards against the migration commit (base manifest not v2 → skip). - gen-manifest --check: v2 schema + resurrection guard (fails if baseline PNGs are ever re-committed); credential-free so every caller can gate. - wasm-build.yml: declares optional read-only S3 secrets, caches + fetches baselines before the lint gate; release.yml/deploy-staging.yml pass them. - screenshot-changelog.yml: triggers on the manifest path instead of PNGs. All 492 objects are seeded and hash-verified in the bucket; fetch/compare degrade to a warn-and-skip without credentials. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
parent
fe57caeb92
commit
6dfa6ebd15
511 changed files with 3214 additions and 617 deletions
|
|
@ -7,8 +7,8 @@ The e2e tests are in /tests, with a README and WHATWORKS md files
|
|||
Test determinism rules (no blind sleeps/ifs, `stableShot` screenshots, retries:0) are in tests/TESTING.md, enforced by `npm run lint:determinism`.
|
||||
The e2e tests are separated per feature
|
||||
Wxwidgets wasm port has hooks for finding positions of UI elements, tests use that
|
||||
The test screenshots are tracked with git, per engine (tests/baseline-screenshots/{chromium,firefox}/); CI's Linux render is the source of truth (tooling: tests/tools/screenshots/, see its README).
|
||||
To update baselines, promote a CI run's render (churn-free — only meaningfully-changed images restage): `cd tests && npm run screenshots:promote -- --run <ci-run-id>`, then commit. `npm run screenshots:check` is the local gate; on each main push CI posts a screenshot-diff + runtime-perf report to Discord.
|
||||
The test screenshot baselines live in a private R2 bucket (content-addressed by sha256), pinned per engine by the committed tests/screenshot-manifest.json; tests/baseline-screenshots/{chromium,firefox}/ is a gitignored local cache — `cd tests && npm run screenshots:fetch` materializes it (needs the R2 credentials in tests/tools/screenshots/README.md). CI's Linux render is the source of truth (tooling: tests/tools/screenshots/, see its README).
|
||||
To update baselines, promote a CI run's render (churn-free — only meaningfully-changed images re-upload): `cd tests && npm run screenshots:promote -- --run <ci-run-id>` (needs the read-write R2 credentials), then commit the manifest diff — never commit PNGs. `npm run screenshots:check` is the local gate (fetch first); on each main push CI posts a screenshot-diff + runtime-perf report to Discord.
|
||||
The tests have log files in tests/logs/{wxwidgets/kicad}/{test-name} after each run where the js console and cpp logs are visible
|
||||
Always check screenshots for validating tests
|
||||
Run e2e tests from /tests folder: `npm run test:e2e` (full CI project set, one merged playwright.config.ts) or `npm run test:kicad` (firefox shortcut) — not playwright directly. One spec/engine: `npx playwright test --project=kicad-firefox kicad/pcbnew.spec.ts`. Web-app suite: `npm run test:web`.
|
||||
|
|
|
|||
Loading…
Reference in a new issue