2026-07-01 18:01:15 +02:00
# Screenshot regression + Discord review tooling
screenshots: R2-hosted manifest becomes the baseline source of truth (morelli cutover)
The committed tests/screenshot-manifest.json is retired. CI now downloads
baselines/pcbjam/manifest.json (manifest v3, written only by the morelli
review app + its seed script) to the gitignored .baseline-manifest.json,
and everything downstream (pull, verify, compare) reads that copy:
- config.ts: MANIFEST_VERSION 3, MANIFEST_PATH .baseline-manifest.json,
R2_BASELINES_MANIFEST_KEY; ManifestEntry grows opaque provenance
- r2-sync.ts: new --manifest mode (atomic fetch; no-creds skip DELETES a
stale copy so the gate skips rather than using old baselines); --push
gone (bytes enter the CAS only via morelli's promote)
- compare.ts: hard-skips when no manifest was fetched — a stale warm
cache can never gate
- wasm-build.yml: fetch-manifest step before the baselines cache; cache
key now hashes the fetched manifest; the gen-manifest --check lint gate
goes with the committed manifest
- deleted: screenshot-manifest.json, promote.ts, changelog.ts,
gen-manifest.ts, screenshot-changelog.yml, promote-screenshots skill
- docs (CLAUDE/README/TESTING/WHATWORKS/tools README): promote flow is
now https://pcbjam-morelli-staging.pcbjam-staging.workers.dev
Validated locally against the real bucket: fetch-manifest (492), cold
pull 492 / warm pull cached=492, no-creds skip chain, compare gate skip.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 11:54:20 +02:00
One comparison engine + a Discord reporter for the e2e screenshots; baseline
updates happen in the morelli review app
(https://pcbjam-morelli-staging.pcbjam-staging.workers.dev, repo
github.com/PCBJam/morelli).
2026-07-01 18:01:15 +02:00
**Source of truth = CI's Linux render.** The dev never authors baselines on the
screenshots: R2-hosted manifest becomes the baseline source of truth (morelli cutover)
The committed tests/screenshot-manifest.json is retired. CI now downloads
baselines/pcbjam/manifest.json (manifest v3, written only by the morelli
review app + its seed script) to the gitignored .baseline-manifest.json,
and everything downstream (pull, verify, compare) reads that copy:
- config.ts: MANIFEST_VERSION 3, MANIFEST_PATH .baseline-manifest.json,
R2_BASELINES_MANIFEST_KEY; ManifestEntry grows opaque provenance
- r2-sync.ts: new --manifest mode (atomic fetch; no-creds skip DELETES a
stale copy so the gate skips rather than using old baselines); --push
gone (bytes enter the CAS only via morelli's promote)
- compare.ts: hard-skips when no manifest was fetched — a stale warm
cache can never gate
- wasm-build.yml: fetch-manifest step before the baselines cache; cache
key now hashes the fetched manifest; the gen-manifest --check lint gate
goes with the committed manifest
- deleted: screenshot-manifest.json, promote.ts, changelog.ts,
gen-manifest.ts, screenshot-changelog.yml, promote-screenshots skill
- docs (CLAUDE/README/TESTING/WHATWORKS/tools README): promote flow is
now https://pcbjam-morelli-staging.pcbjam-staging.workers.dev
Validated locally against the real bucket: fetch-manifest (492), cold
pull 492 / warm pull cached=492, no-creds skip chain, compare gate skip.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 11:54:20 +02:00
Mac (Mac fonts/GL ≠ CI). CI renders on every push and uploads each run's
screenshots to R2 (`runs/pcbjam/<run-id>/` , 30-day retention, upload-run.ts);
when a render change is intentional you * promote * that run's screenshots in
morelli. The environment isn't pinned — if the host's Mesa/fonts drift, the
gate lights up in Discord and you just re-promote (broad + low-intensity
change ⇒ likely drift).
tests: move screenshot baselines from git to R2 (content-addressed manifest v2)
Baseline PNGs (492, 35 MB) leave git: they now live in the private R2 bucket
pcbjam-ci-screenshots as immutable sha256/<hex>.png objects, pinned by the
committed screenshot-manifest.json (v2: name, engine, sha256, bytes, dims).
tests/baseline-screenshots/ becomes a gitignored cache materialized by the new
`npm run screenshots:fetch` (r2-sync.ts + aws4fetch r2-store.ts).
- promote.ts: requires the RW keypair (shell env or gitignored tests/.env),
syncs the cache, uploads new hashes BEFORE rewriting the manifest; the
manifest diff is the only git-visible output. --prune only edits the
manifest — R2 objects are never deleted, old commits still resolve.
- compare.ts: skips the gate (exit 0, no report.json) when the manifest
expects baselines but the cache is empty (secretless callers).
- changelog.ts: diffs the manifest between revs and fetches bytes from R2;
guards against the migration commit (base manifest not v2 → skip).
- gen-manifest --check: v2 schema + resurrection guard (fails if baseline
PNGs are ever re-committed); credential-free so every caller can gate.
- wasm-build.yml: declares optional read-only S3 secrets, caches + fetches
baselines before the lint gate; release.yml/deploy-staging.yml pass them.
- screenshot-changelog.yml: triggers on the manifest path instead of PNGs.
All 492 objects are seeded and hash-verified in the bucket; fetch/compare
degrade to a warn-and-skip without credentials.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-18 16:01:46 +02:00
screenshots: R2-hosted manifest becomes the baseline source of truth (morelli cutover)
The committed tests/screenshot-manifest.json is retired. CI now downloads
baselines/pcbjam/manifest.json (manifest v3, written only by the morelli
review app + its seed script) to the gitignored .baseline-manifest.json,
and everything downstream (pull, verify, compare) reads that copy:
- config.ts: MANIFEST_VERSION 3, MANIFEST_PATH .baseline-manifest.json,
R2_BASELINES_MANIFEST_KEY; ManifestEntry grows opaque provenance
- r2-sync.ts: new --manifest mode (atomic fetch; no-creds skip DELETES a
stale copy so the gate skips rather than using old baselines); --push
gone (bytes enter the CAS only via morelli's promote)
- compare.ts: hard-skips when no manifest was fetched — a stale warm
cache can never gate
- wasm-build.yml: fetch-manifest step before the baselines cache; cache
key now hashes the fetched manifest; the gen-manifest --check lint gate
goes with the committed manifest
- deleted: screenshot-manifest.json, promote.ts, changelog.ts,
gen-manifest.ts, screenshot-changelog.yml, promote-screenshots skill
- docs (CLAUDE/README/TESTING/WHATWORKS/tools README): promote flow is
now https://pcbjam-morelli-staging.pcbjam-staging.workers.dev
Validated locally against the real bucket: fetch-manifest (492), cold
pull 492 / warm pull cached=492, no-creds skip chain, compare gate skip.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 11:54:20 +02:00
**Baselines live entirely in R2, not git.** The PNGs sit in a private
Cloudflare R2 bucket (`pcbjam-ci-screenshots` ), content-addressed as
`sha256/<hex>.png` and immutable; the R2-HOSTED manifest
`baselines/pcbjam/manifest.json` (written only by morelli + its seed script)
pins each `<engine>/<name>` to a hash. The local `baseline-screenshots/` tree
and `.baseline-manifest.json` are gitignored caches —
`npm run screenshots:fetch-manifest && npm run screenshots:fetch` materializes
them. Nothing screenshot-related is committed.
tests: move screenshot baselines from git to R2 (content-addressed manifest v2)
Baseline PNGs (492, 35 MB) leave git: they now live in the private R2 bucket
pcbjam-ci-screenshots as immutable sha256/<hex>.png objects, pinned by the
committed screenshot-manifest.json (v2: name, engine, sha256, bytes, dims).
tests/baseline-screenshots/ becomes a gitignored cache materialized by the new
`npm run screenshots:fetch` (r2-sync.ts + aws4fetch r2-store.ts).
- promote.ts: requires the RW keypair (shell env or gitignored tests/.env),
syncs the cache, uploads new hashes BEFORE rewriting the manifest; the
manifest diff is the only git-visible output. --prune only edits the
manifest — R2 objects are never deleted, old commits still resolve.
- compare.ts: skips the gate (exit 0, no report.json) when the manifest
expects baselines but the cache is empty (secretless callers).
- changelog.ts: diffs the manifest between revs and fetches bytes from R2;
guards against the migration commit (base manifest not v2 → skip).
- gen-manifest --check: v2 schema + resurrection guard (fails if baseline
PNGs are ever re-committed); credential-free so every caller can gate.
- wasm-build.yml: declares optional read-only S3 secrets, caches + fetches
baselines before the lint gate; release.yml/deploy-staging.yml pass them.
- screenshot-changelog.yml: triggers on the manifest path instead of PNGs.
All 492 objects are seeded and hash-verified in the bucket; fetch/compare
degrade to a warn-and-skip without credentials.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-18 16:01:46 +02:00
**Credentials** (S3 API, bucket-scoped, region `auto` ):
```
CI_SCREENSHOTS_S3_ENDPOINT # https://<account-id>.r2.cloudflarestorage.com
CI_SCREENSHOTS_S3_BUCKET # optional, default pcbjam-ci-screenshots
screenshots: R2-hosted manifest becomes the baseline source of truth (morelli cutover)
The committed tests/screenshot-manifest.json is retired. CI now downloads
baselines/pcbjam/manifest.json (manifest v3, written only by the morelli
review app + its seed script) to the gitignored .baseline-manifest.json,
and everything downstream (pull, verify, compare) reads that copy:
- config.ts: MANIFEST_VERSION 3, MANIFEST_PATH .baseline-manifest.json,
R2_BASELINES_MANIFEST_KEY; ManifestEntry grows opaque provenance
- r2-sync.ts: new --manifest mode (atomic fetch; no-creds skip DELETES a
stale copy so the gate skips rather than using old baselines); --push
gone (bytes enter the CAS only via morelli's promote)
- compare.ts: hard-skips when no manifest was fetched — a stale warm
cache can never gate
- wasm-build.yml: fetch-manifest step before the baselines cache; cache
key now hashes the fetched manifest; the gen-manifest --check lint gate
goes with the committed manifest
- deleted: screenshot-manifest.json, promote.ts, changelog.ts,
gen-manifest.ts, screenshot-changelog.yml, promote-screenshots skill
- docs (CLAUDE/README/TESTING/WHATWORKS/tools README): promote flow is
now https://pcbjam-morelli-staging.pcbjam-staging.workers.dev
Validated locally against the real bucket: fetch-manifest (492), cold
pull 492 / warm pull cached=492, no-creds skip chain, compare gate skip.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 11:54:20 +02:00
CI_SCREENSHOTS_S3_ACCESS_KEY_ID # read-only pair for fetch; CI's upload step maps in a write pair
tests: move screenshot baselines from git to R2 (content-addressed manifest v2)
Baseline PNGs (492, 35 MB) leave git: they now live in the private R2 bucket
pcbjam-ci-screenshots as immutable sha256/<hex>.png objects, pinned by the
committed screenshot-manifest.json (v2: name, engine, sha256, bytes, dims).
tests/baseline-screenshots/ becomes a gitignored cache materialized by the new
`npm run screenshots:fetch` (r2-sync.ts + aws4fetch r2-store.ts).
- promote.ts: requires the RW keypair (shell env or gitignored tests/.env),
syncs the cache, uploads new hashes BEFORE rewriting the manifest; the
manifest diff is the only git-visible output. --prune only edits the
manifest — R2 objects are never deleted, old commits still resolve.
- compare.ts: skips the gate (exit 0, no report.json) when the manifest
expects baselines but the cache is empty (secretless callers).
- changelog.ts: diffs the manifest between revs and fetches bytes from R2;
guards against the migration commit (base manifest not v2 → skip).
- gen-manifest --check: v2 schema + resurrection guard (fails if baseline
PNGs are ever re-committed); credential-free so every caller can gate.
- wasm-build.yml: declares optional read-only S3 secrets, caches + fetches
baselines before the lint gate; release.yml/deploy-staging.yml pass them.
- screenshot-changelog.yml: triggers on the manifest path instead of PNGs.
All 492 objects are seeded and hash-verified in the bucket; fetch/compare
degrade to a warn-and-skip without credentials.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-18 16:01:46 +02:00
CI_SCREENSHOTS_S3_SECRET_ACCESS_KEY
```
screenshots: R2-hosted manifest becomes the baseline source of truth (morelli cutover)
The committed tests/screenshot-manifest.json is retired. CI now downloads
baselines/pcbjam/manifest.json (manifest v3, written only by the morelli
review app + its seed script) to the gitignored .baseline-manifest.json,
and everything downstream (pull, verify, compare) reads that copy:
- config.ts: MANIFEST_VERSION 3, MANIFEST_PATH .baseline-manifest.json,
R2_BASELINES_MANIFEST_KEY; ManifestEntry grows opaque provenance
- r2-sync.ts: new --manifest mode (atomic fetch; no-creds skip DELETES a
stale copy so the gate skips rather than using old baselines); --push
gone (bytes enter the CAS only via morelli's promote)
- compare.ts: hard-skips when no manifest was fetched — a stale warm
cache can never gate
- wasm-build.yml: fetch-manifest step before the baselines cache; cache
key now hashes the fetched manifest; the gen-manifest --check lint gate
goes with the committed manifest
- deleted: screenshot-manifest.json, promote.ts, changelog.ts,
gen-manifest.ts, screenshot-changelog.yml, promote-screenshots skill
- docs (CLAUDE/README/TESTING/WHATWORKS/tools README): promote flow is
now https://pcbjam-morelli-staging.pcbjam-staging.workers.dev
Validated locally against the real bucket: fetch-manifest (492), cold
pull 492 / warm pull cached=492, no-creds skip chain, compare gate skip.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 11:54:20 +02:00
CI holds the read-only pair as repo secrets (plus `CI_SCREENSHOTS_S3_WRITE_*`
for the run-upload step); devs get a read pair from the team vault (ask) and
put it in `tests/.env` (gitignored, auto-loaded by r2-store.ts; shell env vars
take precedence):
tests: move screenshot baselines from git to R2 (content-addressed manifest v2)
Baseline PNGs (492, 35 MB) leave git: they now live in the private R2 bucket
pcbjam-ci-screenshots as immutable sha256/<hex>.png objects, pinned by the
committed screenshot-manifest.json (v2: name, engine, sha256, bytes, dims).
tests/baseline-screenshots/ becomes a gitignored cache materialized by the new
`npm run screenshots:fetch` (r2-sync.ts + aws4fetch r2-store.ts).
- promote.ts: requires the RW keypair (shell env or gitignored tests/.env),
syncs the cache, uploads new hashes BEFORE rewriting the manifest; the
manifest diff is the only git-visible output. --prune only edits the
manifest — R2 objects are never deleted, old commits still resolve.
- compare.ts: skips the gate (exit 0, no report.json) when the manifest
expects baselines but the cache is empty (secretless callers).
- changelog.ts: diffs the manifest between revs and fetches bytes from R2;
guards against the migration commit (base manifest not v2 → skip).
- gen-manifest --check: v2 schema + resurrection guard (fails if baseline
PNGs are ever re-committed); credential-free so every caller can gate.
- wasm-build.yml: declares optional read-only S3 secrets, caches + fetches
baselines before the lint gate; release.yml/deploy-staging.yml pass them.
- screenshot-changelog.yml: triggers on the manifest path instead of PNGs.
All 492 objects are seeded and hash-verified in the bucket; fetch/compare
degrade to a warn-and-skip without credentials.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-18 16:01:46 +02:00
```
CI_SCREENSHOTS_S3_ENDPOINT=https://<account-id>.r2.cloudflarestorage.com
screenshots: R2-hosted manifest becomes the baseline source of truth (morelli cutover)
The committed tests/screenshot-manifest.json is retired. CI now downloads
baselines/pcbjam/manifest.json (manifest v3, written only by the morelli
review app + its seed script) to the gitignored .baseline-manifest.json,
and everything downstream (pull, verify, compare) reads that copy:
- config.ts: MANIFEST_VERSION 3, MANIFEST_PATH .baseline-manifest.json,
R2_BASELINES_MANIFEST_KEY; ManifestEntry grows opaque provenance
- r2-sync.ts: new --manifest mode (atomic fetch; no-creds skip DELETES a
stale copy so the gate skips rather than using old baselines); --push
gone (bytes enter the CAS only via morelli's promote)
- compare.ts: hard-skips when no manifest was fetched — a stale warm
cache can never gate
- wasm-build.yml: fetch-manifest step before the baselines cache; cache
key now hashes the fetched manifest; the gen-manifest --check lint gate
goes with the committed manifest
- deleted: screenshot-manifest.json, promote.ts, changelog.ts,
gen-manifest.ts, screenshot-changelog.yml, promote-screenshots skill
- docs (CLAUDE/README/TESTING/WHATWORKS/tools README): promote flow is
now https://pcbjam-morelli-staging.pcbjam-staging.workers.dev
Validated locally against the real bucket: fetch-manifest (492), cold
pull 492 / warm pull cached=492, no-creds skip chain, compare gate skip.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 11:54:20 +02:00
CI_SCREENSHOTS_S3_ACCESS_KEY_ID=<access-key-id>
CI_SCREENSHOTS_S3_SECRET_ACCESS_KEY=<secret>
tests: move screenshot baselines from git to R2 (content-addressed manifest v2)
Baseline PNGs (492, 35 MB) leave git: they now live in the private R2 bucket
pcbjam-ci-screenshots as immutable sha256/<hex>.png objects, pinned by the
committed screenshot-manifest.json (v2: name, engine, sha256, bytes, dims).
tests/baseline-screenshots/ becomes a gitignored cache materialized by the new
`npm run screenshots:fetch` (r2-sync.ts + aws4fetch r2-store.ts).
- promote.ts: requires the RW keypair (shell env or gitignored tests/.env),
syncs the cache, uploads new hashes BEFORE rewriting the manifest; the
manifest diff is the only git-visible output. --prune only edits the
manifest — R2 objects are never deleted, old commits still resolve.
- compare.ts: skips the gate (exit 0, no report.json) when the manifest
expects baselines but the cache is empty (secretless callers).
- changelog.ts: diffs the manifest between revs and fetches bytes from R2;
guards against the migration commit (base manifest not v2 → skip).
- gen-manifest --check: v2 schema + resurrection guard (fails if baseline
PNGs are ever re-committed); credential-free so every caller can gate.
- wasm-build.yml: declares optional read-only S3 secrets, caches + fetches
baselines before the lint gate; release.yml/deploy-staging.yml pass them.
- screenshot-changelog.yml: triggers on the manifest path instead of PNGs.
All 492 objects are seeded and hash-verified in the bucket; fetch/compare
degrade to a warn-and-skip without credentials.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-18 16:01:46 +02:00
```
screenshots: R2-hosted manifest becomes the baseline source of truth (morelli cutover)
The committed tests/screenshot-manifest.json is retired. CI now downloads
baselines/pcbjam/manifest.json (manifest v3, written only by the morelli
review app + its seed script) to the gitignored .baseline-manifest.json,
and everything downstream (pull, verify, compare) reads that copy:
- config.ts: MANIFEST_VERSION 3, MANIFEST_PATH .baseline-manifest.json,
R2_BASELINES_MANIFEST_KEY; ManifestEntry grows opaque provenance
- r2-sync.ts: new --manifest mode (atomic fetch; no-creds skip DELETES a
stale copy so the gate skips rather than using old baselines); --push
gone (bytes enter the CAS only via morelli's promote)
- compare.ts: hard-skips when no manifest was fetched — a stale warm
cache can never gate
- wasm-build.yml: fetch-manifest step before the baselines cache; cache
key now hashes the fetched manifest; the gen-manifest --check lint gate
goes with the committed manifest
- deleted: screenshot-manifest.json, promote.ts, changelog.ts,
gen-manifest.ts, screenshot-changelog.yml, promote-screenshots skill
- docs (CLAUDE/README/TESTING/WHATWORKS/tools README): promote flow is
now https://pcbjam-morelli-staging.pcbjam-staging.workers.dev
Validated locally against the real bucket: fetch-manifest (492), cold
pull 492 / warm pull cached=492, no-creds skip chain, compare gate skip.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 11:54:20 +02:00
Without credentials, the manifest/baseline fetch and the compare gate skip with
a warning (secretless CI callers stay green).
2026-07-01 18:01:15 +02:00
e2e/CI: dual-engine suites, per-engine screenshots, SwiftShader retired, prod web suite, CI-coverage gate
Squash of experiment/ff-big-modules vs main.
Big-module routing removed: native-EH shrank kicad_editor below
SpiderMonkey's x86-64 code budget (runs 29355049705/29356152413 green on
stock Firefox), so BIG_MODULE_SPECS routing and the baseline-only-JIT
crutch are gone — kicad-firefox and kicad-chromium both run the full
suite, with the module compiled the way real users' browsers compile it.
Per-engine screenshots end to end: stableShot/shotPath write
test-results/<engine>/<name>.png; baselines move to
baseline-screenshots/{chromium,firefox}/ and the whole tools/screenshots
pipeline (compare/promote/manifest/spec-map/changelog/Discord) keys on
<engine>/<name>. Previously Firefox and Chromium renders of one spec
overwrote each other and Firefox renders were never actually gated.
Seeded from CI run 29421380806 (92 new firefox baselines, +24 chromium
web-suite shots); manifest generated from the baseline tree.
One merged playwright.config.ts (kicad/asyncify/coroutine/perf as
projects); ~25 dead npm scripts dropped. The web suite is gated in CI for
the first time ever (4 rotted specs fixed, 5 broken lib-bridge specs
triaged as fixme in docs/features/web-e2e-rot/); cheap lint step after
npm ci; last 26 blind-sleep violations fixed.
SwiftShader retired: CI Chromium renders WebGL on ANGLE → Mesa llvmpipe
(--use-gl=angle --use-angle=gl --ignore-gpu-blocklist; the blocklist flag
is mandatory — llvmpipe is blocklisted and WebGL is silently unavailable
without it) in BOTH configs. Under WORKERS=4 congestion SwiftShader
transiently failed the first post-board-load draw and the recovery
cascade ended in a silent permanent Cairo fallback — that engine flip was
the "~1.2% changedRatio both directions" occ-export baseline flake.
Validated 160/160 across two 80-repeat rigs; full analysis in
docs/features/wx-parity-bugs/occ-export-context-eviction.md. Chromium
baselines shift slightly on llvmpipe — promote once from the first green
run. Deflakes the new coverage exposed: presence baselines settle before
capture; presence fixtures declare current file formats; perf gets its
own outputDir so CI evidence survives; occ-export settles the board paint
before the export dialog; menu-item waits (waitForRenderedByLabel before
clickMenuItem) in 4 specs + the TESTING.md rule.
Web suite runs the PROD build, in parallel: webServer becomes backend
`start` + the standalone's e2e:preview (build-preview.mjs: link-wasm →
stash the public/wasm symlink aside during vite build, build-demo.mjs's
move — then vite preview as the persistent server). The wasm middleware
serves /wasm/* in preview and emits COOP/COEP/CORP itself (a pthread
worker script's own response must carry COEP or Chrome kills it with
ERR_BLOCKED_BY_RESPONSE). VITE_* flags bake at build time;
VITE_ALLOW_USER_OVERRIDE joins turbo globalEnv. fullyParallel + default
workers: 5.2m → 1.4m. Determinism fixes the parallel run exposed:
shared-page specs become serial groups; locks.spec grabs alice's exact
item via the new kicadCollabTestSelectByUuid hook (cross-tab "first
footprint" order is not a ysync invariant); quit specs poll page.url()
(quit supersedes its own navigation — NS_BINDING_ABORTED on Firefox).
Suite: 51 passed / 12 skipped / 0 failed in 1.6m.
CI-coverage gate (lint:ci-coverage): every tests/**/*.spec.ts must be
reachable from the npm scripts the workflows invoke — scraped from
.github/workflows/, resolved through package.json, coverage asked from
playwright --list itself. Rules: uncovered-spec + orphan-project (with a
documented LOCAL_ONLY_PROJECTS allowlist). Gating next to
lint:determinism; 138 spec files / 13 projects accounted for.
Product fixes kept from the investigations (reachable on real GPUs too):
wx 7799fd1be5 — paint flags clear before dispatch + Invalidate always
propagates; kicad 3dcfea5e45 — SwiftShader pass-boundary flush +
per-instance font texture + first-frame GL-error drain (GAL recovery
recovers instead of falling back to Cairo) + the user-facing eeschema
switch navigates again under __EMSCRIPTEN__ (project-sync's
FaceRegistered gate had rerouted it into the hidden sync player; caught
by the newly-gated web suite).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018eUxiPApHgGiu9NFyQfhAq
2026-07-17 12:10:40 +02:00
**Everything is per-engine.** Specs write `test-results/<engine>/<name>.png`
(engine derived from the running browser via `stableShot` /`shotPath` ); baselines
live in `baseline-screenshots/<engine>/` ; the canonical key everywhere in this
tooling is `<engine>/<name>` and captions/attachments carry the engine label.
The same spec on chromium + firefox is two independent gated screenshots.
2026-07-01 18:01:15 +02:00
## Files
- `config.ts` — thresholds, baseline dirs, per-engine floors (calibrate!), clustering knobs.
- `image-ops.ts` — PNG load/save, pixelmatch diff (AA-excluded), connected-component boxes, triptych compositing, size-cap resize.
e2e/CI: dual-engine suites, per-engine screenshots, SwiftShader retired, prod web suite, CI-coverage gate
Squash of experiment/ff-big-modules vs main.
Big-module routing removed: native-EH shrank kicad_editor below
SpiderMonkey's x86-64 code budget (runs 29355049705/29356152413 green on
stock Firefox), so BIG_MODULE_SPECS routing and the baseline-only-JIT
crutch are gone — kicad-firefox and kicad-chromium both run the full
suite, with the module compiled the way real users' browsers compile it.
Per-engine screenshots end to end: stableShot/shotPath write
test-results/<engine>/<name>.png; baselines move to
baseline-screenshots/{chromium,firefox}/ and the whole tools/screenshots
pipeline (compare/promote/manifest/spec-map/changelog/Discord) keys on
<engine>/<name>. Previously Firefox and Chromium renders of one spec
overwrote each other and Firefox renders were never actually gated.
Seeded from CI run 29421380806 (92 new firefox baselines, +24 chromium
web-suite shots); manifest generated from the baseline tree.
One merged playwright.config.ts (kicad/asyncify/coroutine/perf as
projects); ~25 dead npm scripts dropped. The web suite is gated in CI for
the first time ever (4 rotted specs fixed, 5 broken lib-bridge specs
triaged as fixme in docs/features/web-e2e-rot/); cheap lint step after
npm ci; last 26 blind-sleep violations fixed.
SwiftShader retired: CI Chromium renders WebGL on ANGLE → Mesa llvmpipe
(--use-gl=angle --use-angle=gl --ignore-gpu-blocklist; the blocklist flag
is mandatory — llvmpipe is blocklisted and WebGL is silently unavailable
without it) in BOTH configs. Under WORKERS=4 congestion SwiftShader
transiently failed the first post-board-load draw and the recovery
cascade ended in a silent permanent Cairo fallback — that engine flip was
the "~1.2% changedRatio both directions" occ-export baseline flake.
Validated 160/160 across two 80-repeat rigs; full analysis in
docs/features/wx-parity-bugs/occ-export-context-eviction.md. Chromium
baselines shift slightly on llvmpipe — promote once from the first green
run. Deflakes the new coverage exposed: presence baselines settle before
capture; presence fixtures declare current file formats; perf gets its
own outputDir so CI evidence survives; occ-export settles the board paint
before the export dialog; menu-item waits (waitForRenderedByLabel before
clickMenuItem) in 4 specs + the TESTING.md rule.
Web suite runs the PROD build, in parallel: webServer becomes backend
`start` + the standalone's e2e:preview (build-preview.mjs: link-wasm →
stash the public/wasm symlink aside during vite build, build-demo.mjs's
move — then vite preview as the persistent server). The wasm middleware
serves /wasm/* in preview and emits COOP/COEP/CORP itself (a pthread
worker script's own response must carry COEP or Chrome kills it with
ERR_BLOCKED_BY_RESPONSE). VITE_* flags bake at build time;
VITE_ALLOW_USER_OVERRIDE joins turbo globalEnv. fullyParallel + default
workers: 5.2m → 1.4m. Determinism fixes the parallel run exposed:
shared-page specs become serial groups; locks.spec grabs alice's exact
item via the new kicadCollabTestSelectByUuid hook (cross-tab "first
footprint" order is not a ysync invariant); quit specs poll page.url()
(quit supersedes its own navigation — NS_BINDING_ABORTED on Firefox).
Suite: 51 passed / 12 skipped / 0 failed in 1.6m.
CI-coverage gate (lint:ci-coverage): every tests/**/*.spec.ts must be
reachable from the npm scripts the workflows invoke — scraped from
.github/workflows/, resolved through package.json, coverage asked from
playwright --list itself. Rules: uncovered-spec + orphan-project (with a
documented LOCAL_ONLY_PROJECTS allowlist). Gating next to
lint:determinism; 138 spec files / 13 projects accounted for.
Product fixes kept from the investigations (reachable on real GPUs too):
wx 7799fd1be5 — paint flags clear before dispatch + Invalidate always
propagates; kicad 3dcfea5e45 — SwiftShader pass-boundary flush +
per-instance font texture + first-frame GL-error drain (GAL recovery
recovers instead of falling back to Cairo) + the user-facing eeschema
switch navigates again under __EMSCRIPTEN__ (project-sync's
FaceRegistered gate had rerouted it into the hidden sync player; caught
by the newly-gated web suite).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018eUxiPApHgGiu9NFyQfhAq
2026-07-17 12:10:40 +02:00
- `compare.ts` — the comparison engine: classify per-engine baselines vs `test-results/<engine>/` → `test-results/screenshot-diff/report.json` + triptych/heatmap PNGs. `--pair` diffs two files.
2026-07-01 18:01:15 +02:00
- `perf-report.ts` — renders the track-only runtime-perf table (loadMs/openMs/FPS) with Δ vs the previous main run (fetched via `gh` ).
- `post-discord.ts` — the always-on CI-on-main report: SHA + e2e status + perf table, then screenshot triptychs (batched, size-capped, flood-collapsed).
- `noise.ts` — calibration: diff two identical-input renders → per-engine noise floor.
tests: move screenshot baselines from git to R2 (content-addressed manifest v2)
Baseline PNGs (492, 35 MB) leave git: they now live in the private R2 bucket
pcbjam-ci-screenshots as immutable sha256/<hex>.png objects, pinned by the
committed screenshot-manifest.json (v2: name, engine, sha256, bytes, dims).
tests/baseline-screenshots/ becomes a gitignored cache materialized by the new
`npm run screenshots:fetch` (r2-sync.ts + aws4fetch r2-store.ts).
- promote.ts: requires the RW keypair (shell env or gitignored tests/.env),
syncs the cache, uploads new hashes BEFORE rewriting the manifest; the
manifest diff is the only git-visible output. --prune only edits the
manifest — R2 objects are never deleted, old commits still resolve.
- compare.ts: skips the gate (exit 0, no report.json) when the manifest
expects baselines but the cache is empty (secretless callers).
- changelog.ts: diffs the manifest between revs and fetches bytes from R2;
guards against the migration commit (base manifest not v2 → skip).
- gen-manifest --check: v2 schema + resurrection guard (fails if baseline
PNGs are ever re-committed); credential-free so every caller can gate.
- wasm-build.yml: declares optional read-only S3 secrets, caches + fetches
baselines before the lint gate; release.yml/deploy-staging.yml pass them.
- screenshot-changelog.yml: triggers on the manifest path instead of PNGs.
All 492 objects are seeded and hash-verified in the bucket; fetch/compare
degrade to a warn-and-skip without credentials.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-18 16:01:46 +02:00
- `r2-store.ts` — minimal aws4fetch S3 client for the CAS bucket (get/put/exists by hash, downloads integrity-checked).
screenshots: R2-hosted manifest becomes the baseline source of truth (morelli cutover)
The committed tests/screenshot-manifest.json is retired. CI now downloads
baselines/pcbjam/manifest.json (manifest v3, written only by the morelli
review app + its seed script) to the gitignored .baseline-manifest.json,
and everything downstream (pull, verify, compare) reads that copy:
- config.ts: MANIFEST_VERSION 3, MANIFEST_PATH .baseline-manifest.json,
R2_BASELINES_MANIFEST_KEY; ManifestEntry grows opaque provenance
- r2-sync.ts: new --manifest mode (atomic fetch; no-creds skip DELETES a
stale copy so the gate skips rather than using old baselines); --push
gone (bytes enter the CAS only via morelli's promote)
- compare.ts: hard-skips when no manifest was fetched — a stale warm
cache can never gate
- wasm-build.yml: fetch-manifest step before the baselines cache; cache
key now hashes the fetched manifest; the gen-manifest --check lint gate
goes with the committed manifest
- deleted: screenshot-manifest.json, promote.ts, changelog.ts,
gen-manifest.ts, screenshot-changelog.yml, promote-screenshots skill
- docs (CLAUDE/README/TESTING/WHATWORKS/tools README): promote flow is
now https://pcbjam-morelli-staging.pcbjam-staging.workers.dev
Validated locally against the real bucket: fetch-manifest (492), cold
pull 492 / warm pull cached=492, no-creds skip chain, compare gate skip.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 11:54:20 +02:00
- `r2-sync.ts` — cache sync: `--manifest` downloads the R2-hosted baseline manifest, `--pull` materializes `baseline-screenshots/` from it (idempotent, deletes unlisted files), `--verify` HEADs every hash.
- `upload-run.ts` — CI-only: upload the run's renders + meta.json to `runs/pcbjam/<run-id>/` for morelli (needs the write pair; no-ops without credentials).
e2e/CI: dual-engine suites, per-engine screenshots, SwiftShader retired, prod web suite, CI-coverage gate
Squash of experiment/ff-big-modules vs main.
Big-module routing removed: native-EH shrank kicad_editor below
SpiderMonkey's x86-64 code budget (runs 29355049705/29356152413 green on
stock Firefox), so BIG_MODULE_SPECS routing and the baseline-only-JIT
crutch are gone — kicad-firefox and kicad-chromium both run the full
suite, with the module compiled the way real users' browsers compile it.
Per-engine screenshots end to end: stableShot/shotPath write
test-results/<engine>/<name>.png; baselines move to
baseline-screenshots/{chromium,firefox}/ and the whole tools/screenshots
pipeline (compare/promote/manifest/spec-map/changelog/Discord) keys on
<engine>/<name>. Previously Firefox and Chromium renders of one spec
overwrote each other and Firefox renders were never actually gated.
Seeded from CI run 29421380806 (92 new firefox baselines, +24 chromium
web-suite shots); manifest generated from the baseline tree.
One merged playwright.config.ts (kicad/asyncify/coroutine/perf as
projects); ~25 dead npm scripts dropped. The web suite is gated in CI for
the first time ever (4 rotted specs fixed, 5 broken lib-bridge specs
triaged as fixme in docs/features/web-e2e-rot/); cheap lint step after
npm ci; last 26 blind-sleep violations fixed.
SwiftShader retired: CI Chromium renders WebGL on ANGLE → Mesa llvmpipe
(--use-gl=angle --use-angle=gl --ignore-gpu-blocklist; the blocklist flag
is mandatory — llvmpipe is blocklisted and WebGL is silently unavailable
without it) in BOTH configs. Under WORKERS=4 congestion SwiftShader
transiently failed the first post-board-load draw and the recovery
cascade ended in a silent permanent Cairo fallback — that engine flip was
the "~1.2% changedRatio both directions" occ-export baseline flake.
Validated 160/160 across two 80-repeat rigs; full analysis in
docs/features/wx-parity-bugs/occ-export-context-eviction.md. Chromium
baselines shift slightly on llvmpipe — promote once from the first green
run. Deflakes the new coverage exposed: presence baselines settle before
capture; presence fixtures declare current file formats; perf gets its
own outputDir so CI evidence survives; occ-export settles the board paint
before the export dialog; menu-item waits (waitForRenderedByLabel before
clickMenuItem) in 4 specs + the TESTING.md rule.
Web suite runs the PROD build, in parallel: webServer becomes backend
`start` + the standalone's e2e:preview (build-preview.mjs: link-wasm →
stash the public/wasm symlink aside during vite build, build-demo.mjs's
move — then vite preview as the persistent server). The wasm middleware
serves /wasm/* in preview and emits COOP/COEP/CORP itself (a pthread
worker script's own response must carry COEP or Chrome kills it with
ERR_BLOCKED_BY_RESPONSE). VITE_* flags bake at build time;
VITE_ALLOW_USER_OVERRIDE joins turbo globalEnv. fullyParallel + default
workers: 5.2m → 1.4m. Determinism fixes the parallel run exposed:
shared-page specs become serial groups; locks.spec grabs alice's exact
item via the new kicadCollabTestSelectByUuid hook (cross-tab "first
footprint" order is not a ysync invariant); quit specs poll page.url()
(quit supersedes its own navigation — NS_BINDING_ABORTED on Firefox).
Suite: 51 passed / 12 skipped / 0 failed in 1.6m.
CI-coverage gate (lint:ci-coverage): every tests/**/*.spec.ts must be
reachable from the npm scripts the workflows invoke — scraped from
.github/workflows/, resolved through package.json, coverage asked from
playwright --list itself. Rules: uncovered-spec + orphan-project (with a
documented LOCAL_ONLY_PROJECTS allowlist). Gating next to
lint:determinism; 138 spec files / 13 projects accounted for.
Product fixes kept from the investigations (reachable on real GPUs too):
wx 7799fd1be5 — paint flags clear before dispatch + Invalidate always
propagates; kicad 3dcfea5e45 — SwiftShader pass-boundary flush +
per-instance font texture + first-frame GL-error drain (GAL recovery
recovers instead of falling back to Cairo) + the user-facing eeschema
switch navigates again under __EMSCRIPTEN__ (project-sync's
FaceRegistered gate had rerouted it into the hidden sync player; caught
by the newly-gated web suite).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018eUxiPApHgGiu9NFyQfhAq
2026-07-17 12:10:40 +02:00
- `spec-map.ts` — best-effort screenshot-name → spec-file attribution for captions (scans `stableShot` /`shotPath` literals).
2026-07-01 18:01:15 +02:00
## npm scripts (run from `tests/`)
```
screenshots: R2-hosted manifest becomes the baseline source of truth (morelli cutover)
The committed tests/screenshot-manifest.json is retired. CI now downloads
baselines/pcbjam/manifest.json (manifest v3, written only by the morelli
review app + its seed script) to the gitignored .baseline-manifest.json,
and everything downstream (pull, verify, compare) reads that copy:
- config.ts: MANIFEST_VERSION 3, MANIFEST_PATH .baseline-manifest.json,
R2_BASELINES_MANIFEST_KEY; ManifestEntry grows opaque provenance
- r2-sync.ts: new --manifest mode (atomic fetch; no-creds skip DELETES a
stale copy so the gate skips rather than using old baselines); --push
gone (bytes enter the CAS only via morelli's promote)
- compare.ts: hard-skips when no manifest was fetched — a stale warm
cache can never gate
- wasm-build.yml: fetch-manifest step before the baselines cache; cache
key now hashes the fetched manifest; the gen-manifest --check lint gate
goes with the committed manifest
- deleted: screenshot-manifest.json, promote.ts, changelog.ts,
gen-manifest.ts, screenshot-changelog.yml, promote-screenshots skill
- docs (CLAUDE/README/TESTING/WHATWORKS/tools README): promote flow is
now https://pcbjam-morelli-staging.pcbjam-staging.workers.dev
Validated locally against the real bucket: fetch-manifest (492), cold
pull 492 / warm pull cached=492, no-creds skip chain, compare gate skip.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 11:54:20 +02:00
npm run screenshots:fetch-manifest # download the R2-hosted baseline manifest (needs read creds)
npm run screenshots:fetch # materialize the baseline cache from R2 (run before check)
npm run screenshots:check # gate: baselines vs test-results → report.json (exit 0; add --fail-on-change to gate)
npm run screenshots:report -- --e2e pass # post the CI report to Discord (main+push only; needs DISCORD_WEBHOOK_URL)
npm run screenshots:noise -- run1/ run2/ # calibrate floors
npm run screenshots:upload-run -- --e2e pass # CI-only: upload the run's renders for morelli
2026-07-01 18:01:15 +02:00
```
screenshots: R2-hosted manifest becomes the baseline source of truth (morelli cutover)
The committed tests/screenshot-manifest.json is retired. CI now downloads
baselines/pcbjam/manifest.json (manifest v3, written only by the morelli
review app + its seed script) to the gitignored .baseline-manifest.json,
and everything downstream (pull, verify, compare) reads that copy:
- config.ts: MANIFEST_VERSION 3, MANIFEST_PATH .baseline-manifest.json,
R2_BASELINES_MANIFEST_KEY; ManifestEntry grows opaque provenance
- r2-sync.ts: new --manifest mode (atomic fetch; no-creds skip DELETES a
stale copy so the gate skips rather than using old baselines); --push
gone (bytes enter the CAS only via morelli's promote)
- compare.ts: hard-skips when no manifest was fetched — a stale warm
cache can never gate
- wasm-build.yml: fetch-manifest step before the baselines cache; cache
key now hashes the fetched manifest; the gen-manifest --check lint gate
goes with the committed manifest
- deleted: screenshot-manifest.json, promote.ts, changelog.ts,
gen-manifest.ts, screenshot-changelog.yml, promote-screenshots skill
- docs (CLAUDE/README/TESTING/WHATWORKS/tools README): promote flow is
now https://pcbjam-morelli-staging.pcbjam-staging.workers.dev
Validated locally against the real bucket: fetch-manifest (492), cold
pull 492 / warm pull cached=492, no-creds skip chain, compare gate skip.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 11:54:20 +02:00
Baseline promotion (single or bulk) happens in morelli — pick the run, review
the diffs, Promote. It copies verbatim bytes into the CAS, updates the R2
manifest atomically (with provenance: which run/branch/user), and snapshots the
previous manifest for revert.
2026-07-01 18:01:15 +02:00
## Activation checklist
2026-08-19 11:57:50 +02:00
- [x] Baseline manifest migrated to R2 (`baselines/pcbjam/manifest.json` , {name, engine} authoritative — written by morelli).
2026-07-01 18:01:15 +02:00
- [x] `scale:'device'` →`'css'` normalized (no-op at CI's DSF=1).
1. Add the `DISCORD_WEBHOOK_URL` repo secret — until then everything is inert.
2. Calibrate: run the suite twice in CI, `screenshots:noise` the two dirs, set `FLOORS` in `config.ts` .
3. First re-baseline: `promote` a clean CI run's render, commit (expect a big, one-time chrome-font diff vs the Mac baselines).
4. Delete the old `scripts/{compare,update-baseline}-screenshots.sh` .
5. Once floors are proven stable, flip the gate to `--fail-on-change` .
