AI_Assistant/pcbjam
Watch
1
0
Fork
You've already forked pcbjam
 0
Code Issues Pull requests Projects Releases 4 Packages Wiki Activity Actions 2
pcbjam/tests/tools/screenshots
History
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Viktor Vaczi f419a1fedd build+perf: wasm-opt the shipped wasm, and measure real frames in CI 
emcc only runs Binaryen at link -O2+ (link.py: should_run_binaryen_optimizer
returns OPT_LEVEL >= 2) and we link at -O1, so the shipped module had never seen
wasm-opt at all — it kept its entire 19.56 MB name section, ~20% of the editor
(-sJSPI sets ASYNCIFY=2, which suppresses wasm-ld's --strip-debug, leaving
wasm-opt as the only thing that would drop it). Step 8.2 runs it post-link and
in-container, so CI's cached compile phase covers it and the host post-process
stays pure-host.

Default -O2, picked by measuring every level on the same module: -O0 already
captures 27% of the raw win (it is mostly the name section), -O2 costs 23 s and
gives the best frame rate, and -O3/-O4/-Os/-Oz cost 48-132 s for at most 1.5%
more brotli — -O4 is not even smaller than -O3. Targets that already link -O2/-Oz
(occ_service, kicad_tools) are skipped by testing for the target_features
section, which emcc strips whenever it ran the optimizer itself, so there is no
hard-coded target list to drift. Feature flags come from the module's own
target_features section and so cannot diverge from the link.

The perf specs reported requestAnimationFrame ticks as "FPS". That is not a frame
rate: rAF fires on the compositor's schedule whether or not the GAL redrew, and
it read 120/s on a board where the renderer completed zero frames in six seconds.
measureInteractionFps now counts completed GAL frames — runs of draws to the
default framebuffer, exactly one per frame in every AA mode — and drives a pure
middle-drag pan after a zoom-to-fit. Mixing wheel zoom into the drive made the
result depend on where the wheel left the view: +-20% across identical repeats,
against +-2% for pan alone. The report gains a GAL fps column with a regression
flag on the 1x number; rAF is kept so historical runs stay comparable.

CI has no GPU, so its number is a software-rasteriser redraw rate — a regression
signal, not a user-facing frame rate. Method and measurements in the bench report.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016X9eh1s5sTx1o9Em9KBuwR
2026-08-22 12:53:00 +02:00
..
compare-dirs.ts test(3d): npm run 3d:review — triptychs/heatmaps for ALL pairs, not just changed 2026-07-03 15:46:23 +02:00
compare.ts screenshots: R2-hosted manifest becomes the baseline source of truth (morelli cutover) 2026-08-19 14:35:08 +02:00
config.ts screenshots: R2-hosted manifest becomes the baseline source of truth (morelli cutover) 2026-08-19 14:35:08 +02:00
font8x8.ts feat(tests): caption posted screenshots (name + spec); raise drift floor to 0.5% 2026-07-02 12:07:41 +02:00
image-ops.ts feat(tests): caption posted screenshots (name + spec); raise drift floor to 0.5% 2026-07-02 12:07:41 +02:00
noise.ts e2e/CI: dual-engine suites, per-engine screenshots, SwiftShader retired, prod web suite, CI-coverage gate 2026-07-17 12:21:54 +02:00
perf-report.ts build+perf: wasm-opt the shipped wasm, and measure real frames in CI 2026-08-22 12:53:00 +02:00
post-discord.ts screenshots: R2-hosted manifest becomes the baseline source of truth (morelli cutover) 2026-08-19 14:35:08 +02:00
r2-store.ts screenshots: sweep remaining committed-manifest wording 2026-08-19 14:35:08 +02:00
r2-sync.ts screenshots: R2-hosted manifest becomes the baseline source of truth (morelli cutover) 2026-08-19 14:35:08 +02:00
README.md screenshots: sweep remaining committed-manifest wording 2026-08-19 14:35:08 +02:00
spec-map.ts e2e/CI: dual-engine suites, per-engine screenshots, SwiftShader retired, prod web suite, CI-coverage gate 2026-07-17 12:21:54 +02:00
upload-run.ts ci: upload each run's screenshots + meta.json to R2 (runs/pcbjam/<run-id>/) 2026-08-19 14:35:08 +02:00

README.md

Screenshot regression + Discord review tooling

One comparison engine + a Discord reporter for the e2e screenshots; baseline updates happen in the morelli review app (https://pcbjam-morelli-staging.pcbjam-staging.workers.dev, repo github.com/PCBJam/morelli).

Source of truth = CI's Linux render. The dev never authors baselines on the Mac (Mac fonts/GL ≠ CI). CI renders on every push and uploads each run's screenshots to R2 (runs/pcbjam/<run-id>/, 30-day retention, upload-run.ts); when a render change is intentional you promote that run's screenshots in morelli. The environment isn't pinned — if the host's Mesa/fonts drift, the gate lights up in Discord and you just re-promote (broad + low-intensity change ⇒ likely drift).

Baselines live entirely in R2, not git. The PNGs sit in a private Cloudflare R2 bucket (pcbjam-ci-screenshots), content-addressed as sha256/<hex>.png and immutable; the R2-HOSTED manifest baselines/pcbjam/manifest.json (written only by morelli + its seed script) pins each <engine>/<name> to a hash. The local baseline-screenshots/ tree and .baseline-manifest.json are gitignored caches — npm run screenshots:fetch-manifest && npm run screenshots:fetch materializes them. Nothing screenshot-related is committed.

Credentials (S3 API, bucket-scoped, region auto):

CI_SCREENSHOTS_S3_ENDPOINT           # https://<account-id>.r2.cloudflarestorage.com
CI_SCREENSHOTS_S3_BUCKET             # optional, default pcbjam-ci-screenshots
CI_SCREENSHOTS_S3_ACCESS_KEY_ID      # read-only pair for fetch; CI's upload step maps in a write pair
CI_SCREENSHOTS_S3_SECRET_ACCESS_KEY

CI holds the read-only pair as repo secrets (plus CI_SCREENSHOTS_S3_WRITE_* for the run-upload step); devs get a read pair from the team vault (ask) and put it in tests/.env (gitignored, auto-loaded by r2-store.ts; shell env vars take precedence):

CI_SCREENSHOTS_S3_ENDPOINT=https://<account-id>.r2.cloudflarestorage.com
CI_SCREENSHOTS_S3_ACCESS_KEY_ID=<access-key-id>
CI_SCREENSHOTS_S3_SECRET_ACCESS_KEY=<secret>

Without credentials, the manifest/baseline fetch and the compare gate skip with a warning (secretless CI callers stay green).

Everything is per-engine. Specs write test-results/<engine>/<name>.png (engine derived from the running browser via stableShot/shotPath); baselines live in baseline-screenshots/<engine>/; the canonical key everywhere in this tooling is <engine>/<name> and captions/attachments carry the engine label. The same spec on chromium + firefox is two independent gated screenshots.

Files

  • config.ts — thresholds, baseline dirs, per-engine floors (calibrate!), clustering knobs.
  • image-ops.ts — PNG load/save, pixelmatch diff (AA-excluded), connected-component boxes, triptych compositing, size-cap resize.
  • compare.ts — the comparison engine: classify per-engine baselines vs test-results/<engine>/ → test-results/screenshot-diff/report.json + triptych/heatmap PNGs. --pair diffs two files.
  • perf-report.ts — renders the track-only runtime-perf table (loadMs/openMs/FPS) with Δ vs the previous main run (fetched via gh).
  • post-discord.ts — the always-on CI-on-main report: SHA + e2e status + perf table, then screenshot triptychs (batched, size-capped, flood-collapsed).
  • noise.ts — calibration: diff two identical-input renders → per-engine noise floor.
  • r2-store.ts — minimal aws4fetch S3 client for the CAS bucket (get/put/exists by hash, downloads integrity-checked).
  • r2-sync.ts — cache sync: --manifest downloads the R2-hosted baseline manifest, --pull materializes baseline-screenshots/ from it (idempotent, deletes unlisted files), --verify HEADs every hash.
  • upload-run.ts — CI-only: upload the run's renders + meta.json to runs/pcbjam/<run-id>/ for morelli (needs the write pair; no-ops without credentials).
  • spec-map.ts — best-effort screenshot-name → spec-file attribution for captions (scans stableShot/shotPath literals).

npm scripts (run from tests/)

npm run screenshots:fetch-manifest   # download the R2-hosted baseline manifest (needs read creds)
npm run screenshots:fetch            # materialize the baseline cache from R2 (run before check)
npm run screenshots:check            # gate: baselines vs test-results → report.json (exit 0; add --fail-on-change to gate)
npm run screenshots:report -- --e2e pass   # post the CI report to Discord (main+push only; needs DISCORD_WEBHOOK_URL)
npm run screenshots:noise -- run1/ run2/   # calibrate floors
npm run screenshots:upload-run -- --e2e pass   # CI-only: upload the run's renders for morelli

Baseline promotion (single or bulk) happens in morelli — pick the run, review the diffs, Promote. It copies verbatim bytes into the CAS, updates the R2 manifest atomically (with provenance: which run/branch/user), and snapshots the previous manifest for revert.

Activation checklist

  • Baseline manifest migrated to R2 (baselines/pcbjam/manifest.json, {name, engine} authoritative — written by morelli).
  • scale:'device'→'css' normalized (no-op at CI's DSF=1).
  1. Add the DISCORD_WEBHOOK_URL repo secret — until then everything is inert.
  2. Calibrate: run the suite twice in CI, screenshots:noise the two dirs, set FLOORS in config.ts.
  3. First re-baseline: promote a clean CI run's render, commit (expect a big, one-time chrome-font diff vs the Mac baselines).
  4. Delete the old scripts/{compare,update-baseline}-screenshots.sh.
  5. Once floors are proven stable, flip the gate to --fail-on-change.