Watch

Viktor Vaczi 4c3a4cacd4 test(determinism): deterministic waits + stableShot screenshots; drop blind sleeps/ifs/retries

Make the Playwright e2e + kicad suites deterministic so screenshot flake stops
tracing to timing races.

- Blind page.waitForTimeout -> condition waits (expect.poll, web-first
  assertions, waitUntil) + readiness helpers (waitForWxApp, waitForCanvasApp).
  Remaining sleeps are documented interaction dwells (annotated).
- Defensive "if element exists" branches -> loud asserts; label-fallback chains
  -> normalized clickMenuItemByText. First-run wizard for/if loops removed by
  seeding calculator/gerbview/pcbnew HTMLs.
- Screenshots: new stableShot(page, name) settles the render in-page (canvas
  hash over rAF) then writes a raw PNG to test-results/ for the existing offline
  gate (tools/screenshots vs baseline-screenshots). Replaces toHaveScreenshot,
  which did inline compare + its own baselines and had decoupled the specs from
  the real gate. scale:'css' pinned.
- retries: 0 in both configs.
- Guard: tests/tools/lint-determinism.ts (npm run lint:determinism) bans blind
  sleeps / toHaveScreenshot / inline retries / swallowed catches in specs;
  documented exceptions carry a marker. Rules in tests/TESTING.md.

Assertions, coverage, and renders unchanged (semantic-equivalence reviewed;
captures pixel-identical modulo inherent timer/timestamp/3d-raytrace variance).
Both suites green at retries:0 (e2e 340, kicad 92); ~35-61% faster.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BVX1pHMvRPYHdp6ZfEawrk

2026-07-07 10:50:24 +02:00

2.4 KiB

Raw Blame History

Testing rules

Determinism rules for the Playwright specs (tests/e2e, tests/kicad, tests/web). Enforced by npm run lint:determinism (tools/lint-determinism.ts). Run specs from tests/ via npm run test:kicad (firefox) / npm run test:e2e (chromium) — not playwright directly.

Waits — never blind

No page.waitForTimeout(n). Wait on a condition: expect.poll(() => predicate), a web-first assertion (expect(locator).toBeVisible()), or waitUntil(page, fn, desc) (throws loudly on timeout).
App readiness: waitForWxApp(page) (canvas visible + element registry populated) for widget/editor harnesses; waitForCanvasApp(page) for registry-less canvas apps.
The only allowed waitForTimeout is an irreducible interaction dwell — a canvas/keyboard commit with no JS-observable signal — and it MUST carry a same-line marker: // eslint-disable-line -- documented interaction dwell: <why>.

No defensive branches

No if (await el.count()) el.click(). Assert the element exists, then act: expect(await clickByLabel(page, 'X'), '...').toBe(true). Use clickMenuItemByText (normalizes & / ... / …) instead of try-A-else-A…-else-A fallback chains.
No swallowed .catch(() => {}). Let it throw, or assert the tolerated outcome. A genuinely best-effort op must carry a marker explaining why.

Screenshots — `stableShot`, compared offline

Capture with stableShot(page, 'name.png', { fullPage }) — it settles the render (in-page canvas-hash over animation frames) then writes a raw PNG to test-results/. It does not assert. Never use Playwright's toHaveScreenshot.
Comparison is offline: npm run screenshots:check diffs test-results/ against the committed baselines in tests/baseline-screenshots/ (+ 3d-regression/, gal-regression/).
CI's Linux render is the source of truth; baselines are promoted from CI (npm run screenshots:promote -- --run <ci-run-id>). A local (Mac) check shows font/render noise and is not the gate.
A continuously-animating state (timer, mid-slide) can't be a stable baseline — drop the shot.

Retries

retries: 0 in both configs. A failure is real; don't mask it with a retry.

Where things are

Per-test logs (JS console + cpp): tests/logs/{wxwidgets,kicad}/<test-name>/.
Guard: npm run lint:determinism. Screenshot gate: npm run screenshots:check.

2.4 KiB Raw Blame History