Make the Playwright e2e + kicad suites deterministic so screenshot flake stops tracing to timing races. - Blind page.waitForTimeout -> condition waits (expect.poll, web-first assertions, waitUntil) + readiness helpers (waitForWxApp, waitForCanvasApp). Remaining sleeps are documented interaction dwells (annotated). - Defensive "if element exists" branches -> loud asserts; label-fallback chains -> normalized clickMenuItemByText. First-run wizard for/if loops removed by seeding calculator/gerbview/pcbnew HTMLs. - Screenshots: new stableShot(page, name) settles the render in-page (canvas hash over rAF) then writes a raw PNG to test-results/ for the existing offline gate (tools/screenshots vs baseline-screenshots). Replaces toHaveScreenshot, which did inline compare + its own baselines and had decoupled the specs from the real gate. scale:'css' pinned. - retries: 0 in both configs. - Guard: tests/tools/lint-determinism.ts (npm run lint:determinism) bans blind sleeps / toHaveScreenshot / inline retries / swallowed catches in specs; documented exceptions carry a marker. Rules in tests/TESTING.md. Assertions, coverage, and renders unchanged (semantic-equivalence reviewed; captures pixel-identical modulo inherent timer/timestamp/3d-raytrace variance). Both suites green at retries:0 (e2e 340, kicad 92); ~35-61% faster. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BVX1pHMvRPYHdp6ZfEawrk
122 lines
4.9 KiB
TypeScript
122 lines
4.9 KiB
TypeScript
import { defineConfig, devices } from '@playwright/test';
|
|
import { execSync } from 'child_process';
|
|
import * as fs from 'fs';
|
|
import * as path from 'path';
|
|
|
|
const PORT_FILE = path.join(__dirname, '.test-port');
|
|
|
|
// Resolve the static-server port for this run.
|
|
//
|
|
// This config file is re-imported by EVERY Playwright process: the main runner
|
|
// (which launches the webServer) and each worker process (which calls
|
|
// page.goto(baseURL)). They must all agree on one port. Playwright also
|
|
// *recreates* a worker mid-run after a test times out or crashes — and that new
|
|
// worker re-imports this config.
|
|
//
|
|
// The previous heuristic ("reuse .test-port if it's <60s old, else pick a new
|
|
// free port") broke exactly there: once a run passed the 60s mark, a recreated
|
|
// worker treated the file as stale, picked a DIFFERENT free port, and every
|
|
// subsequent page.goto hit a dead port (net::ERR_CONNECTION_REFUSED) because the
|
|
// webServer was still listening on the original port. A single failing test
|
|
// thus cascaded into ~all later tests failing.
|
|
//
|
|
// Fix (same as playwright-kicad.config.ts): drop the time window entirely. The
|
|
// main runner always picks a fresh port and writes it; workers always reuse
|
|
// whatever the main runner wrote. The main runner is the only process whose
|
|
// argv carries the `test` command (workers are forked with an empty argv), and
|
|
// it imports this config — and so writes the file — before any worker is
|
|
// spawned.
|
|
function resolvePort(): number {
|
|
const isMainRunner = process.argv.slice(2).includes('test');
|
|
if (!isMainRunner) {
|
|
try {
|
|
const existing = parseInt(fs.readFileSync(PORT_FILE, 'utf-8').trim(), 10);
|
|
if (existing > 0 && existing < 65536) {
|
|
return existing;
|
|
}
|
|
} catch {
|
|
// No readable port file — fall through. Shouldn't happen in a worker,
|
|
// since the main runner writes the file before spawning workers.
|
|
}
|
|
}
|
|
|
|
const port = findFreePort();
|
|
fs.writeFileSync(PORT_FILE, port.toString());
|
|
return port;
|
|
}
|
|
|
|
// Find a free port dynamically using a shell command
|
|
function findFreePort(): number {
|
|
// Use Python to find a free port (works on macOS and Linux)
|
|
try {
|
|
const result = execSync(
|
|
'python3 -c "import socket; s=socket.socket(); s.bind((\'\',0)); print(s.getsockname()[1]); s.close()"',
|
|
// timeout: a broken python3 must fall back to the random port below, not
|
|
// hang the whole run at config-load time.
|
|
{ encoding: 'utf-8', timeout: 5000 }
|
|
);
|
|
return parseInt(result.trim());
|
|
} catch {
|
|
// Fallback to default port range
|
|
return 9000 + Math.floor(Math.random() * 1000);
|
|
}
|
|
}
|
|
|
|
const port = resolvePort();
|
|
|
|
const appsDir = 'apps';
|
|
|
|
export default defineConfig({
|
|
globalSetup: './global-setup.ts',
|
|
testDir: './e2e',
|
|
// In CI, redirect Playwright's start-of-run outputDir cleanup to a throwaway dir so it
|
|
// never wipes test-results/ — the committed-baseline screenshots (page.screenshot to
|
|
// 'test-results/…') must survive across the sequential wx/asyncify/kicad/perf runs for the
|
|
// screenshot compare. Local dev keeps the default (test-results cleaned each single run).
|
|
outputDir: process.env.CI ? 'pw-artifacts/wx' : 'test-results',
|
|
fullyParallel: true,
|
|
forbidOnly: !!process.env.CI,
|
|
// retries:0 — the suite is deterministic (no blind sleeps, no "if element exists"
|
|
// branches; screenshots are captured via stableShot for the offline gate, not asserted
|
|
// inline), so a failure is a real failure, not flake to paper over with a retry.
|
|
retries: 0,
|
|
// Run parallel workers on CI too (Playwright default ≈ 50% of cores), same as
|
|
// local — the serial CI run was the dominant wall-clock cost. Cap (e.g. '50%'
|
|
// or a fixed count) if contention OOMs/flakes; retries:2 covers transient.
|
|
workers: undefined,
|
|
reporter: 'html',
|
|
timeout: 60000, // WASM can be slow to load
|
|
|
|
// No expect.toHaveScreenshot block: screenshots are captured via stableShot() (render-settle +
|
|
// raw PNG to test-results/) and compared OFFLINE by tools/screenshots against tests/baseline-
|
|
// screenshots on CI's deterministic Linux render — Playwright does no inline pixel comparison.
|
|
|
|
use: {
|
|
baseURL: `http://localhost:${port}`,
|
|
trace: 'on-first-retry',
|
|
// Grant clipboard and font permissions for tests
|
|
permissions: ['clipboard-read', 'clipboard-write', 'local-fonts'],
|
|
},
|
|
|
|
projects: [
|
|
{
|
|
name: 'chromium',
|
|
use: {
|
|
...devices['Desktop Chrome'],
|
|
// CI runners have no GPU and several wx specs use WebGL
|
|
// (gal-webgl.spec.ts etc.): newer headless Chromium refuses software
|
|
// WebGL without --enable-unsafe-swiftshader. CI-gated so local runs
|
|
// keep stock behavior (same pattern as playwright-kicad.config.ts).
|
|
...(process.env.CI ? {
|
|
launchOptions: { args: ['--enable-unsafe-swiftshader'] },
|
|
} : {}),
|
|
},
|
|
},
|
|
],
|
|
|
|
webServer: {
|
|
command: `npx serve ${appsDir} -p ${port} -c ../serve.json`,
|
|
port: port,
|
|
reuseExistingServer: !process.env.CI,
|
|
},
|
|
});
|