AI_Assistant/pcbjam
Watch
1
0
Fork
You've already forked pcbjam
 0
Code Issues Pull requests Projects Releases 7 Packages Wiki Activity Actions

test(determinism): deterministic waits + stableShot screenshots; drop blind sleeps/ifs/retries

Browse source 
Make the Playwright e2e + kicad suites deterministic so screenshot flake stops
tracing to timing races.

- Blind page.waitForTimeout -> condition waits (expect.poll, web-first
  assertions, waitUntil) + readiness helpers (waitForWxApp, waitForCanvasApp).
  Remaining sleeps are documented interaction dwells (annotated).
- Defensive "if element exists" branches -> loud asserts; label-fallback chains
  -> normalized clickMenuItemByText. First-run wizard for/if loops removed by
  seeding calculator/gerbview/pcbnew HTMLs.
- Screenshots: new stableShot(page, name) settles the render in-page (canvas
  hash over rAF) then writes a raw PNG to test-results/ for the existing offline
  gate (tools/screenshots vs baseline-screenshots). Replaces toHaveScreenshot,
  which did inline compare + its own baselines and had decoupled the specs from
  the real gate. scale:'css' pinned.
- retries: 0 in both configs.
- Guard: tests/tools/lint-determinism.ts (npm run lint:determinism) bans blind
  sleeps / toHaveScreenshot / inline retries / swallowed catches in specs;
  documented exceptions carry a marker. Rules in tests/TESTING.md.

Assertions, coverage, and renders unchanged (semantic-equivalence reviewed;
captures pixel-identical modulo inherent timer/timestamp/3d-raytrace variance).
Both suites green at retries:0 (e2e 340, kicad 92); ~35-61% faster.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BVX1pHMvRPYHdp6ZfEawrk
This commit is contained in:
Viktor Vaczi 2026-07-07 10:50:24 +02:00
commit 4c3a4cacd4
102 changed files with 2867 additions and 3407 deletions
Download patch file Download diff file Expand all files Collapse all files

71
tests/kicad/symbol_editor.spec.ts
View file

@ -1,60 +1,19 @@
import type { Page } from '@playwright/test';
import { test, expect } from './fixtures';
import { clickByLabel } from '../e2e/utils/element-tracker';
import { waitForEditorReady, stableShot } from '../e2e/utils/element-tracker';
/**
* Symbol Editor WASM E2E Tests
*
* The symbol editor (FRAME_SCH_SYMBOL_EDITOR) is served by the eeschema kiface;
* the standalone `symbol_editor` launcher opens that frame directly. It shares
* the same first-run setup wizard as eeschema/pcbnew, so the launch flow mirrors
* eeschema.spec.ts: wait for the canvas, click through the wizard, then assert the
* editor chrome built. Scope is launch-only — the editor must start, paint a
* canvas + toolbars, populate the element registry, and produce no WASM abort.
* Library load/save and other features are intentionally out of scope here.
* the standalone `symbol_editor` launcher opens that frame directly. symbol_editor.html
* seeds a default KiCad config in preRun, so the shared first-run setup wizard never
* opens — the editor comes straight up. Scope is launch-only: the editor must start,
* paint a canvas + toolbars, populate the element registry, and produce no WASM abort.
*
* Determinism: no waitForTimeout, no wizard click-through loop, screenshots via
* stableShot (stabilizes before comparing).
*/
async function completeWizard(page: Page): Promise<void> {
await expect(page.locator('#canvas')).toBeVisible({ timeout: 90000 });
await page.waitForFunction(() => !!window.wxElementRegistry, null, { timeout: 90000 });
// The frame builds its UI a beat after the registry object appears; wait for
// the registry to actually have entries (the wizard or the editor itself)
// before driving it, otherwise we screenshot a blank canvas.
await page.waitForFunction(() => {
const registry = window.wxElementRegistry;
return !!registry && registry.findAll({}).length > 0;
}, null, { timeout: 90000 });
await page.waitForTimeout(2000);
await page.screenshot({ path: 'test-results/symbol_editor-wizard-00-initial.png', scale: 'css' });
for (let i = 1; i <= 10; i++) {
let clicked = await clickByLabel(page, 'Next >');
if (!clicked) {
clicked = await clickByLabel(page, 'Finish');
if (clicked) {
await page.waitForTimeout(500);
await page.screenshot({
path: `test-results/symbol_editor-wizard-${String(i).padStart(2, '0')}-finish.png`,
scale: 'css'
});
}
break;
}
await page.waitForTimeout(500);
await page.screenshot({
path: `test-results/symbol_editor-wizard-${String(i).padStart(2, '0')}.png`,
scale: 'css'
});
}
await page.waitForTimeout(2000);
}
function hasAbort(testLogger: { consoleLogs: string[]; errors: string[] }): boolean {
return [...testLogger.consoleLogs, ...testLogger.errors].some(line => line.includes('Aborted('));
}
@ -65,8 +24,8 @@ test.describe('symbol_editor WASM', () => {
});
test('app loads, canvas visible, no WASM abort', async ({ page, testLogger }) => {
await completeWizard(page);
await page.screenshot({ path: 'test-results/symbol_editor-01-loaded.png', scale: 'css' });
await waitForEditorReady(page);
await stableShot(page, 'symbol_editor-01-loaded.png');
expect(hasAbort(testLogger), 'no WASM abort during load').toBe(false);
@ -75,12 +34,12 @@ test.describe('symbol_editor WASM', () => {
});
test('canvas + toolbar metrics look sane', async ({ page, testLogger }) => {
await completeWizard(page);
await waitForEditorReady(page);
const metrics = await page.evaluate(() => {
const registry = window.wxElementRegistry;
const all = registry ? registry.findAll({ visible: true }) : [];
const toolbars = all.filter((el: { typeName: string }) => /ToolBar/.test(el.typeName));
const registry = window.wxElementRegistry!;
const all = registry.findAll({ visible: true });
const toolbars = all.filter((el) => /ToolBar/.test(el.typeName));
const glCanvas = document.querySelector('canvas[id^="glcanvas-"]') as HTMLCanvasElement | null;
return {
@ -94,7 +53,7 @@ test.describe('symbol_editor WASM', () => {
};
});
await page.screenshot({ path: 'test-results/symbol_editor-02-metrics.png', scale: 'css' });
await stableShot(page, 'symbol_editor-02-metrics.png');
expect(metrics.registryTotal, 'registry should be populated').toBeGreaterThan(10);
expect(metrics.toolbarCount, 'at least one toolbar should be visible').toBeGreaterThanOrEqual(1);