fix(ci): serialize the heavy 3D-viewer e2e specs (dedicated chromium-ci-3d project)

The three heavy 3D-viewer specs (3d-viewer, 3d-viewer-deadlock, 3d-viewer-models)
crashed on CI (run 28604015154): "Target crashed", "browser has been closed", and a
black frozen canvas. Each boots the 3D-enabled pcbnew build, which pre-warms
~hardwareConcurrency*2+8 (~68 on the 30-core VM) Web Workers AND runs a multi-threaded
CPU raytracer over SwiftShader software-WebGL. Under fullyParallel the three spec FILES
run in separate browser processes concurrently, and the simultaneous raytrace tabs
exhaust the shared GPU process's ~16 live-WebGL-context limit / renderer memory. The
pool pre-warm (the deadlock fix, 7630c7e) is correct and present in the tested binary;
per-file isolation prevented in-process Worker accumulation but NOT this cross-process
concurrency — the gap these specs' first CI exposure hit.

Fix (test-orchestration only; no wasm/submodule changes):
- Route the three specs to a dedicated `chromium-ci-3d` Playwright project (own
  outputDir=pw-artifacts/kicad-3d so its failure traces survive the later perf run's
  wipe; the existing tests/pw-artifacts/** upload already captures it).
- test:kicad:ci runs it as a SECOND `playwright test` pass with --workers=1, so at most
  one raytrace tab is alive at a time (the environment in which a single 3D load is
  reliably green — docs/features/3d-raytracer/README.md §7). An exit-code-aggregating
  sequencer runs both phases and fails if either fails (no silent skip of the 3D pass).
- Cap phase-1 CI workers to 12 as headroom under the WebGL-context cap.
- Log a resource-diagnostic snapshot (hardwareConcurrency, pthread pool, GL-canvas
  count, wasm/JS heap) before each 3D interaction so any recurrence is attributable.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Istvan Matejcsok 2026-07-03 10:49:23 +02:00
commit add7c60975
6 changed files with 87 additions and 8 deletions

View file

@ -118,6 +118,24 @@ const BIG_MODULE_SPECS = [
"**/occ-probe.spec.ts",
];
// The heavy 3D-viewer specs each boot the 3D-enabled pcbnew build, which pre-warms
// ~hardwareConcurrency*2+8 Web Workers AND runs a multi-threaded CPU raytracer over
// SwiftShader software-WebGL. Under fullyParallel these separate spec FILES land in
// separate browser processes CONCURRENTLY, and the simultaneous raytrace tabs exhaust
// Workers / wasm heap / the shared GPU process's ~16 live-WebGL-context limit → the tab
// crashes ("Target crashed" / "browser has been closed" / a black frozen canvas). Per-file
// isolation (own worker) prevents in-process Worker accumulation but NOT this cross-process
// concurrency. They STAY in BIG_MODULE_SPECS (so firefox keeps ignoring them — the ~190 MB
// module OOMs SpiderMonkey/x86), but on CI they run in the dedicated 'chromium-ci-3d' project,
// which test:kicad:ci invokes with --workers=1 so at most ONE raytrace tab is alive at a time.
// (footprint-3d-preview.spec.ts is a placeholder glob — no such file exists yet — harmless.)
const THREE_D_HEAVY_SPECS = [
"**/3d-viewer.spec.ts",
"**/3d-viewer-deadlock.spec.ts",
"**/3d-viewer-models.spec.ts",
"**/footprint-3d-preview.spec.ts",
];
// Runtime-perf specs run ONLY on the Chromium 'perf' project below: they need
// CDP CPU throttling (Chromium-only) and pcbnew needs V8. Excluded from the
// firefox/chromium projects so they don't double-run there.
@ -137,10 +155,13 @@ export default defineConfig({
// as playwright.config.ts): the load-pcb post-load clipboard crash that can
// close the page on Firefox, and the calculator first-run-wizard timing race.
retries: process.env.CI ? 2 : 1,
// Run parallel workers on CI too (Playwright default ≈ 50% of cores), same as
// local — the serial CI run was the dominant wall-clock cost. Cap (e.g. '50%'
// or a fixed count) if contention OOMs/flakes; retries:2 covers transient.
workers: undefined,
// Run parallel workers on CI too, same as local — the serial CI run was the dominant
// wall-clock cost. Capped to a fixed 12 on CI (down from Playwright's default ≈ 50% of
// cores = ~15 on the 30-core VM) as defense-in-depth for the ~16 live-WebGL-context limit
// in Chromium's shared GPU process: ~15 GAL-canvas tabs already brush that cap. The heavy
// 3D-viewer specs (a *second* WebGL context each) no longer run here — they're isolated to
// the serial 'chromium-ci-3d' project — so 12 is only headroom, not the primary fix.
workers: process.env.CI ? 12 : undefined,
reporter: "html",
timeout: 180000, // KiCad WASM needs more time to load (3 minutes)
@ -210,7 +231,29 @@ export default defineConfig({
// --enable-unsafe-swiftshader: newer Chromium refuses software WebGL in
// headless without it.
name: "chromium-ci",
testMatch: BIG_MODULE_SPECS,
// Every big-module spec EXCEPT the heavy 3D-viewer ones — those move to the
// serial 'chromium-ci-3d' project below so they never raytrace concurrently.
testMatch: BIG_MODULE_SPECS.filter((s) => !THREE_D_HEAVY_SPECS.includes(s)),
use: {
...devices["Desktop Chrome"],
viewport: { width: 1280, height: 720 },
launchOptions: {
args: ["--enable-unsafe-swiftshader"],
},
},
},
{
// CI-only serial carrier for the heavy 3D-viewer specs (see THREE_D_HEAVY_SPECS).
// Same runtime as chromium-ci; test:kicad:ci invokes this project in a SEPARATE
// `playwright test` pass with --workers=1, so at most one raytrace tab is alive at a
// time (the environment in which a single 3D load is reliably green — see
// docs/features/3d-raytracer/README.md §6-§7). Own outputDir so its start-of-run
// cleanup doesn't wipe chromium-ci's retained traces, and so its own failure traces
// survive the later `test:perf` run's pw-artifacts/kicad wipe; the CI upload glob
// tests/pw-artifacts/** already captures it, so no workflow change is needed.
name: "chromium-ci-3d",
testMatch: THREE_D_HEAVY_SPECS,
outputDir: process.env.CI ? "pw-artifacts/kicad-3d" : "test-results",
use: {
...devices["Desktop Chrome"],
viewport: { width: 1280, height: 720 },