fix(ci): serialize the heavy 3D-viewer e2e specs (dedicated chromium-ci-3d project)
The three heavy 3D-viewer specs (3d-viewer, 3d-viewer-deadlock, 3d-viewer-models)
crashed on CI (run 28604015154): "Target crashed", "browser has been closed", and a
black frozen canvas. Each boots the 3D-enabled pcbnew build, which pre-warms
~hardwareConcurrency*2+8 (~68 on the 30-core VM) Web Workers AND runs a multi-threaded
CPU raytracer over SwiftShader software-WebGL. Under fullyParallel the three spec FILES
run in separate browser processes concurrently, and the simultaneous raytrace tabs
exhaust the shared GPU process's ~16 live-WebGL-context limit / renderer memory. The
pool pre-warm (the deadlock fix, 7630c7e) is correct and present in the tested binary;
per-file isolation prevented in-process Worker accumulation but NOT this cross-process
concurrency — the gap these specs' first CI exposure hit.
Fix (test-orchestration only; no wasm/submodule changes):
- Route the three specs to a dedicated `chromium-ci-3d` Playwright project (own
outputDir=pw-artifacts/kicad-3d so its failure traces survive the later perf run's
wipe; the existing tests/pw-artifacts/** upload already captures it).
- test:kicad:ci runs it as a SECOND `playwright test` pass with --workers=1, so at most
one raytrace tab is alive at a time (the environment in which a single 3D load is
reliably green — docs/features/3d-raytracer/README.md §7). An exit-code-aggregating
sequencer runs both phases and fails if either fails (no silent skip of the 3D pass).
- Cap phase-1 CI workers to 12 as headroom under the WebGL-context cap.
- Log a resource-diagnostic snapshot (hardwareConcurrency, pthread pool, GL-canvas
count, wasm/JS heap) before each 3D interaction so any recurrence is attributable.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
34d2602132
commit
add7c60975
6 changed files with 87 additions and 8 deletions
|
|
@ -118,6 +118,24 @@ const BIG_MODULE_SPECS = [
|
|||
"**/occ-probe.spec.ts",
|
||||
];
|
||||
|
||||
// The heavy 3D-viewer specs each boot the 3D-enabled pcbnew build, which pre-warms
|
||||
// ~hardwareConcurrency*2+8 Web Workers AND runs a multi-threaded CPU raytracer over
|
||||
// SwiftShader software-WebGL. Under fullyParallel these separate spec FILES land in
|
||||
// separate browser processes CONCURRENTLY, and the simultaneous raytrace tabs exhaust
|
||||
// Workers / wasm heap / the shared GPU process's ~16 live-WebGL-context limit → the tab
|
||||
// crashes ("Target crashed" / "browser has been closed" / a black frozen canvas). Per-file
|
||||
// isolation (own worker) prevents in-process Worker accumulation but NOT this cross-process
|
||||
// concurrency. They STAY in BIG_MODULE_SPECS (so firefox keeps ignoring them — the ~190 MB
|
||||
// module OOMs SpiderMonkey/x86), but on CI they run in the dedicated 'chromium-ci-3d' project,
|
||||
// which test:kicad:ci invokes with --workers=1 so at most ONE raytrace tab is alive at a time.
|
||||
// (footprint-3d-preview.spec.ts is a placeholder glob — no such file exists yet — harmless.)
|
||||
const THREE_D_HEAVY_SPECS = [
|
||||
"**/3d-viewer.spec.ts",
|
||||
"**/3d-viewer-deadlock.spec.ts",
|
||||
"**/3d-viewer-models.spec.ts",
|
||||
"**/footprint-3d-preview.spec.ts",
|
||||
];
|
||||
|
||||
// Runtime-perf specs run ONLY on the Chromium 'perf' project below: they need
|
||||
// CDP CPU throttling (Chromium-only) and pcbnew needs V8. Excluded from the
|
||||
// firefox/chromium projects so they don't double-run there.
|
||||
|
|
@ -137,10 +155,13 @@ export default defineConfig({
|
|||
// as playwright.config.ts): the load-pcb post-load clipboard crash that can
|
||||
// close the page on Firefox, and the calculator first-run-wizard timing race.
|
||||
retries: process.env.CI ? 2 : 1,
|
||||
// Run parallel workers on CI too (Playwright default ≈ 50% of cores), same as
|
||||
// local — the serial CI run was the dominant wall-clock cost. Cap (e.g. '50%'
|
||||
// or a fixed count) if contention OOMs/flakes; retries:2 covers transient.
|
||||
workers: undefined,
|
||||
// Run parallel workers on CI too, same as local — the serial CI run was the dominant
|
||||
// wall-clock cost. Capped to a fixed 12 on CI (down from Playwright's default ≈ 50% of
|
||||
// cores = ~15 on the 30-core VM) as defense-in-depth for the ~16 live-WebGL-context limit
|
||||
// in Chromium's shared GPU process: ~15 GAL-canvas tabs already brush that cap. The heavy
|
||||
// 3D-viewer specs (a *second* WebGL context each) no longer run here — they're isolated to
|
||||
// the serial 'chromium-ci-3d' project — so 12 is only headroom, not the primary fix.
|
||||
workers: process.env.CI ? 12 : undefined,
|
||||
reporter: "html",
|
||||
timeout: 180000, // KiCad WASM needs more time to load (3 minutes)
|
||||
|
||||
|
|
@ -210,7 +231,29 @@ export default defineConfig({
|
|||
// --enable-unsafe-swiftshader: newer Chromium refuses software WebGL in
|
||||
// headless without it.
|
||||
name: "chromium-ci",
|
||||
testMatch: BIG_MODULE_SPECS,
|
||||
// Every big-module spec EXCEPT the heavy 3D-viewer ones — those move to the
|
||||
// serial 'chromium-ci-3d' project below so they never raytrace concurrently.
|
||||
testMatch: BIG_MODULE_SPECS.filter((s) => !THREE_D_HEAVY_SPECS.includes(s)),
|
||||
use: {
|
||||
...devices["Desktop Chrome"],
|
||||
viewport: { width: 1280, height: 720 },
|
||||
launchOptions: {
|
||||
args: ["--enable-unsafe-swiftshader"],
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
// CI-only serial carrier for the heavy 3D-viewer specs (see THREE_D_HEAVY_SPECS).
|
||||
// Same runtime as chromium-ci; test:kicad:ci invokes this project in a SEPARATE
|
||||
// `playwright test` pass with --workers=1, so at most one raytrace tab is alive at a
|
||||
// time (the environment in which a single 3D load is reliably green — see
|
||||
// docs/features/3d-raytracer/README.md §6-§7). Own outputDir so its start-of-run
|
||||
// cleanup doesn't wipe chromium-ci's retained traces, and so its own failure traces
|
||||
// survive the later `test:perf` run's pw-artifacts/kicad wipe; the CI upload glob
|
||||
// tests/pw-artifacts/** already captures it, so no workflow change is needed.
|
||||
name: "chromium-ci-3d",
|
||||
testMatch: THREE_D_HEAVY_SPECS,
|
||||
outputDir: process.env.CI ? "pw-artifacts/kicad-3d" : "test-results",
|
||||
use: {
|
||||
...devices["Desktop Chrome"],
|
||||
viewport: { width: 1280, height: 720 },
|
||||
|
|
|
|||
Loading…
Reference in a new issue