pcbjam/docs/features/wx-parity-bugs/occ-export-context-eviction.md
Viktor Vaczi 63ed1f3c1f e2e/CI: dual-engine suites, per-engine screenshots, SwiftShader retired, prod web suite, CI-coverage gate
Squash of experiment/ff-big-modules vs main.

Big-module routing removed: native-EH shrank kicad_editor below
SpiderMonkey's x86-64 code budget (runs 29355049705/29356152413 green on
stock Firefox), so BIG_MODULE_SPECS routing and the baseline-only-JIT
crutch are gone — kicad-firefox and kicad-chromium both run the full
suite, with the module compiled the way real users' browsers compile it.

Per-engine screenshots end to end: stableShot/shotPath write
test-results/<engine>/<name>.png; baselines move to
baseline-screenshots/{chromium,firefox}/ and the whole tools/screenshots
pipeline (compare/promote/manifest/spec-map/changelog/Discord) keys on
<engine>/<name>. Previously Firefox and Chromium renders of one spec
overwrote each other and Firefox renders were never actually gated.
Seeded from CI run 29421380806 (92 new firefox baselines, +24 chromium
web-suite shots); manifest generated from the baseline tree.

One merged playwright.config.ts (kicad/asyncify/coroutine/perf as
projects); ~25 dead npm scripts dropped. The web suite is gated in CI for
the first time ever (4 rotted specs fixed, 5 broken lib-bridge specs
triaged as fixme in docs/features/web-e2e-rot/); cheap lint step after
npm ci; last 26 blind-sleep violations fixed.

SwiftShader retired: CI Chromium renders WebGL on ANGLE → Mesa llvmpipe
(--use-gl=angle --use-angle=gl --ignore-gpu-blocklist; the blocklist flag
is mandatory — llvmpipe is blocklisted and WebGL is silently unavailable
without it) in BOTH configs. Under WORKERS=4 congestion SwiftShader
transiently failed the first post-board-load draw and the recovery
cascade ended in a silent permanent Cairo fallback — that engine flip was
the "~1.2% changedRatio both directions" occ-export baseline flake.
Validated 160/160 across two 80-repeat rigs; full analysis in
docs/features/wx-parity-bugs/occ-export-context-eviction.md. Chromium
baselines shift slightly on llvmpipe — promote once from the first green
run. Deflakes the new coverage exposed: presence baselines settle before
capture; presence fixtures declare current file formats; perf gets its
own outputDir so CI evidence survives; occ-export settles the board paint
before the export dialog; menu-item waits (waitForRenderedByLabel before
clickMenuItem) in 4 specs + the TESTING.md rule.

Web suite runs the PROD build, in parallel: webServer becomes backend
`start` + the standalone's e2e:preview (build-preview.mjs: link-wasm →
stash the public/wasm symlink aside during vite build, build-demo.mjs's
move — then vite preview as the persistent server). The wasm middleware
serves /wasm/* in preview and emits COOP/COEP/CORP itself (a pthread
worker script's own response must carry COEP or Chrome kills it with
ERR_BLOCKED_BY_RESPONSE). VITE_* flags bake at build time;
VITE_ALLOW_USER_OVERRIDE joins turbo globalEnv. fullyParallel + default
workers: 5.2m → 1.4m. Determinism fixes the parallel run exposed:
shared-page specs become serial groups; locks.spec grabs alice's exact
item via the new kicadCollabTestSelectByUuid hook (cross-tab "first
footprint" order is not a ysync invariant); quit specs poll page.url()
(quit supersedes its own navigation — NS_BINDING_ABORTED on Firefox).
Suite: 51 passed / 12 skipped / 0 failed in 1.6m.

CI-coverage gate (lint:ci-coverage): every tests/**/*.spec.ts must be
reachable from the npm scripts the workflows invoke — scraped from
.github/workflows/, resolved through package.json, coverage asked from
playwright --list itself. Rules: uncovered-spec + orphan-project (with a
documented LOCAL_ONLY_PROJECTS allowlist). Gating next to
lint:determinism; 138 spec files / 13 projects accounted for.

Product fixes kept from the investigations (reachable on real GPUs too):
wx 7799fd1be5 — paint flags clear before dispatch + Invalidate always
propagates; kicad 3dcfea5e45 — SwiftShader pass-boundary flush +
per-instance font texture + first-frame GL-error drain (GAL recovery
recovers instead of falling back to Cairo) + the user-facing eeschema
switch navigates again under __EMSCRIPTEN__ (project-sync's
FaceRegistered gate had rerouted it into the hidden sync player; caught
by the newly-gated web suite).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018eUxiPApHgGiu9NFyQfhAq
2026-07-17 12:21:54 +02:00

6.5 KiB
Raw Blame History

occ-export flake: SwiftShader congestion → GAL recovery cascade (fixed by GL-stack swap)

Status 2026-07-16: fully root-caused with page-level canvas/context telemetry + temp C++ instrumentation in a local docker rig. Final remedy: CI Chromium now runs WebGL on ANGLE → Mesa llvmpipe instead of SwiftShader (tests/playwright.config.ts, CHROMIUM_CI_ARGS) — the same software-GL stack the Firefox projects use, where none of this ever happened. Four small product fixes found along the way were kept (below); everything else was deliberately not hardened — the trigger cannot occur outside congested SwiftShader.

Symptom

chromium/occ-export-{dialog,done}.png bistable on CI only: the board behind the modal STEP-export dialog flips between runs (~1.2% changedRatio both directions). Firefox (llvmpipe) never affected.

The causal chain (every step observed in the rig)

  1. Transient SwiftShader failure at the first post-board-load draw (~10-15% of instances at REPEAT=40-80 WORKERS=4): DoRePaint threw: Requested render buffer size is not supported, plus LRU context evictions the moment a 3rd context appears (21/80 runs; the usual victim is a leaked hidden 881×159 startup GAL canvas, so the CONTEXT_LOST_WEBGL line alone was harmless). No flag stops it — --max-active-webgl-contexts=64, --force-gpu-mem-available-mb=4096, --disable-low-end-device-mode, --disable-gpu-driver-bug-workarounds all verified present and useless.

  2. recoverFromGalError attempt 1 (common/draw_panel_gal.cpp:205) destroys + recreates the WebGL GAL (board canvas glcanvas-2glcanvas-3).

  3. Attempt 1 then always failed on stale cross-context GL state:

    • static g_fontTexture bound on the new context → INVALID_OPERATION: bindTexture: object does not belong to this contextcheckGlError throw (fixed — kept: per-instance m_fontTexture, common/gal/webgl/webgl_gal.{cpp,h});
    • sticky GL error flags left by the old GAL's teardown (its LockCtx targets a lost context, so the deletes land on the current one) → the fresh GAL's first checkGlError ("generating vertices buffer: invalid enum") misattributed the leftover and threw (fixed — kept: drain glGetError() before first-frame init() in WEBGL_GAL::BeginDrawing). With both fixes, reinit-on-WebGL succeeds (validated: 0 Cairo fallbacks / 60 after, vs 100% of recoveries before).
  4. Attempt 2 = silent permanent Cairo fallback (GAL_FALLBACK == GAL_TYPE_CAIRO on emscripten, dialog suppressed). Cairo renders the board through the wx software path — shots LOOK painted but are a different engine → subtle whole-board diffs vs the WebGL baseline. This engine flip IS the original "~1.2% changedRatio both directions" flake. FULLSCREEN_QUAD (singleton VAOs) is the same static-GL-state disease and produced bindVertexArray: object does not belong + one wasm abort in rare runs — NOT fixed (unreachable once recovery never triggers; noted here in case in-app recovery robustness is ever wanted).

  5. wx paint-flag races turned it into a stuck-blank board (both fixed — kept, wxwidgets/src/wasm/window.cpp):

    • Invalidate early-out skipped the upward walk while the sweep can strand a window "flags set, ancestors clear" → every later Refresh() swallowed. Fix: always propagate to the top.
    • PaintSelf/PaintChildren cleared the dirty flags AFTER dispatching paint, so an Invalidate issued reentrantly from inside a paint handler (canvas recreated mid-frame, board-load refresh mid-sweep) was clobbered with nothing re-issuing it. Fix: DoPaint snapshots + clears both flags BEFORE dispatching. (Pre-fix rigs showed stuck half-painted toolbars — this is a real production-reachable bug, not a CI artifact.)

    One rare mode ("grid-mode": board data loaded, UI painted, canvas alive, items never drawn, zoom-to-fit doesn't help; VIEW had all 464 items and the same Clear/DisplayBoard ordering as healthy runs) was still being diagnosed when the GL-stack swap made it — and everything above — unobservable: 80/80 runs clean on llvmpipe, all dialog shots pixel- identical (single 55.65% ratio value across all 80; SwiftShader scattered over 55.556.8%).

The fix that ships

CHROMIUM_CI_ARGS = --use-gl=angle --use-angle=gl --ignore-gpu-blocklist (CI-gated). Requirements, all already present on CI: an X display (xvfb-run -a wraps the whole e2e step) and Mesa (installed by playwright install --with-deps). --ignore-gpu-blocklist is mandatory — llvmpipe is on Chromium's software-GL blocklist and WebGL silently reports unavailable without it (the exact analog of the Firefox projects' webgl.force-enabled). Headless works; headed is NOT needed. Renderer-string ground truth (Chrome falls back silently — always verify): gl.getExtension('WEBGL_debug_renderer_info') must report ANGLE (Mesa, llvmpipe …), not SwiftShader.

Consequence: chromium renders shift slightly → the chromium baseline set must be re-promoted once from the first green CI run (cd tests && npm run screenshots:promote -- --run <id>).

Test-side hardening (kept, independent)

Popup menu items register in the element registry progressively while the menu paints; a coarse ">3 menuitems" gate + single-shot clickMenuItem races (48/20 under contention, pre-existing). Rule (tests/TESTING.md): wait for the specific item via waitForRenderedByLabel(page, label, { elementType: 'menuitem' }) before every clickMenuItem. Applied to occ-export, occ-export-models, 3d-viewer-models, load-pcb; pl_editor already did this; clickMenuItemByText waits internally.

Open leads (not blocking)

  • The leaked hidden 881×159 startup GAL canvas: destroying it would free a context + memory.
  • If in-app GL-error recovery ever matters on real GPUs: fix the FULLSCREEN_QUAD singleton VAOs (per-instance like the font texture) and audit remaining cross-context statics.

Repro rig (for revalidation)

Docker mcr.microsoft.com/playwright:v1.57.0-jammy, worktree bind-mount, container-local node_modules volume, CI=1, xvfb-run -a npx playwright test --project=kicad-chromium --repeat-each=N --workers=4 with a temp spec that loads pic_programmer, screenshots board-load / settled / export-dialog / post-poke, and logs the WebGL renderer string + canvas/context telemetry per instance. Canaries: CONTEXT_LOST_WEBGL, does not belong to this context, board-strip pixel ratio of the shots.