docs: ✏️ cleanup and organize docs
This commit is contained in:
Istvan Matejcsok
parent
8db6cfadc3
commit
8e413f89ec
29 changed files with 85 additions and 3340 deletions
58
docs/README.md
Normal file
58
docs/README.md
Normal file
|
|
@ -0,0 +1,58 @@
|
|||
# Documentation Map
|
||||
|
||||
A central index of the documentation in this repo. The goal of the project is to
|
||||
build KiCad with WASM and run it in a browser.
|
||||
|
||||
> Note: per-area `README.md` files stay next to the code they describe (they're linked
|
||||
> below). Cross-cutting guides live under `docs/`. Per-feature design notes live under
|
||||
> [`features/`](features/).
|
||||
|
||||
## Start here
|
||||
|
||||
- [Project README](../README.md) — overview, prerequisites, quick start, project structure
|
||||
- [CLAUDE.md](../CLAUDE.md) — project/agent context and contribution conventions
|
||||
|
||||
## Build
|
||||
|
||||
- [docs/build.md](build.md) — Docker-based KiCad WASM build system (two-phase build, outputs, memory)
|
||||
- [docker/README.md](../docker/README.md) — Docker build environment, branch-specific containers, troubleshooting
|
||||
- [wasm/README.md](../wasm/README.md) — WASM compatibility layer (overrides/shims without patching KiCad)
|
||||
|
||||
## Debugging & Asyncify
|
||||
|
||||
- [docs/debugging/DEBUG.md](debugging/DEBUG.md) — debugging guide: Asyncify stalls vs crashes, shim/codegen coupling, stub-bisection
|
||||
- [docs/debugging/learning.md](debugging/learning.md) — Asyncify + consecutive modal dialogs: the lock pattern
|
||||
- [docs/research/threading_1.md](research/threading_1.md) — deep dive: the Asyncify single-slot `currData` collision bug and the fix
|
||||
- [docs/research/threading_2.md](research/threading_2.md) — external research: JSPI/WasmFX/state-machine alternatives, QEMU analysis
|
||||
|
||||
## Architecture
|
||||
|
||||
- [wasm/README.md](../wasm/README.md) — WASM compatibility layer structure
|
||||
- [web/README.md](../web/README.md) — web app (create/open KiCad projects), tech stack, URL routing, WASM artifact serving
|
||||
|
||||
## Testing
|
||||
|
||||
- [tests/README.md](../tests/README.md) — Playwright test infrastructure, element registry, logs, screenshots
|
||||
- [tests/WHATWORKS.md](../tests/WHATWORKS.md) — wxWidgets-in-WASM feature coverage matrix and KiCad readiness
|
||||
- [tests/GL_README.md](../tests/GL_README.md) — Emscripten legacy GL immediate-mode quirks (color-per-vertex)
|
||||
- [tests/gal-regression/README.md](../tests/gal-regression/README.md) — GAL visual regression suite (native OpenGL vs WebGL WASM)
|
||||
|
||||
## Feature design docs
|
||||
|
||||
Per-feature design notes and porting records live under [`features/`](features/):
|
||||
|
||||
- [web-init](features/web-init/) — web app spec
|
||||
- [schematic](features/schematic/) — eeschema WASM bring-up
|
||||
- [symbol-editor](features/symbol-editor/) — symbol editor (eeschema kiface launcher)
|
||||
- [gerbview](features/gerbview/) — Gerber viewer port
|
||||
- [pl-editor](features/pl-editor/) — page-layout editor port (incl. file-dialog usability fixes)
|
||||
- [browser-tools](features/browser-tools/) — tool-activation / coroutine deep dives
|
||||
- [fix-asyncify-O2-and-modal-promise-rejection](features/fix-asyncify-O2-and-modal-promise-rejection/) — RTree wasm overflow bug investigation
|
||||
|
||||
### Archived / historical
|
||||
|
||||
[`features/archive/`](features/archive/) holds docs whose work is done or superseded
|
||||
(each carries a status banner):
|
||||
|
||||
- [webgl](features/archive/webgl/) — WebGL-GAL strategy/plan (since implemented in `kicad/common/gal/webgl/`)
|
||||
- [ipc-api](features/archive/ipc-api/) — IPC-API guard cleanup TODO (revert not yet actioned)
|
||||
292
docs/build.md
Normal file
292
docs/build.md
Normal file
|
|
@ -0,0 +1,292 @@
|
|||
# KiCad WASM Build System
|
||||
|
||||
This document describes how to build KiCad for WebAssembly using the Docker-based build system.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### Docker
|
||||
- Docker Desktop with ARM64 support (for Apple Silicon) or x86_64
|
||||
- 10+ GB disk space for build cache
|
||||
- Recommended: 10 CPUs, 32GB RAM allocated to Docker
|
||||
|
||||
### Host Tools
|
||||
|
||||
Binaryen (wasm-opt) is downloaded automatically by the build script. No manual installation needed.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Build KiCad WASM (with debug symbols by default, sequential compilation)
|
||||
./docker/build.sh
|
||||
|
||||
# Build with parallel compilation (faster, requires more RAM)
|
||||
./docker/build.sh -j 4
|
||||
|
||||
# Build optimized release (smaller WASM, no debug symbols)
|
||||
./docker/build.sh --release
|
||||
|
||||
# Interactive shell for debugging
|
||||
./docker/shell.sh
|
||||
```
|
||||
|
||||
**Note:** Builds run sequentially by default (`-j 1`) to avoid memory exhaustion in Docker. Use `-j N` for parallel compilation if you have sufficient RAM (at least 16GB for `-j 4`).
|
||||
|
||||
**Build outputs:**
|
||||
- `build-wasm/kicad-pcbnew/pcbnew/pcbnew.js` - Main WASM loader
|
||||
- `build-wasm/kicad-pcbnew/pcbnew/pcbnew.wasm` - WASM binary
|
||||
- `build-wasm/kicad-pcbnew/pcbnew/pcbnew.wasm.map` - Source map (debug builds)
|
||||
|
||||
## Two-Phase Build
|
||||
|
||||
The build is split into two phases due to memory requirements:
|
||||
|
||||
### Phase 1: Docker Compilation
|
||||
Compiles KiCad to WASM **without** asyncify transformation. This runs inside Docker with 32GB memory limit.
|
||||
|
||||
### Phase 2: Host Asyncify
|
||||
Applies `wasm-opt --asyncify` on the host machine using Binaryen v121 (downloaded automatically to `tools/`). This transformation uses ~20-30GB RAM.
|
||||
|
||||
**Note:** Binaryen v121 is used because v125 has a regression causing crashes in the asyncify liveness analysis.
|
||||
|
||||
### Why Asyncify?
|
||||
|
||||
Asyncify is an Emscripten transformation that allows WASM code to pause and resume execution. This is required for:
|
||||
|
||||
- **Modal dialogs** - `wxDialog::ShowModal()` blocks until user closes the dialog
|
||||
- **Message boxes** - `wxMessageBox()` waits for user response
|
||||
- **Clipboard operations** - Browser clipboard API is async
|
||||
- **Sleep/wait operations** - Any blocking call that needs to yield to the browser
|
||||
|
||||
Without asyncify, modal dialogs would freeze the browser because WASM cannot yield control back to JavaScript's event loop.
|
||||
|
||||
### How It Works
|
||||
|
||||
1. `docker/build.sh` compiles KiCad in Docker (no asyncify flags)
|
||||
2. Output is copied to `./output/` directory
|
||||
3. `wasm-opt --asyncify` runs on host, transforming the WASM binary
|
||||
4. Final output is ready for browser execution
|
||||
|
||||
### Technical Details
|
||||
|
||||
The asyncify transformation:
|
||||
- Instruments every function that might be on the call stack during an async operation
|
||||
- Adds stack save/restore logic to unwind and rewind the WASM stack
|
||||
- Increases binary size by ~20% (141MB → 171MB for KiCad)
|
||||
- Uses `asyncify-imports` pattern matching to identify async entry points
|
||||
|
||||
Import patterns used:
|
||||
- `env.invoke_*` - Exception handling trampolines
|
||||
- `env.__asyncjs__*` - EM_ASYNC_JS functions (like `startModal()`)
|
||||
|
||||
## Docker Architecture
|
||||
|
||||
**Base image:** `emscripten/emsdk:4.0.2-arm64`
|
||||
|
||||
**Volumes:**
|
||||
- Source code bind mount: Project root → `/workspace`
|
||||
- Build cache (named volume): `kicad-build-cache` → `/workspace/build-wasm`
|
||||
- Output bind mount: `./output` → `/workspace/output`
|
||||
|
||||
**Entry scripts:**
|
||||
| Script | Purpose |
|
||||
|--------|---------|
|
||||
| `docker/build.sh` | Run build from host |
|
||||
| `docker/shell.sh` | Interactive shell in container |
|
||||
| `docker/entrypoint.sh` | Sources Emscripten environment |
|
||||
|
||||
## Dependencies
|
||||
|
||||
| Dependency | Version | Build System | Purpose |
|
||||
|-----------|---------|--------------|---------|
|
||||
| GLM | 0.9.9.8 | Header-only | Math library |
|
||||
| Zstd | 1.5.5 | CMake | Compression for project files |
|
||||
| Protobuf | 3.21.12 | CMake | IPC serialization |
|
||||
| FreeType | 2.13.2 | CMake | Font rendering |
|
||||
| HarfBuzz | 8.3.0 | CMake | Text shaping |
|
||||
| Pixman | 0.42.2 | Meson | Pixel manipulation |
|
||||
| Cairo | 1.18.0 | Meson | 2D graphics rendering |
|
||||
| Boost | 1.84.0 | B2 | Locale library |
|
||||
| wxWidgets | 3.3.1 | Autoconf | GUI framework |
|
||||
| OpenCASCADE | 7.8.0 | CMake | 3D geometry (optional) |
|
||||
| ngspice | 45.2 | Autoconf | SPICE simulation (optional) |
|
||||
|
||||
### Build Order
|
||||
|
||||
1. **Header-only:** GLM
|
||||
2. **Compression/serialization:** Zstd, Protobuf
|
||||
3. **Font stack:** FreeType → HarfBuzz
|
||||
4. **Graphics:** Pixman → Cairo
|
||||
5. **Optional:** OpenCASCADE, ngspice
|
||||
6. **GUI framework:** wxWidgets
|
||||
7. **Application:** KiCad PCBnew
|
||||
|
||||
## Build Flags
|
||||
|
||||
| Flag | Description |
|
||||
|------|-------------|
|
||||
| `--full` | Full clean rebuild (all deps + wxWidgets + KiCad) |
|
||||
| `--clean-kicad` | Clean only KiCad build directory |
|
||||
| `--build-deps` | Build dependencies (skipped by default) |
|
||||
| `--release` | Disable debug symbols, enable optimizations |
|
||||
| `--debug` | Enable debug symbols (default) |
|
||||
| `-j N` | Parallel jobs (default: all cores) |
|
||||
|
||||
### Build Modes
|
||||
|
||||
| Mode | Command | Description |
|
||||
|------|---------|-------------|
|
||||
| **Incremental (default)** | `./docker/build.sh` | Fastest for development (~1.5 min) |
|
||||
| **Full rebuild** | `./docker/build.sh --full` | Clean everything and rebuild |
|
||||
| **Rebuild KiCad** | `./docker/build.sh --clean-kicad` | Clean and rebuild KiCad only |
|
||||
| **With dependencies** | `./docker/build.sh --build-deps` | Also rebuild dependencies |
|
||||
|
||||
**Full rebuild removes:**
|
||||
- `build-wasm/stamps/*` - All build stamps
|
||||
- `build-wasm/deps/*` - All dependency builds
|
||||
- `build-wasm/wxwidgets-universal` - wxWidgets build
|
||||
- `build-wasm/sysroot/*` - Installed headers/libraries
|
||||
- `build-wasm/kicad-pcbnew` - KiCad build
|
||||
|
||||
## Incremental Build System
|
||||
|
||||
The build system is optimized for fast development iteration:
|
||||
|
||||
### How It Works
|
||||
- **ccache**: Caches compiled objects by hashing preprocessed source
|
||||
- **wxWidgets**: `configure` runs once, `make` handles file-level dependencies
|
||||
- **KiCad**: CMake tracks dependencies, only recompiles changed files
|
||||
- **Asyncify**: Post-processing runs every build (~1 min, irreducible minimum)
|
||||
|
||||
### Performance
|
||||
|
||||
| Scenario | Time |
|
||||
|----------|------|
|
||||
| No changes | ~1.5 min |
|
||||
| Single file change (KiCad or wxWidgets) | ~1.5 min |
|
||||
| Full rebuild | ~10 min |
|
||||
|
||||
Most time is spent on asyncify post-processing which runs on every build.
|
||||
|
||||
### Debug vs Release
|
||||
|
||||
**Debug (default):**
|
||||
- Compiler: `-g -O0` (DWARF symbols, no optimization)
|
||||
- Linker: `-gsource-map` (JavaScript source maps)
|
||||
- Output: `~30-50MB` WASM with `.wasm.map` file
|
||||
- Use for: Development, debugging WASM exceptions
|
||||
|
||||
**Release:**
|
||||
- Compiler: `-O2` (optimized)
|
||||
- Output: `~15MB` WASM
|
||||
- Use for: Production deployment
|
||||
|
||||
## Stamp-based Caching
|
||||
|
||||
Dependency build progress is tracked with stamp files in `build-wasm/stamps/`:
|
||||
|
||||
```
|
||||
build-wasm/stamps/
|
||||
├── zstd.stamp
|
||||
├── protobuf.stamp
|
||||
├── freetype.stamp
|
||||
├── harfbuzz.stamp
|
||||
├── pixman.stamp
|
||||
├── cairo.stamp
|
||||
└── kicad-pcbnew.stamp
|
||||
```
|
||||
|
||||
**Note:** wxWidgets and KiCad use make/CMake for incremental builds instead of stamps.
|
||||
|
||||
**Clear specific component:** `rm build-wasm/stamps/zstd.stamp`
|
||||
**Clear all stamps:** `rm -f build-wasm/stamps/*.stamp`
|
||||
|
||||
After changing build flags (debug/release), use `--full` to force a complete rebuild.
|
||||
|
||||
## Build Scripts
|
||||
|
||||
| Script | Purpose |
|
||||
|--------|---------|
|
||||
| `docker/build.sh` | Host entry point (starts Docker, runs build) |
|
||||
| `scripts/kicad/build-pcbnew.sh` | KiCad PCBnew build (runs inside Docker) |
|
||||
| `scripts/build-wxuniversal-wasm.sh` | wxWidgets build |
|
||||
| `scripts/build-wasm-test.sh` | Build wxWidgets test apps |
|
||||
| `scripts/deps/build-all-deps.sh` | All dependencies |
|
||||
| `scripts/deps/build-*.sh` | Individual dependency builds |
|
||||
| `scripts/common/env.sh` | Environment setup |
|
||||
| `scripts/common/functions.sh` | Shared utilities |
|
||||
| `scripts/common/versions.sh` | Dependency versions |
|
||||
|
||||
## Build Times
|
||||
|
||||
| Component | Approximate Time |
|
||||
|-----------|-----------------|
|
||||
| Dependencies (all) | 20-60 minutes |
|
||||
| wxWidgets | 10-20 minutes |
|
||||
| KiCad PCBnew | 5-15 minutes |
|
||||
| **Total fresh build** | **1-2 hours** |
|
||||
|
||||
OpenCASCADE is the longest dependency to build (~30 minutes).
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Container freezes during build
|
||||
- Check Docker resource allocation (increase CPU/memory)
|
||||
- Reduce parallel jobs: `./docker/build.sh -j 4`
|
||||
- OpenCASCADE is resource-intensive; consider skipping with separate builds
|
||||
|
||||
### Build fails with missing dependency
|
||||
- Clear the specific stamp: `rm build-wasm/stamps/<dep>.stamp`
|
||||
- Re-run build
|
||||
|
||||
### Incremental build not picking up changes
|
||||
- For KiCad: use `--clean-kicad` to force rebuild
|
||||
- For wxWidgets: delete `build-wasm/wxwidgets-universal/Makefile` to force reconfigure
|
||||
|
||||
### WASM exception with numeric error (e.g., `3788888`)
|
||||
- Build with debug symbols (default): No `--release` flag
|
||||
- Check for `.wasm.map` file
|
||||
- Use Chrome DevTools to debug with source maps
|
||||
|
||||
### Clear build cache completely
|
||||
```bash
|
||||
docker volume rm docker_kicad-build-cache
|
||||
```
|
||||
|
||||
## WASM Compatibility Layer
|
||||
|
||||
The WASM port requires compatibility layers for browser execution:
|
||||
|
||||
| Directory | Purpose |
|
||||
|-----------|---------|
|
||||
| `wasm/kiplatform/` | Platform abstraction (app, UI, printing, etc.) |
|
||||
| `wasm/libcontext/` | Coroutine/fiber implementation for Asyncify |
|
||||
| `wasm/stubs/` | Stub implementations (libgit2, curl) |
|
||||
| `wasm/config/` | Build configuration headers |
|
||||
|
||||
## Emscripten Flags
|
||||
|
||||
Key flags used in the build:
|
||||
|
||||
```
|
||||
-pthread -sUSE_PTHREADS=1 # Threading support
|
||||
-sASYNCIFY=1 # Async coroutine support
|
||||
-sALLOW_MEMORY_GROWTH=1 # Dynamic memory
|
||||
-sINITIAL_MEMORY=256MB # Starting memory
|
||||
-sMAXIMUM_MEMORY=4GB # Maximum memory
|
||||
-sLEGACY_GL_EMULATION # OpenGL compatibility
|
||||
-sMAX_WEBGL_VERSION=2 # WebGL 2.0
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
After building, run the test suite:
|
||||
|
||||
```bash
|
||||
# Copy WASM output to test directory
|
||||
./tests/scripts/setup-kicad-wasm.sh
|
||||
|
||||
# Run KiCad tests
|
||||
cd tests
|
||||
npm install
|
||||
npm run test:kicad # Run Playwright tests
|
||||
```
|
||||
294
docs/debugging/DEBUG.md
Normal file
294
docs/debugging/DEBUG.md
Normal file
|
|
@ -0,0 +1,294 @@
|
|||
# Debugging guide — KiCad / wxWidgets WASM
|
||||
|
||||
A practical reference for debugging this project: the kinds of issues WASM +
|
||||
Asyncify + browser builds throw at you, the tools that actually work here, and
|
||||
the gotchas of our specific build pipeline. It is **not** a writeup of any one
|
||||
bug — for a concrete worked example see [§6](#6-a-worked-example) and the
|
||||
project memory.
|
||||
|
||||
If you're new to this codebase, read [§5 (project gotchas)](#5-project-specific-gotchas)
|
||||
first — most wasted hours come from not knowing how the split build and the
|
||||
shim layer behave.
|
||||
|
||||
---
|
||||
|
||||
## 1. Classes of issue we hit here
|
||||
|
||||
- **Engine-specific intolerance** — the same `pcbnew.wasm` runs in Firefox but
|
||||
not Chrome (or vice versa). Usually a V8-vs-SpiderMonkey difference in how an
|
||||
Asyncify-instrumented or very large function is handled.
|
||||
- **Silent stalls vs. hard crashes** — execution stops making progress with *no*
|
||||
exception, trap, or crash report. Distinguishing "crashed" from "hung" from
|
||||
"stalled" is half the battle (§2.6).
|
||||
- **Asyncify state problems** — unwind/rewind not completing, instrumentation on
|
||||
a function that shouldn't have it, or a function too large once instrumented.
|
||||
- **Shim/codegen coupling** — `inject-dyncall-shims.sh` patches Emscripten output
|
||||
by pattern; a flag change that alters codegen can silently break those patches.
|
||||
- **Tooling blind spots** — async console delivery, stripped name sections,
|
||||
Playwright hiding the renderer's stderr (§4).
|
||||
|
||||
---
|
||||
|
||||
## 2. Tools & techniques
|
||||
|
||||
### 2.1 Stub-bisection *(the workhorse)*
|
||||
Comment out / early-`return` a suspect call, rebuild, and observe a **binary
|
||||
survives-or-fails** outcome. This is the most reliable signal we have because it
|
||||
does **not** depend on reading logs (which lag — see §4). Narrow by halving:
|
||||
disable half the suspects, see which half flips the outcome.
|
||||
- *When:* you can localize a failure to "before/after some call."
|
||||
- *Caveat:* at `-O2`, dead-code elimination removes more around an early `return`
|
||||
than you intend — keep this in mind when a stub "fixes" too much.
|
||||
|
||||
### 2.2 `SHIM_DIAGNOSTICS=1` fast loop *(skip the rebuild)*
|
||||
The only host-side JS step is `inject-dyncall-shims.sh`. Re-run it on a pristine
|
||||
`pcbnew.js` while keeping the already-finalized/asyncified `pcbnew.wasm` — JS-only
|
||||
changes go from a multi-minute rebuild to seconds:
|
||||
```bash
|
||||
cp output/pcbnew.pristine.js output/pcbnew.js
|
||||
SHIM_DIAGNOSTICS=1 ./scripts/common/inject-dyncall-shims.sh output/pcbnew.js
|
||||
cd tests && npm run setup:kicad
|
||||
```
|
||||
See the `wasm-build-fast-iteration` project memory.
|
||||
|
||||
### 2.3 Logging-only diagnostics module (`scripts/common/shims/diagnostics.js`)
|
||||
Injected **only** when `SHIM_DIAGNOSTICS=1` (off by default, safe to leave in
|
||||
tree). Provides hooks that need no rebuild:
|
||||
- Asyncify lifecycle: `doRewind`, `handleSleep` (unwind/rewind markers).
|
||||
- Modal lifecycle.
|
||||
- A **WebGL call tracer** (did any GL call happen before the failure?).
|
||||
- A **dynCall tracer**: wraps the shim-bound `dynCall_ii`/`dynCall_vi` to log
|
||||
`ptr`, `getWasmTableEntry(ptr).name` (the function index), and a JS stack for
|
||||
rare/large table indices. Arm it at the main rewind to bound log volume.
|
||||
- Periodic asyncify-state monitor (catch "JS task queue stopped pumping").
|
||||
|
||||
Output is at `console.log` level (not error/warn). This is the JS-side tracer; the
|
||||
C++ source diagnostics are separate and flag-gated — see §2.9.
|
||||
|
||||
### 2.4 Symbolizing wasm function indices
|
||||
The loaded (post-asyncify) wasm has **no `name` section**, so V8/Firefox report
|
||||
bare function indices (`func[20736]`). The Asyncify pass **preserves function
|
||||
indices**, so a symbol map taken from the *pre-asyncify* wasm is still valid:
|
||||
```bash
|
||||
# the in-container wasm-opt is a STUB; use the real one
|
||||
/emsdk/upstream/bin/wasm-opt.real <pre-asyncify pcbnew.wasm> --symbolmap=/tmp/syms.map
|
||||
# then look up the index, e.g. 20736 -> PCB_EDIT_FRAME::setupUIConditions()
|
||||
```
|
||||
Generate the map from a build that still has names (the debug build's
|
||||
pre-asyncify wasm). See §5 on names/DWARF.
|
||||
|
||||
### 2.5 Cross-engine comparison
|
||||
Run the **same** diagnostics build in Firefox and Chrome and compare state at the
|
||||
**same dispatch point** (e.g. asyncify `state`/`currData` at the suspect
|
||||
`dynCall`). If both reach a point with identical state but only one proceeds, you
|
||||
have isolated an engine-specific bug and can stop looking for a logic error.
|
||||
|
||||
### 2.6 Crash vs. hang vs. stall
|
||||
A failure with no exception is not necessarily a crash. Find the renderer PID and
|
||||
inspect it:
|
||||
```bash
|
||||
ps -axo pid,%cpu,%mem,command | grep -i 'Google Chrome'
|
||||
sample <rendererPID> 3 # what is the main thread doing?
|
||||
```
|
||||
- **Idle in `CFRunLoop`/`mach_msg2_trap`, ~0% CPU** → a *stall* (event loop alive,
|
||||
but nothing scheduled to run). Not a deadlock.
|
||||
- **Blocked on a futex / `Atomics.wait`** → a pthread/lock issue.
|
||||
- **Spinning at 100%** → an infinite loop.
|
||||
- **Gone + a `.ips` report** → a real signal crash.
|
||||
|
||||
To see the **renderer's own stderr** and a real crash reason, launch system
|
||||
Chrome **outside Playwright** (Playwright forces `--disable-breakpad` and only
|
||||
pipes the *browser* process stderr): serve `tests/apps` with the COOP/COEP headers
|
||||
(`tests/serve.json`) and open the page in a normal Chrome with crash reporting on.
|
||||
On-load failures need no interaction to reproduce.
|
||||
|
||||
### 2.7 Build-flag diagnostics
|
||||
- `-sASSERTIONS=2` turns silent UB into named errors. **But** it changes
|
||||
Emscripten codegen and can break `inject-dyncall-shims.sh`'s `sed` patterns
|
||||
(causing a *different*, red-herring failure), and it implicitly enables
|
||||
`STACK_OVERFLOW_CHECK`, whose `___set_stack_limits` our host Asyncify pass
|
||||
strips → pair it with `-sSTACK_OVERFLOW_CHECK=0`. Prefer the §2.3 dynCall
|
||||
tracer on a normal build when you can.
|
||||
- `--pass-arg=asyncify-asserts` (added to the `wasm-opt --asyncify` invocation in
|
||||
`apply-asyncify.sh`) adds Asyncify state-machine runtime checks — use it to
|
||||
validate the removelist (a wrongly-excluded function that *does* unwind is
|
||||
otherwise silent corruption).
|
||||
|
||||
### 2.8 Isolated standalone probes
|
||||
`tests/apps/standalone/coroutine-pthread/` builds minimal C++ probes with the
|
||||
*real* libcontext + Asyncify + pthreads + DYNCALLS + the shim, run via
|
||||
`tests/e2e/coroutine-pthread.spec.ts`. Use these to reproduce a mechanism in
|
||||
isolation. **Reality check:** an isolated probe often *won't* reproduce a bug
|
||||
that needs the full app runtime — don't over-trust a green probe.
|
||||
|
||||
### 2.9 Source diagnostic logging flags (`--diag=`)
|
||||
The KiCad C++ source carries built-in diagnostic logging, **off by default**,
|
||||
enabled per category at build time:
|
||||
```bash
|
||||
./docker/build.sh --debug --diag=gal,coroutine,ctor # or: --diag=all
|
||||
```
|
||||
| `--diag=` value | covers |
|
||||
|---|---|
|
||||
| `gal` | `[DIAG_GAL]` — GAL/WebGL pipeline (paint, context create/lock, init) |
|
||||
| `coroutine` | `[WASM_FCONTEXT]` fiber switches + `[DIAG_TOOL]`/`[DIAG_DISP]` tool dispatch |
|
||||
| `ctor` | `[DIAG_CTOR]` — `PCB_EDIT_FRAME` startup milestones |
|
||||
|
||||
- Each value maps to a `-DKICAD_DIAG_*` define that gates the `KI_DIAG_*` macros
|
||||
in `kicad/include/kicad_wasm_diag.h`. All output goes to **stdout** → it shows
|
||||
as `[KICAD_OUT]` logs, never `[KICAD_ERR]` errors.
|
||||
- **Compile-time:** changing `--diag` changes `CMAKE_CXX_FLAGS`, so it forces a
|
||||
recompile (slow once per flag combo, then ccache-cached). Works with `--debug`
|
||||
or `--release`.
|
||||
- Separate from the JS shim tracer (§2.3), which stays `SHIM_DIAGNOSTICS`-gated.
|
||||
|
||||
---
|
||||
|
||||
## 3. Principles
|
||||
|
||||
1. **Reproduce cleanly first** — a stable engine-X-fails / engine-Y-passes
|
||||
baseline before changing anything.
|
||||
2. **Fix the build infra before iterating** — a flaky build wastes every
|
||||
subsequent experiment.
|
||||
3. **Narrow by bisection**, with binary outcomes, not by staring at logs.
|
||||
4. **Turn silent failures into named ones** (assertions, asyncify-asserts) or
|
||||
into a state comparison across engines.
|
||||
5. **Know the tooling's blind spots** (§4) before trusting what it shows you.
|
||||
|
||||
---
|
||||
|
||||
## 4. Tooling blind spots (read before trusting output)
|
||||
|
||||
- **Console is async** — `printf`/`console.*` from WASM reaches Playwright via
|
||||
CDP asynchronously; the *last delivered* line can lag the real failure point.
|
||||
Use stub-bisection for ground truth, not "the last log line."
|
||||
- **No name section** in the shipped wasm → bare indices (§2.4).
|
||||
- **Asyncify shifts code offsets** — DWARF line info is generated before the host
|
||||
Asyncify pass rewrites the code, so source-line mapping on the *shipped* wasm is
|
||||
stale. Asyncify *does* preserve function indices and names.
|
||||
- **Playwright hides the renderer** — forces `--disable-breakpad`, pipes only the
|
||||
browser process stderr (§2.6).
|
||||
- **macOS `sample`/`.ips`** see wasm frames as numeric offsets, not C++ names.
|
||||
|
||||
---
|
||||
|
||||
## 5. Project-specific gotchas
|
||||
|
||||
- **Split build.** `docker/build.sh` compiles + links inside Docker, but the
|
||||
in-container `wasm-opt` and `wasm-emscripten-finalize` are **stubbed** (they OOM
|
||||
on the large wasm). The real `wasm-emscripten-finalize` and
|
||||
`wasm-opt --asyncify` run **on the host** afterward (`apply-finalize.sh`,
|
||||
`apply-asyncify.sh`). Real binary: `…/upstream/bin/wasm-opt.real`.
|
||||
- **Per-branch Docker volumes.** The compose project name is derived from the git
|
||||
branch, so each branch has its own build-cache volume/container. Switching
|
||||
optimization level (`-O1`↔`-O2`) busts ccache and forces a full recompile.
|
||||
- **COOP/COEP.** SharedArrayBuffer/pthreads need cross-origin isolation headers;
|
||||
serve `tests/apps` with `tests/serve.json` (`npx serve apps -c ../serve.json`).
|
||||
- **The shim layer.** `inject-dyncall-shims.sh` binds bare `dynCall_<sig>` to the
|
||||
real `DYNCALLS=1` exports and patches several Emscripten empty-stub callbacks by
|
||||
`sed` pattern — so codegen-changing flags can silently break it.
|
||||
- **Names / DWARF, concretely.** Neither build keeps a `name` section in the
|
||||
*runtime* wasm (it carries only `external_debug_info` + `target_features`). The
|
||||
**debug** build (`-O1 -g -gseparate-dwarf`) puts full DWARF in a ~1.5 GB
|
||||
`pcbnew.wasm.debug.wasm` sidecar (loaded on demand by DevTools' C/C++ extension);
|
||||
the **release** build (`-O2`, no `-g`) has neither names nor DWARF. So readable
|
||||
symbols come from the debug build's DWARF / the §2.4 symbol map, not from the
|
||||
shipped binary.
|
||||
|
||||
---
|
||||
|
||||
## 6. A worked example
|
||||
|
||||
The **Chrome-only startup stall** (May 2026): V8 could not run the
|
||||
Asyncify-*instrumented* `PCB_EDIT_FRAME::setupUIConditions()` (a huge function
|
||||
that never actually unwinds) when it was invoked from the Asyncify-rewound
|
||||
constructor stack — a silent stall, not a crash; Firefox ran the identical wasm
|
||||
fine. Found with stub-bisection (§2.1) + the dynCall tracer (§2.3) + symbol map
|
||||
(§2.4) + cross-engine state comparison (§2.5) + `sample` (§2.6).
|
||||
|
||||
A **second instance** of the same family (May 28, 2026) hit the line-drawing
|
||||
coroutine: V8 stalled at the first instruction of the asyncify-instrumented
|
||||
`libcontext::wasm_fcontext_entry` trampoline when a new fiber for
|
||||
`pcbnew.InteractiveDrawing.line` was entered. The `[DIAG_TOOL]` log showed
|
||||
the activate dispatching and `[WASM_FCONTEXT]` showed `jump-swap` completing,
|
||||
but `entry-call` (logged on the new fiber's first statement) never fired —
|
||||
the tool's button visually never toggled, and tests on headed Chrome **could
|
||||
not reproduce** it (same wasm, different cumulative asyncify state). Trying
|
||||
to add the trampoline / `COROUTINE::callerStub` to `ASYNCIFY_REMOVE` broke
|
||||
runtime because both functions sit ON the suspend chain (their callees
|
||||
`emscripten_fiber_swap` / suspendable tool bodies), so removing them from
|
||||
instrumentation orphans the rewind — `null function` / `ASM_CONSTS` errors.
|
||||
|
||||
The systemic fix (see [§7](#7-debug-vs-production-builds)) is now **committed
|
||||
default**: run `wasm-opt -O2` as a separate pass after `--asyncify` in
|
||||
`scripts/common/apply-asyncify.sh`. This shrinks every instrumented function
|
||||
back under V8's threshold, including the coroutine trampolines that can't be
|
||||
removelist'd. The legacy `ASYNCIFY_REMOVE` entries (`setupUIConditions`
|
||||
etc.) are kept as a redundant safety net — under `-O2` they're no longer
|
||||
required but are harmless.
|
||||
|
||||
Details: the `chrome-asyncify-rewind-crash` and `bundle-size-asyncify-optimization`
|
||||
project memories, and git history of `apply-asyncify.sh`.
|
||||
|
||||
---
|
||||
|
||||
## 7. Debug vs. production builds
|
||||
|
||||
The committed default is the **debug** build (compiled `-g -gseparate-dwarf`,
|
||||
DWARF sidecar) with `apply-asyncify.sh` running `wasm-opt --asyncify` followed
|
||||
by `wasm-opt -O2` (May 28, 2026). Result: ~187 MB wasm / ~65 MB gzip, full
|
||||
source-level debugging. Switch to release (`./docker/build.sh` without
|
||||
`--debug`) for an even smaller shippable build with no DWARF.
|
||||
|
||||
### What the knobs do
|
||||
Two independent knobs:
|
||||
- **`-g` (debug info)** — whether a source map exists at all. Debug =
|
||||
`-g -gseparate-dwarf` (DWARF sidecar); release = none.
|
||||
- **`-O` (optimization)** — how much the code is rewritten. This is what actually
|
||||
fixes the "function too big for V8" class of bug, because Asyncify emits
|
||||
deliberately verbose instrumentation (spills every live local) and **relies on
|
||||
the optimizer to coalesce it back down**. The Emscripten/Binaryen docs are
|
||||
emphatic that you must optimize when using Asyncify.
|
||||
|
||||
### How the build flow uses both
|
||||
1. **Docker compile + link** (`./docker/build.sh [--debug]`) produces an
|
||||
un-finalized, un-asyncified wasm. `--debug` controls only `-g`; the
|
||||
`-O2` optimisation level is set unconditionally at compile time.
|
||||
2. **Host post-processing** (`scripts/common/apply-finalize.sh` then
|
||||
`scripts/common/apply-asyncify.sh`):
|
||||
- `wasm-opt --asyncify` instruments suspendable functions.
|
||||
- `wasm-opt -O2` (added May 28, 2026) shrinks every instrumented
|
||||
function back under V8's per-function locals limit, fixing the
|
||||
"Chrome-only stall on coroutine entry" class of bug systemically.
|
||||
Without this pass, large asyncify-instrumented functions like
|
||||
`PCB_EDIT_FRAME::setupUIConditions()` or libcontext's
|
||||
`wasm_fcontext_entry` silently stall in Chrome's V8 even though
|
||||
Firefox runs them fine. The two passes are run separately so peak
|
||||
RAM stays ~10–15 GB (one heavy `wasm-opt` at a time).
|
||||
3. **Shim injection** (`scripts/common/inject-dyncall-shims.sh`) adds the
|
||||
asyncify-aware dynCall bindings and the nested-asyncify `handleSleep`
|
||||
wrapper to `pcbnew.js`.
|
||||
|
||||
### The `ASYNCIFY_REMOVE` list (in `apply-asyncify.sh`)
|
||||
With `-O2` after asyncify, no large function should exceed V8's limit anymore,
|
||||
so the removelist is mostly a redundant safety net. Two situations still
|
||||
warrant adding to it:
|
||||
- A function whose subtree does **not** asyncify-suspend (so removing it is
|
||||
always safe) and that you're confident never needs to participate in
|
||||
unwind/rewind. Example: `setupUIConditions()` — registers handlers, never
|
||||
yields.
|
||||
- **Don't** add functions on the asyncify-suspend chain (coroutine
|
||||
trampolines, anything calling `emscripten_fiber_swap` / `EM_ASYNC_JS`):
|
||||
removing them orphans the rewind path and you get `null function` /
|
||||
`ASM_CONSTS[code] is not a function` at runtime.
|
||||
|
||||
### Measured result (May 2026)
|
||||
| build | raw wasm | gzip | source-level debugging |
|
||||
|---|---|---|---|
|
||||
| debug, asyncify only (old default) | 338 MB | 137 MB | full (DWARF sidecar) |
|
||||
| debug + asyncify + `-O2` (current default) | **187 MB** | **65 MB** | full (DWARF sidecar) |
|
||||
| release + asyncify + `-O2` | smaller still | — | none |
|
||||
|
||||
The optimized build passes Chrome **and** Firefox `select draw lines` e2e,
|
||||
fixes the user-reported "line tool doesn't toggle in real Chrome" stall, and
|
||||
makes the test load+run ~2× faster (smaller wasm parses faster). Tradeoff:
|
||||
each build now spends an extra ~10 minutes on the `-O2` pass.
|
||||
40
docs/debugging/learning.md
Normal file
40
docs/debugging/learning.md
Normal file
|
|
@ -0,0 +1,40 @@
|
|||
# Learning Notes
|
||||
|
||||
## Asyncify and Consecutive Modal Dialogs
|
||||
|
||||
### Problem
|
||||
When multiple modal dialogs are triggered in quick succession, Asyncify operations can overlap causing crashes:
|
||||
- "indirect call to null"
|
||||
- "func is not a function"
|
||||
- "index out of bounds"
|
||||
|
||||
### Root Cause
|
||||
Per Emscripten docs: "It is not safe to start an async operation while another is already running."
|
||||
|
||||
When the first modal completes:
|
||||
1. Asyncify begins rewinding the C++ stack
|
||||
2. C++ code triggers second modal before rewind completes
|
||||
3. Second modal's Asyncify operation conflicts with first modal's cleanup
|
||||
4. Asyncify state corruption occurs
|
||||
|
||||
### Key Insight
|
||||
**You cannot use ANY Asyncify mechanism to wait** - neither `EM_ASYNC_JS` await nor `emscripten_sleep()` - while another Asyncify operation is cleaning up. Both use Asyncify internally and cause the same conflict.
|
||||
|
||||
### Solution Pattern
|
||||
1. Use a **global lock** to track when Asyncify is busy
|
||||
2. Check lock with **synchronous JS** (`EM_JS`, not `EM_ASYNC_JS`) - this doesn't use Asyncify
|
||||
3. If locked, **return immediately** instead of waiting
|
||||
4. Release lock via **double setTimeout(0)** to ensure Asyncify fully completes before allowing new operations
|
||||
|
||||
```javascript
|
||||
// Good: Synchronous check (no Asyncify)
|
||||
EM_JS(int, isLocked, (), { return Module._locked ? 1 : 0; });
|
||||
|
||||
// Bad: This uses Asyncify and will cause conflicts
|
||||
while (isLocked()) {
|
||||
emscripten_sleep(10); // Uses Asyncify!
|
||||
}
|
||||
```
|
||||
|
||||
### File Reference
|
||||
`wxwidgets/src/wasm/dialog.cpp` - Modal implementation with lock mechanism
|
||||
154
docs/features/archive/ipc-api/ipc-api.md
Normal file
154
docs/features/archive/ipc-api/ipc-api.md
Normal file
|
|
@ -0,0 +1,154 @@
|
|||
# KiCad IPC API - WASM Fork Changes
|
||||
|
||||
> **ARCHIVED — open cleanup TODO, not yet actioned.** Describes `#ifdef KICAD_IPC_API` guards added to ~18 KiCad source files that could be reverted to reduce fork diff. Archiving this doc does **not** perform the revert — that remains a separate task.
|
||||
|
||||
## Overview
|
||||
|
||||
We added `#ifdef KICAD_IPC_API` guards to 18 KiCad source files. These guards wrap:
|
||||
- `#include` statements for protobuf/API headers
|
||||
- `Serialize()` and `Deserialize()` methods
|
||||
|
||||
**Key insight:** Since our build uses `KICAD_IPC_API=ON` (line 251 of `scripts/kicad/build-pcbnew.sh`), these guards don't actually disable any code - everything compiles. The guards can be safely reverted to reduce fork divergence.
|
||||
|
||||
## Modified Files
|
||||
|
||||
| File | What We Guarded |
|
||||
|------|-----------------|
|
||||
| `common/eda_shape.cpp` | `#include` for API headers, `Serialize()`, `Deserialize()` |
|
||||
| `common/eda_text.cpp` | `#include` for API headers, `Serialize()`, `Deserialize()` |
|
||||
| `common/netclass.cpp` | `#include` for API headers, `Serialize()`, `Deserialize()` |
|
||||
| `include/api/api_utils.h` | Entire namespace content (utility functions) |
|
||||
| `pcbnew/api/api_pcb_utils.h` | Entire file content |
|
||||
| `pcbnew/board_connected_item.cpp` | `#include` for API headers, `Serialize()`, `Deserialize()` |
|
||||
| `pcbnew/board_stackup_manager/board_stackup.cpp` | `#include` for API headers, `Serialize()`, `Deserialize()` |
|
||||
| `pcbnew/footprint.cpp` | `#include` for API headers, `Serialize()`, `Deserialize()` |
|
||||
| `pcbnew/pad.cpp` | `#include` for API headers, `Serialize()`, `Deserialize()` |
|
||||
| `pcbnew/padstack.cpp` | `#include` for API headers, `Serialize()`, `Deserialize()` |
|
||||
| `pcbnew/pcb_dimension.cpp` | `#include` for API headers, `Serialize()`, `Deserialize()` |
|
||||
| `pcbnew/pcb_field.cpp` | `#include` for API headers, `Serialize()`, `Deserialize()` |
|
||||
| `pcbnew/pcb_group.cpp` | `#include` for API headers, `Serialize()`, `Deserialize()` |
|
||||
| `pcbnew/pcb_shape.cpp` | `#include` for API headers, `Serialize()`, `Deserialize()` |
|
||||
| `pcbnew/pcb_text.cpp` | `#include` for API headers, `Serialize()`, `Deserialize()` |
|
||||
| `pcbnew/pcb_textbox.cpp` | `#include` for API headers, `Serialize()`, `Deserialize()` |
|
||||
| `pcbnew/pcb_track.cpp` | `#include` for API headers, `Serialize()`, `Deserialize()` |
|
||||
| `pcbnew/zone.cpp` | `#include` for API headers, `Serialize()`, `Deserialize()` |
|
||||
|
||||
## Example of Our Changes
|
||||
|
||||
**Before (upstream):**
|
||||
```cpp
|
||||
#include <api/api_enums.h>
|
||||
#include <api/api_utils.h>
|
||||
#include <api/board/board_types.pb.h>
|
||||
|
||||
void PAD::Serialize( google::protobuf::Any &aContainer ) const
|
||||
{
|
||||
// ... serialization code
|
||||
}
|
||||
```
|
||||
|
||||
**After (our fork):**
|
||||
```cpp
|
||||
#ifdef KICAD_IPC_API
|
||||
#include <api/api_enums.h>
|
||||
#include <api/api_utils.h>
|
||||
#include <api/board/board_types.pb.h>
|
||||
#endif
|
||||
|
||||
#ifdef KICAD_IPC_API
|
||||
void PAD::Serialize( google::protobuf::Any &aContainer ) const
|
||||
{
|
||||
// ... serialization code
|
||||
}
|
||||
#endif
|
||||
```
|
||||
|
||||
## Why We Added These Guards
|
||||
|
||||
Originally added to allow building with `KICAD_IPC_API=OFF`, which would:
|
||||
- Skip protobuf dependency
|
||||
- Exclude serialization methods that depend on protobuf types
|
||||
|
||||
However, our current build has `KICAD_IPC_API=ON`, making these guards unnecessary.
|
||||
|
||||
## How to Revert
|
||||
|
||||
To revert these files to upstream (base commit `4bfed3f174`):
|
||||
|
||||
```bash
|
||||
cd kicad
|
||||
git checkout 4bfed3f174 -- \
|
||||
common/eda_shape.cpp \
|
||||
common/eda_text.cpp \
|
||||
common/netclass.cpp \
|
||||
include/api/api_utils.h \
|
||||
pcbnew/api/api_pcb_utils.h \
|
||||
pcbnew/board_connected_item.cpp \
|
||||
pcbnew/board_stackup_manager/board_stackup.cpp \
|
||||
pcbnew/footprint.cpp \
|
||||
pcbnew/pad.cpp \
|
||||
pcbnew/padstack.cpp \
|
||||
pcbnew/pcb_dimension.cpp \
|
||||
pcbnew/pcb_field.cpp \
|
||||
pcbnew/pcb_group.cpp \
|
||||
pcbnew/pcb_shape.cpp \
|
||||
pcbnew/pcb_text.cpp \
|
||||
pcbnew/pcb_textbox.cpp \
|
||||
pcbnew/pcb_track.cpp \
|
||||
pcbnew/zone.cpp
|
||||
```
|
||||
|
||||
Then rebuild:
|
||||
```bash
|
||||
./docker/build.sh --clean-kicad
|
||||
```
|
||||
|
||||
## Note: toolbars_pcb_editor.cpp
|
||||
|
||||
This file has `#ifdef KICAD_SCRIPTING` guards (not IPC API guards):
|
||||
|
||||
```cpp
|
||||
#ifdef KICAD_SCRIPTING
|
||||
#include "../scripting/python_scripting.h"
|
||||
#endif
|
||||
...
|
||||
#ifdef KICAD_SCRIPTING
|
||||
bool scriptingAvailable = SCRIPTING::IsWxAvailable();
|
||||
#else
|
||||
bool scriptingAvailable = false;
|
||||
#endif
|
||||
```
|
||||
|
||||
**Cannot be fully reverted** because:
|
||||
- `KICAD_SCRIPTING=OFF` for WASM builds (CMakeLists.txt lines 135-138)
|
||||
- Without guards, build fails (python_scripting.h not built, SCRIPTING class doesn't exist)
|
||||
|
||||
This is a separate concern from the IPC API and is documented here for completeness.
|
||||
|
||||
## Impact of Reversion
|
||||
|
||||
- **Fork diff reduction:** 18 fewer modified files
|
||||
- **Build behavior:** No change (KICAD_IPC_API=ON, code compiles identically)
|
||||
- **Functionality:** No change (serialization methods still available)
|
||||
|
||||
## Related: IPC API Architecture
|
||||
|
||||
KiCad's IPC API uses:
|
||||
- **Protobuf messages** for request/response serialization
|
||||
- **NNG sockets** for transport (stubbed in WASM - see `wasm/stubs/nng_stub.c`)
|
||||
- **API_HANDLER_PCB** class for handling requests (instantiated in PCB_EDIT_FRAME)
|
||||
|
||||
Our stub at `wasm/stubs/api_plugin_stub.cpp` provides no-op implementations for:
|
||||
- `KICAD_API_SERVER::Start()`, `Stop()`, `Running()`
|
||||
- `KICAD_API_SERVER::RegisterHandler()`, `DeregisterHandler()`
|
||||
|
||||
## Future: JavaScript Bridge
|
||||
|
||||
To expose the IPC API to JavaScript (optional enhancement):
|
||||
|
||||
1. Modify `api_plugin_stub.cpp` to store registered handlers
|
||||
2. Create Embind bindings in `wasm/bindings/api_bridge.cpp`
|
||||
3. Expose `KiCadApi_HandleRequest(protobufData)` function
|
||||
4. Use `protobuf.js` on JS side for message creation/parsing
|
||||
|
||||
This would enable JS automation without modifying any KiCad source files.
|
||||
264
docs/features/archive/webgl/0001-opengl-gal-webgl-strategy.md
Normal file
264
docs/features/archive/webgl/0001-opengl-gal-webgl-strategy.md
Normal file
|
|
@ -0,0 +1,264 @@
|
|||
# OpenGL, GAL, and WebGL Strategy for KiCad WASM
|
||||
|
||||
> **ARCHIVED / HISTORICAL** — the WebGL-GAL backend described here was implemented (see `kicad/common/gal/webgl/`, ~22.5k lines). Live test docs: [`tests/gal-regression/README.md`](../../../../tests/gal-regression/README.md). Kept for design rationale.
|
||||
|
||||
## Summary
|
||||
|
||||
This document analyzes KiCad's graphics architecture and evaluates strategies for WebGL rendering in the WASM build.
|
||||
|
||||
## Current Implementation: Emscripten LEGACY_GL_EMULATION
|
||||
|
||||
We currently use Emscripten's legacy OpenGL emulation:
|
||||
|
||||
```bash
|
||||
-sLEGACY_GL_EMULATION -sMAX_WEBGL_VERSION=2
|
||||
```
|
||||
|
||||
**How it works:**
|
||||
|
||||
```
|
||||
KiCad/wxWidgets C++ (glBegin, glVertex, glColor, etc.)
|
||||
│
|
||||
▼
|
||||
gl_immediate_shim.js (our fix for color-per-vertex bug)
|
||||
│
|
||||
▼
|
||||
Emscripten GLImmediate (translates legacy GL → shaders + VBOs)
|
||||
│
|
||||
▼
|
||||
WebGL 2.0 (ES 3.0 compatible, shader-based)
|
||||
```
|
||||
|
||||
**Key components:**
|
||||
|
||||
| Component | File | Purpose |
|
||||
|-----------|------|---------|
|
||||
| wxGLCanvas | `wxwidgets/src/wasm/glcanvas.cpp` | Creates WebGL contexts on dedicated HTML5 canvases |
|
||||
| GL Shim | `wasm/shims/gl_immediate_shim.js` | Fixes color-per-vertex bug in GLImmediate |
|
||||
| GLU Tesselator | `wasm/stubs/glu_wasm_impl.cpp` | Polygon triangulation using earcut (GLU not available in WebGL) |
|
||||
| GLEW Stub | `wasm/cmake/FindGLEW.cmake` | Stubs out GLEW (not needed with Emscripten) |
|
||||
| kiglew.h | `kicad/include/gal/opengl/kiglew.h` | Maps double→float, stubs unsupported functions |
|
||||
|
||||
---
|
||||
|
||||
## GLU Status
|
||||
|
||||
**KiCad uses GLU** for polygon tesselation (`gluTess*` functions for zones, complex shapes).
|
||||
|
||||
**We already solved this** with `/wasm/stubs/glu_wasm_impl.cpp` which implements the GLU tesselator API using KiCad's earcut algorithm. This works with both LEGACY_GL_EMULATION and would work with pure ES3.
|
||||
|
||||
GLU is NOT a blocker.
|
||||
|
||||
---
|
||||
|
||||
## Where KiCad Uses Legacy OpenGL
|
||||
|
||||
Two separate systems use legacy OpenGL:
|
||||
|
||||
### 1. OPENGL_GAL (2D Rendering)
|
||||
|
||||
Used for schematic and PCB editors.
|
||||
|
||||
**Files:**
|
||||
- `kicad/common/gal/opengl/opengl_gal.cpp` - Main implementation
|
||||
- `kicad/common/gal/opengl/gpu_manager.cpp` - VBO management
|
||||
- `kicad/common/gal/opengl/opengl_compositor.cpp` - Framebuffer compositing
|
||||
- `kicad/common/gal/opengl/antialiasing.cpp` - AA effects
|
||||
|
||||
**Legacy GL usage in opengl_gal.cpp:**
|
||||
- Lines 1557-1605: Bitmap rendering with `glBegin(GL_QUADS)`
|
||||
- Lines 2690-2755: Cursor drawing with `glBegin(GL_LINES)`
|
||||
- Lines 566-629, 1558-1602, 2704-2751: Matrix stack operations (`glPushMatrix`, `glPopMatrix`, `glMatrixMode`)
|
||||
|
||||
### 2. 3D Viewer (3D Rendering)
|
||||
|
||||
Separate from GAL, used for 3D board visualization.
|
||||
|
||||
**Files using legacy GL:**
|
||||
- `kicad/3d-viewer/3d_rendering/opengl/render_3d_opengl.cpp`
|
||||
- `kicad/3d-viewer/3d_rendering/opengl/opengl_utils.cpp`
|
||||
- `kicad/3d-viewer/3d_rendering/opengl/layer_triangles.cpp`
|
||||
- `kicad/3d-viewer/3d_rendering/opengl/3d_spheres_gizmo.cpp`
|
||||
- `kicad/3d-viewer/3d_rendering/opengl/3d_model.cpp`
|
||||
- `kicad/3d-viewer/3d_model_viewer/eda_3d_model_viewer.cpp`
|
||||
- `kicad/3d-viewer/3d_canvas/eda_3d_canvas_pivot.cpp`
|
||||
|
||||
---
|
||||
|
||||
## If We Cover GAL, Is That Enough?
|
||||
|
||||
**For 2D editing (schematic + PCB): YES**
|
||||
|
||||
The entire 2D rendering pipeline goes through GAL. A working GAL backend enables full schematic and PCB editing.
|
||||
|
||||
**For 3D viewer: NO**
|
||||
|
||||
The 3D viewer is a separate rendering system. It could be disabled/stubbed initially and added later.
|
||||
|
||||
---
|
||||
|
||||
## The GAL API
|
||||
|
||||
GAL (Graphics Abstraction Layer) is a clean 2D drawing interface defined in `kicad/include/gal/graphics_abstraction_layer.h`.
|
||||
|
||||
**NO raw OpenGL is exposed.** The API consists of ~40 virtual methods:
|
||||
|
||||
```cpp
|
||||
// Drawing primitives
|
||||
virtual void DrawLine(const VECTOR2D& start, const VECTOR2D& end);
|
||||
virtual void DrawSegment(const VECTOR2D& start, const VECTOR2D& end, double width);
|
||||
virtual void DrawPolyline(const std::vector<VECTOR2D>& points);
|
||||
virtual void DrawCircle(const VECTOR2D& center, double radius);
|
||||
virtual void DrawArc(const VECTOR2D& center, double radius, const EDA_ANGLE& start, const EDA_ANGLE& angle);
|
||||
virtual void DrawRectangle(const VECTOR2D& start, const VECTOR2D& end);
|
||||
virtual void DrawPolygon(const SHAPE_POLY_SET& polySet);
|
||||
virtual void DrawBitmap(const BITMAP_BASE& bitmap, double alpha);
|
||||
virtual void DrawGlyph(const KIFONT::GLYPH& glyph);
|
||||
|
||||
// Attributes
|
||||
virtual void SetFillColor(const COLOR4D& color);
|
||||
virtual void SetStrokeColor(const COLOR4D& color);
|
||||
virtual void SetLineWidth(float width);
|
||||
virtual void SetIsFill(bool enabled);
|
||||
virtual void SetIsStroke(bool enabled);
|
||||
virtual void SetLayerDepth(double depth);
|
||||
|
||||
// Transforms (matrix stack)
|
||||
virtual void Save();
|
||||
virtual void Restore();
|
||||
virtual void Transform(const MATRIX3x3D& matrix);
|
||||
virtual void Rotate(double angle);
|
||||
virtual void Translate(const VECTOR2D& translation);
|
||||
virtual void Scale(const VECTOR2D& scale);
|
||||
|
||||
// Rendering control
|
||||
virtual void BeginDrawing();
|
||||
virtual void EndDrawing();
|
||||
virtual void SetTarget(RENDER_TARGET target);
|
||||
virtual void ClearTarget(RENDER_TARGET target);
|
||||
virtual void ClearScreen();
|
||||
|
||||
// Grid and cursor
|
||||
virtual void DrawGrid();
|
||||
virtual void DrawCursor(const VECTOR2D& position);
|
||||
```
|
||||
|
||||
**Existing GAL implementations:**
|
||||
|
||||
| Class | File | Purpose |
|
||||
|-------|------|---------|
|
||||
| `OPENGL_GAL` | `kicad/include/gal/opengl/opengl_gal.h` | OpenGL rendering (uses legacy GL internally) |
|
||||
| `CAIRO_GAL` | `kicad/include/gal/cairo/cairo_gal.h` | Cairo software rendering |
|
||||
| `CALLBACK_GAL` | `kicad/include/callback_gal.h` | Hit testing, no actual rendering |
|
||||
|
||||
---
|
||||
|
||||
## What Breaks with FULL_ES3 (No Legacy Emulation)
|
||||
|
||||
If we removed `LEGACY_GL_EMULATION`:
|
||||
|
||||
| Feature | Used In | Status in FULL_ES3 |
|
||||
|---------|---------|-------------------|
|
||||
| `gluTess*` (tesselation) | Polygon rendering | **Our stub works** |
|
||||
| `glBegin/glEnd` | Bitmap quads, cursor | Not available |
|
||||
| `glVertex/glColor` | Immediate mode drawing | Not available |
|
||||
| `glPushMatrix/glPopMatrix` | Transformations | Not available |
|
||||
| `glMatrixMode` | Matrix switching | Not available |
|
||||
| `GL_QUADS` | Bitmap rendering | Not available (only triangles) |
|
||||
| `glEnableClientState` | Legacy vertex arrays | Not available |
|
||||
|
||||
---
|
||||
|
||||
## Strategy Options
|
||||
|
||||
### Option 1: Keep LEGACY_GL_EMULATION (Current)
|
||||
|
||||
**Pros:**
|
||||
- Works now
|
||||
- Minimal code changes to KiCad
|
||||
- All immediate mode functions available
|
||||
|
||||
**Cons:**
|
||||
- ~200KB binary overhead from GLImmediate
|
||||
- Runtime overhead from emulation
|
||||
- Color-per-vertex bug requires our shim
|
||||
- Some edge cases may not work
|
||||
|
||||
**Build flags:**
|
||||
```bash
|
||||
-sLEGACY_GL_EMULATION -sMAX_WEBGL_VERSION=2
|
||||
```
|
||||
|
||||
### Option 2: Create WEBGL_GAL (New Backend)
|
||||
|
||||
Create a new GAL implementation that uses pure WebGL 2.0 (ES 3.0) without legacy emulation.
|
||||
|
||||
**Pros:**
|
||||
- No emulation overhead
|
||||
- Smaller binary
|
||||
- Cleaner, more maintainable
|
||||
- Better performance
|
||||
|
||||
**Cons:**
|
||||
- Development effort (~2-3 weeks)
|
||||
- Need to maintain separate backend
|
||||
|
||||
**Estimated scope:**
|
||||
```
|
||||
New class: WEBGL_GAL : public GAL
|
||||
|
||||
Files needed:
|
||||
├── webgl_gal.h (~300 lines)
|
||||
├── webgl_gal.cpp (~2500 lines)
|
||||
├── webgl_shaders.cpp (~500 lines)
|
||||
└── webgl_compositor.cpp (~500 lines)
|
||||
|
||||
Total: ~3500-4000 lines
|
||||
```
|
||||
|
||||
**Why it's feasible:**
|
||||
- GAL API is clean - no raw GL leaks through
|
||||
- CAIRO_GAL proves it works (~2500 lines)
|
||||
- All drawing is 2D with simple primitives
|
||||
- KiCad's shaders already exist and work in WebGL
|
||||
|
||||
### Option 3: Cairo-only (Software Rendering)
|
||||
|
||||
Use CAIRO_GAL exclusively, render to HTML5 2D canvas.
|
||||
|
||||
**Pros:**
|
||||
- Already exists
|
||||
- No WebGL needed
|
||||
- Works everywhere
|
||||
|
||||
**Cons:**
|
||||
- Slow (CPU-only)
|
||||
- May struggle with complex boards
|
||||
- No hardware acceleration
|
||||
|
||||
---
|
||||
|
||||
## Recommendation
|
||||
|
||||
**Short-term:** Keep LEGACY_GL_EMULATION - it works and allows rapid development.
|
||||
|
||||
**Medium-term:** Create WEBGL_GAL - cleaner architecture, better performance, removes emulation hacks.
|
||||
|
||||
**For 3D viewer:** Disable initially, add later as separate effort.
|
||||
|
||||
---
|
||||
|
||||
## Key Files Reference
|
||||
|
||||
| Purpose | File |
|
||||
|---------|------|
|
||||
| Build flags | `scripts/common/env.sh` |
|
||||
| wxGLCanvas WASM | `wxwidgets/src/wasm/glcanvas.cpp` |
|
||||
| GL immediate shim | `wasm/shims/gl_immediate_shim.js` |
|
||||
| GLU tesselator | `wasm/stubs/glu_wasm_impl.cpp` |
|
||||
| KiCad GL compat | `kicad/include/gal/opengl/kiglew.h` |
|
||||
| KiCad shaders | `kicad/common/gal/shaders/kicad_*.glsl` |
|
||||
| GAL base class | `kicad/include/gal/graphics_abstraction_layer.h` |
|
||||
| OPENGL_GAL | `kicad/include/gal/opengl/opengl_gal.h` |
|
||||
| CAIRO_GAL | `kicad/include/gal/cairo/cairo_gal.h` |
|
||||
| GL documentation | `tests/GL_README.md` |
|
||||
289
docs/features/archive/webgl/0002-gal-native-test-architecture.md
Normal file
289
docs/features/archive/webgl/0002-gal-native-test-architecture.md
Normal file
|
|
@ -0,0 +1,289 @@
|
|||
# GAL Native Test Harness Architecture
|
||||
|
||||
> **ARCHIVED / HISTORICAL** — the WebGL-GAL backend described here was implemented (see `kicad/common/gal/webgl/`). Live test docs: [`tests/gal-regression/README.md`](../../../../tests/gal-regression/README.md). Kept for design rationale.
|
||||
|
||||
## Overview
|
||||
|
||||
The GAL native test harness is a standalone macOS application that compiles KiCad's actual `OPENGL_GAL` rendering engine against system wxWidgets. It generates baseline PNG screenshots for visual regression testing of WebGL rendering in the WASM build.
|
||||
|
||||
**Purpose**: Compare native OpenGL rendering (ground truth) against WebGL rendering in the browser to detect visual regressions.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ GAL Native Test │
|
||||
├─────────────────────────────────────────────────────────────────────┤
|
||||
│ gal_native_test.cpp (wxApp + wxFrame) │
|
||||
│ └─ Creates OPENGL_GAL on wxGLCanvas │
|
||||
│ └─ Runs 11 test scenarios │
|
||||
│ └─ Captures FBO → PNG for each │
|
||||
├─────────────────────────────────────────────────────────────────────┤
|
||||
│ kicad_stubs.cpp │ gal_test_accessor.cpp │
|
||||
│ • PGM_BASE singleton │ • Template accessor for │
|
||||
│ • ADVANCED_CFG │ private OPENGL_GAL members │
|
||||
│ • KIFONT stubs │ • FBO reading │
|
||||
│ • Observable stubs │ │
|
||||
├─────────────────────────────────────────────────────────────────────┤
|
||||
│ KiCad OPENGL_GAL (from submodule) │
|
||||
│ kicad/common/gal/opengl/*.cpp (18 source files) │
|
||||
├─────────────────────────────────────────────────────────────────────┤
|
||||
│ System Dependencies │
|
||||
│ wxWidgets (via homebrew) │ GLEW │ OpenGL │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## GAL Code Source
|
||||
|
||||
The test harness compiles **KiCad's actual OPENGL_GAL** from the kicad submodule:
|
||||
|
||||
### Source Files (`/kicad/common/gal/opengl/`)
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `opengl_gal.cpp` | Main GAL implementation - drawing primitives |
|
||||
| `opengl_compositor.cpp` | FBO management, layer compositing |
|
||||
| `antialiasing.cpp` | SMAA antialiasing implementation |
|
||||
| `vertex_manager.cpp` | Vertex buffer accumulation |
|
||||
| `vertex_item.cpp` | Individual vertex items |
|
||||
| `vertex_container.cpp` | Vertex storage base class |
|
||||
| `cached_container.cpp` | Cached geometry container |
|
||||
| `cached_container_gpu.cpp` | GPU-resident cached geometry |
|
||||
| `cached_container_ram.cpp` | RAM-backed cached geometry |
|
||||
| `noncached_container.cpp` | Per-frame geometry |
|
||||
| `gpu_manager.cpp` | GPU buffer management |
|
||||
| `shader.cpp` | GLSL shader compilation/linking |
|
||||
| `utils.cpp` | OpenGL utility functions |
|
||||
| `gl_resources.cpp` | OpenGL resource management |
|
||||
| `hidpi_gl_canvas.cpp` | HiDPI-aware GL canvas |
|
||||
| `graphics_abstraction_layer.cpp` | GAL base class |
|
||||
| `color4d.cpp` | Color handling |
|
||||
| `gal_display_options.cpp` | Display options |
|
||||
|
||||
### Build Configuration (`CMakeLists.txt`)
|
||||
|
||||
```cmake
|
||||
# Links against system wxWidgets
|
||||
find_program(WX_CONFIG_EXECUTABLE wx-config
|
||||
HINTS /opt/homebrew/bin /usr/local/bin)
|
||||
|
||||
# Includes KiCad headers from submodule
|
||||
target_include_directories(gal_native_test PRIVATE
|
||||
${KICAD_SOURCE}/include
|
||||
${KICAD_SOURCE}/include/gal
|
||||
${KICAD_SOURCE}/include/gal/opengl
|
||||
${KICAD_SOURCE}/libs/kimath/include
|
||||
${KICAD_SOURCE}/libs/core/include
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Generated Files
|
||||
|
||||
### Location
|
||||
|
||||
`/tests/gal-regression/native/generated/`
|
||||
|
||||
### Purpose
|
||||
|
||||
GLSL shaders must be embedded as strings at runtime. `generate_shaders.py` converts shader source files to C++ hex arrays.
|
||||
|
||||
### Generator Script
|
||||
|
||||
`generate_shaders.py` reads from `/kicad/common/gal/shaders/` and creates:
|
||||
|
||||
| Shader | Generated Files | Purpose |
|
||||
|--------|-----------------|---------|
|
||||
| `kicad.frag` | `glsl_kicad_frag.cpp/h` | Fragment shader (coloring) |
|
||||
| `kicad.vert` | `glsl_kicad_vert.cpp/h` | Vertex shader (transforms) |
|
||||
| `smaa_base.glsl` | `glsl_smaa_base.cpp/h` | SMAA common structures |
|
||||
| `smaa_pass_1_*.glsl` | 3 file pairs | SMAA edge detection |
|
||||
| `smaa_pass_2_*.glsl` | 2 file pairs | SMAA blending weights |
|
||||
| `smaa_pass_3_*.glsl` | 2 file pairs | SMAA neighborhood blending |
|
||||
|
||||
### Generated Code Structure
|
||||
|
||||
```cpp
|
||||
// generated/glsl_kicad_frag.cpp
|
||||
namespace KIGFX {
|
||||
namespace BUILTIN_SHADERS {
|
||||
static unsigned char glsl_kicad_frag_bytes[] = { 0x2f, 0x2a, ... };
|
||||
std::string glsl_kicad_frag = std::string(
|
||||
reinterpret_cast<char const*>(glsl_kicad_frag_bytes), 4233);
|
||||
}}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Test Harness Components
|
||||
|
||||
### `gal_native_test.cpp`
|
||||
|
||||
Main test driver with wxApp/wxFrame:
|
||||
|
||||
1. Creates `OPENGL_GAL` on a wxGLCanvas
|
||||
2. Configures coordinate system for 1:1 world-to-screen mapping
|
||||
3. Iterates through test scenarios
|
||||
4. Captures FBO contents as PNG screenshots
|
||||
|
||||
Key configuration:
|
||||
```cpp
|
||||
// Critical: Set 1:1 world-to-screen mapping
|
||||
// GAL default is for PCB nanometers (3.937e-8), which would
|
||||
// compress pixel coordinates (0-800) to ~0.003 screen pixels
|
||||
m_gal->SetWorldUnitLength(1.0 / ADVANCED_CFG::GetCfg().m_ScreenDPI);
|
||||
```
|
||||
|
||||
### `kicad_stubs.cpp`
|
||||
|
||||
Minimal implementations for KiCad symbols not included in GAL:
|
||||
|
||||
- `PGM_BASE` singleton with `GL_CONTEXT_MANAGER`
|
||||
- `ADVANCED_CFG` with `m_ScreenDPI = 91`
|
||||
- `KIFONT` stubs (returns nullptr for fonts)
|
||||
- `OBSERVABLE_BASE` observer pattern stubs
|
||||
- UI dialog stubs (`DisplayError`, etc.)
|
||||
|
||||
### `gal_test_accessor.cpp`
|
||||
|
||||
Uses C++ template technique to access private members:
|
||||
|
||||
```cpp
|
||||
// Access private OPENGL_GAL members without modifying headers
|
||||
template<typename T, T> struct steal_impl;
|
||||
template<typename T, T ptr>
|
||||
struct steal_impl {
|
||||
friend T get(steal_impl*) { return ptr; }
|
||||
};
|
||||
```
|
||||
|
||||
Provides:
|
||||
- `GetCompositorMainBufferTexture()` - For screenshot reading
|
||||
- `GetCompositorMainFBO()` - FBO ID access
|
||||
- `ReadCompositorFBOPixels()` - Direct pixel readback
|
||||
|
||||
### `gal_test_scenarios.cpp`
|
||||
|
||||
11 rendering test scenarios using the GAL API.
|
||||
|
||||
---
|
||||
|
||||
## Test Coverage Analysis
|
||||
|
||||
### Current Scenarios
|
||||
|
||||
| # | Name | GAL Features Tested |
|
||||
|---|------|-------------------|
|
||||
| 0 | basic-lines | `DrawLine`, `SetStrokeColor`, `SetLineWidth` |
|
||||
| 1 | line-widths | `DrawLine` with varying widths (0.5 to 12.0) |
|
||||
| 2 | circles | `DrawCircle` (filled and stroked) |
|
||||
| 3 | arcs | `DrawArc` |
|
||||
| 4 | rectangles | `DrawRectangle` (filled and stroked) |
|
||||
| 5 | polygons | `DrawPolygon`, `DrawPolyline` |
|
||||
| 6 | alpha-blending | `SetFillColor` with alpha transparency |
|
||||
| 7 | transforms | `Save`, `Restore`, `Rotate`, `Translate`, `Scale` |
|
||||
| 8 | grid-cursor | `DrawGrid`, `DrawCursor` |
|
||||
| 9 | segments | `DrawSegment` |
|
||||
| 10 | complex-scene | `SetLayerDepth` (z-ordering) |
|
||||
|
||||
### Coverage: ~19% of GAL API (14 of 73 methods)
|
||||
|
||||
### Missing Coverage (Priority Order)
|
||||
|
||||
**High Priority** (Critical for KiCad functionality):
|
||||
1. `DrawBitmap()` - Image/icon rendering
|
||||
2. `DrawCurve()` - Bezier curves
|
||||
3. `DrawGlyph()` / `BitmapText()` - Text rendering
|
||||
4. `DrawSegmentChain()` - Complex paths
|
||||
5. `DrawArcSegment()` - Filled arc segments
|
||||
|
||||
**Medium Priority** (Performance-critical features):
|
||||
6. Group methods: `BeginGroup()`, `EndGroup()`, `DrawGroup()`, `ClearCache()`
|
||||
7. Render targets: `SetTarget()`, offscreen rendering
|
||||
8. `SetNegativeDrawMode()` - Gerber-style rendering
|
||||
|
||||
**Low Priority** (Already implicit or not relevant to WASM):
|
||||
9. `EnableDepthTest()` - Implicit in complex-scene
|
||||
10. Context locking - Single-threaded in WASM
|
||||
|
||||
---
|
||||
|
||||
## Code Quality Assessment
|
||||
|
||||
### Strengths
|
||||
|
||||
1. **Clean separation** - Stubs, accessor, scenarios in separate files
|
||||
2. **Minimal stubs** - Only implements what's needed
|
||||
3. **Safe accessor** - Template technique avoids `#define private public`
|
||||
4. **Standard shader embedding** - Common practice for GL applications
|
||||
|
||||
### Architecture Decisions
|
||||
|
||||
| Decision | Rationale |
|
||||
|----------|-----------|
|
||||
| Compile actual KiCad GAL | Ground truth for comparison |
|
||||
| System wxWidgets | Native OpenGL context |
|
||||
| FBO reading | Clean screenshots without window capture |
|
||||
| Hex shader embedding | Runtime shader loading like KiCad |
|
||||
|
||||
---
|
||||
|
||||
## Running the Test
|
||||
|
||||
### Build
|
||||
|
||||
```bash
|
||||
./scripts/build-gal-native-test.sh
|
||||
```
|
||||
|
||||
### Execute
|
||||
|
||||
```bash
|
||||
./tests/gal-regression/native/build/gal_native_test --output ./baselines
|
||||
```
|
||||
|
||||
### Options
|
||||
|
||||
| Flag | Description |
|
||||
|------|-------------|
|
||||
| `--output <dir>` | Output directory for PNG files |
|
||||
| `--width <w>` | Canvas width (default: 800) |
|
||||
| `--height <h>` | Canvas height (default: 600) |
|
||||
| `--show` | Show window instead of headless |
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
### Immediate: Expand Test Coverage
|
||||
|
||||
1. Add `DrawBitmap` scenario with simple test image
|
||||
2. Add `DrawCurve` (Bezier) scenario
|
||||
3. Add text rendering scenario (requires KIFONT implementation)
|
||||
4. Add `DrawSegmentChain` scenario
|
||||
5. Add `DrawArcSegment` scenario
|
||||
|
||||
### Future: WebGL Comparison
|
||||
|
||||
1. Run same scenarios in WASM build
|
||||
2. Compare native vs WebGL screenshots
|
||||
3. Automate regression detection
|
||||
|
||||
---
|
||||
|
||||
## File Reference
|
||||
|
||||
| File | Location |
|
||||
|------|----------|
|
||||
| Test driver | `tests/gal-regression/native/gal_native_test.cpp` |
|
||||
| Stubs | `tests/gal-regression/native/kicad_stubs.cpp` |
|
||||
| Private accessor | `tests/gal-regression/native/gal_test_accessor.cpp` |
|
||||
| Test scenarios | `tests/gal-regression/scenarios/gal_test_scenarios.cpp` |
|
||||
| CMake config | `tests/gal-regression/native/CMakeLists.txt` |
|
||||
| Shader generator | `tests/gal-regression/native/generate_shaders.py` |
|
||||
| Build script | `scripts/build-gal-native-test.sh` |
|
||||
|
|
@ -0,0 +1,235 @@
|
|||
# WebGL GAL Port - Implementation Complete
|
||||
|
||||
> **ARCHIVED / HISTORICAL** — the WebGL-GAL backend described here was implemented (see `kicad/common/gal/webgl/`). Live test docs: [`tests/gal-regression/README.md`](../../../../tests/gal-regression/README.md). Kept for design rationale.
|
||||
|
||||
## Status: ALL PHASES COMPLETE ✅
|
||||
|
||||
**Branch:** `webgl` (21+ commits)
|
||||
|
||||
## Goal
|
||||
Port KiCad's GAL (Graphics Abstraction Layer) from OpenGL to WebGL to enable full KiCad functionality in the browser. Use the existing 28-scenario test suite to verify visual parity between native OpenGL and WebGL implementations.
|
||||
|
||||
## Final State
|
||||
- **WebGL GAL**: Fully integrated into `kicad/common/gal/webgl/` (~27,800 lines)
|
||||
- **GAL Test Suite**: 28 scenarios passing, visual parity with native OpenGL
|
||||
- **KiCad WASM**: Builds and runs successfully with WebGL GAL
|
||||
- **3D Viewer**: Disabled with stubs for WASM builds (`KICAD_BUILD_3D_VIEWER_WASM=OFF`)
|
||||
|
||||
## Key Decisions
|
||||
- **Approach**: Copy and modify existing OpenGL GAL code
|
||||
- **Scope**: Full feature parity (all 28 scenarios)
|
||||
- **Location**: Developed in `tests/gal-regression/wasm/` first, moved to KiCad for integration
|
||||
|
||||
## Architecture
|
||||
|
||||
**Two-backend test architecture:**
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ SAME 28 SCENARIO FILES │
|
||||
│ (scenarios/*.cpp - pure GAL API calls) │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
┌───────────────┴───────────────┐
|
||||
▼ ▼
|
||||
┌─────────────────────────────┐ ┌─────────────────────────┐
|
||||
│ NATIVE TEST HARNESS │ │ WEBGL TEST HARNESS │
|
||||
│ (gal_native_test.cpp) │ │ (gal_webgl_test.cpp) │
|
||||
│ │ │ │
|
||||
│ Uses: OPENGL_GAL │ │ Uses: WEBGL_GAL │
|
||||
│ Runs: macOS native │ │ Runs: Browser/WASM │
|
||||
└─────────────────────────────┘ └─────────────────────────┘
|
||||
│ │
|
||||
▼ ▼
|
||||
┌─────────────────────────────┐ ┌─────────────────────────┐
|
||||
│ output/native/gal-*.png │ │ output/webgl/gal-*.png │
|
||||
└─────────────────────────────┘ └─────────────────────────┘
|
||||
```
|
||||
|
||||
## Master Test Script: `scripts/test-gal-regression.sh`
|
||||
|
||||
**This is the only script we run.** Single command to build, test, and compare everything:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# Single script to build, run, and compare both backends
|
||||
|
||||
# 1. BUILD BOTH
|
||||
scripts/build-gal-native-test.sh
|
||||
scripts/build-gal-webgl-test.sh
|
||||
|
||||
# 2. RUN BOTH TESTS
|
||||
./tests/gal-regression/native/build/gal_native_test --output tests/gal-regression/output/native/
|
||||
npx playwright test gal-webgl.spec.ts # outputs to tests/gal-regression/output/webgl/
|
||||
|
||||
# 3. COMPARE (two-level)
|
||||
compare_screenshots output/native/ baseline/ # Catch native regressions
|
||||
compare_screenshots output/webgl/ output/native/ # Verify WebGL matches native
|
||||
|
||||
# 4. REPORT
|
||||
# Exit 0 if all match, exit 1 if any differ
|
||||
```
|
||||
|
||||
**Two-level comparison:**
|
||||
1. **native vs baseline** → Catches if native code regressed
|
||||
2. **webgl vs native** → Verifies WebGL implementation matches
|
||||
|
||||
**Output structure:**
|
||||
```
|
||||
tests/gal-regression/
|
||||
├── baseline/ # Committed reference screenshots
|
||||
├── output/
|
||||
│ ├── native/ # Fresh native run
|
||||
│ └── webgl/ # WebGL run via Playwright
|
||||
```
|
||||
|
||||
## Phases - ALL COMPLETE ✅
|
||||
|
||||
### Phase 1: Native Test Harness ✅
|
||||
Created unified test infrastructure with 28 scenarios covering 100% of GAL API.
|
||||
|
||||
**Commits:** 490f531 → 051fb87
|
||||
|
||||
**Deliverables:**
|
||||
- [x] `scripts/test-gal-regression.sh` - Master build/test/compare script
|
||||
- [x] `scripts/build-gal-native-test.sh` - Native build script
|
||||
- [x] `tests/gal-regression/native/` - Native C++ test harness using OPENGL_GAL
|
||||
- [x] `tests/gal-regression/scenarios/` - 28 shared test scenarios
|
||||
- [x] `tests/gal-regression/baseline/` - Native OpenGL reference screenshots
|
||||
|
||||
### Phase 2: WebGL GAL Implementation ✅
|
||||
Full port of OPENGL_GAL to WebGL 2.0 / OpenGL ES 3.0.
|
||||
|
||||
**Commits:** a4f444f → 74faa7e
|
||||
|
||||
**Key Changes:**
|
||||
- Replaced legacy `glBegin/glEnd` with VBO-based rendering
|
||||
- Replaced GL matrix stack with glm matrices
|
||||
- Converted GLSL shaders to ES 3.0 (`attribute`→`in`, `varying`→`out`, etc.)
|
||||
- Added VAO support (required for WebGL 2.0)
|
||||
- Replaced GLU tesselator with earcut.hpp
|
||||
|
||||
**Deliverables:**
|
||||
- [x] `tests/gal-regression/wasm/webgl/` - Initial WebGL GAL implementation
|
||||
- [x] `scripts/build-gal-webgl-test.sh` - WASM build script
|
||||
- [x] `tests/e2e/gal-webgl.spec.ts` - Playwright spec for screenshots
|
||||
|
||||
### Phase 3: Complete API Coverage ✅
|
||||
All GAL methods implemented with visual parity.
|
||||
|
||||
**Method Groups (all complete):**
|
||||
- [x] Basic drawing: DrawLine, DrawSegment, DrawCircle, DrawArc
|
||||
- [x] Shapes: DrawRectangle, DrawPolygon, DrawPolyline
|
||||
- [x] Advanced: DrawBezier, DrawBezierArc, DrawArcSegment, DrawSegmentChain
|
||||
- [x] State: Colors, transforms, depth testing, render targets
|
||||
- [x] Groups: BeginGroup, EndGroup, DrawGroup, ChangeGroupColor/Depth
|
||||
- [x] Text: DrawGlyph, DrawGlyphs, BitmapText
|
||||
- [x] Special: DrawGrid, DrawCursor, DrawBitmap
|
||||
|
||||
### Phase 4: KiCad Integration ✅
|
||||
WebGL GAL moved to KiCad source tree with CMake integration.
|
||||
|
||||
**Commit:** 37d973f (KiCad submodule: 1b5bb125d2)
|
||||
|
||||
**Deliverables:**
|
||||
- [x] Move `webgl_gal.*` to `kicad/common/gal/webgl/`
|
||||
- [x] CMake integration for Emscripten builds
|
||||
- [x] 3D viewer disabled with stubs (`KICAD_BUILD_3D_VIEWER_WASM=OFF`)
|
||||
- [x] KiCad WASM builds and e2e tests pass
|
||||
|
||||
---
|
||||
|
||||
## Technical Lessons Learned
|
||||
|
||||
### 1. GLSL ES 3.0 Shader Conversion
|
||||
Desktop OpenGL shaders needed conversion for WebGL 2.0:
|
||||
```glsl
|
||||
attribute → in
|
||||
varying → out (vertex shader) / in (fragment shader)
|
||||
texture2D() → texture()
|
||||
gl_FragColor → explicit out variable
|
||||
```
|
||||
Automated in `generate_shaders.py`.
|
||||
|
||||
### 2. VAO Required for WebGL 2.0
|
||||
WebGL 2.0 requires Vertex Array Objects (VAOs):
|
||||
```cpp
|
||||
glGenVertexArrays(1, &m_vao);
|
||||
glBindVertexArray(m_vao);
|
||||
```
|
||||
|
||||
### 3. Legacy GL Elimination
|
||||
All legacy OpenGL calls replaced:
|
||||
- `glBegin/glEnd` → VBO-based rendering
|
||||
- `glPushMatrix/glPopMatrix` → glm matrix stack
|
||||
- `GL_QUADS` → `GL_TRIANGLES` (quads not supported in WebGL)
|
||||
- `glEnableClientState` → modern vertex attributes
|
||||
|
||||
### 4. GLU Tesselator Replacement
|
||||
GLU not available in WebGL. Implemented using earcut.hpp:
|
||||
- File: `kicad/common/gal/webgl/glu_tess_impl.cpp`
|
||||
- Provides `gluNewTess`, `gluTessBeginPolygon`, etc.
|
||||
|
||||
### 5. Coordinate System & Retina Scaling
|
||||
- White background required for screenshot comparison (alpha compositing)
|
||||
- Retina 2x scaling requires proper `devicePixelRatio` handling
|
||||
- World-to-screen mapping: `SetWorldUnitLength(1.0 / DPI)`
|
||||
|
||||
### 6. Alpha Blending
|
||||
Proper blend functions required for correct transparency:
|
||||
```cpp
|
||||
glBlendFuncSeparate(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA, GL_ONE, GL_ONE_MINUS_SRC_ALPHA);
|
||||
```
|
||||
|
||||
### 7. 3D Viewer Stubbing Strategy
|
||||
When disabling 3D viewer (`KICAD_BUILD_3D_VIEWER_WASM=OFF`):
|
||||
- Guard includes/code with `#ifndef __EMSCRIPTEN__`
|
||||
- Stub classes need wxWidgets event table macros (`BEGIN_EVENT_TABLE`)
|
||||
- Complex classes (PANEL_PREVIEW_3D_MODEL) require all event handler stubs
|
||||
|
||||
## File Structure (Final)
|
||||
|
||||
```
|
||||
kicad/common/gal/webgl/ # WebGL GAL in KiCad source tree
|
||||
├── webgl_gal.cpp # Main implementation (3184 lines)
|
||||
├── webgl_gal.h # Class declaration (618 lines)
|
||||
├── webgl_compositor.cpp # FBO compositing
|
||||
├── webgl_antialiasing.cpp # SMAA implementation
|
||||
├── gpu_manager.cpp # VBO/VAO management
|
||||
├── vertex_manager.cpp # Vertex accumulation
|
||||
├── shader.cpp # GLSL compilation
|
||||
├── glu_tess_impl.cpp # GLU tesselator (earcut)
|
||||
├── earcut.hpp # Polygon triangulation
|
||||
└── ... (20+ files total)
|
||||
|
||||
tests/gal-regression/
|
||||
├── baseline/ # Native OpenGL reference (28 PNGs)
|
||||
├── baseline-webgl/ # WebGL reference (29 PNGs)
|
||||
├── native/ # Native test harness
|
||||
├── wasm/ # WebGL test harness (uses KiCad GAL)
|
||||
└── scenarios/ # Shared 28 test scenarios
|
||||
|
||||
wasm/stubs/
|
||||
├── 3d_canvas_stub.cpp # ~500 lines of 3D stubs
|
||||
├── 3d_viewer_stub.cpp # EDA_3D_VIEWER_FRAME stub
|
||||
└── 3d_scenegraph_stub.cpp # VRML export stubs
|
||||
|
||||
scripts/
|
||||
├── build-gal-native-test.sh # Native build script
|
||||
├── build-gal-webgl-test.sh # WASM build script
|
||||
└── test-gal-regression.sh # Master test script
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Verification Commands
|
||||
|
||||
```bash
|
||||
# Build KiCad WASM
|
||||
docker/build.sh
|
||||
|
||||
# Run GAL regression tests
|
||||
scripts/test-gal-regression.sh
|
||||
|
||||
# Run KiCad e2e tests
|
||||
cd tests && npm run test:kicad
|
||||
```
|
||||
|
|
@ -0,0 +1,329 @@
|
|||
# KiCad WASM Tool Activation Investigation
|
||||
|
||||
## Summary
|
||||
|
||||
This document explains the current investigation into why interactive PCB tools do not work correctly in the browser build of KiCad.
|
||||
|
||||
The visible symptom is simple:
|
||||
|
||||
- In native KiCad, clicking `Draw Lines` in the right toolbar leaves the tool selected, and two clicks on the board create a line.
|
||||
- In the browser build, the tool does not remain selected, and board clicks do not start drawing.
|
||||
|
||||
The important conclusion so far is that this does **not** look like a normal KiCad tool-definition bug. The evidence points much lower in the stack, into the WebAssembly coroutine/runtime path used to emulate KiCad's native coroutine model in the browser.
|
||||
|
||||
## What "Working" Looks Like Natively
|
||||
|
||||
On macOS, Linux, and Windows, the flow for a tool like `Draw Lines` is roughly:
|
||||
|
||||
```text
|
||||
User clicks right toolbar button
|
||||
-> wxAuiToolBar handles mouse up
|
||||
-> ACTION_TOOLBAR emits tool action
|
||||
-> TOOL_MANAGER activates the requested tool
|
||||
-> KiCad starts or resumes the tool coroutine
|
||||
-> User clicks board canvas
|
||||
-> GAL / canvas event is forwarded to the active tool
|
||||
-> Tool consumes the clicks and creates geometry
|
||||
```
|
||||
|
||||
Two details matter here:
|
||||
|
||||
1. KiCad tools are not just plain event handlers. Many of them are coroutine-driven.
|
||||
2. Native builds can rely on real low-level context switching and on the OS windowing system for input routing.
|
||||
|
||||
## Core Concepts
|
||||
|
||||
### `wxAuiToolBar`
|
||||
|
||||
KiCad's right-side drawing toolbar is an AUI toolbar, not a plain `wxToolBar`. That matters because its rendering, hit-testing, and state transitions go through a different path than the simpler wxWidgets toolbar tests.
|
||||
|
||||
### `ACTION_TOOLBAR` and `TOOL_MANAGER`
|
||||
|
||||
The toolbar does not draw lines by itself. Clicking a button activates a named KiCad action, and the tool manager is responsible for making the corresponding interactive tool current.
|
||||
|
||||
### KiCad coroutines
|
||||
|
||||
Interactive tools in KiCad depend on coroutine-style control flow. On native platforms this is implemented with `libcontext`, specifically `make_fcontext()` and `jump_fcontext()`.
|
||||
|
||||
### `wxGLCanvas` / `WEBGL_GAL`
|
||||
|
||||
Once a tool is active, board clicks are handled through KiCad's graphics/input path. In the browser build that path includes a DOM canvas, wxWidgets' WASM port, and KiCad's WebGL GAL layer.
|
||||
|
||||
### Emscripten fibers and Asyncify
|
||||
|
||||
WebAssembly in the browser cannot perform the same kind of native stack switching that desktop KiCad uses. Our port therefore has to emulate it with:
|
||||
|
||||
- Emscripten fibers
|
||||
- Asyncify
|
||||
- generated JavaScript glue around fiber switches
|
||||
|
||||
That emulation layer is the main place where the browser build can diverge from the native behavior.
|
||||
|
||||
## Native Data Flow
|
||||
|
||||
The native path is conceptually:
|
||||
|
||||
```text
|
||||
OS mouse event
|
||||
-> wxWidgets window / child-window dispatch
|
||||
-> wxAuiToolBar::OnLeftUp()
|
||||
-> ACTION_TOOLBAR
|
||||
-> TOOL_MANAGER
|
||||
-> libcontext coroutine switch
|
||||
-> active KiCad tool
|
||||
-> board clicks routed to the tool
|
||||
-> drawing result appears
|
||||
```
|
||||
|
||||
The key point is that native `libcontext` performs real context switching, and the OS owns the final event routing between the toolbar area and the graphics canvas.
|
||||
|
||||
## Browser/WASM Data Flow
|
||||
|
||||
The browser path has more moving parts:
|
||||
|
||||
```text
|
||||
Browser pointer event
|
||||
-> generated wx.js / WASM event bridge
|
||||
-> wxWidgets WASM windowing layer
|
||||
-> wxAuiToolBar or wxGLCanvas target
|
||||
-> KiCad action dispatch
|
||||
-> Emscripten fiber switch
|
||||
-> Asyncify suspend/resume bookkeeping
|
||||
-> KiCad tool coroutine body
|
||||
-> WebGL canvas input handling
|
||||
-> drawing result appears
|
||||
```
|
||||
|
||||
This means the browser build must get all of the following correct at the same time:
|
||||
|
||||
- AUI toolbar hit-testing
|
||||
- tool activation
|
||||
- coroutine entry
|
||||
- coroutine completion / return
|
||||
- WebGL canvas event forwarding
|
||||
|
||||
If any one of those layers is wrong, the tool appears to "not work".
|
||||
|
||||
## What We Observed
|
||||
|
||||
At first glance the problem looked like a toolbar-state problem:
|
||||
|
||||
- the `Draw Lines` button did not stay selected
|
||||
- clicking the board did nothing
|
||||
|
||||
However, once we added a proper E2E test with logging and delayed screenshots, the picture became clearer:
|
||||
|
||||
1. The button state alone was not enough to diagnose the problem.
|
||||
An immediate screenshot can capture hover or pressed state, not true persistent selection.
|
||||
2. After we added explicit `checked`-state tracking and a short delay, it became clear that the tool was still not truly active.
|
||||
3. The deeper failure was not simply "the toolbar forgot its checked state".
|
||||
|
||||
## The First Real Root Cause We Found
|
||||
|
||||
The first concrete bug was in generated JavaScript around Asyncify fiber switching.
|
||||
|
||||
In the generated KiCad JS, the fiber entry callback in `Fibers.finishContextSwitch()` had effectively become a no-op:
|
||||
|
||||
```text
|
||||
(a1 => {})(userData);
|
||||
```
|
||||
|
||||
That means the code switched into the new fiber, but then did not actually call the tool coroutine entry function.
|
||||
|
||||
### Why this matters
|
||||
|
||||
If the entry callback is replaced by an empty function, the tool activation path can appear to run, but the coroutine body never really starts. From the outside, that looks like:
|
||||
|
||||
- the toolbar click "does something"
|
||||
- but the tool never becomes truly active
|
||||
- board clicks are ignored because there is no running interactive tool waiting for them
|
||||
|
||||
## Change 1: Fix the generated fiber entry callback
|
||||
|
||||
We added a new fix in [inject-dyncall-shims.sh](/Users/V/IdeaProjects/kicad-wasm/scripts/common/inject-dyncall-shims.sh) so the generated JS is patched to call the real entry function:
|
||||
|
||||
```text
|
||||
dynCall_vi(entryPoint, userData);
|
||||
```
|
||||
|
||||
### Reasoning
|
||||
|
||||
This is a good change because it fixes an objectively broken generated code path at the WASM/JS boundary. It is not a KiCad workaround.
|
||||
|
||||
## Change 2: Tell Asyncify that fiber swaps are suspension points
|
||||
|
||||
We updated [apply-asyncify.sh](/Users/V/IdeaProjects/kicad-wasm/scripts/common/apply-asyncify.sh) so `env.emscripten_fiber_swap` is included in `ASYNCIFY_IMPORTS`.
|
||||
|
||||
### Reasoning
|
||||
|
||||
Asyncify needs to know which imports can suspend or unwind control flow. Fiber swaps are exactly that kind of boundary. If Asyncify does not model them correctly, the call stack bookkeeping around coroutine switches becomes unreliable.
|
||||
|
||||
This is also a WASM-layer fix, not a KiCad UI workaround.
|
||||
|
||||
## What Happened After Those Changes
|
||||
|
||||
Those two changes moved the investigation forward, but they did not finish the problem.
|
||||
|
||||
After the JS callback fix, the logs started showing that the fiber entry code was actually being reached:
|
||||
|
||||
```text
|
||||
[WASM_FCONTEXT] make
|
||||
[WASM_FCONTEXT] swap
|
||||
[WASM_FCONTEXT] entry
|
||||
```
|
||||
|
||||
That is an important result. It means:
|
||||
|
||||
- the original "entry callback is a no-op" bug was real
|
||||
- we did fix it
|
||||
- but another problem exists after fiber entry
|
||||
|
||||
## The Current Deeper Problem
|
||||
|
||||
After the fiber starts, the startup sequence still stalls before PCBnew fully finishes bringing up the toolbars.
|
||||
|
||||
The clearest evidence comes from the E2E log at:
|
||||
|
||||
[pcbnew-spec-ts-pcbnew-wasm-select-draw-lines-and-draw-on-the-board.log](/Users/V/IdeaProjects/kicad-wasm/tests/logs/kicad/pcbnew/pcbnew-spec-ts-pcbnew-wasm-select-draw-lines-and-draw-on-the-board.log)
|
||||
|
||||
The rendered-element summary currently ends up as:
|
||||
|
||||
```text
|
||||
{"count":13,"byType":{"sash":2,"searchctrl":3,"searchbutton":4,"auipart":4},"tools":[]}
|
||||
```
|
||||
|
||||
That means the browser-side registry can see some UI pieces, but **no rendered toolbar tools at all**.
|
||||
|
||||
So the current failure is no longer best described as "the Draw Lines tool unchecks itself". The stronger diagnosis is:
|
||||
|
||||
- PCBnew startup is being interrupted
|
||||
- the AUI toolbars never fully come online
|
||||
- the tool registry is therefore empty
|
||||
- the test cannot even reach a stable active-tool state
|
||||
|
||||
## Why Native KiCad Works But Browser KiCad Does Not
|
||||
|
||||
Native KiCad works because two hard problems are already solved by the native platform stack:
|
||||
|
||||
1. `libcontext` can use real native context switching semantics.
|
||||
2. The operating system handles input routing across the real window hierarchy.
|
||||
|
||||
The browser build does not get either of those for free.
|
||||
|
||||
Instead, it must emulate them with:
|
||||
|
||||
- generated JS glue
|
||||
- Asyncify instrumentation
|
||||
- Emscripten fibers
|
||||
- a DOM canvas and WebGL canvas bridge
|
||||
|
||||
So the browser failure is not evidence that KiCad's tool logic is broken. It is evidence that our WASM adaptation layer is still incomplete.
|
||||
|
||||
## Why Existing wxWidgets Tests Can Still Look Fine
|
||||
|
||||
This issue can exist even if many wxWidgets tests look correct.
|
||||
|
||||
The reason is that the failing KiCad path is more complex than a normal wx control interaction:
|
||||
|
||||
- KiCad uses `wxAuiToolBar`, not only basic controls
|
||||
- KiCad tool activation goes through `ACTION_TOOLBAR` and `TOOL_MANAGER`
|
||||
- KiCad drawing tools depend on coroutine switching
|
||||
- the board uses a separate WebGL-backed canvas/input path
|
||||
|
||||
Most simple wxWidgets tests do not exercise that exact combination.
|
||||
|
||||
## What We Changed During The Investigation
|
||||
|
||||
The current working tree contains a mix of real fixes, testing support, and experiments.
|
||||
|
||||
### Changes that look fundamentally correct
|
||||
|
||||
| Layer | File | Purpose | Why it makes sense |
|
||||
|------|------|---------|--------------------|
|
||||
| Build/WASM glue | [scripts/common/inject-dyncall-shims.sh](/Users/V/IdeaProjects/kicad-wasm/scripts/common/inject-dyncall-shims.sh) | Fix empty fiber entry callback in generated JS | Repairs objectively broken generated code |
|
||||
| Build/WASM glue | [scripts/common/apply-asyncify.sh](/Users/V/IdeaProjects/kicad-wasm/scripts/common/apply-asyncify.sh) | Add `env.emscripten_fiber_swap` to Asyncify imports | Makes Asyncify aware of fiber-switch suspension points |
|
||||
| Test support | [tests/e2e/utils/element-tracker.ts](/Users/V/IdeaProjects/kicad-wasm/tests/e2e/utils/element-tracker.ts) | Track `checked` state | Lets the test distinguish hover/pressed from real selection |
|
||||
| Test support | [tests/kicad/pcbnew.spec.ts](/Users/V/IdeaProjects/kicad-wasm/tests/kicad/pcbnew.spec.ts) | Add tool-selection/drawing regression and log capture | Reproduces the bug through the real KiCad flow |
|
||||
| Test observability | [wxwidgets/src/aui/auibar.cpp](/Users/V/IdeaProjects/kicad-wasm/wxwidgets/src/aui/auibar.cpp) | Export rendered AUI tool items to the browser registry | Gives Playwright a reliable way to see KiCad AUI tools |
|
||||
|
||||
### Changes that are investigative, not final
|
||||
|
||||
| Layer | File | Purpose | Current assessment |
|
||||
|------|------|---------|--------------------|
|
||||
| KiCad WebGL input | [kicad/common/gal/webgl/webgl_gal.cpp](/Users/V/IdeaProjects/kicad-wasm/kicad/common/gal/webgl/webgl_gal.cpp) | Forward mouse events immediately on WASM instead of posting them | Useful experiment, but not yet proven to be the main fix |
|
||||
| KiCad startup | [kicad/pcbnew/pcb_edit_frame.cpp](/Users/V/IdeaProjects/kicad-wasm/kicad/pcbnew/pcb_edit_frame.cpp) | Skip auto-invoking the selection tool on WASM | Pure debugging aid to see whether startup tool activation was the blocker |
|
||||
| KiCad third-party porting layer | [kicad/thirdparty/libcontext/libcontext.cpp](/Users/V/IdeaProjects/kicad-wasm/kicad/thirdparty/libcontext/libcontext.cpp) | Experimental Emscripten-fiber implementation of `libcontext` | Likely the right conceptual layer, but the current implementation is not clean/final |
|
||||
|
||||
### Changes that are not meaningful
|
||||
|
||||
| File | Note |
|
||||
|------|------|
|
||||
| [kicad/common/tool/action_toolbar.cpp](/Users/V/IdeaProjects/kicad-wasm/kicad/common/tool/action_toolbar.cpp) | Trailing newline / formatting-only diff |
|
||||
| [kicad/common/tool/tools_holder.cpp](/Users/V/IdeaProjects/kicad-wasm/kicad/common/tool/tools_holder.cpp) | Trailing newline / formatting-only diff |
|
||||
| [wxwidgets/build/wasm/wx.js](/Users/V/IdeaProjects/kicad-wasm/wxwidgets/build/wasm/wx.js) | Generated build artifact, not a source-level design change |
|
||||
|
||||
## Why the Current `libcontext` Work Is Still Not "Done"
|
||||
|
||||
The current experiments in [libcontext.cpp](/Users/V/IdeaProjects/kicad-wasm/kicad/thirdparty/libcontext/libcontext.cpp) show that we can start a fiber, but not yet return from it with semantics that match what KiCad expects from native `jump_fcontext()`.
|
||||
|
||||
The native contract is subtle:
|
||||
|
||||
- one context transfers control to another
|
||||
- control can later resume into the previous context
|
||||
- returned values and ownership of "who resumes whom" must remain consistent
|
||||
- cleanup of a finished coroutine must not break the surrounding frame startup
|
||||
|
||||
In the browser build, once the first startup coroutine finishes, that handoff is still wrong. The result is not necessarily an immediate crash anymore, but the UI startup is interrupted before the toolbars are fully present.
|
||||
|
||||
## Current Best Explanation
|
||||
|
||||
The current best explanation is:
|
||||
|
||||
1. A toolbar click is not the primary problem.
|
||||
2. The browser port originally had a broken fiber entry callback, which prevented tool coroutines from starting at all.
|
||||
3. After fixing that, the browser port still mishandles coroutine completion / return.
|
||||
4. That deeper runtime mismatch interrupts PCBnew startup before the AUI tools fully appear.
|
||||
5. Because the tools are not fully rendered and the interactive-tool runtime is not stable, the right-side drawing tools do not remain active and board clicks do not draw.
|
||||
|
||||
## Clean Direction From Here
|
||||
|
||||
The cleanest direction is to keep the fix low in the WASM/runtime layer and avoid papering over the issue in KiCad UI code.
|
||||
|
||||
Recommended direction:
|
||||
|
||||
1. Keep the JS fiber-entry fix in [inject-dyncall-shims.sh](/Users/V/IdeaProjects/kicad-wasm/scripts/common/inject-dyncall-shims.sh).
|
||||
2. Keep the Asyncify import fix in [apply-asyncify.sh](/Users/V/IdeaProjects/kicad-wasm/scripts/common/apply-asyncify.sh).
|
||||
3. Move the `libcontext` solution toward the dedicated WASM layer under [wasm/libcontext](/Users/V/IdeaProjects/kicad-wasm/wasm/libcontext) instead of continuing to patch KiCad's bundled third-party copy in ad hoc ways.
|
||||
4. Remove temporary KiCad startup and input experiments once the lower-layer coroutine behavior is correct.
|
||||
5. Keep the AUI rendered-tool export only if we still want browser E2E tests to target KiCad tools by tooltip and checked state.
|
||||
|
||||
## Reproduction and Evidence
|
||||
|
||||
### Build and run
|
||||
|
||||
```bash
|
||||
./docker/build.sh
|
||||
cd tests
|
||||
npm run test:kicad -- --grep "select draw lines"
|
||||
```
|
||||
|
||||
### Where to look
|
||||
|
||||
- E2E spec:
|
||||
[pcbnew.spec.ts](/Users/V/IdeaProjects/kicad-wasm/tests/kicad/pcbnew.spec.ts)
|
||||
- E2E logs:
|
||||
[tests/logs/kicad/pcbnew](/Users/V/IdeaProjects/kicad-wasm/tests/logs/kicad/pcbnew)
|
||||
- Current startup screenshot:
|
||||
[wizard-00-initial.png](/Users/V/IdeaProjects/kicad-wasm/tests/test-results/wizard-00-initial.png)
|
||||
|
||||
## Bottom Line
|
||||
|
||||
The current evidence says the KiCad browser tool failure is fundamentally a WebAssembly coroutine/runtime problem.
|
||||
|
||||
The right fix direction is:
|
||||
|
||||
- not "change KiCad tool logic"
|
||||
- not "hack the toolbar state"
|
||||
- but "make the WASM coroutine and fiber handoff behave like native KiCad expects"
|
||||
|
||||
The two most defensible changes so far are the generated-JS fiber-entry fix and the Asyncify import update. Everything else should be treated as either observability support or investigation scaffolding until the underlying `libcontext` behavior is corrected.
|
||||
795
docs/features/browser-tools/0002-wasm-coroutine-deep-dive.md
Normal file
795
docs/features/browser-tools/0002-wasm-coroutine-deep-dive.md
Normal file
|
|
@ -0,0 +1,795 @@
|
|||
# KiCad WASM Coroutine Deep Dive
|
||||
|
||||
## Why Our Use Case Is Special
|
||||
|
||||
### Most WASM projects don't need coroutines at all
|
||||
|
||||
When you think of "compile C/C++ to WebAssembly with Emscripten," the typical projects are:
|
||||
|
||||
- **Games** (Unity, Unreal, etc.): They have a main loop that renders frames. The game engine calls `emscripten_set_main_loop(renderFrame, 60, 0)` and Emscripten calls `renderFrame()` 60 times per second. No coroutines needed — everything is event-driven already.
|
||||
|
||||
- **Command-line tools** (ffmpeg, SQLite, etc.): They run, produce output, and exit. Linear execution. No coroutines.
|
||||
|
||||
- **Simple GUI apps**: They handle events through callbacks. Button clicked → run handler. No need to pause mid-function.
|
||||
|
||||
**KiCad is unusual** because its interactive tools use a **synchronous programming model** inside a coroutine:
|
||||
|
||||
```cpp
|
||||
void PCB_TOOL::DrawLine(TOOL_EVENT& evt) {
|
||||
Point p1 = WaitForClick(); // ← PAUSES HERE, waits for user
|
||||
Point p2 = WaitForClick(); // ← PAUSES HERE again
|
||||
CreateLine(p1, p2);
|
||||
}
|
||||
```
|
||||
|
||||
This code looks simple and linear, but `WaitForClick()` can't actually block in a browser. Instead, KiCad uses a coroutine to pause the function, return control to the browser event loop, and resume later when the click arrives. This requires the ability to **save and restore the entire call stack** — which is what libcontext and fibers do.
|
||||
|
||||
### Very few projects need this
|
||||
|
||||
The number of large C/C++ applications that:
|
||||
1. Were designed for desktop with coroutine-based control flow
|
||||
2. Are now being ported to the browser via Emscripten
|
||||
3. Need those coroutines to actually work
|
||||
|
||||
...is very small. QEMU is one. KiCad is another. Maybe a handful of others.
|
||||
|
||||
Because so few people need this, the Emscripten support for it is:
|
||||
- **Functional** (the fiber API exists and works)
|
||||
- **But rough around the edges** (bugs in internal code, poor documentation, edge cases not handled)
|
||||
- **And under-tested** (most users never exercise these code paths)
|
||||
|
||||
### Emscripten's priorities
|
||||
|
||||
Emscripten's main user base is game engines and simple tools. Their effort goes into:
|
||||
- Compilation speed
|
||||
- WASM binary size
|
||||
- Performance of simple programs
|
||||
- SIMD, threading, memory64
|
||||
|
||||
The fiber/coroutine path is a niche feature. The `makeDynCall` bug persists because almost nobody hits it — the people who do (like us) work around it.
|
||||
|
||||
---
|
||||
|
||||
## Concepts From the Ground Up
|
||||
|
||||
### What is "The Stack"?
|
||||
|
||||
Every time you call a function, a new "frame" is pushed onto the call stack holding local variables, return address, and arguments.
|
||||
|
||||
```
|
||||
main() calls drawLine() calls calculatePoint()
|
||||
|
||||
Stack (grows downward):
|
||||
┌──────────────────┐
|
||||
│ main() │ ← local vars of main
|
||||
├──────────────────┤
|
||||
│ drawLine() │ ← local vars of drawLine
|
||||
├──────────────────┤
|
||||
│ calculatePoint() │ ← TOP: local vars of calculatePoint
|
||||
└──────────────────┘
|
||||
```
|
||||
|
||||
**"Stack-local"** = a variable that lives in a frame. When that function returns, the frame is popped and the variable's memory becomes garbage.
|
||||
|
||||
### What is a Coroutine?
|
||||
|
||||
A function that can **pause** mid-execution and **resume** later. KiCad needs this because in a browser you can't block waiting for a mouse click (the page freezes). So a drawing tool must pause after requesting a click, let the browser run, then resume when the click arrives.
|
||||
|
||||
### Stackful vs Stackless
|
||||
|
||||
**Stackless** (C++20 `co_await`): Can only pause at the top level. If `waitForClick()` is 5 calls deep, you can't pause.
|
||||
|
||||
**Stackful** (KiCad): Can pause from **anywhere** in the call stack. The entire stack is saved and restored. KiCad needs this.
|
||||
|
||||
### What is libcontext?
|
||||
|
||||
A small C library (from Boost.Context) that performs context switching via three functions:
|
||||
|
||||
```cpp
|
||||
make_fcontext(stack, size, entry_func); // Create a new context
|
||||
jump_fcontext(&old, new, value); // Switch contexts
|
||||
release_fcontext(ctx); // Free a context
|
||||
```
|
||||
|
||||
On native: ~20 lines of assembly per platform (x86, ARM, etc.) that saves/loads CPU registers. On WASM: impossible natively, must be emulated.
|
||||
|
||||
**libcontext is NOT a separate repo.** It's a directory inside KiCad (`kicad/thirdparty/libcontext/`). Our kicad submodule points to our fork (`VV-EE/kicad-source-mirror.git`), so we already own it. No additional forking needed.
|
||||
|
||||
### The Full Stack of Abstractions
|
||||
|
||||
```
|
||||
KiCad COROUTINE class (kicad/include/tool/coroutine.h)
|
||||
↓ calls
|
||||
libcontext API (make_fcontext / jump_fcontext)
|
||||
↓ implemented with (on WASM)
|
||||
Emscripten Fibers (emscripten_fiber_swap)
|
||||
↓ built on
|
||||
Asyncify (wasm-opt binary transformation)
|
||||
↓ manipulates
|
||||
WebAssembly call stack (in the browser's WASM runtime)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## How Asyncify Actually Works
|
||||
|
||||
### The Core Idea
|
||||
|
||||
WASM doesn't let you save/restore call stacks like native assembly does. Asyncify works around this with a completely different approach: **it rewrites your WASM bytecode** so that every function can cooperatively save its state and return, then later be re-called and skip ahead to where it left off.
|
||||
|
||||
Two globals drive everything:
|
||||
- `__asyncify_state`: 0 = Normal, 1 = Unwinding, 2 = Rewinding
|
||||
- `__asyncify_data`: pointer to a buffer that holds saved state
|
||||
|
||||
### The Asyncify Data Buffer
|
||||
|
||||
Each fiber/coroutine has its own buffer (the "asyncify stack"). Layout:
|
||||
|
||||
```
|
||||
[ptr+0] i32: current stack position (grows upward as data is pushed)
|
||||
[ptr+4] i32: stack end (upper bound)
|
||||
[ptr+8] i32: rewind_id (which WASM export to re-enter during rewind)
|
||||
[ptr+12] ... actual saved data (call indices + serialized local variables)
|
||||
```
|
||||
|
||||
### What the Binary Transformation Does
|
||||
|
||||
Asyncify (via `wasm-opt --asyncify`) rewrites every function in the WASM module. Here's a before/after:
|
||||
|
||||
**Before transformation:**
|
||||
```c
|
||||
void foo(int x) {
|
||||
x = x + 1;
|
||||
x = x / 2;
|
||||
bar(x); // ← this call might trigger a pause
|
||||
while (x & 7) {
|
||||
x = x + 1;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**After transformation (pseudocode of the generated WASM):**
|
||||
```c
|
||||
void foo(int x) {
|
||||
// PRELUDE: if we're rewinding, restore our saved locals
|
||||
if (__asyncify_state == REWINDING) {
|
||||
x = pop_from_asyncify_stack(); // restore x
|
||||
call_index = pop_from_asyncify_stack(); // which call site to skip to
|
||||
}
|
||||
|
||||
// Normal code: skip during rewind
|
||||
if (__asyncify_state == NORMAL) {
|
||||
x = x + 1;
|
||||
x = x / 2;
|
||||
}
|
||||
|
||||
// The call site: execute if normal, OR if rewinding to this specific call
|
||||
if (__asyncify_state == NORMAL || call_index == 0) {
|
||||
bar(x);
|
||||
|
||||
// After the call returns: are we unwinding?
|
||||
if (__asyncify_state == UNWINDING) {
|
||||
push_to_asyncify_stack(0); // save call index (we were at bar())
|
||||
push_to_asyncify_stack(x); // save local variable x
|
||||
return; // cooperatively return up the chain
|
||||
}
|
||||
}
|
||||
|
||||
// Rest of function: skip during rewind
|
||||
if (__asyncify_state == NORMAL) {
|
||||
while (x & 7) {
|
||||
x = x + 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Key insight:** Every function in the call chain gets this treatment. During unwind, each frame saves its state and returns normally. During rewind, each frame skips ahead to the right call site and dives deeper.
|
||||
|
||||
### The Complete Unwind Sequence (Pause)
|
||||
|
||||
When something wants to pause (e.g., `emscripten_fiber_swap`):
|
||||
|
||||
```
|
||||
1. JS sets __asyncify_state = UNWINDING (1)
|
||||
2. JS sets __asyncify_data = pointer to this fiber's buffer
|
||||
3. The call to emscripten_fiber_swap returns to its caller in WASM
|
||||
4. The caller checks: state == UNWINDING? Yes.
|
||||
→ Pushes its call index + locals to asyncify_data buffer
|
||||
→ Returns to ITS caller
|
||||
5. That caller checks: state == UNWINDING? Yes.
|
||||
→ Same thing: push call index + locals, return
|
||||
6. This cascades all the way up until the WASM export returns to JS
|
||||
7. JS: all WASM frames have returned. Call asyncify_stop_unwind().
|
||||
→ __asyncify_state = 0 (Normal)
|
||||
8. The entire WASM call stack is gone. State is saved in the buffer.
|
||||
```
|
||||
|
||||
### The Complete Rewind Sequence (Resume)
|
||||
|
||||
When something wants to resume a paused fiber:
|
||||
|
||||
```
|
||||
1. JS sets __asyncify_state = REWINDING (2)
|
||||
2. JS sets __asyncify_data = pointer to the saved fiber's buffer
|
||||
3. JS calls the same WASM export function that was running before (e.g., main)
|
||||
4. main() enters. Sees state == REWINDING.
|
||||
→ Pops its locals from asyncify_data buffer
|
||||
→ Pops its call index → skips ahead to that call site
|
||||
→ Calls the function at that call site
|
||||
5. That function enters. Sees state == REWINDING.
|
||||
→ Same thing: pop locals, pop call index, skip ahead, call deeper
|
||||
6. This continues until we reach the DEEPEST frame (the one that paused)
|
||||
7. The deepest frame calls asyncify_stop_rewind()
|
||||
→ __asyncify_state = 0 (Normal)
|
||||
8. Execution continues normally from exactly where it paused.
|
||||
```
|
||||
|
||||
### How `emscripten_fiber_swap` Coordinates Two Fibers
|
||||
|
||||
Each `emscripten_fiber_t` struct contains:
|
||||
```c
|
||||
typedef struct {
|
||||
void* stack_base; // C stack top
|
||||
void* stack_limit; // C stack bottom
|
||||
void* stack_ptr; // current C stack pointer (saved on swap)
|
||||
void (*entry)(void*); // entry function (NULL after first call)
|
||||
void* user_data; // argument for entry function
|
||||
asyncify_data_t asyncify_data; // this fiber's own asyncify buffer
|
||||
} emscripten_fiber_t;
|
||||
```
|
||||
|
||||
The swap sequence for switching from Fiber A to Fiber B:
|
||||
|
||||
```
|
||||
Fiber A is running (state = Normal)
|
||||
|
||||
A calls emscripten_fiber_swap(&A, &B):
|
||||
JS side:
|
||||
1. state = Unwinding
|
||||
2. currData = A.asyncify_data (save into A's buffer)
|
||||
3. asyncify_start_unwind(A.asyncify_data)
|
||||
4. Save A's C stack pointer into A.stack_ptr
|
||||
5. Set Fibers.nextFiber = B
|
||||
6. Return (emscripten_fiber_swap returns to caller)
|
||||
|
||||
WASM side:
|
||||
7. A's call chain unwinds: each frame saves state into A.asyncify_data
|
||||
8. All WASM frames return to JS
|
||||
|
||||
JS side (maybeStopUnwind):
|
||||
9. asyncify_stop_unwind() → state = Normal
|
||||
10. Fibers.trampoline() → finishContextSwitch(B)
|
||||
|
||||
finishContextSwitch(B):
|
||||
11. Restore B's C stack pointer + limits
|
||||
12. Is B.entry != NULL? (first time entering B)
|
||||
YES → call B.entry(B.user_data) ← this is where dynCall_vi matters!
|
||||
NO → (B was previously paused)
|
||||
asyncify_start_rewind(B.asyncify_data)
|
||||
doRewind() → calls the saved export, which replays B's call chain
|
||||
|
||||
B is now running.
|
||||
|
||||
Later, B calls emscripten_fiber_swap(&B, &A):
|
||||
Same process in reverse:
|
||||
- B's state is saved into B.asyncify_data
|
||||
- finishContextSwitch(A):
|
||||
A.entry == NULL → rewind into A.asyncify_data
|
||||
A's call chain replays until emscripten_fiber_swap
|
||||
emscripten_fiber_swap's "else" branch runs:
|
||||
state = Normal
|
||||
asyncify_stop_rewind()
|
||||
|
||||
A continues exactly where it left off.
|
||||
```
|
||||
|
||||
### Why This Is Slow
|
||||
|
||||
Every context switch involves:
|
||||
1. Unwinding the entire call stack (every frame saves state and returns)
|
||||
2. Rewinding the entire call stack (every frame re-enters, restores state, skips ahead)
|
||||
|
||||
Native libcontext: save ~15 CPU registers, change stack pointer. Done in nanoseconds.
|
||||
Asyncify: serialize/deserialize every frame. Documented overhead: 20-100% slowdown.
|
||||
|
||||
---
|
||||
|
||||
## The dynCall Problem
|
||||
|
||||
### What dynCall Functions Were
|
||||
|
||||
`dynCall_vi`, `dynCall_ii`, etc. were JavaScript wrapper functions for calling WASM **function pointers** from JS. Naming convention:
|
||||
- `v` = void, `i` = int, `f` = float, `d` = double
|
||||
- First letter = return type, rest = argument types
|
||||
- `dynCall_vi(ptr, arg)` = "call the WASM function at table index `ptr` with one int `arg`, returning void"
|
||||
|
||||
They existed because calling a WASM function pointer from JavaScript requires:
|
||||
1. Looking up the function in the `WebAssembly.Table` by index
|
||||
2. Calling it with the right types
|
||||
|
||||
Before the WebAssembly.Table API stabilized, Emscripten generated one typed wrapper per signature used in the program.
|
||||
|
||||
### Why They Were Removed
|
||||
|
||||
Starting Emscripten 2.0.2 (August 2020), removed for performance:
|
||||
|
||||
The replacement is `getWasmTableEntry(index)` which directly looks up the function in the table:
|
||||
```javascript
|
||||
// Old way:
|
||||
dynCall_vi(funcPtr, arg1);
|
||||
|
||||
// New way:
|
||||
getWasmTableEntry(funcPtr)(arg1);
|
||||
```
|
||||
|
||||
Benchmarks showed the new way is **60-80% faster** and produces smaller JS output.
|
||||
|
||||
### How Emscripten's Internal Code Uses dynCall
|
||||
|
||||
Emscripten's own JS library files (the runtime glue) need to call WASM function pointers too. They use a preprocessor macro called `makeDynCall`:
|
||||
|
||||
```javascript
|
||||
// Inside Emscripten's library_async.js, library_html5.js, etc.
|
||||
// This is a BUILD-TIME macro, expanded by Emscripten's preprocessor
|
||||
|
||||
// Old syntax (pre-2.0.9):
|
||||
{{{ makeDynCall('vi') }}}(funcPtr, arg1)
|
||||
|
||||
// New syntax (2.0.9+):
|
||||
{{{ makeDynCall('vi', 'funcPtr') }}}(arg1)
|
||||
```
|
||||
|
||||
The difference: the old syntax doesn't tell the macro which variable holds the function pointer. The new syntax does.
|
||||
|
||||
### The Silent Degradation Bug
|
||||
|
||||
Here's what happens when the macro expands. Inside Emscripten's `parseTools.mjs`:
|
||||
|
||||
```javascript
|
||||
function makeDynCall(sig, funcPtr) {
|
||||
if (funcPtr === undefined) {
|
||||
// OLD SYNTAX: funcPtr not provided
|
||||
if (DYNCALLS) {
|
||||
// -sDYNCALLS=1 is set: use the generated dynCall_vi function
|
||||
return `dynCall_${sig}`;
|
||||
}
|
||||
// DYNCALLS is false (default since ~2.0.3)
|
||||
// Try to find an exported dynCall_vi... it doesn't exist
|
||||
// Fall through to:
|
||||
return `((args) => {} /* a dynamic function call to signature ${sig},
|
||||
but there are no exported function pointers with that signature,
|
||||
so this path should never be taken. */)`;
|
||||
}
|
||||
// NEW SYNTAX: funcPtr provided → use getWasmTableEntry
|
||||
return `getWasmTableEntry(${funcPtr})`;
|
||||
}
|
||||
```
|
||||
|
||||
**The critical problem:** Emscripten's **own internal library files** still use the old syntax in many places. When `DYNCALLS=false` (the default), the macro generates an empty arrow function `(a1 => {})` instead of actually calling the function.
|
||||
|
||||
The generated comment even says *"this path should never be taken"* — but it IS taken, because the internal library files trigger it.
|
||||
|
||||
### What Breaks
|
||||
|
||||
| Location in Emscripten JS | What the no-op replaces | Effect |
|
||||
|---------------------------|------------------------|--------|
|
||||
| `Fibers.finishContextSwitch` | `dynCall_vi(entryPoint, userData)` | **Fiber entry function never called** — coroutines are dead |
|
||||
| `_emscripten_set_main_loop` | `dynCall_v(callback)` | Main loop callback is a no-op |
|
||||
| `_emscripten_async_call` | `dynCall_vi(callback, arg)` | Timer callbacks never fire |
|
||||
| `___call_sighandler` | `dynCall_vi(handler, sig)` | Signal handlers are no-ops |
|
||||
| `invokeEntryPoint` (pthreads) | `dynCall_ii(entry, arg)` | Thread entry never called |
|
||||
| HTML5 event callbacks | `dynCall_iiii(callback, ...)` | Mouse/keyboard events ignored |
|
||||
|
||||
### Why `finishContextSwitch` Matters Most For Us
|
||||
|
||||
This is the function that runs when a fiber is being entered for the first time. The flow:
|
||||
|
||||
```javascript
|
||||
finishContextSwitch(newFiber) {
|
||||
// ... restore C stack ...
|
||||
|
||||
var entryPoint = /* read from fiber struct */;
|
||||
if (entryPoint !== 0) {
|
||||
// FIRST TIME entering this fiber: call the entry function
|
||||
var userData = /* read from fiber struct */;
|
||||
|
||||
// THIS LINE is what's broken:
|
||||
{{{ makeDynCall('vi', 'entryPoint') }}}(userData);
|
||||
//
|
||||
// With old-syntax makeDynCall and DYNCALLS=false, this becomes:
|
||||
// (a1 => {})(userData);
|
||||
//
|
||||
// The entry function is NEVER CALLED.
|
||||
// The fiber "starts" but its body never runs.
|
||||
} else {
|
||||
// Subsequent entry: rewind via asyncify
|
||||
// This path works fine
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
So: every fiber's FIRST entry goes through `dynCall_vi(entryPoint, userData)`. If that's a no-op, the coroutine body never starts. The fiber appears to start (the swap succeeds) but nothing actually happens inside it.
|
||||
|
||||
### What Our Fix Does
|
||||
|
||||
`inject-dyncall-shims.sh` does two things:
|
||||
|
||||
**1. Generates Asyncify-aware dynCall shims:**
|
||||
```javascript
|
||||
function dynCall_vi(funcPtr, arg1) {
|
||||
var func = getWasmTableEntry(funcPtr);
|
||||
// Track in Asyncify's export call stack so unwind/rewind works
|
||||
Asyncify.exportCallStack.push('dynCall_vi');
|
||||
try {
|
||||
func(arg1);
|
||||
} finally {
|
||||
if (Asyncify.currData) {
|
||||
// We're mid-unwind: set the rewind function
|
||||
Asyncify.setDataRewindFunc(Asyncify.currData);
|
||||
}
|
||||
Asyncify.exportCallStack.pop();
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**2. Patches the six empty arrow function patterns** to call the real shims.
|
||||
|
||||
### Why Not Just Use `-sDYNCALLS=1`?
|
||||
|
||||
You could. Unity does. But there's a subtlety:
|
||||
|
||||
Plain `dynCall_vi` from `-sDYNCALLS=1` is just:
|
||||
```javascript
|
||||
function dynCall_vi(index, a1) { getWasmTableEntry(index)(a1); }
|
||||
```
|
||||
|
||||
It does **not** push/pop `Asyncify.exportCallStack`. That tracking is needed for Asyncify to know which WASM export to re-enter during rewind. Without it, if a fiber swap happens inside an indirect call (function pointer), Asyncify loses track of the call chain and the rewind fails.
|
||||
|
||||
Our shims add this tracking. That's the extra value over `-sDYNCALLS=1`.
|
||||
|
||||
### Is This an Emscripten Bug?
|
||||
|
||||
**Yes, arguably.** The problem is that Emscripten's own internal library files (`library_async.js`, `library_html5.js`, etc.) use the deprecated `makeDynCall` syntax, which silently degrades to no-ops when `DYNCALLS=false`. The generated comment says "this path should never be taken" but it's taken constantly. Worth filing as a bug.
|
||||
|
||||
---
|
||||
|
||||
## QEMU: The Gold Standard Reference
|
||||
|
||||
### What QEMU actually is
|
||||
|
||||
QEMU is a **machine emulator and virtualizer**. It lets you:
|
||||
- Run an ARM Linux system on your x86 laptop
|
||||
- Run Windows inside a virtual machine on Linux
|
||||
- Emulate hardware for embedded development
|
||||
|
||||
It's one of the most important open-source infrastructure projects — it powers much of cloud computing (via KVM/QEMU).
|
||||
|
||||
### Why QEMU uses coroutines
|
||||
|
||||
QEMU's disk I/O layer uses coroutines for the same reason KiCad uses them: to write **synchronous-looking code** that actually runs asynchronously.
|
||||
|
||||
When QEMU needs to read from a virtual disk:
|
||||
```c
|
||||
void handle_disk_read(Request *req) {
|
||||
Buffer data = read_from_disk(req->sector); // ← this might take time
|
||||
send_data_to_guest(req, data);
|
||||
}
|
||||
```
|
||||
|
||||
`read_from_disk()` might need to wait for actual I/O. Instead of blocking (which would freeze the emulator), QEMU pauses the coroutine, processes other events, and resumes when the data is ready. Exactly the same pattern as KiCad's `WaitForClick()`.
|
||||
|
||||
### Why QEMU was recently ported to WASM
|
||||
|
||||
People want to run QEMU in the browser — to provide virtual machines in web-based development environments, education tools, etc. The QEMU project accepted patches to build with Emscripten, and part of that work was making coroutines work in WASM.
|
||||
|
||||
### Why QEMU's solution is relevant to us
|
||||
|
||||
QEMU and KiCad have the **exact same problem**:
|
||||
- Both are large C/C++ codebases
|
||||
- Both use stackful coroutines internally
|
||||
- Both need those coroutines to work when compiled to WASM
|
||||
- Both use Emscripten's fiber API as the backend
|
||||
|
||||
QEMU's solution (`util/coroutine-wasm.c`) was:
|
||||
1. Written by someone who clearly understood the Emscripten fiber API constraints
|
||||
2. Reviewed by the QEMU maintainers
|
||||
3. Accepted into the official QEMU repository
|
||||
4. Has been running in production
|
||||
|
||||
It's only 127 lines. It's the cleanest, most proven reference for "how to do coroutines in Emscripten."
|
||||
|
||||
### QEMU's Full Implementation
|
||||
|
||||
Source: [github.com/qemu/qemu/blob/master/util/coroutine-wasm.c](https://github.com/qemu/qemu/blob/master/util/coroutine-wasm.c)
|
||||
|
||||
**The struct:**
|
||||
```c
|
||||
typedef struct {
|
||||
Coroutine base; // QEMU's base coroutine type
|
||||
void *stack; // C stack buffer (heap-allocated)
|
||||
size_t stack_size;
|
||||
void *asyncify_stack; // Asyncify data buffer (heap-allocated)
|
||||
size_t asyncify_stack_size;
|
||||
CoroutineAction action; // Communication channel (YIELD, TERMINATE, etc.)
|
||||
emscripten_fiber_t fiber; // The Emscripten fiber handle
|
||||
} CoroutineEmscripten;
|
||||
```
|
||||
|
||||
Each coroutine owns **two** heap-allocated buffers: a C stack and an asyncify stack. Both persist for the coroutine's lifetime.
|
||||
|
||||
**The trampoline (most important part):**
|
||||
```c
|
||||
static void coroutine_trampoline(void *co_)
|
||||
{
|
||||
Coroutine *co = co_;
|
||||
|
||||
while (true) { // ← NEVER returns
|
||||
co->entry(co->entry_arg); // Run the coroutine body
|
||||
qemu_coroutine_switch(co, co->caller,
|
||||
COROUTINE_TERMINATE); // Swap back to caller
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Walk-through:
|
||||
|
||||
1. A new QEMU coroutine is created for some I/O operation.
|
||||
2. `emscripten_fiber_init()` is called with `coroutine_trampoline` as the entry function.
|
||||
3. When the coroutine is first entered (someone swaps to it), `coroutine_trampoline` starts running.
|
||||
4. It calls `co->entry(co->entry_arg)` — this is the actual I/O handler.
|
||||
5. The I/O handler might pause (yield) many times while waiting for data. Each yield does a `fiber_swap` back to the caller, and each resume does a `fiber_swap` back to this coroutine. But throughout all of that, `coroutine_trampoline` is still on the stack — we're inside the `co->entry()` call.
|
||||
6. Eventually the I/O handler finishes and returns.
|
||||
7. `coroutine_trampoline` resumes after the `co->entry()` line.
|
||||
8. It calls `qemu_coroutine_switch(co, co->caller, COROUTINE_TERMINATE)` — this swaps back to the caller with a "I'm done" flag.
|
||||
9. The `while(true)` loops back to the top. If nobody ever swaps back to this coroutine, it just stays suspended here forever (which is fine — the fiber is deallocated later).
|
||||
10. The entry function **never returns**. The `while(true)` guarantees it.
|
||||
|
||||
**Creating a coroutine:**
|
||||
```c
|
||||
Coroutine *qemu_coroutine_new(void)
|
||||
{
|
||||
CoroutineEmscripten *co = g_malloc0(sizeof(*co));
|
||||
|
||||
co->stack_size = COROUTINE_STACK_SIZE;
|
||||
co->stack = qemu_alloc_stack(&co->stack_size);
|
||||
|
||||
co->asyncify_stack_size = COROUTINE_STACK_SIZE;
|
||||
co->asyncify_stack = g_malloc0(co->asyncify_stack_size);
|
||||
|
||||
emscripten_fiber_init(
|
||||
&co->fiber,
|
||||
coroutine_trampoline, // the infinite-loop entry
|
||||
&co->base, // user_data
|
||||
co->stack, co->stack_size,
|
||||
co->asyncify_stack, co->asyncify_stack_size
|
||||
);
|
||||
|
||||
return &co->base;
|
||||
}
|
||||
```
|
||||
|
||||
Both stacks are **heap-allocated** and persist for the coroutine's entire lifetime.
|
||||
|
||||
**Context switch:**
|
||||
```c
|
||||
CoroutineAction qemu_coroutine_switch(Coroutine *from_, Coroutine *to_,
|
||||
CoroutineAction action)
|
||||
{
|
||||
CoroutineEmscripten *from = DO_UPCAST(CoroutineEmscripten, base, from_);
|
||||
CoroutineEmscripten *to = DO_UPCAST(CoroutineEmscripten, base, to_);
|
||||
|
||||
set_current(to_);
|
||||
to->action = action; // Tell the target why
|
||||
emscripten_fiber_swap(&from->fiber, &to->fiber); // Swap!
|
||||
return from->action; // Read what caller set
|
||||
}
|
||||
```
|
||||
|
||||
Communication between coroutines uses the `action` field: one side sets it before swapping, the other reads it after resuming.
|
||||
|
||||
**Main thread bootstrap (lazy init):**
|
||||
```c
|
||||
Coroutine *qemu_coroutine_self(void)
|
||||
{
|
||||
Coroutine *self = get_current();
|
||||
if (!self) {
|
||||
// First call: capture the main thread as a fiber
|
||||
CoroutineEmscripten *leaderp = g_malloc0(sizeof(*leaderp));
|
||||
leaderp->asyncify_stack = g_malloc0(leader_asyncify_stack_size);
|
||||
leaderp->asyncify_stack_size = leader_asyncify_stack_size;
|
||||
|
||||
emscripten_fiber_init_from_current_context(
|
||||
&leaderp->fiber,
|
||||
leaderp->asyncify_stack,
|
||||
leaderp->asyncify_stack_size
|
||||
);
|
||||
|
||||
set_leader(leaderp);
|
||||
self = &leaderp->base;
|
||||
set_current(self);
|
||||
}
|
||||
return self;
|
||||
}
|
||||
```
|
||||
|
||||
**Cleanup:**
|
||||
```c
|
||||
void qemu_coroutine_delete(Coroutine *co_)
|
||||
{
|
||||
CoroutineEmscripten *co = DO_UPCAST(CoroutineEmscripten, base, co_);
|
||||
qemu_free_stack(co->stack, co->stack_size);
|
||||
g_free(co->asyncify_stack);
|
||||
g_free(co);
|
||||
}
|
||||
```
|
||||
|
||||
Both stacks freed when coroutine destroyed. No stack-local temporaries, no abandoned frames.
|
||||
|
||||
### The Key Insight: Entry Function Must NEVER Return
|
||||
|
||||
Emscripten's fiber API has a rule: **if the fiber's entry function returns, the program terminates**. This is documented in `fiber.h`:
|
||||
|
||||
> "If entry_func returns, the entire program will end, as if main had returned."
|
||||
|
||||
Why? Because when the entry function returns, control goes... nowhere. The fiber's stack is done. There's no caller to return to (the fiber was started from a swap, not a regular function call). Emscripten handles this by treating it as program exit.
|
||||
|
||||
---
|
||||
|
||||
## How Our Implementation Compares to QEMU
|
||||
|
||||
### Our Code (`kicad/thirdparty/libcontext/libcontext.cpp`)
|
||||
|
||||
```cpp
|
||||
[[noreturn]] void wasm_fcontext_entry(void* aArg)
|
||||
{
|
||||
auto* ctx = static_cast<wasm_fcontext*>(aArg);
|
||||
|
||||
// Step 1: Run the coroutine body
|
||||
ctx->entry(ctx->transfer_value);
|
||||
|
||||
// Step 2: The coroutine body returned. We're in trouble.
|
||||
ctx->running = false;
|
||||
|
||||
// Step 3: Try to swap back to the caller
|
||||
if (ctx->return_to)
|
||||
{
|
||||
// Create a TEMPORARY fiber just so we have something to swap FROM
|
||||
emscripten_fiber_t finished_ctx {};
|
||||
alignas(16) char finished_asyncify_stack[64*1024] {};
|
||||
emscripten_fiber_init_from_current_context(&finished_ctx, ...);
|
||||
|
||||
// Swap to the caller. We'll never come back.
|
||||
emscripten_fiber_swap(&finished_ctx, &ctx->return_to->fiber);
|
||||
}
|
||||
|
||||
// Step 4: If we get here, kill everything
|
||||
emscripten_unwind_to_js_event_loop();
|
||||
}
|
||||
```
|
||||
|
||||
The problems:
|
||||
|
||||
1. **Step 2 is dangerous.** The entry function returned. According to Emscripten docs, this should terminate the program. We're in undefined territory.
|
||||
|
||||
2. **Step 3 creates stack-local buffers.** `finished_ctx` and `finished_asyncify_stack` (64KB!) are on this function's stack. When we swap away, this stack frame is abandoned. But Asyncify's bookkeeping still holds pointers to `finished_ctx` (because `emscripten_fiber_swap` saves the asyncify state into it). If Asyncify ever tries to do anything with those pointers, it's reading garbage memory.
|
||||
|
||||
3. **Step 4 uses `emscripten_unwind_to_js_event_loop()`**. This function says "I'm done with all WASM execution, return to the browser event loop." It tears down the ENTIRE WASM call stack — not just this fiber, but everything. If this happens during KiCad's startup sequence, the startup dies.
|
||||
|
||||
### The Three Differences
|
||||
|
||||
| Issue | Our Code | QEMU |
|
||||
|-------|----------|------|
|
||||
| Entry function returns? | Yes, then handles it | Never - `while(true)` |
|
||||
| Stack-local asyncify buffers? | Yes (64KB on stack) | No - all heap-allocated |
|
||||
| `emscripten_unwind_to_js_event_loop`? | Yes, as fallback | Not used |
|
||||
|
||||
---
|
||||
|
||||
## How Would We Adopt QEMU's Pattern?
|
||||
|
||||
### The change is small
|
||||
|
||||
The fix is to replace our `wasm_fcontext_entry` with a QEMU-style trampoline:
|
||||
|
||||
**QEMU-style replacement:**
|
||||
```cpp
|
||||
[[noreturn]] void wasm_fcontext_entry(void* aArg)
|
||||
{
|
||||
auto* ctx = static_cast<wasm_fcontext*>(aArg);
|
||||
|
||||
while (true) {
|
||||
// Run the coroutine body
|
||||
ctx->entry(ctx->transfer_value);
|
||||
|
||||
// Coroutine finished. Swap back to whoever started us.
|
||||
ctx->running = false;
|
||||
|
||||
if (ctx->return_to) {
|
||||
ctx->return_to->transfer_value = 0;
|
||||
ctx->return_to->running = true;
|
||||
g_current_context = ctx->return_to;
|
||||
emscripten_fiber_swap(&ctx->fiber, &ctx->return_to->fiber);
|
||||
// If we're swapped back to (unlikely), the while(true) loops
|
||||
}
|
||||
}
|
||||
// We never reach here
|
||||
}
|
||||
```
|
||||
|
||||
Key differences:
|
||||
1. `while(true)` ensures we never return from the entry function
|
||||
2. We swap using `ctx->fiber` (the coroutine's own, heap-allocated fiber) instead of creating a stack-local temporary
|
||||
3. No `emscripten_unwind_to_js_event_loop()` — we just stay in the loop
|
||||
|
||||
### How hard is the change?
|
||||
|
||||
**Maybe 15-20 lines changed** in one file (`kicad/thirdparty/libcontext/libcontext.cpp`). The architecture is already right — we use Emscripten fibers, we have a `wasm_fcontext` struct with proper fields, we have `g_current_context` tracking. The only wrong part is the entry-return handling.
|
||||
|
||||
The change is small but **the testing is critical**. After making it:
|
||||
1. Rebuild the WASM module (`docker/build.sh`)
|
||||
2. Run the PCBnew E2E test (`cd tests && npm run test:kicad -- --grep "select draw lines"`)
|
||||
3. Check if toolbars now fully appear (the `tools: []` should become non-empty)
|
||||
4. Check if drawing actually works
|
||||
|
||||
### What could go wrong?
|
||||
|
||||
The biggest risk is that `jump_fcontext`'s semantics don't perfectly match what KiCad's COROUTINE class expects. Specifically:
|
||||
|
||||
- KiCad's COROUTINE uses `jump_fcontext(&old_ctx, new_ctx, value)` where the returned `intptr_t` is a pointer to `INVOCATION_ARGS` that tells the coroutine why it was resumed (FROM_ROOT, FROM_ROUTINE, CONTINUE_AFTER_ROOT).
|
||||
- If the trampoline loop doesn't correctly set the transfer value before swapping back, the caller might misinterpret why the coroutine stopped.
|
||||
|
||||
But this is testable — the E2E test will catch it.
|
||||
|
||||
---
|
||||
|
||||
## Verification of the Investigation Document
|
||||
|
||||
The original investigation document (`0001-kicad-wasm-tool-activation-investigation.md`) was reviewed and verified:
|
||||
|
||||
| Claim | Verdict |
|
||||
|-------|---------|
|
||||
| Fiber entry callback was a no-op | TRUE |
|
||||
| inject-dyncall-shims.sh fixes it | TRUE |
|
||||
| emscripten_fiber_swap needed in ASYNCIFY_IMPORTS | TRUE |
|
||||
| Tools array is empty / startup stalls | TRUE |
|
||||
| Problem is in WASM coroutine layer, not KiCad UI | TRUE (well-supported) |
|
||||
| libcontext impl is "experimental / not clean" | FALSE - it's well-structured production code |
|
||||
| Coroutine return is the remaining issue | PLAUSIBLE but not proven |
|
||||
| KiCad investigative changes exist in files | NOT FOUND (likely already reverted) |
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
### What's already correct
|
||||
- The `wasm_fcontext` struct design
|
||||
- The `make_fcontext` / `jump_fcontext` API mapping to fibers
|
||||
- The `inject-dyncall-shims.sh` JS patching
|
||||
- The `apply-asyncify.sh` import configuration
|
||||
- The main context lazy initialization
|
||||
|
||||
### What needs fixing
|
||||
- `wasm_fcontext_entry`: add `while(true)`, remove stack-local buffers, remove `emscripten_unwind_to_js_event_loop()`
|
||||
- Dead code cleanup: delete `wasm/libcontext/` directory
|
||||
|
||||
### Alternative approaches considered
|
||||
|
||||
| Approach | Viability | Notes |
|
||||
|----------|-----------|-------|
|
||||
| Fix current Asyncify + fibers | HIGH | Adopt QEMU's trampoline pattern |
|
||||
| JSPI (JS Promise Integration) | MEDIUM | Future Asyncify replacement, limited browser support |
|
||||
| Event-driven state machines | NOT VIABLE | Rewrites every KiCad tool |
|
||||
| C++20 stackless coroutines | NOT COMPATIBLE | KiCad needs stackful suspension |
|
||||
| WASM Stack Switching proposal | FUTURE | Not standardized yet |
|
||||
|
||||
### References
|
||||
|
||||
- [QEMU coroutine-wasm.c](https://github.com/qemu/qemu/blob/master/util/coroutine-wasm.c) — gold standard implementation
|
||||
- [Emscripten fiber.h docs](https://emscripten.org/docs/api_reference/fiber.h.html) — API reference
|
||||
- [Asyncify blog post](https://kripken.github.io/blog/wasm/2019/07/16/asyncify.html) — deep technical dive
|
||||
- [Binaryen Asyncify.cpp](https://github.com/WebAssembly/binaryen/blob/main/src/passes/Asyncify.cpp) — the compiler pass source
|
||||
- [Fiber PR #9859](https://github.com/emscripten-core/emscripten/pull/9859) — design discussion
|
||||
- [minicoro](https://github.com/edubart/minicoro) — single-header coroutine lib with WASM support
|
||||
- [Issue #13302](https://github.com/emscripten-core/emscripten/issues/13302) — fiber swap return value bug
|
||||
- [Issue #12733](https://github.com/emscripten-core/emscripten/issues/12733) — dynCall removal discussion
|
||||
|
|
@ -0,0 +1,304 @@
|
|||
# wxAuiToolBar Registration — Tool Selection Fix
|
||||
|
||||
## Context
|
||||
|
||||
The nested Asyncify collision bug (see `0002-wasm-coroutine-deep-dive.md` and `../../research/threading_1.md`) is fixed. KiCad WASM now loads through the startup wizard without crashing, and the full PCBnew UI renders — menus, left drawing-tool sidebar with Line/Circle/Rectangle icons, layer panel, PCB canvas — all visible.
|
||||
|
||||
But tools still don't work end-to-end:
|
||||
|
||||
- **User observation**: clicking the Draw Lines tool in the browser doesn't visibly select it or make it function.
|
||||
- **E2E test**: `select draw lines and draw on the board` can't even attempt a click — it fails earlier at `wxElementRegistry.findAllRendered({ elementType: 'tool' })` because the returned array is empty.
|
||||
|
||||
Both signals point at the same gap: **wxAuiToolBar never registers its tools with the rendered-element registry**. The intended outcome of this fix is to make all wxAuiToolBar buttons (Draw Lines and siblings) clickable, selectable, and functional in the browser build.
|
||||
|
||||
---
|
||||
|
||||
## Root Cause
|
||||
|
||||
### The registry and how it gets populated
|
||||
|
||||
`window.wxElementRegistry` is a JS-side registry of UI elements used by Playwright tests to find controls by type/label/tooltip. For elements that are not standalone wxWindow instances (e.g., toolbar buttons rendered as pixels on a parent canvas), wxWidgets calls a C++ bridge `WasmRegisterRenderedElement()` which invokes the JS helper `wxRenderedElementRegister()`.
|
||||
|
||||
Canonical definition: `wxwidgets/src/wasm/window.cpp:182-214`:
|
||||
```cpp
|
||||
void WasmRegisterRenderedElement(
|
||||
wxWindow* parent,
|
||||
const char* elementType, // "tool", "menuitem", "sash", "auipart", ...
|
||||
const char* subType,
|
||||
int index,
|
||||
const wxString& label,
|
||||
const wxString& tooltip,
|
||||
int screenX, int screenY,
|
||||
int width, int height,
|
||||
bool enabled);
|
||||
```
|
||||
|
||||
### Where it's currently called
|
||||
|
||||
Grepping `wxwidgets/src/` for `WasmRegisterRenderedElement`:
|
||||
|
||||
| File | What it registers |
|
||||
|---|---|
|
||||
| `src/univ/toolbar.cpp:557-607` | Regular wxToolBar items (in `RecalcToolBitmapCache`) |
|
||||
| `src/univ/menu.cpp` | Menu bar items, popup menu items |
|
||||
| `src/aui/framemanager.cpp:2687-2787` | AUI pane captions, close/pin/maximize buttons |
|
||||
| `src/aui/tabart.cpp:392,1137` | Tab headers |
|
||||
| `src/univ/textctrl.cpp:4304-4319` | Text control segments |
|
||||
| `src/propgrid/propgrid.cpp:2509-2537` | Property grid rows |
|
||||
| `src/stc/stc.cpp:5203-5213` | Styled text cells |
|
||||
|
||||
### The gap
|
||||
|
||||
`wxwidgets/src/aui/auibar.cpp` has **zero** `__EMSCRIPTEN__` blocks and **zero** calls to `WasmRegisterRenderedElement`. Verified:
|
||||
```
|
||||
$ grep -nE "__EMSCRIPTEN__|WasmRegister" wxwidgets/src/aui/auibar.cpp
|
||||
(no matches)
|
||||
|
||||
$ git log --oneline -n 10 src/aui/auibar.cpp
|
||||
# Only upstream wxWidgets commits — our fork hasn't modified this file.
|
||||
```
|
||||
|
||||
KiCad's left drawing-tool sidebar is a `wxAuiToolBar` (not `wxToolBar`), which is why its tools are invisible to the registry.
|
||||
|
||||
### Test log evidence
|
||||
|
||||
From `tests/logs/kicad/pcbnew/pcbnew-spec-ts-pcbnew-wasm-select-draw-lines-and-draw-on-the-board.log`:
|
||||
|
||||
```
|
||||
[TEST] rendered summary {"count":42,"byType":{
|
||||
"searchctrl":3,"searchbutton":4,"sash":2,"auipart":4,
|
||||
"combobutton":8,"combotextarea":8,"textctrl":1,"tab":3,"menuitem":9
|
||||
},"tools":[]}
|
||||
```
|
||||
|
||||
Everything else registers. `tools` is the only empty bucket.
|
||||
|
||||
### The log is otherwise clean
|
||||
|
||||
Filtering out diagnostic output (`WASM_FCONTEXT`, `DIAG_*`, `wxLog DEBUG`) leaves only three substantive lines, all informational:
|
||||
```
|
||||
[DIAG_SHOWMODAL] About to call startModal()
|
||||
Debug: EndModal: 5100
|
||||
[DIAG_SHOWMODAL] startModal() returned 5100
|
||||
```
|
||||
|
||||
No exceptions, no crashes, no fiber errors, no `jump-ghost`, `main_refresh=1` stable. The nested Asyncify fix is holding. **The only remaining issue between "UI loads" and "tool works" is this registration gap.**
|
||||
|
||||
### Separating the two signals
|
||||
|
||||
The registry gap directly explains the **test failure**. It does NOT directly explain the **user's manual observation** — registry population is test-only infrastructure and has no effect on in-browser interactivity.
|
||||
|
||||
Hypothesis: once tools are registered, the test will click the tool via coordinates from the registry and produce a log that either shows the click succeeded (no bug — user's "doesn't select" was a display misreading because state wasn't exposed) or shows concrete failure evidence (real activation bug, e.g., another variant of coroutine/asyncify interaction). Either way, the registration fix is strictly additive and unblocks diagnosis.
|
||||
|
||||
---
|
||||
|
||||
## The Fix
|
||||
|
||||
### Summary
|
||||
|
||||
Two small changes, one new block:
|
||||
|
||||
1. **Extend the registry signature** to include `checked` state (needed so the test can verify selection).
|
||||
2. **Add a registration block** to `wxAuiToolBar::OnPaint()` following the `univ/toolbar.cpp` pattern.
|
||||
3. **Update existing callers** to pass a `checked` value (`false` for non-toggleable, real state for `wxItemCheck/wxItemRadio`).
|
||||
|
||||
### Change 1 — extend `WasmRegisterRenderedElement` signature
|
||||
|
||||
**File: `wxwidgets/src/wasm/window.cpp`** (function at line 182)
|
||||
|
||||
Add a `bool checked` parameter and pass it through to the JS helper:
|
||||
|
||||
```cpp
|
||||
void WasmRegisterRenderedElement(
|
||||
wxWindow* parent,
|
||||
const char* elementType,
|
||||
const char* subType,
|
||||
int index,
|
||||
const wxString& label,
|
||||
const wxString& tooltip,
|
||||
int screenX, int screenY,
|
||||
int width, int height,
|
||||
bool enabled,
|
||||
bool checked) // ← NEW
|
||||
{
|
||||
if (!parent) return;
|
||||
uintptr_t parentId = reinterpret_cast<uintptr_t>(parent);
|
||||
|
||||
EM_ASM({
|
||||
var id = $0.toString() + ':' + UTF8ToString($1) + ':' + $2;
|
||||
wxRenderedElementRegister(
|
||||
id,
|
||||
$0.toString(),
|
||||
UTF8ToString($1), // elementType
|
||||
UTF8ToString($3), // subType
|
||||
UTF8ToString($4), // label
|
||||
UTF8ToString($5), // tooltip
|
||||
$6, $7, $8, $9, // x, y, w, h
|
||||
$10 ? true : false, // enabled
|
||||
$2, // index
|
||||
$11 ? true : false // ← checked
|
||||
);
|
||||
},
|
||||
parentId, elementType, index, subType,
|
||||
label.utf8_str().data(), tooltip.utf8_str().data(),
|
||||
screenX, screenY, width, height,
|
||||
enabled, checked);
|
||||
}
|
||||
```
|
||||
|
||||
**File: `wxwidgets/build/wasm/wx.js`** (helper at line 297)
|
||||
|
||||
```javascript
|
||||
function wxRenderedElementRegister(
|
||||
id, parentId, elementType, subType,
|
||||
label, tooltip, screenX, screenY, width, height,
|
||||
enabled, index, checked) // ← NEW
|
||||
{
|
||||
if (window.wxElementRegistry) {
|
||||
window.wxElementRegistry.registerRendered(id, {
|
||||
id, parentId, elementType, subType,
|
||||
label, tooltip,
|
||||
screenX, screenY, width, height,
|
||||
centerX: screenX + Math.floor(width / 2),
|
||||
centerY: screenY + Math.floor(height / 2),
|
||||
enabled,
|
||||
index,
|
||||
checked: !!checked, // ← NEW
|
||||
lastUpdated: Date.now()
|
||||
});
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Also update `wxRenderedElementUpdate` (same file, around line 321) similarly, so subsequent updates can change `checked`.
|
||||
|
||||
### Change 2 — register wxAuiToolBar tools
|
||||
|
||||
**File: `wxwidgets/src/aui/auibar.cpp`** (inside `OnPaint`, after the main item-paint loop, before the overflow paint at line ~2501)
|
||||
|
||||
```cpp
|
||||
#ifdef __EMSCRIPTEN__
|
||||
// Update element registry with toolbar tools (for E2E test automation).
|
||||
// Runs after every paint so state (enabled/checked, layout) stays current.
|
||||
extern void WasmRegisterRenderedElement(
|
||||
wxWindow* parent, const char* elementType, const char* subType,
|
||||
int index, const wxString& label, const wxString& tooltip,
|
||||
int screenX, int screenY, int width, int height,
|
||||
bool enabled, bool checked);
|
||||
extern void WasmUnregisterRenderedElementsByParent(wxWindow* parent);
|
||||
|
||||
WasmUnregisterRenderedElementsByParent(this);
|
||||
|
||||
wxPoint screenPos = GetScreenPosition();
|
||||
for (size_t j = 0, itemCount = m_items.GetCount(); j < itemCount; ++j)
|
||||
{
|
||||
wxAuiToolBarItem& item = m_items.Item(j);
|
||||
|
||||
if (!item.m_sizerItem)
|
||||
continue;
|
||||
if (item.m_kind == wxITEM_SEPARATOR)
|
||||
continue;
|
||||
|
||||
wxRect itemRect = item.m_sizerItem->GetRect();
|
||||
|
||||
// Skip items scrolled off the end (match the paint loop's cutoff)
|
||||
if ((horizontal && itemRect.x + itemRect.width >= last_extent) ||
|
||||
(!horizontal && itemRect.y + itemRect.height >= last_extent))
|
||||
continue;
|
||||
|
||||
const char* subType = (item.m_kind == wxITEM_CONTROL) ? "control" : "button";
|
||||
bool isEnabled = !(item.m_state & wxAUI_BUTTON_STATE_DISABLED);
|
||||
bool isChecked = (item.m_state & wxAUI_BUTTON_STATE_CHECKED) != 0;
|
||||
|
||||
WasmRegisterRenderedElement(
|
||||
this,
|
||||
"tool",
|
||||
subType,
|
||||
static_cast<int>(j),
|
||||
item.m_label,
|
||||
item.m_shortHelp,
|
||||
screenPos.x + itemRect.x,
|
||||
screenPos.y + itemRect.y,
|
||||
itemRect.width,
|
||||
itemRect.height,
|
||||
isEnabled,
|
||||
isChecked
|
||||
);
|
||||
}
|
||||
#endif
|
||||
```
|
||||
|
||||
Notes:
|
||||
- `item.m_label` / `item.m_shortHelp` are the verified field names (`wxwidgets/include/wx/aui/auibar.h:231,235`).
|
||||
- `wxAUI_BUTTON_STATE_CHECKED` is already how `wxAuiToolBar::OnLeftUp` tracks toggle state (see line 2676 `m_actionItem->m_state & wxAUI_BUTTON_STATE_CHECKED`).
|
||||
- Placing the block inside OnPaint means every repaint refreshes the registry, which keeps `checked`/`enabled` state synchronized with visible state without needing a separate update path.
|
||||
|
||||
### Change 3 — update existing callers to pass `checked`
|
||||
|
||||
Every existing `WasmRegisterRenderedElement` call must pass a new final arg. Most don't have meaningful checked state:
|
||||
|
||||
- `src/univ/menu.cpp` — pass `false` (or `menuItem->IsChecked()` for check-menu-items, already available)
|
||||
- `src/aui/framemanager.cpp` (pane parts) — pass `false`
|
||||
- `src/aui/tabart.cpp` — pass `false` for non-selected tabs, `true` for the active tab (`page.active`)
|
||||
- `src/univ/textctrl.cpp`, `src/propgrid/propgrid.cpp`, `src/stc/stc.cpp` — pass `false`
|
||||
- `src/univ/toolbar.cpp` — pass `tool->IsToggled()` (real value for the regular wxToolBar path)
|
||||
|
||||
This is a small mechanical change: add `, false` (or the appropriate value) to each existing call site.
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
### Build
|
||||
|
||||
1. **wxWidgets standalone build** (fast): `./scripts/build-wxuniversal-wasm.sh`
|
||||
2. **KiCad rebuild** (needed because KiCad statically links wxWidgets; this is the slow step): `./docker/build.sh`
|
||||
3. **Setup test artifacts**: handled automatically by `npm run test:kicad`'s `setup:kicad` step.
|
||||
|
||||
### Run
|
||||
|
||||
```bash
|
||||
cd tests
|
||||
npm run test:kicad
|
||||
```
|
||||
|
||||
Expectations for `pcbnew.spec.ts`:
|
||||
|
||||
- **Test 1** (`click through setup wizard to load PCBnew`): still passes (already passing, unaffected by this change).
|
||||
- **Test 2** (`select draw lines and draw on the board`): now proceeds past the `findAllRendered` poll. Three possible outcomes:
|
||||
1. **Passes fully** — tools were simply invisible to the test before; the user's "doesn't select" manual report was a display misreading (likely state changed but they didn't see the visual update, or they tested a stale build).
|
||||
2. **Fails at the `checked` poll (5s)** — tool renders, click reaches it, but activation path (ACTION_TOOLBAR → TOOL_MANAGER → coroutine) has a real functional bug. Follow up using the log.
|
||||
3. **Fails at the initial `findAllRendered` poll still** — registration isn't firing; something wrong with the build/binding. Debug by inspecting the generated `pcbnew.js` for the new signature.
|
||||
|
||||
### Diagnostic signals in the log
|
||||
|
||||
After the click, watch for these patterns:
|
||||
|
||||
- `[WASM_FCONTEXT] entry-call ctx=…` new fiber created after click → activation coroutine started. Any subsequent failure is in tool logic, not plumbing.
|
||||
- No fiber activity at all after the click → click didn't route to ACTION_TOOLBAR. Suspect event routing through the canvas (`wxwidgets/src/wasm/window.cpp` mouse handlers, possibly `kicad/common/gal/webgl/webgl_gal.cpp` which has a WASM-specific uncommitted change).
|
||||
- Fiber starts but never yields / doesn't hit the tool's `Wait()` loop → similar class of coroutine bug to the Asyncify fix, but different trigger.
|
||||
|
||||
### Follow-up scenarios
|
||||
|
||||
If the test still fails after this fix, use the above signals to narrow to:
|
||||
|
||||
- **Rendering-only**: `Refresh(false); Update()` already runs in `wxAuiToolBar::OnLeftUp` at line 2683–2684, so this is unlikely; but if the registry updates yet the canvas visibly doesn't, something is suppressing paint.
|
||||
- **Coroutine activation**: new variant of nested-asyncify (maybe menu → tool → dialog nesting). Extend `coroutine-nested` standalone test with the matching scenario.
|
||||
- **Event routing**: audit the DOM-event → wxWidgets-event bridge. If clicks on coordinates in the canvas aren't reaching wxAuiToolBar, the bridge has regressed.
|
||||
|
||||
---
|
||||
|
||||
## Files Touched
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `wxwidgets/src/wasm/window.cpp` | Add `bool checked` param to `WasmRegisterRenderedElement` signature |
|
||||
| `wxwidgets/build/wasm/wx.js` | Add `checked` arg to `wxRenderedElementRegister` and `wxRenderedElementUpdate` JS helpers |
|
||||
| `wxwidgets/src/aui/auibar.cpp` | **NEW** registration block in `OnPaint()` (~30 lines in `#ifdef __EMSCRIPTEN__`) |
|
||||
| `wxwidgets/src/univ/toolbar.cpp` | Pass `tool->IsToggled()` as new final arg |
|
||||
| `wxwidgets/src/univ/menu.cpp` | Pass `false` (or `IsChecked()` for check items) |
|
||||
| `wxwidgets/src/aui/framemanager.cpp` | Pass `false` |
|
||||
| `wxwidgets/src/aui/tabart.cpp` | Pass `page.active` where appropriate, else `false` |
|
||||
| `wxwidgets/src/univ/textctrl.cpp`, `propgrid/propgrid.cpp`, `stc/stc.cpp` | Pass `false` |
|
||||
|
||||
Net: +~50 lines of new code, ~8 files touched. The wxWidgets fork drift grows by one localized patch — no protocol or architectural change.
|
||||
|
|
@ -0,0 +1,172 @@
|
|||
# RTree `Classify` duplicate-index crash on PCB load (wasm-only)
|
||||
|
||||
## TL;DR
|
||||
|
||||
A debug-build pcbnew in the browser used to abort on every `.kicad_pcb` open with
|
||||
|
||||
```
|
||||
Aborted(Assertion failed: !a_parVars->m_taken[a_index],
|
||||
at: kicad/thirdparty/rtree/geometry/rtree.h:1771, Classify)
|
||||
```
|
||||
|
||||
Native debug builds never hit it. Root cause was a **wasm-only integer overflow** triggered by a single bad template argument in upstream KiCad:
|
||||
|
||||
```cpp
|
||||
// kicad/libs/kimath/src/geometry/shape_poly_set.cpp:1927 ← before
|
||||
RTree<intptr_t, intptr_t, 2, intptr_t> rtree;
|
||||
// ^^^^^^^^ ELEMTYPEREAL = intptr_t
|
||||
```
|
||||
|
||||
Every other RTree instantiation in KiCad uses `double` for the 4th template parameter (the volume/area type). This one used `intptr_t`, which on wasm32 is `int32_t` — too narrow to hold `RectSphericalVolume`'s `sumOfSquares` for typical KiCad nanometer extents (`~10⁸ per axis → ~10¹⁴ area`). On native targets `intptr_t = int64_t` happens to absorb the result, so the bug stays latent.
|
||||
|
||||
Fix: change `intptr_t` to `double` on that one line. After the fix, both `kicad/demos/microwave/microwave.kicad_pcb` and `kicad/demos/pic_programmer/pic_programmer.kicad_pcb` load and render cleanly through pcbnew's File → Open in the wasm debug build — see `tests/kicad/load-pcb.spec.ts` and the baselines under `tests/baseline-screenshots/load-pcb-*.png`.
|
||||
|
||||
---
|
||||
|
||||
## Investigation trail
|
||||
|
||||
### 1. Reproducing and isolating
|
||||
|
||||
The crash reproduced deterministically on every non-empty board (`microwave`, `flat_hierarchy`, `pic_programmer`). The first idea — that the bug had something to do with the file-open path itself — was disproven by the spike test at `tests/kicad/load-pcb.spec.ts`: the menu and dialog drive end-to-end, the abort fires after the file is selected and parsing has begun, well inside `PCB_IO_KICAD_SEXPR::LoadBoard()`.
|
||||
|
||||
The static read of `rtree.h` left two plausible mechanisms:
|
||||
|
||||
1. `seed0 == seed1` in `PickSeeds` because `if (waste >= worst)` never fires, which only happens if `worst` (= `-coverSplitArea - 1`) goes NaN/`-inf`. That points at integer-overflow / UB inside `RectVolume` / `RectSphericalVolume`.
|
||||
2. Out-of-band corruption of `m_taken[]` — re-entrant Insert during iteration, stack/heap overrun, etc.
|
||||
|
||||
Neither could be decided from code alone.
|
||||
|
||||
### 2. Targeted instrumentation
|
||||
|
||||
Phase A added a temporary `KICAD_RTREE_DIAG`-gated block in `kicad/thirdparty/rtree/geometry/rtree.h` (since reverted) that, on the assertion's would-fire condition, dumped:
|
||||
|
||||
- the call-site tag (`PickSeeds-0` / `PickSeeds-1` / `ChoosePartition`, derived from current `m_count[]`),
|
||||
- `a_index`, `a_group`, `m_total`, `m_minFill`, `m_count[]`,
|
||||
- `m_taken[]`, `m_partition[]`,
|
||||
- `m_coverSplitArea`, `m_area[0..1]`,
|
||||
- every branch's `m_min/m_max/span` per dimension,
|
||||
|
||||
and `return`-ed without aborting. Wired up via `./docker/build.sh --diag rtree` (gated through the existing `--diag=<flags>` plumbing in `scripts/kicad/build-pcbnew.sh`).
|
||||
|
||||
### 3. The dump that nailed it
|
||||
|
||||
Two consecutive runs from the diagnostic build (excerpt — first run shown):
|
||||
|
||||
```
|
||||
[RTREE-DIAG] Classify duplicate src=PickSeeds-1 idx=0 grp=1
|
||||
total=9 minFill=4 count=[1,0]
|
||||
coverSplitArea=-2099823776 area=[717833776,0]
|
||||
[RTREE-DIAG] m_taken=100000000 m_partition=0 -1 -1 -1 -1 -1 -1 -1 -1
|
||||
[RTREE-DIAG] branchBuf[0].rect=[67429380..67932300 span=502920]
|
||||
[117822980..117822980 span=0]
|
||||
[RTREE-DIAG] branchBuf[1].rect=[69441060..69943980 span=502920]
|
||||
[117805200..117810280 span=5080]
|
||||
...
|
||||
[RTREE-DIAG] branchBuf[8].rect=[82008980..82511900 span=502920]
|
||||
[117955060..117962680 span=7620]
|
||||
```
|
||||
|
||||
Key observations:
|
||||
|
||||
- `src=PickSeeds-1` plus `m_taken[0]=1` and `idx=0` → `seed0 == seed1 == 0`. Exactly the smoking-gun for hypothesis (1).
|
||||
- Every per-dim `span` is small and positive (`5e5`, `5e3`, etc.). No leaf overflow.
|
||||
- But `coverSplitArea = -2099823776` — **negative**, which `RectSphericalVolume` cannot algebraically produce (`sumOfSquares * unitSphereVolume` is non-negative).
|
||||
- The magnitude is interesting too: `-2099823776 ≈ -INT_MAX × 0.98`. That's not a noisy double; that's an exact 32-bit signed-integer value sitting in a double slot.
|
||||
- `area[0] = 717833776` (run 1) and `-919295316` (run 2) — also "round int32" magnitudes, also wildly wrong vs. the expected `~1.99e11` for the seed-0 branch.
|
||||
|
||||
### 4. The actual bug
|
||||
|
||||
Every value the diag pulled out of a "double" field had the shape of a 32-bit signed integer. That means the field is being computed and stored as `int32_t` somewhere, not as `double`. The `ELEMTYPEREAL` template parameter is supposed to be the wide floating-point type for volume math.
|
||||
|
||||
Grep across the kicad submodule:
|
||||
|
||||
```
|
||||
kicad/include/view/view_rtree.h:36 RTree<VIEW_ITEM*, int, 2, double>
|
||||
kicad/pcbnew/drc/drc_rtree.h:77 RTree<ITEM_WITH_SHAPE*, int, 2, double>
|
||||
kicad/pcbnew/connectivity/connectivity_rtree.h:45 RTree<T, int, 3, double>
|
||||
kicad/pcbnew/connectivity/connectivity_items.h:395 RTree<const SHAPE*, int, 2, double>
|
||||
kicad/eeschema/sch_rtree.h:42 RTree<SCH_ITEM*, int, 3, double>
|
||||
kicad/libs/kimath/include/geometry/shape_index.h RTree<T, int, 2, double>
|
||||
kicad/libs/kimath/src/geometry/shape_poly_set.cpp:1927 RTree<intptr_t, intptr_t, 2, intptr_t> ← OUTLIER
|
||||
```
|
||||
|
||||
Every other instantiation passes `double` for `ELEMTYPEREAL`. `splitCollinearOutlines` (called from `SHAPE_POLY_SET::Simplify`, which fires during any board load that has polygon-bearing items like the microwave demo's RF "footprints" or pic_programmer's pads) passes `intptr_t`.
|
||||
|
||||
On wasm32, `intptr_t` is `int32_t` because pointers are 32-bit. `RectSphericalVolume`'s loop:
|
||||
|
||||
```cpp
|
||||
ELEMTYPEREAL sumOfSquares = 0;
|
||||
for (int index = 0; index < NUMDIMS; ++index) {
|
||||
ELEMTYPEREAL halfExtent =
|
||||
((ELEMTYPEREAL) max[index] - (ELEMTYPEREAL) min[index]) * 0.5f;
|
||||
sumOfSquares += halfExtent * halfExtent;
|
||||
}
|
||||
return sumOfSquares * m_unitSphereVolume;
|
||||
```
|
||||
|
||||
…becomes int32 arithmetic. For span_x = 1.5×10⁷, halfExtent² = 5.7×10¹³ — far past `INT_MAX = 2.15×10⁹`. The multiplication wraps, `sumOfSquares` ends up as garbage (the `-2099823776` we observed), `m_coverSplitArea` follows, `worst = -coverSplitArea - 1` ends up hugely positive, and PickSeeds' `if (waste >= worst)` never fires for any pair. `seed0 = seed1 = 0` (their default), `Classify(0, 0)` succeeds, `Classify(0, 1)` trips the assertion.
|
||||
|
||||
On native (x86_64, arm64), `intptr_t = int64_t` so the same math succeeds even with the wrong template arg. KiCad's CI has therefore never seen the assertion.
|
||||
|
||||
### 5. The fix
|
||||
|
||||
```diff
|
||||
- RTree<intptr_t, intptr_t, 2, intptr_t> rtree;
|
||||
+ // ELEMTYPEREAL must be a wide floating-point type: RectSphericalVolume's
|
||||
+ // sumOfSquares grows quadratically with extents, easily exceeding 2^31 for
|
||||
+ // KiCad nanometer coordinates (~10^8 per axis -> ~10^14 area). All other
|
||||
+ // RTree instantiations in KiCad use `double` for that fourth argument; this
|
||||
+ // one was using `intptr_t`, which silently overflowed on wasm32 (where
|
||||
+ // intptr_t is 32-bit) and tripped an assertion deep in PickSeeds during
|
||||
+ // PCB load. See features/fix-asyncify-O2-and-modal-promise-rejection/
|
||||
+ // rtree-debug-findings.md for the full diagnosis trail.
|
||||
+ RTree<intptr_t, intptr_t, 2, double> rtree;
|
||||
```
|
||||
|
||||
`git -C kicad diff origin/master -- thirdparty/rtree/geometry/rtree.h` is empty — the rtree third-party code stays vanilla upstream. The only divergence from upstream is `libs/kimath/src/geometry/shape_poly_set.cpp:1927`.
|
||||
|
||||
### 6. Verification
|
||||
|
||||
After the fix:
|
||||
|
||||
- `npm run test:kicad:firefox -- load-pcb.spec` → both `microwave` and `pic_programmer` tests pass (~38s total).
|
||||
- `tests/logs/kicad/load-pcb/*.log` contains the `[KICAD] Wrote …{microwave,pic_programmer}.kicad_pcb` injection lines and no `Aborted(` line, no `[RTREE-DIAG]` line.
|
||||
- `tests/baseline-screenshots/load-pcb-microwave.png` shows the microwave's two distinctive RF polygon footprints as horizontal red bars on F.Cu, `Pads: 8` in the status bar.
|
||||
- `tests/baseline-screenshots/load-pcb-pic_programmer.png` shows the pic_programmer's fully-routed multi-IC layout with traces visible across F.Cu.
|
||||
- `pcbnew.spec.ts` (empty-board path) still passes — no regression.
|
||||
|
||||
There remains a separate, pre-existing wasm-port issue downstream: after the board has fully rendered, KiCad's clipboard polling path hits a `RuntimeError: index out of bounds` (and `indirect call to null` on Firefox) inside `__asyncjs__js_clipboardHasText` → `Asyncify.handleSleep`. That's not blocking the load (the screenshot is fully painted by then) and is out of scope for this fix; the load-pcb test explicitly filters its assertion to the two things it cares about — no `[RTREE-DIAG]` and no `Aborted(` — leaving downstream clipboard noise for a follow-up.
|
||||
|
||||
---
|
||||
|
||||
## Reportable summary for upstream KiCad
|
||||
|
||||
Below is a self-contained version suitable for an upstream bug report; pull it as-is.
|
||||
|
||||
> **Title:** RTree `ELEMTYPEREAL = intptr_t` in `SHAPE_POLY_SET::splitCollinearOutlines` overflows on 32-bit-pointer targets
|
||||
>
|
||||
> **File:** `libs/kimath/src/geometry/shape_poly_set.cpp:1927`
|
||||
>
|
||||
> ```cpp
|
||||
> RTree<intptr_t, intptr_t, 2, intptr_t> rtree;
|
||||
> ```
|
||||
>
|
||||
> The 4th template parameter is `ELEMTYPEREAL`, which `RTree::RectVolume` and `RectSphericalVolume` use for the accumulated volume / sum-of-squares math. Every other `RTree<>` instantiation in KiCad passes `double` for that slot (`view_rtree.h`, `drc_rtree.h`, `connectivity_rtree.h`, `connectivity_items.h`, `eeschema/sch_rtree.h`, `kimath/include/geometry/shape_index.h`). This one passes `intptr_t`.
|
||||
>
|
||||
> On 64-bit-pointer hosts `intptr_t == int64_t` and the math fits, so the bug is latent. On 32-bit-pointer targets (wasm32, 32-bit Linux, etc.) `intptr_t == int32_t`, so `RectSphericalVolume`'s `sumOfSquares` overflows for typical KiCad nanometer extents (`~10⁸ per axis → halfExtent² ~5×10¹³`, well past `INT_MAX = 2.15×10⁹`). The wrap turns `m_coverSplitArea` negative, makes `worst = -m_coverSplitArea - 1` huge positive in `PickSeeds`, so `if (waste >= worst)` never fires for any pair. `seed0 == seed1 == 0` survives the loop, `Classify(0, 0)` succeeds, `Classify(0, 1)` trips
|
||||
>
|
||||
> ```
|
||||
> Assertion failed: !a_parVars->m_taken[a_index]
|
||||
> (thirdparty/rtree/geometry/rtree.h, line 1771)
|
||||
> ```
|
||||
>
|
||||
> in debug builds. Release builds silently store a corrupt tree.
|
||||
>
|
||||
> Fix is one character class:
|
||||
>
|
||||
> ```diff
|
||||
> -RTree<intptr_t, intptr_t, 2, intptr_t> rtree;
|
||||
> +RTree<intptr_t, intptr_t, 2, double> rtree;
|
||||
> ```
|
||||
>
|
||||
> Reproduces on any non-empty board on a wasm32 debug build (we've hit it on the `microwave`, `flat_hierarchy`, and `pic_programmer` demos). Should also reproduce on 32-bit Linux debug builds.
|
||||
74
docs/features/gerbview/0001-gerbview-port.md
Normal file
74
docs/features/gerbview/0001-gerbview-port.md
Normal file
|
|
@ -0,0 +1,74 @@
|
|||
# Gerber Viewer (gerbview) WASM port — design notes
|
||||
|
||||
## Goal
|
||||
|
||||
Bring up KiCad's Gerber Viewer (`gerbview`, `FRAME_GERBER`) in the browser to the
|
||||
"boots, canvas visible, click around" level the other ported apps reached. Scope is
|
||||
launch-only — loading actual Gerber/drill files is out of scope for now.
|
||||
|
||||
## Approach
|
||||
|
||||
gerbview is its own standalone program + kiface (unlike symbol_editor, which lived
|
||||
inside the eeschema kiface), so it follows the **pl_editor/pcbnew pattern** almost
|
||||
verbatim: gate the native dynamic-kiface logic behind `if( EMSCRIPTEN )` and link the
|
||||
kiface objects statically into the `gerbview` executable. `gerbview.cpp` (the
|
||||
`KIFACE_GETTER`) is already part of `gerbview_kiface_objects`, so no source hoisting
|
||||
was needed (unlike eeschema). There was no `#ifdef __EMSCRIPTEN__` frame stub to
|
||||
remove (gerbview was never gated out, unlike the symbol editor).
|
||||
|
||||
## Changes (kicad submodule)
|
||||
|
||||
- **`kicad/gerbview/CMakeLists.txt`** — mirror pl_editor's WASM static-link block:
|
||||
- On EMSCRIPTEN, compile `common/single_top.cpp` with `TOP_FRAME=FRAME_GERBER`
|
||||
(no `BUILD_KIWAY_DLL`); wrap the native minimal exe link in `if( NOT EMSCRIPTEN )`.
|
||||
- Hoist the kiface deps into `GERBVIEW_KIFACE_LIBRARIES`; on EMSCRIPTEN link them
|
||||
directly into the `gerbview` exe with `LINKER:--allow-multiple-definition`.
|
||||
- Gate `gerbview.cpp` defs: EMSCRIPTEN → `COMPILING_DLL` (no `BUILD_KIWAY_DLL`, so
|
||||
`KIFACE_GETTER` links statically); else `BUILD_KIWAY_DLL;COMPILING_DLL`.
|
||||
- **`kicad/gerbview/navlib/CMakeLists.txt`** — add an `if( EMSCRIPTEN )` branch that
|
||||
builds `gerbview_navlib` from the WASM stub instead of the real 3Dconnexion plugin
|
||||
(no SpaceMouse driver in the browser). The frame's navlib member uses
|
||||
`NL_GERBVIEW_PLUGIN` under WASM (`#ifndef __linux__`; emscripten doesn't define it).
|
||||
- **`#include <wx/choice.h>`** added to three files that use `wxChoice` (the
|
||||
Cmp/Net/Attr/DCode aux-toolbar combo boxes) but only had the forward declaration:
|
||||
`gerbview/events_called_functions.cpp`, `gerbview/toolbars_gerber.cpp`,
|
||||
`gerbview/tools/gerbview_control.cpp`. Native builds pull `wx/choice.h` transitively;
|
||||
the WASM wxWidgets header config does not, so these failed with "member access into
|
||||
incomplete type 'wxChoice'". Include-what-you-use fix — behavior-neutral, upstream-safe.
|
||||
(`gerbview_frame.cpp` already gets it transitively; the generated `_base.cpp` carries
|
||||
its own includes — both left untouched to keep the fork minimal.)
|
||||
|
||||
## Changes (root repo)
|
||||
|
||||
- **`wasm/stubs/nl_gerbview_plugin_stub.cpp`** (NEW) — no-op `NL_GERBVIEW_PLUGIN`
|
||||
ctor/dtor + `SetCanvas`/`SetFocus`, mirroring `nl_pl_editor_plugin_stub.cpp`.
|
||||
- **`scripts/kicad/build-gerbview.sh`** (NEW) — thin wrapper → `build-kicad-target.sh gerbview`.
|
||||
- **`scripts/kicad/build-kicad-target.sh`** — add `gerbview` to the `pcbnew|eeschema)`
|
||||
case arm (target = subdir = `gerbview`); update usage strings.
|
||||
- **`docker/build.sh`** — add `gerbview` to valid apps, dispatch case, and the `all`
|
||||
loop (now 6 apps).
|
||||
- **`tests/scripts/setup-kicad-wasm.sh`** — `copy_app gerbview`.
|
||||
- **`tests/apps/kicad/gerbview.html`** (NEW) — browser shell (copy of pl_editor.html;
|
||||
title, `thisProgram=/usr/bin/gerbview`, `gerbview.js`).
|
||||
- **`tests/kicad/gerbview.spec.ts`** (NEW) + **`tests/package.json`** — launch-only
|
||||
smoke test (wizard, canvas visible, registry populated, ≥1 toolbar, no abort).
|
||||
|
||||
## Build & verify
|
||||
|
||||
```
|
||||
./docker/build.sh gerbview # seed fresh-branch cache from main first (see build-quirks memory)
|
||||
cd tests && npm run setup:kicad && npm run test:gerbview
|
||||
```
|
||||
|
||||
Expect: the viewer opens — menu bar, top + aux toolbars (with the Cmp/Net/Attr/DCode
|
||||
combos), left tool toolbar, dark gerber canvas with grid + origin crosshair, and the
|
||||
Layers/Items manager pane. `gerbview.spec.ts` passes (2/2, no abort).
|
||||
|
||||
## Known limitations
|
||||
|
||||
- No Gerber/drill files are loaded; the canvas is empty until a file is opened
|
||||
(file loading untested / out of scope).
|
||||
- Symbol-editor-style drawing tools that require an open document behave per native
|
||||
KiCad (some are inactive with no layers loaded).
|
||||
- No persistent storage (MEMFS only).
|
||||
</content>
|
||||
54
docs/features/pl-editor/0001-pl-editor-port.md
Normal file
54
docs/features/pl-editor/0001-pl-editor-port.md
Normal file
|
|
@ -0,0 +1,54 @@
|
|||
# pl_editor (drawing-sheet editor) WASM port — design notes
|
||||
|
||||
## Goal
|
||||
|
||||
Bring up KiCad's `pagelayout_editor` sub-app (also known as `pl_editor`, the drawing-sheet editor) in the browser, to the same "boots, canvas visible, partially usable in-session" level as `pcbnew`. Persistence across sessions not required.
|
||||
|
||||
## Approach
|
||||
|
||||
Mirrors the in-tree pattern that pcbnew/calculator/eeschema use: gate WASM differences behind `if( EMSCRIPTEN )` blocks inside the upstream `pagelayout_editor/CMakeLists.txt`, keeping a single source of truth for the build alongside KiCad's existing platform conditionals (`if( WIN32 )`, `if( APPLE )`).
|
||||
|
||||
(An earlier iteration tried an out-of-tree CMake wrapper to keep the kicad submodule bit-for-bit upstream. It worked, but diverged from the team norm — every other WASM-ported app modifies kicad. We converged to the team pattern; the only kicad-side cost is a ~80-line patch in this app's CMakeLists.txt, all WASM-conditional.)
|
||||
|
||||
## Changes (see kicad.patch + root.patch + wxwidgets.patch)
|
||||
|
||||
### kicad submodule
|
||||
|
||||
- **`kicad/pagelayout_editor/CMakeLists.txt`** — mirrors pcbnew's WASM static-linking pattern:
|
||||
- Drop `BUILD_KIWAY_DLL` from `single_top.cpp` and `pl_editor.cpp` compile defs on EMSCRIPTEN (browser can't `dlopen` a `.kiface` shared library).
|
||||
- Split `pl_editor_kiface` into an OBJECT library (`pl_editor_kiface_objects`) + an empty MODULE; lets the same compiled objects be linked statically into the exe on WASM and dynamically into the `.kiface` module on native.
|
||||
- On EMSCRIPTEN, link `pl_editor` directly against `PL_EDITOR_KIFACE_LIBRARIES` with `LINKER:--allow-multiple-definition` (handles wxWidgets/nanosvg duplicate symbols, same as pcbnew).
|
||||
- **`kicad/pagelayout_editor/navlib/CMakeLists.txt`** — for EMSCRIPTEN, replace the real 3Dconnexion SpaceMouse plugin sources with `wasm/stubs/nl_pl_editor_plugin_stub.cpp` (no USB hardware in the browser).
|
||||
|
||||
### Root repo
|
||||
|
||||
- **`wasm/stubs/nl_pl_editor_plugin_stub.cpp`** — no-op `NL_PL_EDITOR_PLUGIN` ctor/dtor + `SetCanvas`/`SetFocus`, mirroring `nl_pcbnew_plugin_stub.cpp`.
|
||||
- **`scripts/kicad/build-pl_editor.sh`** — thin wrapper around `build-kicad-target.sh pl_editor`.
|
||||
- **`scripts/kicad/build-kicad-target.sh`** — adds `pl_editor` to the `case` (uses upstream target name `pl_editor`, source subdir `pagelayout_editor`).
|
||||
- **`docker/build.sh`** — adds `pl_editor` to the unified app dispatch (valid apps + `all` loop + `kicad_subdir_for`).
|
||||
- **`tests/apps/kicad/pl_editor.html`** — browser shell; `preRun` creates `/home/kicad` and `FS.chdir` there so file dialogs land somewhere friendly instead of MEMFS root.
|
||||
- **`tests/scripts/setup-kicad-wasm.sh`** — `copy_app pl_editor` added to the existing list.
|
||||
- **`tests/e2e/filedialog-folder-nav.spec.ts`** — regression test for the wxFileDialog folder-navigation fix.
|
||||
|
||||
### wxwidgets submodule (file dialog usability fixes)
|
||||
|
||||
These were discovered while bringing up pl_editor's file dialog but apply to any wxWidgets-WASM app:
|
||||
|
||||
- **`wxwidgets/src/generic/filedlgg.cpp`** — `wxGenericFileDialog::OnOk` now navigates into the selected entry when it's a directory instead of closing the dialog and surfacing the folder path to the caller as if it were a file. Without this, KiCad's "Open Drawing Sheet" produced "Unable to load /dev file" when the user selected `/dev` (a directory in MEMFS).
|
||||
- **`wxwidgets/src/wasm/mouse.cpp`** — stateful double-click detection. `EmscriptenMouseEvent` has no click-count field, so `wxEVT_LEFT_DCLICK` literally never fired in the WASM build — breaking `EVT_LIST_ITEM_ACTIVATED` on every listctrl. Now two MOUSEDOWNs of the same button within 500ms emit DCLICK.
|
||||
|
||||
## Build & verify
|
||||
|
||||
```
|
||||
./docker/build.sh pl_editor
|
||||
./tests/scripts/setup-kicad-wasm.sh
|
||||
# serve tests/apps/kicad and open pl_editor.html
|
||||
```
|
||||
|
||||
Expect: window opens, canvas renders, File > Open / Save As dialogs work (folder navigation via single-click + Enter, single-click + OK, or double-click). Dialog lands at `/home/kicad` by default.
|
||||
|
||||
## Known limitations
|
||||
|
||||
- No persistent storage (MEMFS only); files vanish on tab close.
|
||||
- No keyboard accelerator for "navigate to parent directory" beyond the up-arrow button + ".." entry.
|
||||
- Drawing-sheet-specific tooling beyond basic edit/save is untested (out of MVP scope).
|
||||
54
docs/features/schematic/0001-eeschema-iface-stubs.md
Normal file
54
docs/features/schematic/0001-eeschema-iface-stubs.md
Normal file
|
|
@ -0,0 +1,54 @@
|
|||
# 0001 — eeschema IFACE sub-frame stubs (WASM)
|
||||
|
||||
## Why
|
||||
|
||||
Eeschema's `IFACE::CreateKiWindow` in [kicad/eeschema/eeschema.cpp](../../kicad/eeschema/eeschema.cpp) dispatches `FRAME_T` ids to four frame constructors:
|
||||
|
||||
- `FRAME_SCH` → `SCH_EDIT_FRAME` (the schematic editor — what we want)
|
||||
- `FRAME_SCH_SYMBOL_EDITOR` → `SYMBOL_EDIT_FRAME` (symbol library editor)
|
||||
- `FRAME_SCH_VIEWER` → `SYMBOL_VIEWER_FRAME` (symbol library viewer)
|
||||
- `FRAME_SYMBOL_CHOOSER` → `SYMBOL_CHOOSER_FRAME` (symbol chooser dialog)
|
||||
- `FRAME_SIMULATOR` → `SIMULATOR_FRAME` (already wrapped in try/catch — no patch needed)
|
||||
|
||||
The MVP scope for the WASM port is "empty schematic editor + draw a wire", so the three sub-frames (symbol editor / viewer / chooser) are out of scope. They are non-trivial to support in the browser — they need bundled symbol libraries, FS access for `.kicad_sym` lookup, and full dialog plumbing.
|
||||
|
||||
Compiling them out at the source level (removing `EESCHEMA_LIBEDIT_SRCS` from the build) cascades into many CMakeLists.txt and symbol-chooser sources; cleaner to keep the sources compiled and just refuse to instantiate the frames at the IFACE switch.
|
||||
|
||||
## What changed
|
||||
|
||||
`kicad/eeschema/eeschema.cpp`, the three sub-frame cases inside `IFACE::CreateKiWindow` are now guarded:
|
||||
|
||||
```cpp
|
||||
case FRAME_SCH_SYMBOL_EDITOR:
|
||||
#ifdef __EMSCRIPTEN__
|
||||
return nullptr;
|
||||
#else
|
||||
return new SYMBOL_EDIT_FRAME( aKiway, aParent );
|
||||
#endif
|
||||
|
||||
case FRAME_SCH_VIEWER:
|
||||
#ifdef __EMSCRIPTEN__
|
||||
return nullptr;
|
||||
#else
|
||||
return new SYMBOL_VIEWER_FRAME( aKiway, aParent );
|
||||
#endif
|
||||
|
||||
case FRAME_SYMBOL_CHOOSER:
|
||||
{
|
||||
#ifdef __EMSCRIPTEN__
|
||||
return nullptr;
|
||||
#else
|
||||
// … existing body …
|
||||
#endif
|
||||
}
|
||||
```
|
||||
|
||||
`FRAME_SIMULATOR` is untouched — its existing `try/catch (SIMULATOR_INIT_ERR&)` block catches the ngspice init failure that our header stub eventually triggers, and returns nullptr the same way.
|
||||
|
||||
## How to apply
|
||||
|
||||
Captured under `features/schematic/kicad.patch` via `./scripts/create-feature-patches.sh schematic` once the build is green.
|
||||
|
||||
## Tested by
|
||||
|
||||
`tests/kicad/eeschema.spec.ts` — wizard completion + `Draw Wires` tool test. Neither test path exercises the stubbed-out frames.
|
||||
85
docs/features/symbol-editor/0001-symbol-editor-port.md
Normal file
85
docs/features/symbol-editor/0001-symbol-editor-port.md
Normal file
|
|
@ -0,0 +1,85 @@
|
|||
# Symbol Editor WASM port — design notes
|
||||
|
||||
## Goal
|
||||
|
||||
Bring up KiCad's Symbol Editor (`FRAME_SCH_SYMBOL_EDITOR`, the `.kicad_sym`
|
||||
library editor) in the browser, to the same "boots, canvas visible, click
|
||||
around" level as the other ported apps. Scope is launch-only — library
|
||||
load/save and the symbol viewer/chooser sub-frames are out of scope for now.
|
||||
|
||||
## Key insight
|
||||
|
||||
Unlike pcbnew / pl_editor, the symbol editor is **not a separate program**. It is
|
||||
served by the **eeschema kiface**: its sources already compile into
|
||||
`eeschema_kiface_objects` (`EESCHEMA_LIBEDIT_SRCS` + the `symbol_editor_*` tools in
|
||||
`kicad/eeschema/CMakeLists.txt`). KiCad's universal launcher `common/single_top.cpp`
|
||||
opens whichever frame the compile-time `TOP_FRAME` macro names.
|
||||
|
||||
So the port is just **a second launcher executable (`symbol_editor`) that links the
|
||||
same eeschema kiface but compiles `single_top.cpp` with
|
||||
`TOP_FRAME=FRAME_SCH_SYMBOL_EDITOR`** — no new sources, no new kiface, no extracting
|
||||
symbol-editor code. The eeschema kiface (deps, navlib, stubs, libraries) is reused
|
||||
verbatim.
|
||||
|
||||
## Changes
|
||||
|
||||
### kicad submodule (2 files)
|
||||
|
||||
- **`kicad/eeschema/CMakeLists.txt`** — a WASM-only (`if( EMSCRIPTEN )`) block adds the
|
||||
`symbol_editor` executable. Because `single_top.cpp`'s `COMPILE_DEFINITIONS` are
|
||||
directory-scoped (already pinned to `TOP_FRAME=FRAME_SCH` for the `eeschema` exe),
|
||||
we `configure_file`-copy it to a private TU (`symbol_editor_single_top.cpp`) and set
|
||||
that copy's `TOP_FRAME=FRAME_SCH_SYMBOL_EDITOR;PGM_DATA_FILE_EXT="kicad_sym"`. The exe
|
||||
links `EESCHEMA_KIFACE_LIBRARIES` directly with `LINKER:--allow-multiple-definition`,
|
||||
mirroring the eeschema/pcbnew static-link pattern.
|
||||
- **`kicad/eeschema/eeschema.cpp`** — `IFACE::CreateKiWindow`'s `FRAME_SCH_SYMBOL_EDITOR`
|
||||
case was stubbed to `return nullptr` on `__EMSCRIPTEN__` during the eeschema MVP
|
||||
(see `../schematic/0001-eeschema-iface-stubs.md`). That stub is now removed so
|
||||
the frame is constructed in WASM like the native build. This was THE blocker: with the
|
||||
stub, `Kiway.Player(FRAME_SCH_SYMBOL_EDITOR)` returned null, `single_top`'s `OnInit`
|
||||
bailed, and the app sat idle with a blank canvas (no abort, no error).
|
||||
|
||||
The symbol viewer (`FRAME_SCH_VIEWER`) and chooser (`FRAME_SYMBOL_CHOOSER`) remain
|
||||
stubbed — out of scope, and the chooser needs bundled libraries we don't ship.
|
||||
|
||||
### Root repo
|
||||
|
||||
- **`scripts/kicad/build-symbol_editor.sh`** — thin wrapper around
|
||||
`build-kicad-target.sh symbol_editor`.
|
||||
- **`scripts/kicad/build-kicad-target.sh`** — adds `symbol_editor` to the `case`. CMake
|
||||
target is `symbol_editor` but its build subdir is `eeschema` (it's part of that
|
||||
kiface), so a `KICAD_SUBDIR` variable now distinguishes target name from subdir for
|
||||
the output-path log and embind include.
|
||||
- **`docker/build.sh`** — adds `symbol_editor` to valid apps, the `all` loop, and
|
||||
`kicad_subdir_for` (`symbol_editor → eeschema`); artifacts land at
|
||||
`build-wasm/kicad-symbol_editor/eeschema/symbol_editor.{js,wasm}`.
|
||||
- **`tests/apps/kicad/symbol_editor.html`** — browser shell (copy of eeschema.html with
|
||||
title + `thisProgram=/usr/bin/symbol_editor` + `symbol_editor.js`).
|
||||
- **`tests/scripts/setup-kicad-wasm.sh`** — `copy_app symbol_editor` + subdir map entry.
|
||||
- **`tests/kicad/symbol_editor.spec.ts`** + **`tests/package.json`** — launch-scope smoke
|
||||
test (canvas visible, registry populated, toolbars present, no WASM abort), mirroring
|
||||
eeschema's wizard-aware flow.
|
||||
|
||||
### wxwidgets submodule
|
||||
|
||||
No changes needed — the file-dialog and double-click fixes landed with the pl_editor port.
|
||||
|
||||
## Build & verify
|
||||
|
||||
```
|
||||
./docker/build.sh symbol_editor
|
||||
cd tests && npm run setup:kicad && npm run test:symbol_editor
|
||||
# or serve tests/apps/kicad and open symbol_editor.html
|
||||
```
|
||||
|
||||
Expect: the symbol editor window opens — menu bar, top + left + right toolbars
|
||||
(incl. pin/rect/circle/line drawing tools), the symbol library tree pane with the
|
||||
filter box, the gridded canvas with the symbol-origin crosshair, and a status bar.
|
||||
|
||||
## Known limitations
|
||||
|
||||
- No bundled symbol libraries, so the library tree is empty (`SyncLibraries` reports
|
||||
`libCount=0`). Opening/creating/saving `.kicad_sym` files is untested (out of scope).
|
||||
- Symbol viewer and symbol chooser frames are still stubbed out for WASM.
|
||||
- No persistent storage (MEMFS only).
|
||||
</content>
|
||||
323
docs/features/web-init/0001-web-app-spec.md
Normal file
323
docs/features/web-init/0001-web-app-spec.md
Normal file
|
|
@ -0,0 +1,323 @@
|
|||
# 0001 — KiCad-WASM Web App Spec
|
||||
|
||||
Status: **Refined spec, ready for `implement plan`**
|
||||
Branch: `feature/web-init`
|
||||
Scope of this iteration: **create a project, open a project, upload files, open a file in a WASM tool via URL.**
|
||||
|
||||
This document is the agreed design after a clarification pass. Decisions that were
|
||||
explicitly chosen by the user are marked **[decided]**. Items intentionally pushed to a
|
||||
later iteration are marked **[later]**. Sensible defaults that were *not* explicitly
|
||||
discussed are marked **[default]** and are safe to change during planning.
|
||||
|
||||
---
|
||||
|
||||
## 1. Goal
|
||||
|
||||
A single web application that lets a user create/open KiCad projects, upload files into
|
||||
them, and open a file in the matching WASM tool (pcbnew / eeschema / calculator) by
|
||||
visiting a URL such as:
|
||||
|
||||
```
|
||||
/p/project5/pcbnew/nyak.kicad_pcb
|
||||
```
|
||||
|
||||
The WASM apps already exist as `<tool>.js` + `<tool>.wasm` pairs (see `output/` and
|
||||
`tests/apps/kicad/`). They boot into an Emscripten harness and read files from MEMFS.
|
||||
This web app wraps that: it manages projects + files server-side, and on a tool URL it
|
||||
boots the right WASM app and feeds it the project's files via `FS.writeFile`, then drives
|
||||
File→Open on the target file.
|
||||
|
||||
Non-goals this iteration: editing/saving back, collaboration, auth/login.
|
||||
|
||||
---
|
||||
|
||||
## 2. Key decisions (summary table)
|
||||
|
||||
| Area | Decision |
|
||||
|---|---|
|
||||
| Frontend | React + TypeScript + Vite + shadcn/ui **[decided]** |
|
||||
| Backend | Fastify + ts-rest + Zod **[decided]** |
|
||||
| Shared types | `packages/contract` (ts-rest contract + Zod) imported by FE client & BE router **[decided]** |
|
||||
| Realtime | None now; pick stack that allows Hocuspocus/WSS later, no WS endpoints yet **[decided]** |
|
||||
| URL semantics | `/p/:project/:tool/*filepath` — `tool` selects the WASM app, `*filepath` is auto-opened **[decided]** |
|
||||
| Auth/tenancy | No auth now, but data model namespaced by an owner id for later multi-user **[decided]** |
|
||||
| Metadata store | Postgres now, accessed via Drizzle (drizzle-zod shares schemas) **[decided]** |
|
||||
| File blob storage | Pluggable `FileStorage` interface; local-disk impl now, S3 later **[decided]** |
|
||||
| Open behavior | Sync **whole project tree** into MEMFS, then auto-open target **[decided]**; lazy/partial load **[later]** |
|
||||
| Upload | Individual files (multi), folder (preserve structure), and `.zip` of a project **[decided]** |
|
||||
| WASM artifact delivery | Served from a **configurable base location** (URL/dir): local Fastify static from `output/` in dev, public S3 URL in prod **[decided]** |
|
||||
| Save-back / sync | Read-only open now; write interface defined but unused. Save-back + lazy load land together **[later]** |
|
||||
| Monorepo | pnpm + turbo workspace under `web/` **[decided]** |
|
||||
|
||||
---
|
||||
|
||||
## 3. Repository / monorepo layout **[decided: under `web/`]**
|
||||
|
||||
```
|
||||
web/
|
||||
├── package.json # pnpm workspace root
|
||||
├── pnpm-workspace.yaml
|
||||
├── turbo.json
|
||||
├── .env.example
|
||||
├── docker-compose.yml # local Postgres (and later: minio for S3 parity)
|
||||
├── apps/
|
||||
│ ├── frontend/ # Vite + React + TS + shadcn
|
||||
│ └── server/ # Fastify + ts-rest + Drizzle
|
||||
└── packages/
|
||||
├── contract/ # ts-rest contract + Zod schemas (shared)
|
||||
├── storage/ # FileStorage interface + local-disk impl (+ S3 later)
|
||||
└── config/ # shared tsconfig / eslint / env parsing [default]
|
||||
```
|
||||
|
||||
Rationale: isolates the JS/TS app from the C++/WASM build repo at root (`kicad/`,
|
||||
`wxwidgets/`, `scripts/`, `docker/`). The web app consumes WASM artifacts produced by the
|
||||
existing build, it does not build them.
|
||||
|
||||
Root `.gitignore` should ignore `web/**/node_modules`, `web/**/dist`, build caches.
|
||||
|
||||
---
|
||||
|
||||
## 4. Domain model
|
||||
|
||||
### 4.1 Entities (Postgres, via Drizzle) **[decided: Postgres + Drizzle]**
|
||||
|
||||
```
|
||||
owner -- namespace for "no auth now, multi-user later"
|
||||
id uuid pk
|
||||
slug text unique -- e.g. "default" now; becomes real users later
|
||||
created_at timestamptz
|
||||
|
||||
project
|
||||
id uuid pk
|
||||
owner_id uuid fk -> owner.id
|
||||
slug text -- URL segment, unique within owner (e.g. "project5")
|
||||
name text -- human display name
|
||||
created_at timestamptz
|
||||
updated_at timestamptz
|
||||
unique(owner_id, slug)
|
||||
|
||||
project_file -- index of files; bytes live in FileStorage
|
||||
id uuid pk
|
||||
project_id uuid fk -> project.id
|
||||
path text -- POSIX-relative within project, e.g. "pcbnew/nyak.kicad_pcb"
|
||||
size bigint
|
||||
content_type text
|
||||
storage_key text -- opaque key handed to FileStorage
|
||||
created_at timestamptz
|
||||
updated_at timestamptz
|
||||
unique(project_id, path)
|
||||
```
|
||||
|
||||
- **Owner namespace [decided]**: every project belongs to an `owner`. This iteration uses a
|
||||
single seeded owner (`slug = "default"`); the URL omits owner (`/p/:project/...`) and the
|
||||
server resolves it to the default owner. Adding real auth later = populate `owner` per
|
||||
user and prefix routes, **no schema migration needed**.
|
||||
- `project_file.path` is the canonical project-relative path. The `storage_key` decouples
|
||||
the logical path from however the blob backend names things (so renames/S3 layout are free).
|
||||
|
||||
drizzle-zod derives Zod schemas from these tables; those Zod schemas feed the ts-rest
|
||||
contract so DB ↔ API ↔ client share one source of truth.
|
||||
|
||||
### 4.2 What "project" means at the byte level
|
||||
|
||||
A project is a directory tree of files (`.kicad_pro`, `.kicad_pcb`, `.kicad_sch`,
|
||||
`fp-lib-table`, `sym-lib-table`, footprint/symbol lib dirs, etc.). `project_file` rows
|
||||
enumerate the tree; bytes live behind `FileStorage`.
|
||||
|
||||
---
|
||||
|
||||
## 5. Storage abstraction **[decided: pluggable, local now, S3 later]**
|
||||
|
||||
`packages/storage` exposes a single interface. The whole iteration is **read-heavy**; write
|
||||
methods exist so save-back **[later]** needs no redesign.
|
||||
|
||||
```ts
|
||||
export interface FileStorage {
|
||||
// read path
|
||||
exists(key: string): Promise<boolean>;
|
||||
read(key: string): Promise<Uint8Array>;
|
||||
createReadStream(key: string): NodeJS.ReadableStream; // for large files
|
||||
stat(key: string): Promise<{ size: number; contentType?: string }>;
|
||||
list(prefix: string): Promise<string[]>; // keys under a prefix
|
||||
|
||||
// write path (used now only by upload; save-back is [later])
|
||||
write(key: string, data: Uint8Array | NodeJS.ReadableStream, opts?: { contentType?: string }): Promise<void>;
|
||||
delete(key: string): Promise<void>;
|
||||
}
|
||||
```
|
||||
|
||||
Implementations:
|
||||
- `LocalDiskStorage` **[now]** — rooted at a configurable dir (`STORAGE_ROOT`), `key` maps
|
||||
to a path under it. Streams to/from disk.
|
||||
- `S3Storage` **[later]** — same interface over an S3-compatible bucket. `docker-compose`
|
||||
can run MinIO for local S3 parity when we get there.
|
||||
|
||||
Storage key scheme **[default]**: `owners/<owner_id>/projects/<project_id>/<project_file.path>`.
|
||||
Opaque to callers — only `FileStorage` interprets it.
|
||||
|
||||
---
|
||||
|
||||
## 6. WASM artifact delivery **[decided: configurable base location]**
|
||||
|
||||
The big artifacts (`pcbnew.wasm` ~350 MB, `eeschema.wasm` ~180 MB, `calculator.wasm`,
|
||||
their `.js` glue, `wx.js`, `images.tar.gz`) are **app binaries, not user data** — kept
|
||||
separate from `FileStorage`.
|
||||
|
||||
- The frontend resolves every artifact URL from a single configurable base:
|
||||
`WASM_ASSET_BASE_URL` **[decided requirement]**.
|
||||
- **dev**: points at the Fastify server, which serves the artifacts statically from a
|
||||
configurable dir (default `../../output` relative to the server, i.e. repo `output/`).
|
||||
- **prod**: points at a public S3/CDN URL. No code change — just env.
|
||||
- An artifact URL is composed as `${WASM_ASSET_BASE_URL}/${tool}.js` (and the glue then
|
||||
fetches the sibling `.wasm` / `worker.js` / `images.tar.gz` from the same base). The
|
||||
Emscripten `locateFile` hook must be wired to this base so `.wasm`/`.worker.js` resolve
|
||||
correctly regardless of origin.
|
||||
- **Cross-origin caveat [important]**: KiCad WASM uses threads (`.worker.js` present),
|
||||
which needs `SharedArrayBuffer` → the **document** must be served with
|
||||
`Cross-Origin-Opener-Policy: same-origin` and `Cross-Origin-Embedder-Policy: require-corp`.
|
||||
When artifacts come from a different origin (S3/CDN), they must be served with
|
||||
`Cross-Origin-Resource-Policy: cross-origin` (or CORS) so they load under COEP. The
|
||||
Fastify static route sets COOP/COEP/CORP in dev; the prod bucket/CDN must set CORP/CORS.
|
||||
Verify against the existing harness behavior in `tests/apps/kicad/`.
|
||||
|
||||
Artifacts are **not committed to git** (they're build outputs). The build pipeline
|
||||
(`docker/build.sh` → `output/`) remains the source.
|
||||
|
||||
---
|
||||
|
||||
## 7. URL & routing **[decided]**
|
||||
|
||||
Frontend routes (client-side router) **[default: react-router]**:
|
||||
|
||||
| Route | View |
|
||||
|---|---|
|
||||
| `/` | Project list + "Create project" |
|
||||
| `/p/:project` | Project detail: file tree, upload, "open in tool" actions |
|
||||
| `/p/:project/:tool/*filepath` | **Tool view**: boots `:tool` WASM app, auto-opens `*filepath` |
|
||||
|
||||
- `:tool` ∈ `{ pcbnew, eeschema, calculator }` — selects the WASM app **[decided]**.
|
||||
(`calculator` takes no file; opening it ignores `*filepath`.)
|
||||
- `*filepath` is the project-relative path of the file to auto-open
|
||||
(e.g. `pcbnew/nyak.kicad_pcb`). It must match a `project_file.path` row.
|
||||
- `:project` is the project **slug** within the default owner.
|
||||
- Owner is implicit (default owner) now; route gains an owner segment when auth lands
|
||||
**[later]** — `/p/:project/...` is forward-compatible.
|
||||
|
||||
Tool→file-extension mapping (for validation / "open with" UI) **[default]**:
|
||||
`.kicad_pcb → pcbnew`, `.kicad_sch → eeschema`. Mismatches surface a warning but the
|
||||
explicit `:tool` segment wins (per the decided semantics).
|
||||
|
||||
---
|
||||
|
||||
## 8. API (ts-rest contract in `packages/contract`) **[decided: ts-rest + Zod]**
|
||||
|
||||
All endpoints under `/api`. Contract is the single typed source; Fastify router
|
||||
implements it, frontend uses the generated ts-rest react-query client **[default]**.
|
||||
|
||||
```
|
||||
GET /api/projects -> Project[]
|
||||
POST /api/projects { name, slug? } -> Project # create [scope]
|
||||
GET /api/projects/:project -> Project + file tree # open [scope]
|
||||
DELETE /api/projects/:project -> 204 # [default, nice-to-have]
|
||||
|
||||
GET /api/projects/:project/files -> ProjectFile[]
|
||||
GET /api/projects/:project/files/*path -> file bytes (streamed) # used to fill MEMFS
|
||||
POST /api/projects/:project/files (multipart) -> ProjectFile[] # upload [scope]
|
||||
POST /api/projects/:project/files/zip (multipart zip) -> ProjectFile[] # upload-zip [scope]
|
||||
# write/rename/delete of individual files: interface ready, [later] for save-back
|
||||
```
|
||||
|
||||
Upload handling **[decided: files + folder + zip]**:
|
||||
- **Individual files (multi)**: multipart; each part carries its target project-relative
|
||||
path. Streamed to `FileStorage`, one `project_file` row each.
|
||||
- **Folder (preserve structure)**: frontend uses `webkitdirectory`; relative paths derived
|
||||
from `file.webkitRelativePath` and sent as the per-file path. Server preserves the tree.
|
||||
- **Zip**: server unpacks (streaming unzip) into the project tree, creating `project_file`
|
||||
rows per entry. Reject path traversal (`../`) and absolute paths.
|
||||
|
||||
Validation: Zod schemas (shared) validate bodies/params; Fastify JSON-schema serialization
|
||||
for responses. Errors via a consistent ts-rest error shape **[default]**.
|
||||
|
||||
---
|
||||
|
||||
## 9. Open-a-file flow (the core of the iteration)
|
||||
|
||||
Visiting `/p/:project/:tool/*filepath`:
|
||||
|
||||
1. Frontend fetches the project's file tree: `GET /api/projects/:project/files`.
|
||||
2. Frontend loads the WASM glue for `:tool` from `WASM_ASSET_BASE_URL` and instantiates the
|
||||
Emscripten module into a canvas-bearing harness (reuse the proven shell from
|
||||
`tests/apps/kicad/pcbnew.html` — `createCanvas`, `images.tar.gz` prefetch+write,
|
||||
`locateFile`, status/progress UI), ported into a React component.
|
||||
3. **Sync whole project tree into MEMFS** **[decided]**: for every `project_file`, fetch its
|
||||
bytes (`GET .../files/*path`) and `FS.mkdirTree` + `FS.writeFile` at the project root
|
||||
inside MEMFS (mirroring `tests/kicad/utils/fs-inject.ts`). Files land at the path KiCad
|
||||
expects (e.g. under the default projects dir, confirmed by `load-pcb-probe.spec.ts`:
|
||||
`/home/kicad/documents/kicad/9.99/projects/...`). The exact MEMFS mount point for an
|
||||
arbitrary user project is an **open implementation detail** — see §11.
|
||||
4. Drive File→Open on `*filepath` using the existing element-tracker / menu-driving
|
||||
helpers (`tests/e2e/utils/element-tracker.ts`, `tests/kicad/load-pcb.spec.ts`). This UI
|
||||
automation already works for the demo boards and is the reference implementation.
|
||||
5. Render. Read-only — no write-back **[decided]**.
|
||||
|
||||
> **[later]** Lazy/partial loading: instead of syncing the whole tree up front, intercept
|
||||
> MEMFS reads and fetch siblings on demand. This lands **together with save-back/sync**, as
|
||||
> a single coherent iteration (both need MEMFS↔storage plumbing). Not now.
|
||||
|
||||
---
|
||||
|
||||
## 10. Frontend detail **[shadcn decided; rest default]**
|
||||
|
||||
- Vite + React + TS, shadcn/ui components, Tailwind.
|
||||
- Pages: project list (cards + create dialog), project detail (file tree + upload
|
||||
dropzone + per-file "open in pcbnew/eeschema" buttons), tool view (full-viewport WASM
|
||||
canvas + status overlay).
|
||||
- Data layer: ts-rest react-query client generated from the contract.
|
||||
- Upload UX: drag-drop dropzone supporting multi-file, folder (`webkitdirectory`), and
|
||||
`.zip`; progress per file; streamed to backend.
|
||||
- The WASM tool view is a dedicated component that owns the Emscripten lifecycle and tears
|
||||
it down on unmount (WebGL context, MEMFS) to allow switching tools/projects.
|
||||
|
||||
---
|
||||
|
||||
## 11. Open questions for `implement plan` (not blocking this spec)
|
||||
|
||||
1. **MEMFS mount point for arbitrary projects.** Demos rely on KiCad's default projects
|
||||
path. For a user project we must decide where in MEMFS the tree is written and whether
|
||||
pcbnew/eeschema need it under their expected projects dir, or whether File→Open can
|
||||
target an arbitrary MEMFS path. Resolve by probing (extend `load-pcb-probe`).
|
||||
2. **Driving File→Open generically.** Current helpers are tuned to the demo dialog flow
|
||||
(filelist bbox click + filename input + Enter). Confirm it generalizes to arbitrary
|
||||
paths, or expose a cleaner embind "open file" entry point in the WASM layer.
|
||||
3. **eeschema/calculator open flows.** Mirror the pcbnew flow; verify eeschema's File→Open
|
||||
and that calculator (no file) just boots.
|
||||
4. **Large-tree sync performance.** Whole-tree sync of a big project over many HTTP
|
||||
requests may be slow; consider a single tar/zip stream endpoint to fill MEMFS in one
|
||||
shot (still "sync whole tree", just one request). Decide in planning.
|
||||
5. **COOP/COEP in prod** with cross-origin S3 artifacts — validate header matrix.
|
||||
6. **Project slug generation/collision** rules; reserved tool names as slugs.
|
||||
|
||||
---
|
||||
|
||||
## 12. Out of scope this iteration (explicit)
|
||||
|
||||
- Auth / login / real multi-user (data model is ready; UI/enforcement is **[later]**).
|
||||
- WebSocket / realtime / Hocuspocus collaboration (stack chosen to allow it; none built).
|
||||
- Saving or syncing edits back to storage (**[later]**, paired with lazy load).
|
||||
- S3 storage implementation (interface ready; **[later]**).
|
||||
- Editing project files in the browser outside the WASM tools.
|
||||
|
||||
---
|
||||
|
||||
## 13. Definition of done (this iteration)
|
||||
|
||||
- `pnpm install && docker-compose up -d && pnpm dev` in `web/` brings up Postgres,
|
||||
Fastify (`/api` + WASM static), and the Vite frontend.
|
||||
- Create a project from the UI; it appears in the list and in Postgres.
|
||||
- Open the project; upload files via multi-select, folder, and zip — files appear in the
|
||||
tree and in `FileStorage` + `project_file`.
|
||||
- Navigate to `/p/<slug>/pcbnew/<path>.kicad_pcb`; the board renders read-only, equivalent
|
||||
to the existing `tests/kicad/load-pcb` result, sourcing files from project storage.
|
||||
- Same for an eeschema `.kicad_sch` file.
|
||||
- Swapping `WASM_ASSET_BASE_URL` between local `output/` and a remote URL requires no code
|
||||
change.
|
||||
96
docs/features/web-init/0002-url-regex-modal-followup.md
Normal file
96
docs/features/web-init/0002-url-regex-modal-followup.md
Normal file
|
|
@ -0,0 +1,96 @@
|
|||
# Next-session prompt: fix the URL-detection wxRegEx modal on schematic load (WASM)
|
||||
|
||||
Paste everything below the line into a fresh session to work on this.
|
||||
|
||||
---
|
||||
|
||||
Fix the "Invalid regular expression … UTF-8 error" modal that pops up when eeschema
|
||||
(WASM) renders a schematic containing text/symbol fields. Branch: `feature/web-init`.
|
||||
|
||||
## Symptom
|
||||
|
||||
Loading a schematic that has symbol fields / text (e.g.
|
||||
`kicad/demos/ecc83/ecc83-pp.kicad_sch`) renders the schematic but immediately throws a
|
||||
**modal dialog**:
|
||||
|
||||
```
|
||||
KiCad Schematic Editor Error
|
||||
Invalid regular expression '(https?|ftp|file)://([-\w+&@#/%?=~_|!:,.;]*[^.,;<>\s<><73><EFBFBD><EFBFBD><EFBFBD><EFBFBD>])':
|
||||
UTF-8 error: code points 0xd800-0xdfff are not defined
|
||||
```
|
||||
|
||||
(The `<60><><EFBFBD><EFBFBD><EFBFBD><EFBFBD>` are mojibake where `¶` U+00B6 should be.) It is **non-fatal** — the schematic
|
||||
draws behind the dialog — but it blocks the UI and means URL detection in text is broken.
|
||||
An empty / geometry-only schematic does NOT trigger it (that's why the load regression test
|
||||
`tests/kicad/eeschema-load.spec.ts` uses wires/junctions only).
|
||||
|
||||
## Root cause (already localized)
|
||||
|
||||
`kicad/common/string_utils.cpp` has two static `wxRegEx` whose pattern ends in a negated
|
||||
character class containing `¶` (¶, U+00B6):
|
||||
|
||||
- `LinkifyHTML()` ~line 672: `wxS( "\\b(https?|ftp|file)://([-\\w+&@#/%?=~|!:,.;]*[^.,:;<>\\(\\)\\s¶])" )`
|
||||
- `IsURL()` ~line 683: `wxS( "(https?|ftp|file)://([-\\w+&@#/%?=~|!:,.;]*[^.,:;<>\\s¶])" )`
|
||||
|
||||
`IsURL()` is called during field/text rendering — `kicad/eeschema/sch_field.cpp:1042` &
|
||||
`:1081`, `kicad/eeschema/sch_textbox.cpp:352`, `kicad/eeschema/fields_data_model.cpp:377` —
|
||||
so any schematic with symbol reference/value fields hits it. The error is a **regex
|
||||
COMPILE** failure of the static pattern (input-independent): the first `IsURL()` call
|
||||
constructs the static `wxRegEx`, which fails to compile.
|
||||
|
||||
Hypothesis: under the emscripten/wxWidgets-WASM build the `¶` in the `wxS(...)` pattern
|
||||
is mis-encoded (the dialog shows it as invalid UTF-8 in the 0xd800–0xdfff surrogate range),
|
||||
so `wxRegEx` rejects the pattern. Native KiCad compiles it fine, so this is WASM-specific —
|
||||
likely the wide-char literal handling and/or the `wxString → wxRegEx` UTF-8 conversion in
|
||||
the wx-WASM port. NOT yet root-caused to the exact byte; that's step 1.
|
||||
|
||||
## What to do
|
||||
|
||||
1. Reproduce + pin the exact corruption. Add temporary logging that prints the pattern
|
||||
bytes (hex) right before the `wxRegEx` ctor in `IsURL()`/`LinkifyHTML()`, and compare
|
||||
what `¶` becomes in the WASM build vs. what it should be (`0xC2 0xB6`). Determine
|
||||
whether the corruption is at: the C++ wide/narrow literal, the `wxS`/`wxString` storage,
|
||||
or the `wxRegEx` UTF-8 conversion (`src/common/regex.cpp` / the wx-WASM regex backend).
|
||||
2. Fix at the lowest correct layer. Prefer the wx-WASM layer (project rule: keep `kicad/`
|
||||
close to upstream, fix under `__EMSCRIPTEN__` in `wxwidgets/`). Candidate fixes, cheapest
|
||||
first — validate which is actually right after step 1:
|
||||
- If the `wxRegEx` UTF-8 conversion is the bug, fix it in the wx-WASM regex backend so
|
||||
non-ASCII pattern code points (e.g. `¶`) round-trip.
|
||||
- If it's the literal/encoding, build the pattern via `wxString::FromUTF8("…\xC2\xB6…")`
|
||||
instead of `¶`, or otherwise ensure correct encoding. (If this has to live in
|
||||
`string_utils.cpp`, guard it `#ifdef __EMSCRIPTEN__` and keep the upstream pattern for
|
||||
native — minimal divergence; run `scripts/kicad-diff-stats.sh` after.)
|
||||
- Last resort: drop `¶` from the WASM pattern (it only excludes the pilcrow from a
|
||||
URL's trailing char — cosmetic). Note this in a comment if chosen.
|
||||
3. Verify: load a text-bearing schematic and confirm NO modal, schematic renders, and a real
|
||||
URL in a text field is still linkified (don't regress URL detection). Then re-run
|
||||
`tests/kicad/eeschema-load.spec.ts` (must stay green) and ideally add a text-bearing
|
||||
schematic case that would have shown the modal.
|
||||
|
||||
## Context you need
|
||||
|
||||
- The schematic LOAD path now works: the fiber/Asyncify hang was fixed by the trampoline
|
||||
self-heal shim in `scripts/common/inject-dyncall-shims.sh` (section "3c"). Don't touch it.
|
||||
- Build eeschema (reuses prebuilt deps volume, ~5 min, don't build deps from scratch, don't
|
||||
run two builders at once — 32G each on a 37G Docker VM OOMs):
|
||||
`COMPOSE_PROJECT_NAME=kicad-wasm-feature-schematic ./docker/build.sh eeschema --debug`
|
||||
Then `cd web/apps/frontend && npm run link-wasm`. Build scripts log to files
|
||||
(`logs/build/*.log`) — don't pipe them.
|
||||
- wxWidgets-only changes: `scripts/build-wxuniversal-wasm.sh` (on-machine, faster), then
|
||||
relink eeschema.
|
||||
- The webapp's iframe is being removed this round (separate task) to simplify dev — so the
|
||||
app may load eeschema directly rather than in a same-origin iframe. The e2e harness
|
||||
(`tests/apps/kicad/eeschema.html`, loaded directly by Playwright) already runs iframe-free;
|
||||
it's the most reliable repro. Run kicad e2e from `tests/`: `npm run test:eeschema`.
|
||||
- Symbolizing WASM stack frames (browser shows `wasm-function[N]` with no names): the separate
|
||||
`*.debug.wasm` is PRE-asyncify and useless. See memory `eeschema_wasm_symbolization` for the
|
||||
`--profiling-funcs` + asyncify `-g` recipe and the name-section index→name parser. Those are
|
||||
temporary debug build flags — re-add while debugging, remove before committing.
|
||||
- Use debug tools / symbols, don't guess (CLAUDE.md). Temporary `fprintf(stderr,"[TAG] …")`
|
||||
shows as `[KICAD_ERR]` in the browser console; remove before committing.
|
||||
|
||||
## Memory pointers (read these first)
|
||||
|
||||
- `eeschema_schematic_load_crash` — full root-cause history of the load hang + this regex
|
||||
follow-up (the "NEW minor follow-up" note).
|
||||
- `eeschema_wasm_symbolization` — how to get real C++ names into browser WASM stack traces.
|
||||
446
docs/research/threading_1.md
Normal file
446
docs/research/threading_1.md
Normal file
|
|
@ -0,0 +1,446 @@
|
|||
# Research: KiCad WASM Coroutine Architecture & Nested Asyncify Bug
|
||||
|
||||
## TL;DR
|
||||
|
||||
**The bug**: `RuntimeError: index out of bounds` when a startup wizard modal closes. Happens because **`Asyncify.currData` is a single-slot global**. Our fiber-swap shim overwrites it when a tool activates inside the modal's event loop. The modal's subsequent rewind then follows a stale/mismatched call chain and crashes.
|
||||
|
||||
**The fix**: ~10 lines in `scripts/common/inject-dyncall-shims.sh` to wrap `Asyncify.handleSleep` so it saves its own `asyncifyData` before unwind and restores it before `doRewind`.
|
||||
|
||||
**Upstream status**: Documented as Emscripten [Issue #9153](https://github.com/emscripten-core/emscripten/issues/9153), marked **wontfix**. We must work around it.
|
||||
|
||||
---
|
||||
|
||||
## Conceptual Foundations
|
||||
|
||||
### Coroutine vs subroutine
|
||||
|
||||
A **subroutine** has a single entry and a single exit — runs to completion. A **coroutine** is a subroutine that can be **paused** at arbitrary points and **resumed** later.
|
||||
|
||||
Two flavors:
|
||||
|
||||
- **Stackless**: the compiler transforms the function into a state machine. Saves only locals at designated suspension points (`co_await`, `await`). Can only pause at those points, not inside arbitrary callees. C++20 coroutines, JavaScript `async`/`await`.
|
||||
- **Stackful**: the coroutine owns a separate call stack. Can pause from ANY depth — even from inside library functions. Python greenlets, Boost.Context, Lua coroutines, KiCad's `COROUTINE`.
|
||||
|
||||
KiCad needs stackful because `WaitForClick()` is called many frames deep inside tool logic; stackless would require rewriting every tool.
|
||||
|
||||
### Fiber
|
||||
|
||||
A runtime primitive for stackful coroutines: owns its own stack, cooperatively scheduled (unlike threads which are preemptively scheduled by the OS). Native fibers swap CPU registers and the stack pointer — ~20 assembly instructions per platform.
|
||||
|
||||
WASM has no register access and no direct call-stack manipulation. Emscripten provides `emscripten_fiber_t` emulated on top of Asyncify.
|
||||
|
||||
### Asyncify
|
||||
|
||||
A **binary transformation** pass (Binaryen's `wasm-opt --asyncify`). It rewrites every WASM function in the module to add:
|
||||
|
||||
- A prelude: `if (state == REWINDING) { pop_locals(); jump to saved call site }`
|
||||
- Wrapped call sites: `normal_call(); if (state == UNWINDING) { push_locals(); save_call_index(); return; }`
|
||||
|
||||
Three globals drive everything:
|
||||
- `__asyncify_state`: 0=Normal, 1=Unwinding, 2=Rewinding
|
||||
- `__asyncify_data`: pointer to current buffer
|
||||
- JS-side `Asyncify.currData`: **a single-slot pointer to the currently-active async operation's buffer**
|
||||
|
||||
**The structural fault**: `Asyncify.currData` is a global. Emscripten assumes ONE async operation active at a time. When two overlap (EM_ASYNC_JS modal + fiber swap during its event loop), they fight over this slot.
|
||||
|
||||
### Layering in KiCad WASM
|
||||
|
||||
```
|
||||
KiCad tool code (C++)
|
||||
└─ uses COROUTINE<int, TOOL_EVENT&> [kicad/include/tool/coroutine.h]
|
||||
└─ uses libcontext::jump_fcontext [kicad/thirdparty/libcontext/libcontext.cpp]
|
||||
└─ WASM backend: emscripten_fiber_swap
|
||||
└─ emscripten/fiber.h [tools/emsdk/.../fiber.h]
|
||||
└─ uses Asyncify [wasm-opt transform]
|
||||
|
||||
wxWidgets uses EM_ASYNC_JS (parallel Asyncify channel):
|
||||
wxDialog::ShowModal
|
||||
└─ startModal() EM_ASYNC_JS [wxwidgets/src/wasm/dialog.cpp]
|
||||
└─ Asyncify.handleSleep
|
||||
└─ uses the SAME Asyncify.currData
|
||||
```
|
||||
|
||||
Two independent channels share one global. Collision guaranteed.
|
||||
|
||||
---
|
||||
|
||||
## All 8 Suspension Patterns
|
||||
|
||||
### Pattern 1 — Tool First Activation
|
||||
Trigger: `TOOL_MANAGER::dispatchInternal()` finds a matching `Go()` transition; calls `cofunc->Call(event)`.
|
||||
Mechanism: `make_fcontext(callerStub)` → `jump_fcontext` → `emscripten_fiber_swap` → (first time) `dynCall_vi(entryPoint, userData)` → `wasm_fcontext_entry(ctx)` → `callerStub` → tool method.
|
||||
Status per runtime logs: **working** (multiple successful first-entries).
|
||||
|
||||
### Pattern 2 — RunMainStack (dialog from tool)
|
||||
Tool coroutine calls `RunMainStack([&]() { dlg.ShowModal(); })` → `CALL_CONTEXT::RunMainStack` → `jump_fcontext` back to main with `CONTINUE_AFTER_ROOT` → main runs the lambda → lambda's `ShowModal` uses EM_ASYNC_JS → returns → main calls `doResume()` with `FROM_ROOT` → coroutine resumes.
|
||||
The canonical "nested Asyncify" pattern at the tool level. Not yet reached in current test runs (blocked by Pattern 4 failure first).
|
||||
|
||||
### Pattern 3 — Tool Wait/Resume cycle
|
||||
Inside a tool method:
|
||||
```cpp
|
||||
while (TOOL_EVENT* evt = Wait()) { process(evt); }
|
||||
```
|
||||
`Wait()` → `TOOL_MANAGER::ScheduleWait()` → sets `pendingWait`, calls `cofunc->KiYield()` → `jumpOut()` → `jump_fcontext` → fiber swap to main. Later, matching event → `cofunc->Resume()` → swap back. Each Wait/Resume is 2 fiber swaps (4 asyncify operations).
|
||||
Status per logs: **working**.
|
||||
|
||||
### Pattern 4 — Standalone Modal (EM_ASYNC_JS)
|
||||
`wxDialog::ShowModal()` on main stack → `startModal()` EM_ASYNC_JS → Asyncify unwinds main into global `currData` buffer → setTimeout event loop polls `ProcessEvents` every 17ms → `EndModal(code)` resolves Promise → Asyncify rewinds main → result returned.
|
||||
Status: **works alone; fails when fibers run during its event loop**. This is where the current bug manifests.
|
||||
|
||||
### Pattern 5 — Clipboard (EM_ASYNC_JS)
|
||||
`js_writeTextToClipboard`, `js_readTextFromClipboard`, etc. Async browser APIs wrapped in EM_ASYNC_JS. Same single-slot collision hazard as Pattern 4 if called while a fiber is mid-suspension.
|
||||
|
||||
### Pattern 6 — Font enumeration (EM_ASYNC_JS)
|
||||
`js_enumerateFonts()` using Local Font Access API. Fires once at app init, typically before fibers exist. Probably safe.
|
||||
|
||||
### Pattern 7 — Nested/stacked tools
|
||||
Two mechanisms:
|
||||
- **Push/Pop**: `TOOL_MANAGER` pushes old coroutine onto stack when a new tool activates; pops back when new tool finishes.
|
||||
- **FROM_ROUTINE calls**: `child.Call(parentCoroutine, value)` — no CALL_CONTEXT, no root bounce; parent resumes child, child yields back to parent directly.
|
||||
|
||||
Status per logs: **working**.
|
||||
|
||||
### Pattern 8 — Selection tool at startup
|
||||
Not the blocker I originally claimed. PCB_SELECTION_TOOL is the first coroutine, but it Call/Yield/Resume cycles correctly per logs. The startup DOES progress through this pattern without stalling.
|
||||
|
||||
---
|
||||
|
||||
## The Actual Bug — Full Trace Against Source
|
||||
|
||||
### Step-by-step (verified against `tools/emsdk/upstream/emscripten/src/lib/libasync.js`):
|
||||
|
||||
```
|
||||
1. JS calls wasmExports["_ZN8wxDialog9ShowModalEv"]()
|
||||
│ exportCallStack = ["_ZN8wxDialog9ShowModalEv"]
|
||||
│ Asyncify.state = Normal, currData = null
|
||||
↓
|
||||
2. WASM: startModal() is EM_ASYNC_JS
|
||||
│ compiles to: Asyncify.handleAsync(startAsync)
|
||||
│ which calls: Asyncify.handleSleep((wakeUp) => startAsync().then(wakeUp))
|
||||
│
|
||||
│ handleSleep:
|
||||
│ • allocateData() → malloc's BLOCK_A (~12 byte header + stack space)
|
||||
│ • setDataRewindFunc(BLOCK_A):
|
||||
│ bottomOfCallStack = exportCallStack[0] = "_ZN8wxDialog9ShowModalEv"
|
||||
│ rewindId = Asyncify.getCallStackId(bottomOfCallStack)
|
||||
│ HEAP32[(BLOCK_A + 8) >> 2] = rewindId ← modal's re-entry pinned
|
||||
│ • Asyncify.currData = BLOCK_A ← the MODAL's buffer
|
||||
│ • _asyncify_start_unwind(BLOCK_A)
|
||||
│ • WASM unwinds fully. exportCallStack → []
|
||||
│ • Asyncify.state = Normal (unwind complete, awaiting Promise)
|
||||
↓
|
||||
3. JS event loop. setTimeout(runEventLoop, 17ms) fires.
|
||||
│ ccall('ProcessEvents') pushes "ProcessEvents" to exportCallStack.
|
||||
│ WASM: ProcessEvents dispatches queued events.
|
||||
│ One of them: tool activation → cofunc->Call(event).
|
||||
│ That calls jump_fcontext → our _emscripten_fiber_swap override fires.
|
||||
↓
|
||||
4. ★★★ THE FAULT ★★★ — inject-dyncall-shims.sh line ~215:
|
||||
│
|
||||
│ if (Asyncify.state === Asyncify.State.Normal) {
|
||||
│ Asyncify.state = Asyncify.State.Unwinding;
|
||||
│ var asyncifyData = oldFiber + 20; ← fiber's embedded asyncify_data
|
||||
│ // ... sets up __fiber_rewind_<oldFiber> stable rewind target ...
|
||||
│ Asyncify.setDataRewindFunc(asyncifyData, "__fiber_rewind_<oldFiber>");
|
||||
│ Asyncify.currData = asyncifyData; ◄◄◄ OVERWRITES BLOCK_A
|
||||
│ _asyncify_start_unwind(asyncifyData);
|
||||
│ ...
|
||||
│ }
|
||||
│
|
||||
│ At this moment: BLOCK_A's pointer is LOST from Asyncify's view.
|
||||
│ BLOCK_A is still malloc'd; the fiber just changed the "current" slot.
|
||||
↓
|
||||
5. Fiber runs tool body, eventually swaps back. Each fiber swap again writes
|
||||
Asyncify.currData = some_fiber_buffer. Multiple fiber swaps may occur
|
||||
during the modal's event loop.
|
||||
│
|
||||
│ Asyncify.currData is now ANY of these fiber buffers, NEVER restored to BLOCK_A.
|
||||
↓
|
||||
6. User action in modal resolves it. wxDialog::EndModal(5100) is called,
|
||||
which invokes Module._endModal(5100).
|
||||
│ The Promise stored by startModal's setTimeout resolves with 5100.
|
||||
│
|
||||
│ .then(wakeUp) runs from pure JS:
|
||||
│ handleSleep's wakeUp(5100):
|
||||
│ runtimeKeepalivePop();
|
||||
│ handleSleepReturnValue = 5100;
|
||||
│ Asyncify.state = Rewinding;
|
||||
│ _asyncify_start_rewind(Asyncify.currData); ← NOT BLOCK_A!
|
||||
│ Asyncify.doRewind(Asyncify.currData); ← rewinds wrong buffer
|
||||
↓
|
||||
7. ★★★ THE CRASH ★★★
|
||||
│ Asyncify.currData is some fiber's buffer.
|
||||
│ rewind_id at (that fiber + 20 + 8) → name "__fiber_rewind_<fiber>"
|
||||
│ doRewind calls wasmExports["__fiber_rewind_<fiber>"]()
|
||||
│ That wrapper calls wasmExports[entryKey] = __fiber_entry_<fiber>
|
||||
│ __fiber_entry_<fiber> calls dynCall_vi(entryPoint, userData)
|
||||
│ entryPoint was set to 0 when fiber first entered (Emscripten clears it)
|
||||
│ dynCall_vi(0, ...) → getWasmTableEntry(0) → wasmTable.get(0)
|
||||
│ Binaryen's rewind then tries to replay a saved call-index chain
|
||||
│ serialized during the fiber's last unwind — but we're now inside the
|
||||
│ modal's expected context. Call indices point to wrong table entries.
|
||||
│ → RuntimeError: index out of bounds
|
||||
```
|
||||
|
||||
### Matching evidence in log files
|
||||
|
||||
From `tests/logs/kicad/pcbnew/pcbnew-spec-ts-pcbnew-wasm-select-draw-lines-and-draw-on-the-board.log`:
|
||||
|
||||
```
|
||||
[DIAG_MODAL] Modal started (Module._endModal appeared) asyncifyState=0
|
||||
... (many successful fiber operations inside the modal event loop) ...
|
||||
[DIAG_REWIND_FUNC] ... modalActive=true callStack=["ProcessEvents",...]
|
||||
...
|
||||
EndModal: 5100
|
||||
[DIAG_MODAL] EndModal called with code=5100 asyncifyState=0
|
||||
[DIAG_STARTMODAL] endModal called, code=5100
|
||||
[DIAG_STARTMODAL] promise resolved, result=5100
|
||||
🔥 RuntimeError: index out of bounds at pcbnew.wasm:144634078
|
||||
at dynCall_vi (pcbnew.js:6525) ← our shim
|
||||
at dynCall_vi (pcbnew.js:27816) ← our fiber entry wrapper
|
||||
at wrapper (pcbnew.js:17788) ← Emscripten callUserCallback
|
||||
at safeSetTimeout ← modal's event loop
|
||||
```
|
||||
|
||||
The double `dynCall_vi` at lines 6525 and 27816 is explained:
|
||||
- Line 27816 = injected shim (bottom of `pcbnew.js`)
|
||||
- Line 6525 = `Fibers.entryWrapperByFiber[fiber]` = `function() { return dynCall_vi(entryPoint, userData); }` (a fiber-specific wrapper)
|
||||
|
||||
Rewind enters the fiber wrapper → calls dynCall_vi with `entryPoint=0` → crash.
|
||||
|
||||
---
|
||||
|
||||
## Why The First Pass Missed This
|
||||
|
||||
1. **Didn't read runtime logs.** The investigation docs described "startup stalls / toolbars empty" — outdated. Current logs show the app progresses through the wizard and crashes on close. Ground truth was one file away.
|
||||
|
||||
2. **Treated QEMU's pattern as universal.** QEMU's while(true) trampoline fixes entry-function-returns — but `wasm_fcontext_entry`'s return path is never reached in KiCad because `callerStub` always swaps via `jumpOut`. The fix addresses a problem that doesn't occur.
|
||||
|
||||
3. **Didn't audit `inject-dyncall-shims.sh`.** The 350-line script contains the actual bug site (~80 lines of fiber stabilization). Without reading it carefully, couldn't see the `Asyncify.currData` overwrite without save/restore.
|
||||
|
||||
4. **Didn't search Emscripten issues.** Issue #9153 is a wontfix matching our failure exactly. A two-minute search would have located it.
|
||||
|
||||
5. **Confused historical symptom with current symptom.** Investigation docs were written before dynCall shim fixes existed. Those fixes changed the symptom from "startup stall" to "crash on modal close". Docs weren't updated.
|
||||
|
||||
---
|
||||
|
||||
## The Shim's Fiber Stabilization Layer (auditor's notes)
|
||||
|
||||
### Part 1: Per-signature dynCall shims
|
||||
For each `dynCall_*` signature, generate a JS wrapper that:
|
||||
1. Looks up the function in the WASM table
|
||||
2. Pushes a per-call key (`__dyn_SIG_<funcPtr>`) onto `Asyncify.exportCallStack`
|
||||
3. Calls the function
|
||||
4. On return, pops the key and calls `maybeStopUnwind`
|
||||
|
||||
### Part 2: Empty-callback patches (6 patterns)
|
||||
Emscripten 4.x generates `(a1 => {})` no-op stubs when DYNCALLS=0. Six of these are actually called:
|
||||
- HTML5 event callbacks
|
||||
- pthread entry
|
||||
- Signal handlers
|
||||
- Timer callbacks
|
||||
- Main loop iterator
|
||||
- **Fiber entry callback** ← critical
|
||||
|
||||
Each is replaced with the appropriate `dynCall_*` invocation.
|
||||
|
||||
### Part 3: Fiber rewind stabilization (THE BUG SITE)
|
||||
Overrides three functions:
|
||||
- `Asyncify.setDataRewindFunc(ptr, forcedBottomOfCallStack)` — writes a specific fiber-owned rewind ID when forced
|
||||
- `Fibers.finishContextSwitch(newFiber)` — on first entry, registers `__fiber_entry_<fiber>` as a synthetic wasmExport
|
||||
- `_emscripten_fiber_swap(oldFiber, newFiber)` — **this is where `Asyncify.currData` gets clobbered**
|
||||
|
||||
Data structures created but **never cleaned up**:
|
||||
- `Fibers.rewindTargetByFiber[fiberPtr]`
|
||||
- `Fibers.rewindWrapperByFiber[fiberPtr]`
|
||||
- `Fibers.entryWrapperByFiber[fiberPtr]`
|
||||
- `Fibers.entryKeyByFiber[fiberPtr]`
|
||||
- `wasmExports["__fiber_entry_<fiberPtr>"]`, `wasmExports["__fiber_rewind_<fiberPtr>"]`
|
||||
|
||||
Plus `Asyncify.callStackNameToId` / `callStackIdToName` grow by one entry per unique fiber pointer.
|
||||
|
||||
### Secondary hazards (not causing the current crash but loaded footguns)
|
||||
- **Pointer reuse collisions**: When a C++ fiber is freed and memory reused for a new fiber, the new fiber's synthetic exports overwrite the old ones at the same key. Stale references become dangling.
|
||||
- **exportCallStack imbalance on ABORT**: `finally` blocks skip `pop()` when `ABORT` is set. Recoverable errors leave the stack corrupted.
|
||||
- **ID map unbounded growth**: one ID per unique string forever. Long session = thousands of entries.
|
||||
|
||||
---
|
||||
|
||||
## The Fix
|
||||
|
||||
### Root-cause fix: wrap `Asyncify.handleSleep`
|
||||
|
||||
Add to `scripts/common/inject-dyncall-shims.sh`, within the fiber stabilization block:
|
||||
|
||||
```javascript
|
||||
// Save/restore Asyncify.currData around handleSleep to survive
|
||||
// fiber swaps that happen during the sleep's Promise await.
|
||||
// Fixes Emscripten Issue #9153 (wontfix).
|
||||
if (typeof Asyncify !== "undefined" && Asyncify.handleSleep) {
|
||||
var __originalHandleSleep = Asyncify.handleSleep.bind(Asyncify);
|
||||
Asyncify.handleSleep = function(startAsync) {
|
||||
return __originalHandleSleep(function(wakeUp) {
|
||||
// This function runs inside handleSleep AFTER allocateData and
|
||||
// setDataRewindFunc have set Asyncify.currData to THIS sleep's buffer.
|
||||
var myAsyncifyData = Asyncify.currData;
|
||||
return startAsync(function(result) {
|
||||
// wakeUp runs from pure JS with an empty exportCallStack.
|
||||
// Asyncify.currData may have been clobbered by fiber swaps that
|
||||
// ran during our Promise await. Restore our own data before
|
||||
// handleSleep proceeds to doRewind.
|
||||
Asyncify.currData = myAsyncifyData;
|
||||
return wakeUp(result);
|
||||
});
|
||||
});
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
**Why this works**: `handleSleep` pairs `allocateData` (sets currData) with `doRewind` (reads currData). Between those, the Promise awaits. If anything overwrites currData during the await, the rewind uses the wrong buffer. By capturing our own data pointer after `allocateData` and restoring it before `wakeUp` triggers `doRewind`, we guarantee the rewind uses the correct buffer regardless of what fiber swaps did.
|
||||
|
||||
**Why it's safe**: The capture happens synchronously after `handleSleep` sets currData for this sleep; the restore happens in the Promise resolution path before handleSleep's rewind logic runs. Never runs concurrently with the sleep's own rewind.
|
||||
|
||||
### Alternative considered: save/restore in fiber swap
|
||||
|
||||
Could alternatively push/pop `Asyncify.currData` inside `_emscripten_fiber_swap`, but this requires knowing when the outer context is "done" using its currData — handleSleep already knows this (at wakeUp time), so wrapping handleSleep is simpler and more robust.
|
||||
|
||||
### Workaround (if fix is delayed): block tool activation during modal
|
||||
|
||||
In `TOOL_MANAGER::dispatchInternal`, check `wxTheApp->GetTopWindow()->IsModal()` or similar. Queue events; drain on modal close. Doesn't fix the architectural issue but unblocks startup.
|
||||
|
||||
### Hygiene cleanups (separate follow-up PR)
|
||||
|
||||
Independent of the bug fix, these improve `kicad/thirdparty/libcontext/libcontext.cpp`:
|
||||
|
||||
1. Replace `wasm_fcontext_entry` with QEMU-style `while(true)` trampoline — never returns, safer if the entry function's return path were ever accidentally reached
|
||||
2. Remove `emscripten_unwind_to_js_event_loop()` from `wasm_fcontext_entry` and `jump_fcontext` — these would terminate all WASM execution but are never called in practice
|
||||
3. Remove `parking_fiber` / `active_fiber()` / `ensure_parking_context()` — dead code paths related to the entry-returns problem
|
||||
|
||||
Additional shim cleanups:
|
||||
4. Add `Fibers.destroyFiber(fiberPtr)` called from `release_fcontext` via EM_ASM; clears `rewindTargetByFiber`, `rewindWrapperByFiber`, `entryWrapperByFiber`, `entryKeyByFiber`, and `delete wasmExports["__fiber_*_<fiberPtr>"]`
|
||||
5. Balance `exportCallStack` even on ABORT (pop if top matches expected key)
|
||||
|
||||
These don't fix the bug but eliminate several loaded footguns.
|
||||
|
||||
---
|
||||
|
||||
## Test Plan
|
||||
|
||||
### New standalone: `tests/apps/standalone/coroutine-nested/nested_test.cpp`
|
||||
|
||||
Same pattern as `coroutine_test.cpp`. Reuses `kicad_coroutine_harness.h`.
|
||||
|
||||
**Shared helper `AutoClosingDialog`**:
|
||||
- On `wxEVT_SHOW`, starts `wxTimer::StartOnce(delayMs)` → `EndModal(wxID_OK)` on fire
|
||||
- Optional external-close hook for scenarios that need to interleave fiber ops before closing
|
||||
|
||||
**Asyncify state logging macro**:
|
||||
```cpp
|
||||
#define LOG_ASYNCIFY(tag) EM_ASM({
|
||||
console.log('[COROUTINE_TEST] ASYNCIFY ' + UTF8ToString($0) +
|
||||
' state=' + Asyncify.state +
|
||||
' stackLen=' + Asyncify.exportCallStack.length +
|
||||
' currData=' + (Asyncify.currData || 'null') +
|
||||
' tableLen=' + wasmTable.length);
|
||||
}, tag)
|
||||
```
|
||||
|
||||
The `currData` value in these traces is the smoking gun.
|
||||
|
||||
### The 8 scenarios
|
||||
|
||||
| # | Name | Proves |
|
||||
|---|---|---|
|
||||
| 1 | `baseline_modal_alone` | Build and EM_ASYNC_JS work |
|
||||
| 2 | `baseline_fiber_alone` | Fiber swap works |
|
||||
| 3 | `fiber_create_run_destroy_inside_modal` | **TARGET REPRODUCER** |
|
||||
| 4 | `fiber_multi_swap_inside_modal` | Multiple swaps under modal |
|
||||
| 5 | `fiber_yield_across_modal_close` | Dormant fiber across modal boundary |
|
||||
| 6 | `fiber_deep_yield_loop_inside_modal` | Deep stack + many yields under modal |
|
||||
| 7 | `modal_fiber_modal_sequence` | Modal A → fiber → Modal B |
|
||||
| 8 | `nested_fibers_inside_modal` | Fiber-to-fiber (FROM_ROUTINE) under modal |
|
||||
|
||||
### Diagnostic matrix
|
||||
|
||||
| Hypothesis | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| Build/infra broken | FAIL | — | — | — | — | — | — | — |
|
||||
| Fiber port broken | — | FAIL | — | — | — | — | — | — |
|
||||
| Any fiber-under-modal | pass | pass | FAIL | FAIL | FAIL | FAIL | FAIL | FAIL |
|
||||
| Multi-swap only | pass | pass | pass | FAIL | pass | FAIL | maybe | FAIL |
|
||||
| Dormant fiber pin | pass | pass | pass | pass | FAIL | pass | pass | pass |
|
||||
| Consecutive-modal leak | pass | pass | pass | pass | pass | pass | FAIL | pass |
|
||||
| Nested fiber-to-fiber | pass | pass | pass | pass | pass | pass | pass | FAIL |
|
||||
|
||||
**Current build expectation**: Baselines pass; Scenario 3 crashes with `RuntimeError: index out of bounds`.
|
||||
**After fix expectation**: All 8 pass.
|
||||
|
||||
### Build integration
|
||||
|
||||
Add to `tests/apps/Makefile.wasm`:
|
||||
```makefile
|
||||
$(S)/coroutine-nested/nested_test.o: $(S)/coroutine-nested/nested_test.cpp \
|
||||
$(S)/coroutine/kicad_coroutine_harness.h
|
||||
$(CXX) -c $(CXXFLAGS) -I$(KICAD_ROOT)/thirdparty/libcontext \
|
||||
-I$(S)/coroutine $< -o $@
|
||||
|
||||
$(S)/coroutine-nested/nested_test.html: \
|
||||
$(S)/coroutine-nested/nested_test.o \
|
||||
$(S)/coroutine/libcontext.o $(WX_CORE_LIB)
|
||||
$(CXX) $^ $(LDFLAGS_COROUTINE) --pre-js $(JS) --shell-file $(HTML) -o $@
|
||||
../../scripts/common/inject-dyncall-shims.sh $(basename $@).js
|
||||
|
||||
coroutine-nested: $(S)/coroutine-nested/nested_test.html
|
||||
```
|
||||
|
||||
Reuses `LDFLAGS_COROUTINE` (already has `startModal` and `emscripten_fiber_swap` in `ASYNCIFY_IMPORTS`).
|
||||
|
||||
### E2E spec: `tests/e2e/coroutine-nested.spec.ts`
|
||||
|
||||
Three tests mirroring `coroutine.spec.ts`:
|
||||
1. "loads and reports case inventory" — all 8 `CASE` lines present
|
||||
2. "reports zero failures" — SUMMARY parseable, total=8, failed=0, no pageerror
|
||||
3. "per-scenario status (diagnostic)" — `expect.soft` per case for triage view
|
||||
|
||||
---
|
||||
|
||||
## Implementation Sequence
|
||||
|
||||
1. Scaffold: dir, stub `.cpp`, Makefile rule, verify build succeeds
|
||||
2. Implement Baselines 1 and 2 — confirm test infra works
|
||||
3. Implement Scenario 3 — expect crash (successful reproduction)
|
||||
4. Implement Scenarios 4–8 — fill diagnostic matrix
|
||||
5. Apply `handleSleep` save/restore fix
|
||||
6. Re-run nested suite — all 8 pass
|
||||
7. Re-run existing `coroutine` suite — no regression
|
||||
8. Full KiCad E2E — wizard closes cleanly, toolbars populate, Draw Line works
|
||||
9. Separate PR: libcontext.cpp hygiene (while(true), remove unwind, remove parking)
|
||||
|
||||
---
|
||||
|
||||
## File Inventory
|
||||
|
||||
| File | Role |
|
||||
|---|---|
|
||||
| `tests/apps/standalone/coroutine-nested/nested_test.cpp` | NEW — 8-case reproducer |
|
||||
| `tests/apps/Makefile.wasm` | Add `coroutine-nested` target |
|
||||
| `tests/e2e/coroutine-nested.spec.ts` | NEW — E2E for reproducer |
|
||||
| `scripts/common/inject-dyncall-shims.sh` | FIX — add `handleSleep` wrapper |
|
||||
| `kicad/thirdparty/libcontext/libcontext.cpp` | Hygiene follow-up — while(true), remove unwind |
|
||||
| `kicad/include/tool/coroutine.h` | Reference only |
|
||||
| `kicad/common/tool/tool_manager.cpp` | Reference only |
|
||||
| `wxwidgets/src/wasm/dialog.cpp` | Reference only — site of EM_ASYNC_JS startModal |
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- [Emscripten Issue #9153](https://github.com/emscripten-core/emscripten/issues/9153) — "Asyncify: Nested pause function calls does not work" (WONTFIX). Matches our failure exactly.
|
||||
- [Emscripten Issue #13302](https://github.com/emscripten-core/emscripten/issues/13302) — fiber swap return value bug (same single-slot design root)
|
||||
- [Emscripten Issue #12270](https://github.com/emscripten-core/emscripten/issues/12270) — fibers + embind return undefined (WONTFIX)
|
||||
- [Emscripten Issue #12239](https://github.com/emscripten-core/emscripten/issues/12239) — `start is not a function` in doRewind (empty exportCallStack variant)
|
||||
- [Emscripten PR #9859](https://github.com/emscripten-core/emscripten/pull/9859) — fiber API introduction, design discussion
|
||||
- [Asyncify blog post](https://kripken.github.io/blog/wasm/2019/07/16/asyncify.html) — the transformation explained
|
||||
- [QEMU coroutine-wasm.c](https://github.com/qemu/qemu/blob/master/util/coroutine-wasm.c) — reference implementation (solves a different subset of problems)
|
||||
- Local: `tools/emsdk/upstream/emscripten/src/lib/libasync.js` — ground truth for Asyncify JS runtime
|
||||
- Local: `tools/emsdk/upstream/emscripten/system/include/emscripten/fiber.h` — fiber API
|
||||
489
docs/research/threading_2.md
Normal file
489
docs/research/threading_2.md
Normal file
|
|
@ -0,0 +1,489 @@
|
|||
# Research: Asyncify + Coroutines — External Solutions, QEMU Deep Dive, and Why Alternatives Fail
|
||||
|
||||
This document extends `threading_1.md` with external research: how other projects solved (or failed to solve) the exact same problem, the technical details of why JSPI/WasmFX/state-machines don't help, and a deeper analysis of QEMU's working implementation vs our code.
|
||||
|
||||
---
|
||||
|
||||
## How Asyncify Works at the Instruction Level
|
||||
|
||||
Understanding why coroutines break Asyncify requires knowing exactly what the binary transformation does.
|
||||
|
||||
### The Binaryen Pass
|
||||
|
||||
`wasm-opt --asyncify` (implemented in `binaryen/src/passes/Asyncify.cpp`) rewrites every WASM function that can transitively reach an "async import" (a function that might suspend). It transforms each function into a three-state machine:
|
||||
|
||||
- **State 0 (Normal)**: Code runs as-is.
|
||||
- **State 1 (Unwinding)**: Functions return immediately, saving their local variables and a call-site index into a contiguous "asyncify stack" region in linear memory.
|
||||
- **State 2 (Rewinding)**: Functions are re-entered from the top. They read saved call indices to skip forward to the correct inner call, restoring locals along the way.
|
||||
|
||||
Two globals drive everything:
|
||||
```
|
||||
__asyncify_state: 0 = Normal, 1 = Unwinding, 2 = Rewinding
|
||||
__asyncify_data: pointer to the asyncify buffer for the current operation
|
||||
```
|
||||
|
||||
### Asyncify Data Buffer Layout
|
||||
|
||||
Each fiber/coroutine has its own buffer (the "asyncify stack"):
|
||||
```
|
||||
[ptr+0] i32: current stack position (grows upward as data is pushed)
|
||||
[ptr+4] i32: stack end (upper bound — overflow → wasm trap)
|
||||
[ptr+8] i32: rewind_id (which WASM export to re-enter during rewind)
|
||||
[ptr+12] ... actual saved data: alternating call indices + serialized locals
|
||||
```
|
||||
|
||||
### Before/After Transformation
|
||||
|
||||
**Before:**
|
||||
```c
|
||||
void foo(int x) {
|
||||
x = x + 1;
|
||||
x = x / 2;
|
||||
bar(x); // ← might trigger a pause
|
||||
while (x & 7) x = x + 1;
|
||||
}
|
||||
```
|
||||
|
||||
**After (pseudocode of the generated WASM):**
|
||||
```c
|
||||
void foo(int x) {
|
||||
if (__asyncify_state == REWINDING) {
|
||||
x = pop_from_asyncify_stack();
|
||||
call_index = pop_from_asyncify_stack();
|
||||
}
|
||||
|
||||
if (__asyncify_state == NORMAL) {
|
||||
x = x + 1;
|
||||
x = x / 2;
|
||||
}
|
||||
|
||||
if (__asyncify_state == NORMAL || call_index == 0) {
|
||||
bar(x);
|
||||
if (__asyncify_state == UNWINDING) {
|
||||
push_to_asyncify_stack(0); // call index
|
||||
push_to_asyncify_stack(x); // local
|
||||
return; // cooperative return
|
||||
}
|
||||
}
|
||||
|
||||
if (__asyncify_state == NORMAL) {
|
||||
while (x & 7) x = x + 1;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Every function in the call chain gets this treatment. During unwind, each frame saves state and returns. During rewind, each frame skips ahead to the saved call site and dives deeper.
|
||||
|
||||
### The Fundamental Assumption
|
||||
|
||||
**Asyncify assumes a single linear call stack.** The rewind mechanism works by re-entering the outermost export and replaying the call chain from the top down. This requires that the call stack at rewind time is identical to the one at unwind time.
|
||||
|
||||
When `jump_fcontext()` or `emscripten_fiber_swap()` switches the C stack pointer to a different region of memory, the entire call chain changes. This is fine **if and only if** the Asyncify machinery knows about it — which is what `emscripten_fiber_swap` does. Each fiber has its own asyncify buffer, so unwind saves into fiber A's buffer and rewind uses fiber B's buffer. The JS glue orchestrates which buffer is active.
|
||||
|
||||
**What breaks**: If code uses raw stack manipulation (like native boost.context assembly) that bypasses Asyncify entirely. Then Asyncify's bookkeeping points to a call chain that no longer exists.
|
||||
|
||||
### Indirect Calls Compound the Problem
|
||||
|
||||
Because `jump_fcontext` in non-WASM code is called through a function pointer (table call), Asyncify by default conservatively assumes any indirect call may reach an async import. This causes ALL indirect call sites to be instrumented, massively inflating code size. The workaround (`ASYNCIFY_IGNORE_INDIRECT`) skips indirect call analysis but is dangerous if any indirect call IS on the active stack during unwind.
|
||||
|
||||
**Sources**: [Binaryen Asyncify.cpp](https://github.com/WebAssembly/binaryen/blob/main/src/passes/Asyncify.cpp), [Alon Zakai's blog post](https://kripken.github.io/blog/wasm/2019/07/16/asyncify.html), [emscripten #8979](https://github.com/emscripten-core/emscripten/issues/8979)
|
||||
|
||||
---
|
||||
|
||||
## emscripten_fiber_t: The Correct Abstraction
|
||||
|
||||
Emscripten's fiber API (`<emscripten/fiber.h>`) is specifically designed to solve the problem of multiple execution stacks on top of Asyncify. Our code already uses it — the question is whether it's used correctly.
|
||||
|
||||
### How It Works
|
||||
|
||||
Each `emscripten_fiber_t` contains:
|
||||
```c
|
||||
typedef struct {
|
||||
void* stack_base; // C stack top
|
||||
void* stack_limit; // C stack bottom
|
||||
void* stack_ptr; // current C stack pointer (saved on swap)
|
||||
void (*entry)(void*); // entry function (NULL after first call)
|
||||
void* user_data; // argument for entry function
|
||||
asyncify_data_t asyncify_data; // THIS fiber's own asyncify buffer
|
||||
} emscripten_fiber_t;
|
||||
```
|
||||
|
||||
`emscripten_fiber_swap(old_fiber, new_fiber)`:
|
||||
1. Triggers Asyncify unwind of the current call stack into `old_fiber->asyncify_data`
|
||||
2. Switches the C stack pointer to `new_fiber->stack_ptr`
|
||||
3. Either calls `new_fiber->entry` (if first entry) or triggers Asyncify rewind using `new_fiber->asyncify_data`
|
||||
|
||||
**Key**: Because each fiber has its own `asyncify_data`, Asyncify tracks each fiber's call stack independently. Context switches tell Asyncify "save state here, restore state there."
|
||||
|
||||
### The Critical Constraint
|
||||
|
||||
From `fiber.h` documentation:
|
||||
|
||||
> "If entry_func returns, the entire program will end, as if main had returned."
|
||||
|
||||
When the entry function returns, there's no caller to return to — the fiber was started from a swap, not a regular function call. Emscripten treats this as program exit.
|
||||
|
||||
### What This Means For Our Code
|
||||
|
||||
Our `wasm_fcontext_entry()` (libcontext.cpp:249-272) violates this constraint. After `ctx->entry(ctx->transfer_value)` returns, the function tries to swap back using a stack-local parking fiber. This is undefined behavior per the Emscripten docs.
|
||||
|
||||
**Sources**: [fiber.h docs](https://emscripten.org/docs/api_reference/fiber.h.html), [PR #9859](https://github.com/emscripten-core/emscripten/pull/9859), [boost.context #109](https://github.com/boostorg/context/issues/109)
|
||||
|
||||
---
|
||||
|
||||
## The "Cannot Have Multiple Async Operations in Flight" Rule
|
||||
|
||||
Asyncify enforces a hard invariant: only one unwind/rewind cycle can be active at any moment. The global `Asyncify.state` variable tracks this. The assertion "Cannot have multiple async operations in flight at once" fires when:
|
||||
|
||||
- WASM is suspended (state = unwinding or rewinding)
|
||||
- A second call tries to enter the WASM module (e.g., from a JS event handler)
|
||||
|
||||
### Why This Matters for KiCad
|
||||
|
||||
When an interactive tool coroutine is suspended in `Wait()`, the fiber has been unwound and the main fiber rewound. Main is now in Normal state. A browser event fires, `ProcessEvents()` runs, finds a matching event, and resumes the coroutine via `cofunc->Resume()` — this triggers a new fiber swap. This is fine because the previous operation completed.
|
||||
|
||||
But consider RunMainStack: coroutine fiber → main fiber → ShowModal() → EM_ASYNC_JS suspends main. Now main's asyncify state is being managed by the EM_ASYNC_JS mechanism, AND the fiber system has its own asyncify buffers. These are separate paths but share `__asyncify_state`.
|
||||
|
||||
The `emscripten_fiber_swap` mechanism handles this correctly because the fiber JS glue and EM_ASYNC_JS use different code paths. But bugs in the glue code (like the dynCall no-ops) can cause state confusion.
|
||||
|
||||
### The `setTimeout(wakeUp, 0)` Pattern
|
||||
|
||||
When Asyncify.wakeUp() is called while compiled code is still on the JS call stack, it corrupts state. The fix is always to defer: `setTimeout(wakeUp, 0)` ensures the previous operation has fully unwound before starting the next rewind. Our modal dialog code uses `setTimeout(0)` twice (double-deferred) for this reason — documented in `../debugging/learning.md`.
|
||||
|
||||
**Sources**: [emscripten #16291](https://github.com/emscripten-core/emscripten/issues/16291), [emscripten #18412](https://github.com/emscripten-core/emscripten/issues/18412), [emscripten #10515](https://github.com/emscripten-core/emscripten/issues/10515)
|
||||
|
||||
---
|
||||
|
||||
## QEMU WASM: The Gold Standard (Deep Technical Analysis)
|
||||
|
||||
QEMU was compiled to WASM with working coroutines. The patch series "Enable QEMU to run on browsers" (Kohei Tokunaga, April 2025, merged upstream) is the canonical reference for this problem.
|
||||
|
||||
### QEMU's Coroutine Problem
|
||||
|
||||
QEMU's async I/O uses coroutines everywhere — disk reads, network operations, etc. On native systems, QEMU uses `coroutine-ucontext.c` (`ucontext_t` + `sigsetjmp`/`siglongjmp`). Emscripten doesn't support ucontext, so they wrote a new backend.
|
||||
|
||||
### The Implementation: `util/coroutine-wasm.c`
|
||||
|
||||
127 lines. Three functions.
|
||||
|
||||
**The struct:**
|
||||
```c
|
||||
typedef struct {
|
||||
Coroutine base;
|
||||
void *stack; // C stack buffer (heap-allocated, persists)
|
||||
size_t stack_size;
|
||||
void *asyncify_stack; // Asyncify data buffer (heap-allocated, persists)
|
||||
size_t asyncify_stack_size;
|
||||
CoroutineAction action; // Communication: YIELD, TERMINATE, etc.
|
||||
emscripten_fiber_t fiber;
|
||||
} CoroutineEmscripten;
|
||||
```
|
||||
|
||||
Both stacks are heap-allocated and persist for the coroutine's entire lifetime. No stack-local temporaries.
|
||||
|
||||
**The trampoline (most important part):**
|
||||
```c
|
||||
static void coroutine_trampoline(void *co_)
|
||||
{
|
||||
Coroutine *co = co_;
|
||||
|
||||
while (true) { // ← NEVER returns
|
||||
co->entry(co->entry_arg); // Run the coroutine body
|
||||
qemu_coroutine_switch(co, co->caller,
|
||||
COROUTINE_TERMINATE); // Swap back to caller
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Walk-through:
|
||||
1. `emscripten_fiber_init()` is called with `coroutine_trampoline` as entry
|
||||
2. When first swapped to, `coroutine_trampoline` starts running
|
||||
3. Calls `co->entry(co->entry_arg)` — the actual I/O handler
|
||||
4. Handler may yield many times (each yield does fiber_swap back to caller, resume does fiber_swap back)
|
||||
5. Handler finishes and returns
|
||||
6. `coroutine_trampoline` resumes after the `co->entry()` line
|
||||
7. Calls `qemu_coroutine_switch(co, co->caller, COROUTINE_TERMINATE)` — swaps back with "done" flag
|
||||
8. `while(true)` loops. If nobody swaps back, stays suspended forever (fiber freed later)
|
||||
9. **Entry function never returns.**
|
||||
|
||||
**Context switch:**
|
||||
```c
|
||||
CoroutineAction qemu_coroutine_switch(Coroutine *from_, Coroutine *to_,
|
||||
CoroutineAction action)
|
||||
{
|
||||
CoroutineEmscripten *from = DO_UPCAST(CoroutineEmscripten, base, from_);
|
||||
CoroutineEmscripten *to = DO_UPCAST(CoroutineEmscripten, base, to_);
|
||||
|
||||
set_current(to_);
|
||||
to->action = action;
|
||||
emscripten_fiber_swap(&from->fiber, &to->fiber);
|
||||
return from->action;
|
||||
}
|
||||
```
|
||||
|
||||
Simple two-party swap. Communication via the `action` field.
|
||||
|
||||
**Main thread bootstrap (lazy init):**
|
||||
```c
|
||||
Coroutine *qemu_coroutine_self(void)
|
||||
{
|
||||
Coroutine *self = get_current();
|
||||
if (!self) {
|
||||
CoroutineEmscripten *leaderp = g_malloc0(sizeof(*leaderp));
|
||||
leaderp->asyncify_stack = g_malloc0(leader_asyncify_stack_size);
|
||||
leaderp->asyncify_stack_size = leader_asyncify_stack_size;
|
||||
|
||||
emscripten_fiber_init_from_current_context(
|
||||
&leaderp->fiber,
|
||||
leaderp->asyncify_stack,
|
||||
leaderp->asyncify_stack_size
|
||||
);
|
||||
|
||||
set_leader(leaderp);
|
||||
self = &leaderp->base;
|
||||
set_current(self);
|
||||
}
|
||||
return self;
|
||||
}
|
||||
```
|
||||
|
||||
The main execution context is captured lazily as a fiber. Its asyncify stack is heap-allocated.
|
||||
|
||||
**Cleanup:**
|
||||
```c
|
||||
void qemu_coroutine_delete(Coroutine *co_)
|
||||
{
|
||||
CoroutineEmscripten *co = DO_UPCAST(CoroutineEmscripten, base, co_);
|
||||
qemu_free_stack(co->stack, co->stack_size);
|
||||
g_free(co->asyncify_stack);
|
||||
g_free(co);
|
||||
}
|
||||
```
|
||||
|
||||
Both stacks freed. No dangling pointers because the while(true) loop means the fiber is either suspended (waiting inside the loop) or never entered again.
|
||||
|
||||
### QEMU Limitation We Share
|
||||
|
||||
From the patch notes: "Fiber does not support submitting coroutines to other threads." QEMU disabled cross-thread coroutine operations in 9pfs for Emscripten builds. KiCad's tool coroutines are single-threaded by design, so this is not a concern.
|
||||
|
||||
### What QEMU Does NOT Need (That We Do)
|
||||
|
||||
QEMU's coroutine model is simpler than KiCad's:
|
||||
|
||||
| Feature | QEMU | KiCad |
|
||||
|---------|------|-------|
|
||||
| Context switch parties | Always 2: coroutine ↔ caller | 3 types: FROM_ROOT, FROM_ROUTINE, CONTINUE_AFTER_ROOT |
|
||||
| Communication | Simple `action` enum | `INVOCATION_ARGS*` struct via `intptr_t` |
|
||||
| RunMainStack (execute on main from coroutine) | Not needed | Essential for ShowModal from tools |
|
||||
| Nested coroutines | Not used | Parent→child tool invocation |
|
||||
| EM_ASYNC_JS nested inside fiber context | Not applicable | ShowModal inside RunMainStack |
|
||||
|
||||
These differences mean we can't copy QEMU verbatim. We adopt the **trampoline pattern** and **heap-only buffers**, but keep KiCad's richer invocation protocol.
|
||||
|
||||
**Sources**: [ktock/qemu-wasm](https://github.com/ktock/qemu-wasm), [QEMU coroutine-fiber.c patch](https://www.mail-archive.com/qemu-block@nongnu.org/msg119137.html), [PATCH 00/10](https://patchew.org/QEMU/cover.1744032780.git.ktokunaga.mail@gmail.com/)
|
||||
|
||||
---
|
||||
|
||||
## Why JSPI Does NOT Solve Our Problem
|
||||
|
||||
JSPI (JavaScript Promise Integration) is a WebAssembly standard (Phase 4 W3C, Chrome 137+, Firefox 139+) that works at the VM level: the JS engine intercepts Promise returns from WASM-to-JS calls and natively suspends the WASM stack. No binary transformation needed.
|
||||
|
||||
### How JSPI Differs from Asyncify
|
||||
|
||||
| | Asyncify | JSPI |
|
||||
|---|---|---|
|
||||
| Mechanism | Binaryen rewrites WASM binary as state machine | VM natively suspends/resumes WASM stack |
|
||||
| Code size overhead | ~50% | Zero |
|
||||
| Suspension speed | Serialize/deserialize all frames | ~1 microsecond |
|
||||
| Where suspension happens | Inside WASM (any instrumented call) | At WASM→JS boundary only |
|
||||
|
||||
### Why JSPI Cannot Replace Coroutines
|
||||
|
||||
**1. JSPI only suspends at JS-WASM boundaries.**
|
||||
|
||||
A JSPI suspension happens when a WASM function calls a JS function that returns a Promise. Coroutine switches from one C++ coroutine to another (both inside WASM) do not cross a JS boundary. JSPI cannot mediate them.
|
||||
|
||||
```
|
||||
Coroutine A ←→ Coroutine B (intra-WASM, no JS boundary → JSPI can't help)
|
||||
Main stack → JS API (WASM→JS boundary → JSPI works here)
|
||||
```
|
||||
|
||||
**2. No JS frames can be suspended.**
|
||||
|
||||
V8 enforces: JSPI cannot capture JS frames on the stack. When WASM calls JS (which calls back into WASM), only the inner WASM stack can be suspended. This means callback-heavy patterns (like our ProcessEvents loop) need careful architecture.
|
||||
|
||||
The error is: "trying to suspend without a WebAssembly.promising export" — which Qt also hits.
|
||||
|
||||
**3. Emscripten main loop incompatibility.**
|
||||
|
||||
`emscripten_set_main_loop()`, `emscripten_request_animation_frame_loop()`, and `emscripten_set_timeout()` invoke WASM callbacks WITHOUT wrapping them in `WebAssembly.promising()`. Those callbacks cannot be suspended by JSPI.
|
||||
|
||||
Qt's investigation of JSPI (`QT_EMSCRIPTEN_ASYNCIFY=2`) confirmed they hit this exact error for dialog operations. As of early 2026, Qt's JSPI support is still incomplete.
|
||||
|
||||
**4. Each JSPI export runs on a separate stack.**
|
||||
|
||||
JSPI allocates a new stack per suspended export call. Multiple outstanding JSPI suspensions (one per tool coroutine) each get their own stack. But the "switch between" semantics of cooperative coroutines (yield to scheduler → scheduler resumes specific other coroutine) doesn't map onto JSPI's Promise-based model.
|
||||
|
||||
### What JSPI IS Good For
|
||||
|
||||
Async operations that cross JS boundaries: file I/O, network requests, `sleep()`, dialog results. If KiCad's tools could be restructured to yield to JS rather than to another C++ coroutine, JSPI becomes applicable. But the current architecture — `COROUTINE::yield()` switches directly via `jump_fcontext` — has no JS boundary.
|
||||
|
||||
**Verdict**: JSPI could potentially replace EM_ASYNC_JS for modal dialogs (Pattern 4 in threading_1.md). It cannot replace emscripten_fiber_swap for tool coroutines (Patterns 1/2/3/7/8).
|
||||
|
||||
**Sources**: [V8 JSPI blog](https://v8.dev/blog/jspi), [V8 JSPI new API](https://v8.dev/blog/jspi-newapi), [emscripten #22493](https://github.com/emscripten-core/emscripten/issues/22493), [emscripten #22469](https://github.com/emscripten-core/emscripten/issues/22469), [wasm/stack-switching #49](https://github.com/WebAssembly/stack-switching/issues/49)
|
||||
|
||||
---
|
||||
|
||||
## Why WasmFX / Typed Continuations Won't Help (Yet)
|
||||
|
||||
WasmFX is a formal WebAssembly proposal adding native stack-switching instructions: `cont.new`, `resume`, `suspend`, `switch`, `cont.bind`. These would allow efficient, type-safe coroutine/fiber switching entirely within WASM — the "correct long-term solution."
|
||||
|
||||
### Status (April 2026)
|
||||
|
||||
**Not shipped in any browser.** Not enabled in Chrome, Firefox, or Safari. Wasmtime has partial x64-Linux-only experimental support (tracking [issue #10248](https://github.com/bytecodealliance/wasmtime/issues/10248)). The proposal has been under discussion since 2021. A reference interpreter exists, but browser shipping is not imminent.
|
||||
|
||||
**Verdict**: Do not plan around this. If it ships in 2027+, we can revisit. For now, Asyncify + emscripten_fiber_t is the only viable path.
|
||||
|
||||
**Sources**: [WasmFX site](http://wasmfx.dev/), [Stack Switching Explainer](https://github.com/WebAssembly/stack-switching/blob/main/proposals/stack-switching/Explainer.md)
|
||||
|
||||
---
|
||||
|
||||
## Qt for WebAssembly: The Closest GUI Framework Comparison
|
||||
|
||||
Qt is the closest analogy: large C++ GUI framework with blocking modal dialogs (`QDialog::exec()`), nested event loops (`QEventLoop::exec()`), and tools that assume synchronous behavior.
|
||||
|
||||
### Qt's Evolution
|
||||
|
||||
**Pre-6.3 (no Asyncify)**: No support for `exec()`. Forced API change to `show()` + signal/slot callbacks. Broke all sync dialog patterns.
|
||||
|
||||
**Qt 6.3+ with Asyncify**: Added `--enable-asyncify` build option. `QEventLoop::exec()` works by Asyncify-suspending the entire WASM module. The Qt event loop spins inside the Asyncify unwind, JS processes browser events, then Asyncify rewinds when ready.
|
||||
|
||||
**JSPI exploration (ongoing)**: Qt has `-feature-wasm-jspi` but as of early 2026 still hits "attempting to suspend without a WebAssembly.promising export" for dialog operations.
|
||||
|
||||
### Qt's Core Insight
|
||||
|
||||
Qt uses Asyncify not to implement cooperative C++ coroutines, but to make the **main** execution context suspendable at arbitrary call depth. The browser JS event loop becomes the "scheduler." There is no explicit coroutine switching between multiple C++ contexts.
|
||||
|
||||
### What Qt Has NOT Solved
|
||||
|
||||
Interactive tools that use cooperative coroutines (like KiCad's tool framework) are NOT something Qt needs to handle. Qt's model is signal/slot, not coroutine-based. This means Qt's experience validates Asyncify for modal dialogs but tells us nothing about the multi-fiber case.
|
||||
|
||||
**Sources**: [Qt WASM docs](https://doc.qt.io/qt-6/wasm.html), [Qt exec() on WASM](http://qtandeverything.blogspot.com/2019/05/exec-on-qt-webassembly.html), [QTBUG-102827](https://bugreports.qt.io/browse/QTBUG-102827)
|
||||
|
||||
---
|
||||
|
||||
## Python/Pyodide/Greenlet: The Conceptual Match
|
||||
|
||||
Python greenlets are stackful coroutines using `slp_switch` (similar to `jump_fcontext`) for stack switching. The Pyodide and Wasmer teams hit exactly our problem.
|
||||
|
||||
### Pyodide Finding
|
||||
|
||||
From [issue #2664](https://github.com/pyodide/pyodide/issues/2664): Hood Chatham identified a fundamental incompatibility — greenlet's `slp_switch` duplicates call stacks (like `fork()`). JSPI explicitly cannot duplicate stacks. Therefore JSPI alone is insufficient for greenlets. A `continulet` abstraction on top of WASM stack switching was needed.
|
||||
|
||||
### Wasmer's Greenlet Solution (2025)
|
||||
|
||||
Wasmer exposed runtime-level system calls (`wasix_context_create/switch/destroy`) implementing cooperative stack switching. This is specific to the Wasmer runtime, not applicable to browser WASM.
|
||||
|
||||
### Pyodide's `syncify()` / `runPythonSyncifying()`
|
||||
|
||||
Works in Chrome with JSPI or Node.js with `--experimental-wasm-stack-switching`. But requires the outer call to be wrapped in `WebAssembly.promising()` — same limitation.
|
||||
|
||||
### Lesson for Us
|
||||
|
||||
If you need N concurrently-suspended C++ coroutines that switch between each other inside WASM, neither raw Asyncify nor JSPI is sufficient alone. The `emscripten_fiber_t` API (managing one `asyncify_data` per fiber) is the correct and currently only tool. QEMU proved it works. Our implementation just has bugs.
|
||||
|
||||
**Sources**: [Pyodide #2664](https://github.com/pyodide/pyodide/issues/2664), [Wasmer greenlet post](https://wasmer.io/posts/greenlet-support-python-wasm)
|
||||
|
||||
---
|
||||
|
||||
## Other Real-World Examples
|
||||
|
||||
### WordPress Playground (PHP in WASM)
|
||||
|
||||
Uses Asyncify for synchronous PHP networking code. Works because PHP has a single-threaded, single-stack model. No cooperative coroutine switching inside PHP. Not comparable.
|
||||
|
||||
### minicoro (Single-header coroutine library)
|
||||
|
||||
[github.com/edubart/minicoro](https://github.com/edubart/minicoro) — a minimal C coroutine library that explicitly supports Emscripten/WASM via the fiber API. Its WASM backend is essentially a cleaner version of what we're doing in libcontext.cpp. Worth studying for patterns but doesn't add capabilities beyond what emscripten_fiber_t provides.
|
||||
|
||||
---
|
||||
|
||||
## Summary of All Approaches Evaluated
|
||||
|
||||
| Approach | Viability | Handles Intra-WASM Coroutines? | Handles Modal Dialogs? | Browser Support |
|
||||
|----------|-----------|-------------------------------|----------------------|----------------|
|
||||
| **Asyncify + emscripten_fiber_t** (current, with fixes) | **HIGH** | Yes | Yes (via EM_ASYNC_JS) | All browsers |
|
||||
| JSPI | Medium | **No** — only at JS boundary | Yes | Chrome 137+, Firefox 139+ |
|
||||
| WasmFX / Typed Continuations | Future | Yes (native) | Yes | **No browser support** |
|
||||
| C++20 stackless coroutines | Not viable | No — only top-level suspension | N/A | N/A |
|
||||
| Rewrite tools as state machines | Not viable | N/A (eliminates coroutines) | N/A | N/A |
|
||||
| Raw Asyncify handleSleep | Not viable | No — can't recurse | Partial | All browsers |
|
||||
| boost.context assembly for wasm32 | Impossible | N/A | N/A | N/A |
|
||||
|
||||
**Conclusion**: Fix the emscripten_fiber_t usage in libcontext.cpp. Everything else is either not ready, not applicable, or not viable.
|
||||
|
||||
---
|
||||
|
||||
## Concrete Bugs In Our Code (Updated From threading_1.md)
|
||||
|
||||
### Bug 1: Entry Function Returns (libcontext.cpp:249-272)
|
||||
|
||||
The `wasm_fcontext_entry()` function violates the Emscripten rule that fiber entry functions must never return. After the coroutine body finishes:
|
||||
- Creates a `parking_fiber` with `emscripten_fiber_init_from_current_context()` — but the asyncify stack is stack-local (64KB on the C stack)
|
||||
- Swaps from parking_fiber to return_to — the parking_fiber's asyncify state now points to stack memory that's garbage
|
||||
- Falls through to `emscripten_unwind_to_js_event_loop()` — terminates ALL WASM execution
|
||||
|
||||
**Fix**: QEMU-style `while(true)` trampoline. Swap back using `&ctx->fiber` (heap-allocated) instead of parking fiber.
|
||||
|
||||
### Bug 2: Detached Epoch Kills Everything (libcontext.cpp:329-335)
|
||||
|
||||
In `jump_fcontext()`, after `emscripten_fiber_swap()` returns, if `old_ctx->resume_epoch == expected_resume_epoch`, the code calls `emscripten_unwind_to_js_event_loop()`. This is the "ghost resume" detection, but the response (kill everything) is disproportionate.
|
||||
|
||||
**Fix**: Log the ghost and return 0 instead of killing. Let the caller handle the null INVOCATION_ARGS.
|
||||
|
||||
### Bug 3: Parking Fiber Complexity (libcontext.cpp:152-175)
|
||||
|
||||
The entire parking fiber mechanism — `ensure_parking_context()`, `active_fiber()` indirection, `parking_initialized` flag, `parking_asyncify_stack` — exists to handle the case where the entry function returns. With the while(true) trampoline, this case doesn't exist. The complexity can be removed entirely.
|
||||
|
||||
**Fix**: Delete parking infrastructure. Always use `&ctx->fiber`.
|
||||
|
||||
---
|
||||
|
||||
## Key External References
|
||||
|
||||
### Emscripten Issues
|
||||
- [#8979](https://github.com/emscripten-core/emscripten/issues/8979) — Coroutines broken with Asyncify (root issue)
|
||||
- [#9859](https://github.com/emscripten-core/emscripten/pull/9859) — Fiber API implementation PR
|
||||
- [#10515](https://github.com/emscripten-core/emscripten/issues/10515) — Asyncify repeated yield fails
|
||||
- [#13302](https://github.com/emscripten-core/emscripten/issues/13302) — Bad return value with Asyncify and fibers
|
||||
- [#16291](https://github.com/emscripten-core/emscripten/issues/16291) — Cannot have multiple async operations in flight
|
||||
- [#20413](https://github.com/emscripten-core/emscripten/issues/20413) — C++20 coroutines + JSPI
|
||||
- [#22469](https://github.com/emscripten-core/emscripten/issues/22469) — Trying to suspend JS frames with JSPI
|
||||
- [#22493](https://github.com/emscripten-core/emscripten/issues/22493) — Main loop incompatible with JSPI
|
||||
|
||||
### Documentation
|
||||
- [fiber.h docs](https://emscripten.org/docs/api_reference/fiber.h.html) — Emscripten fiber API reference
|
||||
- [Asyncify docs](https://emscripten.org/docs/porting/asyncify.html) — Asyncify porting guide
|
||||
- [Binaryen Asyncify.cpp](https://github.com/WebAssembly/binaryen/blob/main/src/passes/Asyncify.cpp) — Compiler pass source
|
||||
- [Alon Zakai's Asyncify blog](https://kripken.github.io/blog/wasm/2019/07/16/asyncify.html) — Technical deep dive
|
||||
|
||||
### QEMU
|
||||
- [ktock/qemu-wasm](https://github.com/ktock/qemu-wasm) — QEMU WASM repo
|
||||
- [PATCH 00/10](https://patchew.org/QEMU/cover.1744032780.git.ktokunaga.mail@gmail.com/) — Patch series: Enable QEMU to run on browsers
|
||||
- [coroutine-fiber.c patch](https://www.mail-archive.com/qemu-block@nongnu.org/msg119137.html) — The fiber backend
|
||||
|
||||
### JSPI / Stack Switching
|
||||
- [V8 JSPI blog](https://v8.dev/blog/jspi) — Introduction
|
||||
- [V8 JSPI new API](https://v8.dev/blog/jspi-newapi) — Updated API
|
||||
- [WasmFX Explainer](https://github.com/WebAssembly/stack-switching/blob/main/proposals/stack-switching/Explainer.md)
|
||||
- [Wasmtime #10248](https://github.com/bytecodealliance/wasmtime/issues/10248) — Stack switching tracking
|
||||
- [JS frames constraint](https://github.com/WebAssembly/stack-switching/issues/49)
|
||||
|
||||
### Other Projects
|
||||
- [Pyodide #2664](https://github.com/pyodide/pyodide/issues/2664) — Greenlet/stackful coroutines in WASM
|
||||
- [Wasmer greenlet](https://wasmer.io/posts/greenlet-support-python-wasm) — Runtime-level solution
|
||||
- [boost.context #109](https://github.com/boostorg/context/issues/109) — WASM support (not possible)
|
||||
- [minicoro](https://github.com/edubart/minicoro) — Minimal C coroutine lib with WASM support
|
||||
- [Qt WASM docs](https://doc.qt.io/qt-6/wasm.html)
|
||||
- [Qt QTBUG-102827](https://bugreports.qt.io/browse/QTBUG-102827) — Asyncify crash
|
||||
- [WordPress Playground](https://wordpress.github.io/wordpress-playground/developers/architecture/wasm-asyncify/) — PHP WASM asyncify
|
||||
Loading…
Reference in a new issue