Add two-phase build with host asyncify transformation

- Move asyncify from Docker to host to avoid memory issues - Auto-download Binaryen v121 (v125 has regression bug) - Use -O1 for debug builds (V8 local count limit) - Remove asyncify flags from linker (handled by wasm-opt) - Document two-phase build in build.md 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-13 22:13:31 +01:00 · 2025-12-13 22:13:31 +01:00 · b70f098ae2
commit b70f098ae2
parent 0c48855191
6 changed files with 187 additions and 19 deletions
--- a/.gitignore
+++ b/.gitignore
@ -52,3 +52,4 @@ wxwidgets-clean/
 *.log
 *.tmp
 output/
+tools/
--- a/build.md
+++ b/build.md
@ -4,9 +4,14 @@ This document describes how to build KiCad for WebAssembly using the Docker-base

 ## Prerequisites

+### Docker
 - Docker Desktop with ARM64 support (for Apple Silicon) or x86_64
 - 10+ GB disk space for build cache
- Recommended: 10 CPUs, 16GB RAM allocated to Docker
+- Recommended: 10 CPUs, 32GB RAM allocated to Docker
+
+### Host Tools
+
+Binaryen (wasm-opt) is downloaded automatically by the build script. No manual installation needed.

 ## Quick Start

@ -31,6 +36,48 @@ This document describes how to build KiCad for WebAssembly using the Docker-base
 - `build-wasm/kicad-pcbnew/pcbnew/pcbnew.wasm` - WASM binary
 - `build-wasm/kicad-pcbnew/pcbnew/pcbnew.wasm.map` - Source map (debug builds)

+## Two-Phase Build
+
+The build is split into two phases due to memory requirements:
+
+### Phase 1: Docker Compilation
+Compiles KiCad to WASM **without** asyncify transformation. This runs inside Docker with 32GB memory limit.
+
+### Phase 2: Host Asyncify
+Applies `wasm-opt --asyncify` on the host machine using Binaryen v121 (downloaded automatically to `tools/`). This transformation uses ~20-30GB RAM.
+
+**Note:** Binaryen v121 is used because v125 has a regression causing crashes in the asyncify liveness analysis.
+
+### Why Asyncify?
+
+Asyncify is an Emscripten transformation that allows WASM code to pause and resume execution. This is required for:
+
+- **Modal dialogs** - `wxDialog::ShowModal()` blocks until user closes the dialog
+- **Message boxes** - `wxMessageBox()` waits for user response
+- **Clipboard operations** - Browser clipboard API is async
+- **Sleep/wait operations** - Any blocking call that needs to yield to the browser
+
+Without asyncify, modal dialogs would freeze the browser because WASM cannot yield control back to JavaScript's event loop.
+
+### How It Works
+
+1. `docker/build.sh` compiles KiCad in Docker (no asyncify flags)
+2. Output is copied to `./output/` directory
+3. `wasm-opt --asyncify` runs on host, transforming the WASM binary
+4. Final output is ready for browser execution
+
+### Technical Details
+
+The asyncify transformation:
+- Instruments every function that might be on the call stack during an async operation
+- Adds stack save/restore logic to unwind and rewind the WASM stack
+- Increases binary size by ~20% (141MB → 171MB for KiCad)
+- Uses `asyncify-imports` pattern matching to identify async entry points
+
+Import patterns used:
+- `env.invoke_*` - Exception handling trampolines
+- `env.__asyncjs__*` - EM_ASYNC_JS functions (like `startModal()`)
+
 ## Docker Architecture

 **Base image:** `emscripten/emsdk:4.0.2-arm64`
--- a/docker/build.sh
+++ b/docker/build.sh
@ -1,13 +1,25 @@
 #!/bin/bash
-# Build KiCad WASM inside Docker container
+# Build KiCad WASM inside Docker container, then apply asyncify on host
+#
+# The build is split into two phases:
+# 1. Docker: Compile KiCad to WASM (without asyncify)
+# 2. Host: Apply asyncify transformation (uses Binaryen v121)
+#
+# Binaryen is downloaded automatically - no prerequisites needed.
+
 set -e

 cd "$(dirname "$0")/.."

+# Get wasm-opt (downloads Binaryen v121 if not cached)
+echo "Checking wasm-opt..."
+WASM_OPT=$(./scripts/common/get-wasm-opt.sh)
+echo "Using: ${WASM_OPT}"
+
 # Start container if not running
 docker compose -f docker/docker-compose.yml up -d

-# Run build command
+# Run build command (without asyncify - handled on host due to memory requirements)
 docker compose -f docker/docker-compose.yml exec kicad-wasm-builder \
    /workspace/scripts/kicad/build-pcbnew.sh "$@"

@ -16,4 +28,23 @@ echo "Copying build output to ./output/..."
 docker compose -f docker/docker-compose.yml exec kicad-wasm-builder \
    bash -c "mkdir -p /workspace/output && cp /workspace/build-wasm/kicad-pcbnew/pcbnew/pcbnew.{js,wasm,wasm.map,worker.js} /workspace/output/ 2>/dev/null || cp /workspace/build-wasm/kicad-pcbnew/pcbnew/pcbnew.{js,wasm} /workspace/output/"

+# Apply asyncify transformation on host
+# This is done on the host because wasm-opt --asyncify needs 50GB+ RAM for KiCad,
+# which exceeds typical Docker memory limits
+echo ""
+echo "Applying asyncify transformation on host..."
+echo "This may take several minutes and use significant RAM..."
+
+# Asyncify import patterns (functions that trigger async suspension)
+# - env.invoke_* : Exception handling trampolines
+# - env.__asyncjs__* : EM_ASYNC_JS functions
+ASYNCIFY_IMPORTS="env.invoke_*,env.__asyncjs__*"
+
+"${WASM_OPT}" --asyncify \
+    --pass-arg=asyncify-imports@${ASYNCIFY_IMPORTS} \
+    --pass-arg=asyncify-propagate-addlist \
+    output/pcbnew.wasm -o output/pcbnew.wasm
+
+echo ""
 echo "Build complete. Output files in ./output/"
+ls -lh output/
--- a/scripts/common/get-wasm-opt.sh
+++ b/scripts/common/get-wasm-opt.sh
@ -0,0 +1,66 @@
+#!/bin/bash
+# Download and cache Binaryen wasm-opt
+#
+# Usage: ./scripts/tools/get-wasm-opt.sh
+# Output: Prints path to wasm-opt executable
+#
+# Binaryen v121 is used because v125 has a regression causing asyncify crashes.
+# The binary is cached in tools/binaryen-121/
+
+set -e
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
+
+BINARYEN_VERSION="121"
+BINARYEN_DIR="${PROJECT_ROOT}/tools/binaryen-${BINARYEN_VERSION}"
+WASM_OPT="${BINARYEN_DIR}/bin/wasm-opt"
+
+download_binaryen() {
+    # Detect platform
+    local os=$(uname -s)
+    local arch=$(uname -m)
+    local platform=""
+
+    case "${os}-${arch}" in
+        Darwin-arm64)  platform="arm64-macos" ;;
+        Darwin-x86_64) platform="x86_64-macos" ;;
+        Linux-aarch64) platform="aarch64-linux" ;;
+        Linux-x86_64)  platform="x86_64-linux" ;;
+        *)
+            echo "ERROR: Unsupported platform: ${os}-${arch}" >&2
+            exit 1
+            ;;
+    esac
+
+    local url="https://github.com/WebAssembly/binaryen/releases/download/version_${BINARYEN_VERSION}/binaryen-version_${BINARYEN_VERSION}-${platform}.tar.gz"
+    local tarball="${PROJECT_ROOT}/tools/binaryen-${BINARYEN_VERSION}.tar.gz"
+
+    echo "Downloading Binaryen v${BINARYEN_VERSION} for ${platform}..." >&2
+    mkdir -p "${PROJECT_ROOT}/tools"
+    curl -L -o "${tarball}" "${url}"
+
+    echo "Extracting..." >&2
+    tar -xzf "${tarball}" -C "${PROJECT_ROOT}/tools"
+    mv "${PROJECT_ROOT}/tools/binaryen-version_${BINARYEN_VERSION}" "${BINARYEN_DIR}"
+    rm "${tarball}"
+
+    echo "Binaryen v${BINARYEN_VERSION} installed to ${BINARYEN_DIR}" >&2
+}
+
+# Download Binaryen if not cached
+if [ ! -x "${WASM_OPT}" ]; then
+    download_binaryen
+fi
+
+# Verify version
+INSTALLED_VERSION=$("${WASM_OPT}" --version 2>&1 | grep -o '[0-9]\+' | head -1)
+if [ "${INSTALLED_VERSION}" != "${BINARYEN_VERSION}" ]; then
+    echo "WARNING: wasm-opt version mismatch (got ${INSTALLED_VERSION}, expected ${BINARYEN_VERSION})" >&2
+    echo "Re-downloading..." >&2
+    rm -rf "${BINARYEN_DIR}"
+    download_binaryen
+fi
+
+# Output path to wasm-opt (this is the only stdout output)
+echo "${WASM_OPT}"
--- a/scripts/kicad/build-pcbnew.sh
+++ b/scripts/kicad/build-pcbnew.sh
@ -119,16 +119,22 @@ log_info "Building KiCad PCBnew ${KICAD_VERSION} for WASM..."
 # Use environment DEBUG_BUILD if set, otherwise check local --debug flag
 # -fexceptions is required because wxWidgets is built with exceptions enabled
 # -matomics -mbulk-memory are required for shared memory (pthreads)
+# NOTE: We use -O1 for debug builds because -O0 produces WASM with too many
+# locals for V8/Chrome to compile (error: "local count too large").
+# -O1 keeps debug info but optimizes enough to stay under V8's limits.
 if [ "${DEBUG_BUILD:-0}" = "1" ] || [ $DEBUG -eq 1 ]; then
    BUILD_TYPE="Debug"
-    EXTRA_FLAGS="-g -O0 -fexceptions -matomics -mbulk-memory"
-    LINKER_DEBUG_FLAGS="-g -gsource-map -fexceptions"
-    log_info "Building KiCad in DEBUG mode (with source maps)"
+    EXTRA_FLAGS="-g -O1 -fexceptions -matomics -mbulk-memory"
+    # -O0 at link time skips wasm-opt (which can OOM on large WASM with debug symbols)
+    LINKER_DEBUG_FLAGS="-O0 -g -gsource-map -fexceptions"
+    log_info "Building KiCad in DEBUG mode (with source maps, -O1 for WASM compatibility)"
 else
    BUILD_TYPE="Release"
    EXTRA_FLAGS="-O2 -fexceptions -matomics -mbulk-memory"
-    LINKER_DEBUG_FLAGS="-fexceptions"
-    log_info "Building KiCad in RELEASE mode"
+    # -O0 at link time skips wasm-opt (which can OOM on large WASM files)
+    # Compilation is still -O2 for optimized code, but we skip post-link wasm-opt
+    LINKER_DEBUG_FLAGS="-O0 -fexceptions"
+    log_info "Building KiCad in RELEASE mode (skipping wasm-opt due to memory limits)"
 fi

 # Step 6: Create build directory
@ -178,7 +184,7 @@ emcmake cmake "${KICAD_DIR}" \
    -DCMAKE_POLICY_VERSION_MINIMUM=3.5 \
    -DCMAKE_CXX_FLAGS="${EXTRA_FLAGS} -pthread -sUSE_ZLIB=1 -DKICAD_USE_PLATFORM_WASM=1 -I${SYSROOT}/include" \
    -DCMAKE_C_FLAGS="${EXTRA_FLAGS} -pthread -sUSE_ZLIB=1 -I${SYSROOT}/include" \
-    -DCMAKE_EXE_LINKER_FLAGS="${LINKER_DEBUG_FLAGS} -pthread -sUSE_ZLIB=1 -sASYNCIFY=1 -sASYNCIFY_STACK_SIZE=65536 -sUSE_PTHREADS=1 -sPTHREAD_POOL_SIZE=4 -sALLOW_MEMORY_GROWTH=1 -sINITIAL_MEMORY=256MB -sMAXIMUM_MEMORY=4GB -L${SYSROOT}/lib ${STUBS_BUILD}/libgit2_stub.a ${STUBS_BUILD}/libcurl_stub.a" \
+    -DCMAKE_EXE_LINKER_FLAGS="${LINKER_DEBUG_FLAGS} -pthread -sUSE_ZLIB=1 -sUSE_PTHREADS=1 -sPTHREAD_POOL_SIZE=4 -sALLOW_MEMORY_GROWTH=1 -sINITIAL_MEMORY=256MB -sMAXIMUM_MEMORY=4GB -L${SYSROOT}/lib ${STUBS_BUILD}/libgit2_stub.a ${STUBS_BUILD}/libcurl_stub.a" \
    -DCMAKE_PREFIX_PATH="${SYSROOT};${WX_BUILD}" \
    -DwxWidgets_CONFIG_EXECUTABLE="${WX_BUILD}/wx-config" \
    \
--- a/tests/scripts/setup-kicad-wasm.sh
+++ b/tests/scripts/setup-kicad-wasm.sh
@ -1,23 +1,40 @@
 #!/bin/bash
-# Copies KiCad WASM build output from Docker volume to test directory
+# Copies KiCad WASM build output to test directory
+#
+# Priority: Use local output/ directory (populated by docker/build.sh)
+# Fallback: Copy from Docker volume directly

 set -e

 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
 KICAD_TEST="$PROJECT_ROOT/tests/wasm-app/kicad"
+OUTPUT_DIR="$PROJECT_ROOT/output"

 mkdir -p "$KICAD_TEST"

-echo "Copying KiCad WASM files from Docker build..."
-docker compose -f "$PROJECT_ROOT/docker/docker-compose.yml" cp \
-  kicad-wasm-builder:/workspace/build-wasm/kicad-pcbnew/pcbnew/pcbnew.js "$KICAD_TEST/"
-docker compose -f "$PROJECT_ROOT/docker/docker-compose.yml" cp \
-  kicad-wasm-builder:/workspace/build-wasm/kicad-pcbnew/pcbnew/pcbnew.wasm "$KICAD_TEST/"
-
-# Worker file for pthreads (if exists)
-docker compose -f "$PROJECT_ROOT/docker/docker-compose.yml" cp \
-  kicad-wasm-builder:/workspace/build-wasm/kicad-pcbnew/pcbnew/pcbnew.worker.js "$KICAD_TEST/" 2>/dev/null || true
+# Check if output directory has the build files
+if [ -f "$OUTPUT_DIR/pcbnew.js" ] && [ -f "$OUTPUT_DIR/pcbnew.wasm" ]; then
+    echo "Copying KiCad WASM files from output directory..."
+    cp "$OUTPUT_DIR/pcbnew.js" "$KICAD_TEST/"
+    cp "$OUTPUT_DIR/pcbnew.wasm" "$KICAD_TEST/"
+    # Source map for debug symbols (optional)
+    cp "$OUTPUT_DIR/pcbnew.wasm.map" "$KICAD_TEST/" 2>/dev/null || true
+    # Worker file for pthreads (optional)
+    cp "$OUTPUT_DIR/pcbnew.worker.js" "$KICAD_TEST/" 2>/dev/null || true
+else
+    echo "Output directory not found, copying from Docker build..."
+    docker compose -f "$PROJECT_ROOT/docker/docker-compose.yml" cp \
+      kicad-wasm-builder:/workspace/build-wasm/kicad-pcbnew/pcbnew/pcbnew.js "$KICAD_TEST/"
+    docker compose -f "$PROJECT_ROOT/docker/docker-compose.yml" cp \
+      kicad-wasm-builder:/workspace/build-wasm/kicad-pcbnew/pcbnew/pcbnew.wasm "$KICAD_TEST/"
+    # Source map for debug symbols (optional)
+    docker compose -f "$PROJECT_ROOT/docker/docker-compose.yml" cp \
+      kicad-wasm-builder:/workspace/build-wasm/kicad-pcbnew/pcbnew/pcbnew.wasm.map "$KICAD_TEST/" 2>/dev/null || true
+    # Worker file for pthreads (optional)
+    docker compose -f "$PROJECT_ROOT/docker/docker-compose.yml" cp \
+      kicad-wasm-builder:/workspace/build-wasm/kicad-pcbnew/pcbnew/pcbnew.worker.js "$KICAD_TEST/" 2>/dev/null || true
+fi

 # wxWidgets WASM JavaScript glue code (defines JS functions called from WASM)
 echo "Copying wxWidgets WASM glue code..."