diff --git a/build.md b/build.md index 97b6050..f186b87 100644 --- a/build.md +++ b/build.md @@ -124,28 +124,49 @@ Import patterns used: | Flag | Description | |------|-------------| -| `--clean` | Full clean rebuild (all deps + wxWidgets + KiCad) | -| `--no-clean` | Incremental build (don't clean anything) | -| `--skip-deps` | Skip dependency rebuild | +| `--full` | Full clean rebuild (all deps + wxWidgets + KiCad) | +| `--clean-kicad` | Clean only KiCad build directory | +| `--build-deps` | Build dependencies (skipped by default) | | `--release` | Disable debug symbols, enable optimizations | | `--debug` | Enable debug symbols (default) | -| `-j N` | Parallel jobs (default: 1 for sequential builds) | +| `-j N` | Parallel jobs (default: all cores) | -### Clean Modes +### Build Modes -| Mode | Command | What gets cleaned | -|------|---------|-------------------| -| **Full clean** | `./docker/build.sh --clean` | All stamps, deps, wxWidgets, sysroot, KiCad | -| **Default** | `./docker/build.sh` | KiCad build only (reuses deps) | -| **Incremental** | `./docker/build.sh --no-clean` | Nothing (fastest for iteration) | +| Mode | Command | Description | +|------|---------|-------------| +| **Incremental (default)** | `./docker/build.sh` | Fastest for development (~1.5 min) | +| **Full rebuild** | `./docker/build.sh --full` | Clean everything and rebuild | +| **Rebuild KiCad** | `./docker/build.sh --clean-kicad` | Clean and rebuild KiCad only | +| **With dependencies** | `./docker/build.sh --build-deps` | Also rebuild dependencies | -**Full clean removes:** +**Full rebuild removes:** - `build-wasm/stamps/*` - All build stamps - `build-wasm/deps/*` - All dependency builds - `build-wasm/wxwidgets-universal` - wxWidgets build - `build-wasm/sysroot/*` - Installed headers/libraries - `build-wasm/kicad-pcbnew` - KiCad build +## Incremental Build System + +The build system is optimized for fast development iteration: + +### How It Works +- **ccache**: Caches compiled objects by hashing preprocessed source +- **wxWidgets**: `configure` runs once, `make` handles file-level dependencies +- **KiCad**: CMake tracks dependencies, only recompiles changed files +- **Asyncify**: Post-processing runs every build (~1 min, irreducible minimum) + +### Performance + +| Scenario | Time | +|----------|------| +| No changes | ~1.5 min | +| Single file change (KiCad or wxWidgets) | ~1.5 min | +| Full rebuild | ~10 min | + +Most time is spent on asyncify post-processing which runs on every build. + ### Debug vs Release **Debug (default):** @@ -161,7 +182,7 @@ Import patterns used: ## Stamp-based Caching -Build progress is tracked with stamp files in `build-wasm/stamps/`: +Dependency build progress is tracked with stamp files in `build-wasm/stamps/`: ``` build-wasm/stamps/ @@ -171,14 +192,15 @@ build-wasm/stamps/ ├── harfbuzz.stamp ├── pixman.stamp ├── cairo.stamp -├── wxwidgets.stamp └── kicad-pcbnew.stamp ``` +**Note:** wxWidgets and KiCad use make/CMake for incremental builds instead of stamps. + **Clear specific component:** `rm build-wasm/stamps/zstd.stamp` **Clear all stamps:** `rm -f build-wasm/stamps/*.stamp` -After changing build flags (debug/release), clear stamps to force rebuild. +After changing build flags (debug/release), use `--full` to force a complete rebuild. ## Build Scripts @@ -216,8 +238,8 @@ OpenCASCADE is the longest dependency to build (~30 minutes). - Re-run build ### Incremental build not picking up changes -- Clear KiCad stamp: `rm build-wasm/stamps/kicad-pcbnew.stamp` -- Use `--no-clean` flag to avoid full rebuild +- For KiCad: use `--clean-kicad` to force rebuild +- For wxWidgets: delete `build-wasm/wxwidgets-universal/Makefile` to force reconfigure ### WASM exception with numeric error (e.g., `3788888`) - Build with debug symbols (default): No `--release` flag diff --git a/docker/Dockerfile b/docker/Dockerfile index 57a73c1..4a66d13 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -17,8 +17,17 @@ RUN apt-get update && apt-get install -y \ protobuf-compiler \ swig \ unixodbc-dev \ + ccache \ && rm -rf /var/lib/apt/lists/* +# Configure ccache for Emscripten +# CCACHE_DIR in named volume persists across containers +ENV CCACHE_DIR=/workspace/build-wasm/.ccache +ENV CCACHE_MAXSIZE=10G +ENV CCACHE_COMPILERCHECK=content +ENV CCACHE_SLOPPINESS=include_file_mtime,time_macros,pch_defines +ENV CCACHE_BASEDIR=/workspace + WORKDIR /workspace # Entry point that sources Emscripten environment diff --git a/docs/ccache-benchmark.md b/docs/ccache-benchmark.md new file mode 100644 index 0000000..344513d --- /dev/null +++ b/docs/ccache-benchmark.md @@ -0,0 +1,63 @@ +# ccache Build Performance Benchmark + +Testing build times before and after adding ccache to measure the impact. + +## Test Environment +- Machine: Apple Silicon (M4 Max) +- Docker resources: 10 CPUs, 32GB RAM +- Build command: `./docker/build.sh` (defaults to incremental build) + +## Results + +### Step 1: Baseline (before ccache) +Command: `./docker/build.sh --skip-deps` (forces KiCad rebuild, old default) +**Time: 7:34.93** (7 min 35 sec) + +### Step 2: First build with ccache (populating cache) +Command: `./docker/build.sh --skip-deps` (old default) +**Time: 9:35.52** (9 min 35 sec) +Note: Slower than baseline due to ccache overhead when populating cache +Cache stats: 1335 misses, 0 hits, 1.39GB cached + +### Step 3: Single KiCad file change (incremental build) +Changed: `kicad/pcbnew/board.cpp` (added one line) +Command: `./docker/build.sh` (new incremental default) +**Time: 1:45.97** (1 min 46 sec) +- Only `board.cpp` recompiled +- Rest of compilation: instant (CMake detected no changes) +- Most time spent on post-processing (asyncify ~1 min) +- ccache hit rate: 25% (some preprocessed source matched) + +### Step 4: wxWidgets incremental build baseline (configure ran) +Command: `./docker/build.sh` (after implementing skip-configure logic) +**Time: 7:43.87** (7 min 44 sec) +- First build after script changes, so configure ran +- This populates the wxWidgets build state for incremental builds + +### Step 5: No-change rebuild (configure skipped) +Command: `./docker/build.sh` (no changes to any source files) +**Time: 1:31.13** (1 min 31 sec) +- wxWidgets configure skipped (Makefile exists, configure.in unchanged) +- wxWidgets make: instant (nothing to rebuild) +- KiCad CMake/make: instant (nothing changed) +- All time spent on post-processing (asyncify ~1 min) + +### Step 6: Single wxWidgets file change (incremental build) +Changed: `wxwidgets/src/common/memory.cpp` (added one line) +Command: `./docker/build.sh` +**Time: 1:33.99** (1 min 34 sec) +- wxWidgets configure skipped +- Only `memory.cpp` recompiled, library re-archived +- KiCad links against updated wxWidgets +- Most time spent on post-processing (asyncify ~1 min) + +## Summary + +| Scenario | Before | After | Speedup | +|----------|--------|-------|---------| +| Full rebuild (baseline) | 7:35 | 9:35 | -27% (cache populating) | +| KiCad single file change | 7:35 | 1:46 | **4.3x faster** | +| wxWidgets single file change | ~7:35 | 1:34 | **4.8x faster** | +| No changes | ~7:35 | 1:31 | **5x faster** | + +**Note**: Asyncify post-processing takes ~1 min and runs every build. This is the irreducible minimum build time. diff --git a/scripts/build-wxuniversal-wasm.sh b/scripts/build-wxuniversal-wasm.sh index c4fce8f..97efa2c 100755 --- a/scripts/build-wxuniversal-wasm.sh +++ b/scripts/build-wxuniversal-wasm.sh @@ -53,70 +53,90 @@ fi mkdir -p "$BUILD_DIR" cd "$BUILD_DIR" -# Configure with emconfigure -# Key flags based on wxWidgets-wasm: -# --host=emscripten Host system (detected via config.sub) -# --enable-universal Use wxUniversal (draws widgets directly) -# --disable-shared Build static libraries -# --with-opengl Enable OpenGL/WebGL support -# --enable-exceptions Enable C++ exceptions (needed for KiCad debug builds) -# --disable-richtext Not needed for KiCad, simplifies build -# --without-libtiff Avoid external dependencies -# --disable-xlocale Browser environment handles locale - -echo "" -echo "=== Configuring ===" - -# Ensure Emscripten's zlib port is built (works in Docker and on host) -# This populates the cache sysroot with zlib.h and libz.a -echo "Building Emscripten zlib port..." -embuilder build zlib - -# Get Emscripten cache sysroot path (portable across environments) -EM_CACHE_SYSROOT="$(em-config CACHE)/sysroot" -echo "Emscripten cache sysroot: $EM_CACHE_SYSROOT" - -# Set flags for Emscripten compatibility -# Z_HAVE_UNISTD_H ensures zlib includes for read/write/lseek -# Include pcre2 headers from the build directory (generated during configure) -PCRE2_INCLUDE="$BUILD_DIR/3rdparty/pcre/src" - -# Configure debug/release flags based on DEBUG_BUILD environment variable -if [ "${DEBUG_BUILD:-1}" = "1" ]; then - WX_DEBUG_FLAGS="-g -O1" - WX_CONFIGURE_DEBUG="--enable-debug" - echo "Building wxWidgets in DEBUG mode" +# Determine if we need to run configure +# Skip configure if: +# 1. Makefile exists (already configured) +# 2. configure.in hasn't changed since last configure +NEEDS_CONFIGURE=0 +if [ ! -f "$BUILD_DIR/Makefile" ]; then + echo "Not configured yet, will run configure..." + NEEDS_CONFIGURE=1 +elif [ "$WX_SOURCE/configure.in" -nt "$BUILD_DIR/Makefile" ]; then + echo "configure.in changed since last configure, will reconfigure..." + NEEDS_CONFIGURE=1 +elif [ "$WX_SOURCE/configure" -nt "$BUILD_DIR/Makefile" ]; then + echo "configure script changed, will reconfigure..." + NEEDS_CONFIGURE=1 else - WX_DEBUG_FLAGS="-O2" - WX_CONFIGURE_DEBUG="" - echo "Building wxWidgets in RELEASE mode" + echo "Already configured, skipping configure (use clean build to reconfigure)" fi -# Include emscripten cache sysroot for zlib headers -export CFLAGS="-DZ_HAVE_UNISTD_H=1 -I$EM_CACHE_SYSROOT/include ${WX_DEBUG_FLAGS} -fexceptions -pthread -matomics -mbulk-memory" -export CXXFLAGS="-DZ_HAVE_UNISTD_H=1 -I$EM_CACHE_SYSROOT/include -I$PCRE2_INCLUDE ${WX_DEBUG_FLAGS} -fexceptions -pthread -matomics -mbulk-memory" -export LDFLAGS="-L$EM_CACHE_SYSROOT/lib/wasm32-emscripten" +if [ $NEEDS_CONFIGURE -eq 1 ]; then + # Configure with emconfigure + # Key flags based on wxWidgets-wasm: + # --host=emscripten Host system (detected via config.sub) + # --enable-universal Use wxUniversal (draws widgets directly) + # --disable-shared Build static libraries + # --with-opengl Enable OpenGL/WebGL support + # --enable-exceptions Enable C++ exceptions (needed for KiCad debug builds) + # --disable-richtext Not needed for KiCad, simplifies build + # --without-libtiff Avoid external dependencies + # --disable-xlocale Browser environment handles locale -emconfigure "$WX_SOURCE/configure" \ - --host=emscripten \ - --without-subdirs \ - --enable-universal \ - --disable-shared \ - --with-opengl \ - --enable-exceptions \ - --disable-richtext \ - --without-libtiff \ - --disable-xlocale \ - --with-cxx=17 \ - --enable-utf8 \ - --with-zlib=sys \ - ${WX_CONFIGURE_DEBUG} + echo "" + echo "=== Configuring ===" -# Build PCRE first to avoid race condition with parallel builds -# PCRE headers (pcre2.h) must be generated before regex.cpp compiles -echo "" -echo "=== Building PCRE first (dependency) ===" -emmake make -C 3rdparty/pcre + # Ensure Emscripten's zlib port is built (works in Docker and on host) + # This populates the cache sysroot with zlib.h and libz.a + echo "Building Emscripten zlib port..." + embuilder build zlib + + # Get Emscripten cache sysroot path (portable across environments) + EM_CACHE_SYSROOT="$(em-config CACHE)/sysroot" + echo "Emscripten cache sysroot: $EM_CACHE_SYSROOT" + + # Set flags for Emscripten compatibility + # Z_HAVE_UNISTD_H ensures zlib includes for read/write/lseek + # Include pcre2 headers from the build directory (generated during configure) + PCRE2_INCLUDE="$BUILD_DIR/3rdparty/pcre/src" + + # Configure debug/release flags based on DEBUG_BUILD environment variable + if [ "${DEBUG_BUILD:-1}" = "1" ]; then + WX_DEBUG_FLAGS="-g -O1" + WX_CONFIGURE_DEBUG="--enable-debug" + echo "Building wxWidgets in DEBUG mode" + else + WX_DEBUG_FLAGS="-O2" + WX_CONFIGURE_DEBUG="" + echo "Building wxWidgets in RELEASE mode" + fi + + # Include emscripten cache sysroot for zlib headers + export CFLAGS="-DZ_HAVE_UNISTD_H=1 -I$EM_CACHE_SYSROOT/include ${WX_DEBUG_FLAGS} -fexceptions -pthread -matomics -mbulk-memory" + export CXXFLAGS="-DZ_HAVE_UNISTD_H=1 -I$EM_CACHE_SYSROOT/include -I$PCRE2_INCLUDE ${WX_DEBUG_FLAGS} -fexceptions -pthread -matomics -mbulk-memory" + export LDFLAGS="-L$EM_CACHE_SYSROOT/lib/wasm32-emscripten" + + emconfigure "$WX_SOURCE/configure" \ + --host=emscripten \ + --without-subdirs \ + --enable-universal \ + --disable-shared \ + --with-opengl \ + --enable-exceptions \ + --disable-richtext \ + --without-libtiff \ + --disable-xlocale \ + --with-cxx=17 \ + --enable-utf8 \ + --with-zlib=sys \ + ${WX_CONFIGURE_DEBUG} + + # Build PCRE first to avoid race condition with parallel builds + # PCRE headers (pcre2.h) must be generated before regex.cpp compiles + echo "" + echo "=== Building PCRE first (dependency) ===" + emmake make -C 3rdparty/pcre +fi # Build wxWidgets echo "" diff --git a/scripts/common/env.sh b/scripts/common/env.sh index be59199..a2575c9 100755 --- a/scripts/common/env.sh +++ b/scripts/common/env.sh @@ -34,6 +34,12 @@ export EMSDK_QUIET=1 export EMCC_CFLAGS="-fPIC -DEMSCRIPTEN" export EMCC_CXXFLAGS="-fPIC -DEMSCRIPTEN -std=c++17" +# Use ccache if available for faster rebuilds +if command -v ccache &> /dev/null; then + export CC="ccache emcc" + export CXX="ccache em++" +fi + # Debug mode (default: ON, use --release to disable) # This can be overridden by setting DEBUG_BUILD=0 before sourcing this file DEBUG_BUILD="${DEBUG_BUILD:-1}" diff --git a/scripts/common/functions.sh b/scripts/common/functions.sh index eab3ef2..36e8b76 100755 --- a/scripts/common/functions.sh +++ b/scripts/common/functions.sh @@ -193,81 +193,6 @@ remove_stamp() { rm -f "$stamp_file" } -# Compute hash of source files in a directory -# Usage: compute_source_hash /path/to/source "*.cpp" "*.h" -compute_source_hash() { - local source_dir="$1" - shift - local patterns=("$@") - - # Build find command for all patterns - local find_args=() - for pattern in "${patterns[@]}"; do - if [ ${#find_args[@]} -gt 0 ]; then - find_args+=("-o") - fi - find_args+=("-name" "$pattern") - done - - # Hash all matching files (sorted for consistency) - # Use md5 on macOS, md5sum on Linux (Docker) - if command -v md5 &>/dev/null; then - find "$source_dir" -type f \( "${find_args[@]}" \) 2>/dev/null | \ - sort | \ - xargs cat 2>/dev/null | \ - md5 - else - find "$source_dir" -type f \( "${find_args[@]}" \) 2>/dev/null | \ - sort | \ - xargs cat 2>/dev/null | \ - md5sum | \ - cut -d' ' -f1 - fi -} - -# Create stamp with source hash -# Usage: create_source_stamp "wxwidgets" /path/to/source "*.cpp" "*.h" -create_source_stamp() { - local name="$1" - local source_dir="$2" - shift 2 - local patterns=("$@") - - local stamp_dir="${BUILD_ROOT:-$PROJECT_ROOT/build-wasm}/stamps" - mkdir -p "$stamp_dir" - local stamp_file="$stamp_dir/$name.stamp" - - local hash=$(compute_source_hash "$source_dir" "${patterns[@]}") - echo "$hash" > "$stamp_file" - log_info "Created stamp: $name (hash: ${hash:0:8}...)" -} - -# Check if source stamp is still valid -# Usage: check_source_stamp "wxwidgets" /path/to/source "*.cpp" "*.h" -# Returns: 0 if valid (no rebuild needed), 1 if invalid (rebuild needed) -check_source_stamp() { - local name="$1" - local source_dir="$2" - shift 2 - local patterns=("$@") - - local stamp_file="${BUILD_ROOT:-$PROJECT_ROOT/build-wasm}/stamps/$name.stamp" - - if [ ! -f "$stamp_file" ]; then - return 1 # No stamp, need build - fi - - local stored_hash=$(cat "$stamp_file") - local current_hash=$(compute_source_hash "$source_dir" "${patterns[@]}") - - if [ "$stored_hash" = "$current_hash" ]; then - return 0 # Up to date - else - log_info "Source changed for $name (${stored_hash:0:8}... -> ${current_hash:0:8}...)" - return 1 # Changed, need rebuild - fi -} - # Build if stamp doesn't exist build_if_needed() { local name="$1" diff --git a/scripts/kicad/build-pcbnew.sh b/scripts/kicad/build-pcbnew.sh index 641e03a..4b8c583 100755 --- a/scripts/kicad/build-pcbnew.sh +++ b/scripts/kicad/build-pcbnew.sh @@ -6,17 +6,22 @@ # ./scripts/kicad/build-pcbnew.sh [options] # # Options: -# --clean Full clean rebuild (dependencies + KiCad) -# --no-clean Skip cleaning the build directory (default: clean KiCad only) -# --skip-deps Skip building dependencies +# --full Full clean rebuild (dependencies + KiCad) +# --clean-kicad Clean only KiCad build directory (not deps) +# --build-deps Build dependencies (default: skip) # --debug Build with debug symbols (default) # --release Build optimized without debug symbols # -j N Parallel compilation jobs (default: 1) # -# Stamp System: -# Dependencies use source-hash stamps to detect when rebuilds are needed. -# wxWidgets rebuilds automatically when source files (*.cpp, *.h, *.c) change. -# To force a rebuild, delete the stamp: rm -f build-wasm/stamps/wxwidgets.stamp +# Defaults (optimized for development): +# - Incremental build (no clean) +# - Skip dependencies +# - ccache enabled for faster rebuilds +# +# Incremental Build System: +# - wxWidgets: configure runs once, make handles file-level dependencies +# - KiCad: CMake tracks dependencies, only recompiles changed files +# - ccache: Caches compiled objects for faster rebuilds set -e @@ -31,23 +36,25 @@ KICAD_STAMP="${BUILD_ROOT}/stamps/kicad-pcbnew.stamp" WASM_LAYER="${PROJECT_ROOT}/wasm" WX_BUILD="${BUILD_ROOT}/wxwidgets-universal" -# Parse arguments - clean KiCad by default -NO_CLEAN=0 +# Parse arguments - incremental build by default (optimized for development) +NO_CLEAN=1 FULL_CLEAN=0 -SKIP_DEPS=0 +SKIP_DEPS=1 DEBUG=0 while [[ $# -gt 0 ]]; do case $1 in - --clean) + --full) FULL_CLEAN=1 + NO_CLEAN=0 + SKIP_DEPS=0 shift ;; - --no-clean) - NO_CLEAN=1 + --clean-kicad) + NO_CLEAN=0 shift ;; - --skip-deps) - SKIP_DEPS=1 + --build-deps) + SKIP_DEPS=0 shift ;; --debug) @@ -88,7 +95,7 @@ elif [ $NO_CLEAN -eq 0 ]; then log_info "Cleaning KiCad PCBnew build directory..." rm -rf "${KICAD_BUILD}" "${KICAD_STAMP}" else - log_info "Skipping clean (--no-clean specified)" + log_info "Incremental build (use --clean-kicad or --full to clean)" fi # Step 2: Build dependencies @@ -97,23 +104,21 @@ if [ $SKIP_DEPS -eq 0 ]; then log_info "Building dependencies..." "${SCRIPT_DIR}/../deps/build-all-deps.sh" --with-occ else - log_info "Skipping dependencies (--skip-deps specified)" + log_info "Skipping dependencies (use --build-deps or --full to build)" fi -# Step 3: Check if already built (only relevant with --no-clean) -if [ $NO_CLEAN -eq 1 ] && check_stamp "${KICAD_STAMP}"; then - log_info "KiCad PCBnew already built, skipping..." - exit 0 -fi +# Note: We don't check the KiCad stamp here for incremental builds. +# CMake handles dependency tracking - it will detect changed source files +# and only recompile what's needed. The stamp is created at the end for +# scripts that want to know if KiCad was ever built successfully. -# Step 4: Build wxWidgets if source changed or not present -WX_SOURCE="${PROJECT_ROOT}/wxwidgets/src" -if [ ! -f "${WX_BUILD}/lib/libwx_baseu-3.2.a" ] || \ - ! check_source_stamp "wxwidgets" "$WX_SOURCE" "*.cpp" "*.h" "*.c"; then - log_info "Building wxWidgets..." - "${SCRIPT_DIR}/../build-wxuniversal-wasm.sh" --no-clean - create_source_stamp "wxwidgets" "$WX_SOURCE" "*.cpp" "*.h" "*.c" -fi +# Step 4: Build wxWidgets (incremental - only recompiles changed files) +# The wxWidgets build script handles: +# - Skipping configure if already configured +# - make handles per-file dependency tracking +# - ccache handles compilation caching +log_info "Building wxWidgets..." +"${SCRIPT_DIR}/../build-wxuniversal-wasm.sh" --no-clean log_info "Building KiCad PCBnew ${KICAD_VERSION} for WASM..." @@ -218,7 +223,16 @@ log_info "KiCad WASM support verified" # Step 7: Configure KiCad with CMake # We use CMAKE_MODULE_PATH to inject our compatibility layer log_info "Configuring KiCad with CMake..." + +# Use ccache if available (CMAKE_*_COMPILER_LAUNCHER is the proper CMake way) +CCACHE_OPTS="" +if command -v ccache &> /dev/null; then + CCACHE_OPTS="-DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache" + log_info "Using ccache for compilation" +fi + emcmake cmake "${KICAD_DIR}" \ + ${CCACHE_OPTS} \ -DCMAKE_BUILD_TYPE=${BUILD_TYPE} \ -DCMAKE_INSTALL_PREFIX="${SYSROOT}" \ -DCMAKE_MODULE_PATH="${WASM_LAYER}/cmake" \