bench: wasm-opt benchmark harness — Hetzner config sweeps + local QEMU VM

Tooling that produced the perf findings (committed for rerunnability): - scripts/bench/wasm-opt-bench.sh + o2-config-sweep.sh + sweep.conf: replay asyncify/-O2 over a cached fixture under allocator/THP/core matrices on the Hetzner runner. - scripts/bench/setup-vm.sh + cloud-init/ + vm-build.sh: local QEMU (HVF) Ubuntu guest with Docker CE to verify Linux builds without burning paid runners. - .gitignore: bench fixtures/results and the VM image stay local. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 07:47:06 +02:00 · 2026-06-11 07:47:06 +02:00 · df25275843
commit df25275843
parent 680eb9dc15
9 changed files with 657 additions and 0 deletions
--- a/scripts/bench/README.md
+++ b/scripts/bench/README.md
@ -0,0 +1,104 @@
+# wasm-opt allocator/core benchmark
+
+Fast, local feedback loop for the CI perf issue: the host-side `wasm-opt`/asyncify
+pass (`scripts/common/apply-asyncify.sh`) is slow on the glibc Linux CI runner
+because glibc `malloc` collapses into per-arena lock (`futex`) contention under
+many threads. We preload **jemalloc** to fix it. This harness measures the effect
+locally instead of paying for a full ~160-min Hetzner CI run per experiment.
+
+**Key idea:** `wasm-opt`/asyncify is a standalone pass over an *already-compiled*
+`.wasm` (`docker/build.sh:194`). So we build the eeschema `.wasm` **once** on the
+Mac, copy it into a Linux VM, and replay only the optimizer across a matrix of
+`{glibc, jemalloc} × core counts`. No KiCad compile happens in the VM.
+
+## Caveats (read before trusting numbers)
+
+- **This Mac is 10 cores / 32 GB.** The local core sweep tops out at ~10 threads.
+  The CI runner is 32 vCPU, so the *full* 32-thread contention is **not**
+  reproducible here — the local run shows the **direction** (jemalloc vs glibc,
+  and how wall-clock scales with cores), not CI's absolute worst case. The final
+  32-thread pick must still be confirmed on the real runner.
+- **aarch64 ≠ CI's x86_64.** The VM is aarch64 (HVF, near-native speed). The
+  glibc arena-lock pathology is arch-independent, so the **ratios** transfer;
+  **absolute seconds do not** match the x86_64 AMD runner.
+- asyncify peaks ~10–15 GB RAM → the VM gets 20 GB; close other heavy apps.
+
+## 1. Build the fixture (once, on the Mac)
+
+```bash
+mkdir -p bench
+./docker/build.sh eeschema --build-deps          # long: full cold build
+# Pull the Docker-side raw (pre-finalize) wasm while the builder container is up:
+docker compose -f docker/docker-compose.yml cp \
+  kicad-wasm-builder:/workspace/build-wasm/kicad-eeschema/eeschema/eeschema.wasm \
+  bench/eeschema.raw.wasm
+# Finalize it to match what asyncify actually consumes in the pipeline:
+./scripts/common/apply-finalize.sh bench/eeschema.raw.wasm bench/eeschema.finalized.wasm
+```
+
+Fallback if `compose cp` fails:
+`docker compose -f docker/docker-compose.yml exec kicad-wasm-builder cat <path> > bench/eeschema.raw.wasm`
+
+## 2. Provision and boot the VM (Mac)
+
+```bash
+brew install qemu                 # one-time
+./scripts/bench/setup-vm.sh prepare
+./scripts/bench/setup-vm.sh run   # serial console; quit with Ctrl-a x
+```
+
+Wait ~1–2 min for cloud-init (installs git/curl/time/libjemalloc2/strace).
+
+## 3. Load repo + fixture into the VM
+
+```bash
+# from the Mac, in another terminal:
+scp -P 2222 -o StrictHostKeyChecking=no bench/eeschema.finalized.wasm bench@localhost:~/
+./scripts/bench/setup-vm.sh ssh
+# inside the VM:
+git clone <this-repo-url> repo && cd repo
+git checkout istvanmatejcsok/feat/ci-hetzner-allcores
+mkdir -p bench && mv ~/eeschema.finalized.wasm bench/
+# get-wasm-opt.sh auto-downloads Binaryen v121 aarch64-linux on first use
+```
+
+## 4. Run the benchmark (in the VM)
+
+```bash
+STRACE=1 ./scripts/bench/wasm-opt-bench.sh
+```
+
+Sweeps `CORES="1 4 8 10"` × `{glibc, jemalloc}`, writing `bench/results.csv`
+(wall-clock + peak RSS per cell) and per-cell logs under `bench/results/`. With
+`STRACE=1` it also records `futex` syscall share per allocator (expect ~99% on
+glibc, far lower with jemalloc). Override the sweep with e.g. `CORES="8 10"`.
+
+## Interpreting
+
+- jemalloc rows should show **lower wall-clock** than glibc, widening as cores rise.
+- glibc wall-clock that *stops improving* (or worsens) with more cores = the
+  arena-lock storm; jemalloc should keep scaling.
+- These ratios justify the `LD_PRELOAD` fix; use them (plus one real-runner
+  confirmation) to choose CI's `BINARYEN_CORES`.
+
+Artifacts (`bench/*.wasm`, `bench/results*`, `scripts/bench/vm/`) are gitignored.
+
+## 5. Full Docker build in the VM (CI dry-run) — vm-build.sh
+
+Verifies CI orchestration changes (docker/build.sh, compose limits, pipelining)
+on Linux+Docker without burning a Hetzner slot. The guest is aarch64/HVF:
+a *functional* CI proxy, not an x86 performance proxy.
+
+```bash
+# one-time: bigger disk + Docker-enabled cloud-init, then boot
+VM_DISK=80G ./scripts/bench/setup-vm.sh prepare
+./scripts/bench/setup-vm.sh run            # leave running in its own terminal
+
+# from the Mac: cold calculator build inside the guest (deps + docker image)
+./scripts/bench/vm-build.sh                # = calculator --build-deps
+
+# pipeline smoke test (deps already in the guest volume from the previous run)
+KICAD_PIPELINE=1 ./scripts/bench/vm-build.sh calculator,pl_editor
+```
+
+Prints the guest build wall time at the end; build logs stream through ssh.
--- a/scripts/bench/cloud-init/meta-data
+++ b/scripts/bench/cloud-init/meta-data
@ -0,0 +1,2 @@
+instance-id: kicad-wasmopt-bench
+local-hostname: kicad-bench
--- a/scripts/bench/cloud-init/user-data
+++ b/scripts/bench/cloud-init/user-data
@ -0,0 +1,40 @@
+#cloud-config
+# NoCloud seed for the wasm-opt benchmark VM. setup-vm.sh substitutes
+# __SSH_PUBKEY__ with your ~/.ssh/id_ed25519.pub before building the seed ISO.
+hostname: kicad-bench
+users:
+  - name: bench
+    sudo: "ALL=(ALL) NOPASSWD:ALL"
+    shell: /bin/bash
+    lock_passwd: false
+    # Fallback console/SSH password (key auth is preferred). Change if you care.
+    plain_text_passwd: bench
+    ssh_authorized_keys:
+      - __SSH_PUBKEY__
+ssh_pwauth: true
+package_update: true
+# Docker CE from the official repo (mirrors the CI workflow's install step) so
+# scripts/bench/vm-build.sh can run the full docker/build.sh pipeline in-guest.
+apt:
+  sources:
+    docker.list:
+      source: "deb [signed-by=$KEY_FILE] https://download.docker.com/linux/ubuntu noble stable"
+      keyid: 9DC858229FC7DD38854AE2D88D81803C0EBFCD88
+packages:
+  - git
+  - curl
+  - ca-certificates
+  - time
+  - libjemalloc2
+  - strace
+  - rsync
+  - docker-ce
+  - docker-ce-cli
+  - containerd.io
+  - docker-buildx-plugin
+  - docker-compose-plugin
+runcmd:
+  # The docker group only exists after the package installs, so the user's
+  # `groups:` stanza can't grant it — add membership here (applies to new
+  # ssh sessions, which is all vm-build.sh uses).
+  - usermod -aG docker bench
--- a/scripts/bench/o2-config-sweep.sh
+++ b/scripts/bench/o2-config-sweep.sh
@ -0,0 +1,166 @@
+#!/bin/bash
+# o2-config-sweep.sh — replay wasm-opt over a PREBUILT asyncified fixture under a
+# matrix of {Binaryen version, optimization passes, thread count, allocator/THP}.
+# RUNS ON THE LINUX CI BOX.
+#
+# WHY: the on-box diagnostic (docs/ci-build-slowness-findings.md) proved the
+# ~80-min `-O2` cost is ~90% FUTEX LOCK CONTENTION inside wasm-opt (Binaryen's
+# global type mutex), NOT the allocator and NOT memory management (madvise=0,
+# THP compaction=0, identical under glibc and mimalloc). So the levers that can
+# move wall-clock are: a NEWER Binaryen (the devs cut this contention after our
+# pinned v121) and FEWER threads (less lock contention) — plus reducing the -O2
+# work itself (lighter passes). Allocator/THP are dead ends, kept only as controls.
+#
+# CONFIG NAME GRAMMAR:  <preset>[@<cores>]
+#   preset sets Binaryen version + passes + allocator/THP (see config_env).
+#   optional @<cores> overrides BINARYEN_CORES for that one cell (sweep threads).
+#   e.g. "v130-O2@8" = Binaryen 130, -O2, 8 threads.
+#
+# TWO MODES:
+#   CAP_SECONDS=0 (default): run to completion → true wall-clock + output size.
+#   CAP_SECONDS>0: WINDOWED sample — the lock storm is steady-state, so measure
+#     it over a [60s, CAP-30s] window and kill the pass. Reports, per cell:
+#       win_sysfrac   = system CPU fraction in the window (lock contention; LOWER better)
+#       win_usercores = REAL-work cores in the window (progress rate; HIGHER better
+#                       → predicts shorter wall for the same pass set)
+#       win_tlb       = TLB-shootdown interrupts in the window
+#     ~CAP/60 min per cell, so the whole matrix fits one CI run.
+#
+# Usage: CONFIGS="baseline v130-O2 baseline@8" CAP_SECONDS=600 \
+#          ./scripts/bench/o2-config-sweep.sh <asyncified.wasm>
+# Env: CONFIGS, CORES (default nproc), CAP_SECONDS (default 0), DIAGNOSTIC (1 =
+#      perf kernel-symbol sample, full mode). Output under /bench/o2-results/.
+set -uo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO="$(cd "${SCRIPT_DIR}/../.." && pwd)"
+
+FIXTURE="${1:?usage: o2-config-sweep.sh <asyncified-fixture.wasm>}"
+CONFIGS="${CONFIGS:-baseline}"
+CORES_DEFAULT="${CORES:-$(nproc)}"
+CAP_SECONDS="${CAP_SECONDS:-0}"
+DIAGNOSTIC="${DIAGNOSTIC:-0}"
+
+[[ "$(uname -s)" == "Linux" ]] || { echo "ERROR: run on the Linux CI box." >&2; exit 1; }
+[[ -f "${FIXTURE}" ]] || { echo "ERROR: fixture not found: ${FIXTURE}" >&2; exit 1; }
+command -v /usr/bin/time >/dev/null || { echo "ERROR: apt-get install -y time" >&2; exit 1; }
+
+ARCH="$(uname -m)"
+WASM_OPT_121="$("${REPO}/scripts/common/get-wasm-opt.sh" 2>/dev/null)"   # pinned v121
+JEMALLOC="$(ls /usr/lib/${ARCH}-linux-gnu/libjemalloc.so.2 2>/dev/null | head -1 || true)"
+MIMALLOC="$(ls /usr/lib/${ARCH}-linux-gnu/libmimalloc.so* 2>/dev/null | sort | tail -1 || true)"
+
+OUTDIR="${REPO}/bench/o2-results"; mkdir -p "${OUTDIR}"
+CSV="${OUTDIR}/results.csv"
+echo "config,cores,ver,flags,thp,mode,wall,wall_s,user_s,sys_s,vol_ctxsw,win_sysfrac,win_usercores,win_tlb,madvise,out_bytes,preload" > "${CSV}"
+
+sudo sysctl -w kernel.perf_event_paranoid=-1 >/dev/null 2>&1 || true
+PERF=""; command -v perf >/dev/null 2>&1 && perf stat -e syscalls:sys_enter_madvise -- true >/dev/null 2>&1 && PERF="yes"
+echo "perf syscall counting: ${PERF:-NO}"
+CLK="$(getconf CLK_TCK 2>/dev/null || echo 100)"
+
+# Resolve (download+cache) a standalone wasm-opt for a Binaryen version. 121 = the
+# pinned one from get-wasm-opt.sh; anything else is fetched from GitHub releases.
+get_wasm_opt_ver() {
+    local ver="$1"
+    if [[ -z "$ver" || "$ver" == "121" ]]; then echo "${WASM_OPT_121}"; return 0; fi
+    local dir="${REPO}/build-wasm/tools/binaryen-${ver}" bin="${REPO}/build-wasm/tools/binaryen-${ver}/bin/wasm-opt"
+    if [[ ! -x "$bin" ]]; then
+        local url="https://github.com/WebAssembly/binaryen/releases/download/version_${ver}/binaryen-version_${ver}-${ARCH}-linux.tar.gz"
+        mkdir -p "${REPO}/build-wasm/tools"
+        echo "  downloading Binaryen v${ver}..." >&2
+        curl -fsSL -o "/tmp/binaryen-${ver}.tgz" "$url" || { echo "  ERR: download v${ver} failed ($url)" >&2; return 1; }
+        tar -xzf "/tmp/binaryen-${ver}.tgz" -C "${REPO}/build-wasm/tools" || return 1
+        mv "${REPO}/build-wasm/tools/binaryen-version_${ver}" "$dir" 2>/dev/null || true
+    fi
+    [[ -x "$bin" ]] && echo "$bin" || return 1
+}
+
+snap_cpu() { awk '/^cpu /{u=$2;s=$4;t=0;for(i=2;i<=NF;i++)t+=$i;print u,s,t}' /proc/stat; }
+snap_tlb() { awk '/TLB/{for(i=2;i<=NF;i++)if($i~/^[0-9]+$/)s+=$i}END{print s+0}' /proc/interrupts; }
+snap_vm()  { awk '/^compact_stall /{c=$2}/^thp_fault_alloc /{t=$2}END{print c+0,t+0}' /proc/vmstat; }
+thp_state(){ grep -oP '\[\K[^]]+' /sys/kernel/mm/transparent_hugepage/enabled 2>/dev/null || echo "?"; }
+set_thp()  { [[ "$1" == "asis" ]] && return 0
+    echo "$1"|sudo tee /sys/kernel/mm/transparent_hugepage/enabled >/dev/null 2>&1 || true
+    echo "$1"|sudo tee /sys/kernel/mm/transparent_hugepage/defrag  >/dev/null 2>&1 || true; }
+
+# preset -> BIN_VER, OPTARGS (pass list), PRELOAD, THP_WANT, EXTRA env
+BIN_VER=""; OPTARGS=(); PRELOAD=""; THP_WANT="asis"; EXTRA=()
+LIGHT=(--flatten --simplify-locals --coalesce-locals --reorder-locals --vacuum)
+config_env() {
+    BIN_VER="121"; OPTARGS=(-O2); PRELOAD=""; THP_WANT="asis"; EXTRA=()
+    case "$1" in
+        baseline)        ;;                                   # v121 -O2 (current CI = control)
+        v130-O2)         BIN_VER="130" ;;                     # newer Binaryen, same passes — lock fixed?
+        v130-O1)         BIN_VER="130"; OPTARGS=(-O1) ;;
+        v121-O1)         OPTARGS=(-O1) ;;                      # lighter passes (less work)
+        v130-light)      BIN_VER="130"; OPTARGS=("${LIGHT[@]}") ;;
+        v121-light)      OPTARGS=("${LIGHT[@]}") ;;
+        mimalloc-retain) PRELOAD="${MIMALLOC}"; EXTRA=(MIMALLOC_PURGE_DELAY=-1 MIMALLOC_ALLOW_THP=0) ;;  # control (proven dead)
+        thp-off)         THP_WANT="never" ;;                  # control (proven dead)
+        *) echo "ERROR: unknown preset '$1'" >&2; return 1 ;;
+    esac
+    if [[ -n "${PRELOAD}" && ! -e "${PRELOAD}" ]]; then echo "WARN: preload for '$1' missing — SKIP" >&2; return 2; fi
+}
+
+field() { grep -F "$1" "$2" 2>/dev/null | tail -1 | sed 's/.*: //' | tr -d ' '; }
+
+run_config() {
+    local cfg="$1" base cores
+    base="${cfg%@*}"; if [[ "$cfg" == *"@"* ]]; then cores="${cfg##*@}"; else cores="${CORES_DEFAULT}"; fi
+    if ! config_env "${base}"; then echo "${cfg},${cores},,,skip,skip,SKIPPED,,,,,,,,,," >> "${CSV}"; return 0; fi
+
+    local wopt; wopt="$(get_wasm_opt_ver "${BIN_VER}")" || {
+        echo "  cannot resolve Binaryen v${BIN_VER} — SKIP"; echo "${cfg},${cores},${BIN_VER},,skip,skip,SKIP-NOBIN,,,,,,,,,," >> "${CSV}"; return 0; }
+    set_thp "${THP_WANT}"; local thp_now; thp_now="$(thp_state)"
+    local flags="${OPTARGS[*]}"
+    local timef="${OUTDIR}/${cfg//[@ ]/_}.time" statf="${OUTDIR}/${cfg//[@ ]/_}.perfstat" logf="${OUTDIR}/${cfg//[@ ]/_}.log"
+    local outw="/tmp/o2-out.wasm"
+    cp "${FIXTURE}" /tmp/o2-in.wasm
+    local -a runenv=(BINARYEN_CORES="${cores}" "${EXTRA[@]}"); [[ -n "${PRELOAD}" ]] && runenv+=("LD_PRELOAD=${PRELOAD}")
+
+    echo ""; echo "=== ${cfg}  (binaryen=$("${wopt}" --version 2>&1 | grep -oE '[0-9]+' | head -1), flags='${flags}', cores=${cores}, THP=${thp_now}, preload=${PRELOAD:-none}, extra=${EXTRA[*]:-none}) ==="
+
+    local mode="full" wall wall_s user sys volcsw sysf usercores tlbd madv outsz
+    sysf=NA; usercores=NA; tlbd=NA; madv=NA
+
+    if [[ "${CAP_SECONDS}" -gt 0 ]]; then
+        mode="cap${CAP_SECONDS}"
+        ( /usr/bin/time -v -o "${timef}" timeout "${CAP_SECONDS}" env "${runenv[@]}" "${wopt}" "${OPTARGS[@]}" /tmp/o2-in.wasm -o "${outw}" > "${logf}" 2>&1 ) &
+        local rp=$!
+        sleep 60
+        local cu0 cs0 ct0 tlb0; read -r cu0 cs0 ct0 <<<"$(snap_cpu)"; tlb0="$(snap_tlb)"
+        local win=$(( CAP_SECONDS>120 ? CAP_SECONDS-90 : 30 )); sleep "${win}"
+        local cu1 cs1 ct1 tlb1; read -r cu1 cs1 ct1 <<<"$(snap_cpu)"; tlb1="$(snap_tlb)"
+        wait "${rp}" 2>/dev/null || true
+        local dtot=$((ct1-ct0)) dsys=$((cs1-cs0)) duser=$((cu1-cu0))
+        sysf="$(awk -v s=${dsys} -v t=${dtot} 'BEGIN{print (t>0)?sprintf("%.2f",s/t):"NA"}')"
+        usercores="$(awk -v u=${duser} -v c=${CLK} -v w=${win} 'BEGIN{print (w>0)?sprintf("%.2f",(u/c)/w):"NA"}')"
+        tlbd=$((tlb1-tlb0)); wall="cap@${CAP_SECONDS}"; wall_s="${CAP_SECONDS}"
+    else
+        if [[ -n "${PERF}" ]]; then
+            /usr/bin/time -v -o "${timef}" perf stat -o "${statf}" -e syscalls:sys_enter_madvise,syscalls:sys_enter_munmap \
+                env "${runenv[@]}" "${wopt}" "${OPTARGS[@]}" /tmp/o2-in.wasm -o "${outw}" > "${logf}" 2>&1
+            madv="$(grep -E 'sys_enter_madvise' "${statf}" 2>/dev/null | awk '{gsub(/,/,"",$1);print $1}' | head -1)"
+        else
+            /usr/bin/time -v -o "${timef}" env "${runenv[@]}" "${wopt}" "${OPTARGS[@]}" /tmp/o2-in.wasm -o "${outw}" > "${logf}" 2>&1
+        fi
+        if [[ $? -ne 0 ]]; then echo "  FAILED"; tail -4 "${logf}"; echo "${cfg},${cores},${BIN_VER},${flags// /+},${thp_now},full,FAILED,,,,,,,,,," >> "${CSV}"; return 0; fi
+        wall="$(field 'Elapsed (wall clock) time' "${timef}")"
+        wall_s="$(awk -F: '{if(NF==3)print $1*3600+$2*60+$3;else if(NF==2)print $1*60+$2;else print $1}' <<<"${wall}")"
+    fi
+
+    user="$(field 'User time (seconds)' "${timef}")"; sys="$(field 'System time (seconds)' "${timef}")"
+    volcsw="$(field 'Voluntary context switches' "${timef}")"; outsz="$(stat -c %s "${outw}" 2>/dev/null || echo NA)"
+    printf '%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s\n' \
+        "${cfg}" "${cores}" "${BIN_VER}" "${flags// /+}" "${thp_now}" "${mode}" "${wall}" "${wall_s}" \
+        "${user}" "${sys}" "${volcsw}" "${sysf}" "${usercores}" "${tlbd}" "${madv}" "${outsz}" "${PRELOAD:-none}" >> "${CSV}"
+    echo "  mode=${mode} wall=${wall} user=${user}s sys=${sys}s vol_ctxsw=${volcsw} out=${outsz}B"
+    echo "  WINDOW sys_frac=${sysf} (lock contention)  user_cores=${usercores} (real-work rate)  tlb=${tlbd}  madvise=${madv}"
+    rm -f "${outw}" /tmp/o2-in.wasm
+}
+
+echo "Fixture: ${FIXTURE} ($(stat -c %s "${FIXTURE}" 2>/dev/null) bytes)"
+echo "Configs: ${CONFIGS}  CORES_DEFAULT=${CORES_DEFAULT}  CAP_SECONDS=${CAP_SECONDS}  THP(init)=$(thp_state)"
+for cfg in ${CONFIGS}; do run_config "${cfg}"; done
+echo ""; echo "=== results (${CSV}) ==="; column -t -s, "${CSV}" || cat "${CSV}"
--- a/scripts/bench/setup-vm.sh
+++ b/scripts/bench/setup-vm.sh
@ -0,0 +1,92 @@
+#!/bin/bash
+# Provision a local QEMU aarch64 + HVF Ubuntu 24.04 VM for the wasm-opt benchmark.
+# RUNS ON THE macOS HOST (Apple Silicon). See scripts/bench/README.md.
+#
+# Subcommands:
+#   prepare   download cloud image, build NoCloud seed ISO, copy UEFI vars
+#   run       boot the VM (foreground, serial console; ssh on :2222)
+#   ssh       ssh into the running VM
+#   (none)    = prepare (if needed) then run
+#
+# Env overrides: VM_CORES (default 10), VM_MEM (default 20G), SSH_PORT (2222),
+# VM_DISK (default +30G grow; use VM_DISK=80G for full Docker builds via
+# scripts/bench/vm-build.sh — emsdk image + deps + build tree need ~50 GB).
+
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+VMDIR="${SCRIPT_DIR}/vm"
+CLOUD_INIT="${SCRIPT_DIR}/cloud-init"
+
+VM_CORES="${VM_CORES:-10}"          # Mac has 10 cores; this caps the local sweep
+VM_MEM="${VM_MEM:-20G}"             # asyncify peaks ~10-15 GB; leave headroom for macOS
+VM_DISK="${VM_DISK:-30G}"           # disk grow beyond the ~3.5 GB cloud image
+SSH_PORT="${SSH_PORT:-2222}"
+IMG_URL="https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-arm64.img"
+
+QEMU_SHARE="$(brew --prefix qemu)/share/qemu"
+FW_CODE="${QEMU_SHARE}/edk2-aarch64-code.fd"
+FW_VARS_SRC="${QEMU_SHARE}/edk2-arm-vars.fd"
+
+BASE_IMG="${VMDIR}/ubuntu-24.04-arm64.img"
+DISK="${VMDIR}/disk.qcow2"
+SEED="${VMDIR}/seed.iso"
+FW_VARS="${VMDIR}/edk2-vars.fd"
+PUBKEY="${HOME}/.ssh/id_ed25519.pub"
+
+prepare() {
+    mkdir -p "${VMDIR}"
+    command -v qemu-system-aarch64 >/dev/null || { echo "Install qemu: brew install qemu" >&2; exit 1; }
+    [[ -f "${PUBKEY}" ]] || { echo "No SSH pubkey at ${PUBKEY}" >&2; exit 1; }
+
+    if [[ ! -f "${BASE_IMG}" ]]; then
+        echo "Downloading Ubuntu 24.04 arm64 cloud image..."
+        curl -fL -o "${BASE_IMG}" "${IMG_URL}"
+    fi
+
+    # Fresh working disk each prepare: copy the cloud image and grow it (the
+    # cloud image is ~3.5 GB; binaryen + fixture + scratch need more headroom).
+    echo "Creating working disk (${DISK}, +${VM_DISK})..."
+    cp "${BASE_IMG}" "${DISK}"
+    qemu-img resize "${DISK}" "+${VM_DISK}"
+
+    # Writable UEFI vars store (copy of the template).
+    cp "${FW_VARS_SRC}" "${FW_VARS}"
+
+    # Build the NoCloud seed ISO (label must be CIDATA), injecting the pubkey.
+    echo "Building cloud-init seed ISO..."
+    local tmp; tmp="$(mktemp -d)"
+    sed "s|__SSH_PUBKEY__|$(cat "${PUBKEY}")|" "${CLOUD_INIT}/user-data" > "${tmp}/user-data"
+    cp "${CLOUD_INIT}/meta-data" "${tmp}/meta-data"
+    rm -f "${SEED}"
+    hdiutil makehybrid -iso -joliet -default-volume-name CIDATA -o "${SEED}" "${tmp}" >/dev/null
+    rm -rf "${tmp}"
+    echo "Prepared. Boot with: $0 run"
+}
+
+run() {
+    [[ -f "${DISK}" && -f "${SEED}" && -f "${FW_VARS}" ]] || { echo "Run '$0 prepare' first." >&2; exit 1; }
+    echo "Booting VM: ${VM_CORES} vCPU, ${VM_MEM} RAM, ssh -> localhost:${SSH_PORT} (user: bench)"
+    echo "First boot runs cloud-init (installs packages); wait ~1-2 min before ssh."
+    echo "Quit the serial console with: Ctrl-a x"
+    exec qemu-system-aarch64 \
+        -machine virt -accel hvf -cpu host \
+        -smp "${VM_CORES}" -m "${VM_MEM}" \
+        -drive "if=pflash,format=raw,readonly=on,file=${FW_CODE}" \
+        -drive "if=pflash,format=raw,file=${FW_VARS}" \
+        -drive "if=virtio,format=qcow2,file=${DISK}" \
+        -drive "if=virtio,format=raw,file=${SEED}" \
+        -netdev "user,id=n0,hostfwd=tcp::${SSH_PORT}-:22" \
+        -device virtio-net,netdev=n0 \
+        -nographic
+}
+
+do_ssh() { exec ssh -p "${SSH_PORT}" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null bench@localhost "$@"; }
+
+case "${1:-}" in
+    prepare) prepare ;;
+    run)     run ;;
+    ssh)     shift; do_ssh "$@" ;;
+    "")      [[ -f "${DISK}" && -f "${SEED}" ]] || prepare; run ;;
+    *)       echo "Usage: $0 {prepare|run|ssh}" >&2; exit 1 ;;
+esac
--- a/scripts/bench/sweep.conf
+++ b/scripts/bench/sweep.conf
@ -0,0 +1,33 @@
+# Parameters for the wasm-opt config sweep (.github/workflows/wasm-opt-bench.yml).
+# Edit, commit, push to a bench/** branch to launch a run.
+#
+# Config grammar: <preset>[@<cores>]  (see scripts/bench/o2-config-sweep.sh)
+#   presets: baseline(v121 -O2) v130-O2 v130-O1 v121-O1 v130-light v121-light
+#            mimalloc-retain thp-off (last two are proven-dead controls)
+#   @<cores> overrides BINARYEN_CORES for that cell (thread sweep).
+# CAP_SECONDS_CONF: >0 = windowed storm sample (~CAP/60 min/cell, whole matrix in
+#   one run); 0 = run each to completion (true wall-clock). FIXTURE_RUN_ID_CONF:
+#   reuse a prior run's cached fixture (blank = build it ~40 min).
+#
+# VERDICT from run #1 (27197360957): the -O2 cost is ~90% FUTEX LOCK CONTENTION
+# in Binaryen (type mutex), NOT allocator/THP (madvise=0, compaction=0, identical
+# under glibc & mimalloc; mimalloc-retain saved only 8%). So allocator/THP are
+# dead; the levers are NEWER Binaryen (devs fixed this after v121) and FEWER
+# threads, plus lighter passes. Run #2 triages those windowed (fast):
+#   win_sysfrac  = lock contention (LOWER better)
+#   win_usercores = real-work rate (HIGHER better → shorter wall for same passes)
+
+# ---- Run #2: windowed triage (10 min/cell) on the cached run-#1 fixture ----
+#   baseline      = v121 -O2 @32c  (control: expect sys_frac~0.88, user_cores~2.6)
+#   v130-O2       = newer Binaryen @32c  (does the version fix the lock?)
+#   baseline@8    = v121 -O2 @8c   (do fewer threads cut contention?)
+#   v130-O2@8     = newer + fewer threads
+#   v121-O1       = lighter passes @32c (does less work progress faster?)
+CONFIGS_CONF="baseline v130-O2 baseline@8 v130-O2@8 v121-O1"
+CORES_CONF=""
+FIXTURE_RUN_ID_CONF="27197360957"
+DIAGNOSTIC_CONF="0"
+CAP_SECONDS_CONF="600"
+
+# ---- Run #3 (planned): full run (CAP_SECONDS_CONF="0") of the winner(s) for true
+#      wall-clock + output validity, then Chrome e2e. e.g. CONFIGS="v130-O2 v130-O2@8"
--- a/scripts/bench/vm-build.sh
+++ b/scripts/bench/vm-build.sh
@ -0,0 +1,100 @@
+#!/bin/bash
+# Run a real docker/build.sh build INSIDE the local QEMU Linux VM, as a cheap
+# stand-in for the Hetzner CI runner (Linux + Docker + the same scripts), so CI
+# orchestration changes can be verified without burning a Hetzner slot.
+# RUNS ON THE macOS HOST. The guest is aarch64 under HVF — near-native speed,
+# a correct *functional* proxy for CI but NOT an x86 performance proxy.
+#
+# Usage:
+#   ./scripts/bench/vm-build.sh [build.sh args...]
+#     default args: calculator --build-deps
+#
+# Examples:
+#   ./scripts/bench/vm-build.sh                                  # cold calculator build
+#   KICAD_PIPELINE=1 ./scripts/bench/vm-build.sh calculator,pl_editor   # pipeline smoke test
+#
+# Prereqs (one-time):
+#   VM_DISK=80G ./scripts/bench/setup-vm.sh prepare   # 80G: emsdk image + deps + build tree
+#   ./scripts/bench/setup-vm.sh run                   # boot in another terminal, wait for cloud-init
+#
+# Env: SSH_PORT (2222), KICAD_PIPELINE/KICAD_PIPELINE_JOBS (forwarded to the guest),
+#      VM_BUILD_JOBS (compile -j inside the guest; default = guest nproc).
+
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
+
+SSH_PORT="${SSH_PORT:-2222}"
+GUEST="bench@localhost"
+GUEST_DIR="kicad-wasm"
+SSH_OPTS=(-p "${SSH_PORT}" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null)
+
+vssh() { ssh "${SSH_OPTS[@]}" "${GUEST}" "$@"; }
+
+if ! vssh true 2>/dev/null; then
+    echo "ERROR: VM not reachable on localhost:${SSH_PORT}." >&2
+    echo "Boot it first: ./scripts/bench/setup-vm.sh run  (wait ~2 min for cloud-init)" >&2
+    exit 1
+fi
+
+# Wait until cloud-init finished installing Docker (first boot takes a few min).
+echo "Waiting for cloud-init to finish (installs Docker on first boot)..."
+vssh "cloud-init status --wait >/dev/null 2>&1 || true"
+vssh "docker info >/dev/null 2>&1" || {
+    echo "ERROR: Docker not usable in the guest. Re-run setup-vm.sh prepare with the" >&2
+    echo "updated cloud-init (installs docker-ce) and boot a fresh VM." >&2
+    exit 1
+}
+
+# Sync the working tree (incl. submodule content, no .git) into the guest.
+# Same exclude set as build.sh's container-sync, plus the VM images themselves.
+echo "Syncing repo to ${GUEST}:~/${GUEST_DIR} ..."
+rsync -az --delete \
+    -e "ssh ${SSH_OPTS[*]}" \
+    --exclude="build-wasm" \
+    --exclude="output" \
+    --exclude=".git" \
+    --exclude="logs" \
+    --exclude=".idea" \
+    --exclude="node_modules" \
+    --exclude="tools/emsdk" \
+    --exclude="scripts/bench/vm" \
+    --exclude="tests/test-results" \
+    --exclude="tests/playwright-report" \
+    "${PROJECT_ROOT}/" "${GUEST}:${GUEST_DIR}/"
+
+BUILD_ARGS=("$@")
+[ ${#BUILD_ARGS[@]} -eq 0 ] && BUILD_ARGS=(calculator --build-deps)
+# Append a guest-sized -j unless the caller already passed one.
+JOBS_FLAG='-j "${JOBS}"'
+[[ " ${BUILD_ARGS[*]} " == *" -j "* ]] && JOBS_FLAG=""
+
+# Guest-side env:
+# - COMPOSE_PROJECT_NAME: no .git in the guest tree, so build.sh can't derive a
+#   branch name — pin the project name explicitly.
+# - KICAD_DOCKER_CPUS/MEM: the compose dev-Mac caps (10 CPU / 32G) point at a
+#   guest that may have fewer cores and definitely has less RAM; size to guest.
+# - KICAD_NO_MONITOR/KICAD_LOG_NESTED: no TTY dashboard over ssh, stream output.
+REMOTE_CMD=$(cat <<EOF
+set -e
+cd ${GUEST_DIR}
+GUEST_CORES=\$(nproc)
+GUEST_MEM_G=\$(awk '/MemTotal/{printf "%d", \$2/1024/1024 - 3}' /proc/meminfo)
+export COMPOSE_PROJECT_NAME=kicad-wasm-vm
+export KICAD_DOCKER_CPUS=\${GUEST_CORES}
+export KICAD_DOCKER_MEM=\${GUEST_MEM_G}G
+export KICAD_NO_MONITOR=1 KICAD_LOG_NESTED=1
+export KICAD_PIPELINE=${KICAD_PIPELINE:-0} KICAD_PIPELINE_JOBS=${KICAD_PIPELINE_JOBS:-2}
+JOBS=${VM_BUILD_JOBS:-\$GUEST_CORES}
+echo "=== VM build: \$(uname -m), \${GUEST_CORES} cores, docker mem \${KICAD_DOCKER_MEM}, -j \${JOBS}, args: ${BUILD_ARGS[*]} ==="
+time ./docker/build.sh ${BUILD_ARGS[*]} ${JOBS_FLAG}
+ls -lh output/
+EOF
+)
+
+START=$(date +%s)
+vssh "${REMOTE_CMD}"
+END=$(date +%s)
+echo ""
+echo "=== VM build wall time: $(( (END - START) / 60 ))m $(( (END - START) % 60 ))s (args: ${BUILD_ARGS[*]}) ==="
--- a/scripts/bench/wasm-opt-bench.sh
+++ b/scripts/bench/wasm-opt-bench.sh
@ -0,0 +1,114 @@
+#!/bin/bash
+# wasm-opt allocator/core benchmark — RUNS INSIDE THE LINUX VM.
+#
+# Times the host-side wasm-opt/asyncify pass (scripts/common/apply-asyncify.sh)
+# over a prebuilt eeschema .wasm across a matrix of {glibc, jemalloc} x core
+# counts, to find why the step is slow on glibc CI and what BINARYEN_CORES helps.
+#
+# Why this isolates the right thing: wasm-opt/asyncify is a standalone pass over
+# an already-compiled .wasm (see docker/build.sh:194). We never compile KiCad
+# here — we just replay the optimizer over a fixture built once on the host.
+#
+# Usage (in the VM, from the repo root):
+#   ./scripts/bench/wasm-opt-bench.sh [fixture.wasm]
+# Env:
+#   CORES="1 4 8 10"   core counts to sweep (BINARYEN_CORES)
+#   ALLOCS="glibc jemalloc"
+#   STRACE=1           also run a syscall-count pass per allocator at max cores
+#
+# Output: a CSV table on stdout (also tee'd to bench/results.csv) plus per-cell
+# logs under bench/results/ (each holds apply-asyncify's own per-pass `time -v`).
+
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO="$(cd "${SCRIPT_DIR}/../.." && pwd)"
+
+FIXTURE="${1:-${REPO}/bench/eeschema.finalized.wasm}"
+CORES="${CORES:-1 4 8 10}"
+ALLOCS="${ALLOCS:-glibc jemalloc}"
+OUTDIR="${REPO}/bench/results"
+CSV="${REPO}/bench/results.csv"
+
+if [[ ! -f "${FIXTURE}" ]]; then
+    echo "ERROR: fixture not found: ${FIXTURE}" >&2
+    echo "Create it on the host (see scripts/bench/README.md) and scp it in." >&2
+    exit 1
+fi
+if [[ "$(uname -s)" != "Linux" ]]; then
+    echo "ERROR: run this inside the Linux VM (glibc is the point); host is $(uname -s)." >&2
+    exit 1
+fi
+command -v /usr/bin/time >/dev/null || { echo "ERROR: install GNU time (apt-get install -y time)" >&2; exit 1; }
+
+mkdir -p "${OUTDIR}"
+echo "cores,alloc,wall_clock,wall_s,peak_rss_kb,preload" > "${CSV}"
+
+# Convert GNU time's "Elapsed (wall clock)" field ([h:]m:ss[.ss]) to seconds.
+to_seconds() {
+    awk -F: '{ if (NF==3) print $1*3600+$2*60+$3; else if (NF==2) print $1*60+$2; else print $1 }'
+}
+
+run_cell() {
+    local cores="$1" alloc="$2"
+    local logf="${OUTDIR}/${alloc}-c${cores}.log"
+    local timef="${OUTDIR}/${alloc}-c${cores}.time"
+
+    cp "${FIXTURE}" /tmp/bench-in.wasm
+
+    # glibc baseline forces no preload; jemalloc leaves WASM_OPT_PRELOAD unset so
+    # apply-asyncify.sh auto-detects the system libjemalloc.
+    local -a env_prefix=(BINARYEN_CORES="${cores}")
+    if [[ "${alloc}" == "glibc" ]]; then
+        env_prefix+=(WASM_OPT_PRELOAD=none)
+    fi
+
+    echo ">>> ${alloc}  BINARYEN_CORES=${cores}" >&2
+    if ! env "${env_prefix[@]}" /usr/bin/time -v -o "${timef}" \
+            "${REPO}/scripts/common/apply-asyncify.sh" /tmp/bench-in.wasm /tmp/bench-out.wasm \
+            >"${logf}" 2>&1; then
+        echo "    FAILED (see ${logf})" >&2
+        echo "${cores},${alloc},FAILED,,," >> "${CSV}"
+        return 0
+    fi
+
+    local wall maxrss preload wall_s
+    wall=$(grep -F "Elapsed (wall clock)" "${timef}" | awk '{print $NF}')
+    maxrss=$(grep -F "Maximum resident set size" "${timef}" | awk '{print $NF}')
+    preload=$(grep -m1 -F "LD_PRELOAD=" "${logf}" | sed 's/.*LD_PRELOAD=//' | tr -d ' ')
+    wall_s=$(printf '%s' "${wall}" | to_seconds)
+    echo "    wall=${wall} (${wall_s}s)  peakRSS=${maxrss}KB  preload=${preload}" >&2
+    echo "${cores},${alloc},${wall},${wall_s},${maxrss},${preload}" >> "${CSV}"
+}
+
+for c in ${CORES}; do
+    for a in ${ALLOCS}; do
+        run_cell "${c}" "${a}"
+    done
+done
+
+# Optional: confirm the futex storm collapses with jemalloc. strace -c adds heavy
+# overhead, so this is a separate, single-pass-per-allocator measurement at the
+# highest core count, not part of the timing matrix above.
+if [[ "${STRACE:-0}" == "1" ]]; then
+    command -v strace >/dev/null || { echo "strace not installed; skipping" >&2; STRACE=0; }
+fi
+if [[ "${STRACE:-0}" == "1" ]]; then
+    maxc="$(echo ${CORES} | tr ' ' '\n' | sort -n | tail -1)"
+    WASM_OPT="$("${REPO}/scripts/common/get-wasm-opt.sh" 2>/dev/null)"
+    for a in ${ALLOCS}; do
+        cp "${FIXTURE}" /tmp/bench-in.wasm
+        local_preload=""
+        [[ "${a}" == "jemalloc" ]] && local_preload="$(ls /usr/lib/$(uname -m)-linux-gnu/libjemalloc.so.2 2>/dev/null || true)"
+        echo ">>> strace ${a} (asyncify pass, BINARYEN_CORES=${maxc})" >&2
+        env BINARYEN_CORES="${maxc}" ${local_preload:+LD_PRELOAD=${local_preload}} \
+            strace -f -c -e trace=futex,mmap,munmap -o "${OUTDIR}/strace-${a}.txt" \
+            "${WASM_OPT}" --asyncify /tmp/bench-in.wasm -o /tmp/bench-out.wasm \
+            >"${OUTDIR}/strace-${a}.log" 2>&1 || echo "    strace ${a} failed (see log)" >&2
+    done
+    echo "strace summaries: ${OUTDIR}/strace-*.txt" >&2
+fi
+
+echo ""
+echo "=== results (${CSV}) ==="
+column -t -s, "${CSV}"