perf(build): 4h05m -> 1h15m full CI build — docker caps, pipelined wasm-opt, Binaryen 130

Three orchestration fixes, validated end-to-end on the Hetzner ccx53
(run 27280051992, 1h14m41s vs 4h05m baseline, e2e identical):

- docker-compose CPU/memory caps were hardcoded to dev-Mac defaults
  (10 CPUs / 32G); now env-tunable via KICAD_DOCKER_CPUS/KICAD_DOCKER_MEM
  (CI exports nproc/110G — the 32-core runner was compiling on 10 cores).
- docker/build.sh: split build_app into compile_app (container) +
  postprocess_app (host-side dyncall shims + finalize + asyncify + -O2)
  and added KICAD_PIPELINE=1 mode that overlaps each tool's host-side
  wasm-opt with the next tool's container compile (max 2 concurrent
  postprocesses — pcbnew -O2 peaks ~34G RSS). Also: comma-separated app
  lists for cheap pipeline repros.
- get-wasm-opt.sh: Binaryen default 121 -> 130 (fixes the v121 -O2 lock
  convoy, ~9x) and BINARYEN_BUILD_FROM_SOURCE=1 support: the official
  x86_64-linux release tarballs (Alpine/musl, no LTO, assertions on) run
  asyncify 4x slower than a stock gcc -O3+LTO build with sha256-identical
  output; self-build takes ~5 min and is instantly repaid.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Istvan Matejcsok 2026-06-11 07:46:48 +02:00
commit 680eb9dc15
6 changed files with 289 additions and 33 deletions

View file

@ -10,7 +10,7 @@
# (e.g. v121+72 dev) and a standalone release (v121) corrupts asyncify
# metadata, causing "func is not a function" errors at runtime.
#
# Falls back to downloading standalone Binaryen v121 if emsdk is not installed
# Falls back to downloading standalone Binaryen v130 if emsdk is not installed
# locally (e.g. CI environments that only use Docker).
set -e
@ -19,8 +19,13 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
# --- Prefer emsdk-bundled Binaryen (matches the Emscripten that compiled the WASM) ---
# Override: set BINARYEN_VERSION in the env to FORCE a specific standalone release
# (skips the emsdk preference). Used to A/B Binaryen versions — e.g. v121 carries a
# wasm::Type lock-contention bug that makes the host-side -O2 pass ~9x slower than
# v130 (see docs/ci-build-slowness-findings.md). Validate any bump with the e2e
# suite: a Binaryen/emsdk skew can corrupt asyncify metadata ("func is not a function").
EMSDK_WASM_OPT="${PROJECT_ROOT}/tools/emsdk/upstream/bin/wasm-opt"
if [ -x "${EMSDK_WASM_OPT}" ]; then
if [ -z "${BINARYEN_VERSION:-}" ] && [ -x "${EMSDK_WASM_OPT}" ]; then
EMSDK_VERSION=$("${EMSDK_WASM_OPT}" --version 2>&1 || true)
echo "Using emsdk-bundled Binaryen: ${EMSDK_VERSION}" >&2
echo "${EMSDK_WASM_OPT}"
@ -30,7 +35,50 @@ fi
# --- Fallback: download standalone Binaryen (for CI or environments without local emsdk) ---
echo "emsdk Binaryen not found at ${EMSDK_WASM_OPT}, falling back to standalone download..." >&2
BINARYEN_VERSION="121"
# Default v130: v121 has a wasm::Type lock convoy that makes -O2 ~9x slower on
# many-core Linux (docs/ci-build-slowness-findings.md). v130 output validated by
# the full e2e suite locally (31/31) and Chromium-green on CI run 27226030304.
BINARYEN_VERSION="${BINARYEN_VERSION:-130}"
# BINARYEN_BUILD_FROM_SOURCE=1: compile wasm-opt ourselves instead of using the
# official Linux release tarballs, which are badly built — measured on the
# calculator fixture, identical output sha256: asyncify 4x faster on x86
# (CI run 27276830256: 3:50 -> 0:58) and 13x on aarch64; -O2 equal. The macOS
# tarball is well-built (self-build is ~12% SLOWER there), so this is only
# worth enabling on Linux CI. Needs cmake, ninja, g++, git. ~5 min on 32 cores,
# cached in build-wasm/tools after the first call.
if [[ "${BINARYEN_BUILD_FROM_SOURCE:-0}" == "1" ]]; then
SELF_DIR="${PROJECT_ROOT}/build-wasm/tools/binaryen-${BINARYEN_VERSION}-selfbuilt"
SELF_WASM_OPT="${SELF_DIR}/bin/wasm-opt"
if [ ! -x "${SELF_WASM_OPT}" ]; then
SRC_DIR="${PROJECT_ROOT}/build-wasm/tools/binaryen-src-${BINARYEN_VERSION}"
BUILD_DIR="${PROJECT_ROOT}/build-wasm/tools/binaryen-build-${BINARYEN_VERSION}"
echo "Building Binaryen v${BINARYEN_VERSION} from source (one-time, ~5 min)..." >&2
if [ ! -d "${SRC_DIR}" ]; then
git clone -q --depth 1 --branch "version_${BINARYEN_VERSION}" \
--recurse-submodules --shallow-submodules \
https://github.com/WebAssembly/binaryen.git "${SRC_DIR}" >&2
fi
cmake -S "${SRC_DIR}" -B "${BUILD_DIR}" -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_CXX_FLAGS="-Wno-maybe-uninitialized" \
-DCMAKE_INTERPROCEDURAL_OPTIMIZATION=ON -DBUILD_TESTS=OFF >&2
ninja -C "${BUILD_DIR}" wasm-opt wasm-emscripten-finalize >&2
# wasm-opt links lib/libbinaryen.so via rpath $ORIGIN/../lib.
mkdir -p "${SELF_DIR}/bin" "${SELF_DIR}/lib"
cp "${BUILD_DIR}/bin/wasm-opt" "${BUILD_DIR}/bin/wasm-emscripten-finalize" "${SELF_DIR}/bin/"
cp "${BUILD_DIR}"/lib/libbinaryen.* "${SELF_DIR}/lib/" 2>/dev/null || true
echo "Self-built Binaryen installed to ${SELF_DIR}" >&2
fi
SELF_VERSION=$("${SELF_WASM_OPT}" --version 2>&1 | grep -o '[0-9]\+' | head -1)
if [ "${SELF_VERSION}" != "${BINARYEN_VERSION}" ]; then
echo "ERROR: self-built wasm-opt version mismatch (got ${SELF_VERSION}, expected ${BINARYEN_VERSION})" >&2
exit 1
fi
echo "Using self-built Binaryen v${BINARYEN_VERSION} (${SELF_WASM_OPT})" >&2
echo "${SELF_WASM_OPT}"
exit 0
fi
BINARYEN_DIR="${PROJECT_ROOT}/build-wasm/tools/binaryen-${BINARYEN_VERSION}"
WASM_OPT="${BINARYEN_DIR}/bin/wasm-opt"