perf(build): 4h05m -> 1h15m full CI build — docker caps, pipelined wasm-opt, Binaryen 130
Three orchestration fixes, validated end-to-end on the Hetzner ccx53 (run 27280051992, 1h14m41s vs 4h05m baseline, e2e identical): - docker-compose CPU/memory caps were hardcoded to dev-Mac defaults (10 CPUs / 32G); now env-tunable via KICAD_DOCKER_CPUS/KICAD_DOCKER_MEM (CI exports nproc/110G — the 32-core runner was compiling on 10 cores). - docker/build.sh: split build_app into compile_app (container) + postprocess_app (host-side dyncall shims + finalize + asyncify + -O2) and added KICAD_PIPELINE=1 mode that overlaps each tool's host-side wasm-opt with the next tool's container compile (max 2 concurrent postprocesses — pcbnew -O2 peaks ~34G RSS). Also: comma-separated app lists for cheap pipeline repros. - get-wasm-opt.sh: Binaryen default 121 -> 130 (fixes the v121 -O2 lock convoy, ~9x) and BINARYEN_BUILD_FROM_SOURCE=1 support: the official x86_64-linux release tarballs (Alpine/musl, no LTO, assertions on) run asyncify 4x slower than a stock gcc -O3+LTO build with sha256-identical output; self-build takes ~5 min and is instantly repaid. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
parent
e6ff74a11e
commit
680eb9dc15
6 changed files with 289 additions and 33 deletions
|
|
@ -10,7 +10,7 @@
|
|||
# (e.g. v121+72 dev) and a standalone release (v121) corrupts asyncify
|
||||
# metadata, causing "func is not a function" errors at runtime.
|
||||
#
|
||||
# Falls back to downloading standalone Binaryen v121 if emsdk is not installed
|
||||
# Falls back to downloading standalone Binaryen v130 if emsdk is not installed
|
||||
# locally (e.g. CI environments that only use Docker).
|
||||
|
||||
set -e
|
||||
|
|
@ -19,8 +19,13 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|||
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
|
||||
|
||||
# --- Prefer emsdk-bundled Binaryen (matches the Emscripten that compiled the WASM) ---
|
||||
# Override: set BINARYEN_VERSION in the env to FORCE a specific standalone release
|
||||
# (skips the emsdk preference). Used to A/B Binaryen versions — e.g. v121 carries a
|
||||
# wasm::Type lock-contention bug that makes the host-side -O2 pass ~9x slower than
|
||||
# v130 (see docs/ci-build-slowness-findings.md). Validate any bump with the e2e
|
||||
# suite: a Binaryen/emsdk skew can corrupt asyncify metadata ("func is not a function").
|
||||
EMSDK_WASM_OPT="${PROJECT_ROOT}/tools/emsdk/upstream/bin/wasm-opt"
|
||||
if [ -x "${EMSDK_WASM_OPT}" ]; then
|
||||
if [ -z "${BINARYEN_VERSION:-}" ] && [ -x "${EMSDK_WASM_OPT}" ]; then
|
||||
EMSDK_VERSION=$("${EMSDK_WASM_OPT}" --version 2>&1 || true)
|
||||
echo "Using emsdk-bundled Binaryen: ${EMSDK_VERSION}" >&2
|
||||
echo "${EMSDK_WASM_OPT}"
|
||||
|
|
@ -30,7 +35,50 @@ fi
|
|||
# --- Fallback: download standalone Binaryen (for CI or environments without local emsdk) ---
|
||||
echo "emsdk Binaryen not found at ${EMSDK_WASM_OPT}, falling back to standalone download..." >&2
|
||||
|
||||
BINARYEN_VERSION="121"
|
||||
# Default v130: v121 has a wasm::Type lock convoy that makes -O2 ~9x slower on
|
||||
# many-core Linux (docs/ci-build-slowness-findings.md). v130 output validated by
|
||||
# the full e2e suite locally (31/31) and Chromium-green on CI run 27226030304.
|
||||
BINARYEN_VERSION="${BINARYEN_VERSION:-130}"
|
||||
|
||||
# BINARYEN_BUILD_FROM_SOURCE=1: compile wasm-opt ourselves instead of using the
|
||||
# official Linux release tarballs, which are badly built — measured on the
|
||||
# calculator fixture, identical output sha256: asyncify 4x faster on x86
|
||||
# (CI run 27276830256: 3:50 -> 0:58) and 13x on aarch64; -O2 equal. The macOS
|
||||
# tarball is well-built (self-build is ~12% SLOWER there), so this is only
|
||||
# worth enabling on Linux CI. Needs cmake, ninja, g++, git. ~5 min on 32 cores,
|
||||
# cached in build-wasm/tools after the first call.
|
||||
if [[ "${BINARYEN_BUILD_FROM_SOURCE:-0}" == "1" ]]; then
|
||||
SELF_DIR="${PROJECT_ROOT}/build-wasm/tools/binaryen-${BINARYEN_VERSION}-selfbuilt"
|
||||
SELF_WASM_OPT="${SELF_DIR}/bin/wasm-opt"
|
||||
if [ ! -x "${SELF_WASM_OPT}" ]; then
|
||||
SRC_DIR="${PROJECT_ROOT}/build-wasm/tools/binaryen-src-${BINARYEN_VERSION}"
|
||||
BUILD_DIR="${PROJECT_ROOT}/build-wasm/tools/binaryen-build-${BINARYEN_VERSION}"
|
||||
echo "Building Binaryen v${BINARYEN_VERSION} from source (one-time, ~5 min)..." >&2
|
||||
if [ ! -d "${SRC_DIR}" ]; then
|
||||
git clone -q --depth 1 --branch "version_${BINARYEN_VERSION}" \
|
||||
--recurse-submodules --shallow-submodules \
|
||||
https://github.com/WebAssembly/binaryen.git "${SRC_DIR}" >&2
|
||||
fi
|
||||
cmake -S "${SRC_DIR}" -B "${BUILD_DIR}" -G Ninja \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DCMAKE_CXX_FLAGS="-Wno-maybe-uninitialized" \
|
||||
-DCMAKE_INTERPROCEDURAL_OPTIMIZATION=ON -DBUILD_TESTS=OFF >&2
|
||||
ninja -C "${BUILD_DIR}" wasm-opt wasm-emscripten-finalize >&2
|
||||
# wasm-opt links lib/libbinaryen.so via rpath $ORIGIN/../lib.
|
||||
mkdir -p "${SELF_DIR}/bin" "${SELF_DIR}/lib"
|
||||
cp "${BUILD_DIR}/bin/wasm-opt" "${BUILD_DIR}/bin/wasm-emscripten-finalize" "${SELF_DIR}/bin/"
|
||||
cp "${BUILD_DIR}"/lib/libbinaryen.* "${SELF_DIR}/lib/" 2>/dev/null || true
|
||||
echo "Self-built Binaryen installed to ${SELF_DIR}" >&2
|
||||
fi
|
||||
SELF_VERSION=$("${SELF_WASM_OPT}" --version 2>&1 | grep -o '[0-9]\+' | head -1)
|
||||
if [ "${SELF_VERSION}" != "${BINARYEN_VERSION}" ]; then
|
||||
echo "ERROR: self-built wasm-opt version mismatch (got ${SELF_VERSION}, expected ${BINARYEN_VERSION})" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "Using self-built Binaryen v${BINARYEN_VERSION} (${SELF_WASM_OPT})" >&2
|
||||
echo "${SELF_WASM_OPT}"
|
||||
exit 0
|
||||
fi
|
||||
BINARYEN_DIR="${PROJECT_ROOT}/build-wasm/tools/binaryen-${BINARYEN_VERSION}"
|
||||
WASM_OPT="${BINARYEN_DIR}/bin/wasm-opt"
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue