build(wasm): standardize asyncify wasm-opt tail to -O1 everywhere
The Binaryen wasm-opt shrink after Asyncify was the only -O2 in the pipeline: -O2 by default (local + tag release) but -O1 in main CI. The C++/wx/deps compile is already -O1 everywhere (DEBUG_BUILD defaults to 1; nothing passes --release in CI/release). Pin the tail to -O1 too: - apply-asyncify.sh: BINARYEN_OPT_LEVEL default -O2 -> -O1 - release.yml: opt_level -O2 -> -O1 (demo now ships -O1) - ci-ubicloud.yml / wasm-build.yml / docker/build.sh: refresh stale -O2 comments Because both callers now build at -O1, the FINAL cache key converges, so a tag release FINAL-cache-hits main's build and skips the asyncify-tail rebuild entirely (previously the ~1-2h -O2 wasm-opt was rerun per release). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
89a433fda9
commit
e7640d0a97
5 changed files with 43 additions and 38 deletions
12
.github/workflows/ci-ubicloud.yml
vendored
12
.github/workflows/ci-ubicloud.yml
vendored
|
|
@ -2,10 +2,10 @@ name: CI full build + e2e (Ubicloud)
|
||||||
|
|
||||||
# Main/PR gate: build all 6 KiCad WASM tools + run the wxWidgets + KiCad e2e
|
# Main/PR gate: build all 6 KiCad WASM tools + run the wxWidgets + KiCad e2e
|
||||||
# suites on a Ubicloud runner. The actual recipe lives in the reusable
|
# suites on a Ubicloud runner. The actual recipe lives in the reusable
|
||||||
# wasm-build.yml (the SINGLE build definition) — this is just the main/PR caller,
|
# wasm-build.yml (the SINGLE build definition) — this is just the main/PR caller.
|
||||||
# pinned to -O1 for fast feedback. The tag release (release.yml) calls the same
|
# The asyncify wasm-opt tail is -O1 everywhere (main, release, local); the tag
|
||||||
# recipe at -O2 (the shipped demo opt level) and then publishes, so the build can
|
# release (release.yml) calls the same recipe at the same -O1 and then publishes,
|
||||||
# never diverge from what ships.
|
# so the build can never diverge from what ships.
|
||||||
#
|
#
|
||||||
# Routine edits under scripts/ do NOT bust the WASM output cache — only the subset
|
# Routine edits under scripts/ do NOT bust the WASM output cache — only the subset
|
||||||
# in scripts/deploy/wasm-cache-hash.mjs does. For inputs the sc-hash can't see
|
# in scripts/deploy/wasm-cache-hash.mjs does. For inputs the sc-hash can't see
|
||||||
|
|
@ -36,8 +36,8 @@ jobs:
|
||||||
build:
|
build:
|
||||||
uses: ./.github/workflows/wasm-build.yml
|
uses: ./.github/workflows/wasm-build.yml
|
||||||
with:
|
with:
|
||||||
# -O1: faster asyncify shrink, sufficient for the test gate (the demo ships
|
# -O1 asyncify shrink — the level we ship (release.yml uses the same -O1).
|
||||||
# -O2 via release.yml). 3D viewer ON so 3d-viewer.spec.ts has a viewer.
|
# 3D viewer ON so 3d-viewer.spec.ts has a viewer.
|
||||||
opt_level: "-O1"
|
opt_level: "-O1"
|
||||||
build_3d_viewer: "ON"
|
build_3d_viewer: "ON"
|
||||||
run_tests: true
|
run_tests: true
|
||||||
|
|
|
||||||
18
.github/workflows/release.yml
vendored
18
.github/workflows/release.yml
vendored
|
|
@ -1,10 +1,11 @@
|
||||||
name: release
|
name: release
|
||||||
|
|
||||||
# Release pipeline on a vX.Y.Z tag, in order:
|
# Release pipeline on a vX.Y.Z tag, in order:
|
||||||
# 1) build — the SAME wasm-build.yml recipe as CI, but at -O2 (the shipped demo
|
# 1) build — the SAME wasm-build.yml recipe as CI, at the SAME -O1 asyncify tail
|
||||||
# opt level) + run the e2e gate; reuses main's opt-independent base
|
# + run the e2e gate. Because the opt level now matches main, this
|
||||||
# cache so only the asyncify/-O2 tail rebuilds. Uploads output/.
|
# FINAL-cache-hits main's build and skips the asyncify tail rebuild.
|
||||||
# 2) publish-wasm — push the -O2 build to the CDN (content-addressed, idempotent)
|
# Uploads output/.
|
||||||
|
# 2) publish-wasm — push the build to the CDN (content-addressed, idempotent)
|
||||||
# and write manifest-<tag>.json. The registry now reflects THIS tag.
|
# and write manifest-<tag>.json. The registry now reflects THIS tag.
|
||||||
# 3) deploy-demo — build the standalone pinned to that manifest + deploy to
|
# 3) deploy-demo — build the standalone pinned to that manifest + deploy to
|
||||||
# demo.pcbjam.com. So the demo can never point at stale wasm.
|
# demo.pcbjam.com. So the demo can never point at stale wasm.
|
||||||
|
|
@ -49,17 +50,18 @@ jobs:
|
||||||
- id: tag
|
- id: tag
|
||||||
run: echo "tag=${{ github.event.inputs.tag || github.ref_name }}" >> "$GITHUB_OUTPUT"
|
run: echo "tag=${{ github.event.inputs.tag || github.ref_name }}" >> "$GITHUB_OUTPUT"
|
||||||
|
|
||||||
# 1) Build at -O2 (shipped) + e2e gate, reusing main's base cache. Uploads the
|
# 1) Build (-O1, identical to main) + e2e gate. Reuses main's FINAL cache (same
|
||||||
# publishable output/ as the 'wasm-output' artifact for the publish job.
|
# opt level) so the asyncify tail isn't rebuilt. Uploads the publishable
|
||||||
|
# output/ as the 'wasm-output' artifact for the publish job.
|
||||||
build:
|
build:
|
||||||
uses: ./.github/workflows/wasm-build.yml
|
uses: ./.github/workflows/wasm-build.yml
|
||||||
with:
|
with:
|
||||||
opt_level: "-O2"
|
opt_level: "-O1"
|
||||||
build_3d_viewer: "ON"
|
build_3d_viewer: "ON"
|
||||||
run_tests: true
|
run_tests: true
|
||||||
upload_output: true
|
upload_output: true
|
||||||
|
|
||||||
# 2) Publish the -O2 build to the CDN (content-addressed; unchanged tools reuse)
|
# 2) Publish the build to the CDN (content-addressed; unchanged tools reuse)
|
||||||
# and write manifest-<tag>.json. Cheap runner — just downloads + uploads.
|
# and write manifest-<tag>.json. Cheap runner — just downloads + uploads.
|
||||||
publish-wasm:
|
publish-wasm:
|
||||||
needs: [meta, build]
|
needs: [meta, build]
|
||||||
|
|
|
||||||
30
.github/workflows/wasm-build.yml
vendored
30
.github/workflows/wasm-build.yml
vendored
|
|
@ -1,27 +1,29 @@
|
||||||
name: wasm-build (reusable)
|
name: wasm-build (reusable)
|
||||||
|
|
||||||
# THE single KiCad WASM build+test recipe, called by ci-ubicloud.yml (main/PR,
|
# THE single KiCad WASM build+test recipe, called by ci-ubicloud.yml (main/PR) and
|
||||||
# -O1 for fast feedback) and release.yml (tag, -O2 for the shipped demo). There
|
# release.yml (tag). Both build the SAME way — the asyncify `wasm-opt` tail is -O1
|
||||||
# is no other build definition — publishing/deploy consume this job's output, so
|
# everywhere — so the build can never diverge from what ships (the bug that shipped
|
||||||
# the build can never diverge from what ships (the bug that shipped a -O1 / 3D-off
|
# a -O1 / 3D-off demo while CI built -O1 / 3D-on). The opt_level input remains as an
|
||||||
# demo while CI built -O1 / 3D-on).
|
# escape hatch for a one-off -O2 build, but is -O1 for both real callers, so a tag
|
||||||
|
# release FINAL-cache-hits main's build and skips the asyncify tail entirely.
|
||||||
#
|
#
|
||||||
# Two-tier output cache around docker/build.sh's --compile-only / --postprocess-only
|
# Two-tier output cache around docker/build.sh's --compile-only / --postprocess-only
|
||||||
# split. -O1 and -O2 differ ONLY in the final `wasm-opt -O` shrink; everything
|
# split. The expensive container compile (→ base wasm) is opt- and binaryen-
|
||||||
# before (docker compile → base wasm → asyncify instrument) is identical. So:
|
# independent; only the host `asyncify + wasm-opt -O` tail depends on the binaryen
|
||||||
# - BASE cache (opt-INDEPENDENT key): the --compile-only output (base wasm +
|
# fork + opt level. So:
|
||||||
# sysroot headers). Warmed by main's -O1 CI; an -O2 release restores it and
|
# - BASE cache (compile-input key): the --compile-only output (base wasm +
|
||||||
# skips the expensive compile, running only the asyncify/-O2 tail.
|
# sysroot headers). Reused whenever only the binaryen fork / asyncify config
|
||||||
# - FINAL cache (opt-SPECIFIC key): the post-processed output; fast-path for
|
# changed — the compile is skipped and just the tail reruns.
|
||||||
# re-running the same opt+SHA (e.g. a re-deploy).
|
# - FINAL cache (base + binaryen SHA + opt level): the post-processed output;
|
||||||
# Both keys now include the 3D-viewer flag, so a 3D-on and 3D-off build can never
|
# fast-path for re-running the same SHA (a tag release reusing main, a re-deploy).
|
||||||
|
# Both keys include the 3D-viewer flag, so a 3D-on and 3D-off build can never
|
||||||
# poison each other's cache.
|
# poison each other's cache.
|
||||||
|
|
||||||
on:
|
on:
|
||||||
workflow_call:
|
workflow_call:
|
||||||
inputs:
|
inputs:
|
||||||
opt_level:
|
opt_level:
|
||||||
description: "Binaryen wasm-opt shrink level for the asyncify tail (-O1 fast / -O2 shipped)"
|
description: "Binaryen wasm-opt shrink level for the asyncify tail (-O1 everywhere; escape hatch for a one-off -O2)"
|
||||||
type: string
|
type: string
|
||||||
default: "-O1"
|
default: "-O1"
|
||||||
build_3d_viewer:
|
build_3d_viewer:
|
||||||
|
|
|
||||||
|
|
@ -22,15 +22,15 @@
|
||||||
#
|
#
|
||||||
# The build is split into two phases:
|
# The build is split into two phases:
|
||||||
# 1. Docker: Compile KiCad to WASM (without asyncify)
|
# 1. Docker: Compile KiCad to WASM (without asyncify)
|
||||||
# 2. Host: dyncall shims + finalize + asyncify + -O2 (Binaryen submodule via build-wasm-opt.sh)
|
# 2. Host: dyncall shims + finalize + asyncify + wasm-opt -O1 (Binaryen submodule via build-wasm-opt.sh)
|
||||||
#
|
#
|
||||||
# KICAD_PIPELINE=1 (multi-app builds only): run phase 2 of each app in the
|
# KICAD_PIPELINE=1 (multi-app builds only): run phase 2 of each app in the
|
||||||
# background while the next app compiles in the container. wasm-opt is
|
# background while the next app compiles in the container. wasm-opt is
|
||||||
# Amdahl-capped at ~4 effective cores, so on a many-core CI box the container
|
# Amdahl-capped at ~4 effective cores, so on a many-core CI box the container
|
||||||
# would otherwise sit idle for the 1-2h of host-side wasm-opt (run 27226030304:
|
# would otherwise sit idle for the 1-2h of host-side wasm-opt (run 27226030304:
|
||||||
# 103 min of the 4h was tools serialized behind each other's wasm-opt). At most
|
# 103 min of the 4h was tools serialized behind each other's wasm-opt). At most
|
||||||
# KICAD_PIPELINE_JOBS (default 2) postprocesses run concurrently — pcbnew's -O2
|
# KICAD_PIPELINE_JOBS (default 2) postprocesses run concurrently — pcbnew's wasm-opt
|
||||||
# peaks ~34 GB RSS, so 2 fits the 128 GB CI box but NOT a dev Mac: leave
|
# pass peaks ~34 GB RSS, so 2 fits the 128 GB CI box but NOT a dev Mac: leave
|
||||||
# KICAD_PIPELINE unset locally.
|
# KICAD_PIPELINE unset locally.
|
||||||
#
|
#
|
||||||
# Binaryen is downloaded automatically - no prerequisites needed.
|
# Binaryen is downloaded automatically - no prerequisites needed.
|
||||||
|
|
@ -118,7 +118,7 @@ echo "Building app: ${APP_NAME}"
|
||||||
# Build phase (cache split). The container compile produces an OPT-INDEPENDENT
|
# Build phase (cache split). The container compile produces an OPT-INDEPENDENT
|
||||||
# base wasm; the only opt-DEPENDENT work is the final `wasm-opt -O$LEVEL` shrink
|
# base wasm; the only opt-DEPENDENT work is the final `wasm-opt -O$LEVEL` shrink
|
||||||
# inside the host post-process (apply-asyncify.sh). Splitting them lets CI cache
|
# inside the host post-process (apply-asyncify.sh). Splitting them lets CI cache
|
||||||
# the expensive compile once (shared across -O1/-O2) and re-run just the
|
# the expensive compile once (shared regardless of the asyncify tail) and re-run just the
|
||||||
# asyncify/-O tail per opt level — see .github/workflows/.
|
# asyncify/-O tail per opt level — see .github/workflows/.
|
||||||
# (default) both — compile in-container, then host post-process.
|
# (default) both — compile in-container, then host post-process.
|
||||||
# --compile-only — only the in-container compile → base wasm in output/.
|
# --compile-only — only the in-container compile → base wasm in output/.
|
||||||
|
|
@ -248,7 +248,7 @@ compile_app() {
|
||||||
}
|
}
|
||||||
|
|
||||||
# Phase 2 of one app: host-side post-processing (dyncall shims, finalize,
|
# Phase 2 of one app: host-side post-processing (dyncall shims, finalize,
|
||||||
# asyncify + -O2). Pure host work on output/${app}.* — independent of the
|
# asyncify + wasm-opt -O1). Pure host work on output/${app}.* — independent of the
|
||||||
# container, which is what makes it safe to run in the background while the
|
# container, which is what makes it safe to run in the background while the
|
||||||
# next app compiles.
|
# next app compiles.
|
||||||
postprocess_app() {
|
postprocess_app() {
|
||||||
|
|
@ -280,7 +280,7 @@ postprocess_app() {
|
||||||
# Apply asyncify transformation on host. The converter is a synchronous node
|
# Apply asyncify transformation on host. The converter is a synchronous node
|
||||||
# CLI built with ASYNCIFY=0, so asyncify is unnecessary and would be wrong.
|
# CLI built with ASYNCIFY=0, so asyncify is unnecessary and would be wrong.
|
||||||
# apply-asyncify always runs the --hoist-cpp-catches pass FIRST (native wasm-EH is the only build
|
# apply-asyncify always runs the --hoist-cpp-catches pass FIRST (native wasm-EH is the only build
|
||||||
# mode) so Asyncify can suspend from inside C++ catch arms, then asyncify + removelist + -O2.
|
# mode) so Asyncify can suspend from inside C++ catch arms, then asyncify + removelist + wasm-opt -O1.
|
||||||
if [ "$app" != "sym_convert" ]; then
|
if [ "$app" != "sym_convert" ]; then
|
||||||
kw_stage asyncify
|
kw_stage asyncify
|
||||||
./scripts/common/apply-asyncify.sh "${out_dir}/${app}.wasm" "${out_dir}/${app}.wasm"
|
./scripts/common/apply-asyncify.sh "${out_dir}/${app}.wasm" "${out_dir}/${app}.wasm"
|
||||||
|
|
@ -304,7 +304,7 @@ pipeline_running_count() {
|
||||||
}
|
}
|
||||||
|
|
||||||
# Launch postprocess_app in the background, capped at KICAD_PIPELINE_JOBS
|
# Launch postprocess_app in the background, capped at KICAD_PIPELINE_JOBS
|
||||||
# concurrent jobs (default 2: pcbnew's -O2 peaks ~34 GB RSS; two postprocesses
|
# concurrent jobs (default 2: pcbnew's wasm-opt pass peaks ~34 GB RSS; two postprocesses
|
||||||
# plus the container compile fit the 128 GB CI box). Portable poll loop instead
|
# plus the container compile fit the 128 GB CI box). Portable poll loop instead
|
||||||
# of `wait -n` (absent in macOS bash 3.2).
|
# of `wait -n` (absent in macOS bash 3.2).
|
||||||
pipeline_postprocess() {
|
pipeline_postprocess() {
|
||||||
|
|
|
||||||
|
|
@ -128,9 +128,10 @@ fi
|
||||||
|
|
||||||
# 3. Post-asyncify shrink. Asyncify spills every live local; without coalescing, large
|
# 3. Post-asyncify shrink. Asyncify spills every live local; without coalescing, large
|
||||||
# coroutine-entry functions exceed V8's per-function locals limit and stall/crash the renderer.
|
# coroutine-entry functions exceed V8's per-function locals limit and stall/crash the renderer.
|
||||||
# -O1 already runs CoalesceLocals and suffices (CI uses it); -O2 is the default. NOT -Os/-Oz
|
# -O1 runs CoalesceLocals (enough to keep instrumented functions under V8's local limit) and is the
|
||||||
# (they break the asyncify runtime — Binaryen #4484). See docs/debugging/DEBUG.md §6-7.
|
# level we ship EVERYWHERE — main CI, tag releases, and local builds all use it. NOT -Os/-Oz (they
|
||||||
BINARYEN_OPT_LEVEL="${BINARYEN_OPT_LEVEL:--O2}"
|
# break the asyncify runtime — Binaryen #4484). See docs/debugging/DEBUG.md §6-7.
|
||||||
|
BINARYEN_OPT_LEVEL="${BINARYEN_OPT_LEVEL:--O1}"
|
||||||
echo "Running wasm-opt ${BINARYEN_OPT_LEVEL} (shrink instrumented functions under V8's local limit)..."
|
echo "Running wasm-opt ${BINARYEN_OPT_LEVEL} (shrink instrumented functions under V8's local limit)..."
|
||||||
"${PRELOAD_CMD[@]}" "${TIME_CMD[@]}" "${WASM_OPT}" "${BINARYEN_OPT_LEVEL}" "${FEAT[@]}" ${G} "${OUTPUT_WASM}" -o "${OUTPUT_WASM}"
|
"${PRELOAD_CMD[@]}" "${TIME_CMD[@]}" "${WASM_OPT}" "${BINARYEN_OPT_LEVEL}" "${FEAT[@]}" ${G} "${OUTPUT_WASM}" -o "${OUTPUT_WASM}"
|
||||||
|
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue