pcbjam/docs/ci-build-slowness-findings.md

# CI build slowness — root-cause findings (handoff)

> Why the Hetzner CI build takes ~2 h. The bottleneck is the host-side
> `wasm-opt -O2` pass (~76–88 min). **This doc was substantially revised** after
> pulling the per-pass `/usr/bin/time -v` counters out of the real CI runs and a
> multi-source deep-research pass: the earlier "irreducible 3.5 CPU-hours of
> work, environment only moves it ±15%" conclusion was **wrong**. See
> "Correction log" at the bottom for what changed and why.

## 🆕 RUN #4 VERDICT (CI run 27226030304, 2026-06-09): v130 works; the remaining 4h is ORCHESTRATION
Run #4 built **all 6 tools** on v130 on the ccx53 in **4h05m** (18:10→22:15 UTC).
The v121 lock convoy is confirmed dead on-box: pcbnew's `-O2` ran with **system
time 81 s / 1,495 voluntary ctx-switches** (v121: 114,075 s / 180 M). The 4h has
three *new*, measured causes — all orchestration, none of them wasm-opt pathology:

| Phase | Wall | Cause |
|---|---|---|
| setup | 4 min | fine |
| deps + wx + pcbnew compile | 50 min (18:12→19:02) | **docker-compose capped the container at `cpus: '10'`** (dev-Mac default) on the 32-core box, with `-j 32` oversubscribed on top |
| pcbnew asyncify 24 min + `-O2` 66 min | 90 min (19:02→20:32) | pcbnew is **338 MB** pre-O2 (eeschema: 188 MB). `-O2` = 13,993 s user @ **354% CPU** (BINARYEN_CORES=8) — *real* compute, Amdahl-capped at ~4 effective cores. This is the irreducible critical path. |
| 5 remaining tools, strictly sequential | 103 min (20:32→22:15) | each tool's host-side wasm-opt **blocked** the next tool's container compile; wasm-opt uses ~4 of 32 cores while the container idles |

eeschema's `-O2` was 10:38 — exactly the bench prediction, so the bench fixture
generalizes. The Mac does pcbnew's asyncify+O2 in ~35 min @ 6 cores (arm64
per-core advantage); the CI gap beyond that is orchestration, fixed by:
1. **Lift the compose caps in CI** — `KICAD_DOCKER_CPUS`/`KICAD_DOCKER_MEM` env
   interpolation in docker-compose.yml; CI sets nproc/110G (was 10/32G).
2. **Pipeline host-side wasm-opt with the next tool's compile** —
   `KICAD_PIPELINE=1` in docker/build.sh backgrounds dyncall+finalize+asyncify+O2
   (max `KICAD_PIPELINE_JOBS=2` concurrent; pcbnew `-O2` peaks 33.6 GB RSS).
   CI-only: a 32 GB dev Mac can't stack two postprocesses.
3. **BINARYEN_CORES=16** in the validate workflow (bench: 32c=8:02 vs 8c≈10:00
   on the eeschema fixture — mild win, and two concurrent postprocesses share
   the box with the compile).
4. **Binaryen default bumped 121→130** in get-wasm-opt.sh (validated: local
   31/31 e2e; CI run #4 Chromium fully green).

Expected: ~2h–2h20m (floor = deps + pcbnew compile + pcbnew's 90-min wasm-opt
chain). Below that needs Lever E (shrink pcbnew's `-O2` work) and/or caching
deps across runs (the ephemeral runner rebuilds deps+wx every time, ~30–50 min).

**Run #4 e2e: 16 passed / 14 failed — ALL 14 Firefox-only**, across every tool
("GL canvas has zero dimensions", wizard never appears), while Chromium passed
everything. Headless-Firefox/WebGL environment problem on the Hetzner VM, not a
v130 regression (locally Firefox passes; and `-O2` exists for V8's locals limit,
i.e. Chromium is the engine that matters for the corruption check). Open issue,
tracked separately from build time.

Correction: the "all 6 tools rc=0 in ~1h04m" local claim below was wrong — the
actual log (`logs/build/20260609-191138.log`) spans 19:11→21:15 ≈ **2h04m**
(`-j 3`, BINARYEN_CORES=6). The conclusion (v130 builds a working KiCad) stands.

## ✅ FINAL VALIDATION (run 27280051992, 2026-06-10): all 6 tools in **1h14m41s** (was 4h05m, 3.3x)
Full cold `all` build on the ccx53 with everything adopted (compose caps lifted,
KICAD_PIPELINE=1, BINARYEN_CORES=16, self-built wasm-opt v130):
- pcbnew: asyncify **5:12** (was 24:07 — the self-built binary at pcbnew scale),
  `-O2` **52:09** (was 1:06:13). Its 58-min postprocess fully overlapped ALL
  five other tools' compiles + postprocesses (eeschema asyncify 1:00 — was 7:22).
- Critical path is now ≈ deps + pcbnew compile + pcbnew asyncify+`-O2` ≈ the
  whole 1h15. Going below ~1h10 requires shrinking pcbnew's `-O2` input —
  see the wasm-EH (-fwasm-exceptions / KICAD_WASM_EH) experiment.
- e2e: 16 passed / 14 failed — the failures are the SAME 14 Firefox-only
  environmental tests as run #4 (Chromium 100% green). No regression; Firefox
  headless-GL on the Hetzner VM remains a separate open issue.

## 🆕 2026-06-10 EXPERIMENT DAY: orchestration verified on-box + the release-tarball discovery

**Repro run 27273412419** (calculator,pl_editor, pipelined, no e2e, ccx53): all
orchestration fixes confirmed on the real runner — deps 14 min (was 28 at the
10-CPU cap), pipeline overlap engaged (calculator's wasm-opt ran during
pl_editor's -j32 compile), whole step **32 min**.

**The official Linux Binaryen release tarballs are badly built.** Measured on
identical fixtures with sha256-identical outputs:
- x86_64 (ccx53, run 27276830256, BINARYEN_CORES=16): asyncify 3:50 → **0:58
  (4x)** with a stock gcc -O3+LTO self-build; -O2 equal (4:22 vs 4:20).
- aarch64 (M-chip QEMU VM): asyncify **13x** faster self-built; clang -O3+LTO
  also beats the tarball's -O2 by 3.5%. With a good binary, Linux ≈ macOS on
  the same silicon (2:59 vs 2:53) — the "Linux is slow" gap was binary quality.
- macOS arm64: the official tarball is *well*-built (self-build 12% slower) —
  keep the tarball on dev Macs; self-build is a Linux-CI-only fix.
→ Adopted: `BINARYEN_BUILD_FROM_SOURCE=1` in get-wasm-opt.sh (one-time ~5-min
build per ephemeral runner, cached in build-wasm/tools, builds wasm-opt +
wasm-emscripten-finalize). Good upstream-issue material (WebAssembly/binaryen).

**Allocator: dead lever on v130** (VM sweep, calculator fixture): glibc 5:37,
jemalloc 5:47, mimalloc 6:59. The jemalloc preload is harmless legacy now.

**arm64 cloud runners: ruled out** (run 27273412432, ubicloud-standard-8-arm):
calculator asyncify 10:42 / -O2 8:40 — Ampere burns ~1.6x the EPYC cycles and
~2x the M-chip's on the Amdahl-bound wasm-opt. Per-core speed is what matters.

**Projection for `all`** with everything adopted: setup 3 + deps 14 + pcbnew
compile ~10 + binaryen build 5 + pcbnew asyncify ~6 + pcbnew -O2 ~60, other 5
tools fully overlapped ≈ **~1h35-40m** (from 4h05m). Next levers beyond that:
Lever E on pcbnew's -O2 (user ruled out -O1; pass-subset/removelist remain) and
deps caching across runners.

## ✅ THE FIX (bench run 27210317273): upgrade Binaryen 121 → 130
Measured on the cached 188 MB fixture, same `-O2`, identical output size:

| Binaryen | cores | `-O2` wall | system time | ctx-switches | effective cores |
|---|---|---|---|---|---|
| **121** (current) | 32 | **1:12:52** | 89% | 180,000,000 | ~2.6 |
| **130** | 32 | **8:02** | **1%** | **3,599** | **~10** |
| 130 | 8 | ~10:00 | 1% | 848 | ~5.8 |

**~9× faster.** v130 eliminates the `wasm::Type` lock convoy: system time 89%→1%,
context-switches 180M→3,599, and the work that was capped at ~2.6 parallel cores now
scales to ~10. This is the whole story — the "more cores = slower" and "weak Mac beats
strong Linux" symptoms were all downstream of the v121 contention bug, fixed by v130.
Ruled out on-box: `-O1` (still 88% system — lock is pass-independent) and fewer threads
(still ~4 effective cores — lock caps it regardless). It is the **version**, full stop.

**Remaining work = validation (run #3):** the speed win is solid, but it was measured
running v130's `-O2` on a *v121*-asyncified module. `get-wasm-opt.sh` warns that
Binaryen/emsdk skew can corrupt asyncify metadata, so the real change is to build the
**whole** asyncify+`-O2` step on v130 (now selectable via `BINARYEN_VERSION=130`) and
run the **Chrome e2e suite** to confirm the app still loads. If e2e passes, bump the
default in `get-wasm-opt.sh` (and check the emsdk-bundled Binaryen matches). If it hits
"func is not a function", the emsdk Binaryen also needs bumping.

## ✅ VALIDATED (local cold build, 2026-06-09): v130 builds a WORKING KiCad
Full from-source cold build of **all 6 tools** with `BINARYEN_VERSION=130` (no
artifacts/fixture reuse), then the KiCad e2e suite in Firefox + Chromium:

- **Build:** `BINARYEN_VERSION=130 ./docker/build.sh all --build-deps -j 3` → all 6
  tools rc=0 in ~1h04m. Each used the standalone **binaryen-130** wasm-opt for the
  asyncify+`-O2` step (confirmed in logs). pcbnew `-O2` shrank it 338 MB → 187 MB, so
  the optimizer ran correctly — **no asyncify-metadata corruption, no "func is not a
  function".** Sizes: pcbnew 187M, eeschema 99M, symbol_editor 99M, pl_editor 53M,
  gerbview 50M, calculator 38M.
- **kicad e2e (`npm run test:kicad`): 31 passed / 1 skipped / 0 failed** across
  Firefox + Chromium. Every tool renders and passes in-browser on v130.
- `-j 3` (not the default `-j 10`) is required **locally only** — Docker Desktop's
  15.6 GB VM OOM-kills the OpenCASCADE compile at `-j 10` on a fresh `--build-deps`.
  The 128 GB Hetzner CI box has no such limit and uses `-j $(nproc)`.

CI cross-check (run #4, validate workflow, `build all`) is the on-box confirmation.
NB: run #3 (eeschema-only build) showed 26 e2e failures — those were the 5 **unbuilt**
tools' missing wasm + Firefox flake, **not** a v130 regression: the identical eeschema
Firefox tests that failed there (`eeschema-ui` Delete/Backspace, text-tool dialog) all
**pass** in this full-build run. → Safe to bump the `get-wasm-opt.sh` default to 130.

## ⚠️ VERDICT (measured on-box, bench run 27197360957) — supersedes the memory theory
**The `-O2` cost is ~90% FUTEX LOCK CONTENTION inside wasm-opt, not memory
management.** `perf` on the live ccx53 shows ~92% of CPU in
`do_futex → _raw_spin_lock → native_queued_spin_lock_slowpath` — threads spinning
in the kernel on a contended lock — **identical under glibc (92%) and mimalloc
(90%)**. The lock is Binaryen's own global type mutex (`wasm::Type`), hit by every
worker thread; more threads → worse contention (a lock convoy). Hard evidence it is
**not** the allocator/THP/madvise theory below:
- `madvise` calls = **0** (perf syscall count). The "purge storm" does not exist here.
- THP `compact_stall` Δ = **0**, `thp_fault_alloc` Δ = **0**. No compaction. (THP=madvise mode.)
- `mimalloc-retain` (`MIMALLOC_PURGE_DELAY=-1`) vs baseline: **1:12:52 → 1:06:51,
  only 8% faster**, both ~88.7% system, both ~90% futex-spinlock. Allocator is irrelevant.

So **jemalloc / mimalloc / retain-configs / THP=never are all DEAD ENDS** (now proven
on-box, not just argued). The levers that can actually move wall-clock:
1. **Newer Binaryen** — the devs cut this exact `wasm::Type` contention after our
   pinned **v121** (latest is v130). Highest-value; coupled to the emsdk Binaryen,
   needs Chrome e2e. **The #1 thing to test.**
2. **Fewer threads** (`BINARYEN_CORES=4–8`) — fewer threads on the one lock = far
   less contention + far less wasted CPU. But `-O2` only does ~2.6 cores of *real*
   work at any cores (true at 8 and 32), so this is mostly an **efficiency/cost win,
   probably not a big wall-clock win** — the ~2.6× parallelism is the wall floor.
3. **Less `-O2` work** — the fixture is **188 MB** of asyncify bloat and `-O2` runs
   every pass over all of it (~11,500 CPU-s of real work = the wall floor). A lighter
   pass set (`-O1`/targeted) or a bigger asyncify removelist cuts that floor; needed
   to get toward ~30 min. Validate output in Chrome (it exists for V8's locals limit).

Everything below this section about "madvise TLB-shootdowns" and "THP compaction"
was the pre-measurement hypothesis and is **WRONG for this workload** — kept only as
the reasoning trail. Trust this section.

## (superseded hypothesis) The memory-storm theory
The `-O2` pass is **not** CPU-bound on optimization work. It is bound by a
**kernel page-management storm** on glibc Linux: the allocator constantly returns
freed pages to the OS (`madvise(MADV_DONTNEED)`/`munmap`), and each return forces
**cross-core TLB-shootdown work** (plus, on Ubuntu 24.04, very likely
Transparent-Huge-Page compaction). That work is **system (kernel) time**, it
**scales super-linearly with thread count**, and it is **largely allocator-choice
independent** — which is exactly why swapping in *default* jemalloc only helped
~15%. [SUPERSEDED: on-box perf shows the system time is futex spinlock, not
TLB-shootdowns; madvise=0, compaction=0. The "futex" the prior strace saw was
lock contention, not allocator arenas. See the VERDICT above.]

## The measurements that settle it
Per-pass `/usr/bin/time -v`, pulled from the real CI runs
(`emergence-engineering/pcbjam`, workflow "CI"). The earlier table omitted the
**context-switch and page-fault counters — those are the diagnostic gold.**

### `wasm-opt -O2` pass (the ~80-min bottleneck)
| metric | 8c glibc (run 27139529490) | 32c glibc (27144910231) | 32c jemalloc-default (27186569662) |
|---|---|---|---|
| BINARYEN_CORES | 8 | 32 | 32 |
| wall clock | **1:23:57** (5037 s) | **1:28:30** (5310 s) | 1:16:23 (4583 s) |
| user time | 12,807 s | 13,043 s | 12,177 s |
| **system time** | **18,030 s (58%)** | **114,075 s (90%)** | **95,525 s (89%)** |
| % CPU | 612% | 2393% | 2349% |
| peak RSS | 39.8 GB | 39.7 GB | 39.0 GB |
| minor page faults | 73.7 M | 67.2 M | 69.7 M |
| **voluntary ctx-switches** | **674,829,131** | **180,672,277** | **149,367,474** |
| involuntary ctx-switches | 31,675 | 1,235,328 | 1,429,770 |

### `--asyncify` pass (same machine, only ~6 GB RSS — shows the storm too)
| metric | 8c glibc | 32c glibc | 32c jemalloc-default |
|---|---|---|---|
| wall | 9:26 (566 s) | **10:51 (651 s)** | 8:28 (508 s) |
| user / system | 1,317 / 2,715 s | 1,187 / **19,045 s** | 994 / 14,630 s |
| voluntary ctx-switches | 61.8 M | 10.4 M | 7.3 M |

### What these numbers prove
1. **Cores anti-scale.** Under the *same* allocator, 32c is *slower* than 8c
   (-O2: 5310 vs 5037 s; asyncify: 651 vs 566 s). System time scales **~6.3×
   for a 4× core bump** (18k→114k) while user time stays flat. That super-linear-
   in-cores, flat-in-user-work shape is the fingerprint of **cross-core kernel
   coordination (TLB shootdowns / compaction)**, not of the optimization work.
2. **Only ~2.5 cores of real work ever happen.** user ÷ wall ≈ 12,800 ÷ 5,000 ≈
   **2.5** at both 8c and 32c. `wasm-opt -O2` barely parallelizes on this module
   (a few asyncify-created monster functions dominate — Amdahl). The other ~20
   "busy" cores at 32c are burning **kernel** time, not optimizing.
3. **The contention is allocator-independent.** glibc and jemalloc both sit at
   **150–680 M voluntary context-switches** and ~70 M page faults; jemalloc-
   default shaved only ~16% of system time and ~14% of wall. A real arena-lock
   problem would have collapsed under jemalloc. It didn't → the cost is **not**
   in the allocator's arenas.
4. **The ctx-switch *inversion* (8c=675 M vs 32c=181 M, yet 32c has 6× the system
   time)** means at 32c threads stop *sleeping* on locks and instead *spin in the
   kernel* (TLB-shootdown IPIs / page-table locks) — consistent with the
   shootdown model, not userspace mutex spinning (which would be *user* time).

## Root cause
**A kernel virtual-memory storm driven by allocator page-return traffic.** Both
glibc and (default) jemalloc periodically hand freed pages back to the OS via
`madvise(MADV_DONTNEED)`/`munmap`. On x86-64 each such return triggers a
**TLB shootdown** — the OS sends IPIs to the other cores running the process's
threads to flush their TLBs — which costs more the more cores exist (hence the
6.3× system-time blow-up 8c→32c). On Ubuntu 24.04 with THP active, faulting
threads can additionally stall in **direct compaction** (`__alloc_pages_slowpath`
→ `try_to_compact_pages`) plus background `khugepaged`. macOS doesn't hit this
(different VM/TLB + allocator, bare-metal, only 10 cores), which is why the
weaker Mac is faster. **Which term dominates — shootdowns vs compaction — was not
yet measured on this exact module; Phase 0 of the experiment plan measures it.**

## Deep-research corroboration (primary sources)
A fan-out research pass (Linux kernel THP docs, jemalloc/mimalloc tuning docs,
glibc-maintainer write-up, Binaryen issues) independently reached the same
diagnosis and supplied the key precedent:
- **Binaryen #5561** — 48-core AMD EPYC, 128 GB, Ubuntu: `wasm-opt` went
  **58m35s → 3m43s** (system time 2,395 min → 40 s) just by switching the
  allocator to **mimalloc** (which by default purges far less aggressively).
  Same anti-scaling signature as ours. This is the upside ceiling.
- **Binaryen #6338** — a 10× `wasm-opt` slowdown "fixed by using Emscripten's
  mimalloc port." Multiplicative, not marginal.
- jemalloc `TUNING.md`: decay time is "a trade-off between CPU and memory" — the
  default *keeps* issuing `madvise`, which is precisely why a default jemalloc
  swap didn't help. The fix is to **disable decay** (retain memory).
- Live alternative hypothesis (do not ignore): Binaryen's own `wasm::Type` global
  mutex (mutrace: 41.8 M locks / 10.7 M contentions in #5561) can contribute
  futex/system time **independent of malloc** — unfixed by any allocator/THP
  change. If allocator-retain + THP-off underperform, this is the next suspect,
  and it points at **upgrading Binaryen** (see below).

### Corrections from adversarial verification (don't repeat these overclaims)
- ❌ "MADV_DONTNEED broadcasts IPIs to *all* CPUs and is KVM-amplified" — refuted.
  Shootdowns go only to cores that ran the process's threads; no special VM
  penalty was substantiated. (Still scales with thread count.)
- ❌ "`dirty_decay_ms:-1` is the documented official fix and fully stops madvise"
  — failed verification; one report saw madvise persist anyway. **Every retain
  config must be strace/perf-verified to confirm madvise actually drops to ~0.**
- ❌ "THP high-system-time symptom definitely matches ours" — the *mechanism* is
  well-documented but attribution to THP for *this* module is an inference;
  measure it (Phase 0), don't assume.

## Levers, ranked (exact flags)
All are env/sysctl only (no pipeline change) except D/E. 128 GB RAM vs ~40 GB
peak makes "never return memory" safe.

| # | Lever | Exact change | Confidence | Risk |
|---|---|---|---|---|
| **A** | Fewer threads | `BINARYEN_CORES=8` (sweep 4–16) | High — *already in our data* (8c ties/beats 32c) | none |
| **B** | Allocator retain (KILLS the purge) | jemalloc `MALLOC_CONF=dirty_decay_ms:-1,muzzy_decay_ms:-1,background_thread:true` · mimalloc `MIMALLOC_PURGE_DELAY=-1` · glibc `MALLOC_TRIM_THRESHOLD_=-1 MALLOC_MMAP_MAX_=0` | High mechanism; **must verify madvise→0** | low (more RAM) |
| **C** | Disable THP | `echo never | sudo tee /sys/kernel/mm/transparent_hugepage/{enabled,defrag}` | High mechanism | low |
| **B+C** | Stack | retain allocator + THP=never | High | low |
| **D** | Newer Binaryen / bundled mimalloc | bump `BINARYEN_VERSION` past 121 (`scripts/common/get-wasm-opt.sh`) | Med (#5561/#6338) | **Med-High**: must match the emsdk Binaryen or asyncify metadata corrupts ("func is not a function"); needs e2e |
| **E** | Shrink the `-O2` input | bigger `asyncify-removelist` / `-O1`/targeted passes vs full `-O2` | Med | Med — needs Chrome e2e per change |
| — | Contention-only controls (expected to NOT fix it) | `MALLOC_ARENA_MAX=4`, `MIMALLOC_PURGE_DECOMMITS=0` | — | — |

**Lever / merely-reduces-contention distinction:** retain configs (B) *kill* the
madvise/munmap traffic; `MALLOC_ARENA_MAX` only trims arena-lock cost (which
isn't our bottleneck) — included as a control to confirm the diagnosis.

## Expected outcome / the 30-min question
At 8c, if `-O2` parallelized perfectly with zero system time it'd be ~27 min
(12,807 s ÷ 8 ÷ 60). The gap to 84 min is the storm **plus** parallelism capped
at ~2.5× — and 32c tying 8c implies **contention is what caps the parallelism**.
So killing the storm should cut system time *and* unlock real core scaling →
**~30 min is plausible** but bounded by Binaryen's true serial fraction (unknown
until measured). Going clearly below likely also needs Lever D and/or E. The
#5561 15.8× is a ceiling demonstration, not a prediction (our *user* time is real).

## The experiment plan (single Hetzner slot at a time)
Running one full CI per config is wasteful (re-compile + e2e). Instead a
dedicated bench builds the `-O2` **input once**, caches it as an artifact, and
replays the sweep over it — many data points per slot, later runs skip the build.

- **Phase 0 (in run #1):** run `baseline` with `DIAGNOSTIC=1` to attribute the
  kernel time. Decision rule:
  - `native_flush_tlb_multi` / `smp_call_function_many` high + `madvise` flood →
    **Lever B** (allocator retain).
  - `try_to_compact_pages` / `compaction_*` / `__alloc_pages_slowpath` + rising
    `/proc/vmstat:compact_stall` → **Lever C** (THP=never).
  - `wasm::Type::*` / futex with **no** madvise flood → **Lever D** (Binaryen ver).
- **Phase 1:** sweep B / C / B+C over the cached fixture; confirm each drops
  voluntary ctx-switches and `madvise` count by 10–100×.
- **Phase 2:** validate the winner end-to-end (full build + Chrome e2e) — `-O2`
  exists to keep asyncify functions under V8's locals limit, so a faster config
  that corrupts the module is worthless.

## The bench harness (how the next agent runs it)
Added in this branch:
- **`.github/workflows/wasm-opt-bench.yml`** — ephemeral Hetzner ccx53, builds-or-
  downloads the fixture, runs the sweep, uploads `o2-bench-results-<run_id>`.
  Triggers on **push to `bench/**`** (can't collide with main CI `[main]` or the
  feature-branch Hetzner CI). `concurrency: wasm-opt-bench` (one VM at a time).
- **`scripts/bench/o2-config-sweep.sh`** — replays `wasm-opt -O2` over the fixture
  under each preset; records `time -v` counters + a perf-stat `madvise/munmap`
  count; `DIAGNOSTIC=1` adds vmstat/interrupts deltas + a perf kernel-symbol
  sample. Preset menu is in the script's `config_env`.
- **`scripts/bench/sweep.conf`** — committed run parameters (CONFIGS / CORES /
  FIXTURE_RUN_ID / DIAGNOSTIC). Edit + commit + push to `bench/**` to launch.

**Drive it:**
1. Run #1 (this branch): `CONFIGS_CONF="baseline mimalloc-retain"`,
   `FIXTURE_RUN_ID_CONF=""` (builds + caches fixture), `DIAGNOSTIC_CONF=1`.
2. Note run #1's id → set `FIXTURE_RUN_ID_CONF` to it so later runs skip the
   ~40-min build, then sweep `"thp-off jemalloc-retain glibc-retain
   thp-off+mimalloc-retain"`.
3. Core sweep the winner: `CONFIGS="<winner>"`, `CORES_CONF=8` (then 16, 4).

Pull results: `gh run download <run_id> -n o2-bench-results-<run_id>`, read
`results.csv` (the `vol_ctxsw`, `sys_s`, and `madvise` columns tell the story).

## Why local QEMU is NOT a fair proxy (asked & answered)
The existing `scripts/bench/` QEMU harness can only run the *asyncify* pass
locally — three independent blockers for `-O2`: (1) **RAM** — `-O2` needs ~40 GB,
a 32 GB Mac caps a guest at ~20 GB → OOM; (2) **arch** — the storm is x86 IPI
TLB-shootdowns, but Apple Silicon is aarch64 with hardware-broadcast `TLBI`
(different mechanism/scaling); (3) **core count** — can't reach the 32-core regime
where it's worst. Use QEMU only for the free "does this config stop madvise"
smoke test on the asyncify pass; do the real numbers on the disposable ccx53.

## Branch / repo state at handoff
- Bench work on branch **`bench/wasm-opt-allocator-sweep`** (off
  `istvanmatejcsok/feat/ci-hetzner-allcores`). Adds the three files above.
- CI uses **standalone Binaryen v121** (`get-wasm-opt.sh` fallback; CI has no
  local emsdk). Lever D must keep this in sync with the emsdk Binaryen.
- Repo `emergence-engineering/pcbjam`. Main CI runs on **Ubicloud** (`[main]`);
  the Hetzner ccx53 only spins for the two `ci-hetzner` feature branches and the
  new `bench/**` branch. "One Hetzner slot at a time" is the operative limit.
- `apply-asyncify.sh`: jemalloc auto-preload + `ASYNCIFY_ONLY=1` (stops before
  -O2 — used to build the bench fixture) + `WASM_OPT_PRELOAD=none` sentinel.

## Useful commands
- Pull per-pass timing from a finished CI run's live log:
  `gh run view <id> --repo emergence-engineering/pcbjam --log | grep -iE "Running wasm-opt|User time|System time|Elapsed \(wall|Maximum resident|context switches|page faults"`
- Older runs (no live log): `gh run download <id> -n e2e-logs` → `logs/build/*.log`.
- Launch a bench run: edit `scripts/bench/sweep.conf`, commit, `git push` to the
  `bench/**` branch.