pcbjam/docs/features/async/22-absorbing-libcontext.md

22 — Absorbing libcontext: one scheduler for JS ↔ Asyncify ↔ fibers

Status: PLAN (2026-08-06), not started. Continues 20 after its D-1/D0/D1 landed, its D2 was reverted, and its D3 met its goal by other means. Read 20 §10 (work log) and 21 (park-site table) first; this doc assumes both.

Decisions already taken by the user: absorb (the scheduler takes over libcontext's wasm backend — one fiber system, not two cooperating ones), and the work continues on feature/async-mailbox.

1. The bug this exists to kill

The blue screen: RuntimeError: index out of bounds (and its siblings unreachable executed, indirect call to null) raised from Asyncify.doRewind, killing the tab during board loads. Its shape, in the field logs' own words: a context gets recovered twice, or recovered by a different fiber than the one that parked it, or recovered from a buffer another party has since overwritten (aliased-wake-live: restoring currData=X over Y).

This is not doc 19's hang. Doc 19 is fixed (a quasi-modal's nested loop no longer parks on a coroutine stack), and that fix removed one ingredient — concurrent parks on fiber stacks — without touching the cause. Anyone reading doc 20 alone may conclude the migration is optional; measured against the blue screen it is not.

2. Why the guards cannot fix it (the real diagnosis)

Asyncify.currData is a single global slot naming "the context currently unwinding or rewinding". Today three layers each decide swaps and resumes with partial information:

layer	decides	guards it invented
libcontext (kicad/thirdparty/libcontext/libcontext.cpp)	every fiber↔fiber swap	swap_suspended, jump-refused-parked, jump-refused-hot-main, ghost-resume epochs
the JS shim (scripts/common/shims/asyncify-scheduler.js)	wake delivery, resume admission	__internallyParked, __parkSleepBuf, __validSuspensions, fiber-resume-refused, aliased-wake-live
wx (wxwidgets/src/wasm/evtloop.cpp)	whether dispatch may run	wxWasmDispatchDepth interlock, zero/restore around parks

The bookkeeping is already centralised — the decision is not. S2 made currData a single-writer accessor with a stray tripwire, and the shim already wraps Fibers.finishContextSwitch and authorises the write. So the shim sees every swap; what it cannot do is prevent one, because the decision to swap was made in libcontext, which never told it. Each layer is therefore reduced to answering "is entering this fiber safe?" by inference, and each invented its own heuristic. That thicket is the patchwork.

Centralising means moving the decision, not adding more observation.

3. Why the phases knotted (do not repeat this)

Doc 20 sequenced D2 (dispatch on a context) → D3 (waits yield contexts). Both orders fail:

• D2 alone (measured, reverted): a quasi-modal opened from a tick handler suspends the dispatch context inside a still-in-place wait, adding an Asyncify layer beneath every libcontext fiber. The coroutine-nested battery died at fiber_create_run_destroy_inside_modal (aliased-wake-live → fiber-resume-refused). Growing a context pool made it worse (8 contexts burned in 30 ms — a context suspended in a modal is consumed for that modal's lifetime); falling back to entry-stack dispatch did not help either, because the extra layer exists the moment the context is suspended.
• D3 alone (measured, guarded): a wait must yield the context that owns its stack, and the doc-19 path's owner is a tool coroutine, not a scheduler context. yield_park now refuses that case (foreignStackRefusals) rather than saving the tool fiber's stack into the host context's fiber struct.

So the dependency is a knot: dispatch needs waits migrated, waits need tool coroutines to be contexts, tool coroutines are libcontext's. Absorbing libcontext is what unties it, which is why it comes first here rather than last.

4. Target architecture

browser tick ─► scheduler.drain()          ONE decision point
                  │  registry = truth; at most one transition in flight
                  ├─► dispatch context     (wx handlers)
                  ├─► tool contexts        (KiCad COROUTINEs — libcontext's clients)
                  ├─► wait contexts        (modal / nested / popup)
                  └─► bridge contexts      (lib, 3D, occ, ngspice, clipboard, font)

Invariants, all already implemented and tested in wx/wasm/private/sched_context.h:

1. Star, not mesh — contexts yield only to the scheduler; only the scheduler resumes. A resume is a registry lookup, never an inference.
2. One transition in flight; drain() is not re-entrant.
3. Wakes never resume inline — mark_ready queues, drain resumes from a clean stack.
4. A context yields only its own stack — enforced by stack-range ownership check.
5. Buffers are owned: one context, one stack, one asyncify buffer, never shared.

5. Phases

Estimates assume one engineer familiar with this stack, and are grounded in this run's measured cycle costs (§8). The A / BCD / E / F split matters: A is landable alone because it is behaviour-preserving; B+C+D must land together (§3).

Phase A — absorb libcontext's wasm backend (3–5 d) · landable alone

Replace the body of libcontext's wasm implementation so make_fcontext / jump_fcontext / release_fcontext become thin adapters over pcbjam_sched, keeping the exact same observable semantics (symmetric swap, same return values, same INVOCATION_ARGS contract). The registry now knows every tool fiber: its stack range, its buffer, its status.

• Keep libcontext's existing guards as tripwires that must never fire; do not delete.
• No caller changes. KiCad still calls jump_fcontext; TOOL_MANAGER is untouched.
• Gate: coroutine trio + coroutine-nested + races + full kicad suite unchanged, and zero tripwire firings across the fuzz suites.
• Why it is safe alone: semantics are identical; only the party doing the bookkeeping changes. This is the de-risking step D2 never had.

Phase B — tool coroutines become scheduler contexts (1–2 wk) · the bulk

Flip the topology from symmetric to star: COROUTINE::Resume/Call become mark-ready-and-drain; a coroutine yields to the scheduler rather than to its resumer.

• The seam is CALL_CONTEXT/doCall/doResume in kicad/include/tool/coroutine.h plus TOOL_MANAGER's m_activeState handling.
• RunMainStack must keep working — it is now "run on the scheduler's caller context".
• Expect this to be most of the risk and most of the calendar time.

Seam read in full after Phase A landed (2026-08-06) — the crux, stated precisely. TOOL_MANAGER's contract is SYNCHRONOUS and the star's is not, and that single mismatch is the phase:

• dispatchInternal does st->cofunc->Call( st->initialEvent ) and then, on the very next line, if( !st->cofunc->Running() ) finishTool( st ) (common/tool/tool_manager.cpp:870-874). So Call must run the coroutine to its first yield before returning. Same shape at ShutdownTool (:592) and the pending-wait resume (:808).
• The star says a resume happens from a clean stack via drain(), and drain() deliberately refuses re-entry — a context calling it would turn the star into a cycle. But dispatch RUNS on a stack the scheduler may own (that is Phase D), so "Call → drain" is re-entrant by construction unless dispatch is already a context. This is the same knot doc 20 §3 measured from the other side, and it is why B, C and D cannot be separated.
• CALL_CONTEXT::Continue's CONTINUE_AFTER_ROOT loop (coroutine.h:175-183) is a second, hand-rolled scheduler: the coroutine jumps to the ROOT stack, the root runs m_mainStackFunction, then resumes the coroutine. Under the star this is just "park with a reason, let the scheduler run the functor, mark ready" — the loop should disappear rather than be ported.

A Phase A measurement that changes B's shape: grace-ring-over-capacity: 33 — KiCad holds 30+ coroutines that never finished and whose owners are gone, and still jumps into them. Under the star every one needs a live context (128 KB + buffer each, vs libcontext's 512 K), so B must fix the lifetime, not just the topology: give COROUTINE an owning handle and destroy the context when the tool state is popped. The grace ring is the measurement of exactly how much that is worth, and it should fall out of the tree at the end of B, not survive into C.

Divergence policy (§10 q4), now answerable: Phase A touched KiCad only inside thirdparty/libcontext — vendored code, no upstream conflict. B is the first phase to restructure coroutine.h + tool_manager.cpp, which CLAUDE.md's "keep the fork close to upstream" rule makes a real cost. Decide before starting: either accept the divergence, or keep the star adapter entirely inside libcontext + sched_context.h and leave both KiCad files untouched, which is likely possible because the whole seam is jump_fcontext calls.

Phase C — waits yield the owning context (3–5 d)

wxWasmYieldUntil becomes a context yield: look up the owner of the current stack (the mechanism exists), park it, and let resolveWait mark it ready. The doc-19 main-stack bounce (wx/wasm/private/mainstack.h) becomes unnecessary and should be removed in the same flip — it is a mitigation for exactly the condition this phase eliminates.

Phase D — dispatch on a context (3–5 d)

Doc 20's D2, retried. The reverted implementation is recoverable from git (design-b D2 in the work log) and was correct apart from its dependencies; the pool idea in it is wrong and must not come back — one dispatch context suffices once waits yield.

B + C + D land as ONE commit. Each alone regresses. Build them on a sub-branch, keep every guard as a tripwire, flip once, gate hard.

Sub-branch feature/async-star-flip (2026-08-06): the core mechanism is built and proven, the wiring is not. Landed there: fiber_transfer(from, to, value) — libcontext's symmetric swap expressed as a star transition (park the source, make the target runnable, the scheduler performs every entry) — plus fiber_start (the lane's entry point, since a transfer needs a running context to park) and drain_all (the top-level pump, because a star transition only makes work runnable).

The question that decided whether B is possible at all is answered YES and pinned by star_transfer_call_is_synchronous: parking the caller and resuming it when the callee yields is indistinguishable, in the caller's own C++ frame, from a synchronous return. So TOOL_MANAGER's Call(); if( !Running() ) contract survives the flip untouched — which is what makes a KiCad-minimal Phase B realistic.

The constraint that fixes the order of the remaining wiring — ONE root, not two. ensure_scheduler_context() adopts whatever stack first calls drain(), and libcontext's ensure_main_context() adopts whatever stack first calls jump_fcontext. In production both are the main stack, so the scheduler fiber and the libcontext root would be two emscripten_fiber_ts describing the SAME stack — mutual corruption the moment either is entered. In Phase A this was harmless (nothing in KiCad ever called drain()); at the flip it is fatal. Therefore:

1. D first, inside the flip: dispatch moves onto its own context, so the main stack is only ever the scheduler and nothing else runs there.
2. libcontext's root then adopts the running context (the dispatch context), never the main stack.
3. Only then may jump_fcontext become fiber_transfer, because every caller is now provably on a context and can park.
4. C follows naturally: wxWasmYieldUntil yields the owning context, resolveWait marks it ready, and the doc-19 mainstack bounce comes out.

The tick becomes fiber_start(dispatch, …) + drain_all() from a fresh JS task.

Phase E — bridges on contexts (1 wk)

Doc 21 §1's K1–K7 and W4/W5 through one wasm_await_promise-style helper. After this no handleSleep park happens on any fiber stack — assert it. Take each bridge's deep-park high-water from its own beacon and size buffers from that (doc 21 §2b); D1's ~34 B/frame is a synthetic floor, not a budget.

Phase F — delete the guard thicket (2–3 d)

Only now, and only guards that have been provably silent: libcontext's refusals, the shim's quarantine/consume-once, the wx interlock and its zero/restore sites, the open/fiber-busy gates. Flip fiber-resume-park.spec.ts from "refused" to "resumed after its park resolves".

Total: ~4–6 weeks, of which B is half.

6. Gates — what "done" means

The batteries never reproduced the blue screen, so they cannot certify it. Use all three:

1. The crash's own repro (the only proof that matters): REPRO_PROFILE + a ≥120 MB project + 2–4 warm loads; watch for the 68/N counters and kill-fiber signature. Target: rootHotTotal == 0, zero aliased-wake-live, zero refusals.
2. The full net: kicad suite, wx battery, coroutine trio, races, sched-context, quasimodal-strand (now a green regression pin), the drift/collab fuzz suites.
3. Tripwire silence: every guard retained during A–E must record zero firings across a full fuzz run before Phase F deletes it.

Known-pre-existing failures to expect and NOT chase: occ-probe glb format matrix, and e2e/modal.spec.ts:125 (environment-sensitive; fails on unchanged binaries).

7. Invariants and traps learned in this run

Hard-won; each cost at least one build+test cycle.

1. Fiber C stacks must be 16-byte aligned. EM_ASM puts its argument buffer on the running stack and the glue asserts buf % 16 == 0; emscripten's malloc gives 8. A misaligned context traps in readEmAsmArgs on every EM_ASM (found via 107 wx failures). AlignedBuffer in sched_context.h handles it — do not hand a fiber a std::vector<char>::data().
2. emscripten_stack_get_base()/end() describe the CURRENT stack, not the main one. finishContextSwitch calls emscripten_stack_set_limits with the incoming fiber's bounds, so a live query reports "on the main stack" from everywhere and detects nothing — silently. Capture the main stack's bounds once, where you are provably on it.
3. Every emscripten_fiber_t needs its own asyncify buffer, including the host side of a swap. A zero-initialised one has a null buffer and traps on first swap. This briefly looked like proof that fiber-on-context nesting was impossible; it is not — fiber_nests_in_context pins that it works.
4. A wait must yield the context that owns its stack. Yielding a context you merely sit on top of saves the wrong stack into its fiber struct: silent, total, undetectable after the fact.
5. Partial migration is worse than none — measured twice (§3). If a change leaves one participant parking in place while another expects contexts, it will regress.
6. A parked context must not be freed, and a resume must never be inferred. Both are registry lookups; keep them that way.
7. Test-infrastructure trap: tests/apps/gal-webgl + build-wasm/sysroot hold gitignored artifacts no normal build produces. Losing them looks like a 29-test mass regression. Fix: scripts/setup-worktree.sh. When triaging a mass failure, first run find tests/apps -name '*.wasm' -newermt <last known good> — it settles "my change vs the environment" in seconds.

8. Cycle costs (for planning)

Measured on the dev Mac, warm caches: wx rebuild ~5 min · all wx test apps ~10 min · docker/build.sh kicad_editor ~15 min · wx+asyncify+coroutine battery ~5 min · full kicad suite ~8 min. So one full "change → verdict" loop on a KiCad-side change is ~25–30 min, and a wx-only change ~20 min. Budget phases accordingly; batch experiments per build.

9. Where the pieces are

what	where
context primitives + registry + memory gate	wxwidgets/include/wx/wasm/private/sched_context.h (header-only)
its harness + gate	tests/apps/standalone/sched-context/, tests/asyncify/sched-context.spec.ts
the scheduler shim (mailbox, waits, wakes, guards)	scripts/common/shims/asyncify-scheduler.js
libcontext's wasm backend (Phase A's target)	kicad/thirdparty/libcontext/libcontext.cpp
tool coroutine seam (Phase B)	kicad/include/tool/coroutine.h, kicad/common/tool/tool_manager.cpp
wait bridges (Phase C)	wxwidgets/src/wasm/evtloop.cpp, dialog.cpp, window.cpp
the doc-19 mitigation to remove at Phase C	wxwidgets/include/wx/wasm/private/mainstack.h, wasm/bindings/main_stack_runner.h, TOOL_MANAGER::RunOnMainStackIfActiveTool
park-site inventory	21 §1
doc-19 regression pin	tests/kicad/quasimodal-strand.spec.ts

Branch: feature/async-mailbox, unpushed across 6 repos. Baseline tags: mailbox-s0-baseline, d-1-pre-delete. Tag before each phase's flip.

10. Work log

Phase A — DONE, gate met (2026-08-06) ✅

Final gate: full kicad suite 139 passed / 1 failed — the failure is the known pre-existing occ-probe glb format matrix, i.e. the baseline exactly. wx battery 346 passed / 1 failed (pre-existing e2e/modal.spec.ts:125) / 3 skipped; coroutine trio + nested + races + asyncify 48/48; sched-context 2/2 including three new fiber-lane scenarios. Tripwire sweep over every suite log: zero sched-divergence-*, zero swap-lost, zero fiber_swap() refusals, zero FIBER-SWAP-NONENTERABLE, zero FIBER-RELEASE-RUNNING, zero jump-into-reclaimed, zero grace-ring-evict. The registry's view agreed with libcontext's protocol on every swap in the suite — which is the whole claim Phase A had to earn before Phase B moves the decision.

Landed (working tree, uncommitted):

• sched_context.h grew the fiber lane: fiber_adopt_current / fiber_create (caller-owned C stack adopted, registry-owned asyncify buffer) / fiber_swap / fiber_enterable / fiber_release, with its own counters so the D1 star memory gate keeps meaning what it meant. Suspended-capture high-water is sampled before the resume consumes it (afterwards it always reads 0) — that number is Phase E's sizing input.
• libcontext.cpp's wasm backend became the adapter: wasm_fcontext keeps only protocol state (return_to, transfer_value, epochs, refcounts); the emscripten_fiber_t + buffer live in the registry; every swap (jump and trampoline return) goes through pcbjam_sched::fiber_swap. swap_suspended stays AUTHORITATIVE for the parked-jump refusal and the registry only observes, beaconing sched-divergence-* on disagreement (see bug 3).
• thirdparty/libcontext/CMakeLists.txt gained the wx include path under EMSCRIPTEN. Harness + spec: 3 fiber-lane scenarios, fiber stats assertions.

Four real bugs, every one caught by a tripwire rather than by guessing. Each is worth reading before Phase B, because three of them are the same mistake:

1. Never infer the swap's from side. The first cut derived it from the lane's "current", which goes stale across a handleSleep park exactly like g_current_context does — the wrong context got marked Suspended, the real swapper stayed "Running" forever, and every later jump into it was wrongly refused (a dispatch the legacy guard would have allowed). libcontext KNOWS who is swapping out; fiber_swap(from, to) now mirrors its answer, keeping the two layers in lockstep by construction.

2. A release must always release. libcontext's refcount drop deletes the struct unconditionally; the registry refusing a "Running" release while the caller frees anyway left a permanent ghost (plus a dangling g_current_context — the garbage-id beacon) that poisoned every later enterability answer. Now: release always, FIBER-RELEASE-RUNNING beacon.

3. The registry must not OVERRULE the protocol, only record it. Keying the parked-jump refusal on fiber_enterable() refused a dispatch the legacy backend permitted: Symbol Properties stopped opening (quasimodal-strand, a fiber the registry still called Running while swap_suspended said validly suspended). Phase A moves the BOOKKEEPING, not the decision. swap_suspended is authoritative again; every disagreement is now a recorded fact instead of a behaviour change, and those recordings are the evidence Phase B's flip gets designed on.

4. Absorbing the buffer un-hid a KiCad use-after-free. TOOL_MANAGER keeps raw fcontext_ts that outlive the COROUTINE owning them and jumps into them after the last refcount is gone. This "worked" for years for a reason worth writing down: the freed block was ~512 K (the asyncify buffer lived INSIDE the struct), and malloc parks that size in a large bin, so freed stayed readable essentially forever. Moving the buffer into the registry left a ~64-byte struct — recycled on the next call — so sched_id read back as a freelist pointer (0x16554B0), the swap was refused, and EVERY canvas tool wedged ("click never landed a selection", "box-select never selected anything", 19 suite failures). The change did not introduce the bug; it removed the size accident that concealed it.

The rule all four share, and the one to carry into B–E: the registry may only record what the protocol actually did — it may not guess it, and it may not refuse anything the legacy code permitted. Phase A is behaviour-preserving or it is nothing.

The grace ring (bug 4's containment, and a Phase B input). Released contexts are kept, not freed: the struct is never deleted (protocol state only now — tens of bytes, cheaper than the accident it replaces, and it makes a stale jump land on a real sched_id), and the FIBER (512 K + registry entry) is bounded by a 32-entry ring. Eviction takes only coroutines that actually FINISHED — evicting by age alone dropped long-lived tool coroutine #1 that TOOL_MANAGER re-enters, and m-move plus lock-resist failed. An evicted context keeps its struct with sched_id zeroed so a later jump takes the ghost contract callers already handle.

Measured, and it is the number Phase B needs: grace-ring-over-capacity: 33 fired in 4 specs — i.e. 33 released coroutines were live with NONE evictable, because none had finished. So KiCad really does hold 30+ never-finished coroutines whose owners are gone, and the ring cannot reclaim them: worst case that is ~16 MB of retained asyncify buffers, and it grows with the working set rather than being capped by the ring. This is not a leak Phase A may fix (that would change behaviour); it is exactly what Phase B removes by giving coroutines scheduler-owned lifetimes, and it should be re-measured after that flip.

Process traps paid for in this run:

• Build against the libs volume (COMPOSE_PROJECT_NAME=kicad-wasm-libs). A bare docker/build.sh creates a cold per-branch volume whose sysroot lacks glm and dies at the gl1 shim.
• npx playwright test does NOT sync output/*.wasm into tests/apps/kicad/; only npm run test:kicad (which runs setup:kicad first) does. A retest after a rebuild that shows byte-identical beacons is testing the OLD binary — this cost a full diagnosis cycle.
• A Docker VM at 100% disk fails builds with exit 137 and simultaneously crash-loops the user's unrelated postgres containers (could not write lock file). Diagnose and report; never prune or delete Docker state to clear it.

Baseline tags phase-a-pre on root/pcbjam/kicad/wxwidgets.

11. Open questions

1. pthreads. Doc 21 §2 settled that every Asyncify park is main-thread and the lib bridge's worker path is a blocking proxy. Phase A must re-check that libcontext is never driven from a worker before assuming the scheduler is main-thread-only.
2. Buffer sizing. Still unmeasured for real park sites (§5 Phase E). libcontext's 512 K is inherited, D1's 128 K is a guess with a synthetic floor behind it.
3. D5 (main loop as a context) stays optional. The 68/1 class is currently held closed by v0.1.28's scheduling trick; decide with Phase E's telemetry in hand, not before.
4. Does Phase B change upstream-divergence policy? It touches coroutine.h and tool_manager.cpp structurally. Worth deciding up front whether that divergence is acceptable or whether the adapter should stay entirely inside libcontext.