From 586fbadfe9064d220ea39afa7498aae1460620d5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Gerg=C5=91=20T=C3=B6rcsv=C3=A1ri?= Date: Thu, 6 Aug 2026 20:19:42 +0200 Subject: [PATCH] design-b Phase A: absorb libcontext's wasm backend (doc 22) Bumps wxwidgets (fiber lane in sched_context.h) and kicad (libcontext's wasm backend becomes an adapter over it), so there is now ONE party performing every emscripten_fiber_swap and recording who is on the CPU. Behaviour is preserved throughout - libcontext still decides, the registry observes and beacons any disagreement - which is the de-risking step doc 20's D2 never had. Harness + spec gain three fiber-lane scenarios: a fresh fiber enters at its entry and a swap suspends the swapper (fiber_roundtrip), releasing a suspended fiber is legal while a stale id refuses instead of use-after-free (fiber_release_suspended), and symmetric swaps leave a parked star context undisturbed (fiber_and_star_coexist). The spec asserts the lane's counters, including the tripwire that must stay zero: fiberNonEnterableSwaps. Doc 22 gains the Phase A work log: the gate (kicad suite 139/1, the 1 being the pre-existing occ-probe glb matrix; wx 346/1/3; batteries 48/48; zero tripwires anywhere), the four bugs the tripwires caught and the single rule behind three of them, the grace ring and the 33-coroutine measurement Phase B needs from it, and the process traps this run paid for. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EarUW9DS1c1sSW4ZNrkGQS --- .../features/async/22-absorbing-libcontext.md | 107 ++++++++++- kicad | 2 +- .../sched-context/sched_context_test.cpp | 173 ++++++++++++++++++ tests/asyncify/sched-context.spec.ts | 41 +++++ wxwidgets | 2 +- 5 files changed, 322 insertions(+), 3 deletions(-) diff --git a/docs/features/async/22-absorbing-libcontext.md b/docs/features/async/22-absorbing-libcontext.md index 639aa62..a1033ff 100644 --- a/docs/features/async/22-absorbing-libcontext.md +++ b/docs/features/async/22-absorbing-libcontext.md @@ -212,7 +212,112 @@ a wx-only change ~20 min. Budget phases accordingly; batch experiments per build Branch: `feature/async-mailbox`, unpushed across 6 repos. Baseline tags: `mailbox-s0-baseline`, `d-1-pre-delete`. Tag before each phase's flip. -## 10. Open questions +## 10. Work log + +### Phase A — DONE, gate met (2026-08-06) ✅ + +Final gate: full kicad suite **139 passed / 1 failed** — the failure is the known +pre-existing `occ-probe` `glb` format matrix, i.e. the baseline exactly. wx battery +346 passed / 1 failed (pre-existing `e2e/modal.spec.ts:125`) / 3 skipped; coroutine +trio + nested + races + asyncify 48/48; sched-context 2/2 including three new +fiber-lane scenarios. **Tripwire sweep over every suite log: zero +`sched-divergence-*`, zero `swap-lost`, zero `fiber_swap()` refusals, zero +`FIBER-SWAP-NONENTERABLE`, zero `FIBER-RELEASE-RUNNING`, zero +`jump-into-reclaimed`, zero `grace-ring-evict`.** The registry's view agreed with +libcontext's protocol on every swap in the suite — which is the whole claim Phase A +had to earn before Phase B moves the decision. + +**Landed (working tree, uncommitted):** + +- `sched_context.h` grew the **fiber lane**: `fiber_adopt_current` / `fiber_create` + (caller-owned C stack adopted, registry-owned asyncify buffer) / `fiber_swap` / + `fiber_enterable` / `fiber_release`, with its own counters so the D1 star memory + gate keeps meaning what it meant. Suspended-capture high-water is sampled + **before** the resume consumes it (afterwards it always reads 0) — that number is + Phase E's sizing input. +- `libcontext.cpp`'s wasm backend became the adapter: `wasm_fcontext` keeps only + protocol state (return_to, transfer_value, epochs, refcounts); the + `emscripten_fiber_t` + buffer live in the registry; every swap (jump and + trampoline return) goes through `pcbjam_sched::fiber_swap`. `swap_suspended` + stays AUTHORITATIVE for the parked-jump refusal and the registry only observes, + beaconing `sched-divergence-*` on disagreement (see bug 3). +- `thirdparty/libcontext/CMakeLists.txt` gained the wx include path under + EMSCRIPTEN. Harness + spec: 3 fiber-lane scenarios, fiber stats assertions. + +**Four real bugs, every one caught by a tripwire rather than by guessing.** Each is +worth reading before Phase B, because three of them are the same mistake: + +1. **Never infer the swap's `from` side.** The first cut derived it from the + lane's "current", which goes stale across a handleSleep park exactly like + `g_current_context` does — the wrong context got marked Suspended, the real + swapper stayed "Running" forever, and every later jump into it was wrongly + refused (a dispatch the legacy guard would have allowed). libcontext KNOWS who + is swapping out; `fiber_swap(from, to)` now mirrors its answer, keeping the + two layers in lockstep by construction. +2. **A release must always release.** libcontext's refcount drop deletes the + struct unconditionally; the registry refusing a "Running" release while the + caller frees anyway left a permanent ghost (plus a dangling + `g_current_context` — the garbage-id beacon) that poisoned every later + enterability answer. Now: release always, `FIBER-RELEASE-RUNNING` beacon. + +3. **The registry must not OVERRULE the protocol, only record it.** Keying the + parked-jump refusal on `fiber_enterable()` refused a dispatch the legacy + backend permitted: Symbol Properties stopped opening (`quasimodal-strand`, + a fiber the registry still called Running while `swap_suspended` said validly + suspended). Phase A moves the BOOKKEEPING, not the decision. `swap_suspended` + is authoritative again; every disagreement is now a recorded fact instead of a + behaviour change, and those recordings are the evidence Phase B's flip gets + designed on. +4. **Absorbing the buffer un-hid a KiCad use-after-free.** `TOOL_MANAGER` keeps raw + `fcontext_t`s that outlive the `COROUTINE` owning them and jumps into them after + the last refcount is gone. This "worked" for years for a reason worth writing + down: the freed block was ~512 K (the asyncify buffer lived INSIDE the struct), + and malloc parks that size in a large bin, so freed stayed readable essentially + forever. Moving the buffer into the registry left a ~64-byte struct — recycled + on the next call — so `sched_id` read back as a freelist pointer (`0x16554B0`), + the swap was refused, and EVERY canvas tool wedged ("click never landed a + selection", "box-select never selected anything", 19 suite failures). The change + did not introduce the bug; it removed the size accident that concealed it. + +**The rule all four share, and the one to carry into B–E: the registry may only +record what the protocol actually did — it may not guess it, and it may not refuse +anything the legacy code permitted.** Phase A is behaviour-preserving or it is +nothing. + +**The grace ring (bug 4's containment, and a Phase B input).** Released contexts are +kept, not freed: the struct is never deleted (protocol state only now — tens of +bytes, cheaper than the accident it replaces, and it makes a stale jump land on a +real `sched_id`), and the FIBER (512 K + registry entry) is bounded by a 32-entry +ring. Eviction takes only coroutines that actually FINISHED — evicting by age alone +dropped long-lived tool coroutine #1 that `TOOL_MANAGER` re-enters, and `m`-move +plus lock-resist failed. An evicted context keeps its struct with `sched_id` zeroed +so a later jump takes the ghost contract callers already handle. + +**Measured, and it is the number Phase B needs:** `grace-ring-over-capacity: 33` +fired in 4 specs — i.e. 33 released coroutines were live with NONE evictable, +because none had finished. So KiCad really does hold 30+ never-finished coroutines +whose owners are gone, and the ring cannot reclaim them: worst case that is ~16 MB +of retained asyncify buffers, and it grows with the working set rather than being +capped by the ring. This is not a leak Phase A may fix (that would change +behaviour); it is exactly what Phase B removes by giving coroutines +scheduler-owned lifetimes, and it should be re-measured after that flip. + +**Process traps paid for in this run:** + +- Build against the **libs volume** (`COMPOSE_PROJECT_NAME=kicad-wasm-libs`). A bare + `docker/build.sh` creates a cold per-branch volume whose sysroot lacks glm and + dies at the gl1 shim. +- `npx playwright test` does NOT sync `output/*.wasm` into `tests/apps/kicad/`; only + `npm run test:kicad` (which runs `setup:kicad` first) does. A retest after a + rebuild that shows byte-identical beacons is testing the OLD binary — this cost a + full diagnosis cycle. +- A Docker VM at 100% disk fails builds with exit 137 and simultaneously crash-loops + the user's unrelated postgres containers (`could not write lock file`). Diagnose + and report; never prune or delete Docker state to clear it. + +Baseline tags `phase-a-pre` on root/pcbjam/kicad/wxwidgets. + +## 11. Open questions 1. **pthreads.** Doc 21 §2 settled that every Asyncify park is main-thread and the lib bridge's worker path is a blocking proxy. Phase A must re-check that libcontext is never diff --git a/kicad b/kicad index 2c777ef..73dc150 160000 --- a/kicad +++ b/kicad @@ -1 +1 @@ -Subproject commit 2c777efedee85b5ee5cd8e46fbd10736cecc1729 +Subproject commit 73dc150bd459983ece0825fddc0c857c506b5199 diff --git a/tests/apps/standalone/sched-context/sched_context_test.cpp b/tests/apps/standalone/sched-context/sched_context_test.cpp index c3275b4..417ad8d 100644 --- a/tests/apps/standalone/sched-context/sched_context_test.cpp +++ b/tests/apps/standalone/sched-context/sched_context_test.cpp @@ -523,6 +523,176 @@ void Scenario_ForeignStackRefused() report( "foreign_stack_refused", true ); } +// --------------------------------------------------------------------------- +// FIBER LANE scenarios (doc 22 Phase A) — the symmetric semantics libcontext's +// wasm backend adapts onto. These pin what the adapter relies on: a fresh +// fiber enters at its entry, a swap suspends the swapper, enterability is +// registry truth, releasing a suspended fiber is legal, a stale id refuses +// instead of use-after-free, and the star lane is undisturbed throughout. +// --------------------------------------------------------------------------- +ContextId g_fiber_root = 0; + +ContextId fiber_root() +{ + // Adopted once for the whole battery, like libcontext's main context. + if( !g_fiber_root ) + g_fiber_root = fiber_adopt_current( 64 * 1024, "test-root" ); + + return g_fiber_root; +} + +alignas( 16 ) char g_fiber_a_stack[64 * 1024]; +ContextId g_fiber_a = 0; +int g_fiber_a_runs = 0; +int g_fiber_a_saw_current = -1; +int g_fiber_a_saw_root_enterable = -1; +int g_fiber_a_saw_self_enterable = -1; + +void fiber_a_entry( void* ) +{ + // Never returns, like libcontext's trampoline: run, swap back, repeat. + for( ;; ) + { + ++g_fiber_a_runs; + g_fiber_a_saw_current = ( fiber_current() == g_fiber_a ) ? 1 : 0; + g_fiber_a_saw_root_enterable = fiber_enterable( g_fiber_root ) ? 1 : 0; + g_fiber_a_saw_self_enterable = fiber_enterable( g_fiber_a ) ? 1 : 0; + fiber_swap( g_fiber_a, g_fiber_root ); + } +} + +void Scenario_FiberRoundtrip() +{ + if( !fiber_root() ) + return report( "fiber_roundtrip", false, "could not adopt the root" ); + + g_fiber_a_runs = 0; + g_fiber_a = fiber_create( fiber_a_entry, nullptr, g_fiber_a_stack, + sizeof( g_fiber_a_stack ), 32 * 1024, "fiber-a" ); + + if( !g_fiber_a ) + return report( "fiber_roundtrip", false, "fiber_create failed" ); + + if( status_of( g_fiber_a ) != Status::Fresh || !fiber_enterable( g_fiber_a ) ) + return report( "fiber_roundtrip", false, "a fresh fiber should be enterable" ); + + if( g_fiber_a_runs != 0 ) + return report( "fiber_roundtrip", false, "body ran before the first swap" ); + + if( !fiber_swap( g_fiber_root, g_fiber_a ) ) + return report( "fiber_roundtrip", false, "first swap refused" ); + + // Back here: the fiber ran once and swapped back to the root. + if( g_fiber_a_runs != 1 ) + return report( "fiber_roundtrip", false, + "runs=" + std::to_string( g_fiber_a_runs ) + " after first swap" ); + + if( fiber_current() != g_fiber_root ) + return report( "fiber_roundtrip", false, "root is not fiber_current() again" ); + + if( status_of( g_fiber_a ) != Status::Suspended || !fiber_enterable( g_fiber_a ) ) + return report( "fiber_roundtrip", false, + std::string( "suspended fiber status: " ) + + status_name( status_of( g_fiber_a ) ) ); + + // What the fiber observed mid-run: it was current, the suspended root was + // enterable, and it itself (Running) was not — registry truth, no guessing. + if( g_fiber_a_saw_current != 1 ) + return report( "fiber_roundtrip", false, "fiber did not see itself as current" ); + + if( g_fiber_a_saw_root_enterable != 1 ) + return report( "fiber_roundtrip", false, "suspended root was not enterable" ); + + if( g_fiber_a_saw_self_enterable != 0 ) + return report( "fiber_roundtrip", false, "a RUNNING fiber claimed to be enterable" ); + + // Re-entry takes the loop again — the second swap resumes, not restarts. + if( !fiber_swap( g_fiber_root, g_fiber_a ) || g_fiber_a_runs != 2 ) + return report( "fiber_roundtrip", false, "second swap did not re-run the body" ); + + report( "fiber_roundtrip", true ); +} + +void Scenario_FiberReleaseSuspended() +{ + // Fiber A is Suspended after the roundtrip. Releasing it mid-suspend is + // LEGAL — libcontext refcounts drop never-finished coroutines — and the + // stale id must then refuse instead of resuming freed state. + if( !g_fiber_a || status_of( g_fiber_a ) != Status::Suspended ) + return report( "fiber_release_suspended", false, "precondition: fiber A suspended" ); + + if( !fiber_release( g_fiber_a ) ) + return report( "fiber_release_suspended", false, "release refused" ); + + if( fiber_enterable( g_fiber_a ) ) + return report( "fiber_release_suspended", false, "released fiber still enterable" ); + + if( fiber_swap( g_fiber_root, g_fiber_a ) ) + return report( "fiber_release_suspended", false, + "swap into a released fiber was allowed (use-after-free)" ); + + report( "fiber_release_suspended", true ); +} + +alignas( 16 ) char g_fiber_b_stack[64 * 1024]; +ContextId g_fiber_b = 0; +int g_fiber_b_runs = 0; +int g_coexist_stage = 0; +int g_coexist_result = 0; + +void fiber_b_entry( void* ) +{ + for( ;; ) + { + ++g_fiber_b_runs; + fiber_swap( g_fiber_b, g_fiber_root ); + } +} + +void body_coexist_park( void* ) +{ + g_coexist_stage = 1; + g_coexist_result = yield_park( "coexist-park" ); + g_coexist_stage = 2; +} + +void Scenario_FiberAndStarCoexist() +{ + // One registry, two lanes: a PARKED star context must sit undisturbed + // while symmetric fiber swaps happen, and resume cleanly afterwards — + // production Phase A runs tool fibers while D1's star layer idles. + g_coexist_stage = 0; + g_fiber_b_runs = 0; + + const ContextId star = create( body_coexist_park, nullptr, "coexist-star" ); + drain(); // parks + + if( g_coexist_stage != 1 || status_of( star ) != Status::Parked ) + return report( "fiber_and_star_coexist", false, "star context did not park" ); + + g_fiber_b = fiber_create( fiber_b_entry, nullptr, g_fiber_b_stack, + sizeof( g_fiber_b_stack ), 32 * 1024, "fiber-b" ); + + if( !g_fiber_b || !fiber_swap( g_fiber_root, g_fiber_b ) || g_fiber_b_runs != 1 ) + return report( "fiber_and_star_coexist", false, "fiber roundtrip failed" ); + + if( status_of( star ) != Status::Parked ) + return report( "fiber_and_star_coexist", false, + "fiber swaps disturbed the parked star context" ); + + if( !mark_ready( star, 7 ) ) + return report( "fiber_and_star_coexist", false, "mark_ready failed" ); + + drain(); + + if( g_coexist_stage != 2 || g_coexist_result != 7 ) + return report( "fiber_and_star_coexist", false, "star context did not resume" ); + + destroy( star ); + fiber_release( g_fiber_b ); + report( "fiber_and_star_coexist", true ); +} + // --------------------------------------------------------------------------- // scenario 10: a REAL async wake — JS timeout resolves the park // The park's wake crosses a JS turn, which is the shape every production @@ -609,6 +779,9 @@ int main() Scenario_DeepPark(); Scenario_FiberNestsInContext(); Scenario_ForeignStackRefused(); + Scenario_FiberRoundtrip(); + Scenario_FiberReleaseSuspended(); + Scenario_FiberAndStarCoexist(); // Async last: it emits STATS + SUMMARY when its wake lands. Scenario_AsyncWake(); diff --git a/tests/asyncify/sched-context.spec.ts b/tests/asyncify/sched-context.spec.ts index e26f365..2e93cb8 100644 --- a/tests/asyncify/sched-context.spec.ts +++ b/tests/asyncify/sched-context.spec.ts @@ -24,6 +24,9 @@ const SCENARIOS = [ 'deep_park_sizing', 'fiber_nests_in_context', 'foreign_stack_refused', + 'fiber_roundtrip', + 'fiber_release_suspended', + 'fiber_and_star_coexist', 'async_wake', ]; @@ -55,6 +58,21 @@ type Stats = { cStackBytes: number; asyncifyBytes: number; asyncifyHighWater: number; + // Fiber lane (doc 22 Phase A) — libcontext's clients, separate counters so + // this battery's star assertions keep meaning what they meant. + fiberLive: number; + fiberPeakLive: number; + fiberCreated: number; + fiberReleased: number; + fiberSwaps: number; + fiberRefusals: number; + fiberReleasedSuspended: number; + fiberReleasedRunning: number; + fiberNonEnterableSwaps: number; + fiberRunning: number; + fiberBytes: number; + fiberPeakBytes: number; + fiberAsyncifyHighWater: number; }; function findLine(logs: string[], marker: string): string | undefined { @@ -195,5 +213,28 @@ test.describe('Design B D1 — scheduler contexts', () => { // No buffer-pressure beacon should have fired (>75% use). const pressure = testLogger.consoleLogs.filter((l) => l.includes('BUFFER-PRESSURE')); expect(pressure, `buffer pressure: ${pressure.join(' || ')}`).toHaveLength(0); + + // --- fiber lane (doc 22 Phase A): libcontext semantics over the registry -- + // The adopted root is the only fiber that outlives the battery (libcontext's + // main context never dies); everything else was released. + expect(stats.fiberLive, 'only the adopted root fiber remains').toBe(1); + expect(stats.fiberRunning, 'the root is the fiber lane current').not.toBe(0); + expect(stats.fiberCreated, 'root + fiber A + fiber B').toBe(3); + expect(stats.fiberReleased, 'fibers A and B were released').toBe(2); + expect(stats.fiberSwaps, 'symmetric swaps happened').toBeGreaterThanOrEqual(6); + // Both releases happened mid-suspend — libcontext's refcount-drop shape. + expect(stats.fiberReleasedSuspended, 'suspended releases are legal and counted').toBe(2); + // One deliberate stale-id swap was refused (use-after-free made loud). + expect(stats.fiberRefusals, 'a stale fiber id was refused').toBeGreaterThanOrEqual(1); + // THE Phase A tripwires: no swap ever entered a non-enterable fiber, and + // no release ever hit a fiber the registry believed was running. + expect(stats.fiberNonEnterableSwaps, 'zero swaps into stale rewind state').toBe(0); + expect(stats.fiberReleasedRunning, 'zero releases of a running fiber').toBe(0); + // The sizing input Phase E reads: a suspended fiber's capture was measured + // (sampled before the resume consumes it, when the buffer is non-empty). + expect( + stats.fiberAsyncifyHighWater, + 'fiber-lane asyncify use was measured while suspended', + ).toBeGreaterThan(0); }); }); diff --git a/wxwidgets b/wxwidgets index 3d37db3..719fd98 160000 --- a/wxwidgets +++ b/wxwidgets @@ -1 +1 @@ -Subproject commit 3d37db3bf1cdffaaefe657047d7f6129f81fdb01 +Subproject commit 719fd98798a79301d7c0f89e085bf0a1a289d72d