tooling(sourcetrail): code-graph setup + asyncify allocator-suspend plan
sourcetrail/: Sourcetrail 2021.4.19 indexing pipeline for the wasm port — compile-db transform (rsp expansion, PCH strip, libc++-11 pinning, path rewrites), removelist-candidate analysis over the indexed call graph, README with regen steps and the hard-won tricks (relative --project-file hang, SDK-header poisoning). Heavy artifacts (.srctrldb, compile dbs, libcxx headers) stay untracked via the folder's .gitignore. docs/asyncify-allocator-suspend/plan.md: verified plan for the nanosleep-shim zero-duration guard (mimalloc mi_atomic_yield=sleep(0) can suspend malloc on the main thread), the red/green contention test, and the adversarially verified removelist additions with measured payoff (80.9k -> 60.0k instrumented functions). Planned, not yet executed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P2NHhbzuEHqP2JmcrSD96D
This commit is contained in:
Viktor Vaczi
parent
80199f42f4
commit
33e23e0a65
7 changed files with 571 additions and 0 deletions
131
docs/asyncify-allocator-suspend/plan.md
Normal file
131
docs/asyncify-allocator-suspend/plan.md
Normal file
|
|
@ -0,0 +1,131 @@
|
||||||
|
<!-- STATUS: PLANNED, NOT EXECUTED (saved 2026-08-10). Verification findings herein are real
|
||||||
|
(measured against the Aug 6 kicad_editor build, emsdk 4.0.2); the shim guard, the test,
|
||||||
|
and the removelist additions have NOT been applied yet. -->
|
||||||
|
|
||||||
|
# Nanosleep shim zero-duration guard + mimalloc contention red/green test
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
The asyncify investigation established that **memory allocation can suspend on the main thread**: emscripten's vendored mimalloc uses `sleep(0)` as its spin-wait yield (`mi_atomic_yield` fallback — wasm32-emscripten matches no arch case in `include/mimalloc/atomic.h`), and `wasm/shims/nanosleep_yield.c` unconditionally converts main-thread `nanosleep` into an Asyncify event-loop yield. Under cross-thread delayed-free contention, malloc's slow path can unwind mid-allocation — a latent reentrancy hazard that also blocks the planned allocator removelist entries (~14K-function instrumentation win).
|
||||||
|
|
||||||
|
Adversarial verification (real kicad_editor binary + emsdk 4.0.2 sources) proved: the `sleep(0)` chain is the allocator's **only** suspend path; all four mimalloc spin sites sleep constant 0; mimalloc is the module's **only zero-duration sleeper** (all other nanosleep callers are ≥1 ms constants); `std::this_thread::yield`/`sched_yield` don't route through nanosleep. Hence an `ms == 0` early return in the shim severs the path completely and affects nothing else.
|
||||||
|
|
||||||
|
**This task:** (1) the zero-duration guard in the shim; (2) a standalone red/green C++ wasm test that exercises the mimalloc contention path and detects main-thread event-loop turns — RED before the fix, GREEN after; (3) add the verified-safe memory-touching entry families to `scripts/common/asyncify-removelist.txt` (user-proposed list, adversarially verified against the real module first).
|
||||||
|
|
||||||
|
Out of scope (explicit follow-ups): full kicad_editor rebuild + 3D e2e suite run with the new removelist (the new entries only take effect at the next app build's post-process anyway).
|
||||||
|
|
||||||
|
## Part 1 — shim fix
|
||||||
|
|
||||||
|
**File:** `pcbjam/wasm/shims/nanosleep_yield.c` (52 lines).
|
||||||
|
|
||||||
|
In `nanosleep()`: after computing `ms`, wrap the yield/sleep branch in `if (ms > 0.0)` — zero-duration requests return 0 immediately on both main thread and workers (a 0 ms blocking sleep is a no-op anyway; `sleep(0)` arrives as exactly `{0,0}` per musl, so the guard fires deterministically). Extend the header comment: mimalloc's `mi_atomic_yield` is `sleep(0)` (spin-politeness hint, must NOT become an event-loop yield mid-malloc); a zero-duration sleep never promised an event-loop turn; verified 2026-08-10 that mimalloc is the only zero-duration caller in the module.
|
||||||
|
|
||||||
|
Rebuild consequences: only binaries that link the shim — future KiCad app builds, `pthread-ondemand`, and the new test. Already-built `output/*.wasm` unchanged. (CI: touching `wasm/**` busts the testapps cache — expected.)
|
||||||
|
|
||||||
|
## Part 2 — the test
|
||||||
|
|
||||||
|
### App: `tests/apps/standalone/mimalloc-storm/mimalloc_storm_test.cpp`
|
||||||
|
|
||||||
|
Plain `int main()` app (non-wx — template: `standalone/coroutine-pthread/main_repro.cpp`), pthreads + asyncify + **`-sMALLOC=mimalloc`** (used nowhere in the test tree today) + the nanosleep shim.
|
||||||
|
|
||||||
|
**Storm** (turns the nanosecond `MI_DELAYED_FREEING` windows into a hit-rate game we control):
|
||||||
|
- Main thread allocates batches of small same-size blocks (64 B → pages fill fast → full pages enter delayed-free mode; ~200k blocks ≈ ~200 full pages/round).
|
||||||
|
- W worker `std::thread`s free the previous round's blocks in tight loops (every cross-thread free of a full page runs the two-CAS `DELAYED_FREEING` bracket).
|
||||||
|
- Main concurrently churns alloc/free in the same size class and calls `mi_collect(true)` each round (`extern "C" void mi_collect(bool)` — public API, headers not on include path; reaches `_mi_heap_delayed_free_all`, the spin site).
|
||||||
|
- Bounded by ITERATION counts (no wall-clock). Early-exit once ≥50 turns observed (keeps pre-fix RED runs fast); parameters tuned empirically at the red-validation step.
|
||||||
|
|
||||||
|
**Detector** (did main return to the event loop mid-storm?):
|
||||||
|
- `EM_ASM` arms a self-re-arming `setTimeout(0)` that increments `globalThis.__stormTurns`; a synchronous C++ storm can only let it run if an Asyncify unwind happened inside. Main checks the counter via `EM_ASM_INT` every k iterations.
|
||||||
|
- Phase 2 assertion (shim's load-bearing behavior unchanged): re-arm marker, `nanosleep(5 ms)` on main → the marker MUST have run (nonzero sleeps still yield).
|
||||||
|
- Console contract (repo idiom — `EM_ASM` console.log markers, self-terminating, documented in the file header):
|
||||||
|
- `[MIMALLOC_STORM] START threads=W blocks=B rounds=R`
|
||||||
|
- `[MIMALLOC_STORM] SUMMARY stormTurns=N sleepTurned=0|1 completed=1`
|
||||||
|
|
||||||
|
### Build rules: `tests/apps/Makefile.wasm`
|
||||||
|
|
||||||
|
- `LDFLAGS_MIMALLOC_STORM` modeled on `LDFLAGS_COROUTINE_PTHREAD_NOWX` (line ~875: EH_FLAGS, ALLOW_MEMORY_GROWTH, ASYNCIFY=1 + stack size, DYNCALLS, -pthread, pool=`navigator.hardwareConcurrency`, STRICT=0, no wx) **plus `-sMALLOC=mimalloc`**.
|
||||||
|
- `.o` rule for the app; `nanosleep_yield.o` reuse per the `pthread-ondemand` pattern (`Makefile.wasm:681`, compiled `-c -pthread`); `.html` rule; `.PHONY: mimalloc-storm`; `all:` accumulation line.
|
||||||
|
- No `mallinfo_stub.c` (no OCC). No new suspending import (`env.__asyncjs__*` already in `scripts/common/asyncify-imports.txt`). Post-link asyncify (`apply-asyncify.sh --no-removelist`) + dyncall injection happen automatically via `build-wasm-test.sh`'s `find -newer` fan-out — no driver-script changes.
|
||||||
|
|
||||||
|
### Spec: `tests/e2e/mimalloc-storm.spec.ts`
|
||||||
|
|
||||||
|
- Placement in `tests/e2e/` → runs under the existing `wx-chromium` project; **zero playwright-config/CI edits** and passes `lint:ci-coverage`.
|
||||||
|
- Pattern copied from `tests/e2e/coroutine-pthread.spec.ts` (the non-wx precedent): `testLogger` fixture, best-effort `tryLoadApp(...).catch(() => {})` with the documented marker comment, then `expect.poll` for the `SUMMARY` line; parse it; assert `stormTurns === 0`, `sleepTurned === 1`, `completed === 1`, and no page errors (favicon-filtered).
|
||||||
|
- Determinism compliance: no `waitForTimeout`, no inline retries, bounded C++ iterations; the app always emits a terminal SUMMARY (self-capping), so the poll is bounded.
|
||||||
|
|
||||||
|
## Part 3 — asyncify-removelist additions (VERIFIED)
|
||||||
|
|
||||||
|
Adversarial verification against the real kicad_editor module is complete: **all proposed families SAFE**, conditional on the Part 1 shim guard shipping first. Add to `scripts/common/asyncify-removelist.txt` (exact patterns — see traps below):
|
||||||
|
|
||||||
|
```
|
||||||
|
# --- Memory-touching families (safe ONLY with the nanosleep zero-duration guard:
|
||||||
|
# --- mimalloc's mi_atomic_yield is sleep(0); with the guard, no allocation path suspends.
|
||||||
|
# --- Also conditional: no one registers mi_register_deferred_free/output/error hooks or a
|
||||||
|
# --- suspending std::new_handler (all unregistered as of 2026-08-10 audit).
|
||||||
|
std::__2::basic_string<*>*
|
||||||
|
std::__2::char_traits<*>*
|
||||||
|
std::__2::vector<*>*
|
||||||
|
std::__2::__tree*
|
||||||
|
std::__2::to_string*
|
||||||
|
std::__2::__itoa*
|
||||||
|
std::__2::__split_buffer<*>*
|
||||||
|
std::__2::__shared_ptr_emplace<*>*
|
||||||
|
std::__2::__shared_ptr_pointer<*>*
|
||||||
|
std::__2::__shared_count*
|
||||||
|
std::__2::__shared_weak_count*
|
||||||
|
std::__2::deque<*>*
|
||||||
|
std::__2::__hash_table<*>*
|
||||||
|
std::__2::map<*>*
|
||||||
|
std::__2::__list_imp<*>*
|
||||||
|
std::__2::allocator*
|
||||||
|
boost::uuids::*
|
||||||
|
operator new*
|
||||||
|
operator delete*
|
||||||
|
aligned_alloc
|
||||||
|
mi_*
|
||||||
|
_mi_*
|
||||||
|
sbrk
|
||||||
|
# Deliberately NOT std::__2::__function* — type-erased std::function invocation (operator())
|
||||||
|
# is how tool/dialog callbacks run; 111 __func::operator() bodies reach startModal/wxMilliSleep
|
||||||
|
# directly (e.g. ShowPreferences, file dialogs), so __function frames must stay instrumented.
|
||||||
|
# Container entries above are still safe: containers only touch callables via their
|
||||||
|
# clone/destroy lifecycle ops, none of which reach a suspend (verified 2026-08-10).
|
||||||
|
```
|
||||||
|
|
||||||
|
**Authoring traps (verified the hard way):**
|
||||||
|
- **Bracket balance:** Binaryen's list parser tracks `<>()[]{}` nesting — an unbalanced pattern like `std::__2::basic_string<*` **aborts the whole asyncify pass** (`Fatal: failed to parse lists`). Template patterns must close the bracket: `basic_string<*>*`.
|
||||||
|
- **Breadth anchoring:** bare `basic_string*` sweeps in `basic_string_view`/`basic_stringbuf`/`basic_stringstream` (31 extra names); the `<`-anchored form excludes them. `__tree*` deliberately includes `__tree_node_base`/iterators (verified). Bare `vector*`/`char_traits*` (no namespace) are wrong forms.
|
||||||
|
|
||||||
|
**Key evidence:** zero direct suspend paths post-guard in every family; `__tree` comparators are monomorphized (no `call_indirect` in e.g. `__find_equal` with `std::less`); container-of-`std::function` ops (e.g. TOOL_MANAGER's `deque<function<void()>>::push_back`) only touch the functor's clone/move/destroy lifecycle slot — of 12,880 `__function` lifecycle ops in the module, zero reach a suspend; all 111 suspend-reaching `__function` members are `operator()` invocation bodies (the excluded family). `boost::uuids` (KIID) seeds `mt19937` once via synchronous `getentropy` (wasi import, not asyncify-relevant).
|
||||||
|
|
||||||
|
**Measured payoff** (single-threaded verbose runs, real hoisted module):
|
||||||
|
|
||||||
|
| Run | Removelist | Instrumented | Module size |
|
||||||
|
|---|---|---|---|
|
||||||
|
| A | current file | 80,871 (74%) | 253 MB |
|
||||||
|
| B | + allocator family | 66,499 (60%) | 218 MB |
|
||||||
|
| **C** | **+ all families above** | **60,047 (55%)** | **204 MB** |
|
||||||
|
|
||||||
|
Run C is a strict subset of B (sanity: zero additions); only non-matching warnings are the 5 pre-existing OCC entries (expected — OCC absent from this app). Audit validity: this binary / emsdk 4.0.2; re-run on emsdk bump or libc++ ABI-namespace change (artifacts in `$W`: `analysis.pkl`, `logC.txt`, `runC.wasm`).
|
||||||
|
|
||||||
|
## Part 4 — execution sequence
|
||||||
|
|
||||||
|
1. **Build the test against the UNPATCHED shim**: `./scripts/build-wasm-test.sh mimalloc-storm` (targeted — the default `all` target is currently broken by pre-existing working-tree deletions, see caveat). Run the spec: `cd tests && npx playwright test --project=wx-chromium e2e/mimalloc-storm.spec.ts`. Expect the spec to FAIL with `stormTurns > 0` — that failure IS the red validation (proves the storm reaches the mimalloc yield). Record the observed turn count; tune B/R/W if turns are marginal (<~10).
|
||||||
|
2. **Apply the shim fix** (Part 1), rebuild the same target (make tracks the shim as an explicit prerequisite), re-run the spec → GREEN: `stormTurns === 0`, storm completes (no-deadlock proof), `sleepTurned === 1`.
|
||||||
|
3. `cd tests && npm run lint:determinism` (spec must pass the linter).
|
||||||
|
4. **Removelist update:** add the verified entries to `scripts/common/asyncify-removelist.txt` with a documented block: the guard dependency (entries safe ONLY with the zero-duration nanosleep guard), the dormant-slot condition (deferred-free/new_handler/output hooks unregistered), and the `std::__2::__function*` exclusion rationale. Sanity-check the new list against the scratch module: one `wasm-opt --asyncify` run on `$W/hoisted.wasm` with the updated file confirms the expected instrumented-function count and that no entry matches zero functions unexpectedly.
|
||||||
|
5. Stop before committing — present the diff; commit on main via the user's `/git-feature-commit` flow on request (trunk-based repo).
|
||||||
|
|
||||||
|
## Caveats found during exploration
|
||||||
|
|
||||||
|
- **Pre-existing working-tree deletions** (not ours; do not touch): `tests/apps/standalone/asyncify-races/races_test.cpp`, `tests/asyncify/asyncify-races.spec.ts`, `tests/web/eeschema-fp-selector.spec.ts` are tracked but deleted, which breaks `Makefile.wasm`'s `all` target and full `npm run test:e2e`. We build only our target and run only our spec; flag the state to the user at the end.
|
||||||
|
- `build-wasm-test.sh` requires `build-wasm/wxwidgets/wx-config` (wx libs built) and stubs the emsdk in-link asyncify — the emsdk lives at `tools/emsdk/` (installed via `scripts/setup-emsdk.sh` if absent).
|
||||||
|
- Reference precedents: `races_test.cpp` (via `git show HEAD:...`) for Asyncify-state probing from C++ if the setTimeout detector needs corroboration; `pthread_ondemand_test.cpp` for the watchdog/marker idiom.
|
||||||
|
|
||||||
|
## Verification summary
|
||||||
|
|
||||||
|
- RED observed pre-fix (spec fails on `stormTurns > 0`), GREEN post-fix on identical parameters — the core deliverable.
|
||||||
|
- GREEN also proves: spin completes without the yield (no deadlock), 5 ms sleep still turns the event loop (worker-boot behavior preserved).
|
||||||
|
- `lint:determinism` clean.
|
||||||
|
- Removelist sanity run on `$W/hoisted.wasm` reproduces run C's numbers (60,047 instrumented / 204 MB pre-O1) with no unexpected non-matching-pattern warnings.
|
||||||
|
- Follow-ups NOT here: full editor rebuild + 3D e2e suite with the new removelist (entries take effect at the next app build's host post-process).
|
||||||
9
sourcetrail/.gitignore
vendored
Normal file
9
sourcetrail/.gitignore
vendored
Normal file
|
|
@ -0,0 +1,9 @@
|
||||||
|
# Generated/machine-local artifacts — only the README, scripts, and project
|
||||||
|
# files are tracked. Everything below is rebuilt by the steps in README.md.
|
||||||
|
*.srctrldb
|
||||||
|
*.srctrldb_tmp
|
||||||
|
*.srctrlbm
|
||||||
|
compile_commands.json
|
||||||
|
sample_compile_commands.json
|
||||||
|
libcxx-11/
|
||||||
|
libcxx-11.src.tar.xz
|
||||||
113
sourcetrail/README.md
Normal file
113
sourcetrail/README.md
Normal file
|
|
@ -0,0 +1,113 @@
|
||||||
|
# Sourcetrail code graph for the KiCad WASM port
|
||||||
|
|
||||||
|
Interactive symbol-level code graph (classes, calls, includes, inheritance) of the
|
||||||
|
merged `kicad_editor` build — all 2,176 TUs: pcbnew, eeschema, common, 3D viewer,
|
||||||
|
and the `wasm/` port layer — indexed by [Sourcetrail](https://github.com/CoatiSoftware/Sourcetrail).
|
||||||
|
Lives in `pcbjam/sourcetrail/`: README, scripts and project files are tracked; the heavy
|
||||||
|
artifacts (index db, compile db, libc++ headers) are ignored via the local `.gitignore`
|
||||||
|
and rebuilt with the steps below.
|
||||||
|
|
||||||
|
## What's in this folder
|
||||||
|
|
||||||
|
| File | Purpose |
|
||||||
|
|---|---|
|
||||||
|
| `kicad-wasm.srctrlprj` / `.srctrldb` | Sourcetrail project + indexed database (~440 MB) |
|
||||||
|
| `compile_commands.json` | Transformed compile db the indexer consumes |
|
||||||
|
| `transform_compile_db.py` | Turns the container's emscripten compile db into the above |
|
||||||
|
| `libcxx-11/` | Pinned libc++ 11.1.0 headers (see Tricks) |
|
||||||
|
| `asyncify_candidates.py` | Ranks subsystems safe for the asyncify removelist |
|
||||||
|
| `sample.srctrlprj` + `sample_compile_commands.json` | 6-file smoke test for the pipeline |
|
||||||
|
|
||||||
|
## Install
|
||||||
|
|
||||||
|
- **Sourcetrail 2021.4.19** — the last free release (the maintained fork paywalls binaries):
|
||||||
|
<https://github.com/CoatiSoftware/Sourcetrail/releases/tag/2021.4.19>, macOS dmg → `/Applications`,
|
||||||
|
then `xattr -dr com.apple.quarantine /Applications/Sourcetrail.app`. It's x86_64 — needs Rosetta 2
|
||||||
|
on Apple Silicon (works fine).
|
||||||
|
- Nothing else: python3 stdlib only; header mirrors are exported from the Docker build.
|
||||||
|
|
||||||
|
## Run
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# browse (GUI)
|
||||||
|
open -a Sourcetrail /Users/V/IdeaProjects/pcbjam-private/pcbjam/sourcetrail/kicad-wasm.srctrlprj
|
||||||
|
|
||||||
|
# (re)index from the terminal — ABSOLUTE project path, see Tricks
|
||||||
|
/Applications/Sourcetrail.app/Contents/MacOS/Sourcetrail index \
|
||||||
|
--project-file /Users/V/IdeaProjects/pcbjam-private/pcbjam/sourcetrail/kicad-wasm.srctrlprj
|
||||||
|
```
|
||||||
|
|
||||||
|
Full index ≈ 12 min all-cores. In the GUI there is no single whole-project graph: search a
|
||||||
|
symbol (Cmd+F — e.g. `BOARD`, `KIPLATFORM`) and click it; the graph pane centers on it and
|
||||||
|
expands as you click nodes/edges.
|
||||||
|
|
||||||
|
### Refreshing after a KiCad rebuild
|
||||||
|
|
||||||
|
The compile db comes from the `main`-branch Docker build cache (KiCad's CMake exports it
|
||||||
|
by default — no reconfigure needed). From `pcbjam/`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
source scripts/common/versions.sh
|
||||||
|
COMPOSE_PROJECT_NAME=kicad-wasm-main docker compose -f docker/docker-compose.yml up -d --build
|
||||||
|
|
||||||
|
# compile db + generated sources/headers + .rsp files + deps/wx includes
|
||||||
|
docker compose -p kicad-wasm-main -f docker/docker-compose.yml exec -T kicad-wasm-builder bash -c \
|
||||||
|
"cd /workspace && { find build-wasm/kicad-kicad_editor \( -name '*.h' -o -name '*.hpp' -o -name '*.hxx' \
|
||||||
|
-o -name '*.hh' -o -name '*.inc' -o -name '*.rsp' -o -name '*.cc' -o -name '*.cpp' -o -name '*.cxx' \
|
||||||
|
-o -name 'compile_commands.json' \) -type f; echo build-wasm/sysroot/include; \
|
||||||
|
echo build-wasm/wxwidgets/lib/wx/include; } | tar -cf - -T -" | tar -xf - -C .
|
||||||
|
|
||||||
|
# emscripten sysroot headers -> tools/emsdk mirror (only after an emsdk bump)
|
||||||
|
docker compose -p kicad-wasm-main -f docker/docker-compose.yml exec -T kicad-wasm-builder \
|
||||||
|
tar -cf - -C / emsdk/upstream/emscripten/cache/sysroot/include | tar -xf - -C tools/
|
||||||
|
|
||||||
|
cd sourcetrail
|
||||||
|
python3 transform_compile_db.py ../kicad-kicad_editor/compile_commands.json compile_commands.json
|
||||||
|
# then the `Sourcetrail index` command above
|
||||||
|
```
|
||||||
|
|
||||||
|
## Tricks (why this isn't just "point Sourcetrail at the cdb")
|
||||||
|
|
||||||
|
- **The bundled clang is ~LLVM 11.** Modern libc++ (emsdk 4.x's or the macOS SDK's) does not
|
||||||
|
parse under it. The transform pins `libcxx-11/` via `-nostdinc++` and takes C headers from the
|
||||||
|
emsdk musl sysroot mirror (`tools/emsdk/`). Expected residue: ~14 errors, all in the
|
||||||
|
libc++11/musl locale seam (`_CTYPE_*`, `strtoull_l`, one fatal `xlocale.h`) — harmless to the graph.
|
||||||
|
- **Transform surgery:** `@CMakeFiles/*.rsp` response files are expanded inline (old clang's cdb
|
||||||
|
loader can't); PCH is stripped (`-Xclang -include-pch` of clang-20 `.pch` binaries) and replaced
|
||||||
|
with `-include cmake_pch.hxx`; emscripten-only flags (`-sFOO`, `-fwasm-exceptions`) dropped,
|
||||||
|
`--target=wasm32-unknown-emscripten` + `-fexceptions` added; paths rewritten
|
||||||
|
`/workspace` → `pcbjam/`, `/emsdk` → `pcbjam/tools/emsdk/`.
|
||||||
|
- **CLI hangs on relative `--project-file` paths.** Silently — idle event loop, log stops after
|
||||||
|
"Maven executable path detection". Always pass absolute paths.
|
||||||
|
- **Global header paths are deliberately empty** in
|
||||||
|
`~/Library/Application Support/Sourcetrail/ApplicationSettings.xml`. First launch auto-filled
|
||||||
|
macOS-26-SDK paths, which poison every parse (see clang-11 point). Don't re-run header path
|
||||||
|
detection from Preferences; `has_prefilled_header_search_paths=1` keeps it from coming back.
|
||||||
|
- **"N files (126 complete)" undersells the index.** A file counts as complete only if *every* TU
|
||||||
|
touching it had zero errors; the 14 std-header errors are included nearly everywhere, so the
|
||||||
|
flag cascades. The symbols/references themselves are all recorded.
|
||||||
|
|
||||||
|
## Asyncify removelist candidates
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 asyncify_candidates.py
|
||||||
|
```
|
||||||
|
|
||||||
|
Computes, over the indexed call graph (187K call edges + override pseudo-edges for virtual
|
||||||
|
dispatch), which functions can NEVER reach a suspend point, aggregated per module. Seeds =
|
||||||
|
every `ShowModal`/`Yield`/`Sleep`/progress-dialog/`COROUTINE` function (**strict**), plus
|
||||||
|
`ProcessEvent`-style synchronous dispatch (**lenient** — a dispatched handler may suspend and
|
||||||
|
unwind through the dispatcher). Functions clean under *lenient* are candidates for
|
||||||
|
`scripts/common/asyncify-removelist.txt` (matching rules are documented in that file:
|
||||||
|
one prefix wildcard per symbol, e.g. `SHAPE_POLY_SET::*`).
|
||||||
|
|
||||||
|
Headline results from the 2026-08-07 index: `kiapi` generated protobuf (15.2K funcs),
|
||||||
|
`libs/kimath` (7.3K), `clipper2` (6.3K), `nlohmann_json`/`fmt`/`pegtl`/`zint` are 100% clean;
|
||||||
|
`pcbnew/router` is 1,820/1,827 clean (the 7 are the `Wait()` tool-integration layer — the PNS
|
||||||
|
shove/optimizer core never suspends).
|
||||||
|
|
||||||
|
**Caveats:** the C++ graph can't see calls through `std::function`, event tables, or raw function
|
||||||
|
pointers, and asyncify operates on the post-inlining *wasm* call graph, not C++ symbols. Before
|
||||||
|
shipping an entry: ground-truth with Binaryen's asyncify verbose/advise output in
|
||||||
|
`apply-asyncify.sh`, and rely on e2e — a wrong removal traps loudly (`unreachable`) at the first
|
||||||
|
unwind through it. Tweak seeds / path depths at the top of the script.
|
||||||
125
sourcetrail/asyncify_candidates.py
Normal file
125
sourcetrail/asyncify_candidates.py
Normal file
|
|
@ -0,0 +1,125 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Rank subsystems by asyncify-removelist safety using the Sourcetrail db.
|
||||||
|
|
||||||
|
Taint = "a suspend point is reachable from this function" (reverse BFS over
|
||||||
|
call edges from suspend primitives, with base->override pseudo-edges for
|
||||||
|
virtual dispatch). Two levels:
|
||||||
|
strict — direct suspend primitives only (Yield/Sleep/ShowModal/progress/
|
||||||
|
coroutine/fiber)
|
||||||
|
lenient — strict + synchronous event dispatch (ProcessEvent & co), since a
|
||||||
|
dispatched handler may suspend and unwind through the dispatcher
|
||||||
|
|
||||||
|
Functions never tainted even in lenient mode are removelist candidates.
|
||||||
|
Blind spots (validate with ASYNCIFY_ADVISE before shipping): std::function /
|
||||||
|
event-table indirection, function pointers, wx/libc internals outside the cdb.
|
||||||
|
"""
|
||||||
|
import re
|
||||||
|
import sqlite3
|
||||||
|
from collections import defaultdict, deque
|
||||||
|
|
||||||
|
DB = "kicad-wasm.srctrldb"
|
||||||
|
PCBJAM = "/Users/V/IdeaProjects/pcbjam-private/pcbjam/"
|
||||||
|
FUNC_TYPES = (4096, 8192)
|
||||||
|
|
||||||
|
STRICT_PATTERNS = [
|
||||||
|
r"\tnShowModal", # every ShowModal incl. DIALOG_SHIM/KIDIALOG wrappers
|
||||||
|
r"\tnShowWindowModal",
|
||||||
|
r"\tnShowQuasiModal",
|
||||||
|
r"mwxYield", r"mwxSafeYield", r"mwxYieldIfNeeded",
|
||||||
|
r"\tnYield", r"\tnYieldFor", r"\tnDoYieldFor", r"\tnSafeYield",
|
||||||
|
r"\tnSleep", r"mwxSleep", r"mwxMilliSleep", r"mwxMicroSleep",
|
||||||
|
r"sleep_for", r"sleep_until", r"nanosleep", r"emscripten_sleep",
|
||||||
|
r"emscripten_fiber", r"__asyncjs__",
|
||||||
|
r"mwxGenericProgressDialog", r"mwxProgressDialog",
|
||||||
|
r"mwxMessageBox", r"mwxExecute",
|
||||||
|
r"mCOROUTINE<",
|
||||||
|
]
|
||||||
|
LENIENT_EXTRA = [
|
||||||
|
r"\tnProcessEvent\t", r"\tnSafelyProcessEvent", r"\tnProcessPendingEvents",
|
||||||
|
r"\tnHandleEvent\t", r"\tnProcessEventLocally",
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
con = sqlite3.connect(DB)
|
||||||
|
|
||||||
|
funcs = {} # id -> serialized_name
|
||||||
|
for nid, name in con.execute(
|
||||||
|
f"SELECT id, serialized_name FROM node WHERE type IN {FUNC_TYPES}"):
|
||||||
|
funcs[nid] = name
|
||||||
|
|
||||||
|
# callee -> callers (reverse call graph)
|
||||||
|
rev = defaultdict(list)
|
||||||
|
for s, t in con.execute("SELECT source_node_id, target_node_id FROM edge WHERE type=8"):
|
||||||
|
rev[t].append(s)
|
||||||
|
# virtual dispatch: caller of Base::f may land in Derived::f, so taint of
|
||||||
|
# Derived::f must flow to callers of Base::f -> pseudo-edge callee=Derived,
|
||||||
|
# caller-side=Base is wrong; we need: if Derived tainted then Base tainted
|
||||||
|
# is NOT true. Correct: call to Base::f can dispatch to Derived::f, so if
|
||||||
|
# Derived::f suspends, callers of Base::f suspend. Model: rev[Derived] gets
|
||||||
|
# nothing; instead treat Base::f as a caller of every override Derived::f.
|
||||||
|
for s, t in con.execute("SELECT source_node_id, target_node_id FROM edge WHERE type=32"):
|
||||||
|
# override edge: Derived::f (source) -> Base::f (target)
|
||||||
|
rev[s].append(t) # taint flows Derived -> Base -> Base's callers
|
||||||
|
|
||||||
|
def seeds_for(patterns):
|
||||||
|
pats = [re.compile(p) for p in patterns]
|
||||||
|
return {nid for nid, name in funcs.items() if any(p.search(name) for p in pats)}
|
||||||
|
|
||||||
|
def closure(seed_ids):
|
||||||
|
seen = set(seed_ids)
|
||||||
|
q = deque(seed_ids)
|
||||||
|
while q:
|
||||||
|
n = q.popleft()
|
||||||
|
for caller in rev.get(n, ()):
|
||||||
|
if caller not in seen:
|
||||||
|
seen.add(caller)
|
||||||
|
q.append(caller)
|
||||||
|
return seen
|
||||||
|
|
||||||
|
strict_seeds = seeds_for(STRICT_PATTERNS)
|
||||||
|
lenient_seeds = strict_seeds | seeds_for(STRICT_PATTERNS + LENIENT_EXTRA)
|
||||||
|
print(f"functions: {len(funcs)} strict seeds: {len(strict_seeds)} "
|
||||||
|
f"lenient seeds: {len(lenient_seeds)}")
|
||||||
|
|
||||||
|
tainted_strict = closure(strict_seeds)
|
||||||
|
tainted_lenient = closure(lenient_seeds)
|
||||||
|
|
||||||
|
# function -> file (prefer definition scope locations, type=1)
|
||||||
|
loc = {}
|
||||||
|
for typ in (1, 0):
|
||||||
|
for eid, path in con.execute(
|
||||||
|
"SELECT o.element_id, f.path FROM occurrence o "
|
||||||
|
"JOIN source_location sl ON sl.id=o.source_location_id "
|
||||||
|
"JOIN file f ON f.id=sl.file_node_id WHERE sl.type=?", (typ,)):
|
||||||
|
if eid in funcs and eid not in loc:
|
||||||
|
loc[eid] = path
|
||||||
|
|
||||||
|
def module(path):
|
||||||
|
if not path.startswith(PCBJAM):
|
||||||
|
return None # std/emsdk/deps headers — not our code
|
||||||
|
rel = path[len(PCBJAM):]
|
||||||
|
parts = rel.split("/")
|
||||||
|
depth = 3 if parts[0] in ("kicad", "build-wasm") else 2
|
||||||
|
return "/".join(parts[:depth]) if len(parts) > depth else "/".join(parts[:-1])
|
||||||
|
|
||||||
|
stats = defaultdict(lambda: [0, 0, 0]) # module -> [total, strict, lenient]
|
||||||
|
for nid in funcs:
|
||||||
|
m = module(loc.get(nid, ""))
|
||||||
|
if m is None:
|
||||||
|
continue
|
||||||
|
stats[m][0] += 1
|
||||||
|
if nid in tainted_strict:
|
||||||
|
stats[m][1] += 1
|
||||||
|
if nid in tainted_lenient:
|
||||||
|
stats[m][2] += 1
|
||||||
|
|
||||||
|
rows = [(m, t, s, l, t - l) for m, (t, s, l) in stats.items() if t >= 20]
|
||||||
|
rows.sort(key=lambda r: -r[4])
|
||||||
|
print(f"\n{'module':<44}{'funcs':>7}{'strict✗':>9}{'lenient✗':>9}{'clean':>7}{'clean%':>8}")
|
||||||
|
for m, t, s, l, clean in rows[:45]:
|
||||||
|
print(f"{m:<44}{t:>7}{s:>9}{l:>9}{clean:>7}{100*clean//t:>7}%")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
21
sourcetrail/kicad-wasm.srctrlprj
Normal file
21
sourcetrail/kicad-wasm.srctrlprj
Normal file
|
|
@ -0,0 +1,21 @@
|
||||||
|
<?xml version="1.0" encoding="utf-8" ?>
|
||||||
|
<config>
|
||||||
|
<source_groups>
|
||||||
|
<source_group_9a1b2c3d-0000-4000-8000-kicadwasm0001>
|
||||||
|
<build_file_path>
|
||||||
|
<compilation_db_path>./compile_commands.json</compilation_db_path>
|
||||||
|
</build_file_path>
|
||||||
|
<indexed_header_paths>
|
||||||
|
<indexed_header_path>/Users/V/IdeaProjects/pcbjam-private/pcbjam/kicad</indexed_header_path>
|
||||||
|
<indexed_header_path>/Users/V/IdeaProjects/pcbjam-private/pcbjam/wasm</indexed_header_path>
|
||||||
|
<indexed_header_path>/Users/V/IdeaProjects/pcbjam-private/pcbjam/wxwidgets/include</indexed_header_path>
|
||||||
|
<indexed_header_path>/Users/V/IdeaProjects/pcbjam-private/pcbjam/build-wasm/kicad-kicad_editor</indexed_header_path>
|
||||||
|
<indexed_header_path>/Users/V/IdeaProjects/pcbjam-private/pcbjam/build-wasm/wxwidgets/lib/wx/include</indexed_header_path>
|
||||||
|
</indexed_header_paths>
|
||||||
|
<name>KiCad WASM</name>
|
||||||
|
<status>enabled</status>
|
||||||
|
<type>C/C++ from Compilation Database</type>
|
||||||
|
</source_group_9a1b2c3d-0000-4000-8000-kicadwasm0001>
|
||||||
|
</source_groups>
|
||||||
|
<version>8</version>
|
||||||
|
</config>
|
||||||
21
sourcetrail/sample.srctrlprj
Normal file
21
sourcetrail/sample.srctrlprj
Normal file
|
|
@ -0,0 +1,21 @@
|
||||||
|
<?xml version="1.0" encoding="utf-8" ?>
|
||||||
|
<config>
|
||||||
|
<source_groups>
|
||||||
|
<source_group_9a1b2c3d-0000-4000-8000-kicadwasm0001>
|
||||||
|
<build_file_path>
|
||||||
|
<compilation_db_path>./sample_compile_commands.json</compilation_db_path>
|
||||||
|
</build_file_path>
|
||||||
|
<indexed_header_paths>
|
||||||
|
<indexed_header_path>/Users/V/IdeaProjects/pcbjam-private/pcbjam/kicad</indexed_header_path>
|
||||||
|
<indexed_header_path>/Users/V/IdeaProjects/pcbjam-private/pcbjam/wasm</indexed_header_path>
|
||||||
|
<indexed_header_path>/Users/V/IdeaProjects/pcbjam-private/pcbjam/wxwidgets/include</indexed_header_path>
|
||||||
|
<indexed_header_path>/Users/V/IdeaProjects/pcbjam-private/pcbjam/build-wasm/kicad-kicad_editor</indexed_header_path>
|
||||||
|
<indexed_header_path>/Users/V/IdeaProjects/pcbjam-private/pcbjam/build-wasm/wxwidgets/lib/wx/include</indexed_header_path>
|
||||||
|
</indexed_header_paths>
|
||||||
|
<name>KiCad WASM sample</name>
|
||||||
|
<status>enabled</status>
|
||||||
|
<type>C/C++ from Compilation Database</type>
|
||||||
|
</source_group_9a1b2c3d-0000-4000-8000-kicadwasm0001>
|
||||||
|
</source_groups>
|
||||||
|
<version>8</version>
|
||||||
|
</config>
|
||||||
151
sourcetrail/transform_compile_db.py
Normal file
151
sourcetrail/transform_compile_db.py
Normal file
|
|
@ -0,0 +1,151 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Transform the in-container emscripten compile_commands.json into one
|
||||||
|
Sourcetrail 2021.4.19 (bundled clang ~11) can index on the host.
|
||||||
|
|
||||||
|
- expands @CMakeFiles/....rsp response files inline
|
||||||
|
- strips emscripten-only and PCH flags (old clang can't load clang-20 .pch)
|
||||||
|
- replaces the PCH with `-include cmake_pch.hxx` so those headers still parse
|
||||||
|
- injects libc++ 11 headers (-nostdinc++) + emscripten sysroot includes
|
||||||
|
- rewrites /workspace -> host pcbjam, /emsdk -> host tools/emsdk mirror
|
||||||
|
|
||||||
|
Usage: transform_compile_db.py [--no-wasm-target] <in.json> <out.json>
|
||||||
|
"""
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import shlex
|
||||||
|
import sys
|
||||||
|
|
||||||
|
PCBJAM = "/Users/V/IdeaProjects/pcbjam-private/pcbjam"
|
||||||
|
ST_DIR = f"{PCBJAM}/sourcetrail"
|
||||||
|
EMSDK_HOST = f"{PCBJAM}/tools/emsdk"
|
||||||
|
|
||||||
|
PATH_MAP = [("/workspace/", PCBJAM + "/"), ("/emsdk/", EMSDK_HOST + "/")]
|
||||||
|
|
||||||
|
DROP_EXACT = {
|
||||||
|
"-fwasm-exceptions",
|
||||||
|
"-Winvalid-pch",
|
||||||
|
"--emit-symbol-map",
|
||||||
|
}
|
||||||
|
DROP_PREFIX_RE = re.compile(r"^-s[A-Z][A-Z_0-9]*(=.*)?$|^-gseparate-dwarf")
|
||||||
|
|
||||||
|
|
||||||
|
def host_path(p: str) -> str:
|
||||||
|
for src, dst in PATH_MAP:
|
||||||
|
if p.startswith(src):
|
||||||
|
return dst + p[len(src):]
|
||||||
|
return p
|
||||||
|
|
||||||
|
|
||||||
|
def rewrite_tok(tok: str) -> str:
|
||||||
|
for src, dst in PATH_MAP:
|
||||||
|
tok = tok.replace(src, dst)
|
||||||
|
return tok
|
||||||
|
|
||||||
|
|
||||||
|
def expand_rsp(tokens, directory):
|
||||||
|
out = []
|
||||||
|
for tok in tokens:
|
||||||
|
if tok.startswith("@"):
|
||||||
|
rsp = tok[1:]
|
||||||
|
if not os.path.isabs(rsp):
|
||||||
|
rsp = os.path.join(directory, rsp)
|
||||||
|
rsp_host = host_path(rsp)
|
||||||
|
with open(rsp_host) as f:
|
||||||
|
out.extend(shlex.split(f.read()))
|
||||||
|
else:
|
||||||
|
out.append(tok)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def transform(tokens, directory, wasm_target=True):
|
||||||
|
tokens = expand_rsp(tokens, directory)
|
||||||
|
|
||||||
|
argv0 = tokens[0]
|
||||||
|
lang_cxx = argv0.endswith("++")
|
||||||
|
out = []
|
||||||
|
|
||||||
|
i = 1
|
||||||
|
pch_headers = []
|
||||||
|
while i < len(tokens):
|
||||||
|
tok = tokens[i]
|
||||||
|
if tok == "-Xclang" and i + 1 < len(tokens):
|
||||||
|
nxt = tokens[i + 1]
|
||||||
|
if nxt == "-fno-pch-timestamp":
|
||||||
|
i += 2
|
||||||
|
continue
|
||||||
|
if nxt == "-include-pch":
|
||||||
|
# -Xclang -include-pch -Xclang <path.pch>
|
||||||
|
if i + 3 < len(tokens) and tokens[i + 2] == "-Xclang":
|
||||||
|
pch = tokens[i + 3]
|
||||||
|
hdr = pch[:-4] if pch.endswith(".pch") else pch
|
||||||
|
if os.path.exists(host_path(hdr)):
|
||||||
|
pch_headers.append(hdr)
|
||||||
|
i += 4
|
||||||
|
continue
|
||||||
|
i += 2
|
||||||
|
continue
|
||||||
|
out.extend([tok, nxt])
|
||||||
|
i += 2
|
||||||
|
continue
|
||||||
|
if tok in DROP_EXACT or DROP_PREFIX_RE.match(tok):
|
||||||
|
i += 1
|
||||||
|
continue
|
||||||
|
if not wasm_target and tok in ("-matomics", "-mbulk-memory", "-pthread"):
|
||||||
|
i += 1
|
||||||
|
continue
|
||||||
|
out.append(tok)
|
||||||
|
i += 1
|
||||||
|
|
||||||
|
for hdr in pch_headers:
|
||||||
|
out.extend(["-include", hdr])
|
||||||
|
|
||||||
|
# Rewrite container paths in the ORIGINAL tokens only, then prepend the
|
||||||
|
# host-path injections — rewriting after injection would re-fire on the
|
||||||
|
# /emsdk/ substring inside the host tools/emsdk mirror path.
|
||||||
|
out = [rewrite_tok(t) for t in out]
|
||||||
|
|
||||||
|
inject = ["-isystem", f"{EMSDK_HOST}/upstream/emscripten/cache/sysroot/include",
|
||||||
|
"-fexceptions"]
|
||||||
|
if lang_cxx:
|
||||||
|
inject = ["-nostdinc++", "-isystem", f"{ST_DIR}/libcxx-11/include"] + inject
|
||||||
|
if wasm_target:
|
||||||
|
inject = ["--target=wasm32-unknown-emscripten"] + inject
|
||||||
|
|
||||||
|
return ["clang++" if lang_cxx else "clang"] + inject + out
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
args = sys.argv[1:]
|
||||||
|
wasm_target = True
|
||||||
|
if args and args[0] == "--no-wasm-target":
|
||||||
|
wasm_target = False
|
||||||
|
args = args[1:]
|
||||||
|
src, dst = args
|
||||||
|
|
||||||
|
with open(src) as f:
|
||||||
|
db = json.load(f)
|
||||||
|
|
||||||
|
out_db = []
|
||||||
|
missing = 0
|
||||||
|
for e in db:
|
||||||
|
tokens = shlex.split(e["command"])
|
||||||
|
directory = e["directory"]
|
||||||
|
new_tokens = transform(tokens, directory, wasm_target)
|
||||||
|
file_host = host_path(e["file"])
|
||||||
|
if not os.path.exists(file_host):
|
||||||
|
missing += 1
|
||||||
|
continue
|
||||||
|
out_db.append({
|
||||||
|
"directory": host_path(directory),
|
||||||
|
"command": " ".join(shlex.quote(t) for t in new_tokens),
|
||||||
|
"file": file_host,
|
||||||
|
})
|
||||||
|
|
||||||
|
with open(dst, "w") as f:
|
||||||
|
json.dump(out_db, f, indent=1)
|
||||||
|
print(f"wrote {len(out_db)} entries to {dst} ({missing} skipped: file missing on host)")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
Loading…
Reference in a new issue