diff --git a/docs/asyncify-allocator-suspend/plan.md b/docs/asyncify-allocator-suspend/plan.md new file mode 100644 index 0000000..41bddaf --- /dev/null +++ b/docs/asyncify-allocator-suspend/plan.md @@ -0,0 +1,131 @@ + + +# Nanosleep shim zero-duration guard + mimalloc contention red/green test + +## Context + +The asyncify investigation established that **memory allocation can suspend on the main thread**: emscripten's vendored mimalloc uses `sleep(0)` as its spin-wait yield (`mi_atomic_yield` fallback — wasm32-emscripten matches no arch case in `include/mimalloc/atomic.h`), and `wasm/shims/nanosleep_yield.c` unconditionally converts main-thread `nanosleep` into an Asyncify event-loop yield. Under cross-thread delayed-free contention, malloc's slow path can unwind mid-allocation — a latent reentrancy hazard that also blocks the planned allocator removelist entries (~14K-function instrumentation win). + +Adversarial verification (real kicad_editor binary + emsdk 4.0.2 sources) proved: the `sleep(0)` chain is the allocator's **only** suspend path; all four mimalloc spin sites sleep constant 0; mimalloc is the module's **only zero-duration sleeper** (all other nanosleep callers are ≥1 ms constants); `std::this_thread::yield`/`sched_yield` don't route through nanosleep. Hence an `ms == 0` early return in the shim severs the path completely and affects nothing else. + +**This task:** (1) the zero-duration guard in the shim; (2) a standalone red/green C++ wasm test that exercises the mimalloc contention path and detects main-thread event-loop turns — RED before the fix, GREEN after; (3) add the verified-safe memory-touching entry families to `scripts/common/asyncify-removelist.txt` (user-proposed list, adversarially verified against the real module first). + +Out of scope (explicit follow-ups): full kicad_editor rebuild + 3D e2e suite run with the new removelist (the new entries only take effect at the next app build's post-process anyway). + +## Part 1 — shim fix + +**File:** `pcbjam/wasm/shims/nanosleep_yield.c` (52 lines). + +In `nanosleep()`: after computing `ms`, wrap the yield/sleep branch in `if (ms > 0.0)` — zero-duration requests return 0 immediately on both main thread and workers (a 0 ms blocking sleep is a no-op anyway; `sleep(0)` arrives as exactly `{0,0}` per musl, so the guard fires deterministically). Extend the header comment: mimalloc's `mi_atomic_yield` is `sleep(0)` (spin-politeness hint, must NOT become an event-loop yield mid-malloc); a zero-duration sleep never promised an event-loop turn; verified 2026-08-10 that mimalloc is the only zero-duration caller in the module. + +Rebuild consequences: only binaries that link the shim — future KiCad app builds, `pthread-ondemand`, and the new test. Already-built `output/*.wasm` unchanged. (CI: touching `wasm/**` busts the testapps cache — expected.) + +## Part 2 — the test + +### App: `tests/apps/standalone/mimalloc-storm/mimalloc_storm_test.cpp` + +Plain `int main()` app (non-wx — template: `standalone/coroutine-pthread/main_repro.cpp`), pthreads + asyncify + **`-sMALLOC=mimalloc`** (used nowhere in the test tree today) + the nanosleep shim. + +**Storm** (turns the nanosecond `MI_DELAYED_FREEING` windows into a hit-rate game we control): +- Main thread allocates batches of small same-size blocks (64 B → pages fill fast → full pages enter delayed-free mode; ~200k blocks ≈ ~200 full pages/round). +- W worker `std::thread`s free the previous round's blocks in tight loops (every cross-thread free of a full page runs the two-CAS `DELAYED_FREEING` bracket). +- Main concurrently churns alloc/free in the same size class and calls `mi_collect(true)` each round (`extern "C" void mi_collect(bool)` — public API, headers not on include path; reaches `_mi_heap_delayed_free_all`, the spin site). +- Bounded by ITERATION counts (no wall-clock). Early-exit once ≥50 turns observed (keeps pre-fix RED runs fast); parameters tuned empirically at the red-validation step. + +**Detector** (did main return to the event loop mid-storm?): +- `EM_ASM` arms a self-re-arming `setTimeout(0)` that increments `globalThis.__stormTurns`; a synchronous C++ storm can only let it run if an Asyncify unwind happened inside. Main checks the counter via `EM_ASM_INT` every k iterations. +- Phase 2 assertion (shim's load-bearing behavior unchanged): re-arm marker, `nanosleep(5 ms)` on main → the marker MUST have run (nonzero sleeps still yield). +- Console contract (repo idiom — `EM_ASM` console.log markers, self-terminating, documented in the file header): + - `[MIMALLOC_STORM] START threads=W blocks=B rounds=R` + - `[MIMALLOC_STORM] SUMMARY stormTurns=N sleepTurned=0|1 completed=1` + +### Build rules: `tests/apps/Makefile.wasm` + +- `LDFLAGS_MIMALLOC_STORM` modeled on `LDFLAGS_COROUTINE_PTHREAD_NOWX` (line ~875: EH_FLAGS, ALLOW_MEMORY_GROWTH, ASYNCIFY=1 + stack size, DYNCALLS, -pthread, pool=`navigator.hardwareConcurrency`, STRICT=0, no wx) **plus `-sMALLOC=mimalloc`**. +- `.o` rule for the app; `nanosleep_yield.o` reuse per the `pthread-ondemand` pattern (`Makefile.wasm:681`, compiled `-c -pthread`); `.html` rule; `.PHONY: mimalloc-storm`; `all:` accumulation line. +- No `mallinfo_stub.c` (no OCC). No new suspending import (`env.__asyncjs__*` already in `scripts/common/asyncify-imports.txt`). Post-link asyncify (`apply-asyncify.sh --no-removelist`) + dyncall injection happen automatically via `build-wasm-test.sh`'s `find -newer` fan-out — no driver-script changes. + +### Spec: `tests/e2e/mimalloc-storm.spec.ts` + +- Placement in `tests/e2e/` → runs under the existing `wx-chromium` project; **zero playwright-config/CI edits** and passes `lint:ci-coverage`. +- Pattern copied from `tests/e2e/coroutine-pthread.spec.ts` (the non-wx precedent): `testLogger` fixture, best-effort `tryLoadApp(...).catch(() => {})` with the documented marker comment, then `expect.poll` for the `SUMMARY` line; parse it; assert `stormTurns === 0`, `sleepTurned === 1`, `completed === 1`, and no page errors (favicon-filtered). +- Determinism compliance: no `waitForTimeout`, no inline retries, bounded C++ iterations; the app always emits a terminal SUMMARY (self-capping), so the poll is bounded. + +## Part 3 — asyncify-removelist additions (VERIFIED) + +Adversarial verification against the real kicad_editor module is complete: **all proposed families SAFE**, conditional on the Part 1 shim guard shipping first. Add to `scripts/common/asyncify-removelist.txt` (exact patterns — see traps below): + +``` +# --- Memory-touching families (safe ONLY with the nanosleep zero-duration guard: +# --- mimalloc's mi_atomic_yield is sleep(0); with the guard, no allocation path suspends. +# --- Also conditional: no one registers mi_register_deferred_free/output/error hooks or a +# --- suspending std::new_handler (all unregistered as of 2026-08-10 audit). +std::__2::basic_string<*>* +std::__2::char_traits<*>* +std::__2::vector<*>* +std::__2::__tree* +std::__2::to_string* +std::__2::__itoa* +std::__2::__split_buffer<*>* +std::__2::__shared_ptr_emplace<*>* +std::__2::__shared_ptr_pointer<*>* +std::__2::__shared_count* +std::__2::__shared_weak_count* +std::__2::deque<*>* +std::__2::__hash_table<*>* +std::__2::map<*>* +std::__2::__list_imp<*>* +std::__2::allocator* +boost::uuids::* +operator new* +operator delete* +aligned_alloc +mi_* +_mi_* +sbrk +# Deliberately NOT std::__2::__function* — type-erased std::function invocation (operator()) +# is how tool/dialog callbacks run; 111 __func::operator() bodies reach startModal/wxMilliSleep +# directly (e.g. ShowPreferences, file dialogs), so __function frames must stay instrumented. +# Container entries above are still safe: containers only touch callables via their +# clone/destroy lifecycle ops, none of which reach a suspend (verified 2026-08-10). +``` + +**Authoring traps (verified the hard way):** +- **Bracket balance:** Binaryen's list parser tracks `<>()[]{}` nesting — an unbalanced pattern like `std::__2::basic_string<*` **aborts the whole asyncify pass** (`Fatal: failed to parse lists`). Template patterns must close the bracket: `basic_string<*>*`. +- **Breadth anchoring:** bare `basic_string*` sweeps in `basic_string_view`/`basic_stringbuf`/`basic_stringstream` (31 extra names); the `<`-anchored form excludes them. `__tree*` deliberately includes `__tree_node_base`/iterators (verified). Bare `vector*`/`char_traits*` (no namespace) are wrong forms. + +**Key evidence:** zero direct suspend paths post-guard in every family; `__tree` comparators are monomorphized (no `call_indirect` in e.g. `__find_equal` with `std::less`); container-of-`std::function` ops (e.g. TOOL_MANAGER's `deque>::push_back`) only touch the functor's clone/move/destroy lifecycle slot — of 12,880 `__function` lifecycle ops in the module, zero reach a suspend; all 111 suspend-reaching `__function` members are `operator()` invocation bodies (the excluded family). `boost::uuids` (KIID) seeds `mt19937` once via synchronous `getentropy` (wasi import, not asyncify-relevant). + +**Measured payoff** (single-threaded verbose runs, real hoisted module): + +| Run | Removelist | Instrumented | Module size | +|---|---|---|---| +| A | current file | 80,871 (74%) | 253 MB | +| B | + allocator family | 66,499 (60%) | 218 MB | +| **C** | **+ all families above** | **60,047 (55%)** | **204 MB** | + +Run C is a strict subset of B (sanity: zero additions); only non-matching warnings are the 5 pre-existing OCC entries (expected — OCC absent from this app). Audit validity: this binary / emsdk 4.0.2; re-run on emsdk bump or libc++ ABI-namespace change (artifacts in `$W`: `analysis.pkl`, `logC.txt`, `runC.wasm`). + +## Part 4 — execution sequence + +1. **Build the test against the UNPATCHED shim**: `./scripts/build-wasm-test.sh mimalloc-storm` (targeted — the default `all` target is currently broken by pre-existing working-tree deletions, see caveat). Run the spec: `cd tests && npx playwright test --project=wx-chromium e2e/mimalloc-storm.spec.ts`. Expect the spec to FAIL with `stormTurns > 0` — that failure IS the red validation (proves the storm reaches the mimalloc yield). Record the observed turn count; tune B/R/W if turns are marginal (<~10). +2. **Apply the shim fix** (Part 1), rebuild the same target (make tracks the shim as an explicit prerequisite), re-run the spec → GREEN: `stormTurns === 0`, storm completes (no-deadlock proof), `sleepTurned === 1`. +3. `cd tests && npm run lint:determinism` (spec must pass the linter). +4. **Removelist update:** add the verified entries to `scripts/common/asyncify-removelist.txt` with a documented block: the guard dependency (entries safe ONLY with the zero-duration nanosleep guard), the dormant-slot condition (deferred-free/new_handler/output hooks unregistered), and the `std::__2::__function*` exclusion rationale. Sanity-check the new list against the scratch module: one `wasm-opt --asyncify` run on `$W/hoisted.wasm` with the updated file confirms the expected instrumented-function count and that no entry matches zero functions unexpectedly. +5. Stop before committing — present the diff; commit on main via the user's `/git-feature-commit` flow on request (trunk-based repo). + +## Caveats found during exploration + +- **Pre-existing working-tree deletions** (not ours; do not touch): `tests/apps/standalone/asyncify-races/races_test.cpp`, `tests/asyncify/asyncify-races.spec.ts`, `tests/web/eeschema-fp-selector.spec.ts` are tracked but deleted, which breaks `Makefile.wasm`'s `all` target and full `npm run test:e2e`. We build only our target and run only our spec; flag the state to the user at the end. +- `build-wasm-test.sh` requires `build-wasm/wxwidgets/wx-config` (wx libs built) and stubs the emsdk in-link asyncify — the emsdk lives at `tools/emsdk/` (installed via `scripts/setup-emsdk.sh` if absent). +- Reference precedents: `races_test.cpp` (via `git show HEAD:...`) for Asyncify-state probing from C++ if the setTimeout detector needs corroboration; `pthread_ondemand_test.cpp` for the watchdog/marker idiom. + +## Verification summary + +- RED observed pre-fix (spec fails on `stormTurns > 0`), GREEN post-fix on identical parameters — the core deliverable. +- GREEN also proves: spin completes without the yield (no deadlock), 5 ms sleep still turns the event loop (worker-boot behavior preserved). +- `lint:determinism` clean. +- Removelist sanity run on `$W/hoisted.wasm` reproduces run C's numbers (60,047 instrumented / 204 MB pre-O1) with no unexpected non-matching-pattern warnings. +- Follow-ups NOT here: full editor rebuild + 3D e2e suite with the new removelist (entries take effect at the next app build's host post-process). diff --git a/sourcetrail/.gitignore b/sourcetrail/.gitignore new file mode 100644 index 0000000..429e4ad --- /dev/null +++ b/sourcetrail/.gitignore @@ -0,0 +1,9 @@ +# Generated/machine-local artifacts — only the README, scripts, and project +# files are tracked. Everything below is rebuilt by the steps in README.md. +*.srctrldb +*.srctrldb_tmp +*.srctrlbm +compile_commands.json +sample_compile_commands.json +libcxx-11/ +libcxx-11.src.tar.xz diff --git a/sourcetrail/README.md b/sourcetrail/README.md new file mode 100644 index 0000000..06d96a7 --- /dev/null +++ b/sourcetrail/README.md @@ -0,0 +1,113 @@ +# Sourcetrail code graph for the KiCad WASM port + +Interactive symbol-level code graph (classes, calls, includes, inheritance) of the +merged `kicad_editor` build — all 2,176 TUs: pcbnew, eeschema, common, 3D viewer, +and the `wasm/` port layer — indexed by [Sourcetrail](https://github.com/CoatiSoftware/Sourcetrail). +Lives in `pcbjam/sourcetrail/`: README, scripts and project files are tracked; the heavy +artifacts (index db, compile db, libc++ headers) are ignored via the local `.gitignore` +and rebuilt with the steps below. + +## What's in this folder + +| File | Purpose | +|---|---| +| `kicad-wasm.srctrlprj` / `.srctrldb` | Sourcetrail project + indexed database (~440 MB) | +| `compile_commands.json` | Transformed compile db the indexer consumes | +| `transform_compile_db.py` | Turns the container's emscripten compile db into the above | +| `libcxx-11/` | Pinned libc++ 11.1.0 headers (see Tricks) | +| `asyncify_candidates.py` | Ranks subsystems safe for the asyncify removelist | +| `sample.srctrlprj` + `sample_compile_commands.json` | 6-file smoke test for the pipeline | + +## Install + +- **Sourcetrail 2021.4.19** — the last free release (the maintained fork paywalls binaries): + , macOS dmg → `/Applications`, + then `xattr -dr com.apple.quarantine /Applications/Sourcetrail.app`. It's x86_64 — needs Rosetta 2 + on Apple Silicon (works fine). +- Nothing else: python3 stdlib only; header mirrors are exported from the Docker build. + +## Run + +```bash +# browse (GUI) +open -a Sourcetrail /Users/V/IdeaProjects/pcbjam-private/pcbjam/sourcetrail/kicad-wasm.srctrlprj + +# (re)index from the terminal — ABSOLUTE project path, see Tricks +/Applications/Sourcetrail.app/Contents/MacOS/Sourcetrail index \ + --project-file /Users/V/IdeaProjects/pcbjam-private/pcbjam/sourcetrail/kicad-wasm.srctrlprj +``` + +Full index ≈ 12 min all-cores. In the GUI there is no single whole-project graph: search a +symbol (Cmd+F — e.g. `BOARD`, `KIPLATFORM`) and click it; the graph pane centers on it and +expands as you click nodes/edges. + +### Refreshing after a KiCad rebuild + +The compile db comes from the `main`-branch Docker build cache (KiCad's CMake exports it +by default — no reconfigure needed). From `pcbjam/`: + +```bash +source scripts/common/versions.sh +COMPOSE_PROJECT_NAME=kicad-wasm-main docker compose -f docker/docker-compose.yml up -d --build + +# compile db + generated sources/headers + .rsp files + deps/wx includes +docker compose -p kicad-wasm-main -f docker/docker-compose.yml exec -T kicad-wasm-builder bash -c \ + "cd /workspace && { find build-wasm/kicad-kicad_editor \( -name '*.h' -o -name '*.hpp' -o -name '*.hxx' \ + -o -name '*.hh' -o -name '*.inc' -o -name '*.rsp' -o -name '*.cc' -o -name '*.cpp' -o -name '*.cxx' \ + -o -name 'compile_commands.json' \) -type f; echo build-wasm/sysroot/include; \ + echo build-wasm/wxwidgets/lib/wx/include; } | tar -cf - -T -" | tar -xf - -C . + +# emscripten sysroot headers -> tools/emsdk mirror (only after an emsdk bump) +docker compose -p kicad-wasm-main -f docker/docker-compose.yml exec -T kicad-wasm-builder \ + tar -cf - -C / emsdk/upstream/emscripten/cache/sysroot/include | tar -xf - -C tools/ + +cd sourcetrail +python3 transform_compile_db.py ../kicad-kicad_editor/compile_commands.json compile_commands.json +# then the `Sourcetrail index` command above +``` + +## Tricks (why this isn't just "point Sourcetrail at the cdb") + +- **The bundled clang is ~LLVM 11.** Modern libc++ (emsdk 4.x's or the macOS SDK's) does not + parse under it. The transform pins `libcxx-11/` via `-nostdinc++` and takes C headers from the + emsdk musl sysroot mirror (`tools/emsdk/`). Expected residue: ~14 errors, all in the + libc++11/musl locale seam (`_CTYPE_*`, `strtoull_l`, one fatal `xlocale.h`) — harmless to the graph. +- **Transform surgery:** `@CMakeFiles/*.rsp` response files are expanded inline (old clang's cdb + loader can't); PCH is stripped (`-Xclang -include-pch` of clang-20 `.pch` binaries) and replaced + with `-include cmake_pch.hxx`; emscripten-only flags (`-sFOO`, `-fwasm-exceptions`) dropped, + `--target=wasm32-unknown-emscripten` + `-fexceptions` added; paths rewritten + `/workspace` → `pcbjam/`, `/emsdk` → `pcbjam/tools/emsdk/`. +- **CLI hangs on relative `--project-file` paths.** Silently — idle event loop, log stops after + "Maven executable path detection". Always pass absolute paths. +- **Global header paths are deliberately empty** in + `~/Library/Application Support/Sourcetrail/ApplicationSettings.xml`. First launch auto-filled + macOS-26-SDK paths, which poison every parse (see clang-11 point). Don't re-run header path + detection from Preferences; `has_prefilled_header_search_paths=1` keeps it from coming back. +- **"N files (126 complete)" undersells the index.** A file counts as complete only if *every* TU + touching it had zero errors; the 14 std-header errors are included nearly everywhere, so the + flag cascades. The symbols/references themselves are all recorded. + +## Asyncify removelist candidates + +```bash +python3 asyncify_candidates.py +``` + +Computes, over the indexed call graph (187K call edges + override pseudo-edges for virtual +dispatch), which functions can NEVER reach a suspend point, aggregated per module. Seeds = +every `ShowModal`/`Yield`/`Sleep`/progress-dialog/`COROUTINE` function (**strict**), plus +`ProcessEvent`-style synchronous dispatch (**lenient** — a dispatched handler may suspend and +unwind through the dispatcher). Functions clean under *lenient* are candidates for +`scripts/common/asyncify-removelist.txt` (matching rules are documented in that file: +one prefix wildcard per symbol, e.g. `SHAPE_POLY_SET::*`). + +Headline results from the 2026-08-07 index: `kiapi` generated protobuf (15.2K funcs), +`libs/kimath` (7.3K), `clipper2` (6.3K), `nlohmann_json`/`fmt`/`pegtl`/`zint` are 100% clean; +`pcbnew/router` is 1,820/1,827 clean (the 7 are the `Wait()` tool-integration layer — the PNS +shove/optimizer core never suspends). + +**Caveats:** the C++ graph can't see calls through `std::function`, event tables, or raw function +pointers, and asyncify operates on the post-inlining *wasm* call graph, not C++ symbols. Before +shipping an entry: ground-truth with Binaryen's asyncify verbose/advise output in +`apply-asyncify.sh`, and rely on e2e — a wrong removal traps loudly (`unreachable`) at the first +unwind through it. Tweak seeds / path depths at the top of the script. diff --git a/sourcetrail/asyncify_candidates.py b/sourcetrail/asyncify_candidates.py new file mode 100644 index 0000000..78372e9 --- /dev/null +++ b/sourcetrail/asyncify_candidates.py @@ -0,0 +1,125 @@ +#!/usr/bin/env python3 +"""Rank subsystems by asyncify-removelist safety using the Sourcetrail db. + +Taint = "a suspend point is reachable from this function" (reverse BFS over +call edges from suspend primitives, with base->override pseudo-edges for +virtual dispatch). Two levels: + strict — direct suspend primitives only (Yield/Sleep/ShowModal/progress/ + coroutine/fiber) + lenient — strict + synchronous event dispatch (ProcessEvent & co), since a + dispatched handler may suspend and unwind through the dispatcher + +Functions never tainted even in lenient mode are removelist candidates. +Blind spots (validate with ASYNCIFY_ADVISE before shipping): std::function / +event-table indirection, function pointers, wx/libc internals outside the cdb. +""" +import re +import sqlite3 +from collections import defaultdict, deque + +DB = "kicad-wasm.srctrldb" +PCBJAM = "/Users/V/IdeaProjects/pcbjam-private/pcbjam/" +FUNC_TYPES = (4096, 8192) + +STRICT_PATTERNS = [ + r"\tnShowModal", # every ShowModal incl. DIALOG_SHIM/KIDIALOG wrappers + r"\tnShowWindowModal", + r"\tnShowQuasiModal", + r"mwxYield", r"mwxSafeYield", r"mwxYieldIfNeeded", + r"\tnYield", r"\tnYieldFor", r"\tnDoYieldFor", r"\tnSafeYield", + r"\tnSleep", r"mwxSleep", r"mwxMilliSleep", r"mwxMicroSleep", + r"sleep_for", r"sleep_until", r"nanosleep", r"emscripten_sleep", + r"emscripten_fiber", r"__asyncjs__", + r"mwxGenericProgressDialog", r"mwxProgressDialog", + r"mwxMessageBox", r"mwxExecute", + r"mCOROUTINE<", +] +LENIENT_EXTRA = [ + r"\tnProcessEvent\t", r"\tnSafelyProcessEvent", r"\tnProcessPendingEvents", + r"\tnHandleEvent\t", r"\tnProcessEventLocally", +] + + +def main(): + con = sqlite3.connect(DB) + + funcs = {} # id -> serialized_name + for nid, name in con.execute( + f"SELECT id, serialized_name FROM node WHERE type IN {FUNC_TYPES}"): + funcs[nid] = name + + # callee -> callers (reverse call graph) + rev = defaultdict(list) + for s, t in con.execute("SELECT source_node_id, target_node_id FROM edge WHERE type=8"): + rev[t].append(s) + # virtual dispatch: caller of Base::f may land in Derived::f, so taint of + # Derived::f must flow to callers of Base::f -> pseudo-edge callee=Derived, + # caller-side=Base is wrong; we need: if Derived tainted then Base tainted + # is NOT true. Correct: call to Base::f can dispatch to Derived::f, so if + # Derived::f suspends, callers of Base::f suspend. Model: rev[Derived] gets + # nothing; instead treat Base::f as a caller of every override Derived::f. + for s, t in con.execute("SELECT source_node_id, target_node_id FROM edge WHERE type=32"): + # override edge: Derived::f (source) -> Base::f (target) + rev[s].append(t) # taint flows Derived -> Base -> Base's callers + + def seeds_for(patterns): + pats = [re.compile(p) for p in patterns] + return {nid for nid, name in funcs.items() if any(p.search(name) for p in pats)} + + def closure(seed_ids): + seen = set(seed_ids) + q = deque(seed_ids) + while q: + n = q.popleft() + for caller in rev.get(n, ()): + if caller not in seen: + seen.add(caller) + q.append(caller) + return seen + + strict_seeds = seeds_for(STRICT_PATTERNS) + lenient_seeds = strict_seeds | seeds_for(STRICT_PATTERNS + LENIENT_EXTRA) + print(f"functions: {len(funcs)} strict seeds: {len(strict_seeds)} " + f"lenient seeds: {len(lenient_seeds)}") + + tainted_strict = closure(strict_seeds) + tainted_lenient = closure(lenient_seeds) + + # function -> file (prefer definition scope locations, type=1) + loc = {} + for typ in (1, 0): + for eid, path in con.execute( + "SELECT o.element_id, f.path FROM occurrence o " + "JOIN source_location sl ON sl.id=o.source_location_id " + "JOIN file f ON f.id=sl.file_node_id WHERE sl.type=?", (typ,)): + if eid in funcs and eid not in loc: + loc[eid] = path + + def module(path): + if not path.startswith(PCBJAM): + return None # std/emsdk/deps headers — not our code + rel = path[len(PCBJAM):] + parts = rel.split("/") + depth = 3 if parts[0] in ("kicad", "build-wasm") else 2 + return "/".join(parts[:depth]) if len(parts) > depth else "/".join(parts[:-1]) + + stats = defaultdict(lambda: [0, 0, 0]) # module -> [total, strict, lenient] + for nid in funcs: + m = module(loc.get(nid, "")) + if m is None: + continue + stats[m][0] += 1 + if nid in tainted_strict: + stats[m][1] += 1 + if nid in tainted_lenient: + stats[m][2] += 1 + + rows = [(m, t, s, l, t - l) for m, (t, s, l) in stats.items() if t >= 20] + rows.sort(key=lambda r: -r[4]) + print(f"\n{'module':<44}{'funcs':>7}{'strict✗':>9}{'lenient✗':>9}{'clean':>7}{'clean%':>8}") + for m, t, s, l, clean in rows[:45]: + print(f"{m:<44}{t:>7}{s:>9}{l:>9}{clean:>7}{100*clean//t:>7}%") + + +if __name__ == "__main__": + main() diff --git a/sourcetrail/kicad-wasm.srctrlprj b/sourcetrail/kicad-wasm.srctrlprj new file mode 100644 index 0000000..bb40c45 --- /dev/null +++ b/sourcetrail/kicad-wasm.srctrlprj @@ -0,0 +1,21 @@ + + + + + + ./compile_commands.json + + + /Users/V/IdeaProjects/pcbjam-private/pcbjam/kicad + /Users/V/IdeaProjects/pcbjam-private/pcbjam/wasm + /Users/V/IdeaProjects/pcbjam-private/pcbjam/wxwidgets/include + /Users/V/IdeaProjects/pcbjam-private/pcbjam/build-wasm/kicad-kicad_editor + /Users/V/IdeaProjects/pcbjam-private/pcbjam/build-wasm/wxwidgets/lib/wx/include + + KiCad WASM + enabled + C/C++ from Compilation Database + + + 8 + diff --git a/sourcetrail/sample.srctrlprj b/sourcetrail/sample.srctrlprj new file mode 100644 index 0000000..237fcf4 --- /dev/null +++ b/sourcetrail/sample.srctrlprj @@ -0,0 +1,21 @@ + + + + + + ./sample_compile_commands.json + + + /Users/V/IdeaProjects/pcbjam-private/pcbjam/kicad + /Users/V/IdeaProjects/pcbjam-private/pcbjam/wasm + /Users/V/IdeaProjects/pcbjam-private/pcbjam/wxwidgets/include + /Users/V/IdeaProjects/pcbjam-private/pcbjam/build-wasm/kicad-kicad_editor + /Users/V/IdeaProjects/pcbjam-private/pcbjam/build-wasm/wxwidgets/lib/wx/include + + KiCad WASM sample + enabled + C/C++ from Compilation Database + + + 8 + diff --git a/sourcetrail/transform_compile_db.py b/sourcetrail/transform_compile_db.py new file mode 100644 index 0000000..efaa8b3 --- /dev/null +++ b/sourcetrail/transform_compile_db.py @@ -0,0 +1,151 @@ +#!/usr/bin/env python3 +"""Transform the in-container emscripten compile_commands.json into one +Sourcetrail 2021.4.19 (bundled clang ~11) can index on the host. + +- expands @CMakeFiles/....rsp response files inline +- strips emscripten-only and PCH flags (old clang can't load clang-20 .pch) +- replaces the PCH with `-include cmake_pch.hxx` so those headers still parse +- injects libc++ 11 headers (-nostdinc++) + emscripten sysroot includes +- rewrites /workspace -> host pcbjam, /emsdk -> host tools/emsdk mirror + +Usage: transform_compile_db.py [--no-wasm-target] +""" +import json +import os +import re +import shlex +import sys + +PCBJAM = "/Users/V/IdeaProjects/pcbjam-private/pcbjam" +ST_DIR = f"{PCBJAM}/sourcetrail" +EMSDK_HOST = f"{PCBJAM}/tools/emsdk" + +PATH_MAP = [("/workspace/", PCBJAM + "/"), ("/emsdk/", EMSDK_HOST + "/")] + +DROP_EXACT = { + "-fwasm-exceptions", + "-Winvalid-pch", + "--emit-symbol-map", +} +DROP_PREFIX_RE = re.compile(r"^-s[A-Z][A-Z_0-9]*(=.*)?$|^-gseparate-dwarf") + + +def host_path(p: str) -> str: + for src, dst in PATH_MAP: + if p.startswith(src): + return dst + p[len(src):] + return p + + +def rewrite_tok(tok: str) -> str: + for src, dst in PATH_MAP: + tok = tok.replace(src, dst) + return tok + + +def expand_rsp(tokens, directory): + out = [] + for tok in tokens: + if tok.startswith("@"): + rsp = tok[1:] + if not os.path.isabs(rsp): + rsp = os.path.join(directory, rsp) + rsp_host = host_path(rsp) + with open(rsp_host) as f: + out.extend(shlex.split(f.read())) + else: + out.append(tok) + return out + + +def transform(tokens, directory, wasm_target=True): + tokens = expand_rsp(tokens, directory) + + argv0 = tokens[0] + lang_cxx = argv0.endswith("++") + out = [] + + i = 1 + pch_headers = [] + while i < len(tokens): + tok = tokens[i] + if tok == "-Xclang" and i + 1 < len(tokens): + nxt = tokens[i + 1] + if nxt == "-fno-pch-timestamp": + i += 2 + continue + if nxt == "-include-pch": + # -Xclang -include-pch -Xclang + if i + 3 < len(tokens) and tokens[i + 2] == "-Xclang": + pch = tokens[i + 3] + hdr = pch[:-4] if pch.endswith(".pch") else pch + if os.path.exists(host_path(hdr)): + pch_headers.append(hdr) + i += 4 + continue + i += 2 + continue + out.extend([tok, nxt]) + i += 2 + continue + if tok in DROP_EXACT or DROP_PREFIX_RE.match(tok): + i += 1 + continue + if not wasm_target and tok in ("-matomics", "-mbulk-memory", "-pthread"): + i += 1 + continue + out.append(tok) + i += 1 + + for hdr in pch_headers: + out.extend(["-include", hdr]) + + # Rewrite container paths in the ORIGINAL tokens only, then prepend the + # host-path injections — rewriting after injection would re-fire on the + # /emsdk/ substring inside the host tools/emsdk mirror path. + out = [rewrite_tok(t) for t in out] + + inject = ["-isystem", f"{EMSDK_HOST}/upstream/emscripten/cache/sysroot/include", + "-fexceptions"] + if lang_cxx: + inject = ["-nostdinc++", "-isystem", f"{ST_DIR}/libcxx-11/include"] + inject + if wasm_target: + inject = ["--target=wasm32-unknown-emscripten"] + inject + + return ["clang++" if lang_cxx else "clang"] + inject + out + + +def main(): + args = sys.argv[1:] + wasm_target = True + if args and args[0] == "--no-wasm-target": + wasm_target = False + args = args[1:] + src, dst = args + + with open(src) as f: + db = json.load(f) + + out_db = [] + missing = 0 + for e in db: + tokens = shlex.split(e["command"]) + directory = e["directory"] + new_tokens = transform(tokens, directory, wasm_target) + file_host = host_path(e["file"]) + if not os.path.exists(file_host): + missing += 1 + continue + out_db.append({ + "directory": host_path(directory), + "command": " ".join(shlex.quote(t) for t in new_tokens), + "file": file_host, + }) + + with open(dst, "w") as f: + json.dump(out_db, f, indent=1) + print(f"wrote {len(out_db)} entries to {dst} ({missing} skipped: file missing on host)") + + +if __name__ == "__main__": + main()