tooling(sourcetrail): code-graph setup + asyncify allocator-suspend plan

sourcetrail/: Sourcetrail 2021.4.19 indexing pipeline for the wasm port —
compile-db transform (rsp expansion, PCH strip, libc++-11 pinning, path
rewrites), removelist-candidate analysis over the indexed call graph, README
with regen steps and the hard-won tricks (relative --project-file hang,
SDK-header poisoning). Heavy artifacts (.srctrldb, compile dbs, libcxx
headers) stay untracked via the folder's .gitignore.

docs/asyncify-allocator-suspend/plan.md: verified plan for the nanosleep-shim
zero-duration guard (mimalloc mi_atomic_yield=sleep(0) can suspend malloc on
the main thread), the red/green contention test, and the adversarially
verified removelist additions with measured payoff (80.9k -> 60.0k
instrumented functions). Planned, not yet executed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P2NHhbzuEHqP2JmcrSD96D

sourcetrail/.gitignore

@@ -0,0 +1,9 @@
+# Generated/machine-local artifacts — only the README, scripts, and project
+# files are tracked. Everything below is rebuilt by the steps in README.md.
+*.srctrldb
+*.srctrldb_tmp
+*.srctrlbm
+compile_commands.json
+sample_compile_commands.json
+libcxx-11/
+libcxx-11.src.tar.xz

sourcetrail/README.md

@@ -0,0 +1,113 @@
+# Sourcetrail code graph for the KiCad WASM port
+
+Interactive symbol-level code graph (classes, calls, includes, inheritance) of the
+merged `kicad_editor` build — all 2,176 TUs: pcbnew, eeschema, common, 3D viewer,
+and the `wasm/` port layer — indexed by [Sourcetrail](https://github.com/CoatiSoftware/Sourcetrail).
+Lives in `pcbjam/sourcetrail/`: README, scripts and project files are tracked; the heavy
+artifacts (index db, compile db, libc++ headers) are ignored via the local `.gitignore`
+and rebuilt with the steps below.
+
+## What's in this folder
+
+| File | Purpose |
+|---|---|
+| `kicad-wasm.srctrlprj` / `.srctrldb` | Sourcetrail project + indexed database (~440 MB) |
+| `compile_commands.json` | Transformed compile db the indexer consumes |
+| `transform_compile_db.py` | Turns the container's emscripten compile db into the above |
+| `libcxx-11/` | Pinned libc++ 11.1.0 headers (see Tricks) |
+| `asyncify_candidates.py` | Ranks subsystems safe for the asyncify removelist |
+| `sample.srctrlprj` + `sample_compile_commands.json` | 6-file smoke test for the pipeline |
+
+## Install
+
+- **Sourcetrail 2021.4.19** — the last free release (the maintained fork paywalls binaries):
+  <https://github.com/CoatiSoftware/Sourcetrail/releases/tag/2021.4.19>, macOS dmg → `/Applications`,
+  then `xattr -dr com.apple.quarantine /Applications/Sourcetrail.app`. It's x86_64 — needs Rosetta 2
+  on Apple Silicon (works fine).
+- Nothing else: python3 stdlib only; header mirrors are exported from the Docker build.
+
+## Run
+
+```bash
+# browse (GUI)
+open -a Sourcetrail /Users/V/IdeaProjects/pcbjam-private/pcbjam/sourcetrail/kicad-wasm.srctrlprj
+
+# (re)index from the terminal — ABSOLUTE project path, see Tricks
+/Applications/Sourcetrail.app/Contents/MacOS/Sourcetrail index \
+  --project-file /Users/V/IdeaProjects/pcbjam-private/pcbjam/sourcetrail/kicad-wasm.srctrlprj
+```
+
+Full index ≈ 12 min all-cores. In the GUI there is no single whole-project graph: search a
+symbol (Cmd+F — e.g. `BOARD`, `KIPLATFORM`) and click it; the graph pane centers on it and
+expands as you click nodes/edges.
+
+### Refreshing after a KiCad rebuild
+
+The compile db comes from the `main`-branch Docker build cache (KiCad's CMake exports it
+by default — no reconfigure needed). From `pcbjam/`:
+
+```bash
+source scripts/common/versions.sh
+COMPOSE_PROJECT_NAME=kicad-wasm-main docker compose -f docker/docker-compose.yml up -d --build
+
+# compile db + generated sources/headers + .rsp files + deps/wx includes
+docker compose -p kicad-wasm-main -f docker/docker-compose.yml exec -T kicad-wasm-builder bash -c \
+  "cd /workspace && { find build-wasm/kicad-kicad_editor \( -name '*.h' -o -name '*.hpp' -o -name '*.hxx' \
+   -o -name '*.hh' -o -name '*.inc' -o -name '*.rsp' -o -name '*.cc' -o -name '*.cpp' -o -name '*.cxx' \
+   -o -name 'compile_commands.json' \) -type f; echo build-wasm/sysroot/include; \
+   echo build-wasm/wxwidgets/lib/wx/include; } | tar -cf - -T -" | tar -xf - -C .
+
+# emscripten sysroot headers -> tools/emsdk mirror (only after an emsdk bump)
+docker compose -p kicad-wasm-main -f docker/docker-compose.yml exec -T kicad-wasm-builder \
+  tar -cf - -C / emsdk/upstream/emscripten/cache/sysroot/include | tar -xf - -C tools/
+
+cd sourcetrail
+python3 transform_compile_db.py ../kicad-kicad_editor/compile_commands.json compile_commands.json
+# then the `Sourcetrail index` command above
+```
+
+## Tricks (why this isn't just "point Sourcetrail at the cdb")
+
+- **The bundled clang is ~LLVM 11.** Modern libc++ (emsdk 4.x's or the macOS SDK's) does not
+  parse under it. The transform pins `libcxx-11/` via `-nostdinc++` and takes C headers from the
+  emsdk musl sysroot mirror (`tools/emsdk/`). Expected residue: ~14 errors, all in the
+  libc++11/musl locale seam (`_CTYPE_*`, `strtoull_l`, one fatal `xlocale.h`) — harmless to the graph.
+- **Transform surgery:** `@CMakeFiles/*.rsp` response files are expanded inline (old clang's cdb
+  loader can't); PCH is stripped (`-Xclang -include-pch` of clang-20 `.pch` binaries) and replaced
+  with `-include cmake_pch.hxx`; emscripten-only flags (`-sFOO`, `-fwasm-exceptions`) dropped,
+  `--target=wasm32-unknown-emscripten` + `-fexceptions` added; paths rewritten
+  `/workspace` → `pcbjam/`, `/emsdk` → `pcbjam/tools/emsdk/`.
+- **CLI hangs on relative `--project-file` paths.** Silently — idle event loop, log stops after
+  "Maven executable path detection". Always pass absolute paths.
+- **Global header paths are deliberately empty** in
+  `~/Library/Application Support/Sourcetrail/ApplicationSettings.xml`. First launch auto-filled
+  macOS-26-SDK paths, which poison every parse (see clang-11 point). Don't re-run header path
+  detection from Preferences; `has_prefilled_header_search_paths=1` keeps it from coming back.
+- **"N files (126 complete)" undersells the index.** A file counts as complete only if *every* TU
+  touching it had zero errors; the 14 std-header errors are included nearly everywhere, so the
+  flag cascades. The symbols/references themselves are all recorded.
+
+## Asyncify removelist candidates
+
+```bash
+python3 asyncify_candidates.py
+```
+
+Computes, over the indexed call graph (187K call edges + override pseudo-edges for virtual
+dispatch), which functions can NEVER reach a suspend point, aggregated per module. Seeds =
+every `ShowModal`/`Yield`/`Sleep`/progress-dialog/`COROUTINE` function (**strict**), plus
+`ProcessEvent`-style synchronous dispatch (**lenient** — a dispatched handler may suspend and
+unwind through the dispatcher). Functions clean under *lenient* are candidates for
+`scripts/common/asyncify-removelist.txt` (matching rules are documented in that file:
+one prefix wildcard per symbol, e.g. `SHAPE_POLY_SET::*`).
+
+Headline results from the 2026-08-07 index: `kiapi` generated protobuf (15.2K funcs),
+`libs/kimath` (7.3K), `clipper2` (6.3K), `nlohmann_json`/`fmt`/`pegtl`/`zint` are 100% clean;
+`pcbnew/router` is 1,820/1,827 clean (the 7 are the `Wait()` tool-integration layer — the PNS
+shove/optimizer core never suspends).
+
+**Caveats:** the C++ graph can't see calls through `std::function`, event tables, or raw function
+pointers, and asyncify operates on the post-inlining *wasm* call graph, not C++ symbols. Before
+shipping an entry: ground-truth with Binaryen's asyncify verbose/advise output in
+`apply-asyncify.sh`, and rely on e2e — a wrong removal traps loudly (`unreachable`) at the first
+unwind through it. Tweak seeds / path depths at the top of the script.

sourcetrail/asyncify_candidates.py

@@ -0,0 +1,125 @@
+#!/usr/bin/env python3
+"""Rank subsystems by asyncify-removelist safety using the Sourcetrail db.
+
+Taint = "a suspend point is reachable from this function" (reverse BFS over
+call edges from suspend primitives, with base->override pseudo-edges for
+virtual dispatch).  Two levels:
+  strict  — direct suspend primitives only (Yield/Sleep/ShowModal/progress/
+            coroutine/fiber)
+  lenient — strict + synchronous event dispatch (ProcessEvent & co), since a
+            dispatched handler may suspend and unwind through the dispatcher
+
+Functions never tainted even in lenient mode are removelist candidates.
+Blind spots (validate with ASYNCIFY_ADVISE before shipping): std::function /
+event-table indirection, function pointers, wx/libc internals outside the cdb.
+"""
+import re
+import sqlite3
+from collections import defaultdict, deque
+
+DB = "kicad-wasm.srctrldb"
+PCBJAM = "/Users/V/IdeaProjects/pcbjam-private/pcbjam/"
+FUNC_TYPES = (4096, 8192)
+
+STRICT_PATTERNS = [
+    r"\tnShowModal",  # every ShowModal incl. DIALOG_SHIM/KIDIALOG wrappers
+    r"\tnShowWindowModal",
+    r"\tnShowQuasiModal",
+    r"mwxYield", r"mwxSafeYield", r"mwxYieldIfNeeded",
+    r"\tnYield", r"\tnYieldFor", r"\tnDoYieldFor", r"\tnSafeYield",
+    r"\tnSleep", r"mwxSleep", r"mwxMilliSleep", r"mwxMicroSleep",
+    r"sleep_for", r"sleep_until", r"nanosleep", r"emscripten_sleep",
+    r"emscripten_fiber", r"__asyncjs__",
+    r"mwxGenericProgressDialog", r"mwxProgressDialog",
+    r"mwxMessageBox", r"mwxExecute",
+    r"mCOROUTINE<",
+]
+LENIENT_EXTRA = [
+    r"\tnProcessEvent\t", r"\tnSafelyProcessEvent", r"\tnProcessPendingEvents",
+    r"\tnHandleEvent\t", r"\tnProcessEventLocally",
+]
+
+
+def main():
+    con = sqlite3.connect(DB)
+
+    funcs = {}  # id -> serialized_name
+    for nid, name in con.execute(
+            f"SELECT id, serialized_name FROM node WHERE type IN {FUNC_TYPES}"):
+        funcs[nid] = name
+
+    # callee -> callers (reverse call graph)
+    rev = defaultdict(list)
+    for s, t in con.execute("SELECT source_node_id, target_node_id FROM edge WHERE type=8"):
+        rev[t].append(s)
+    # virtual dispatch: caller of Base::f may land in Derived::f, so taint of
+    # Derived::f must flow to callers of Base::f -> pseudo-edge callee=Derived,
+    # caller-side=Base is wrong; we need: if Derived tainted then Base tainted
+    # is NOT true. Correct: call to Base::f can dispatch to Derived::f, so if
+    # Derived::f suspends, callers of Base::f suspend. Model: rev[Derived] gets
+    # nothing; instead treat Base::f as a caller of every override Derived::f.
+    for s, t in con.execute("SELECT source_node_id, target_node_id FROM edge WHERE type=32"):
+        # override edge: Derived::f (source) -> Base::f (target)
+        rev[s].append(t)  # taint flows Derived -> Base -> Base's callers
+
+    def seeds_for(patterns):
+        pats = [re.compile(p) for p in patterns]
+        return {nid for nid, name in funcs.items() if any(p.search(name) for p in pats)}
+
+    def closure(seed_ids):
+        seen = set(seed_ids)
+        q = deque(seed_ids)
+        while q:
+            n = q.popleft()
+            for caller in rev.get(n, ()):
+                if caller not in seen:
+                    seen.add(caller)
+                    q.append(caller)
+        return seen
+
+    strict_seeds = seeds_for(STRICT_PATTERNS)
+    lenient_seeds = strict_seeds | seeds_for(STRICT_PATTERNS + LENIENT_EXTRA)
+    print(f"functions: {len(funcs)}  strict seeds: {len(strict_seeds)}  "
+          f"lenient seeds: {len(lenient_seeds)}")
+
+    tainted_strict = closure(strict_seeds)
+    tainted_lenient = closure(lenient_seeds)
+
+    # function -> file (prefer definition scope locations, type=1)
+    loc = {}
+    for typ in (1, 0):
+        for eid, path in con.execute(
+                "SELECT o.element_id, f.path FROM occurrence o "
+                "JOIN source_location sl ON sl.id=o.source_location_id "
+                "JOIN file f ON f.id=sl.file_node_id WHERE sl.type=?", (typ,)):
+            if eid in funcs and eid not in loc:
+                loc[eid] = path
+
+    def module(path):
+        if not path.startswith(PCBJAM):
+            return None  # std/emsdk/deps headers — not our code
+        rel = path[len(PCBJAM):]
+        parts = rel.split("/")
+        depth = 3 if parts[0] in ("kicad", "build-wasm") else 2
+        return "/".join(parts[:depth]) if len(parts) > depth else "/".join(parts[:-1])
+
+    stats = defaultdict(lambda: [0, 0, 0])  # module -> [total, strict, lenient]
+    for nid in funcs:
+        m = module(loc.get(nid, ""))
+        if m is None:
+            continue
+        stats[m][0] += 1
+        if nid in tainted_strict:
+            stats[m][1] += 1
+        if nid in tainted_lenient:
+            stats[m][2] += 1
+
+    rows = [(m, t, s, l, t - l) for m, (t, s, l) in stats.items() if t >= 20]
+    rows.sort(key=lambda r: -r[4])
+    print(f"\n{'module':<44}{'funcs':>7}{'strict✗':>9}{'lenient✗':>9}{'clean':>7}{'clean%':>8}")
+    for m, t, s, l, clean in rows[:45]:
+        print(f"{m:<44}{t:>7}{s:>9}{l:>9}{clean:>7}{100*clean//t:>7}%")
+
+
+if __name__ == "__main__":
+    main()

sourcetrail/kicad-wasm.srctrlprj

@@ -0,0 +1,21 @@
+<?xml version="1.0" encoding="utf-8" ?>
+<config>
+    <source_groups>
+        <source_group_9a1b2c3d-0000-4000-8000-kicadwasm0001>
+            <build_file_path>
+                <compilation_db_path>./compile_commands.json</compilation_db_path>
+            </build_file_path>
+            <indexed_header_paths>
+                <indexed_header_path>/Users/V/IdeaProjects/pcbjam-private/pcbjam/kicad</indexed_header_path>
+                <indexed_header_path>/Users/V/IdeaProjects/pcbjam-private/pcbjam/wasm</indexed_header_path>
+                <indexed_header_path>/Users/V/IdeaProjects/pcbjam-private/pcbjam/wxwidgets/include</indexed_header_path>
+                <indexed_header_path>/Users/V/IdeaProjects/pcbjam-private/pcbjam/build-wasm/kicad-kicad_editor</indexed_header_path>
+                <indexed_header_path>/Users/V/IdeaProjects/pcbjam-private/pcbjam/build-wasm/wxwidgets/lib/wx/include</indexed_header_path>
+            </indexed_header_paths>
+            <name>KiCad WASM</name>
+            <status>enabled</status>
+            <type>C/C++ from Compilation Database</type>
+        </source_group_9a1b2c3d-0000-4000-8000-kicadwasm0001>
+    </source_groups>
+    <version>8</version>
+</config>

sourcetrail/sample.srctrlprj

@@ -0,0 +1,21 @@
+<?xml version="1.0" encoding="utf-8" ?>
+<config>
+    <source_groups>
+        <source_group_9a1b2c3d-0000-4000-8000-kicadwasm0001>
+            <build_file_path>
+                <compilation_db_path>./sample_compile_commands.json</compilation_db_path>
+            </build_file_path>
+            <indexed_header_paths>
+                <indexed_header_path>/Users/V/IdeaProjects/pcbjam-private/pcbjam/kicad</indexed_header_path>
+                <indexed_header_path>/Users/V/IdeaProjects/pcbjam-private/pcbjam/wasm</indexed_header_path>
+                <indexed_header_path>/Users/V/IdeaProjects/pcbjam-private/pcbjam/wxwidgets/include</indexed_header_path>
+                <indexed_header_path>/Users/V/IdeaProjects/pcbjam-private/pcbjam/build-wasm/kicad-kicad_editor</indexed_header_path>
+                <indexed_header_path>/Users/V/IdeaProjects/pcbjam-private/pcbjam/build-wasm/wxwidgets/lib/wx/include</indexed_header_path>
+            </indexed_header_paths>
+            <name>KiCad WASM sample</name>
+            <status>enabled</status>
+            <type>C/C++ from Compilation Database</type>
+        </source_group_9a1b2c3d-0000-4000-8000-kicadwasm0001>
+    </source_groups>
+    <version>8</version>
+</config>

sourcetrail/transform_compile_db.py

@@ -0,0 +1,151 @@
+#!/usr/bin/env python3
+"""Transform the in-container emscripten compile_commands.json into one
+Sourcetrail 2021.4.19 (bundled clang ~11) can index on the host.
+
+- expands @CMakeFiles/....rsp response files inline
+- strips emscripten-only and PCH flags (old clang can't load clang-20 .pch)
+- replaces the PCH with `-include cmake_pch.hxx` so those headers still parse
+- injects libc++ 11 headers (-nostdinc++) + emscripten sysroot includes
+- rewrites /workspace -> host pcbjam, /emsdk -> host tools/emsdk mirror
+
+Usage: transform_compile_db.py [--no-wasm-target] <in.json> <out.json>
+"""
+import json
+import os
+import re
+import shlex
+import sys
+
+PCBJAM = "/Users/V/IdeaProjects/pcbjam-private/pcbjam"
+ST_DIR = f"{PCBJAM}/sourcetrail"
+EMSDK_HOST = f"{PCBJAM}/tools/emsdk"
+
+PATH_MAP = [("/workspace/", PCBJAM + "/"), ("/emsdk/", EMSDK_HOST + "/")]
+
+DROP_EXACT = {
+    "-fwasm-exceptions",
+    "-Winvalid-pch",
+    "--emit-symbol-map",
+}
+DROP_PREFIX_RE = re.compile(r"^-s[A-Z][A-Z_0-9]*(=.*)?$|^-gseparate-dwarf")
+
+
+def host_path(p: str) -> str:
+    for src, dst in PATH_MAP:
+        if p.startswith(src):
+            return dst + p[len(src):]
+    return p
+
+
+def rewrite_tok(tok: str) -> str:
+    for src, dst in PATH_MAP:
+        tok = tok.replace(src, dst)
+    return tok
+
+
+def expand_rsp(tokens, directory):
+    out = []
+    for tok in tokens:
+        if tok.startswith("@"):
+            rsp = tok[1:]
+            if not os.path.isabs(rsp):
+                rsp = os.path.join(directory, rsp)
+            rsp_host = host_path(rsp)
+            with open(rsp_host) as f:
+                out.extend(shlex.split(f.read()))
+        else:
+            out.append(tok)
+    return out
+
+
+def transform(tokens, directory, wasm_target=True):
+    tokens = expand_rsp(tokens, directory)
+
+    argv0 = tokens[0]
+    lang_cxx = argv0.endswith("++")
+    out = []
+
+    i = 1
+    pch_headers = []
+    while i < len(tokens):
+        tok = tokens[i]
+        if tok == "-Xclang" and i + 1 < len(tokens):
+            nxt = tokens[i + 1]
+            if nxt == "-fno-pch-timestamp":
+                i += 2
+                continue
+            if nxt == "-include-pch":
+                # -Xclang -include-pch -Xclang <path.pch>
+                if i + 3 < len(tokens) and tokens[i + 2] == "-Xclang":
+                    pch = tokens[i + 3]
+                    hdr = pch[:-4] if pch.endswith(".pch") else pch
+                    if os.path.exists(host_path(hdr)):
+                        pch_headers.append(hdr)
+                    i += 4
+                    continue
+                i += 2
+                continue
+            out.extend([tok, nxt])
+            i += 2
+            continue
+        if tok in DROP_EXACT or DROP_PREFIX_RE.match(tok):
+            i += 1
+            continue
+        if not wasm_target and tok in ("-matomics", "-mbulk-memory", "-pthread"):
+            i += 1
+            continue
+        out.append(tok)
+        i += 1
+
+    for hdr in pch_headers:
+        out.extend(["-include", hdr])
+
+    # Rewrite container paths in the ORIGINAL tokens only, then prepend the
+    # host-path injections — rewriting after injection would re-fire on the
+    # /emsdk/ substring inside the host tools/emsdk mirror path.
+    out = [rewrite_tok(t) for t in out]
+
+    inject = ["-isystem", f"{EMSDK_HOST}/upstream/emscripten/cache/sysroot/include",
+              "-fexceptions"]
+    if lang_cxx:
+        inject = ["-nostdinc++", "-isystem", f"{ST_DIR}/libcxx-11/include"] + inject
+    if wasm_target:
+        inject = ["--target=wasm32-unknown-emscripten"] + inject
+
+    return ["clang++" if lang_cxx else "clang"] + inject + out
+
+
+def main():
+    args = sys.argv[1:]
+    wasm_target = True
+    if args and args[0] == "--no-wasm-target":
+        wasm_target = False
+        args = args[1:]
+    src, dst = args
+
+    with open(src) as f:
+        db = json.load(f)
+
+    out_db = []
+    missing = 0
+    for e in db:
+        tokens = shlex.split(e["command"])
+        directory = e["directory"]
+        new_tokens = transform(tokens, directory, wasm_target)
+        file_host = host_path(e["file"])
+        if not os.path.exists(file_host):
+            missing += 1
+            continue
+        out_db.append({
+            "directory": host_path(directory),
+            "command": " ".join(shlex.quote(t) for t in new_tokens),
+            "file": file_host,
+        })
+
+    with open(dst, "w") as f:
+        json.dump(out_db, f, indent=1)
+    print(f"wrote {len(out_db)} entries to {dst} ({missing} skipped: file missing on host)")
+
+
+if __name__ == "__main__":
+    main()