From f94114efb8e8b8c5ada8a9c4c61d4036bb0dd27f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Gerg=C5=91=20T=C3=B6rcsv=C3=A1ri?= Date: Wed, 3 Jun 2026 13:21:24 +0200 Subject: [PATCH] docs(yjs-bridge): add collaborative Yjs bridge design Approved design for porting KiCad-wasm editors to collaborative editing via Yjs: - 0001 general (tool-agnostic) bridge contract: structured CRDT, unified ChangeSource + apply, the 3 serialization mechanisms, 4-commit phasing - 0002 pl_editor (first PoC): identity, snapshot-differ adapter, field mapping - 0003 eeschema, 0004 pcbnew: native-listener/COMMIT adapters Co-Authored-By: Claude Opus 4.8 (1M context) --- .../yjs-bridge/0001-general-bridge-design.md | 228 ++++++++++++++++++ features/yjs-bridge/0002-pl-editor-bridge.md | 121 ++++++++++ features/yjs-bridge/0003-eeschema-bridge.md | 88 +++++++ features/yjs-bridge/0004-pcbnew-bridge.md | 78 ++++++ 4 files changed, 515 insertions(+) create mode 100644 features/yjs-bridge/0001-general-bridge-design.md create mode 100644 features/yjs-bridge/0002-pl-editor-bridge.md create mode 100644 features/yjs-bridge/0003-eeschema-bridge.md create mode 100644 features/yjs-bridge/0004-pcbnew-bridge.md diff --git a/features/yjs-bridge/0001-general-bridge-design.md b/features/yjs-bridge/0001-general-bridge-design.md new file mode 100644 index 0000000..dc3357f --- /dev/null +++ b/features/yjs-bridge/0001-general-bridge-design.md @@ -0,0 +1,228 @@ +# Yjs collaborative bridge — general design (tool-agnostic) + +Branch: `feature/yjs-bridge` (root + kicad + wxwidgets). +Goal: a bidirectional bridge between a KiCad editor's in-memory document model and a Yjs +`Y.Doc`, so local UI edits become CRDT ops and remote CRDT ops become model mutations. + +This file is the **shared architecture**. Per-tool specifics live in: +- `0002-pl-editor-bridge.md` — drawing-sheet editor (first PoC). +- `0003-eeschema-bridge.md` — schematic editor. +- `0004-pcbnew-bridge.md` — PCB editor. + +Each tool doc states **only its differences** from this file. + +Decisions taken (2026-06-03): +- **CRDT representation: structured** — `Y.Array` of one `Y.Map` per item (stable id + fields). +- **First PoC transport: in-browser, no server** — two tabs share a `Y.Doc` via + `BroadcastChannel`/`y-webrtc`. Server persistence is a later milestone. +- **First PoC target: pl_editor.** +- **One unified bridge contract for all three tools — no profile split** (see §1). + +--- + +## 1. The unified bridge contract (one shape, per-tool adapters) + +There is **one** bridge contract, identical for every tool. The C++ side exposes exactly two +things, and the JS side is fully generic: + +``` + ┌───────────────────────── identical across all tools ─────────────────────────┐ + C++ editor → ChangeSource → per-item delta JSON → reconciler.ts → Y.Doc(items: Y.Array) + C++ editor ← apply(delta) ← per-item delta JSON ← reconciler.ts ← Y.Doc ← provider ← peer + └─ per-tool adapter ─┘ └──────────── generic, schema-agnostic ──┘ +``` + +- **`ChangeSource` (emit):** produces a stream of **per-item change events** — `added` / + `removed` / `changed`, each carrying `{id, type, …fields}`. Emitted to JS as delta JSON. +- **`apply(delta)`:** consumes the same per-item delta JSON and applies it to the model. + +Both directions speak the **same per-item vocabulary** for every tool. What differs is only the +**per-tool adapter** that *implements* `ChangeSource`/`apply` — an internal detail invisible to +the wire format, the JS reconciler, and the other tools: + +| Tool | `ChangeSource` implementation | `apply` implementation | Identity | +|------|-------------------------------|------------------------|----------| +| eeschema | native `SCHEMATIC_LISTENER` | `SCH_COMMIT` Add/Modify/Remove + Push | native `KIID` | +| pcbnew | native `BOARD_LISTENER` (`OnBoardCompositeUpdate`) | `BOARD_COMMIT` Add/Modify/Remove + Push | native `KIID` | +| pl_editor | **in-bridge snapshot-differ** fired from `OnModify` | per-item mutate `DS_DATA_MODEL` by uuid + `SyncDrawItems` | **added** `KIID` (`0002`) | + +eeschema/pcbnew get per-item events *natively* from their commit/listener systems. pl_editor +has none, so its adapter **derives** the same per-item events by uuid-diffing successive +whole-model snapshots inside the bridge (trivial at its scale) — and applies per-item by uuid. +The derivation is encapsulated in pl_editor's adapter; nothing downstream can tell the +difference. (We deliberately do **not** rewrite pl_editor's edit/undo core to a real `COMMIT` — +that would diverge heavily from upstream for no runtime benefit; decided 2026-06-03.) + +So the only real per-tool work is: (1) pl_editor must **add** `KIID`+`(uuid)` to its format +(§2, `0002`); (2) eeschema/pcbnew must run the correct **recompute** on apply (connectivity / +ERC / ratsnest / hierarchy) by going through their native `COMMIT`. + +--- + +## 2. Item identity (shared semantics) + +A structured CRDT needs a per-item key that is (a) **identical on every client** and +(b) **stable across undo/redo**. The only way to guarantee both is for the id to live in the +**serialized form** of the item (so re-parsing — which undo does — preserves it, and two +clients opening the same file derive the same ids): + +- **eeschema/pcbnew:** satisfied natively. Items derive from `EDA_ITEM` + (`include/eda_item.h:516` → `const KIID m_Uuid`) and every item writes `(uuid …)`. Nothing to + do; the uuid flows through KiCad ↔ JSON ↔ Y.Doc as the single source of truth, with **no + cross-layer identity assumptions**. +- **pl_editor:** *not* satisfied natively — items have no `KIID` and `.kicad_wks` has no uuid + token. We add **the same `KIID` + `(uuid …)`** the others use (not a bespoke id), so the three + tools share one identity scheme (`0002`). Once added, identity behaves identically. + +New items minted locally get a fresh `KIID`; it appears in the next emit, the reconciler keys +a new `Y.Map` by it, and remote peers create the item with that same uuid on apply. + +**Backfill caveat for files lacking uuids.** When a tool *adds* uuids to a format (pl_editor), +legacy/upstream files have none, so they're backfilled on load. `KIID` backfill is **random, +not a deterministic content-hash** — so two clients cold-opening the *same never-saved* file +would disagree. Resolve via the **seed-once rule**: the client that finds the `Y.Doc` empty +seeds it and its backfilled uuids win; joiners adopt the `Y.Doc` instead of their own parse. +The first save then persists uuids, making all later opens consistent. (Deterministic +content-seed backfill is an optional later hardening; see `0002`.) + +--- + +## 3. Wire schema + representation granularity + +**The contract is minimal and schema-agnostic:** an array of objects, each with a string `id` +and `type`, plus arbitrary additional fields. The JS reconciler hardcodes **no** field names. + +Each item's fields can be represented at one of two granularities — **chosen per item-type on +the C++ side; the JS reconciler is identical either way** (it just diffs values keyed by field +name): + +- **Decomposed scalars** — `{id, type, x, y, text, …}`. Enables *field-level* merge (two + people editing different fields of the same item never conflict). Best for small/hot types. +- **Opaque blob** — `{id, type, sexpr: "(segment …)"}` reusing the tool's native per-item + serializer. *Item-level* merge granularity (concurrent edits to the same item conflict). + Near-zero C++ mapping. Best for the long tail of complex types. + +### Where the field values come from (don't hand-write N serializers) +There are three C++ mechanisms to produce the JSON for an item, in decreasing genericity. Pick +per tool; a tool may mix them and the JS reconciler is unaffected: + +1. **`PROPERTY_MANAGER` reflection — write-once, all `EDA_ITEM` types (the "do it once" lever).** + `EDA_ITEM`-based items register their fields generically (`REGISTER_TYPE` + `AddProperty`, + e.g. `pcbnew/board_item.cpp:427+` — *"Position X"*, *"Layer"*, *"Locked"*, …). Enumerate + `propMgr.GetProperties(item->GetTypeId())`, read each via `PROPERTY_BASE::GetValue → wxAny` + → JSON; write back via `PROPERTY_BASE::Set`. **One converter covers every registered type** + with field-level merge. Cost paid once: a `wxAny↔JSON` type bridge (ints/strings/bools, + enums, `EDA_ANGLE`/`VECTOR2I` units). Coverage isn't 100% — a few unregistered fields still + need (3). Primary path for **pcbnew** (broad coverage) and **eeschema** (partial). +2. **Native per-item S-expr blob — write-once *per tool that has one*.** pcbnew + `PCB_IO_KICAD_SEXPR::Format(BOARD_ITEM*)`; pl_editor `DS_DATA_MODEL_IO::Format(model, item)` + / `SaveInString({item}, &str)`. Serializes *any* of that tool's items with no per-type code; + item-level merge only. (eeschema has no clean per-item formatter — document-level only.) +3. **Hand-mapped per-type fields.** Explicit, total control, works everywhere (incl. pl_editor, + which has neither reflection nor `EDA_ITEM`). Most code — reserve for tools with few types + (pl_editor's 5) or the hot fields you want field-level merge on. + +(There is also `SERIALIZABLE`/`Serialize→protobuf::Any` on `EDA_ITEM`, but it targets the IPC +API — protobuf, partial coverage — wrong fit for JSON; not used here.) + +A tool may mix granularities/mechanisms and refine over time **without any JS change**. Nested +data (polygon corner lists, bitmap blobs) is an opaque blob field or a nested `Y.Array`/`Y.Map`. +Numeric coordinates use the model's native units; the reconciler **quantizes before comparing** +to avoid spurious float diffs. + +Optional **Zod schema as a validation guard only** at the apply boundary (reject garbage, +coerce number/bool/string) — never as a structural requirement. + +--- + +## 4. The JS reconciler (`web/.../collab/`, identical for all tools) + +A single generic module, parameterized only by the `Y.Doc` and the two bridge entry points: + +- **Down (model → Y):** keep the last-seen item-set keyed by `id`. On each emitted per-item + delta, write **only** the changes into the `Y.Array`, inside one `Y.transaction` + tagged with a local origin. (If an adapter ever emits a whole snapshot instead of a delta — + it shouldn't, but pl_editor's differ could — the reconciler still diffs it to minimal ops, so + the wire stays granular regardless.) +- **Up (Y → model):** observe the `Y.Array`; on remote-origin events, build the changed-item + delta JSON and call the C++ `apply` hook. Ignore events whose origin is our own tag + (standard Yjs echo-suppression). +- **Generic mapping:** an item ⇆ `Y.Map` is a flat copy of scalar fields; the diff is + value-equality per key. Adding a C++ field needs zero JS change. + +The reconciler is **adapter-blind** — it only ever sees per-item deltas keyed by uuid. + +--- + +## 5. Loop / re-entrancy safety (shared) + +- **`s_applyingRemote` guard** in the C++ bridge: set around `apply`; emit is suppressed while + set. Prevents apply → (OnModify | listener fire) → emit → Y write → observe → apply … loops. + Needed by every adapter (a native listener would otherwise re-fire on our own apply; the + pl_editor differ would otherwise re-diff our own apply). +- **Asyncify timing:** apply is invoked from a JS callback and runs synchronously on the main + thread between frames (same context as the existing `OnFileDropped`/`OpenFileCallback` + hooks). It must not run mid-Asyncify-unwind — gate applies on the tool's ready/idle state and + queue any that arrive during a suspension. +- **Yjs origin tags** on all reconciler transactions; observers skip their own origin. + +--- + +## 6. Transport & persistence (shared, phased) + +- **PoC:** two tabs, one `Y.Doc`, `BroadcastChannel` (or `y-webrtc`). Zero backend work. +- **Later (M-server):** `y-websocket` provider (standalone or folded into the Fastify server in + `web/apps/server`) for networked sync + persistence; periodically materialize the Y.Doc back + to the canonical `.kicad_*` file via the tool's normal save path. + +--- + +## 7. Shared C++ bridge interface + +A thin, `#ifdef __EMSCRIPTEN__`-guarded interface each tool implements, so JS is uniform: + +- **Emit → JS:** `EM_ASM({ window.kicadCollab.onDelta(UTF8ToString($0)) }, json)` — `json` is a + per-item delta. Same JS entry point for every tool/adapter. +- **Apply ← JS:** `EMSCRIPTEN_KEEPALIVE void kicadCollabApply(const char* json)` — dispatches to + the active tool's adapter; wrapped in the `s_applyingRemote` guard. +- Copy the proven wx-layer pattern (`EM_ASM` out, `ccall`→`EMSCRIPTEN_KEEPALIVE` in). +- The collab code that touches tool models lives in the **kicad** fork; keep it tightly guarded + and run `kicad-diff-stats.sh`. eeschema/pcbnew adapters add almost no surface (listener + subclass + COMMIT calls). pl_editor adds the most: the `(uuid)` format token + the snapshot- + differ adapter — but still leaves its edit/undo core untouched. + +--- + +## 8. Phasing — one commit per step + +1. **pl_editor interface/identity change.** `KIID m_Uuid` on `DS_DATA_ITEM` + `(uuid …)` + keyword/parse(backfill)/save via the shared `FormatUuid`. Independently testable (uuid + round-trip + legacy-file backfill-on-save). Commit. (`0002` §identity) +2. **pl_editor bridge + JSON converters (all 5 item types) + tests.** The snapshot-differ + `ChangeSource`, per-item `apply` by uuid, the generic JS reconciler, `BroadcastChannel` + transport, two-tab demo. Text/line/rect as scalars, poly/bitmap as blobs (`0002`). Commit. +3. **eeschema JSON converters + bridge + tests.** Adapter = `SCHEMATIC_LISTENER` + `SCH_COMMIT`; + converters reflection-first (§3 mechanism 1) + hand-map gaps; multi-sheet hierarchy + symbol + instances. Commit. (`0003`) +4. **pcbnew JSON converters + bridge + tests.** Adapter = `BOARD_LISTENER` + `BOARD_COMMIT`; + converters = reflection (1) for the bulk + native blob (2) for the long tail; mandatory + connectivity/ratsnest recompute; large-board batched initial-sync. Commit. (`0004`) +5. **Later — y-websocket + persistence** across all tools. + +pl_editor goes first (steps 1–2) because its adapter is the only one we write from scratch and +it carries the least that can break (no connectivity, tiny docs); once the contract holds +there, the eeschema/pcbnew adapters are a thin re-use over native machinery. + +--- + +## 9. Cross-cutting risks +- **Asyncify timing** of apply (§5) — needs a real under-load test. +- **Representation-granularity drift** — mixing blob and scalar per type must stay consistent + across clients (the `type`→granularity decision lives in C++ and must be deterministic). +- **Coordinate precision** — quantize before diffing (§3). +- **Recompute side-effects (eeschema/pcbnew)** — applying outside the normal UI flow must still + trigger connectivity/ERC/ratsnest/hierarchy; always go through the native COMMIT. +- **Initial sync cost (pcbnew)** — seeding a 100k-item board into a fresh `Y.Doc` is itself + heavy; needs a batched/own-phase strategy (see `0004`). +- **Fork divergence** — pl_editor changes the `.kicad_wks` file format (fork-only files; see + `0002`). eeschema/pcbnew adapters are near-zero divergence. diff --git a/features/yjs-bridge/0002-pl-editor-bridge.md b/features/yjs-bridge/0002-pl-editor-bridge.md new file mode 100644 index 0000000..2ead121 --- /dev/null +++ b/features/yjs-bridge/0002-pl-editor-bridge.md @@ -0,0 +1,121 @@ +# pl_editor (drawing-sheet editor) — bridge specifics + +Differences from `0001-general-bridge-design.md`. pl_editor implements the **same unified +bridge contract** as the other two (per-item `ChangeSource` + per-item `apply`); it is the +**first PoC**. It has none of the native machinery, so its `ChangeSource` is the one adapter we +write from scratch — an **in-bridge snapshot-differ** — while its edit/undo core stays +untouched (low upstream divergence; Route 1, decided 2026-06-03). + +## What pl_editor lacks natively (why its adapter is custom) +- `DS_DATA_ITEM` does **not** derive from `EDA_ITEM` (`include/drawing_sheet/ds_data_item.h:95`) + → no `KIID`, and the `COMMIT` base operates on `EDA_ITEM*` so it can't take data items. +- **No COMMIT** anywhere in `pagelayout_editor/` (0 uses); undo is a whole-model S-expr snapshot + (`SaveCopyInUndoList` → `DS_PROXY_UNDO_ITEM`) wired at 13 sites across 8 files. +- **No listener/observer** of any kind. +- Single change chokepoint: `PL_EDITOR_FRAME::OnModify()` (`pl_editor_frame.cpp:379`). + +We do **not** rewrite that core. Instead the adapter derives per-item events by diffing, and +applies per-item by uuid — producing the exact same observable behavior as `SCH_COMMIT`/ +`BOARD_COMMIT`, encapsulated entirely in the bridge. + +## Model +- Singleton `DS_DATA_MODEL::GetTheInstance()`; `std::vector m_list` + (`include/drawing_sheet/ds_data_model.h`). Flat, single-document — no sheets/screens, no + connectivity/ERC (a big simplification vs eeschema/pcbnew). +- Item types: `DS_TEXT`, `DS_SEGMENT`, `DS_RECT`, `DS_POLYPOLYGON`, `DS_BITMAP`. +- `DS_DRAW_ITEM_*` render proxies are rebuilt by `DS_DATA_ITEM::SyncDrawItems(view)`; ephemeral. + +## Identity — add the SAME `KIID`/`(uuid …)` as eeschema/pcbnew +Use the identical type + token + writer (general §2), not a bespoke id: +- Add `KIID m_Uuid;` to `DS_DATA_ITEM` (`include/kiid.h`). Default-constructed → **random**, so + a new or uuid-less item is auto-backfilled in memory for free. +- Serialize with the shared helper `KICAD_FORMAT::FormatUuid( m_out, item->m_Uuid )` + (`include/io/kicad/kicad_io_utils.h:55`) in `ds_data_model_io.cpp`, exactly as sch/pcb do. + +### Save + load + backfill +- **Format:** add `uuid` to `common/drawing_sheet/drawing_sheet.keywords` (absent today). The + wks lexer is **strict** (unknown token → `default:` → `Expecting()`/throws), so the token must + be declared + handled — no silent passthrough. +- **Load:** `case T_uuid: item->m_Uuid = KIID( FromUTF8() ); NeedRIGHT(); break;` in each item + parser (`drawing_sheet_parser.cpp`). If the token is absent, the parser leaves the default + random `m_Uuid` in place → **automatic backfill**. +- **Save:** always write `(uuid …)` per item → the first save of a legacy/upstream uuid-less + `.kicad_wks` **upgrades the file** in place. +- **Cross-client consistency:** files with uuids (our output, and any file after one save) → + all clients agree. A never-saved uuid-less legacy file gets *random* backfill + (`KIID(invalid)` → random, confirmed in `common/kiid.cpp` — not a deterministic hash), so two + cold opens disagree; handled by the **seed-once rule** (general §2). Optional deterministic + content-seed backfill (boost `name_generator` over type+name+pos+index) is deferred hardening. +- **Tradeoff:** files now carry `(uuid …)`; stock desktop KiCad's strict parser rejects the + unknown token → fork-only files. Backfill makes the upgrade graceful. + +## `ChangeSource` adapter — in-bridge snapshot-differ (emits per-item deltas) +The bridge holds the **last emitted snapshot** (a `uuid → fields` map). On `OnModify()`: +1. Serialize current `m_list` to a `uuid → fields` map — **all 5 types, none invisible** + (rationale below): text/segment/rect as scalars; poly/bitmap as an opaque S-expr blob. +2. **Diff vs the cached map by uuid:** present-only-in-new → `added`; only-in-old → `removed`; + in-both-with-changed-fields → `changed`. +3. Emit the per-item delta JSON via `EM_ASM window.kicadCollab.onDelta(json)`; update the cache. + +This yields the same `added/removed/changed` per-item events the native listeners produce. The +diff is O(items) — negligible for drawing sheets. *(We could instead call the differ directly +from `OnModify`; optionally wrap it as a one-method `DS_DATA_MODEL` listener purely for +registration symmetry with sch/pcb — nice-to-have, not required.)* + +## `apply(delta)` — per-item mutation by uuid +For each op in the incoming delta (inside the `s_applyingRemote` guard, general §5): +- **changed:** find `DS_DATA_ITEM` by `m_Uuid` in `m_list`; set the changed fields. +- **added:** create the item (with the given uuid), `DS_DATA_MODEL::Append`. +- **removed:** find by uuid, `Remove` + delete. +Then `SyncDrawItems` on touched items + canvas refresh, and **re-select by uuid** so a remote +apply doesn't drop local selection (a bonus the uuid unlocks). Optionally +`SaveCopyInUndoList()` once after a remote batch so local undo has a coherent checkpoint. **No +connectivity/ERC** — none exists for drawing sheets. + +## Text-insert path (working UI flow targeted first) +`PL_DRAWING_TOOLS::PlaceItem` (`tools/pl_drawing_tools.cpp:72`) → `AddDrawingSheetItem(DS_TEXT)` +(`pl_editor_frame.cpp:823`) → `new DS_DATA_ITEM_TEXT("Text")` (fresh `KIID`) → `Append` → +`SyncDrawItems` → `SaveCopyInUndoList()` → `OnModify()` ← differ runs here → emits an `added`. + +## Field mapping — all 5 types in one pass, at two fidelities +Cover every type so none is invisible to collaboration (see rationale). pl_editor has no +reflection (`DS_*` is not `EDA_ITEM`), so scalars are hand-mapped (general §3 mechanism 3) and +the awkward two reuse the existing per-item serializer (mechanism 2): + +Shared: `id`(m_Uuid) · `type` · `name`(m_Name) · `x,y`(m_Pos.m_Pos, mm) · `anchor`(m_Pos.m_Anchor) +- **text** (scalars): `text`(m_TextBase) · `orient`(°) · `hjustify`/`vjustify` · `italic` · + `bold` · `sizeX,sizeY`(m_TextSize, mm) — field-level merge. +- **segment/rect** (scalars): `ex,ey`(m_End.m_Pos) · `eanchor` · `linewidth`(m_LineWidth). +- **poly/bitmap** (opaque blob): `sexpr` field from `DS_DATA_MODEL_IO::Format(model, item)` / + `SaveInString({item}, &str)` — *no* nested corner-list / base64 field mapper to write; + item-level merge. Refine to scalars later only if field-level merge is ever wanted. + +**Why all 5, not just text/line/rect:** the adapter is a whole-model differ, so a type absent +from the snapshot is **invisible** — a user adding a polygon or logo bitmap simply wouldn't +sync. Including poly/bitmap as blobs closes that correctness gap at near-zero cost (the +serializer already exists). **Bitmap is safe** despite its large base64 payload: the differ +emits a field only when its value changes (string compare), so a static image is cached once +and never re-emitted — just keep it out of any per-keystroke re-encode path. + +On **apply**, a blob item is reconstructed by parsing its `sexpr` back into a `DS_DATA_ITEM` +(wrap as a minimal model string → reuse `SetPageLayout`/the parser; a tiny parse-one helper). + +## Build steps (maps to phasing commits 1–2 in general §8) +**Commit 1 — interface/identity:** `KIID m_Uuid` on `DS_DATA_ITEM` + `(uuid)` +keyword/parse(backfill)/save via `FormatUuid`. Build once; verify a `.kicad_wks` round-trips +uuids and a legacy file is backfilled on save. +**Commit 2 — bridge + converters (all 5 types) + tests:** +1. `common/drawing_sheet/ds_collab_bridge.{h,cpp}` (`#ifdef __EMSCRIPTEN__`): the snapshot-differ + `ChangeSource` (cache + diff, emit on `OnModify`), `kicadCollabApply` per-item-by-uuid + (incl. blob parse-one for poly/bitmap), `s_applyingRemote` guard. +2. Web: `yjs` + `BroadcastChannel` provider; the generic reconciler (general §4) + `WasmTool` + wiring; optional Zod guard. Two-tab demo. +3. Verify: insert/move text & line & rect in tab A → mirrors in tab B and back; poly/bitmap + round-trip via blob; selection survives remote apply (uuid re-select); e2e + screenshots. + +## Risks specific to pl_editor +- Carries the **format-change divergence** (general §9) — the only tool that does. +- Its `ChangeSource` is a **derived** differ (not native tracking). Correct and cheap here, but + the differ pattern is pl_editor's alone — eeschema/pcbnew use native events. +- Never-saved legacy-file cold-open identity race — mitigated by seed-once; deterministic + backfill deferred. diff --git a/features/yjs-bridge/0003-eeschema-bridge.md b/features/yjs-bridge/0003-eeschema-bridge.md new file mode 100644 index 0000000..6809825 --- /dev/null +++ b/features/yjs-bridge/0003-eeschema-bridge.md @@ -0,0 +1,88 @@ +# eeschema (schematic editor) — bridge specifics + +Differences from `0001-general-bridge-design.md`. eeschema implements the **same unified bridge +contract** (general §1); its `ChangeSource`/`apply` adapter is a thin re-use of **native** +machinery, so most of pl_editor's hard work (`0002`) disappears here. Two genuinely new things: +**multi-sheet hierarchy** and **symbol instances**. + +## Adapter — native (no derived differ, no format change) — and *why* +eeschema has the full machinery natively: +- **Identity free:** `SCH_ITEM` ⊂ `EDA_ITEM` → `const KIID m_Uuid` (`include/eda_item.h:516`), + serialized as `(uuid …)` on every item by `SCH_IO_KICAD_SEXPR` + (`eeschema/sch_io/kicad_sexpr/sch_io_kicad_sexpr.cpp:745,1047,1189,1322,…`; parsed in + `…_parser.cpp:2777,3476,…`). **No format change** (contrast `0002`). +- **Granular change set:** `SCH_COMMIT` (`eeschema/sch_commit.h`), `Push()` loop + (`eeschema/sch_commit.cpp:228–397`) tags each entry `CHT_ADD` / `CHT_REMOVE` / `CHT_MODIFY`. +- **Native observer — hook without patching core:** `SCHEMATIC_LISTENER` + (`eeschema/schematic.h:60–71`): + ```cpp + virtual void OnSchItemsAdded ( SCHEMATIC&, std::vector& ); + virtual void OnSchItemsRemoved ( SCHEMATIC&, std::vector& ); + virtual void OnSchItemsChanged ( SCHEMATIC&, std::vector& ); + virtual void OnSchSheetChanged ( SCHEMATIC& ); + ``` + Register via `schematic->AddListener(bridge)` (`schematic.h:359`). `SCH_COMMIT::Push` fires + these in bulk (`sch_commit.cpp:399–417`). → **emit is a listener subclass; near-zero fork + divergence.** + +So: no derived differ and no format patch here. The `ChangeSource` is the listener; `apply` +goes through `SCH_COMMIT`. + +## Model — multi-sheet (the main structural difference) +- `SCHEMATIC` (`eeschema/schematic.h:87`) → virtual `m_rootSheet`, `m_topLevelSheets`, + current `m_currentSheet`. Each `SCH_SHEET` owns one `SCH_SCREEN` whose items live in an + `EE_RTREE` (`eeschema/sch_screen.h:98–117`). +- **Yjs shape:** because `m_Uuid` is globally unique, a **flat uuid-keyed `Y.Array`** still + works, but each item's `Y.Map` must carry a **`screen`/`sheetPath` field** so apply knows + which `SCH_SCREEN` to target. (Alternative: a `Y.Map` of screens, each a `Y.Array` of items + — more structure, mirrors the model; decide during the eeschema commit. Flat+scope-field is + simpler to start.) +- `OnSchSheetChanged` / hierarchy refresh must propagate as its own doc field/event. + +## Symbol instances (the other new wrinkle) +A `SCH_SYMBOL` placed in a shared sheet appears on multiple sheet *instances*; per-instance +data (reference, unit) lives in `m_instanceReferences` (`SCH_SYMBOL_INSTANCE`). Editing the +shared symbol affects all instances; a remote change to reference/unit must update the right +**instance record**, not just the symbol. Treat instance data as nested fields keyed by +sheet-path within the symbol's item. + +## ChangeSource (emit) +`SCHEMATIC_LISTENER` subclass → on `OnSchItemsAdded/Removed/Changed`, emit per-item JSON. +**Converter strategy: reflection-first** (general §3 mechanism 1) — `SCH_ITEM : EDA_ITEM`, so +use `PROPERTY_MANAGER` to serialize registered fields write-once; **hand-map (mechanism 3) the +gaps** where properties aren't registered. (Blobs are *not* an option here — see serialization +note.) Include the `screen`/`sheetPath` scope on every item. + +## apply — go through SCH_COMMIT +```cpp +SCH_COMMIT commit( toolMgr ); +// add: create SCH_ITEM, commit.Add(item, screen) +// modify: item = schematic->ResolveItem(uuid); commit.Modify(item, screen); …edit fields… +// remove: commit.Remove(item, screen) +commit.Push( "collab" ); // triggers the recompute below +``` +`Push` (`sch_commit.cpp`) runs the required side-effects: +- **Connectivity/ERC/netlist:** `frame->RecalculateConnections(...)` when any connectable item + changed (`HasConnectivityChanges`); local vs global cleanup chosen automatically. +- **Hierarchy:** `schematic->RefreshHierarchy()` + nav update when a `SCH_SHEET` changed. + +## Apply gotchas (enumerated) +- Connectivity is **bidirectional** — adding/removing a wire re-nets connected pins; never skip + `RecalculateConnections`. +- **Symbol instances** — update the per-sheet instance record, not only the shared symbol. +- **Groups** — keep parent/child links in sync (`group->AddItem/RemoveItem`). +- **`SCH_FIELD` remove is special** — it hides, not deletes (`sch_commit.cpp:308`). +- **ERC markers** are uuid-tied; deletes orphan them (respect exclusions). +- **Selection** is transient UI state — never propagate. + +## Serialization note (differs from pcbnew) +eeschema's I/O plugin is **document-level**, not per-item — there is **no** clean public +`Format(single SCH_ITEM)`, so the opaque-S-expr-blob path (general §3 mechanism 2) is *not* +available here. This is exactly why the converter is **reflection-first + hand-mapped gaps**: +fields via `PROPERTY_MANAGER` (or accessors), and on apply build complex items via `Clone()` + +field setters rather than re-parsing a blob. + +## Scale +~500–5000 items typical (10k+ for large hierarchical designs). Granular deltas (listener) are +strongly preferred; whole-document serialization is expensive (hierarchy + uuid resolve + +connectivity). A pl_editor-style derived differ would be wrong at this scale — use the listener. diff --git a/features/yjs-bridge/0004-pcbnew-bridge.md b/features/yjs-bridge/0004-pcbnew-bridge.md new file mode 100644 index 0000000..4f5e915 --- /dev/null +++ b/features/yjs-bridge/0004-pcbnew-bridge.md @@ -0,0 +1,78 @@ +# pcbnew (PCB editor) — bridge specifics + +Differences from `0001-general-bridge-design.md`. pcbnew implements the **same unified bridge +contract** (general §1) with the **richest** native hooks of the three. Two things dominate its +specifics: **mandatory connectivity/ratsnest recompute** and **scale** (10k–100k+ items). + +## Adapter — native (richest of the three) — and *why* +- **Identity free:** `BOARD_ITEM` ⊂ `EDA_ITEM` → `const KIID m_Uuid` (`include/eda_item.h:516`), + serialized as `(uuid …)` on every item by `PCB_IO_KICAD_SEXPR` (e.g. track + `pcb_io_kicad_sexpr.cpp:2619`, footprint `:1170`, zone `:2647`). Fast lookup via + `std::unordered_map m_itemByIdCache` (`board.h:1458`). **No format change.** +- **Granular change set:** `BOARD_COMMIT::Push` (`pcbnew/board_commit.cpp:176–631`) tags each + entry `CHT_ADD/REMOVE/MODIFY` and collects bulk add/remove/changed vectors. +- **Native observer — hook without patching core:** `BOARD_LISTENER` (`board.h:283–301`) is the + richest interface — singular *and* bulk callbacks, plus a combined one: + ```cpp + OnBoardItemAdded/­ItemsAdded · OnBoardItemRemoved/­ItemsRemoved + OnBoardItemChanged/­ItemsChanged · OnBoardNetSettingsChanged + OnBoardHighlightNetChanged · OnBoardRatsnestChanged + OnBoardCompositeUpdate(board, added[], removed[], changed[]) // ← one event, all three sets + ``` + Registered via `board->AddListener(bridge)` (`board.cpp:2964`); fired through + `InvokeListeners(...)` (`board.h:1424`). `BOARD_COMMIT::Push` → `OnItemsCompositeUpdate` + (`board_commit.cpp:580` → `board.cpp:3001`). **`OnBoardCompositeUpdate` is the ideal single + emit hook.** + +## Model — typed containers (vs flat list / vs rtree) +8 typed containers, not one list (`board.h:1447–1455`): `m_footprints`, `m_tracks`, `m_zones`, +`m_drawings`, `m_groups`, `m_generators`, `m_markers`, `m_points`. The Yjs shape is still a +**flat uuid-keyed `Y.Array`** — `type` tells apply which container `BOARD::Add` routes to; +`m_itemByIdCache` resolves uuid→item in O(1). + +## ChangeSource (emit) +`BOARD_LISTENER::OnBoardCompositeUpdate` → emit per-item JSON for added/changed (uuid+type+ +fields) and uuid-only for removed. **Converter strategy: reflection for the bulk + native blob +for the long tail** — pcbnew uniquely has *both* generic mechanisms (general §3): +- (1) `BOARD_ITEM` is broadly `INSPECTABLE` (`board_item.cpp:427+`) → `PROPERTY_MANAGER` + serializes most fields **write-once** with field-level merge. +- (2) `PCB_IO_KICAD_SEXPR::Format(const BOARD_ITEM*)` (`pcb_io_kicad_sexpr.h:351`, per-type + `format(PCB_TRACK*/PAD*/ZONE*/…)`) gives a **free opaque blob** for any type reflection + misses. + +So: reflection-decompose the hot types you want field-level merge on (track width/endpoints, +position), blob everything else — **no hand-written per-type serializers**. + +## apply — go through BOARD_COMMIT; recompute is MANDATORY +```cpp +BOARD_COMMIT commit( toolMgr ); +// add: item = deserialize(...); commit.Add(item) +// modify: item = board->GetItem(uuid); commit.Modify(item); …edit… +// remove: commit.Remove(board->GetItem(uuid)) +commit.Push( "collab" ); // updates connectivity + recomputes ratsnest +``` +`Push` updates `CONNECTIVITY_DATA` per item and calls `RecalculateRatsnest`. **Never** use +direct `BOARD::Add/Remove` for remote ops unless you also manually +`GetConnectivity()->Add/Update/Remove` + `RecalculateRatsnest()` + `OnRatsnestChanged()` — +the commit path does it correctly, so use it. + +## Apply gotchas (enumerated) +- **Connectivity + ratsnest are mandatory** after any structural/net change — skipping + corrupts net feedback. This is the single biggest difference from pl_editor. +- **Zones** are expensive — outline edits need a re-fill (not automatic). +- **Footprints** own child items (pads, fp text); their uuids live in `m_itemByIdCache` too — + a footprint add/remove cascades (handled by `BOARD::Add`, `board.cpp:1239`). +- **Vias/teardrops/DRC** — drill changes ripple; `BOARD_COMMIT::Push` handles within commit. +- **Groups** — membership tracked by the commit. +- **Net settings** (`OnBoardNetSettingsChanged`) are board-global, not per-item — model as a + separate doc field, not an items entry. + +## Scale — the defining constraint +Real boards: 5k–20k items typical, 50k–500k for large designs. Implications: +- **Snapshot-on-change is non-viable** (per-commit serialization → MB; remote parse blocks UI). + Granular deltas via the listener are **essential** here — never a whole-document approach. +- **Initial sync is itself heavy:** seeding a fresh `Y.Doc` from a 100k-item board must be a + **batched, own-phase** operation (chunked transactions, possibly off the first paint), not a + single emit. Flag as its own sub-task within the pcbnew commit. +- Prefer **opaque-blob** items to keep per-item payloads small and avoid a huge field-mapping + surface across pcbnew's many types.