pcbjam/docs/features/web-init/0001-web-app-spec.md

# 0001 — KiCad-WASM Web App Spec

Status: **Refined spec, ready for `implement plan`**
Branch: `feature/web-init`
Scope of this iteration: **create a project, open a project, upload files, open a file in a WASM tool via URL.**

This document is the agreed design after a clarification pass. Decisions that were
explicitly chosen by the user are marked **[decided]**. Items intentionally pushed to a
later iteration are marked **[later]**. Sensible defaults that were *not* explicitly
discussed are marked **[default]** and are safe to change during planning.

---

## 1. Goal

A single web application that lets a user create/open KiCad projects, upload files into
them, and open a file in the matching WASM tool (pcbnew / eeschema / calculator) by
visiting a URL such as:

```
/p/project5/pcbnew/nyak.kicad_pcb
```

The WASM apps already exist as `<tool>.js` + `<tool>.wasm` pairs (see `output/` and
`tests/apps/kicad/`). They boot into an Emscripten harness and read files from MEMFS.
This web app wraps that: it manages projects + files server-side, and on a tool URL it
boots the right WASM app and feeds it the project's files via `FS.writeFile`, then drives
File→Open on the target file.

Non-goals this iteration: editing/saving back, collaboration, auth/login.

---

## 2. Key decisions (summary table)

| Area | Decision |
|---|---|
| Frontend | React + TypeScript + Vite + shadcn/ui **[decided]** |
| Backend | Fastify + ts-rest + Zod **[decided]** |
| Shared types | `packages/contract` (ts-rest contract + Zod) imported by FE client & BE router **[decided]** |
| Realtime | None now; pick stack that allows Hocuspocus/WSS later, no WS endpoints yet **[decided]** |
| URL semantics | `/p/:project/:tool/*filepath` — `tool` selects the WASM app, `*filepath` is auto-opened **[decided]** |
| Auth/tenancy | No auth now, but data model namespaced by an owner id for later multi-user **[decided]** |
| Metadata store | Postgres now, accessed via Drizzle (drizzle-zod shares schemas) **[decided]** |
| File blob storage | Pluggable `FileStorage` interface; local-disk impl now, S3 later **[decided]** |
| Open behavior | Sync **whole project tree** into MEMFS, then auto-open target **[decided]**; lazy/partial load **[later]** |
| Upload | Individual files (multi), folder (preserve structure), and `.zip` of a project **[decided]** |
| WASM artifact delivery | Served from a **configurable base location** (URL/dir): local Fastify static from `output/` in dev, public S3 URL in prod **[decided]** |
| Save-back / sync | Read-only open now; write interface defined but unused. Save-back + lazy load land together **[later]** |
| Monorepo | pnpm + turbo workspace under `web/` **[decided]** |

---

## 3. Repository / monorepo layout  **[decided: under `web/`]**

```
web/
├── package.json                  # pnpm workspace root
├── pnpm-workspace.yaml
├── turbo.json
├── .env.example
├── docker-compose.yml            # local Postgres (and later: minio for S3 parity)
├── apps/
│   ├── frontend/                 # Vite + React + TS + shadcn
│   └── server/                   # Fastify + ts-rest + Drizzle
└── packages/
    ├── contract/                 # ts-rest contract + Zod schemas (shared)
    ├── storage/                  # FileStorage interface + local-disk impl (+ S3 later)
    └── config/                   # shared tsconfig / eslint / env parsing  [default]
```

Rationale: isolates the JS/TS app from the C++/WASM build repo at root (`kicad/`,
`wxwidgets/`, `scripts/`, `docker/`). The web app consumes WASM artifacts produced by the
existing build, it does not build them.

Root `.gitignore` should ignore `web/**/node_modules`, `web/**/dist`, build caches.

---

## 4. Domain model

### 4.1 Entities (Postgres, via Drizzle)  **[decided: Postgres + Drizzle]**

```
owner            -- namespace for "no auth now, multi-user later"
  id             uuid pk
  slug           text unique         -- e.g. "default" now; becomes real users later
  created_at     timestamptz

project
  id             uuid pk
  owner_id       uuid fk -> owner.id
  slug           text                -- URL segment, unique within owner (e.g. "project5")
  name           text                -- human display name
  created_at     timestamptz
  updated_at     timestamptz
  unique(owner_id, slug)

project_file                          -- index of files; bytes live in FileStorage
  id             uuid pk
  project_id     uuid fk -> project.id
  path           text                -- POSIX-relative within project, e.g. "pcbnew/nyak.kicad_pcb"
  size           bigint
  content_type   text
  storage_key    text                -- opaque key handed to FileStorage
  created_at     timestamptz
  updated_at     timestamptz
  unique(project_id, path)
```

- **Owner namespace [decided]**: every project belongs to an `owner`. This iteration uses a
  single seeded owner (`slug = "default"`); the URL omits owner (`/p/:project/...`) and the
  server resolves it to the default owner. Adding real auth later = populate `owner` per
  user and prefix routes, **no schema migration needed**.
- `project_file.path` is the canonical project-relative path. The `storage_key` decouples
  the logical path from however the blob backend names things (so renames/S3 layout are free).

drizzle-zod derives Zod schemas from these tables; those Zod schemas feed the ts-rest
contract so DB ↔ API ↔ client share one source of truth.

### 4.2 What "project" means at the byte level

A project is a directory tree of files (`.kicad_pro`, `.kicad_pcb`, `.kicad_sch`,
`fp-lib-table`, `sym-lib-table`, footprint/symbol lib dirs, etc.). `project_file` rows
enumerate the tree; bytes live behind `FileStorage`.

---

## 5. Storage abstraction  **[decided: pluggable, local now, S3 later]**

`packages/storage` exposes a single interface. The whole iteration is **read-heavy**; write
methods exist so save-back **[later]** needs no redesign.

```ts
export interface FileStorage {
  // read path
  exists(key: string): Promise<boolean>;
  read(key: string): Promise<Uint8Array>;
  createReadStream(key: string): NodeJS.ReadableStream;   // for large files
  stat(key: string): Promise<{ size: number; contentType?: string }>;
  list(prefix: string): Promise<string[]>;                // keys under a prefix

  // write path (used now only by upload; save-back is [later])
  write(key: string, data: Uint8Array | NodeJS.ReadableStream, opts?: { contentType?: string }): Promise<void>;
  delete(key: string): Promise<void>;
}
```

Implementations:
- `LocalDiskStorage` **[now]** — rooted at a configurable dir (`STORAGE_ROOT`), `key` maps
  to a path under it. Streams to/from disk.
- `S3Storage` **[later]** — same interface over an S3-compatible bucket. `docker-compose`
  can run MinIO for local S3 parity when we get there.

Storage key scheme **[default]**: `owners/<owner_id>/projects/<project_id>/<project_file.path>`.
Opaque to callers — only `FileStorage` interprets it.

---

## 6. WASM artifact delivery  **[decided: configurable base location]**

The big artifacts (`pcbnew.wasm` ~350 MB, `eeschema.wasm` ~180 MB, `calculator.wasm`,
their `.js` glue, `wx.js`, `images.tar.gz`) are **app binaries, not user data** — kept
separate from `FileStorage`.

- The frontend resolves every artifact URL from a single configurable base:
  `WASM_ASSET_BASE_URL` **[decided requirement]**.
  - **dev**: points at the Fastify server, which serves the artifacts statically from a
    configurable dir (default `../../output` relative to the server, i.e. repo `output/`).
  - **prod**: points at a public S3/CDN URL. No code change — just env.
- An artifact URL is composed as `${WASM_ASSET_BASE_URL}/${tool}.js` (and the glue then
  fetches the sibling `.wasm` / `worker.js` / `images.tar.gz` from the same base). The
  Emscripten `locateFile` hook must be wired to this base so `.wasm`/`.worker.js` resolve
  correctly regardless of origin.
- **Cross-origin caveat [important]**: KiCad WASM uses threads (`.worker.js` present),
  which needs `SharedArrayBuffer` → the **document** must be served with
  `Cross-Origin-Opener-Policy: same-origin` and `Cross-Origin-Embedder-Policy: require-corp`.
  When artifacts come from a different origin (S3/CDN), they must be served with
  `Cross-Origin-Resource-Policy: cross-origin` (or CORS) so they load under COEP. The
  Fastify static route sets COOP/COEP/CORP in dev; the prod bucket/CDN must set CORP/CORS.
  Verify against the existing harness behavior in `tests/apps/kicad/`.

Artifacts are **not committed to git** (they're build outputs). The build pipeline
(`docker/build.sh` → `output/`) remains the source.

---

## 7. URL & routing  **[decided]**

Frontend routes (client-side router) **[default: react-router]**:

| Route | View |
|---|---|
| `/` | Project list + "Create project" |
| `/p/:project` | Project detail: file tree, upload, "open in tool" actions |
| `/p/:project/:tool/*filepath` | **Tool view**: boots `:tool` WASM app, auto-opens `*filepath` |

- `:tool` ∈ `{ pcbnew, eeschema, calculator }` — selects the WASM app **[decided]**.
  (`calculator` takes no file; opening it ignores `*filepath`.)
- `*filepath` is the project-relative path of the file to auto-open
  (e.g. `pcbnew/nyak.kicad_pcb`). It must match a `project_file.path` row.
- `:project` is the project **slug** within the default owner.
- Owner is implicit (default owner) now; route gains an owner segment when auth lands
  **[later]** — `/p/:project/...` is forward-compatible.

Tool→file-extension mapping (for validation / "open with" UI) **[default]**:
`.kicad_pcb → pcbnew`, `.kicad_sch → eeschema`. Mismatches surface a warning but the
explicit `:tool` segment wins (per the decided semantics).

---

## 8. API (ts-rest contract in `packages/contract`)  **[decided: ts-rest + Zod]**

All endpoints under `/api`. Contract is the single typed source; Fastify router
implements it, frontend uses the generated ts-rest react-query client **[default]**.

```
GET    /api/projects                       -> Project[]
POST   /api/projects                       { name, slug? } -> Project          # create [scope]
GET    /api/projects/:project              -> Project + file tree              # open  [scope]
DELETE /api/projects/:project              -> 204                              # [default, nice-to-have]

GET    /api/projects/:project/files                  -> ProjectFile[]
GET    /api/projects/:project/files/*path            -> file bytes (streamed)  # used to fill MEMFS
POST   /api/projects/:project/files                  (multipart) -> ProjectFile[]   # upload [scope]
POST   /api/projects/:project/files/zip              (multipart zip) -> ProjectFile[] # upload-zip [scope]
# write/rename/delete of individual files: interface ready, [later] for save-back
```

Upload handling **[decided: files + folder + zip]**:
- **Individual files (multi)**: multipart; each part carries its target project-relative
  path. Streamed to `FileStorage`, one `project_file` row each.
- **Folder (preserve structure)**: frontend uses `webkitdirectory`; relative paths derived
  from `file.webkitRelativePath` and sent as the per-file path. Server preserves the tree.
- **Zip**: server unpacks (streaming unzip) into the project tree, creating `project_file`
  rows per entry. Reject path traversal (`../`) and absolute paths.

Validation: Zod schemas (shared) validate bodies/params; Fastify JSON-schema serialization
for responses. Errors via a consistent ts-rest error shape **[default]**.

---

## 9. Open-a-file flow (the core of the iteration)

Visiting `/p/:project/:tool/*filepath`:

1. Frontend fetches the project's file tree: `GET /api/projects/:project/files`.
2. Frontend loads the WASM glue for `:tool` from `WASM_ASSET_BASE_URL` and instantiates the
   Emscripten module into a canvas-bearing harness (reuse the proven shell from
   `tests/apps/kicad/pcbnew.html` — `createCanvas`, `images.tar.gz` prefetch+write,
   `locateFile`, status/progress UI), ported into a React component.
3. **Sync whole project tree into MEMFS** **[decided]**: for every `project_file`, fetch its
   bytes (`GET .../files/*path`) and `FS.mkdirTree` + `FS.writeFile` at the project root
   inside MEMFS (mirroring `tests/kicad/utils/fs-inject.ts`). Files land at the path KiCad
   expects (e.g. under the default projects dir, confirmed by `load-pcb-probe.spec.ts`:
   `/home/kicad/documents/kicad/9.99/projects/...`). The exact MEMFS mount point for an
   arbitrary user project is an **open implementation detail** — see §11.
4. Drive File→Open on `*filepath` using the existing element-tracker / menu-driving
   helpers (`tests/e2e/utils/element-tracker.ts`, `tests/kicad/load-pcb.spec.ts`). This UI
   automation already works for the demo boards and is the reference implementation.
5. Render. Read-only — no write-back **[decided]**.

> **[later]** Lazy/partial loading: instead of syncing the whole tree up front, intercept
> MEMFS reads and fetch siblings on demand. This lands **together with save-back/sync**, as
> a single coherent iteration (both need MEMFS↔storage plumbing). Not now.

---

## 10. Frontend detail  **[shadcn decided; rest default]**

- Vite + React + TS, shadcn/ui components, Tailwind.
- Pages: project list (cards + create dialog), project detail (file tree + upload
  dropzone + per-file "open in pcbnew/eeschema" buttons), tool view (full-viewport WASM
  canvas + status overlay).
- Data layer: ts-rest react-query client generated from the contract.
- Upload UX: drag-drop dropzone supporting multi-file, folder (`webkitdirectory`), and
  `.zip`; progress per file; streamed to backend.
- The WASM tool view is a dedicated component that owns the Emscripten lifecycle and tears
  it down on unmount (WebGL context, MEMFS) to allow switching tools/projects.

---

## 11. Open questions for `implement plan` (not blocking this spec)

1. **MEMFS mount point for arbitrary projects.** Demos rely on KiCad's default projects
   path. For a user project we must decide where in MEMFS the tree is written and whether
   pcbnew/eeschema need it under their expected projects dir, or whether File→Open can
   target an arbitrary MEMFS path. Resolve by probing (extend `load-pcb-probe`).
2. **Driving File→Open generically.** Current helpers are tuned to the demo dialog flow
   (filelist bbox click + filename input + Enter). Confirm it generalizes to arbitrary
   paths, or expose a cleaner embind "open file" entry point in the WASM layer.
3. **eeschema/calculator open flows.** Mirror the pcbnew flow; verify eeschema's File→Open
   and that calculator (no file) just boots.
4. **Large-tree sync performance.** Whole-tree sync of a big project over many HTTP
   requests may be slow; consider a single tar/zip stream endpoint to fill MEMFS in one
   shot (still "sync whole tree", just one request). Decide in planning.
5. **COOP/COEP in prod** with cross-origin S3 artifacts — validate header matrix.
6. **Project slug generation/collision** rules; reserved tool names as slugs.

---

## 12. Out of scope this iteration (explicit)

- Auth / login / real multi-user (data model is ready; UI/enforcement is **[later]**).
- WebSocket / realtime / Hocuspocus collaboration (stack chosen to allow it; none built).
- Saving or syncing edits back to storage (**[later]**, paired with lazy load).
- S3 storage implementation (interface ready; **[later]**).
- Editing project files in the browser outside the WASM tools.

---

## 13. Definition of done (this iteration)

- `pnpm install && docker-compose up -d && pnpm dev` in `web/` brings up Postgres,
  Fastify (`/api` + WASM static), and the Vite frontend.
- Create a project from the UI; it appears in the list and in Postgres.
- Open the project; upload files via multi-select, folder, and zip — files appear in the
  tree and in `FileStorage` + `project_file`.
- Navigate to `/p/<slug>/pcbnew/<path>.kicad_pcb`; the board renders read-only, equivalent
  to the existing `tests/kicad/load-pcb` result, sourcing files from project storage.
- Same for an eeschema `.kicad_sch` file.
- Swapping `WASM_ASSET_BASE_URL` between local `output/` and a remote URL requires no code
  change.