build+perf: wasm-opt the shipped wasm, and measure real frames in CI
emcc only runs Binaryen at link -O2+ (link.py: should_run_binaryen_optimizer returns OPT_LEVEL >= 2) and we link at -O1, so the shipped module had never seen wasm-opt at all — it kept its entire 19.56 MB name section, ~20% of the editor (-sJSPI sets ASYNCIFY=2, which suppresses wasm-ld's --strip-debug, leaving wasm-opt as the only thing that would drop it). Step 8.2 runs it post-link and in-container, so CI's cached compile phase covers it and the host post-process stays pure-host. Default -O2, picked by measuring every level on the same module: -O0 already captures 27% of the raw win (it is mostly the name section), -O2 costs 23 s and gives the best frame rate, and -O3/-O4/-Os/-Oz cost 48-132 s for at most 1.5% more brotli — -O4 is not even smaller than -O3. Targets that already link -O2/-Oz (occ_service, kicad_tools) are skipped by testing for the target_features section, which emcc strips whenever it ran the optimizer itself, so there is no hard-coded target list to drift. Feature flags come from the module's own target_features section and so cannot diverge from the link. The perf specs reported requestAnimationFrame ticks as "FPS". That is not a frame rate: rAF fires on the compositor's schedule whether or not the GAL redrew, and it read 120/s on a board where the renderer completed zero frames in six seconds. measureInteractionFps now counts completed GAL frames — runs of draws to the default framebuffer, exactly one per frame in every AA mode — and drives a pure middle-drag pan after a zoom-to-fit. Mixing wheel zoom into the drive made the result depend on where the wheel left the view: +-20% across identical repeats, against +-2% for pan alone. The report gains a GAL fps column with a regression flag on the 1x number; rAF is kept so historical runs stay comparable. CI has no GPU, so its number is a software-rasteriser redraw rate — a regression signal, not a user-facing frame rate. Method and measurements in the bench report. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016X9eh1s5sTx1o9Em9KBuwR
This commit is contained in:
Viktor Vaczi
parent
745b8f413a
commit
f419a1fedd
7 changed files with 536 additions and 56 deletions
|
|
@ -1,12 +1,12 @@
|
|||
import { test, expect } from './fixtures';
|
||||
import * as path from 'path';
|
||||
import { measureLoad, measureOpenRender, measureFps, setThrottle, recordPerf } from './utils/perf-utils';
|
||||
import { measureLoad, measureOpenRender, measureInteractionFps, setThrottle, recordPerf } from './utils/perf-utils';
|
||||
|
||||
/**
|
||||
* eeschema runtime-perf (TRACK-ONLY, no gating).
|
||||
*
|
||||
* Measures the CURRENT build: cold load, open+render of the demo schematic, and
|
||||
* sustained pan/zoom FPS across CPU-throttle rates. Numbers are logged and written
|
||||
* sustained pan FPS across CPU-throttle rates. Numbers are logged and written
|
||||
* to tests/test-results/perf-eeschema.json (CI uploads it). The only assertions are
|
||||
* "the app booted and the doc opened" — never a perf threshold (would flake CI).
|
||||
* Runs on the Chromium 'perf' project (CDP throttling); pass --headed for real-GPU FPS.
|
||||
|
|
@ -29,12 +29,14 @@ test.describe('eeschema perf', () => {
|
|||
await page.keyboard.press('Escape').catch(() => {}); // eslint-disable-line -- best-effort Escape (may not apply in all states)
|
||||
|
||||
const cdp = await page.context().newCDPSession(page);
|
||||
const fps: { throttle: number; fps: number }[] = [];
|
||||
// galFps is the real frame rate (completed GAL frames). fps is the legacy
|
||||
// rAF tick count, kept so historical CI numbers stay comparable.
|
||||
const fps: { throttle: number; fps: number; galFps: number }[] = [];
|
||||
for (const rate of THROTTLES) {
|
||||
await setThrottle(cdp, rate);
|
||||
const f = await measureFps(page, FPS_SECS);
|
||||
console.log(`[perf] eeschema FPS @ ${rate}x = ${f}`);
|
||||
fps.push({ throttle: rate, fps: f });
|
||||
const f = await measureInteractionFps(page, FPS_SECS);
|
||||
console.log(`[perf] eeschema @ ${rate}x: GAL ${f.galFps} fps (rAF ${f.rafFps})`);
|
||||
fps.push({ throttle: rate, fps: f.rafFps, galFps: f.galFps });
|
||||
}
|
||||
await setThrottle(cdp, 1);
|
||||
|
||||
|
|
|
|||
|
|
@ -1,12 +1,12 @@
|
|||
import { test, expect } from './fixtures';
|
||||
import * as path from 'path';
|
||||
import { measureLoad, measureOpenRender, measureFps, setThrottle, recordPerf } from './utils/perf-utils';
|
||||
import { measureLoad, measureOpenRender, measureInteractionFps, setThrottle, recordPerf } from './utils/perf-utils';
|
||||
|
||||
/**
|
||||
* pcbnew runtime-perf (TRACK-ONLY, no gating). Mirrors eeschema-perf for the board editor.
|
||||
*
|
||||
* Measures the CURRENT build: cold load, open+render of the demo board, and sustained
|
||||
* pan/zoom FPS across CPU-throttle rates → tests/test-results/perf-pcbnew.json (CI uploads it).
|
||||
* pan FPS across CPU-throttle rates → tests/test-results/perf-pcbnew.json (CI uploads it).
|
||||
* Runs on the Chromium 'perf' project (pcbnew's big module OOMs Firefox/SpiderMonkey anyway,
|
||||
* and CDP throttling is Chromium-only). Only asserts booted + opened; never a perf threshold.
|
||||
*/
|
||||
|
|
@ -27,12 +27,14 @@ test.describe('pcbnew perf', () => {
|
|||
await page.keyboard.press('Escape').catch(() => {}); // eslint-disable-line -- best-effort Escape (may not apply in all states)
|
||||
|
||||
const cdp = await page.context().newCDPSession(page);
|
||||
const fps: { throttle: number; fps: number }[] = [];
|
||||
// galFps is the real frame rate (completed GAL frames). fps is the legacy
|
||||
// rAF tick count, kept so historical CI numbers stay comparable.
|
||||
const fps: { throttle: number; fps: number; galFps: number }[] = [];
|
||||
for (const rate of THROTTLES) {
|
||||
await setThrottle(cdp, rate);
|
||||
const f = await measureFps(page, FPS_SECS);
|
||||
console.log(`[perf] pcbnew FPS @ ${rate}x = ${f}`);
|
||||
fps.push({ throttle: rate, fps: f });
|
||||
const f = await measureInteractionFps(page, FPS_SECS);
|
||||
console.log(`[perf] pcbnew @ ${rate}x: GAL ${f.galFps} fps (rAF ${f.rafFps})`);
|
||||
fps.push({ throttle: rate, fps: f.rafFps, galFps: f.galFps });
|
||||
}
|
||||
await setThrottle(cdp, 1);
|
||||
|
||||
|
|
|
|||
|
|
@ -166,10 +166,137 @@ export async function setThrottle(cdp: CDPSession, rate: number): Promise<void>
|
|||
await cdp.send('Emulation.setCPUThrottlingRate', { rate });
|
||||
}
|
||||
|
||||
/**
|
||||
* Count completed GAL frames instead of rAF ticks.
|
||||
*
|
||||
* A GAL frame ends with the compositor blitting to the DEFAULT framebuffer, so a
|
||||
* run of draws issued while no framebuffer is bound is exactly one frame. The run
|
||||
* has to be collapsed: the number of present draws per frame depends on the AA
|
||||
* mode (1 under supersampling, 2 under AA_NONE, +1 when the crosshair is drawn),
|
||||
* but a run *boundary* happens once per frame in every mode — so no divisor.
|
||||
*
|
||||
* Wrapping the prototypes works even though the context already exists: methods
|
||||
* resolve on the prototype at call time, not at context creation. The initial
|
||||
* framebuffer binding is assumed to be the default and self-corrects on the first
|
||||
* bindFramebuffer, which the GAL issues several times per frame.
|
||||
*/
|
||||
async function installGalFrameCounter(page: Page): Promise<void> {
|
||||
await page.evaluate(() => {
|
||||
const w = window as unknown as { __galFrames?: number; __galHooked?: boolean };
|
||||
w.__galFrames = 0;
|
||||
if (w.__galHooked) return;
|
||||
w.__galHooked = true;
|
||||
const protos = [
|
||||
(window as unknown as { WebGL2RenderingContext?: { prototype: object } }).WebGL2RenderingContext,
|
||||
(window as unknown as { WebGLRenderingContext?: { prototype: object } }).WebGLRenderingContext,
|
||||
].filter(Boolean) as Array<{ prototype: Record<string, unknown> }>;
|
||||
const state = new WeakMap<object, { fb: unknown; inPresent: boolean }>();
|
||||
const st = (ctx: object) => {
|
||||
let s = state.get(ctx);
|
||||
if (!s) { s = { fb: null, inPresent: false }; state.set(ctx, s); }
|
||||
return s;
|
||||
};
|
||||
const DRAWS = ['drawArrays', 'drawElements', 'drawArraysInstanced', 'drawElementsInstanced', 'drawRangeElements'];
|
||||
for (const proto of protos) {
|
||||
for (const name of ['bindFramebuffer', ...DRAWS]) {
|
||||
const orig = proto.prototype[name] as ((...a: unknown[]) => unknown) | undefined;
|
||||
if (typeof orig !== 'function') continue;
|
||||
const isDraw = DRAWS.indexOf(name) >= 0;
|
||||
proto.prototype[name] = function (this: object, ...args: unknown[]) {
|
||||
const s = st(this);
|
||||
if (!isDraw) {
|
||||
s.fb = args[1];
|
||||
if (args[1]) s.inPresent = false;
|
||||
} else if (s.fb === null || s.fb === undefined) {
|
||||
if (!s.inPresent) { s.inPresent = true; w.__galFrames = (w.__galFrames ?? 0) + 1; }
|
||||
} else {
|
||||
s.inPresent = false;
|
||||
}
|
||||
return orig.apply(this, args);
|
||||
};
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
/** Zoom-to-fit, so every measurement starts from the same visible geometry. */
|
||||
async function resetViewToFit(page: Page, cx: number, cy: number): Promise<void> {
|
||||
await page.mouse.move(cx, cy);
|
||||
await page.keyboard.press('Escape').catch(() => {}); // eslint-disable-line -- best-effort
|
||||
await page.keyboard.press('Home').catch(() => {}); // eslint-disable-line -- best-effort
|
||||
await page.waitForFunction(() => true, null, { timeout: 5000 });
|
||||
}
|
||||
|
||||
/**
|
||||
* Sustained interaction FPS, reported two ways.
|
||||
*
|
||||
* `galFps` is the real one: completed GAL frames per second (see
|
||||
* installGalFrameCounter). `rafFps` is the legacy main-thread requestAnimationFrame
|
||||
* count, kept only so historical CI numbers stay comparable — it is NOT a frame
|
||||
* rate. rAF ticks on the compositor's schedule whether or not the GAL redrew, so
|
||||
* it can read 120 while the renderer is completely stalled (measured: the 80 MB
|
||||
* jetson board on a software rasteriser renders 0 frames while rAF reports 120).
|
||||
*
|
||||
* The drive is a pure middle-drag PAN. Wheel zoom used to be mixed into the same
|
||||
* loop, and it makes the metric unusable: zooming continuously changes how much
|
||||
* geometry is on screen, so the result depends on where the wheel happens to
|
||||
* leave the view. Measured spread across three identical repeats was ±20%
|
||||
* (34.7 / 42.1 / 27.1 fps) with zoom in the loop, versus ±2% (19.9 / 19.0 / 19.6)
|
||||
* for pan alone. Pan also keeps the workload honest — it continuously reveals
|
||||
* geometry that has to be cached, which is the expensive path.
|
||||
*
|
||||
* The view is zoomed to fit first, so every run starts from the same visible
|
||||
* geometry (the whole board — the worst case) rather than inheriting whatever
|
||||
* zoom level the previous measurement left behind.
|
||||
*/
|
||||
export async function measureInteractionFps(
|
||||
page: Page,
|
||||
seconds: number,
|
||||
): Promise<{ rafFps: number; galFps: number }> {
|
||||
const box = await page.locator(MAIN_CANVAS).boundingBox();
|
||||
if (!box) return { rafFps: 0, galFps: 0 };
|
||||
const cx = box.x + box.width / 2;
|
||||
const cy = box.y + box.height / 2;
|
||||
await resetViewToFit(page, cx, cy);
|
||||
await installGalFrameCounter(page);
|
||||
|
||||
type W = { __perfFrames: number; __perfRAF?: number; __galFrames?: number };
|
||||
await page.evaluate(() => {
|
||||
const w = window as unknown as W;
|
||||
if (w.__perfRAF !== undefined) cancelAnimationFrame(w.__perfRAF);
|
||||
w.__perfFrames = 0;
|
||||
w.__galFrames = 0;
|
||||
const loop = () => {
|
||||
w.__perfFrames++;
|
||||
w.__perfRAF = requestAnimationFrame(loop);
|
||||
};
|
||||
w.__perfRAF = requestAnimationFrame(loop);
|
||||
});
|
||||
const start = Date.now();
|
||||
let k = 0;
|
||||
await page.mouse.move(cx, cy);
|
||||
await page.mouse.down({ button: 'middle' });
|
||||
while (Date.now() - start < seconds * 1000) {
|
||||
await page.mouse.move(cx + Math.round(140 * Math.sin(k / 6)), cy + Math.round(90 * Math.cos(k / 7)));
|
||||
k++;
|
||||
}
|
||||
await page.mouse.up({ button: 'middle' });
|
||||
const elapsed = Date.now() - start;
|
||||
const counts = await page.evaluate(() => {
|
||||
const w = window as unknown as W;
|
||||
if (w.__perfRAF !== undefined) cancelAnimationFrame(w.__perfRAF);
|
||||
return { raf: w.__perfFrames, gal: w.__galFrames ?? 0 };
|
||||
});
|
||||
const secs = elapsed / 1000;
|
||||
return { rafFps: +(counts.raf / secs).toFixed(1), galFps: +(counts.gal / secs).toFixed(1) };
|
||||
}
|
||||
|
||||
/**
|
||||
* Sustained interaction FPS: drive real pan/zoom on #canvas (the emscripten input
|
||||
* surface — glcanvas-* can be display:none) for `seconds`, counting main-thread rAF
|
||||
* frames. Whatever throttle is currently set applies.
|
||||
*
|
||||
* @deprecated rAF ticks are not GAL frames — use measureInteractionFps().galFps.
|
||||
*/
|
||||
export async function measureFps(page: Page, seconds: number): Promise<number> {
|
||||
const box = await page.locator(MAIN_CANVAS).boundingBox();
|
||||
|
|
|
|||
|
|
@ -2,16 +2,17 @@
|
|||
* Renders the track-only runtime-perf block for the CI-on-main Discord comment.
|
||||
*
|
||||
* The perf e2e (tests/kicad/{eeschema,pcbnew}-perf.spec.ts) already writes
|
||||
* test-results/perf-{app}.json — schema { app, when, loadMs, openMs, fps:[{throttle,fps}] }.
|
||||
* test-results/perf-{app}.json — schema { app, when, loadMs, openMs, fps:[{throttle,fps,galFps}] }.
|
||||
* We read those, fetch the PREVIOUS successful main run's perf via `gh run
|
||||
* download` (so we can show a Δ without committing a baseline — stays
|
||||
* no-write-back), and format an aligned monospace table (Discord doesn't render
|
||||
* markdown tables, so it goes in a ``` code block).
|
||||
*
|
||||
* Track-only: nothing here gates the build. A regression past REGRESSION_PCT on
|
||||
* the stable metrics (loadMs/openMs) is only flagged (a `*`), never failed. FPS
|
||||
* is CPU-bound/noisy on CI's headless SwiftShader path, so it's shown but marked
|
||||
* indicative.
|
||||
* the stable metrics (loadMs/openMs) is only flagged (a `*`), never failed. Both
|
||||
* FPS columns are measured on CI's software rasteriser (ANGLE over Mesa llvmpipe,
|
||||
* under Xvfb — there is no GPU on the runner), so they are a regression signal
|
||||
* only and never a user-facing frame rate.
|
||||
*
|
||||
* CLI (from tests/):
|
||||
* tsx tools/screenshots/perf-report.ts [--results DIR] [--prev DIR] [--repo owner/repo]
|
||||
|
|
@ -26,7 +27,8 @@ export const PERF_APPS = ['eeschema', 'pcbnew'] as const;
|
|||
const REGRESSION_PCT = 10; // stable-metric regression past this is flagged with `*`
|
||||
const CI_WORKFLOW = 'ci-ubicloud.yml';
|
||||
|
||||
export type Fps = { throttle: number; fps: number };
|
||||
/** `fps` is the legacy rAF tick count; `galFps` is the real frame rate. */
|
||||
export type Fps = { throttle: number; fps: number; galFps?: number };
|
||||
export type PerfData = { app: string; when?: string; loadMs: number; openMs: number; fps: Fps[] };
|
||||
|
||||
export function readPerf(dir: string): Map<string, PerfData> {
|
||||
|
|
@ -101,10 +103,32 @@ function fmtMetric(cur: number, prev?: number): string {
|
|||
return `${cur} ${arrow}${Math.abs(p).toFixed(0)}%${flag}`;
|
||||
}
|
||||
|
||||
function fmtFps(fps: Fps[]): string {
|
||||
/**
|
||||
* GAL fps with a Δ vs the previous main run, flagged on REGRESSION.
|
||||
*
|
||||
* Higher is better here, so the sign convention is inverted relative to
|
||||
* fmtMetric: a DROP past the threshold gets the `*`. Only the 1x-throttle
|
||||
* number drives the flag — it is the one that repeats within ~2% (pan-only
|
||||
* drive, zoom-to-fit before each measurement), so it is safe to act on.
|
||||
* CI has no GPU, so this is a software-rasteriser redraw rate: useful precisely
|
||||
* because it is consistent, not because it is the user-facing frame rate.
|
||||
*/
|
||||
function fmtGalFps(fps: Fps[], prev?: Fps[]): string {
|
||||
const triple = fmtFps(fps, 'galFps');
|
||||
const cur = fps.find((f) => f.throttle === 1)?.galFps;
|
||||
const was = prev?.find((f) => f.throttle === 1)?.galFps;
|
||||
if (cur === undefined || was === undefined || was === 0) return triple;
|
||||
const p = pct(cur, was); // + = faster than before
|
||||
const arrow = p > 0 ? '▲' : p < 0 ? '▼' : '·';
|
||||
const flag = -p > REGRESSION_PCT ? '*' : '';
|
||||
return `${triple} ${arrow}${Math.abs(p).toFixed(0)}%${flag}`;
|
||||
}
|
||||
|
||||
function fmtFps(fps: Fps[], key: 'fps' | 'galFps' = 'fps'): string {
|
||||
return [1, 4, 6].map((t) => {
|
||||
const hit = fps.find((f) => f.throttle === t);
|
||||
return hit ? Math.round(hit.fps) : '–';
|
||||
const v = hit?.[key];
|
||||
return v === undefined ? '–' : Math.round(v);
|
||||
}).join('/');
|
||||
}
|
||||
|
||||
|
|
@ -120,7 +144,7 @@ export function buildPerfReport(opts: { resultsDir?: string; prevDir?: string |
|
|||
if (cur.size === 0) return { block: '', regressed: false };
|
||||
const prev = opts.prevDir ? readPerf(opts.prevDir) : new Map<string, PerfData>();
|
||||
|
||||
const headers = ['app', 'loadMs (Δ)', 'openMs (Δ)', 'FPS 1/4/6'];
|
||||
const headers = ['app', 'loadMs (Δ)', 'openMs (Δ)', 'GAL fps 1/4/6 (Δ@1x)', 'rAF 1/4/6'];
|
||||
const rows: string[][] = [];
|
||||
let regressed = false;
|
||||
for (const app of PERF_APPS) {
|
||||
|
|
@ -129,8 +153,9 @@ export function buildPerfReport(opts: { resultsDir?: string; prevDir?: string |
|
|||
const p = prev.get(app);
|
||||
const loadCell = fmtMetric(c.loadMs, p?.loadMs);
|
||||
const openCell = fmtMetric(c.openMs, p?.openMs);
|
||||
if (loadCell.endsWith('*') || openCell.endsWith('*')) regressed = true;
|
||||
rows.push([app, loadCell, openCell, fmtFps(c.fps)]);
|
||||
const galCell = fmtGalFps(c.fps, p?.fps);
|
||||
if (loadCell.endsWith('*') || openCell.endsWith('*') || galCell.endsWith('*')) regressed = true;
|
||||
rows.push([app, loadCell, openCell, galCell, fmtFps(c.fps, 'fps')]);
|
||||
}
|
||||
if (rows.length === 0) return { block: '', regressed: false };
|
||||
|
||||
|
|
@ -138,7 +163,11 @@ export function buildPerfReport(opts: { resultsDir?: string; prevDir?: string |
|
|||
const line = (cells: string[]) => cells.map((c, i) => pad(c, widths[i])).join(' ');
|
||||
const body = [line(headers), rows.map((r) => line(r)).join('\n')].join('\n');
|
||||
const footnote = `${prev.size ? 'Δ vs previous main run. ' : 'no prior main run for Δ. '}` +
|
||||
`* = >${REGRESSION_PCT}% slower (track-only, non-gating). FPS is CI-headless — indicative only.`;
|
||||
`* = >${REGRESSION_PCT}% slower (track-only, non-gating). ` +
|
||||
`GAL fps = completed GAL frames on CI's software rasteriser (llvmpipe) — a regression signal, ` +
|
||||
`NOT user-facing frame rate; its Δ/* are computed on the 1x number. rAF is the legacy tick count, ` +
|
||||
`kept for continuity: it ticks on the compositor's schedule whether or not anything rendered, so ` +
|
||||
`it can read 120 while the renderer is stalled.`;
|
||||
return { block: '**Runtime perf** (eeschema + pcbnew)\n```\n' + body + '\n```\n' + footnote, regressed };
|
||||
}
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue