Add baseline screenshots and visual regression testing
Add infrastructure for visual regression testing of wxWidgets WASM rendering: - Add 121 baseline screenshots capturing all UI states tested by Playwright - Add scripts/compare-screenshots.sh to compare test results vs baseline - Update tests/WHATWORKS.md with documentation on screenshot comparison - Update wxwidgets submodule with dialog refactoring The baseline screenshots cover: - Main app tabs (Controls, Text Input, Drawing, Lists, OpenGL, Grid, Dialogs) - Standalone test apps (menu, toolbar, layout, aui, clipboard, filedialog, etc.) - Dialog/MessageBox rendering (info, yes/no, error, custom dialogs) - OpenGL rendering tests (immediate mode, vertex arrays, matrix ops) - UI interactions (dropdowns, tab switching, z-ordering) Use ./scripts/compare-screenshots.sh to check for visual regressions. Small byte differences (<2%) are typically timestamps, not visual changes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
|
|
@ -227,6 +227,80 @@ cd tests/wasm-app && ./build-test-apps.sh
|
|||
|
||||
---
|
||||
|
||||
## Baseline Screenshots
|
||||
|
||||
The test suite captures screenshots during test runs for visual regression testing. These screenshots are compared against a baseline to detect unexpected visual changes.
|
||||
|
||||
### Directory Structure
|
||||
|
||||
```
|
||||
tests/
|
||||
├── baseline-screenshots/ # Known-good reference screenshots (121 files)
|
||||
├── test-results/ # Screenshots from latest test run
|
||||
└── wasm-app/
|
||||
└── e2e/ # Playwright test specs
|
||||
```
|
||||
|
||||
### Comparing Screenshots
|
||||
|
||||
Use the comparison script to check for visual regressions:
|
||||
|
||||
```bash
|
||||
./scripts/compare-screenshots.sh
|
||||
```
|
||||
|
||||
This will:
|
||||
- Compare all baseline screenshots with current test results
|
||||
- Report identical, different, and missing screenshots
|
||||
- Show file size differences (useful for detecting changes)
|
||||
|
||||
Example output:
|
||||
```
|
||||
=== Screenshot Comparison ===
|
||||
Baseline: /path/to/tests/baseline-screenshots
|
||||
Test Results: /path/to/tests/test-results
|
||||
|
||||
=== Comparing Baseline Screenshots ===
|
||||
DIFFERENT: dialogs-msgbox-info-open.png (baseline: 47468B, current: 47478B, diff: 10B / .02%)
|
||||
|
||||
=== Summary ===
|
||||
Total baseline screenshots: 121
|
||||
Identical: 54
|
||||
Different: 67
|
||||
Missing from test results: 0
|
||||
```
|
||||
|
||||
### Understanding Differences
|
||||
|
||||
Small byte differences (<2%) are typically caused by:
|
||||
- **Timestamps** in event logs (e.g., `[19:45:35]` vs `[18:49:35]`)
|
||||
- **PNG compression** variations between runs
|
||||
- **Anti-aliasing** differences
|
||||
|
||||
These are **not** visual regressions. Visually inspect screenshots if you see larger differences.
|
||||
|
||||
### Updating Baseline Screenshots
|
||||
|
||||
After verifying changes are intentional, update the baseline:
|
||||
|
||||
```bash
|
||||
# Copy current screenshots to baseline
|
||||
cp tests/test-results/*.png tests/baseline-screenshots/
|
||||
|
||||
# Or selectively update specific screenshots
|
||||
cp tests/test-results/dialog-*.png tests/baseline-screenshots/
|
||||
```
|
||||
|
||||
### Running Tests with Screenshots
|
||||
|
||||
```bash
|
||||
cd tests/wasm-app
|
||||
npm test # Run all tests (saves screenshots to test-results/)
|
||||
npx playwright test --ui # Interactive mode with screenshot preview
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Screenshots Reference
|
||||
|
||||
| Screenshot | Shows |
|
||||
|
|
|
|||
BIN
tests/baseline-screenshots/01-loading.png
Normal file
|
After Width: | Height: | Size: 5.9 KiB |
BIN
tests/baseline-screenshots/03-after-load.png
Normal file
|
After Width: | Height: | Size: 30 KiB |
BIN
tests/baseline-screenshots/04-controls-tab.png
Normal file
|
After Width: | Height: | Size: 34 KiB |
BIN
tests/baseline-screenshots/05-text-input-tab.png
Normal file
|
After Width: | Height: | Size: 32 KiB |
BIN
tests/baseline-screenshots/06-text-input-typed.png
Normal file
|
After Width: | Height: | Size: 34 KiB |
BIN
tests/baseline-screenshots/07-drawing-tab.png
Normal file
|
After Width: | Height: | Size: 36 KiB |
BIN
tests/baseline-screenshots/08-drawing-done.png
Normal file
|
After Width: | Height: | Size: 46 KiB |
BIN
tests/baseline-screenshots/09-lists-tab.png
Normal file
|
After Width: | Height: | Size: 47 KiB |
BIN
tests/baseline-screenshots/10-lists-clicked.png
Normal file
|
After Width: | Height: | Size: 44 KiB |
BIN
tests/baseline-screenshots/10a-choice-dropdown-open.png
Normal file
|
After Width: | Height: | Size: 44 KiB |
BIN
tests/baseline-screenshots/10b-choice-selected.png
Normal file
|
After Width: | Height: | Size: 43 KiB |
BIN
tests/baseline-screenshots/10c-choice-dropdown-reopen.png
Normal file
|
After Width: | Height: | Size: 43 KiB |
BIN
tests/baseline-screenshots/11-file-menu.png
Normal file
|
After Width: | Height: | Size: 80 KiB |
BIN
tests/baseline-screenshots/12-help-menu.png
Normal file
|
After Width: | Height: | Size: 80 KiB |
BIN
tests/baseline-screenshots/13-final.png
Normal file
|
After Width: | Height: | Size: 44 KiB |
BIN
tests/baseline-screenshots/14-opengl-tab.png
Normal file
|
After Width: | Height: | Size: 79 KiB |
BIN
tests/baseline-screenshots/15-opengl-tests.png
Normal file
|
After Width: | Height: | Size: 79 KiB |
BIN
tests/baseline-screenshots/aui-01-loaded.png
Normal file
|
After Width: | Height: | Size: 32 KiB |
BIN
tests/baseline-screenshots/aui-02-panels.png
Normal file
|
After Width: | Height: | Size: 32 KiB |
BIN
tests/baseline-screenshots/aui-03-close-clicked.png
Normal file
|
After Width: | Height: | Size: 32 KiB |
BIN
tests/baseline-screenshots/aui-04-dragged.png
Normal file
|
After Width: | Height: | Size: 32 KiB |
BIN
tests/baseline-screenshots/aui-05-multi-panel.png
Normal file
|
After Width: | Height: | Size: 32 KiB |
BIN
tests/baseline-screenshots/clipboard-01-loaded.png
Normal file
|
After Width: | Height: | Size: 35 KiB |
BIN
tests/baseline-screenshots/clipboard-02-copy-clicked.png
Normal file
|
After Width: | Height: | Size: 35 KiB |
BIN
tests/baseline-screenshots/clipboard-03-paste-clicked.png
Normal file
|
After Width: | Height: | Size: 35 KiB |
BIN
tests/baseline-screenshots/clipboard-04-check-clicked.png
Normal file
|
After Width: | Height: | Size: 42 KiB |
BIN
tests/baseline-screenshots/clipboard-05-clear-clicked.png
Normal file
|
After Width: | Height: | Size: 42 KiB |
BIN
tests/baseline-screenshots/clipboard-06-full-flow.png
Normal file
|
After Width: | Height: | Size: 48 KiB |
BIN
tests/baseline-screenshots/dialog-01-loaded.png
Normal file
|
After Width: | Height: | Size: 34 KiB |
BIN
tests/baseline-screenshots/dialog-02-info-clicked.png
Normal file
|
After Width: | Height: | Size: 42 KiB |
BIN
tests/baseline-screenshots/dialog-03-yesno-clicked.png
Normal file
|
After Width: | Height: | Size: 45 KiB |
BIN
tests/baseline-screenshots/dialog-04-error-clicked.png
Normal file
|
After Width: | Height: | Size: 42 KiB |
BIN
tests/baseline-screenshots/dialog-05-custom-clicked.png
Normal file
|
After Width: | Height: | Size: 34 KiB |
BIN
tests/baseline-screenshots/dialogs-00-initial.png
Normal file
|
After Width: | Height: | Size: 30 KiB |
BIN
tests/baseline-screenshots/dialogs-01-tab-selected.png
Normal file
|
After Width: | Height: | Size: 46 KiB |
BIN
tests/baseline-screenshots/dialogs-custom-closed.png
Normal file
|
After Width: | Height: | Size: 56 KiB |
BIN
tests/baseline-screenshots/dialogs-custom-open.png
Normal file
|
After Width: | Height: | Size: 56 KiB |
BIN
tests/baseline-screenshots/dialogs-full-flow.png
Normal file
|
After Width: | Height: | Size: 56 KiB |
BIN
tests/baseline-screenshots/dialogs-msgbox-error-closed.png
Normal file
|
After Width: | Height: | Size: 46 KiB |
BIN
tests/baseline-screenshots/dialogs-msgbox-error-open.png
Normal file
|
After Width: | Height: | Size: 46 KiB |
BIN
tests/baseline-screenshots/dialogs-msgbox-info-closed.png
Normal file
|
After Width: | Height: | Size: 46 KiB |
BIN
tests/baseline-screenshots/dialogs-msgbox-info-open.png
Normal file
|
After Width: | Height: | Size: 46 KiB |
BIN
tests/baseline-screenshots/dialogs-msgbox-yesno-closed.png
Normal file
|
After Width: | Height: | Size: 46 KiB |
BIN
tests/baseline-screenshots/dialogs-msgbox-yesno-open.png
Normal file
|
After Width: | Height: | Size: 46 KiB |
BIN
tests/baseline-screenshots/dialogs-timer-initial.png
Normal file
|
After Width: | Height: | Size: 46 KiB |
BIN
tests/baseline-screenshots/dialogs-timer-multiple.png
Normal file
|
After Width: | Height: | Size: 46 KiB |
BIN
tests/baseline-screenshots/dialogs-timer-running.png
Normal file
|
After Width: | Height: | Size: 46 KiB |
BIN
tests/baseline-screenshots/dialogs-timer-started.png
Normal file
|
After Width: | Height: | Size: 46 KiB |
BIN
tests/baseline-screenshots/dialogs-timer-stopped.png
Normal file
|
After Width: | Height: | Size: 46 KiB |
BIN
tests/baseline-screenshots/filedialog-01-loaded.png
Normal file
|
After Width: | Height: | Size: 24 KiB |
BIN
tests/baseline-screenshots/filedialog-02-open-clicked.png
Normal file
|
After Width: | Height: | Size: 24 KiB |
BIN
tests/baseline-screenshots/filedialog-03-save-clicked.png
Normal file
|
After Width: | Height: | Size: 24 KiB |
BIN
tests/baseline-screenshots/filedialog-04-multiple-clicked.png
Normal file
|
After Width: | Height: | Size: 24 KiB |
BIN
tests/baseline-screenshots/filedialog-05-all-buttons.png
Normal file
|
After Width: | Height: | Size: 24 KiB |
BIN
tests/baseline-screenshots/gl-01-opengl-tab.png
Normal file
|
After Width: | Height: | Size: 69 KiB |
BIN
tests/baseline-screenshots/gl-02-vertex-arrays-selected.png
Normal file
|
After Width: | Height: | Size: 75 KiB |
BIN
tests/baseline-screenshots/gl-03-after-wait.png
Normal file
|
After Width: | Height: | Size: 75 KiB |
BIN
tests/baseline-screenshots/gl-run-all-tests.png
Normal file
|
After Width: | Height: | Size: 69 KiB |
BIN
tests/baseline-screenshots/gl-test-0-immediate-mode.png
Normal file
|
After Width: | Height: | Size: 73 KiB |
BIN
tests/baseline-screenshots/gl-test-1-matrix-ops.png
Normal file
|
After Width: | Height: | Size: 72 KiB |
BIN
tests/baseline-screenshots/gl-test-2-vertex-arrays.png
Normal file
|
After Width: | Height: | Size: 82 KiB |
BIN
tests/baseline-screenshots/gl-test-3-state-mgmt.png
Normal file
|
After Width: | Height: | Size: 80 KiB |
BIN
tests/baseline-screenshots/gl-test-4-texture-coords.png
Normal file
|
After Width: | Height: | Size: 80 KiB |
BIN
tests/baseline-screenshots/glcanvas-01-on-opengl-tab.png
Normal file
|
After Width: | Height: | Size: 70 KiB |
|
After Width: | Height: | Size: 38 KiB |
BIN
tests/baseline-screenshots/glcanvas-03-on-drawing-tab.png
Normal file
|
After Width: | Height: | Size: 36 KiB |
BIN
tests/baseline-screenshots/glcanvas-04-on-lists-tab.png
Normal file
|
After Width: | Height: | Size: 43 KiB |
BIN
tests/baseline-screenshots/grid-tab-final.png
Normal file
|
After Width: | Height: | Size: 41 KiB |
BIN
tests/baseline-screenshots/layout-01-loaded.png
Normal file
|
After Width: | Height: | Size: 64 KiB |
BIN
tests/baseline-screenshots/layout-02-splitter.png
Normal file
|
After Width: | Height: | Size: 64 KiB |
BIN
tests/baseline-screenshots/layout-03-sash-dragged.png
Normal file
|
After Width: | Height: | Size: 64 KiB |
BIN
tests/baseline-screenshots/layout-04-scrolled-left.png
Normal file
|
After Width: | Height: | Size: 66 KiB |
BIN
tests/baseline-screenshots/layout-05-scrolled-right.png
Normal file
|
After Width: | Height: | Size: 63 KiB |
BIN
tests/baseline-screenshots/layout-06-combined.png
Normal file
|
After Width: | Height: | Size: 64 KiB |
BIN
tests/baseline-screenshots/menu-01-loaded.png
Normal file
|
After Width: | Height: | Size: 28 KiB |
BIN
tests/baseline-screenshots/menu-02-menubar.png
Normal file
|
After Width: | Height: | Size: 28 KiB |
BIN
tests/baseline-screenshots/menu-03-file-clicked.png
Normal file
|
After Width: | Height: | Size: 28 KiB |
BIN
tests/baseline-screenshots/menu-04-edit-clicked.png
Normal file
|
After Width: | Height: | Size: 31 KiB |
BIN
tests/baseline-screenshots/menu-05-all-menus.png
Normal file
|
After Width: | Height: | Size: 29 KiB |
BIN
tests/baseline-screenshots/opengl-after-tests.png
Normal file
|
After Width: | Height: | Size: 70 KiB |
BIN
tests/baseline-screenshots/opengl-before-debug.png
Normal file
|
After Width: | Height: | Size: 70 KiB |
BIN
tests/baseline-screenshots/opengl-render.png
Normal file
|
After Width: | Height: | Size: 70 KiB |
BIN
tests/baseline-screenshots/opengl-tab-initial.png
Normal file
|
After Width: | Height: | Size: 70 KiB |
BIN
tests/baseline-screenshots/searchctrl-01-visible.png
Normal file
|
After Width: | Height: | Size: 41 KiB |
BIN
tests/baseline-screenshots/searchctrl-02-typing.png
Normal file
|
After Width: | Height: | Size: 41 KiB |
BIN
tests/baseline-screenshots/searchctrl-03-after-enter.png
Normal file
|
After Width: | Height: | Size: 41 KiB |
BIN
tests/baseline-screenshots/spinctrl-01-visible.png
Normal file
|
After Width: | Height: | Size: 41 KiB |
BIN
tests/baseline-screenshots/spinctrl-02-after-click.png
Normal file
|
After Width: | Height: | Size: 41 KiB |
BIN
tests/baseline-screenshots/tabswitch-01-controls.png
Normal file
|
After Width: | Height: | Size: 30 KiB |
BIN
tests/baseline-screenshots/tabswitch-02-opengl.png
Normal file
|
After Width: | Height: | Size: 70 KiB |
BIN
tests/baseline-screenshots/tabswitch-03-lists.png
Normal file
|
After Width: | Height: | Size: 37 KiB |
BIN
tests/baseline-screenshots/tabswitch-04-opengl.png
Normal file
|
After Width: | Height: | Size: 76 KiB |
BIN
tests/baseline-screenshots/tabswitch-05-drawing.png
Normal file
|
After Width: | Height: | Size: 39 KiB |
BIN
tests/baseline-screenshots/tabswitch-06-controls.png
Normal file
|
After Width: | Height: | Size: 45 KiB |
BIN
tests/baseline-screenshots/timer-01-loaded.png
Normal file
|
After Width: | Height: | Size: 35 KiB |
BIN
tests/baseline-screenshots/timer-02-started.png
Normal file
|
After Width: | Height: | Size: 35 KiB |
BIN
tests/baseline-screenshots/timer-05-fast-started.png
Normal file
|
After Width: | Height: | Size: 35 KiB |
BIN
tests/baseline-screenshots/timer-08-reset.png
Normal file
|
After Width: | Height: | Size: 35 KiB |
BIN
tests/baseline-screenshots/toolbar-01-loaded.png
Normal file
|
After Width: | Height: | Size: 36 KiB |