Add baseline screenshots and visual regression testing
Add infrastructure for visual regression testing of wxWidgets WASM rendering: - Add 121 baseline screenshots capturing all UI states tested by Playwright - Add scripts/compare-screenshots.sh to compare test results vs baseline - Update tests/WHATWORKS.md with documentation on screenshot comparison - Update wxwidgets submodule with dialog refactoring The baseline screenshots cover: - Main app tabs (Controls, Text Input, Drawing, Lists, OpenGL, Grid, Dialogs) - Standalone test apps (menu, toolbar, layout, aui, clipboard, filedialog, etc.) - Dialog/MessageBox rendering (info, yes/no, error, custom dialogs) - OpenGL rendering tests (immediate mode, vertex arrays, matrix ops) - UI interactions (dropdowns, tab switching, z-ordering) Use ./scripts/compare-screenshots.sh to check for visual regressions. Small byte differences (<2%) are typically timestamps, not visual changes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
107
scripts/compare-screenshots.sh
Executable file
|
|
@ -0,0 +1,107 @@
|
|||
#!/bin/bash
|
||||
|
||||
# Compare test screenshots with baseline screenshots
|
||||
# This script compares all PNG files in baseline-screenshots with test-results
|
||||
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
|
||||
BASELINE_DIR="$PROJECT_ROOT/tests/baseline-screenshots"
|
||||
TEST_RESULTS_DIR="$PROJECT_ROOT/tests/test-results"
|
||||
|
||||
# Check if directories exist
|
||||
if [ ! -d "$BASELINE_DIR" ]; then
|
||||
echo "ERROR: Baseline directory not found: $BASELINE_DIR"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ ! -d "$TEST_RESULTS_DIR" ]; then
|
||||
echo "ERROR: Test results directory not found: $TEST_RESULTS_DIR"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "=== Screenshot Comparison ==="
|
||||
echo "Baseline: $BASELINE_DIR"
|
||||
echo "Test Results: $TEST_RESULTS_DIR"
|
||||
echo ""
|
||||
|
||||
# Counters
|
||||
total=0
|
||||
identical=0
|
||||
different=0
|
||||
missing_current=0
|
||||
extra_current=0
|
||||
|
||||
# Compare all baseline screenshots
|
||||
echo "=== Comparing Baseline Screenshots ==="
|
||||
for baseline in "$BASELINE_DIR"/*.png; do
|
||||
filename=$(basename "$baseline")
|
||||
current="$TEST_RESULTS_DIR/$filename"
|
||||
total=$((total + 1))
|
||||
|
||||
if [ ! -f "$current" ]; then
|
||||
echo "MISSING: $filename (not in test results)"
|
||||
missing_current=$((missing_current + 1))
|
||||
continue
|
||||
fi
|
||||
|
||||
# Compare using cmp (byte-by-byte comparison)
|
||||
if cmp -s "$baseline" "$current"; then
|
||||
identical=$((identical + 1))
|
||||
# Only show identical files if verbose
|
||||
if [ "$1" = "-v" ] || [ "$1" = "--verbose" ]; then
|
||||
echo "IDENTICAL: $filename"
|
||||
fi
|
||||
else
|
||||
different=$((different + 1))
|
||||
|
||||
# Get file sizes
|
||||
baseline_size=$(stat -f%z "$baseline" 2>/dev/null || stat -c%s "$baseline")
|
||||
current_size=$(stat -f%z "$current" 2>/dev/null || stat -c%s "$current")
|
||||
diff_bytes=$((current_size - baseline_size))
|
||||
|
||||
# Calculate percent difference
|
||||
if [ "$baseline_size" -gt 0 ]; then
|
||||
diff_pct=$(echo "scale=2; ($diff_bytes * 100) / $baseline_size" | bc)
|
||||
else
|
||||
diff_pct="N/A"
|
||||
fi
|
||||
|
||||
echo "DIFFERENT: $filename (baseline: ${baseline_size}B, current: ${current_size}B, diff: ${diff_bytes}B / ${diff_pct}%)"
|
||||
fi
|
||||
done
|
||||
|
||||
# Check for extra files in test results not in baseline
|
||||
echo ""
|
||||
echo "=== Checking for Extra Screenshots ==="
|
||||
for current in "$TEST_RESULTS_DIR"/*.png; do
|
||||
filename=$(basename "$current")
|
||||
baseline="$BASELINE_DIR/$filename"
|
||||
|
||||
if [ ! -f "$baseline" ]; then
|
||||
extra_current=$((extra_current + 1))
|
||||
current_size=$(stat -f%z "$current" 2>/dev/null || stat -c%s "$current")
|
||||
echo "EXTRA: $filename (${current_size}B, not in baseline)"
|
||||
fi
|
||||
done
|
||||
|
||||
# Summary
|
||||
echo ""
|
||||
echo "=== Summary ==="
|
||||
echo "Total baseline screenshots: $total"
|
||||
echo "Identical: $identical"
|
||||
echo "Different: $different"
|
||||
echo "Missing from test results: $missing_current"
|
||||
echo "Extra in test results: $extra_current"
|
||||
|
||||
# Exit with error if there are differences
|
||||
if [ "$different" -gt 0 ] || [ "$missing_current" -gt 0 ]; then
|
||||
echo ""
|
||||
echo "WARNING: There are differences between baseline and test results!"
|
||||
exit 1
|
||||
else
|
||||
echo ""
|
||||
echo "SUCCESS: All screenshots match baseline!"
|
||||
exit 0
|
||||
fi
|
||||
|
|
@ -227,6 +227,80 @@ cd tests/wasm-app && ./build-test-apps.sh
|
|||
|
||||
---
|
||||
|
||||
## Baseline Screenshots
|
||||
|
||||
The test suite captures screenshots during test runs for visual regression testing. These screenshots are compared against a baseline to detect unexpected visual changes.
|
||||
|
||||
### Directory Structure
|
||||
|
||||
```
|
||||
tests/
|
||||
├── baseline-screenshots/ # Known-good reference screenshots (121 files)
|
||||
├── test-results/ # Screenshots from latest test run
|
||||
└── wasm-app/
|
||||
└── e2e/ # Playwright test specs
|
||||
```
|
||||
|
||||
### Comparing Screenshots
|
||||
|
||||
Use the comparison script to check for visual regressions:
|
||||
|
||||
```bash
|
||||
./scripts/compare-screenshots.sh
|
||||
```
|
||||
|
||||
This will:
|
||||
- Compare all baseline screenshots with current test results
|
||||
- Report identical, different, and missing screenshots
|
||||
- Show file size differences (useful for detecting changes)
|
||||
|
||||
Example output:
|
||||
```
|
||||
=== Screenshot Comparison ===
|
||||
Baseline: /path/to/tests/baseline-screenshots
|
||||
Test Results: /path/to/tests/test-results
|
||||
|
||||
=== Comparing Baseline Screenshots ===
|
||||
DIFFERENT: dialogs-msgbox-info-open.png (baseline: 47468B, current: 47478B, diff: 10B / .02%)
|
||||
|
||||
=== Summary ===
|
||||
Total baseline screenshots: 121
|
||||
Identical: 54
|
||||
Different: 67
|
||||
Missing from test results: 0
|
||||
```
|
||||
|
||||
### Understanding Differences
|
||||
|
||||
Small byte differences (<2%) are typically caused by:
|
||||
- **Timestamps** in event logs (e.g., `[19:45:35]` vs `[18:49:35]`)
|
||||
- **PNG compression** variations between runs
|
||||
- **Anti-aliasing** differences
|
||||
|
||||
These are **not** visual regressions. Visually inspect screenshots if you see larger differences.
|
||||
|
||||
### Updating Baseline Screenshots
|
||||
|
||||
After verifying changes are intentional, update the baseline:
|
||||
|
||||
```bash
|
||||
# Copy current screenshots to baseline
|
||||
cp tests/test-results/*.png tests/baseline-screenshots/
|
||||
|
||||
# Or selectively update specific screenshots
|
||||
cp tests/test-results/dialog-*.png tests/baseline-screenshots/
|
||||
```
|
||||
|
||||
### Running Tests with Screenshots
|
||||
|
||||
```bash
|
||||
cd tests/wasm-app
|
||||
npm test # Run all tests (saves screenshots to test-results/)
|
||||
npx playwright test --ui # Interactive mode with screenshot preview
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Screenshots Reference
|
||||
|
||||
| Screenshot | Shows |
|
||||
|
|
|
|||
BIN
tests/baseline-screenshots/01-loading.png
Normal file
|
After Width: | Height: | Size: 5.9 KiB |
BIN
tests/baseline-screenshots/03-after-load.png
Normal file
|
After Width: | Height: | Size: 30 KiB |
BIN
tests/baseline-screenshots/04-controls-tab.png
Normal file
|
After Width: | Height: | Size: 34 KiB |
BIN
tests/baseline-screenshots/05-text-input-tab.png
Normal file
|
After Width: | Height: | Size: 32 KiB |
BIN
tests/baseline-screenshots/06-text-input-typed.png
Normal file
|
After Width: | Height: | Size: 34 KiB |
BIN
tests/baseline-screenshots/07-drawing-tab.png
Normal file
|
After Width: | Height: | Size: 36 KiB |
BIN
tests/baseline-screenshots/08-drawing-done.png
Normal file
|
After Width: | Height: | Size: 46 KiB |
BIN
tests/baseline-screenshots/09-lists-tab.png
Normal file
|
After Width: | Height: | Size: 47 KiB |
BIN
tests/baseline-screenshots/10-lists-clicked.png
Normal file
|
After Width: | Height: | Size: 44 KiB |
BIN
tests/baseline-screenshots/10a-choice-dropdown-open.png
Normal file
|
After Width: | Height: | Size: 44 KiB |
BIN
tests/baseline-screenshots/10b-choice-selected.png
Normal file
|
After Width: | Height: | Size: 43 KiB |
BIN
tests/baseline-screenshots/10c-choice-dropdown-reopen.png
Normal file
|
After Width: | Height: | Size: 43 KiB |
BIN
tests/baseline-screenshots/11-file-menu.png
Normal file
|
After Width: | Height: | Size: 80 KiB |
BIN
tests/baseline-screenshots/12-help-menu.png
Normal file
|
After Width: | Height: | Size: 80 KiB |
BIN
tests/baseline-screenshots/13-final.png
Normal file
|
After Width: | Height: | Size: 44 KiB |
BIN
tests/baseline-screenshots/14-opengl-tab.png
Normal file
|
After Width: | Height: | Size: 79 KiB |
BIN
tests/baseline-screenshots/15-opengl-tests.png
Normal file
|
After Width: | Height: | Size: 79 KiB |
BIN
tests/baseline-screenshots/aui-01-loaded.png
Normal file
|
After Width: | Height: | Size: 32 KiB |
BIN
tests/baseline-screenshots/aui-02-panels.png
Normal file
|
After Width: | Height: | Size: 32 KiB |
BIN
tests/baseline-screenshots/aui-03-close-clicked.png
Normal file
|
After Width: | Height: | Size: 32 KiB |
BIN
tests/baseline-screenshots/aui-04-dragged.png
Normal file
|
After Width: | Height: | Size: 32 KiB |
BIN
tests/baseline-screenshots/aui-05-multi-panel.png
Normal file
|
After Width: | Height: | Size: 32 KiB |
BIN
tests/baseline-screenshots/clipboard-01-loaded.png
Normal file
|
After Width: | Height: | Size: 35 KiB |
BIN
tests/baseline-screenshots/clipboard-02-copy-clicked.png
Normal file
|
After Width: | Height: | Size: 35 KiB |
BIN
tests/baseline-screenshots/clipboard-03-paste-clicked.png
Normal file
|
After Width: | Height: | Size: 35 KiB |
BIN
tests/baseline-screenshots/clipboard-04-check-clicked.png
Normal file
|
After Width: | Height: | Size: 42 KiB |
BIN
tests/baseline-screenshots/clipboard-05-clear-clicked.png
Normal file
|
After Width: | Height: | Size: 42 KiB |
BIN
tests/baseline-screenshots/clipboard-06-full-flow.png
Normal file
|
After Width: | Height: | Size: 48 KiB |
BIN
tests/baseline-screenshots/dialog-01-loaded.png
Normal file
|
After Width: | Height: | Size: 34 KiB |
BIN
tests/baseline-screenshots/dialog-02-info-clicked.png
Normal file
|
After Width: | Height: | Size: 42 KiB |
BIN
tests/baseline-screenshots/dialog-03-yesno-clicked.png
Normal file
|
After Width: | Height: | Size: 45 KiB |
BIN
tests/baseline-screenshots/dialog-04-error-clicked.png
Normal file
|
After Width: | Height: | Size: 42 KiB |
BIN
tests/baseline-screenshots/dialog-05-custom-clicked.png
Normal file
|
After Width: | Height: | Size: 34 KiB |
BIN
tests/baseline-screenshots/dialogs-00-initial.png
Normal file
|
After Width: | Height: | Size: 30 KiB |
BIN
tests/baseline-screenshots/dialogs-01-tab-selected.png
Normal file
|
After Width: | Height: | Size: 46 KiB |
BIN
tests/baseline-screenshots/dialogs-custom-closed.png
Normal file
|
After Width: | Height: | Size: 56 KiB |
BIN
tests/baseline-screenshots/dialogs-custom-open.png
Normal file
|
After Width: | Height: | Size: 56 KiB |
BIN
tests/baseline-screenshots/dialogs-full-flow.png
Normal file
|
After Width: | Height: | Size: 56 KiB |
BIN
tests/baseline-screenshots/dialogs-msgbox-error-closed.png
Normal file
|
After Width: | Height: | Size: 46 KiB |
BIN
tests/baseline-screenshots/dialogs-msgbox-error-open.png
Normal file
|
After Width: | Height: | Size: 46 KiB |
BIN
tests/baseline-screenshots/dialogs-msgbox-info-closed.png
Normal file
|
After Width: | Height: | Size: 46 KiB |
BIN
tests/baseline-screenshots/dialogs-msgbox-info-open.png
Normal file
|
After Width: | Height: | Size: 46 KiB |
BIN
tests/baseline-screenshots/dialogs-msgbox-yesno-closed.png
Normal file
|
After Width: | Height: | Size: 46 KiB |
BIN
tests/baseline-screenshots/dialogs-msgbox-yesno-open.png
Normal file
|
After Width: | Height: | Size: 46 KiB |
BIN
tests/baseline-screenshots/dialogs-timer-initial.png
Normal file
|
After Width: | Height: | Size: 46 KiB |
BIN
tests/baseline-screenshots/dialogs-timer-multiple.png
Normal file
|
After Width: | Height: | Size: 46 KiB |
BIN
tests/baseline-screenshots/dialogs-timer-running.png
Normal file
|
After Width: | Height: | Size: 46 KiB |
BIN
tests/baseline-screenshots/dialogs-timer-started.png
Normal file
|
After Width: | Height: | Size: 46 KiB |
BIN
tests/baseline-screenshots/dialogs-timer-stopped.png
Normal file
|
After Width: | Height: | Size: 46 KiB |
BIN
tests/baseline-screenshots/filedialog-01-loaded.png
Normal file
|
After Width: | Height: | Size: 24 KiB |
BIN
tests/baseline-screenshots/filedialog-02-open-clicked.png
Normal file
|
After Width: | Height: | Size: 24 KiB |
BIN
tests/baseline-screenshots/filedialog-03-save-clicked.png
Normal file
|
After Width: | Height: | Size: 24 KiB |
BIN
tests/baseline-screenshots/filedialog-04-multiple-clicked.png
Normal file
|
After Width: | Height: | Size: 24 KiB |
BIN
tests/baseline-screenshots/filedialog-05-all-buttons.png
Normal file
|
After Width: | Height: | Size: 24 KiB |
BIN
tests/baseline-screenshots/gl-01-opengl-tab.png
Normal file
|
After Width: | Height: | Size: 69 KiB |
BIN
tests/baseline-screenshots/gl-02-vertex-arrays-selected.png
Normal file
|
After Width: | Height: | Size: 75 KiB |
BIN
tests/baseline-screenshots/gl-03-after-wait.png
Normal file
|
After Width: | Height: | Size: 75 KiB |
BIN
tests/baseline-screenshots/gl-run-all-tests.png
Normal file
|
After Width: | Height: | Size: 69 KiB |
BIN
tests/baseline-screenshots/gl-test-0-immediate-mode.png
Normal file
|
After Width: | Height: | Size: 73 KiB |
BIN
tests/baseline-screenshots/gl-test-1-matrix-ops.png
Normal file
|
After Width: | Height: | Size: 72 KiB |
BIN
tests/baseline-screenshots/gl-test-2-vertex-arrays.png
Normal file
|
After Width: | Height: | Size: 82 KiB |
BIN
tests/baseline-screenshots/gl-test-3-state-mgmt.png
Normal file
|
After Width: | Height: | Size: 80 KiB |
BIN
tests/baseline-screenshots/gl-test-4-texture-coords.png
Normal file
|
After Width: | Height: | Size: 80 KiB |
BIN
tests/baseline-screenshots/glcanvas-01-on-opengl-tab.png
Normal file
|
After Width: | Height: | Size: 70 KiB |
|
After Width: | Height: | Size: 38 KiB |
BIN
tests/baseline-screenshots/glcanvas-03-on-drawing-tab.png
Normal file
|
After Width: | Height: | Size: 36 KiB |
BIN
tests/baseline-screenshots/glcanvas-04-on-lists-tab.png
Normal file
|
After Width: | Height: | Size: 43 KiB |
BIN
tests/baseline-screenshots/grid-tab-final.png
Normal file
|
After Width: | Height: | Size: 41 KiB |
BIN
tests/baseline-screenshots/layout-01-loaded.png
Normal file
|
After Width: | Height: | Size: 64 KiB |
BIN
tests/baseline-screenshots/layout-02-splitter.png
Normal file
|
After Width: | Height: | Size: 64 KiB |
BIN
tests/baseline-screenshots/layout-03-sash-dragged.png
Normal file
|
After Width: | Height: | Size: 64 KiB |
BIN
tests/baseline-screenshots/layout-04-scrolled-left.png
Normal file
|
After Width: | Height: | Size: 66 KiB |
BIN
tests/baseline-screenshots/layout-05-scrolled-right.png
Normal file
|
After Width: | Height: | Size: 63 KiB |
BIN
tests/baseline-screenshots/layout-06-combined.png
Normal file
|
After Width: | Height: | Size: 64 KiB |
BIN
tests/baseline-screenshots/menu-01-loaded.png
Normal file
|
After Width: | Height: | Size: 28 KiB |
BIN
tests/baseline-screenshots/menu-02-menubar.png
Normal file
|
After Width: | Height: | Size: 28 KiB |
BIN
tests/baseline-screenshots/menu-03-file-clicked.png
Normal file
|
After Width: | Height: | Size: 28 KiB |
BIN
tests/baseline-screenshots/menu-04-edit-clicked.png
Normal file
|
After Width: | Height: | Size: 31 KiB |
BIN
tests/baseline-screenshots/menu-05-all-menus.png
Normal file
|
After Width: | Height: | Size: 29 KiB |
BIN
tests/baseline-screenshots/opengl-after-tests.png
Normal file
|
After Width: | Height: | Size: 70 KiB |
BIN
tests/baseline-screenshots/opengl-before-debug.png
Normal file
|
After Width: | Height: | Size: 70 KiB |
BIN
tests/baseline-screenshots/opengl-render.png
Normal file
|
After Width: | Height: | Size: 70 KiB |
BIN
tests/baseline-screenshots/opengl-tab-initial.png
Normal file
|
After Width: | Height: | Size: 70 KiB |
BIN
tests/baseline-screenshots/searchctrl-01-visible.png
Normal file
|
After Width: | Height: | Size: 41 KiB |
BIN
tests/baseline-screenshots/searchctrl-02-typing.png
Normal file
|
After Width: | Height: | Size: 41 KiB |
BIN
tests/baseline-screenshots/searchctrl-03-after-enter.png
Normal file
|
After Width: | Height: | Size: 41 KiB |
BIN
tests/baseline-screenshots/spinctrl-01-visible.png
Normal file
|
After Width: | Height: | Size: 41 KiB |
BIN
tests/baseline-screenshots/spinctrl-02-after-click.png
Normal file
|
After Width: | Height: | Size: 41 KiB |
BIN
tests/baseline-screenshots/tabswitch-01-controls.png
Normal file
|
After Width: | Height: | Size: 30 KiB |
BIN
tests/baseline-screenshots/tabswitch-02-opengl.png
Normal file
|
After Width: | Height: | Size: 70 KiB |
BIN
tests/baseline-screenshots/tabswitch-03-lists.png
Normal file
|
After Width: | Height: | Size: 37 KiB |
BIN
tests/baseline-screenshots/tabswitch-04-opengl.png
Normal file
|
After Width: | Height: | Size: 76 KiB |
BIN
tests/baseline-screenshots/tabswitch-05-drawing.png
Normal file
|
After Width: | Height: | Size: 39 KiB |
BIN
tests/baseline-screenshots/tabswitch-06-controls.png
Normal file
|
After Width: | Height: | Size: 45 KiB |
BIN
tests/baseline-screenshots/timer-01-loaded.png
Normal file
|
After Width: | Height: | Size: 35 KiB |
BIN
tests/baseline-screenshots/timer-02-started.png
Normal file
|
After Width: | Height: | Size: 35 KiB |
BIN
tests/baseline-screenshots/timer-05-fast-started.png
Normal file
|
After Width: | Height: | Size: 35 KiB |
BIN
tests/baseline-screenshots/timer-08-reset.png
Normal file
|
After Width: | Height: | Size: 35 KiB |