Add baseline screenshots and visual regression testing

Add infrastructure for visual regression testing of wxWidgets WASM rendering:

- Add 121 baseline screenshots capturing all UI states tested by Playwright
- Add scripts/compare-screenshots.sh to compare test results vs baseline
- Update tests/WHATWORKS.md with documentation on screenshot comparison
- Update wxwidgets submodule with dialog refactoring

The baseline screenshots cover:
- Main app tabs (Controls, Text Input, Drawing, Lists, OpenGL, Grid, Dialogs)
- Standalone test apps (menu, toolbar, layout, aui, clipboard, filedialog, etc.)
- Dialog/MessageBox rendering (info, yes/no, error, custom dialogs)
- OpenGL rendering tests (immediate mode, vertex arrays, matrix ops)
- UI interactions (dropdowns, tab switching, z-ordering)

Use ./scripts/compare-screenshots.sh to check for visual regressions.
Small byte differences (<2%) are typically timestamps, not visual changes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Viktor Vaczi 2025-11-29 19:55:12 +01:00
commit 961455ef54
124 changed files with 182 additions and 1 deletions

107
scripts/compare-screenshots.sh Executable file
View file

@ -0,0 +1,107 @@
#!/bin/bash
# Compare test screenshots with baseline screenshots
# This script compares all PNG files in baseline-screenshots with test-results
set -e
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
BASELINE_DIR="$PROJECT_ROOT/tests/baseline-screenshots"
TEST_RESULTS_DIR="$PROJECT_ROOT/tests/test-results"
# Check if directories exist
if [ ! -d "$BASELINE_DIR" ]; then
echo "ERROR: Baseline directory not found: $BASELINE_DIR"
exit 1
fi
if [ ! -d "$TEST_RESULTS_DIR" ]; then
echo "ERROR: Test results directory not found: $TEST_RESULTS_DIR"
exit 1
fi
echo "=== Screenshot Comparison ==="
echo "Baseline: $BASELINE_DIR"
echo "Test Results: $TEST_RESULTS_DIR"
echo ""
# Counters
total=0
identical=0
different=0
missing_current=0
extra_current=0
# Compare all baseline screenshots
echo "=== Comparing Baseline Screenshots ==="
for baseline in "$BASELINE_DIR"/*.png; do
filename=$(basename "$baseline")
current="$TEST_RESULTS_DIR/$filename"
total=$((total + 1))
if [ ! -f "$current" ]; then
echo "MISSING: $filename (not in test results)"
missing_current=$((missing_current + 1))
continue
fi
# Compare using cmp (byte-by-byte comparison)
if cmp -s "$baseline" "$current"; then
identical=$((identical + 1))
# Only show identical files if verbose
if [ "$1" = "-v" ] || [ "$1" = "--verbose" ]; then
echo "IDENTICAL: $filename"
fi
else
different=$((different + 1))
# Get file sizes
baseline_size=$(stat -f%z "$baseline" 2>/dev/null || stat -c%s "$baseline")
current_size=$(stat -f%z "$current" 2>/dev/null || stat -c%s "$current")
diff_bytes=$((current_size - baseline_size))
# Calculate percent difference
if [ "$baseline_size" -gt 0 ]; then
diff_pct=$(echo "scale=2; ($diff_bytes * 100) / $baseline_size" | bc)
else
diff_pct="N/A"
fi
echo "DIFFERENT: $filename (baseline: ${baseline_size}B, current: ${current_size}B, diff: ${diff_bytes}B / ${diff_pct}%)"
fi
done
# Check for extra files in test results not in baseline
echo ""
echo "=== Checking for Extra Screenshots ==="
for current in "$TEST_RESULTS_DIR"/*.png; do
filename=$(basename "$current")
baseline="$BASELINE_DIR/$filename"
if [ ! -f "$baseline" ]; then
extra_current=$((extra_current + 1))
current_size=$(stat -f%z "$current" 2>/dev/null || stat -c%s "$current")
echo "EXTRA: $filename (${current_size}B, not in baseline)"
fi
done
# Summary
echo ""
echo "=== Summary ==="
echo "Total baseline screenshots: $total"
echo "Identical: $identical"
echo "Different: $different"
echo "Missing from test results: $missing_current"
echo "Extra in test results: $extra_current"
# Exit with error if there are differences
if [ "$different" -gt 0 ] || [ "$missing_current" -gt 0 ]; then
echo ""
echo "WARNING: There are differences between baseline and test results!"
exit 1
else
echo ""
echo "SUCCESS: All screenshots match baseline!"
exit 0
fi

View file

@ -227,6 +227,80 @@ cd tests/wasm-app && ./build-test-apps.sh
---
## Baseline Screenshots
The test suite captures screenshots during test runs for visual regression testing. These screenshots are compared against a baseline to detect unexpected visual changes.
### Directory Structure
```
tests/
├── baseline-screenshots/ # Known-good reference screenshots (121 files)
├── test-results/ # Screenshots from latest test run
└── wasm-app/
└── e2e/ # Playwright test specs
```
### Comparing Screenshots
Use the comparison script to check for visual regressions:
```bash
./scripts/compare-screenshots.sh
```
This will:
- Compare all baseline screenshots with current test results
- Report identical, different, and missing screenshots
- Show file size differences (useful for detecting changes)
Example output:
```
=== Screenshot Comparison ===
Baseline: /path/to/tests/baseline-screenshots
Test Results: /path/to/tests/test-results
=== Comparing Baseline Screenshots ===
DIFFERENT: dialogs-msgbox-info-open.png (baseline: 47468B, current: 47478B, diff: 10B / .02%)
=== Summary ===
Total baseline screenshots: 121
Identical: 54
Different: 67
Missing from test results: 0
```
### Understanding Differences
Small byte differences (<2%) are typically caused by:
- **Timestamps** in event logs (e.g., `[19:45:35]` vs `[18:49:35]`)
- **PNG compression** variations between runs
- **Anti-aliasing** differences
These are **not** visual regressions. Visually inspect screenshots if you see larger differences.
### Updating Baseline Screenshots
After verifying changes are intentional, update the baseline:
```bash
# Copy current screenshots to baseline
cp tests/test-results/*.png tests/baseline-screenshots/
# Or selectively update specific screenshots
cp tests/test-results/dialog-*.png tests/baseline-screenshots/
```
### Running Tests with Screenshots
```bash
cd tests/wasm-app
npm test # Run all tests (saves screenshots to test-results/)
npx playwright test --ui # Interactive mode with screenshot preview
```
---
## Screenshots Reference
| Screenshot | Shows |

Binary file not shown.

After

Width:  |  Height:  |  Size: 5.9 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 30 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 34 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 34 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 36 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 46 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 47 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 44 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 44 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 43 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 43 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 80 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 80 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 44 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 79 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 79 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 35 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 35 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 35 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 42 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 42 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 48 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 34 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 42 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 45 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 42 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 34 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 30 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 46 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 56 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 56 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 56 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 46 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 46 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 46 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 46 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 46 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 46 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 46 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 46 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 46 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 46 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 46 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 24 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 24 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 24 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 24 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 24 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 69 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 75 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 75 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 69 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 73 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 72 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 82 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 80 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 80 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 70 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 38 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 36 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 43 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 41 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 64 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 64 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 64 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 66 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 63 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 64 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 28 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 28 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 28 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 31 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 29 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 70 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 70 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 70 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 70 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 41 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 41 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 41 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 41 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 41 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 30 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 70 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 37 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 76 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 39 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 45 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 35 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 35 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 35 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 35 KiB

Some files were not shown because too many files have changed in this diff Show more