Visual regression testing

The screenshot matrix (#846) is the regression net that keeps four themes × five layouts from rotting: 46 Playwright tests producing 74 checked-in baselines — key screens × 4 public themes × 2 widths, the five blessed theme×layout pairing rows, and a per-theme edge-state pass.

Running

dev-up                    # the suite drives the RUNNING dev server (localhost:6650)
pnpm test:visual          # compare against checked-in baselines
pnpm test:visual:update   # refresh baselines after an intentional visual change

Override the target with VISUAL_BASE_URL=…. The runner is @playwright/test (pinned to the same 1.61.x as the playwright dep; shares its browsers), config in playwright.config.ts, specs in e2e/visual/.

What's in the matrix

• matrix.spec.ts — home, stepper first question, results (both tabs, the stepper driven deterministically by always picking the first option), compare picker, and compare matrix (?systems= ids resolved from the live catalog, longest name included) — each × nanawall/graphite/linen/pressroom × 768/1920, plus the blessed pairings (NanaWall×Editorial, Graphite×Standard, Linen×Spotlight, Pressroom×Magazine, NanaWall×Spread) at 1920: home + the pairing's signature screen. Magazine and Spread are LAYOUT-axis variants (#979) exercised here under a fitting palette.
• edges.spec.ts — per theme: sparse-media compare (zero-image matrix headers), longest-name matrix (with a functional H4 assertion on the NW Acoustical 645 column header), 1-of-N partial results, and the most restrictive deterministic answer path. The no-match dead end is currently unreachable with the live catalog (every deterministic path still recommends); the restrictive-path baseline stands in — see the spec header.

Determinism

Theme and layout are seeded directly into app_settings per describe-block (e2e/visual/harness.ts → wrangler d1 execute --local) — the same rows hooks.server.ts resolves, so the capture matches production rendering with no admin clicking and no test-only override in app code. The suite runs one worker (that seeding is global server state), disables animations, waits for document.fonts.ready, and tolerates a 2% pixel diff (photography/antialias wobble; layout regressions are far larger).

Baselines & diffs

Baselines live in e2e/visual/__screenshots__/<platform>/ and are platform- keyed — the checked-in set is darwin (the canonical dev machine). A diff failure prints the expected/actual/diff image paths per test; the images land in .bob-artifacts/visual-results/. Update intentionally changed baselines with pnpm test:visual:update and commit the PNGs with the change that caused them.

In CI (.github/workflows/ci.yml), the visual job is manual dispatch only: it needs a seeded dev environment, and a bare runner's D1 lacks the Drupal-synced catalog. When dispatched it boots the dev server, runs the matrix, prints per-test diff paths in the job log, and uploads .bob-artifacts/visual-results as a visual-diffs artifact on failure. The first linux run needs linux baselines (--update-snapshots on the runner, then commit) — until then treat the job as an on-demand smoke, not a gate.

Adding a screen

Add a test() to the right describe block (it inherits the seeded presentation), navigate, and call the shared shoot()/shootWidths() helper with a stable baseline name. Then pnpm test:visual:update and commit the new PNGs. Keep names theme-prefixed (<theme>-<screen>[-<width>]) so the matrix stays greppable.