Behavioral diffing for the AI era. A self-hosted GitHub Action that compares frontend runtime behavior — not code — between a base branch and a pull request.
Instead of reading AI-generated diffs line-by-line, reviewers see what actually changed: performance metrics, DOM structure, accessibility violations, screenshots, interaction videos, console errors, and network calls.
- Why
- How It Works
- Metrics Collected
- Diffing Layer
- PR Report
- Configuration
- Tech Stack
- Repo Structure
- MVP Deliverables
- Design Philosophy
- Non-Goals (MVP)
- Stretch Goals
AI tools generate large PRs that are tedious to review manually. Most reviewers don't care about the code — they care about what it does:
- Did performance regress?
- Did the UI break?
- Are there new console errors?
- Did the DOM structure change unexpectedly?
- Did accessibility get worse?
BehaviorDiff answers these questions automatically and posts a summary directly on the PR.
When a PR is opened or updated, BehaviorDiff runs both branches in parallel on the GitHub Actions runner:
- Checkout — Clone both base and PR branches into separate directories
- Install — Run install commands for both (
npm ciby default) - Build — Run build commands for both (
npm run buildby default) - Serve — Start both servers on different ports (e.g., base on
:3000, PR on:3001)- Poll each server's health endpoint until it responds 200 (configurable timeout)
- Capture — For each configured page and viewport size, launch Playwright against both servers:
- Take full-page screenshots
- Record video of scripted interactions
- Serialize the DOM tree
- Collect console logs, warnings, and errors
- Record network requests (count, bytes, timing, endpoints)
- Measure performance metrics (FCP, LCP, CLS, TTI, DOMContentLoaded, navigation timing)
- Run
axe-coreaccessibility audit - Measure JS bundle size
- Diff — Compare all captured data between base and PR
- Report — Generate a markdown summary and post it as a PR comment, with artifacts uploaded to GitHub Actions Artifacts
Both branches run simultaneously on the same runner. Each gets its own working directory and port. This is straightforward on GitHub Actions — the runner has enough resources for two Node servers, and Playwright can target both by URL. The config must define two distinct ports (or the tool auto-assigns them).
Before capturing, the runner polls each server (HTTP GET to root or a configured health path) with a timeout (default: 60s). If a server fails to start, the run fails with a clear error indicating which branch didn't start.
- First Contentful Paint (FCP)
- Largest Contentful Paint (LCP)
- Cumulative Layout Shift (CLS)
- Time to Interactive (TTI)
- DOMContentLoaded
- Navigation timing
- JS bundle size
- JS execution time
- Errors
- Warnings
- Logs
- Request count
- Total bytes transferred
- Slowest requests
- New/removed endpoints
- Full-page screenshots (per viewport)
- Visual diff images (pixelmatch)
- Interaction videos (Playwright video recording)
- Serialized DOM tree snapshot
- Structural diff (added/removed/changed elements)
- Attribute changes (class, style, data-, aria-)
axe-coreviolation count (base vs PR)- New violations introduced
- Violations resolved
- Severity breakdown (critical, serious, moderate, minor)
For each metric category, compute:
- Numeric deltas — e.g., LCP +420ms, bundle size -8%
- Regression detection — flag metrics that exceed configured thresholds
- New errors — console errors or a11y violations not present in base
- Resolved issues — warnings or violations present in base but gone in PR
- Visual diffs — pixel-level image comparison via
pixelmatch - DOM diffs — structural comparison of serialized DOM trees, reporting added/removed/changed nodes and attributes
- Network diffs — new endpoints called, removed endpoints, request count changes
After diffing, a markdown comment is posted to the PR:
## BehaviorDiff Report
### Summary
⚠️ LCP +420ms (threshold: 200ms)
⚠️ 2 new console errors
⚠️ 1 new a11y violation (serious)
✅ JS bundle size -8%
✅ 3 a11y violations resolved
### Pages Tested
| Page | Viewports | Visual Changes | DOM Changes |
|------|-----------|---------------|-------------|
| /login | desktop, mobile | Yes | No |
| /dashboard | desktop, mobile | Yes | Yes |
### Network Changes
- +3 requests to `/api/profile`
- Removed: `/api/legacy-auth`
### Accessibility
- New: "Images must have alternate text" (serious) on /dashboard
- Resolved: "Color contrast insufficient" on /login
### Artifacts
- [Screenshots & Diffs](link-to-artifacts)
- [Interaction Videos](link-to-artifacts)
- [Full Report JSON](link-to-artifacts)The report includes:
- Emoji indicators for quick scanning
- Tables for multi-page/multi-viewport results
- Links to uploaded artifacts (screenshots, videos, full JSON data)
Users define a behaviordiff.config.yml in their repo root:
# Pages to test
pages:
- url: /
name: home
- url: /login
name: login
- url: /dashboard
name: dashboard
# Viewports to capture
viewports:
- name: desktop
width: 1280
height: 720
- name: mobile
width: 375
height: 812
# Build commands
commands:
install: npm ci
build: npm run build
start: npm run start
# Port configuration
ports:
base: 3000
pr: 3001
# Server readiness
healthCheck:
path: /
timeout: 60 # seconds
# Regression thresholds
thresholds:
lcp: 200 # ms
cls: 0.05
fcp: 100 # ms
bundleSize: 5 # percent increase
# Scripted interactions (optional)
# Path to a Playwright script that runs against each page
interactions: ./behaviordiff.interactions.tsUsers can optionally provide a Playwright interaction script. This file exports functions named after each page:
// behaviordiff.interactions.ts
import { Page } from 'playwright';
export async function login(page: Page) {
await page.fill('#email', 'test@example.com');
await page.fill('#password', 'password');
await page.click('button[type="submit"]');
await page.waitForURL('/dashboard');
}
export async function dashboard(page: Page) {
await page.click('[data-tab="settings"]');
await page.waitForSelector('.settings-panel');
}Videos are recorded for all interaction runs automatically.
- Node.js + TypeScript — Core implementation
- GitHub Actions — CI/CD integration (Action)
- Playwright — Browser automation, screenshots, video recording
- pixelmatch — Visual diff image comparison
- axe-core — Accessibility auditing
- jsdom / custom serializer — DOM tree serialization and diffing
behaviordiff/
├── action.yml # GitHub Action definition (inputs, outputs, branding)
├── src/
│ ├── index.ts # Action entry point / orchestrator
│ ├── config.ts # YAML config parsing with defaults
│ ├── types.ts # Shared TypeScript interfaces
│ ├── runner/ # Build, serve, and capture logic
│ │ ├── build.ts # Install + build orchestration
│ │ ├── serve.ts # Server management + health checks
│ │ └── capture.ts # Playwright capture (screenshots, video, DOM, metrics)
│ ├── diff/ # Comparison logic
│ │ ├── index.ts # Diff orchestrator
│ │ ├── metrics.ts # Numeric diffing + threshold checks
│ │ ├── visual.ts # Screenshot diffing (DOM-aware overlays)
│ │ ├── dom.ts # DOM tree diffing (LCS matching)
│ │ ├── network.ts # Network request diffing
│ │ ├── console.ts # Console log diffing
│ │ └── a11y.ts # Accessibility violation diffing
│ ├── intelligence/ # Opinionated analysis layer
│ │ ├── index.ts # Intelligence orchestrator
│ │ ├── signals.ts # Signal extraction from diff data
│ │ ├── causality.ts # Causal relationship detection
│ │ ├── risk.ts # Risk assessment and blocking logic
│ │ ├── categorization.ts # User-facing vs safe vs unknown
│ │ ├── noise-reduction.ts# Flaky/insignificant change filtering
│ │ └── recommendation.ts # Actionable merge recommendation
│ └── report/ # Markdown report generation
│ ├── markdown.ts # Standard markdown report
│ ├── opinionated-markdown.ts # Intelligence-powered report
│ ├── comment.ts # PR comment posting
│ └── artifacts.ts # GitHub Actions artifact upload
├── tests/ # Vitest test suite
├── fixtures/ # Minimal test fixtures (inline HTML apps)
├── scripts/ # Developer utilities (local-run.ts)
└── .github/workflows/ # Sample GitHub Actions workflow
Note: No sample Next.js app in the repo. Test fixtures are minimal inline HTML apps to keep the repo lean. Real-world testing should use a separate repo.
- Working GitHub Action installable with a single workflow YAML
- Parallel branch execution (build + serve both branches simultaneously)
- Playwright capture: screenshots, video, DOM, console, network, performance, a11y
- Diffing: visual, DOM, metrics, network, console, accessibility
- Markdown PR comment with summary, tables, and artifact links
- Multi-viewport support (desktop + mobile at minimum)
-
behaviordiff.config.ymlconfiguration - Optional Playwright interaction scripts
- README with setup instructions
- Sample GitHub Actions workflow file
- Architecture diagram (ASCII or markdown)
- Zero SaaS backend — Everything runs on the GitHub Actions runner
- Transparency — All data is JSON, all comparisons are deterministic
- Reproducibility — Same inputs produce same outputs
- Fast feedback — Parallel execution, no heavyweight tools (no Lighthouse)
- Easy install — One workflow YAML + one config file
- Behavioral over textual — Compare what the app does, not what the code says
- Backend API testing beyond browser-observed network calls
- Web dashboard or hosted UI
- Cloud accounts or proprietary services
- Risk scoring (defer to post-MVP when real-world calibration data exists)
- Lighthouse reports (overlap with direct Playwright metrics)
- Flaky detection via repeated runs
node_modulescaching between runs- Baseline storage (persist metrics from main branch for trend tracking)
- Video side-by-side comparison view (HTML artifact)
- Component-level isolation (test individual components, not just pages)
- Custom reporter plugins