test: validate frontend-preview phase with real Playwright integration on a React/Vue project

[VECTORG99]

Summary

ACF's frontend-preview phase (5) is designed but untested. It claims to launch a dev server, navigate to affected routes, capture before/after screenshots, and produce a visual diff. None of this has been tested with real Playwright integration. We need to validate the frontend-preview phase on a real React/Vue/Svelte project.

Context

• Frontend-preview: skills/05-frontend-preview/SKILL.md
• Roadmap: docs/ROADMAP.md — Phase 2 (Beta) item
• Playwright MCP server is available (listed in CLAUDE.md)
• The skill references: Playwright, dev server, route detection from diff, screenshot capture

Motivation

Frontend-preview is ACF's most ambitious phase — it bridges code and visual output. In 30 years of building developer tools, visual diff tools have always been "6 months away" because they're genuinely hard:

1. Route detection — parsing a diff to know which routes changed is non-trivial
2. Dev server lifecycle — starting, reusing, and not killing the user's server
3. Screenshot determinism — animations, lazy loading, and async data make screenshots flaky
4. Visual diff — pixel comparison is noisy; semantic diff is hard
5. Attachment — GitHub doesn't render images in PR bodies well; need to upload as assets

ACF's frontend-preview SKILL.md describes the ideal flow but doesn't solve these challenges. It says "navigate to route, screenshot before/after" — but how? With Playwright MCP? With a standalone script? What if the dev server is already running? What if the route requires auth?

We need to test this on a real project and document what works, what doesn't, and what needs to change in the skill.

Affected Files

• skills/05-frontend-preview/SKILL.md — update based on test findings
• .devin/skills/05-frontend-preview/SKILL.md — mirror
• docs/ARCHITECTURE.md — update frontend-preview section
• tests/frontend-preview/ (new) — test fixtures and scripts

Acceptance Criteria

• Test project selected (a real React/Vue/Svelte project with a dev server)
• Frontend-preview tested end-to-end:
  • Dev server starts (or reuses existing)
  • Route detection from a sample diff works (at least 1 route identified)
  • Screenshot captured before the change
  • Change applied
  • Screenshot captured after the change
  • Visual diff produced (even a simple side-by-side is acceptable for v1)
• Findings documented in a comment on this issue:
  • Did route detection work? How accurate?
  • Did the dev server lifecycle work? Issues encountered?
  • Were screenshots deterministic? Flakiness?
  • How was the visual diff produced? Tool used?
  • Were screenshots attached to the PR? How?
  • What needs to change in the SKILL.md?
• skills/05-frontend-preview/SKILL.md updated based on findings:
  • Concrete instructions (not just "navigate to route")
  • Playwright MCP integration steps (if used)
  • Dev server detection logic (check if port is in use)
  • Screenshot stability techniques (wait for network idle, disable animations)
  • Attachment method (GitHub asset upload vs base64 in PR body)
• Mirrors kept in sync

Validation

• A PR with a frontend change has a visual diff attached
• The frontend-preview SKILL.md has concrete, tested instructions (not aspirational)

Complexity

Complex — visual diff tools are genuinely hard. This may reveal that frontend-preview needs to be split into multiple sub-skills or moved to a later phase.

Notes

• Consider using the Playwright MCP server (available in the environment) for the test.
• Consider using Playwright's screenshot comparison for the visual diff.
• If screenshot determinism is too hard, consider DOM diffing instead (more stable, less visual).
• This is a Phase 2 (Beta) issue — it validates an ambitious feature.
• If frontend-preview proves too hard for v1, mark it as experimental and focus on the other 7 phases.

[VECTORG99]

Assessment: needs-human

This issue requires testing frontend-preview on a real React/Vue/Svelte project with a dev server — that's an explicit acceptance criterion ("Test project selected (a real React/Vue/Svelte project with a dev server)").

What was done (automated)

• Reviewed skills/05-frontend-preview/SKILL.md — the skill design is sound:
  • Route detection from diff (file → route mapping)
  • Dev server lifecycle (detect running server on common ports)
  • Stash safety (only stash/pop when clean, surface conflicts)
  • Lightweight degradation mode (list routes + describe changes when Playwright unavailable)
  • Screenshot attachment to PR body
• The skill has a clear degradation path and doesn't block PRs

What needs human verification

• Selecting a real frontend project (React/Vue/Svelte with a dev server)
• Running the full flow: dev server start → route detection → before screenshot → change applied → after screenshot → visual diff
• Testing Playwright MCP integration
• Testing screenshot determinism (animations, lazy loading, async data)
• Testing GitHub asset upload vs base64 in PR body
• Documenting findings and updating the SKILL.md with concrete instructions

Why this can't be automated here

The test requires:

1. A real frontend project (not available in this environment)
2. Playwright running against a live dev server
3. Visual inspection of screenshot quality and diff accuracy

These are inherently human-in-the-loop tasks. The skill's design is complete; it needs empirical validation in a real frontend project.

Recommendation

Keep this as needs-human. When a frontend project is available (e.g., artemisa, homedir frontend), run the test and update the SKILL.md based on findings. The skill's lightweight mode is a safe fallback until then.

Generated with Devin