[UX-Automation] Setup Infrastructure for AI-Driven UX Audits

[BeSovereign]

Summary

Implement the technical foundation to allow an LLM-powered agent to control a browser and perform semantic UX audits. This infrastructure should wrap a headless browser with an AI automation layer (e.g., Stagehand or Browser-use).

Technical Context

• Base Directory: tests/ux-audit/ in the freeshard repository.
• Recommended Stack: TypeScript with Stagehand (Playwright-based LLM automation) or Python with browser-use.
• Goal: Create a reusable runner script that accepts a semantic instruction and returns a structured UX report.

Implementation Details for the Agent

1. Environment Setup:
   • Initialize tests/ux-audit/package.json with dependencies for AI-driven automation.
   • Configure access to an LLM provider (OpenAI/Anthropic) via environment variables (e.g., OPENAI_API_KEY).
2. The Audit Runner:
   • Create tests/ux-audit/runner.ts (or .py).
   • The runner should:
     • Launch a browser instance (Chromium preferred).
     • Pass a high-level goal to the AI agent.
     • Capture screenshots/logs for each step of the AI's journey.
     • Export the final result as reports/ux-audit-{{timestamp}}.md.
3. Justfile Integration:
   • Add a command just audit-ui to trigger the default UX audit flow.

Open Technical Points / Decisions

• Browser Choice: The agent should default to Chromium, but allow overriding via AUDIT_BROWSER=firefox.
• Artifact Storage: Decide on a retention policy for screenshots taken by the AI during the audit.

Acceptance Criteria

• A script exists that can semantically navigate to the landing page.
• The script can handle dynamic redirects (e.g., to the login page).
• The audit report is generated as a Markdown file.

[BeSovereign]

Resolution: Infrastructure Specifications for AI-Driven UX Audits

Following a technical review of the open points, we have defined the following implementation constraints for the UX audit infrastructure:

1. Framework & Stack

• Choice: Stagehand (TypeScript).
• Rationale: Built on Playwright, ensuring full consistency with the core E2E testing stack ([Testing] Setup Playwright E2E Framework for UI and Integration Validation #96) and allowing shared helper utilities.

2. AI Model Strategy (The "Brain")

The infrastructure must support an interchangeable model strategy via environment variables. The implementing agent should prioritize these two candidates:

• Option A: Claude 3.5 Sonnet
  • Strength: Exceptional at identifying visual nuances (spacing, contrast, typography) and describing layout inconsistencies.
  • Implementation: Native support in Stagehand via Anthropic SDK.
• Option B: Gemini 1.5 Pro
  • Strength: Massive context window allows referencing the full design-guide.pdf and GEMINI.md simultaneously during the audit. Superior native multimodal capabilities.
  • Implementation: Requires an adapter or proxy (e.g., via LiteLLM or custom Google Generative AI integration) as Stagehand is primarily optimized for OpenAI/Anthropic.

3. Artifact & Screenshot Management

• Strategy: Use Playwright Trace-Files (.zip) as the primary artifact.
• Rationale: Traces capture the entire execution (DOM, network, screenshots) and can be replayed for debugging. In CI (GitHub Actions), retain traces for 30 days.

4. Browser & Viewport

• Default: Chromium (highest reliability for AI element detection).
• Expansion: The runner must support a --mobile flag via the Justfile to simulate a mobile viewport (e.g., iPhone 13) to audit responsiveness.

5. Justfile Integration

Add the following commands to the project root justfile:

• just audit-ui: Runs the standard desktop audit.
• just audit-ui-mobile: Runs the audit with a mobile viewport simulation.

This specification closes the open technical points. Proceed with implementation following these guidelines.

[ClaydeCode]

Hey @BeSovereign, Max here. I actually drove this review myself, no mindless clanker running loose :D

Good base in both this and #96. I like the auth-state caching idea and the clean layout you laid out. But I wanna refocus the scope before we build, so we don't bake in stuff that bites us later.

Let's give it its own repo, fully black box.
I don't think the UX framework should live in freeshard (backend, Python) or in the frontend repo. New dedicated repo, pulls our published images, drives the UI from the outside like a real user. Keeps both product repos clean of a Node/Playwright toolchain, and it tests the actual shipped thing instead of internals. The test stack can also churn on its own pace without messing with product CI.

Start lean: fresh shard, trial from the landing page.
First version is dead simple. The agent gets dropped on the public landing page and left alone. It signs up for a trial shard, goes through onboarding, pokes around. Nice side effect: this also tells us if the landing page → signup flow is clear enough for someone with no guidance. That's real UX signal. The self-hosting / docker-compose.yml boot path can come later as a second mode.

A few things I wanna cut, and why:

• No auth.setup.ts / storage-state caching. That's an optimization for hundreds of fast deterministic tests. Our audit is a handful of long sessions, so the agent can just log in through the UI like a user. No payoff here.
• No shard-api.ts helper. That's only for seeding state out-of-band. Black box + fresh shard means there's nothing to seed. If we ever audit a populated UI we can revisit, but not now.
• Defer the AI audit layer itself. The design guide exists but it's not applied to the frontend yet, so there's nothing to actually audit against right now. Once it's live this turns into a real conformance check (UI vs design guide) instead of fuzzy "semantic UX". I think that's worth waiting for. It also gives us an actual success criterion.
• Small thing: the model picks (Claude 3.5 Sonnet / Gemini 1.5 Pro) and the GEMINI.md / design-guide.pdf refs are a bit stale or don't exist yet. We'll just re-pick current models when we get there.

So the net scope: new repo, agent does everything through the browser, starts on the landing page with a trial shard, no helpers, no seeding. Self-hosting and the AI audit layer come once we need them.

Keeps it minimal and unblocks us right away. What do you think?