WhyMe QA

A reusable QA platform for code-generating MCP servers. Evaluates generated code with executable proof: if it compiles and passes tests, it's objectively usable.

Quick Start

# Install
pnpm install
pnpm build

# Configure environment (copy and edit with your values)
cp .env.example .env

# List available scenarios
npx whyme-qa list -d scenarios/

# Run a scenario against an MCP server
npx whyme-qa run -s scenarios/counter.yaml

# Run with workspace preserved for debugging
npx whyme-qa run -s scenarios/counter.yaml --keep-workspace

How It Works

1. Scenario YAML defines the MCP server, tool calls, expected output, and verification gates
2. Runner connects to the MCP server, executes tool calls, materializes generated code
3. Gate scripts verify the code: compile, lint, test, WASM build, activation check
4. Agent loop (optional) — if gates fail, an LLM reviews errors and fixes the code, re-running gates up to N iterations
5. Reporter generates scores, JUnit XML, and Markdown reports with iteration history

Architecture

whyme-qa/
├── packages/
│   ├── runner/      # CLI + scenario orchestrator + agent loop
│   ├── adapters/    # MCP client + gate execution + LLM client
│   ├── reporter/    # Scoring, JUnit, Markdown reports
│   └── schemas/     # TypeScript type definitions
├── gates/
│   └── stylus/      # Shell scripts for Stylus verification
├── scenarios/       # YAML scenario definitions
├── .gitlab/
│   └── duo/flows/   # GitLab Duo flow definitions
└── .gitlab-ci.yml   # CI pipeline configuration

Data Flow

MCP Server → File Extraction → Workspace → Gates → [Agent Loop] → Score → Report
                                                        │
                                              LLM (via OpenRouter)
                                              Read files + gate errors
                                              Return fixed files
                                              Re-run all gates
                                              Repeat until pass or max iterations

Verification Gates (Stylus)

Gate	Command	Points
Format	cargo fmt --check	5
Clippy	cargo clippy -- -D warnings	10
Compile	cargo build --release	10
Unit Tests	cargo test --all	30
WASM Build	cargo build --target wasm32-unknown-unknown	10
Stylus Check	cargo stylus check	20
Security	cargo audit	15

MCP Response Formats

The runner's extractFilesFromResponse supports three response formats:

1. Generic {files: {path: content}} — a JSON object with a files key mapping file paths to content strings
2. ARBuilder-style — JSON with top-level keys code, cargo_toml, main_rs, stylus_toml, rust_toolchain_toml that map to Stylus project files. Also handles a tests key that is appended to src/lib.rs (or placed in tests/mod.rs if no code key is present)
3. --- FILE: path --- delimited blocks — plain text with file boundaries marked by --- FILE: <path> --- headers

ARBuilder Key	Mapped File
code	src/lib.rs
cargo_toml	Cargo.toml
main_rs	src/main.rs
stylus_toml	Stylus.toml
rust_toolchain_toml	rust-toolchain.toml

Writing Scenarios

See scenarios/counter.yaml for an example. Key fields:

• sut.mcp_server — how to start the MCP server under test
• plan.steps — ordered tool calls to execute (use ARBuilder's prompt param, not description)
• workspace.required_files — files that must exist after generation
• gates — which verification gates to run
• agent — (optional) agent loop config: model, max_iterations, api_key_env

Environment Variable Substitution

Scenario YAML files support ${VAR} and ${VAR:-default} placeholders that resolve from process.env at load time. This keeps secrets and machine-specific paths out of committed files:

sut:
  mcp_server:
    command: "${ARBBUILDER_PYTHON:-python3}"
    cwd: "${ARBBUILDER_DIR}"
    env:
      OPENROUTER_API_KEY: "${OPENROUTER_API_KEY}"

Agent Loop

When configured, the agent loop uses an LLM to iteratively fix code that fails verification gates. After the initial MCP-generated code is tested, if any gates fail the agent:

1. Reads all workspace files and gate error output
2. Sends them to an LLM (via OpenRouter) asking it to fix the issues
3. Applies the LLM's returned file changes to the workspace
4. Re-runs all enabled gates
5. Repeats until all gates pass or max_iterations is reached

Configure per scenario in YAML:

agent:
  enabled: true
  model: "anthropic/claude-sonnet-4"
  max_iterations: 5
  api_key_env: OPENROUTER_API_KEY
  temperature: 0.2

CLI Reference

# Run a scenario
whyme-qa run -s <scenario.yaml> [-o <output-dir>] [-g <gates-dir>] [--keep-workspace]

# Run without agent loop (even if configured in scenario)
whyme-qa run -s <scenario.yaml> --no-agent

# Override agent model and iterations
whyme-qa run -s <scenario.yaml> --agent-model "anthropic/claude-sonnet-4" --max-iterations 10

# List scenarios
whyme-qa list [-d <scenarios-dir>]

CI/CD Pipeline

The .gitlab-ci.yml defines four stages:

Stage	Job	Description
lint	lint	Builds all packages and runs type checking
test	unit_tests	Runs unit tests, uploads reports as artifacts
verify	e2e_counter	Runs the counter scenario end-to-end (manual trigger)
report	(reserved)	Future reporting jobs

The e2e_counter job is manual (when: manual) and allow_failure: true, so it does not block the pipeline. JUnit XML reports are collected as GitLab test report artifacts.

A GitLab Duo flow definition is also provided at .gitlab/duo/flows/whyme_qa.yaml for use with GitLab Duo Workflow automation.

License

Apache-2.0