feat: 4-tier integration testing framework (#16)

## Summary

- **Adds a comprehensive integration testing framework** that proves the
core user contract: models serve inference, tool calling works, LiteLLM
routing works, and coding agents can connect via the OpenAI client.
- **Adds catalog validation to CI** so broken HuggingFace repo URLs
(like the qwen3.5-8b 404 in #15) are caught before merge.
- **Adds nightly and pre-release CI workflows** for smoke tests and full
stack integration.

## Test Tiers

| Tier | Marker | What it proves | When it runs |
|------|--------|----------------|--------------|
| **1. Catalog Validation** | `catalog_validation` | All HF repos exist,
fields valid, capabilities consistent | Every PR (CI) |
| **2. Model Smoke** | `smoke` | Each catalog model loads and serves
inference, tool calling, thinking | Nightly |
| **3. Stack Integration** | `integration` | Full lifecycle, LiteLLM
routing, concurrent requests, clean shutdown | Pre-release |
| **4. Harness Compatibility** | `harness` | OpenAI Python client works
(chat, streaming, tool calling, multi-turn) | Pre-release |

## New Make Targets

```
make test-catalog      # Tier 1 — fast, requires network
make test-smoke        # Tier 2 — slow, requires macOS + vllm-mlx
make test-integration  # Tier 3 — slow, requires macOS + vllm-mlx + litellm
make test-harness      # Tier 4 — slow, requires above + openai package
```

## Key Infrastructure

- **Dynamic port allocation** — no hardcoded ports, no conflicts with
running stacks
- **ServiceManager** — context manager with guaranteed cleanup (SIGTERM
→ SIGKILL → port verification)
- **Persistent model cache** — `~/.mlx-stack-test-cache/models/` avoids
re-downloading across runs
- **Compatibility matrix** — JSON report of pass/fail per model per
capability
- **Skip decorators** — graceful degradation for non-macOS, insufficient
memory, missing deps

## Verification

- `make check` passes (lint + typecheck + 1,421 unit tests)
- New tests correctly deselected from default `make test` run (168
deselected)
- No changes to existing test behavior

## Test plan

- [ ] `make check` passes on CI (lint, typecheck, unit tests)
- [ ] `make test-catalog` validates catalog entries against HuggingFace
API
- [ ] Nightly workflow triggers and runs smoke tests on smallest models
- [ ] Pre-release workflow triggers on release creation

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: weklund

.github/workflows/ci.yml

@@ -49,3 +49,31 @@ jobs:
         with:
           name: coverage-report
           path: coverage.xml
+
+  catalog-validation:
+    name: Catalog Validation
+    runs-on: macos-latest
+    needs: ci
+    strategy:
+      matrix:
+        python-version: ["3.13"]
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v6
+        with:
+          fetch-depth: 0
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v7
+        with:
+          enable-cache: true
+
+      - name: Install Python ${{ matrix.python-version }}
+        run: uv python install ${{ matrix.python-version }}
+
+      - name: Install dependencies
+        run: make install
+
+      - name: Validate catalog entries
+        run: make test-catalog

.github/workflows/integration-nightly.yml

@@ -0,0 +1,58 @@
+name: Nightly Integration
+
+on:
+  schedule:
+    - cron: "0 6 * * *"  # 6 AM UTC daily
+  workflow_dispatch:
+    inputs:
+      model:
+        description: "Specific model ID to test (blank = all non-gated models that fit in memory)"
+        required: false
+
+concurrency:
+  group: nightly-integration
+  cancel-in-progress: true
+
+jobs:
+  smoke:
+    name: Model Smoke Tests
+    runs-on: macos-latest
+    timeout-minutes: 120
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v6
+        with:
+          fetch-depth: 0
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v7
+        with:
+          enable-cache: true
+
+      - name: Install Python 3.13
+        run: uv python install 3.13
+
+      - name: Install project dependencies
+        run: make install
+
+      - name: Install vllm-mlx
+        run: uv tool install vllm-mlx==0.2.6
+
+      - name: Run smoke tests
+        run: |
+          MODEL_FILTER="${{ github.event.inputs.model }}"
+          if [ -n "$MODEL_FILTER" ]; then
+            uv run pytest -m smoke -k "$MODEL_FILTER" -v --tb=long
+          else
+            uv run pytest -m smoke -v --tb=long
+          fi
+        env:
+          HF_TOKEN: ${{ secrets.HF_TOKEN }}
+
+      - name: Upload compatibility matrix
+        uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: compatibility-matrix
+          path: ~/.mlx-stack-test-cache/reports/

.github/workflows/integration-prerelease.yml

@@ -0,0 +1,53 @@
+name: Pre-Release Integration
+
+on:
+  release:
+    types: [created]
+  workflow_dispatch:
+
+concurrency:
+  group: prerelease-integration
+  cancel-in-progress: true
+
+jobs:
+  stack-integration:
+    name: Full Stack Integration
+    runs-on: macos-latest
+    timeout-minutes: 60
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v6
+        with:
+          fetch-depth: 0
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v7
+        with:
+          enable-cache: true
+
+      - name: Install Python 3.13
+        run: uv python install 3.13
+
+      - name: Install project dependencies
+        run: uv sync --dev
+
+      - name: Install openai for harness tests
+        run: uv pip install openai>=1.0
+
+      - name: Install vllm-mlx and litellm
+        run: |
+          uv tool install vllm-mlx==0.2.6
+          uv tool install "litellm[proxy]==1.83.0"
+
+      - name: Run stack integration tests
+        run: uv run pytest -m "integration or harness" -v --tb=long
+        env:
+          HF_TOKEN: ${{ secrets.HF_TOKEN }}
+
+      - name: Upload test results
+        uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: integration-results
+          path: ~/.mlx-stack-test-cache/reports/

.gitignore

@@ -34,3 +34,4 @@ htmlcov/
 
 # Ruff
 .ruff_cache/
+REQUIREMENTS.md

Makefile

@@ -1,4 +1,4 @@
-.PHONY: install lint typecheck test check
+.PHONY: install lint typecheck test check test-catalog test-smoke test-integration test-harness
 
 ## Install dev dependencies
 install:
@@ -18,3 +18,19 @@ test:
 
 ## Run all checks (same as CI)
 check: lint typecheck test
+
+## Run catalog validation (requires network, no models)
+test-catalog:
+	uv run pytest -m catalog_validation -v --tb=short
+
+## Run per-model smoke tests (requires macOS + vllm-mlx)
+test-smoke:
+	uv run pytest -m smoke -v --tb=long
+
+## Run full stack integration tests (requires macOS + vllm-mlx + litellm)
+test-integration:
+	uv run pytest -m integration -v --tb=long
+
+## Run harness compatibility tests (requires above + openai)
+test-harness:
+	uv run pytest -m harness -v --tb=long

pyproject.toml

@@ -53,9 +53,12 @@ packages = ["src/mlx_stack"]
 [tool.pytest.ini_options]
 testpaths = ["tests"]
 pythonpath = ["src"]
-addopts = "-x -q --tb=short -m 'not integration'"
+addopts = "-x -q --tb=short -m 'not integration and not smoke and not catalog_validation and not harness'"
 markers = [
     "integration: real system integration tests (launchctl, etc.)",
+    "smoke: per-model smoke tests requiring model download and inference",
+    "catalog_validation: catalog data integrity checks (requires network, no models)",
+    "harness: external tool compatibility tests (OpenAI client, streaming)",
 ]
 
 [tool.ruff]

src/mlx_stack/cli/main.py

@@ -25,6 +25,7 @@
 from mlx_stack.cli.profile import profile as profile_command
 from mlx_stack.cli.pull import pull as pull_command
 from mlx_stack.cli.recommend import recommend as recommend_command
+from mlx_stack.cli.setup import setup as setup_command
 from mlx_stack.cli.status import status as status_command
 from mlx_stack.cli.up import up as up_command
 from mlx_stack.cli.watch import watch as watch_command
@@ -73,6 +74,7 @@ def format_help(self, ctx: click.Context, formatter: click.HelpFormatter) -> Non
             }
 
             command_categories = {
+                "setup": "Setup & Configuration",
                 "profile": "Setup & Configuration",
                 "config": "Setup & Configuration",
                 "init": "Setup & Configuration",
@@ -165,6 +167,7 @@ def cli(ctx: click.Context) -> None:
 # These will be replaced by real implementations in subsequent features.
 
 
+cli.add_command(setup_command, "setup")
 cli.add_command(profile_command, "profile")
 cli.add_command(recommend_command, "recommend")
 cli.add_command(init_command, "init")