Skip to content

e2e-testing.md

Latest commit

 

History

History
183 lines (128 loc) · 5.83 KB

e2e-testing.md

File metadata and controls

183 lines (128 loc) · 5.83 KB
Raw

End-to-End Testing for Claude Code MCP

This document explains how to run and maintain the end-to-end tests for the Claude Code MCP server.

Overview

The e2e tests are designed to validate the Claude Code MCP server's functionality in real-world scenarios. Since the Claude CLI requires authentication and isn't easily installable in automated environments, the tests use a mock Claude CLI for automated testing and provide optional integration tests for local development.

Test Structure

The e2e tests are organized into several files:

  • src/__tests__/e2e.test.ts - Main e2e test suite with mock Claude CLI
  • src/__tests__/edge-cases.test.ts - Edge case and error handling tests
  • src/__tests__/utils/mcp-client.ts - Mock MCP client for testing
  • src/__tests__/utils/claude-mock.ts - Mock Claude CLI implementation

Running Tests

Quick Start

# Install dependencies
npm install

# Build the project
npm run build

# Run all tests (unit + e2e)
npm test

# Run only e2e tests with mocks
npm run test:e2e

# Run unit tests only
npm run test:unit

Local Integration Testing

When Claude CLI is installed locally, you can run the full integration tests:

# Run all tests including integration tests
npm run test:e2e:local

The integration tests are marked with .skip() by default and will only run when you have Claude CLI installed and authenticated.

Live CLI E2E

Use test:live for release-time checks against real installed AI CLIs. By default this suite runs the built dist/bin/ai-cli.js command and verifies the real run -> ps -> peek -> wait -> result -> cleanup flow, including persisted exit-status.json. It can also exercise the built MCP server surface.

Live E2E is opt-in and is not part of normal npm test:

# Shows the opt-in guard and does not call external CLIs
npm run test:live

# Default live backends: claude,codex
ACM_LIVE_E2E=1 npm run test:live

# Choose exact backends
ACM_LIVE_E2E=1 ACM_LIVE_E2E_AGENTS=claude,codex npm run test:live

# Full release-time coverage
ACM_LIVE_E2E=1 ACM_LIVE_E2E_AGENTS=all npm run test:live

# Include both ai-cli and MCP server surfaces
ACM_LIVE_E2E=1 ACM_LIVE_E2E_SURFACE=all ACM_LIVE_E2E_AGENTS=claude,codex npm run test:live

Useful environment variables:

  • ACM_LIVE_E2E_AGENTS: comma-separated claude,codex,gemini,forge,opencode, or all. Defaults to claude,codex.
  • ACM_LIVE_E2E_SURFACE: cli, mcp, or all. Defaults to cli.
  • ACM_LIVE_E2E_TIMEOUT_SECONDS: timeout passed to ai-cli wait. Defaults to 240.
  • ACM_LIVE_E2E_COMMAND_TIMEOUT_MS: process/request timeout used by the live test harness. Defaults to ACM_LIVE_E2E_TIMEOUT_SECONDS + 60s.
  • ACM_LIVE_E2E_TOKEN: expected token in the model response. Defaults to ACM_LIVE_E2E_OK.
  • ACM_LIVE_E2E_ASSERT_TOKEN=0: disables response-token assertion if a provider adds unavoidable formatting.
  • ACM_LIVE_E2E_<AGENT>_MODEL: override a default model, for example ACM_LIVE_E2E_CODEX_MODEL=gpt-5.6-sol. The default Codex value is gpt-5.4.

Keep this suite out of mandatory CI unless the runner has authenticated CLI sessions, network access, and an acceptable cost budget.

Test Scenarios

Basic Operations

  • Tool registration and discovery
  • Simple prompt execution
  • Error handling
  • Default working directory behavior

Working Directory Handling

  • Custom working directory support
  • Non-existent directory handling
  • Permission errors

Edge Cases

  • Input validation (missing/invalid parameters)
  • Special characters in prompts
  • Concurrent request handling
  • Large prompt handling
  • Path traversal prevention

Integration Tests (Local Only)

  • File creation with real Claude CLI
  • Git operations
  • Complex multi-step workflows

Mock Claude CLI

The tests use a mock Claude CLI that simulates basic Claude behavior. The mock:

  1. Creates a fake executable at ~/.claude/local/claude
  2. Responds to basic commands based on prompt patterns
  3. Simulates errors for testing error handling

The mock is automatically set up before tests run and cleaned up afterwards.

Writing New Tests

When adding new e2e tests:

  1. Use the MCPTestClient for communicating with the server
  2. Set up test directories in beforeEach and clean up in afterEach
  3. Use descriptive test names that explain the scenario
  4. Add appropriate assertions for both success and failure cases

Example:

it('should handle complex file operations', async () => {
  const response = await client.callTool('claude_code', {
    prompt: 'Create multiple files and organize them',
    workFolder: testDir,
  });

  expect(response).toBeTruthy();
  // Add specific assertions about the result
});

Debugging Tests

To debug e2e tests:

  1. Enable debug mode by setting MCP_CLAUDE_DEBUG=true
  2. Add console.log statements in test code
  3. Use the VSCode debugger with the test runner
  4. Check server stderr output for debug logs

CI/CD Considerations

The e2e tests are designed to run in CI environments without Claude CLI:

  • Mock tests run automatically in CI
  • Integration tests are skipped unless explicitly enabled
  • Tests use temporary directories to avoid conflicts
  • All tests clean up after themselves

Common Issues

Tests Timing Out

  • Increase timeout in vitest.config.e2e.ts
  • Check if the mock Claude CLI is set up correctly
  • Verify the server is building properly

Mock Not Found

  • Ensure the mock setup runs in beforeAll
  • Check file permissions on the mock executable
  • Verify the mock path matches the server's expectations

Integration Tests Failing

  • Ensure Claude CLI is installed and authenticated
  • Check that you're running the local test command
  • Verify Claude CLI is accessible in your PATH

Future Improvements

  • Add performance benchmarking tests
  • Implement stress testing scenarios
  • Add tests for specific Claude Code features
  • Create visual regression tests for output formatting