TEST_RUNNER_ENV_IMPLEMENTATION_PLAN.md

TEST_RUNNER_ Environment Variables Implementation Plan

Problem Statement

GitHub Issue: #101 - Support TEST_RUNNER_ prefixed env vars

Core Need: Enable conditional test behavior by passing TEST_RUNNER_ prefixed environment variables from MCP client configurations to xcodebuild test processes. This addresses the specific use case of disabling runsForEachTargetApplicationUIConfiguration for faster development testing.

Background Context

xcodebuild Environment Variable Support

From the xcodebuild man page:

TEST_RUNNER_<VAR>   Set an environment variable whose name is prefixed
                    with TEST_RUNNER_ to have that variable passed, with
                    its prefix stripped, to all test runner processes
                    launched during a test action. For example,
                    TEST_RUNNER_Foo=Bar xcodebuild test ... sets the
                    environment variable Foo=Bar in the test runner's
                    environment.

User Requirements

Users want to configure their MCP server with TEST_RUNNER_ prefixed environment variables:

{
  "mcpServers": {
    "XcodeBuildMCP": {
      "type": "stdio",
      "command": "npx",
      "args": ["-y", "xcodebuildmcp@latest"],
      "env": {
        "TEST_RUNNER_USE_DEV_MODE": "YES"
      }
    }
  }
}

And have tests that can conditionally execute based on these variables:

func testFoo() throws {
  let useDevMode = ProcessInfo.processInfo.environment["USE_DEV_MODE"] == "YES"
  guard useDevMode else {
    XCTFail("Test requires USE_DEV_MODE to be true")
    return
  }
  // Test logic here...
}

Current Architecture Analysis

XcodeBuildMCP Execution Flow

1. All Xcode commands flow through executeXcodeBuildCommand() function
2. Generic CommandExecutor interface handles all command execution
3. Test tools exist for device/simulator/macOS platforms
4. Zod schemas provide parameter validation and type safety

Key Files in Current Architecture

• src/utils/CommandExecutor.ts - Command execution interface
• src/utils/build-utils.ts - Contains executeXcodeBuildCommand
• src/mcp/tools/device/test_device.ts - Device testing tool
• src/mcp/tools/simulator/test_sim.ts - Simulator testing tool
• src/mcp/tools/macos/test_macos.ts - macOS testing tool
• src/utils/test/index.ts - Shared test logic for simulator

Solution Analysis

Design Options Considered

1. Automatic Detection (❌ Rejected)

   • Scan process.env for TEST_RUNNER_ variables and always pass them
   • Issue: Security risk of environment variable leakage
   • Issue: Unpredictable behavior based on server environment

2. Explicit Parameter (✅ Chosen)

   • Add testRunnerEnv parameter to test tools
   • Users explicitly specify which variables to pass
   • Benefits: Secure, predictable, well-validated

3. Hybrid Approach (🤔 Future Enhancement)

   • Both automatic + explicit with explicit overriding
   • Issue: Adds complexity, deferred for future consideration

Expert Analysis Summary

RepoPrompt Analysis: Comprehensive architectural plan emphasizing security, type safety, and integration with existing patterns.

Gemini Analysis: Confirmed explicit approach as optimal, highlighting:

• Security benefits of explicit allow-list approach
• Architectural soundness of extending CommandExecutor
• Recommendation for automatic prefix handling for better UX

Recommended Solution: Explicit Parameter with Automatic Prefix Handling

Key Design Decisions

1. Security-First: Only explicitly provided variables are passed (no automatic process.env scanning)
2. User Experience: Automatic prefix handling - users provide unprefixed keys
3. Architecture: Extend execution layer generically for future extensibility
4. Validation: Zod schema enforcement with proper type safety

User Experience Design

Input (what users specify):

{
  "testRunnerEnv": {
    "USE_DEV_MODE": "YES",
    "runsForEachTargetApplicationUIConfiguration": "NO"
  }
}

Output (what gets passed to xcodebuild):

TEST_RUNNER_USE_DEV_MODE=YES \
TEST_RUNNER_runsForEachTargetApplicationUIConfiguration=NO \
xcodebuild test ...

Implementation Plan

Phase 0: Test-Driven Development Setup

Objective: Create reproduction test to validate issue and later prove fix works

Tasks:

• Create test in example_projects/iOS/MCPTest that checks for environment variable
• Run current test tools to demonstrate limitation (test should fail)
• Document baseline behavior

Test Code Example:

func testEnvironmentVariablePassthrough() throws {
  let useDevMode = ProcessInfo.processInfo.environment["USE_DEV_MODE"] == "YES"
  guard useDevMode else {
    XCTFail("Test requires USE_DEV_MODE=YES via TEST_RUNNER_USE_DEV_MODE")
    return
  }
  XCTAssertTrue(true, "Environment variable successfully passed through")
}

Phase 1: Core Infrastructure Updates

Objective: Extend CommandExecutor and build utilities to support environment variables

1.1 Update CommandExecutor Interface

File: src/utils/CommandExecutor.ts

Changes:

• Add CommandExecOptions type for execution options
• Update CommandExecutor type signature to accept optional execution options

export type CommandExecOptions = {
  cwd?: string;
  env?: Record<string, string | undefined>;
};

export type CommandExecutor = (
  args: string[],
  description?: string,
  quiet?: boolean,
  opts?: CommandExecOptions
) => Promise<CommandResponse>;

1.2 Update Execution Facade

File: src/utils/execution/index.ts

Changes:

• Re-export CommandExecOptions type

export type { CommandExecutor, CommandResponse, CommandExecOptions } from '../CommandExecutor.js';

1.3 Update Default Command Executor

File: src/utils/command.ts

Changes:

• Modify getDefaultCommandExecutor to merge opts.env with process.env when spawning

// In the returned function:
const env = { ...process.env, ...(opts?.env ?? {}) };
// Pass env and opts?.cwd to spawn/exec call

1.4 Create Environment Variable Utility

File: src/utils/environment.ts

Changes:

• Add normalizeTestRunnerEnv function

export function normalizeTestRunnerEnv(
  userVars?: Record<string, string | undefined>
): Record<string, string> {
  const result: Record<string, string> = {};
  if (userVars) {
    for (const [key, value] of Object.entries(userVars)) {
      if (value !== undefined) {
        result[`TEST_RUNNER_${key}`] = value;
      }
    }
  }
  return result;
}

1.5 Update executeXcodeBuildCommand

File: src/utils/build-utils.ts

Changes:

• Add optional execOpts?: CommandExecOptions parameter (6th parameter)
• Pass execution options through to CommandExecutor calls

export async function executeXcodeBuildCommand(
  build: { /* existing fields */ },
  runtime: { /* existing fields */ },
  preferXcodebuild = false,
  action: 'build' | 'test' | 'archive' | 'analyze' | string,
  executor: CommandExecutor = getDefaultCommandExecutor(),
  execOpts?: CommandExecOptions, // NEW
): Promise<ToolResponse>

Phase 2: Test Tool Integration

Objective: Add testRunnerEnv parameter to all test tools and wire through execution

2.1 Update Device Test Tool

File: src/mcp/tools/device/test_device.ts

Changes:

• Add testRunnerEnv to Zod schema with validation
• Import and use normalizeTestRunnerEnv
• Pass execution options to executeXcodeBuildCommand

Schema Addition:

testRunnerEnv: z
  .record(z.string(), z.string().optional())
  .optional()
  .describe('Test runner environment variables (TEST_RUNNER_ prefix added automatically)')

Usage:

const execEnv = normalizeTestRunnerEnv(params.testRunnerEnv);
const testResult = await executeXcodeBuildCommand(
  { /* build params */ },
  { /* runtime params */ },
  params.preferXcodebuild ?? false,
  'test',
  executor,
  { env: execEnv } // NEW
);

2.2 Update macOS Test Tool

File: src/mcp/tools/macos/test_macos.ts

Changes: Same pattern as device test tool

• Schema addition for testRunnerEnv
• Import normalizeTestRunnerEnv
• Pass execution options to executeXcodeBuildCommand

2.3 Update Simulator Test Tool and Logic

File: src/mcp/tools/simulator/test_sim.ts

Changes:

• Add testRunnerEnv to schema
• Pass through to handleTestLogic

File: src/utils/test/index.ts

Changes:

• Update handleTestLogic signature to accept testRunnerEnv?: Record<string, string | undefined>
• Import and use normalizeTestRunnerEnv
• Pass execution options to executeXcodeBuildCommand

Phase 3: Testing and Validation

Objective: Comprehensive testing coverage for new functionality

3.1 Unit Tests

File: src/utils/__tests__/environment.test.ts

Tests:

• Test normalizeTestRunnerEnv with various inputs
• Verify prefix addition
• Verify undefined filtering
• Verify empty input handling

3.2 Integration Tests

Files: Update existing test files for test tools

Tests:

• Verify testRunnerEnv parameter is properly validated
• Verify environment variables are passed through CommandExecutor
• Mock executor to verify correct env object construction

3.3 Tool Export Validation

Files: Test files in each tool directory

Tests:

• Verify schema exports include new testRunnerEnv field
• Verify parameter typing is correct

Phase 4: End-to-End Validation

Objective: Prove the fix works with real xcodebuild scenarios

4.1 Reproduction Test Validation

Tasks:

• Run reproduction test from Phase 0 with new testRunnerEnv parameter
• Verify test passes (proving env var was successfully passed)
• Document the before/after behavior

4.2 Real-World Scenario Testing

Tasks:

• Test with actual iOS project using runsForEachTargetApplicationUIConfiguration
• Verify performance difference when variable is set
• Test with multiple environment variables
• Test edge cases (empty values, special characters)

Security Considerations

Security Benefits

• No Environment Leakage: Only explicit user-provided variables are passed
• Command Injection Prevention: Environment variables passed as separate object, not interpolated into command string
• Input Validation: Zod schemas prevent malformed inputs
• Prefix Enforcement: Only TEST_RUNNER_ prefixed variables can be set

Security Best Practices

• Never log environment variable values (keys only for debugging)
• Filter out undefined values to prevent accidental exposure
• Validate all user inputs through Zod schemas
• Document supported TEST_RUNNER_ variables from Apple's documentation

Architectural Benefits

Clean Integration

• Extends existing CommandExecutor pattern generically
• Maintains backward compatibility (all existing calls remain valid)
• Follows established Zod validation patterns
• Consistent API across all test tools

Future Extensibility

• CommandExecOptions can support additional execution options (timeout, cwd, etc.)
• Pattern can be extended to other tools that need environment variables
• Generic approach allows for non-TEST_RUNNER_ use cases in the future

File Modification Summary

New Files

• src/utils/__tests__/environment.test.ts - Unit tests for environment utilities

Modified Files

• src/utils/CommandExecutor.ts - Add execution options types
• src/utils/execution/index.ts - Re-export new types
• src/utils/command.ts - Update default executor to handle env
• src/utils/environment.ts - Add normalizeTestRunnerEnv utility
• src/utils/build-utils.ts - Update executeXcodeBuildCommand signature
• src/mcp/tools/device/test_device.ts - Add schema and integration
• src/mcp/tools/macos/test_macos.ts - Add schema and integration
• src/mcp/tools/simulator/test_sim.ts - Add schema and pass-through
• src/utils/test/index.ts - Update handleTestLogic for simulator path
• Test files for each modified tool - Add validation tests

Success Criteria

1. Functionality: Users can pass testRunnerEnv parameter to test tools and have variables appear in test runner environment
2. Security: No unintended environment variable leakage from server process
3. Usability: Users specify unprefixed variable names for better UX
4. Compatibility: All existing test tool calls continue to work unchanged
5. Validation: Comprehensive test coverage proves the feature works end-to-end

Future Enhancements (Out of Scope)

1. Configuration Profiles: Allow users to define common TEST_RUNNER_ variable sets in config files
2. Variable Discovery: Help users discover available TEST_RUNNER_ variables
3. Build Tool Support: Extend to build tools if Apple adds similar BUILD_RUNNER_ support
4. Performance Monitoring: Track impact of environment variable passing on build times

Implementation Timeline

• Phase 0: 1-2 hours (reproduction test setup)
• Phase 1: 4-6 hours (infrastructure changes)
• Phase 2: 3-4 hours (tool integration)
• Phase 3: 4-5 hours (testing)
• Phase 4: 2-3 hours (validation)

Total Estimated Time: 14-20 hours

Conclusion

This implementation plan provides a secure, user-friendly, and architecturally sound solution for TEST_RUNNER_ environment variable support. The explicit parameter approach with automatic prefix handling balances security concerns with user experience, while the test-driven development approach ensures we can prove the solution works as intended.

The plan leverages XcodeBuildMCP's existing patterns and provides a foundation for future environment variable needs across the tool ecosystem.