Skip to content

TESTING.md

Latest commit

 

History

History
190 lines (143 loc) · 4.42 KB

TESTING.md

File metadata and controls

190 lines (143 loc) · 4.42 KB

Testing Guide

This document describes the testing strategy for the Customer Service Bot project.

Test Suites

1. Unit Tests (Default)

The default test suite runs comprehensive unit tests with mocked APIs. These tests:

  • Run quickly without external dependencies
  • Mock all API calls (OpenRouter and local order API)
  • Test all bot functionality in isolation
  • Are suitable for CI/CD pipelines

Run with:

./run_tests.sh

Test files:

  • tests/test_api.py - Tests for the mock order API
  • tests/test_bot.py - Tests for bot logic with mocked LLM
  • Other test files (if present)

Coverage:

  • Order tracking and cancellation logic
  • Policy enforcement (10-day cancellation window)
  • Tool execution and error handling
  • Conversation history management
  • Metrics tracking

2. Live Integration Tests

A minimal test suite that makes real API calls to OpenRouter. These tests:

  • Verify actual LLM integration works correctly
  • Test real-world API responses and error handling
  • Validate tool calling with live LLM
  • Consume actual API tokens (costs money)

Prerequisites:

  1. Set your OpenRouter API key:

    export OPENROUTER_API_KEY='your-api-key-here'

  2. Optionally, start the local order API (if testing end-to-end):

    uvicorn api.main:app --reload

Run with:

./run_tests.sh --live

Test file:

  • tests/test_live_integration.py - Live API integration tests

Coverage:

  • Basic LLM chat completions
  • Tool/function calling with real LLM
  • Multi-turn conversations with context
  • Error handling with live API
  • Session metrics and conversation export

Note: Live tests will be automatically skipped if OPENROUTER_API_KEY is not set.

Test Organization

Unit Tests

tests/
├── test_api.py              # Mock API endpoint tests
├── test_bot.py              # Bot logic tests (mocked LLM)
└── test_live_integration.py # Live API tests (separate)

Mocking Strategy

Unit tests use unittest.mock to mock:

  • OpenRouter API responses (bot.llm_client.chat)
  • Order API HTTP calls (requests.get, requests.post)
  • Time-sensitive operations (datetime)

Live tests use minimal mocking:

  • Only mock the local order API (to avoid dependency)
  • Let real OpenRouter API calls go through
  • Test actual LLM behavior and responses

Running Tests

Quick Start

# Run all unit tests (no API key needed)
./run_tests.sh

# Run live integration tests (requires API key)
export OPENROUTER_API_KEY='sk-or-v1-...'
./run_tests.sh --live

Manual Test Execution

If you prefer to run pytest directly:

# Setup environment
conda activate customerservice

# Unit tests only
pytest tests/ -v --ignore=tests/test_live_integration.py

# Live tests only
pytest tests/test_live_integration.py -v -s

# All tests (will skip live if no API key)
pytest tests/ -v

Test Options

  • -v - Verbose output
  • -s - Show print statements
  • -k pattern - Run tests matching pattern
  • --ignore=path - Ignore specific test file

CI/CD Integration

For automated testing in CI/CD:

# GitHub Actions example
- name: Run unit tests
  run: ./run_tests.sh
  
# Optional: Run live tests (with API key secret)
- name: Run live tests
  env:
    OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
  run: ./run_tests.sh --live

Test Development Guidelines

Writing Unit Tests

  • Always mock external dependencies
  • Use @pytest.fixture for reusable setup
  • Test both success and failure scenarios
  • Keep tests fast (< 1s per test)

Writing Live Tests

  • Keep the test suite minimal (token costs)
  • Add pytestmark = pytest.mark.skipif(...) to skip without API key
  • Use patch for local API calls (avoid external dependencies)
  • Test actual LLM behavior, not implementation details

Troubleshooting

"OPENROUTER_API_KEY not set"

Set your API key:

export OPENROUTER_API_KEY='your-key-here'

"Connection refused" errors

Start the local API server:

uvicorn api.main:app --reload

Tests timing out

  • Check internet connection for live tests
  • Verify API key is valid
  • Check OpenRouter API status

Import errors

Reinstall dependencies:

conda activate customerservice
pip install -r requirements.txt

Test Metrics

Current test coverage:

  • Unit tests: 36 tests covering all core functionality
  • Live tests: 11 tests for integration validation
  • Total runtime: ~0.6s (unit) / ~10-30s (live)