EAS Phone AI Agent

A modular, production-grade voice AI agent for phone intake, built with Python. It features:

• Modular architecture (audio input, STT, TTS, LLM, dialog manager)
• Configurable prompts and responses (YAML)
• Centralized logging and config
• Hot-reload for prompts
• Class-based APIs for all modules
• Modern packaging (pyproject.toml)

Setup

1. Clone the repo
2. Install dependencies (editable mode recommended):

   pip install -e .[dev]

3. Download model files (see scripts/download_models.py or use your own)
4. Set up your .env with API keys for OpenAI, ElevenLabs, etc.

Usage

python -m eas_phone_ai_agent.main

Architecture

• eas_phone_ai_agent/
  • audio_input.py — Audio/VAD input
  • stt.py — Whisper STT wrapper
  • tts_output.py — ElevenLabs TTS wrapper
  • llm_client.py — OpenAI LLM wrapper
  • dialog_manager.py — Intake state machine
  • utils/ — Logging, config, helpers
  • config/ — Prompts and canned responses (YAML)
• main.py — Wires up the pipeline

Audio Input & VAD

• Device Selection:
  • The agent selects the microphone device in this order:
    1. Device index passed as argument (rare, for advanced use)
    2. AUDIO_DEVICE_INDEX environment variable (recommended for production)
    3. Auto-selects a preferred mic (e.g., SteelSeries Microphone) or first available input device
• Mic Index Config:
  • Set AUDIO_DEVICE_INDEX in your .env to force a specific device (see sounddevice.query_devices() for indices)
• VAD & Channel Mixing:
  • Uses WebRTC VAD for voice activity detection
  • Handles mono/stereo input, down-mixes to mono as needed
  • Logs RMS values for debugging VAD thresholds
• Troubleshooting:
  • If you get only silence, check your device index and permissions
  • Use python -m sounddevice or sounddevice.query_devices() to list available devices
  • If you see repeated warnings about all-zero frames, your mic may be muted or misconfigured

Speech-to-Text (STT)

• Model Selection:
  • Uses Faster Whisper for speech-to-text.
  • Model size (e.g., 'small', 'medium') and quantization can be configured in code or via environment variables (future).
  • Device selection (CPU/GPU) is configurable.
• Error Handling:
  • Automatic retries on transient errors (with exponential backoff).
  • Returns an empty string on unrecoverable errors or bad audio input.
• Validation:
  • Built-in email and phone number validation helpers.
• Troubleshooting:
  • If you get empty transcripts, check your audio input and model config.
  • For best performance, use a GPU and quantized models if available.

Text-to-Speech (TTS)

• Voice & Model Config:
  • Uses ElevenLabs API for TTS.
  • Voice ID, model ID, speed, volume, and style can be set via environment variables or arguments.
  • Example ENV vars: ELEVENLABS_VOICE_ID, ELEVENLABS_TTS_MODEL, TTS_SPEED, TTS_VOLUME, TTS_STYLE.
• SSML Support:
  • (Planned) Supports SSML for richer prosody and control. Wrap text in <speak>...</speak> tags for future compatibility.
• Error Handling:
  • Retries on API errors, logs failures, and returns silence on unrecoverable errors.
  • Handles barge-in (interrupting playback) and playback races.
• Troubleshooting:
  • If you hear no audio, check your API key, device, and volume settings.
  • For best results, use a supported ElevenLabs voice/model and ensure your API key is valid.

Language Model (LLM)

• Model Config:
  • Uses OpenAI GPT models for chat and summary tasks.
  • Model names, API key, and temperature can be set via environment variables or arguments.
  • Example ENV vars: OPENAI_API_KEY, LLM_FAST_MODEL, LLM_HIGH_QUALITY, LLM_FAST_TEMP, LLM_HIGH_TEMP.
• Prompt Management:
  • Prompts are managed in code and can be externalized to YAML for versioning and customization.
• Error Handling:
  • Retries on API errors, logs failures, and returns empty string on unrecoverable errors.
  • Handles rate limits (429), timeouts, and other transient errors with exponential backoff.
• Troubleshooting:
  • If you get empty responses, check your API key, model config, and prompt formatting.
  • For best results, use the recommended models and keep your API key secure.

Dialog Manager

• Flow Management:
  • Manages the intake call flow as a state machine, progressing through all required fields and handling user responses.
  • Prompts, field labels, and silence nudges are loaded from YAML config for easy customization.
• Extensibility:
  • Easily add new intake flows or fields by extending the state sequence and adding handler methods.
  • Plugin structure allows for custom logic, fallback nodes, and mid-flow changes.
• Error Handling:
  • Handles unknown or off-script user input gracefully, with fallback prompts and state recovery.
  • Logs analytics and state transitions for every call.
• Troubleshooting:
  • If the flow gets stuck, check your prompt YAML and state sequence for mismatches.
  • For custom flows, ensure new states have corresponding prompts and handlers.

Main Pipeline & Entry Point

• How to Run:
  • Start the agent with python -m eas_phone_ai_agent.main or use the provided console script.
• Async Setup & Shutdown:
  • Handles async event loop setup, cross-platform signal handling, and graceful shutdown.
  • Health check server runs in the background for monitoring.
• Troubleshooting:
  • If the agent fails to start, check your environment variables and API keys.
  • On Windows, event loop policy is set automatically; on Unix, signal handlers are used for shutdown.

Utilities & Helpers

• Config Loader:
  • Loads YAML prompt config with hot-reload and thread safety.
  • Handles version logging and error handling.
• Extractors:
  • Provides context/state tracking and field extraction from user input.
  • Includes regex helpers for phone, email, and off-script detection.
• Logging & Analytics:
  • Centralized logging setup with Rich console and rotating file handler (JSON format).
  • Decorators for logging function calls and retrying on error (sync/async).
  • Analytics helpers for CSV output and log rotation.

Config & Prompts

• Prompt YAML:
  • All prompts, field labels, and silence nudges are defined in config/prompts.yaml.
  • Edit or add new prompts/fields as needed. Use the version field to track changes.
• Versioning:
  • Update the version field in the YAML when making changes for traceability and hot-reload.
• Testing Prompt Changes:
  • Use the test suite to ensure new prompts/fields are picked up and handled correctly.
  • If a prompt is missing, the agent will fall back to a default message.
• Extending Flows:
  • Add new fields to field_labels and prewritten_responses.
  • Update the state sequence in DialogManager to include new steps.

Analytics & Logging

• Analytics CSVs:
  • All call/session analytics are logged to analytics_log.csv and analytics_summary.csv in the project root.
  • Each turn and session summary is recorded for later analysis.
• Schema Validation:
  • The analytics CSVs follow a consistent schema (timestamp, state, transcript, sentiment, value, etc.).
  • Use the test suite to validate schema and ensure no data loss.
• Log Rotation:
  • The main agent log (agent.log) uses rotating file handlers to prevent disk overuse.
  • Log files are rotated automatically when they reach 5MB, with up to 3 backups kept.
• Troubleshooting:
  • If logs or analytics are missing, check file permissions and logging config.
  • Use the test suite to verify logging and analytics output.

Testing

pytest

Model Files

• Not included in the repo. Add to .gitignore and use Git LFS or a download script of your own.

Development

Running Tests

pytest

Updating Prompts

Edit eas_phone_ai_agent/config/prompts.yaml to change the agent's language and prompts. Ensure all state names match those in DialogManager.

Starting the Agent Locally

python -m eas_phone_ai_agent.main

YAML Prompt Reference

See eas_phone_ai_agent/config/prompts.yaml for all prompt keys and their text.

Sample Conversation

User: Hello
Agent: Good morning! What is your name?
User: John Doe
Agent: Thanks, John. What's your phone number?
...etc.

License

MIT

Logging and Log Rotation

Logs are written to both the terminal (with color) and to a file (default: agent.log).

Log rotation is enabled by default using a rotating file handler. You can configure log rotation via environment variables:

• EAS_AGENT_LOG: Path to the log file (default: agent.log)
• EAS_AGENT_LOG_MAX_BYTES: Maximum size in bytes before rotating (default: 5242880, i.e., 5MB)
• EAS_AGENT_LOG_BACKUP_COUNT: Number of rotated log files to keep (default: 3)

Example:

EAS_AGENT_LOG=my_agent.log EAS_AGENT_LOG_MAX_BYTES=1048576 EAS_AGENT_LOG_BACKUP_COUNT=5 python -m eas_phone_ai_agent.main