Python Voice Agent with OpenAI Realtime API

A real-time voice agent built with Python that uses OpenAI's Realtime API for natural voice conversations.

Features

• Real-time audio streaming: Capture audio from your microphone and stream it to OpenAI
• Natural conversations: Use OpenAI's Realtime API for low-latency voice interactions
• Audio playback: Play back AI responses through your speakers
• WebSocket-based: Efficient bidirectional communication with OpenAI

Architecture

The voice agent consists of several key components:

1. Audio Capture Module (audio_capture.py): Handles microphone input and audio streaming
2. OpenAI Client Module (openai_client.py): Manages WebSocket connection to OpenAI Realtime API
3. Audio Playback Module (audio_playback.py): Handles audio output and playback
4. Main Orchestrator (main.py): Coordinates all components and manages the conversation flow

Requirements

• Python 3.9 or higher
• OpenAI API key with Realtime API access
• Microphone and speakers
• Internet connection

Installation

Step 1: Clone or create the project directory

cd python-voice-agent

Step 2: Create a virtual environment

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

Step 3: Install dependencies

pip install -r requirements.txt

Step 4: Configure environment variables

Copy the example environment file and add your OpenAI API key:

cp .env.example .env

Edit .env and add your OpenAI API key:

OPENAI_API_KEY=your_api_key_here

Usage

Basic Usage

Run the voice agent:

python main.py

The agent will:

1. Initialize the connection to OpenAI Realtime API
2. Start listening to your microphone
3. Begin the conversation - just start speaking!
4. Press Ctrl+C to stop

Configuration Options

You can modify settings in config.py:

• SAMPLE_RATE: Audio sample rate (default: 24000 Hz)
• CHANNELS: Audio channels (default: 1 - mono)
• CHUNK_SIZE: Audio chunk size for streaming (default: 4096)
• MODEL: OpenAI model to use (default: "gpt-4o-realtime-preview-2024-12-17")

How It Works

1. Audio Capture

The AudioCapture class uses PyAudio to:

• Open an audio stream from your microphone
• Capture audio in chunks
• Convert audio to base64-encoded format
• Stream audio data to the OpenAI client

2. OpenAI Realtime API Connection

The OpenAIRealtimeClient class:

• Establishes a WebSocket connection to OpenAI's Realtime API
• Sends audio chunks for processing
• Receives transcriptions and audio responses
• Handles conversation state and events

3. Audio Playback

The AudioPlayback class:

• Receives audio responses from OpenAI
• Decodes base64 audio data
• Plays audio through your speakers
• Manages playback queue for smooth output

4. Event Flow

User speaks → Microphone → Audio Capture → Base64 encoding
                                              ↓
                                    OpenAI Realtime API
                                              ↓
                                    Audio Response (base64)
                                              ↓
Speaker ← Audio Playback ← Base64 decoding ← Response

API Reference

AudioCapture Class

class AudioCapture:
    def start_stream(self, callback)
    def stop_stream()
    def is_active()

OpenAIRealtimeClient Class

class OpenAIRealtimeClient:
    async def connect()
    async def send_audio(audio_data: bytes)
    async def disconnect()
    def set_audio_callback(callback)

AudioPlayback Class

class AudioPlayback:
    def play_audio(audio_data: bytes)
    def stop()
    def is_playing()

Troubleshooting

"No API key found"

• Make sure you've created a .env file with your OPENAI_API_KEY
• Verify the API key is valid and has Realtime API access

"No default input device"

• Check that your microphone is connected and enabled
• Verify microphone permissions on your OS

"WebSocket connection failed"

• Check your internet connection
• Verify your API key has access to the Realtime API
• Check OpenAI service status

Audio quality issues

• Adjust SAMPLE_RATE in config.py
• Check your microphone settings
• Ensure a stable internet connection

Project Structure

python-voice-agent/
├── README.md                 # This file
├── requirements.txt          # Python dependencies
├── .env.example             # Environment variables template
├── .env                     # Your API keys (gitignored)
├── config.py                # Configuration settings
├── audio_capture.py         # Microphone input handling
├── audio_playback.py        # Audio output handling
├── openai_client.py         # OpenAI Realtime API integration
├── main.py                  # Main orchestrator
└── .gitignore              # Git ignore rules

Development Plan

This project was built following these steps:

1. Initial Setup: Created project structure and configuration files
2. Dependencies: Set up Python packages (websockets, pyaudio, openai, etc.)
3. Audio Capture: Implemented microphone input with PyAudio
4. OpenAI Integration: Built WebSocket client for Realtime API
5. Audio Playback: Created audio output system
6. Orchestration: Integrated all components in main.py
7. Documentation: Added comprehensive docs and comments

Technical Details

Audio Format

• Sample Rate: 24000 Hz (required by OpenAI Realtime API)
• Encoding: 16-bit PCM
• Channels: Mono (1 channel)
• Transport: Base64-encoded for WebSocket transmission

WebSocket Protocol

The OpenAI Realtime API uses WebSocket for bidirectional streaming:

• Events from client: input_audio_buffer.append, conversation.item.create
• Events from server: response.audio.delta, response.text.delta, response.done

Performance Considerations

• Audio is streamed in chunks to minimize latency
• Base64 encoding adds ~33% overhead but enables WebSocket transport
• Async I/O ensures non-blocking audio processing

Contributing

To extend this voice agent:

1. Add custom instructions in openai_client.py
2. Implement voice activity detection in audio_capture.py
3. Add conversation history management
4. Implement function calling capabilities

License

MIT License - feel free to use and modify as needed.

Resources

• OpenAI Realtime API Documentation
• PyAudio Documentation
• Websockets Documentation

Support

For issues or questions:

• Check the troubleshooting section above
• Review OpenAI API documentation
• Verify your environment setup

---

Note: This project requires an OpenAI API key with access to the Realtime API. The Realtime API may incur costs based on usage.