Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,38 @@
## Unreleased

### Added

- **xAI (Grok) voice provider suite** — all three xAI Voice APIs land in both
SDKs, wired end-to-end with real pricing:
- **`XaiRealtime` engine (Grok Voice Agent API)**: real-time speech-to-speech
over `wss://api.x.ai/v1/realtime` (default model `grok-voice-latest`, voice
`eve`). The protocol is OpenAI-Realtime-GA-compatible, so the adapter
subclasses the GA adapter and inherits audio streaming, barge-in, and the
full tool-calling bridge (`transfer_call`/`end_call` included). xAI-specific
session knobs are exposed opt-in: `reasoning_effort` (`"high"`/`"none"`),
VAD `threshold`/`prefix_padding_ms`/`idle_timeout_ms`, ASR
`language_hint`/`keyterms`, output `speed`, pronunciation `replace` map,
session `resumption`, and raw `server_tools` passthrough for xAI
server-side tools (`web_search`, `x_search`, `mcp`, `file_search`).
- **`XaiSTT`**: streaming speech-to-text over `wss://api.x.ai/v1/stt`
(binary frames, `transcript.created` handshake, chunk/utterance finals,
Smart Turn end-of-turn detection, keyterm biasing, diarization) plus a
batch `transcribe()` helper for `POST /v1/stt` (word timestamps, ITN
formatting, 25 languages).
- **`XaiTTS`**: one-shot streaming synthesis via `POST /v1/tts` with the
26-voice Grok roster, `for_twilio()` (native G.711 µ-law 8 kHz) /
`for_telnyx()` (PCM16 16 kHz) carrier presets, and a
`create_custom_voice()` helper for `POST /v1/custom-voices` voice cloning.
- **Pricing** (official, docs.x.ai): Realtime $0.05/min, TTS $15.00/1M chars,
STT $0.20/hr streaming ($0.10/hr batch). `calculate_realtime_cost` /
`calculateRealtimeCost` gain an optional duration argument to support
per-minute realtime billing (token-based path unchanged).
`libraries/python/getpatter/providers/xai_{stt,tts,realtime}.py`,
`libraries/typescript/src/providers/xai-{stt,tts,realtime}.ts`,
`engines/xai.*`, `pricing.*`, docs pages under
`docs/{python,typescript}-sdk/providers/xai-*.mdx`. Beta: validated against
the xAI API spec and mocked protocol tests; not yet exercised on a live call.

### Fixed

- **`gemini-3.1-flash-live-preview` is now actually usable** (field-debugged on
Expand Down
5 changes: 3 additions & 2 deletions docs/concepts.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@ agent = phone.agent(
|---|---|---|
| `OpenAIRealtime()` | Speech-to-speech via OpenAI's Realtime API | Lowest latency, general-purpose |
| `ElevenLabsConvAI()` | Managed conversational agent on ElevenLabs | Premium voice quality |
| `XaiRealtime()` | Speech-to-speech via xAI's Grok Voice Agent | Reasoning + built-in web / X search |

**Pipeline mode (full control)** — you pick each stage:

Expand All @@ -100,9 +101,9 @@ Want fully custom LLM logic (multi-model routing, local models, an internal gate

Three independent stages — swap any of them for a different vendor or local model:

- **STT (Speech-to-Text)** — transcribes caller audio in real time. Providers: `DeepgramSTT`, `WhisperSTT`, `CartesiaSTT`, `SonioxSTT`, `SpeechmaticsSTT` (Python-only), `AssemblyAISTT`. See [STT](/python-sdk/stt).
- **STT (Speech-to-Text)** — transcribes caller audio in real time. Providers: `DeepgramSTT`, `WhisperSTT`, `CartesiaSTT`, `SonioxSTT`, `SpeechmaticsSTT` (Python-only), `AssemblyAISTT`, `XaiSTT`. See [STT](/python-sdk/stt).
- **LLM (Large Language Model)** — generates the reply. Pass a class instance via `llm=`: `OpenAILLM`, `AnthropicLLM`, `GroqLLM`, `CerebrasLLM`, `GoogleLLM`. Tool calling works across all five. For anything else, use `on_message`. See [LLM](/python-sdk/llm).
- **TTS (Text-to-Speech)** — synthesizes the reply audio. Providers: `ElevenLabsTTS`, `OpenAITTS`, `CartesiaTTS`, `RimeTTS`, `LMNTTTS`. See [TTS](/python-sdk/tts).
- **TTS (Text-to-Speech)** — synthesizes the reply audio. Providers: `ElevenLabsTTS`, `OpenAITTS`, `CartesiaTTS`, `RimeTTS`, `LMNTTTS`, `XaiTTS`. See [TTS](/python-sdk/tts).

Pick engine mode when you want minimum code. Pick pipeline mode when you need a specific LLM, a custom voice, or fine-grained control over latency / costs.

Expand Down
18 changes: 12 additions & 6 deletions docs/docs.json
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,8 @@
"python-sdk/providers/openai-realtime-2",
"python-sdk/providers/gemini-live",
"python-sdk/providers/ultravox-realtime",
"python-sdk/providers/elevenlabs-convai"
"python-sdk/providers/elevenlabs-convai",
"python-sdk/providers/xai-realtime"
]
},
{
Expand All @@ -126,7 +127,8 @@
"python-sdk/providers/whisper",
"python-sdk/providers/soniox",
"python-sdk/providers/speechmatics",
"python-sdk/providers/telnyx-stt"
"python-sdk/providers/telnyx-stt",
"python-sdk/providers/xai-stt"
]
},
{
Expand All @@ -152,7 +154,8 @@
"python-sdk/providers/openai-tts",
"python-sdk/providers/cartesia-tts",
"python-sdk/providers/inworld",
"python-sdk/providers/telnyx-tts"
"python-sdk/providers/telnyx-tts",
"python-sdk/providers/xai-tts"
]
},
{
Expand Down Expand Up @@ -247,7 +250,8 @@
"typescript-sdk/providers/openai-realtime-2",
"typescript-sdk/providers/gemini-live",
"typescript-sdk/providers/ultravox-realtime",
"typescript-sdk/providers/elevenlabs-convai"
"typescript-sdk/providers/elevenlabs-convai",
"typescript-sdk/providers/xai-realtime"
]
},
{
Expand All @@ -261,7 +265,8 @@
"typescript-sdk/providers/whisper",
"typescript-sdk/providers/soniox",
"typescript-sdk/providers/speechmatics",
"typescript-sdk/providers/telnyx-stt"
"typescript-sdk/providers/telnyx-stt",
"typescript-sdk/providers/xai-stt"
]
},
{
Expand All @@ -287,7 +292,8 @@
"typescript-sdk/providers/openai-tts",
"typescript-sdk/providers/cartesia-tts",
"typescript-sdk/providers/inworld",
"typescript-sdk/providers/telnyx-tts"
"typescript-sdk/providers/telnyx-tts",
"typescript-sdk/providers/xai-tts"
]
},
{
Expand Down
47 changes: 46 additions & 1 deletion docs/python-sdk/engines.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,12 @@ icon: "bolt"

An **engine** is an end-to-end speech-to-speech runtime. Pass an engine instance to `phone.agent(engine=...)` and Patter wires the audio stream straight through to the provider — no separate STT or TTS is needed.

Patter ships with three engine classes today:
Patter ships several engine classes:

- [`OpenAIRealtime`](#openairealtime) — OpenAI's Realtime API (v1-beta family, `gpt-realtime-mini` / `gpt-realtime` / `gpt-4o-*-realtime-preview`)
- [`OpenAIRealtime2`](#openairealtime2) — OpenAI's GA Realtime API (`gpt-realtime-2`), separate marker because the GA endpoint speaks a different `session.update` wire shape
- [`ElevenLabsConvAI`](#elevenlabsconvai) — ElevenLabs Conversational AI
- [`XaiRealtime`](#xairealtime) — xAI Grok Voice Agent (OpenAI-GA-compatible)

Each class ships as both a **flat alias** (`from getpatter import OpenAIRealtime`) and a **namespaced** class (`from getpatter.engines import openai` → `openai.Realtime()`). They are equivalent.

Expand Down Expand Up @@ -159,6 +160,50 @@ engine = elevenlabs_engine.ConvAI() # reads env
engine = elevenlabs_engine.ConvAI(agent_id="agent_abc123", voice="rachel")
```

## XaiRealtime

xAI's **Grok Voice Agent** — an OpenAI-Realtime-GA-compatible speech-to-speech engine with on-by-default reasoning and server-side tools (`web_search`, `x_search`, `mcp`, `file_search`).

```python
import asyncio
from getpatter import Patter, Twilio, XaiRealtime

phone = Patter(carrier=Twilio(), phone_number="+15550001234") # TWILIO_* from env

agent = phone.agent(
engine=XaiRealtime(voice="eve"), # XAI_API_KEY from env
system_prompt="You are a friendly receptionist.",
first_message="Hello! How can I help?",
)

async def main():
await phone.serve(agent)

asyncio.run(main())
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `api_key` | `str` | `""` | xAI API key. Reads from `XAI_API_KEY` when empty. |
| `model` | `str` | `"grok-voice-latest"` | Grok voice model. `grok-voice-latest` tracks the flagship (`grok-voice-think-fast-1.0`). |
| `voice` | `str` | `"eve"` | Built-in or custom voice ID. |
| `reasoning_effort` | `"high" \| "none" \| None` | `None` | `"high"` (xAI server default) enables reasoning; `"none"` disables it for lower latency. |

Namespaced form:

```python
from getpatter.engines import xai

engine = xai.XaiRealtime() # reads XAI_API_KEY
engine = xai.XaiRealtime(voice="eve", reasoning_effort="none")
```

For the full session-option surface — VAD tuning, language hint, keyterms, pronunciation replacements, session resumption, and server-side tools — see [xAI Realtime — full reference](/python-sdk/providers/xai-realtime).

<Note>
**Beta** — spec-validated, not yet live-call-validated.
</Note>

## What's Next

<CardGroup cols={2}>
Expand Down
15 changes: 15 additions & 0 deletions docs/python-sdk/metrics.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -245,13 +245,16 @@ Provider-level defaults are listed below. Per-model rates live under `DEFAULT_PR
| Cartesia STT (ink-whisper) | per minute | $0.0025 |
| Soniox | per minute | $0.002 |
| Speechmatics (Pro) | per minute | $0.004 |
| xAI STT (streaming) | per minute | $0.003333 ($0.20/hr) |
| ElevenLabs (`eleven_flash_v2_5`) | per 1k chars | $0.06 |
| OpenAI TTS (`tts-1`) | per 1k chars | $0.015 |
| Cartesia TTS (`sonic-2`) | per 1k chars | $0.030 |
| Rime (`mistv2`) | per 1k chars | $0.030 |
| LMNT (`aurora`) | per 1k chars | $0.050 |
| Inworld (`inworld-tts-2`) | per 1k chars | $0.020 |
| xAI TTS | per 1k chars | $0.015 |
| OpenAI Realtime (`gpt-realtime-mini` / `gpt-4o-mini-realtime-preview`) | per token | $10/M audio in · $20/M audio out · $0.60/M text in · $2.40/M text out (cached: $0.30/M audio · $0.06/M text) |
| xAI Realtime (Grok Voice Agent) | per minute | $0.05 ($3.00/hr) |
| Twilio (US inbound local) | per minute | $0.0085 (rounded up to whole minute, per Twilio) |
| Telnyx | per minute | $0.007 |

Expand All @@ -271,6 +274,8 @@ Provider-level defaults are listed below. Per-model rates live under `DEFAULT_PR
| OpenAI Transcribe (`openai_transcribe`) | `gpt-4o-transcribe` *(default)* | $0.006/min |
| OpenAI Transcribe | `gpt-4o-mini-transcribe` | $0.003/min |
| OpenAI Transcribe | `whisper-1` | $0.006/min |
| xAI (`xai`) | streaming *(default)* | $0.003333/min ($0.20/hr) |
| xAI | batch / REST (`xai_transcribe`) | $0.10/hr (documented; not metered on the pipeline) |

#### TTS — per-model rates

Expand All @@ -289,6 +294,7 @@ Provider-level defaults are listed below. Per-model rates live under `DEFAULT_PR
| LMNT | `aurora` *(default)* / `blizzard` | $0.050/1k |
| Inworld | `inworld-tts-2` *(default)* | $0.020/1k |
| Inworld | `inworld-tts-1.5-max` / `inworld-tts-1.5` | $0.025/1k |
| xAI (`xai_tts`) | default | $0.015/1k ($15.00/1M chars) |

#### OpenAI Realtime — per-model rates

Expand All @@ -303,6 +309,15 @@ Provider-level defaults are listed below. Per-model rates live under `DEFAULT_PR
`gpt-4o-realtime-preview` is roughly **10x** the cost of `gpt-realtime-mini` for audio. Switching realtime models has direct billing impact — confirm the model on `agent.realtime.model` matches the rate you expect.
</Warning>

#### xAI Realtime — rates

Unlike OpenAI Realtime (token-based), the xAI Grok Voice Agent is billed **per minute** of session audio, so Patter meters it from the call's `duration_seconds` rather than token counts.

| Provider | Unit | Rate |
|----------|------|------|
| xAI Realtime (`xai_realtime`) | per minute | $0.05/min ($3.00/hr) |
| xAI Realtime — text input | per message | $0.004 per `conversation.item.create` *(documented constant; not metered by Patter metrics v1)* |

<Note>
Twilio defaults match US **inbound** local. Override `pricing.twilio.price` for US toll-free inbound (~$0.022/min) or US outbound local (~$0.014/min). Default pricing is based on publicly listed provider rates and may become stale — check the provider's pricing page or pass your own overrides for authoritative numbers.
</Note>
Expand Down
Loading
Loading