Updated documentation for Model Intelligence

fern/assistants/model-intelligence/metrics-methodology.mdx

@@ -0,0 +1,93 @@
+---
+title: Performance metrics and methodology reference
+subtitle: See how Vapi sources and calculates latency, cost, and quality metrics.
+description: The performance metrics reference explains how Vapi sources and calculates latency, cost, and quality data for each supported transcriber, model, and voice.
+slug: assistants/model-intelligence/metrics-methodology
+---
+
+Vapi measures latency from production calls. Cost metrics are calculated estimates. Quality metrics come from Vapi or third-party benchmarks.
+
+## Source summary
+
+| Metric | Component | What it measures | Source |
+|---|---|---|---|
+| **Latency** | Transcriber, model, voice | Typical response time | Median from Vapi production calls |
+| **Cost** | Transcriber, model, voice | Estimated cost per minute | [Calculated estimate based on typical usage](/assistants/model-intelligence/understanding-cost) |
+| **Word error rate (WER)** | Transcriber | The percentage of words the transcriber gets wrong | [Daily benchmark](https://www.daily.co/blog/benchmarking-stt-for-voice-agents/) |
+| **Intelligence** | Model | Reasoning and task capability | [Artificial Analysis benchmark](https://artificialanalysis.ai/models) |
+| **Humanness** | Voice | How human the voice sounds | [Vapi Humanness Index](https://humannessindex.vapi.ai/) |
+
+## Latency
+
+Vapi reports the median response time for each component based on production calls.
+
+| Component | Latency measurement |
+|---|---|
+| Transcriber | Time to convert speech to text |
+| Model | Time to first token |
+| Voice | Time to first audio from the text-to-speech provider |
+
+The displayed total is simply the sum of the three component medians, so treat it as a rough estimate rather than an exact figure.
+
+<Note>
+The total excludes [endpointing](/customization/speech-configuration) and transport time.
+</Note>
+
+See [how latency works](/assistants/model-intelligence/understanding-latency) for help understanding latency and why more capable models tend to be slower.
+
+## Cost
+
+In order to estimate cost, we came up with a number of assumptions based on actual Vapi data. However, costs vary greatly depending on actual call performance and agent behavior. The displayed cost is an estimated rate per minute, use it to compare models rather than predict your exact bill.
+
+<Note>
+For contracted customers, certain pricing may vary depending on your agreement. Please contact your account team for the most accurate estimate.
+</Note>
+
+| Component | Estimation basis |
+|---|---|
+| Transcriber | Audio minutes for the caller and assistant |
+| Model | Prompt size, tool definitions, prompt caching, and provider rates |
+| Voice | Characters spoken based on typical speaking volume |
+
+LLM cost depends on the assistant's configuration. Vapi uses the assistant's prompt and tool definitions, so two assistants on the same model can show different estimates. The calculation also uses each provider's cached-input rate when prompt caching is supported.
+
+<Note>
+The estimate assumes a moderate cache-hit rate, so actual costs for large prompts are often lower. It excludes growing conversation history, so actual costs for long calls are often higher.
+</Note>
+
+See [how cost works](/assistants/model-intelligence/understanding-cost) for the formulas and assumptions behind each estimate.
+
+## Quality
+
+Each component has a quality metric that helps you compare performance.
+
+### Transcriber word error rate (WER)
+
+Word error rate (WER) measures the percentage of words a transcriber gets wrong. Lower values are better. A WER of 5% means about 1 in 20 words is transcribed incorrectly.
+
+Vapi sources WER from [Daily](https://www.daily.co/blog/benchmarking-stt-for-voice-agents/), a third-party speech-to-text benchmark. This provides a consistent comparison across providers instead of relying on self-reported accuracy.
+
+### Intelligence metric for models
+
+The **Intelligence** metric measures a model's reasoning and task capability. Vapi sources the score from the [Artificial Analysis](https://artificialanalysis.ai/models) Intelligence Index. This third-party benchmark scores models from 0 to 100 across reasoning and knowledge tasks.
+
+Voice agents run LLMs with reasoning turned off to keep latency low. The displayed scores reflect reasoning-off performance. The smartest voice agents run at around 20-30 intelligence score, be sure to compare models to each other for a relative benchmark.
+
+### Humanness metric for voices
+
+The **Humanness** metric measures how natural a voice sounds. Vapi measures it with the first-party [**Vapi Humanness Index**](https://humannessindex.vapi.ai/). The index uses a blind listening test to score voices from 1 to 100. A higher score means the voice is harder to distinguish from a human.
+
+## Data refresh schedule
+
+Vapi refreshes performance metrics weekly as models change and more call data becomes available. The metrics reflect each model at its last update rather than a continuous live feed.
+
+## Related
+
+<CardGroup cols={2}>
+  <Card title="How voice agent component costs and estimates work" href="/assistants/model-intelligence/understanding-cost">
+    The full formulas and assumptions behind the cost estimate.
+  </Card>
+  <Card title="How voice-to-voice latency works in voice agents" href="/assistants/model-intelligence/understanding-latency">
+    What voice-to-voice latency measures and why it varies by model.
+  </Card>
+</CardGroup>

fern/assistants/model-intelligence/overview.mdx

@@ -0,0 +1,41 @@
+---
+title: Model Intelligence for transcribers, models, and voices
+subtitle: Choose assistant components with curated Model Presets and Performance Metrics.
+description: Model Intelligence helps you choose transcribers, models, and voices with curated presets and weekly refreshed performance metrics for latency, cost, and quality.
+slug: assistants/model-intelligence/overview
+---
+
+Model Intelligence is a Vapi feature that helps you choose the transcriber, model, and voice for your assistant. Choose a preset for a strong default combination or compare every option using weekly refreshed performance metrics.
+
+## Key concepts
+
+| Term | Definition |
+|---|---|
+| **Model Presets** | Curated configurations that bundle a transcriber, model, and voice tuned for one goal. |
+| **Performance metrics** | The latency, cost, and quality metrics shown on every component, so you can compare options directly. |
+| **Customized** | The state an assistant moves to when you change any single component away from its preset. |
+
+## How it works
+
+When building a voice agent, Vapi orchestrates the transcriber, model, and voice. You can swap each component for any supported provider and model, which gives you flexibility when building an assistant.
+
+**Model Presets** bundle the three components into a curated combination. Choose **Balanced**, **High Intelligence**, **Ultra Fast**, or **Cost Saver** based on your goal. Presets provide a dependable setup without requiring you to tune each component.
+
+**Performance metrics** show latency, cost, and quality data for every transcriber, model, and voice. The data appears on component panels and the dropdown menu for each component. Use it to compare options and build the combination your assistant needs. See the [performance metrics reference](/assistants/model-intelligence/metrics-methodology) for how each metric is sourced and calculated.
+
+If you change one component from its preset, the assistant moves to **Customized**. Every other preset setting remains unchanged, and you can continue editing any component.
+
+## Start with presets and optimize with performance metrics
+
+[Model Presets](/assistants/model-intelligence/presets) give you a dependable starting point when you do not know which component to choose. [Performance metrics](/assistants/model-intelligence/metrics-methodology) help you compare options and optimize your configuration with data instead of guessing.
+
+## Related
+
+<CardGroup cols={2}>
+  <Card title="Model Presets for transcribers, models, and voices" href="/assistants/model-intelligence/presets">
+    See all four presets, when to use each, and how to apply or customize one.
+  </Card>
+  <Card title="Performance metrics and methodology reference" href="/assistants/model-intelligence/metrics-methodology">
+    How every latency, cost, and quality metric is sourced.
+  </Card>
+</CardGroup>

fern/assistants/model-intelligence/presets.mdx

@@ -0,0 +1,119 @@
+---
+title: Model Presets for transcribers, models, and voices
+subtitle: Configure an assistant's transcriber, model, and voice with one preset.
+description: Model presets optimize for balanced performance, reasoning, low latency, or low cost. Learn when to use each preset and how to apply or customize its configuration.
+slug: assistants/model-intelligence/presets
+---
+
+**Model Presets** are curated configurations that bundle a transcriber, model, and voice for a specific goal. Use a preset for a dependable setup or as a starting point for further customization.
+
+## How it works
+
+New assistants use **Balanced** as the default. When you start from one of Vapi’s use-case templates, the assistant uses the preset that best fits the template.
+
+Presets are a starting point rather than a lock - you can fine-tune or swap out any component to better fit your use case.
+
+## Presets at a glance
+
+| Preset | Optimizes for | Best for |
+|---|---|---|
+| **Balanced** | A strong all-round mix of quality, speed, and cost | Most assistants; start here if you're unsure |
+| **High Intelligence** | Reasoning and conversation quality | Complex, nuanced, or high-stakes conversations |
+| **Ultra Fast** | Lowest latency | Fast, responsive, high-volume conversations |
+| **Cost Saver** | Lowest cost per minute | Simple, high-volume calls where cost matters most |
+
+## Choose a preset
+
+Choose the preset that best matches your use case. Start with **Balanced** if you are unsure.
+
+### Balanced
+
+**Balanced** is the default for new assistants and the best starting point for most use cases. It provides a strong mix of quality, responsiveness, and cost.
+
+Choose **Balanced** when any of the following apply.
+
+- You're building a new assistant and aren't sure which preset fits.
+- Your use case covers support, scheduling, FAQs, or qualification.
+- You want an all-around dependable setup out of the box.
+
+<Note>
+If you later need more reasoning, faster responses, or lower cost, switch to the preset built for that.
+</Note>
+
+### High Intelligence
+
+**High Intelligence** prioritizes reasoning and conversation quality. Use it when the assistant needs to handle nuance, follow multi-step logic, or reason reliably across tools and context.
+
+Choose **High Intelligence** when any of the following apply.
+
+- Conversations are complex, open-ended, or high-stakes.
+- The assistant needs to reason through multi-step problems or use tools reliably.
+- Accuracy and quality matter more than speed or cost.
+
+<Note>
+More capable models respond a little slower and cost more per minute than **Balanced**. See [how latency works](/assistants/model-intelligence/understanding-latency) for why more capable models take longer to respond.
+</Note>
+
+### Ultra Fast
+
+**Ultra Fast** prioritizes low latency and responsive conversations. Use it when response speed matters more than complex reasoning.
+
+Choose **Ultra Fast** when any of the following apply.
+
+- Responsiveness is the priority and replies should feel fast.
+- Calls are high-volume and relatively straightforward.
+- The flow is scripted or transactional rather than open-ended.
+
+<Note>
+The fastest models are smaller and less capable, so **Ultra Fast** may not suit complex reasoning or heavy tool use. Choose **Balanced** or **High Intelligence** if difficult tasks produce quality issues.
+</Note>
+
+### Cost Saver
+
+**Cost Saver** prioritizes the lowest cost per minute. Use it for high-volume calls when cost is the main constraint.
+
+Choose **Cost Saver** when any of the following apply.
+
+- Call volume is high and you're optimizing spend.
+- Interactions are simple or scripted.
+- A small quality trade-off is acceptable in exchange for lower cost.
+
+<Note>
+The lowest-cost models are less capable, so this preset fits simpler interactions best. See [how cost works](/assistants/model-intelligence/understanding-cost) for what drives cost per minute.
+</Note>
+
+## The Customized state
+
+Swapping a model out of any component moves your assistant to **Customized** and the **Performance Metrics** displayed will update to match the new configuration.
+
+## Apply a Model Preset
+
+<Steps>
+  <Step title="Open the Assistant tab">
+    Open your assistant in the Vapi Dashboard, then open **Assistant**.
+  </Step>
+  <Step title="Choose a preset">
+    Choose an option under **Model Presets**. The transcriber, model, voice, and displayed totals update to match the preset.
+  </Step>
+  <Step title="Customize components if needed">
+    Click the pencil icon on the transcriber, model, or voice panel. Use the dropdown menu in the settings panel to switch out the model or provider. Your assistant moves to **Customized**. **Performance Metrics** update to reflect your choices.
+  </Step>
+  <Step title="Publish your changes">
+    Click **Publish** to apply your changes.
+  </Step>
+</Steps>
+
+## Verify the configuration
+
+The option you chose is highlighted under **Model Presets**. Confirm the component panels and displayed totals match the preset or your custom configuration.
+
+## Related
+
+<CardGroup cols={2}>
+  <Card title="Model Intelligence for transcribers, models, and voices" href="/assistants/model-intelligence/overview">
+    Learn how presets and performance metrics help you choose components.
+  </Card>
+  <Card title="Performance metrics and methodology reference" href="/assistants/model-intelligence/metrics-methodology">
+    How every latency, cost, and quality metric is sourced.
+  </Card>
+</CardGroup>

fern/assistants/model-intelligence/understanding-cost.mdx

@@ -0,0 +1,97 @@
+---
+title: How voice agent component costs and estimates work
+subtitle: Learn what drives costs across the transcriber, model, and voice.
+description: Understand how Vapi estimates transcriber, model, and voice costs per minute, including prompt caching assumptions and the effects of conversation history.
+slug: assistants/model-intelligence/understanding-cost
+---
+
+The cost to run a voice agent mainly depends on the costs of the transcriber, LLM, and voice. Each component uses a different billing unit, so we've made some assumptions in order to provide a way for you to compare estimated cost per minute.
+
+<Note>
+The Cost number within **Performance Metrics** provide estimates for comparison rather than exact quotes. Use them to compare models consistently and understand what affects cost. See [Vapi pricing](/pricing) for current pricing. For contracted customers, certain pricing may depend on your agreement, please contact your account team for contract-specific rates.
+</Note>
+
+## Key concepts
+
+| Term | Definition |
+|---|---|
+| Input tokens per minute | Tokens sent to the model each minute based on your prompt size, tool-definition size, and a typical request rate. |
+| Effective input rate | The per-token input rate blended between standard and cached rates when the model supports prompt caching. |
+| Output tokens per minute | Tokens the model generates each minute based on typical Vapi call data. |
+
+## Component billing units
+
+Each component uses a different billing unit.
+
+| Component | Billing unit |
+|---|---|
+| Transcriber | Minutes of audio |
+| Model | 1 million input and output tokens |
+| Voice | Characters of spoken text |
+
+### Transcriber cost
+
+Transcription is billed per minute of audio. Vapi transcribes audio from both the caller and the assistant. The estimate therefore includes two audio channels. This cost scales with call duration rather than your prompt or configuration.
+
+### Voice cost
+
+The Voice model is billed per character of spoken text. The estimate uses the number of characters spoken in a typical minute. Cost increases when the assistant speaks more.
+
+### LLM cost
+
+Model cost varies more than transcriber or voice cost because it heavily depends on the assistant's configuration. Vapi calculates the estimate from the assistant's actual prompt and tool definitions, so two assistants using the same LLM model can show different cost estimates.
+
+The estimate uses each provider's input and output token rates.
+
+| Calculation | Formula |
+|---|---|
+| **Input cost** | Input tokens per minute × effective input rate |
+| **Output cost** | Output tokens per minute × output rate |
+| **Estimated model cost per minute** | Input cost + output cost |
+
+#### Input tokens per minute
+
+**Input tokens per minute** is the largest cost factor you can control. Vapi calculates it from your [system prompt](/prompting-guide), [tool definitions](/tools), and a typical number of model requests per minute. Longer prompts and more tools increase the estimate, while leaner configurations reduce it.
+
+Vapi estimates 5 requests per turn. We use number of requests rather than conversational turns, since a single turn can trigger multiple requests, and providers bill each request.
+
+#### Effective input rate
+
+**Effective input rate** accounts for prompt caching. Cached input is stable prompt content that providers reuse across requests. Many providers offer a discounted rate for this input.
+
+When a model supports caching, Vapi blends its standard and cached rates based on an assumed 50% cache hit rate. Otherwise, Vapi uses the standard rate. The calculation uses the actual rates for each model and provider.
+
+#### Output tokens per minute
+
+Vapi estimates 150 **output tokens per minute** based on typical generated speech in production call data. It prices those tokens at the model’s output rate.
+
+## Estimate limitations
+
+Two assumptions can cause actual model cost to differ from the estimate.
+
+| Scenario | Typical effect | Reason |
+|---|---|---|
+| Large prompts with caching supported | Actual cost is often lower | Actual cache use is usually higher than the estimate assumes |
+| Long calls | Actual cost is often higher | Growing conversation history is sent again with each request |
+
+These deliberate simplifications keep estimates comparable across models. Vapi may refine them over time.
+
+## Reduce cost
+
+You can reduce cost in four ways.
+
+- Shorten your system prompt and tool definitions. This often has the largest effect on model cost.
+- Use models that support prompt caching for large, stable prompts. Providers charge less for prompt content reused from cache, which lowers model cost.
+- Choose a lower-cost model when the use case allows. The [Cost Saver preset](/assistants/model-intelligence/presets#cost-saver) optimizes components for the lowest cost per minute.
+- Keep calls focused. Shorter, more focused conversations cost less and tend to provide a better experience.
+
+## Related
+
+<CardGroup cols={2}>
+  <Card title="Model presets for transcribers, models, and voices" href="/assistants/model-intelligence/presets">
+    Compare presets or apply Cost Saver.
+  </Card>
+  <Card title="Performance metrics and methodology reference" href="/assistants/model-intelligence/metrics-methodology">
+    See how Vapi calculates model cost estimates.
+  </Card>
+</CardGroup>