From 76e68f3456a2e5ce8f7a20ef203eda1ac222fc73 Mon Sep 17 00:00:00 2001 From: Konstantinos Passadis | Azure MVP | MCT Date: Sat, 8 Aug 2026 21:29:33 +0000 Subject: [PATCH 01/11] Add Azure API Management AI Gateway instructions APIM as a GenAI gateway for Microsoft Foundry and OpenAI-compatible LLM APIs: llm-* token-limit/token-metric policies, managed-identity auth, backend pools with circuit breakers, semantic caching, and content safety. Corrects the azure-openai-*/api-key/element-ordering defaults the model produces unprompted. Regenerates docs/README.instructions.md. --- docs/README.instructions.md | 1 + .../azure-apim-ai-gateway.instructions.md | 196 ++++++++++++++++++ 2 files changed, 197 insertions(+) create mode 100644 instructions/azure-apim-ai-gateway.instructions.md diff --git a/docs/README.instructions.md b/docs/README.instructions.md index 0583836bd..c4546df99 100644 --- a/docs/README.instructions.md +++ b/docs/README.instructions.md @@ -31,6 +31,7 @@ See [CONTRIBUTING.md](../CONTRIBUTING.md#adding-instructions) for guidelines on | [ASP.NET REST API Development](../instructions/aspnet-rest-apis.instructions.md)
[![Install in VS Code](https://img.shields.io/badge/VS_Code-Install-0098FF?style=flat-square&logo=visualstudiocode&logoColor=white)](https://aka.ms/awesome-copilot/install/instructions?url=vscode%3Achat-instructions%2Finstall%3Furl%3Dhttps%3A%2F%2Fraw.githubusercontent.com%2Fgithub%2Fawesome-copilot%2Fmain%2Finstructions%2Faspnet-rest-apis.instructions.md)
[![Install in VS Code Insiders](https://img.shields.io/badge/VS_Code_Insiders-Install-24bfa5?style=flat-square&logo=visualstudiocode&logoColor=white)](https://aka.ms/awesome-copilot/install/instructions?url=vscode-insiders%3Achat-instructions%2Finstall%3Furl%3Dhttps%3A%2F%2Fraw.githubusercontent.com%2Fgithub%2Fawesome-copilot%2Fmain%2Finstructions%2Faspnet-rest-apis.instructions.md) | Guidelines for building REST APIs with ASP.NET | | [Astro Development Instructions](../instructions/astro.instructions.md)
[![Install in VS Code](https://img.shields.io/badge/VS_Code-Install-0098FF?style=flat-square&logo=visualstudiocode&logoColor=white)](https://aka.ms/awesome-copilot/install/instructions?url=vscode%3Achat-instructions%2Finstall%3Furl%3Dhttps%3A%2F%2Fraw.githubusercontent.com%2Fgithub%2Fawesome-copilot%2Fmain%2Finstructions%2Fastro.instructions.md)
[![Install in VS Code Insiders](https://img.shields.io/badge/VS_Code_Insiders-Install-24bfa5?style=flat-square&logo=visualstudiocode&logoColor=white)](https://aka.ms/awesome-copilot/install/instructions?url=vscode-insiders%3Achat-instructions%2Finstall%3Furl%3Dhttps%3A%2F%2Fraw.githubusercontent.com%2Fgithub%2Fawesome-copilot%2Fmain%2Finstructions%2Fastro.instructions.md) | Astro 7 development standards and best practices for content-driven websites | | [AWS AppSync Event API Instructions](../instructions/aws-appsync.instructions.md)
[![Install in VS Code](https://img.shields.io/badge/VS_Code-Install-0098FF?style=flat-square&logo=visualstudiocode&logoColor=white)](https://aka.ms/awesome-copilot/install/instructions?url=vscode%3Achat-instructions%2Finstall%3Furl%3Dhttps%3A%2F%2Fraw.githubusercontent.com%2Fgithub%2Fawesome-copilot%2Fmain%2Finstructions%2Faws-appsync.instructions.md)
[![Install in VS Code Insiders](https://img.shields.io/badge/VS_Code_Insiders-Install-24bfa5?style=flat-square&logo=visualstudiocode&logoColor=white)](https://aka.ms/awesome-copilot/install/instructions?url=vscode-insiders%3Achat-instructions%2Finstall%3Furl%3Dhttps%3A%2F%2Fraw.githubusercontent.com%2Fgithub%2Fawesome-copilot%2Fmain%2Finstructions%2Faws-appsync.instructions.md) | Production-grade guidance for AWS AppSync Event API handlers using APPSYNC_JS runtime restrictions, utilities, modules, and datasource patterns | +| [Azure API Management as an AI Gateway Instructions](../instructions/azure-apim-ai-gateway.instructions.md)
[![Install in VS Code](https://img.shields.io/badge/VS_Code-Install-0098FF?style=flat-square&logo=visualstudiocode&logoColor=white)](https://aka.ms/awesome-copilot/install/instructions?url=vscode%3Achat-instructions%2Finstall%3Furl%3Dhttps%3A%2F%2Fraw.githubusercontent.com%2Fgithub%2Fawesome-copilot%2Fmain%2Finstructions%2Fazure-apim-ai-gateway.instructions.md)
[![Install in VS Code Insiders](https://img.shields.io/badge/VS_Code_Insiders-Install-24bfa5?style=flat-square&logo=visualstudiocode&logoColor=white)](https://aka.ms/awesome-copilot/install/instructions?url=vscode-insiders%3Achat-instructions%2Finstall%3Furl%3Dhttps%3A%2F%2Fraw.githubusercontent.com%2Fgithub%2Fawesome-copilot%2Fmain%2Finstructions%2Fazure-apim-ai-gateway.instructions.md) | Configure Azure API Management as an AI (GenAI) gateway in front of Microsoft Foundry and other LLM APIs: token-limit and token-metric policies, managed-identity auth, backend load balancing with circuit breakers, semantic caching, and content safety — plus the policy mistakes Copilot makes by default. | | [Azure DevOps Pipeline YAML Best Practices](../instructions/azure-devops-pipelines.instructions.md)
[![Install in VS Code](https://img.shields.io/badge/VS_Code-Install-0098FF?style=flat-square&logo=visualstudiocode&logoColor=white)](https://aka.ms/awesome-copilot/install/instructions?url=vscode%3Achat-instructions%2Finstall%3Furl%3Dhttps%3A%2F%2Fraw.githubusercontent.com%2Fgithub%2Fawesome-copilot%2Fmain%2Finstructions%2Fazure-devops-pipelines.instructions.md)
[![Install in VS Code Insiders](https://img.shields.io/badge/VS_Code_Insiders-Install-24bfa5?style=flat-square&logo=visualstudiocode&logoColor=white)](https://aka.ms/awesome-copilot/install/instructions?url=vscode-insiders%3Achat-instructions%2Finstall%3Furl%3Dhttps%3A%2F%2Fraw.githubusercontent.com%2Fgithub%2Fawesome-copilot%2Fmain%2Finstructions%2Fazure-devops-pipelines.instructions.md) | Best practices for Azure DevOps Pipeline YAML files | | [Azure Durable Functions C# Development](../instructions/azure-durable-functions-csharp.instructions.md)
[![Install in VS Code](https://img.shields.io/badge/VS_Code-Install-0098FF?style=flat-square&logo=visualstudiocode&logoColor=white)](https://aka.ms/awesome-copilot/install/instructions?url=vscode%3Achat-instructions%2Finstall%3Furl%3Dhttps%3A%2F%2Fraw.githubusercontent.com%2Fgithub%2Fawesome-copilot%2Fmain%2Finstructions%2Fazure-durable-functions-csharp.instructions.md)
[![Install in VS Code Insiders](https://img.shields.io/badge/VS_Code_Insiders-Install-24bfa5?style=flat-square&logo=visualstudiocode&logoColor=white)](https://aka.ms/awesome-copilot/install/instructions?url=vscode-insiders%3Achat-instructions%2Finstall%3Furl%3Dhttps%3A%2F%2Fraw.githubusercontent.com%2Fgithub%2Fawesome-copilot%2Fmain%2Finstructions%2Fazure-durable-functions-csharp.instructions.md) | Guidelines and best practices for building Azure Durable Functions in C# using the isolated worker model | | [Azure Functions C# Development](../instructions/azure-functions-csharp.instructions.md)
[![Install in VS Code](https://img.shields.io/badge/VS_Code-Install-0098FF?style=flat-square&logo=visualstudiocode&logoColor=white)](https://aka.ms/awesome-copilot/install/instructions?url=vscode%3Achat-instructions%2Finstall%3Furl%3Dhttps%3A%2F%2Fraw.githubusercontent.com%2Fgithub%2Fawesome-copilot%2Fmain%2Finstructions%2Fazure-functions-csharp.instructions.md)
[![Install in VS Code Insiders](https://img.shields.io/badge/VS_Code_Insiders-Install-24bfa5?style=flat-square&logo=visualstudiocode&logoColor=white)](https://aka.ms/awesome-copilot/install/instructions?url=vscode-insiders%3Achat-instructions%2Finstall%3Furl%3Dhttps%3A%2F%2Fraw.githubusercontent.com%2Fgithub%2Fawesome-copilot%2Fmain%2Finstructions%2Fazure-functions-csharp.instructions.md) | Guidelines and best practices for building Azure Functions in C# using the isolated worker model | diff --git a/instructions/azure-apim-ai-gateway.instructions.md b/instructions/azure-apim-ai-gateway.instructions.md new file mode 100644 index 000000000..bc9820a07 --- /dev/null +++ b/instructions/azure-apim-ai-gateway.instructions.md @@ -0,0 +1,196 @@ +--- +description: 'Configure Azure API Management as an AI (GenAI) gateway in front of Microsoft Foundry and other LLM APIs: token-limit and token-metric policies, managed-identity auth, backend load balancing with circuit breakers, semantic caching, and content safety — plus the policy mistakes Copilot makes by default.' +applyTo: "**/*.xml, **/policies/**, **/*.bicep" +--- + +# Azure API Management as an AI Gateway Instructions + +Guidance for putting **Azure API Management (APIM)** in front of **Microsoft Foundry** model deployments (Azure OpenAI and other providers) and any OpenAI-compatible LLM API, using APIM's **AI gateway** policy set. When this guidance conflicts with your training data, **follow this file** and verify against Microsoft Learn: https://learn.microsoft.com/azure/api-management/genai-gateway-capabilities + +> **Field note (why this file exists):** In Copilot-assisted APIM work, the default behavior is to (1) reach for the older provider-specific `azure-openai-*` policies instead of the current provider-agnostic `llm-*` ones, (2) throttle LLM traffic with request-count policies (`rate-limit-by-key`) that don't understand tokens, (3) authenticate to the model backend with an `api-key` header pulled from a named value instead of a managed identity, and (4) emit policy elements in an arbitrary order — which APIM rejects, because **AI gateway policy elements are order-sensitive**. These instructions front-load those corrections so Copilot produces a valid, secure gateway policy on the first pass. When in doubt, ground against Microsoft Learn or the Microsoft Docs MCP server — the AI gateway policy surface changes frequently. + +## Core rules + +- **Prefer the provider-agnostic `llm-*` policies** (`llm-token-limit`, `llm-emit-token-metric`, `llm-semantic-cache-lookup`/`-store`, `llm-content-safety`). They work across OpenAI Chat Completions/Responses, Anthropic Messages (v2 tiers), and Google Vertex AI. Only use the `azure-openai-*` variants when the API is exclusively Azure OpenAI and you have a reason to. +- **Throttle by tokens, not by call count**, for LLM APIs. `rate-limit-by-key` counts requests and is blind to token cost; use `llm-token-limit`. +- **Authenticate to Foundry with a managed identity**, never a stored key. Give APIM's identity the **Cognitive Services OpenAI User** role on the Foundry resource. +- **Respect policy element order.** Set elements and child elements in the order documented for each policy, and keep `` in each section (`inbound`, `backend`, `outbound`, `on-error`). +- **The `llm-*` and `azure-openai-*` metric/token policies are not available on the Consumption tier.** Check tier support before recommending them. +- Prefer configuring an APIM **backend** resource (with managed-identity credentials) over inline `authentication-managed-identity` + `set-header`; importing a Foundry API wires this up automatically. + +## Token rate limiting and quotas — `llm-token-limit` + +Enforce a tokens-per-minute (TPM) rate limit, a token quota over a fixed window, or both, keyed off any counter (subscription id, IP, JWT claim, custom header). Set `estimate-prompt-tokens="true"` to reject over-limit prompts **before** they hit the backend. + +```xml + + +``` + +- Exceeding **`tokens-per-minute`** returns `429 Too Many Requests`; exceeding **`token-quota`** returns `403 Forbidden`. Handle both distinctly in clients — a 403 here is a quota exhaustion signal, not an auth failure. +- `token-quota-period` must be one of `Hourly`, `Daily`, `Weekly`, `Monthly`, `Yearly`. +- At least one of `tokens-per-minute` or (`token-quota` + `token-quota-period`) is required; `counter-key` is always required. +- Use a policy expression for `counter-key` to limit per app/team/tenant, e.g. `@(context.Request.Headers.GetValueOrDefault("x-team-id","anon"))`. + +## Observability — `llm-emit-token-metric` + +Emit prompt/completion/total token metrics to **Application Insights** so you can attribute spend per consumer. Add dimensions to slice the metric later in Azure Monitor. + +```xml + + + + + + +``` + +- Requires an Application Insights logger wired to the APIM instance. Also enable LLM request logging to capture prompts/completions for auditing. +- Not available on the Consumption tier. + +## Authentication — managed identity, not keys + +Give APIM's managed identity the **Cognitive Services OpenAI User** role on the Foundry resource, then authenticate at the gateway. Inline form: + +```xml + + + + @("Bearer " + (string)context.Variables["managed-id-access-token"]) + +``` + +Preferred form: configure a **backend** with managed-identity credentials to `https://cognitiveservices.azure.com/` and reference it with ``. This is what APIM sets up when you import a Foundry API directly. + +## Resiliency — backend pools, load balancing, and circuit breakers + +Do **not** hand-roll retry/failover across multiple Foundry endpoints in application code. Define an APIM **backend pool** and let the gateway load-balance (round-robin, weighted, priority, session-aware) with a **circuit breaker** that honors the backend `Retry-After` header. Use `priority` to prefer PTU endpoints and fall back to pay-as-you-go. + +```bicep +resource pool 'Microsoft.ApiManagement/service/backends@2023-09-01-preview' = { + name: '${apimName}/foundry-pool' + properties: { + description: 'Load-balanced Foundry endpoints' + type: 'Pool' + pool: { + services: [ + { id: backend1.id, priority: 1, weight: 1 } // PTU — preferred + { id: backend2.id, priority: 2, weight: 1 } // PayGo — fallback + ] + } + } +} + +resource backend1 'Microsoft.ApiManagement/service/backends@2023-09-01-preview' = { + name: '${apimName}/foundry-ptu' + properties: { + url: 'https://.openai.azure.com/openai' + protocol: 'http' + circuitBreaker: { + rules: [ + { + name: 'trip-on-5xx' + failureCondition: { + count: 3 + interval: 'PT1H' + statusCodeRanges: [ { min: 500, max: 599 } ] + errorReasons: [ 'Server errors' ] + } + tripDuration: 'PT1H' + acceptRetryAfter: true + } + ] + } + } +} +``` + +## Semantic caching — `llm-semantic-cache-lookup` / `-store` + +Cache completions by vector proximity of the prompt to reduce token spend and latency. This is **not** in-memory caching: it requires an external **RediSearch-compatible cache** (e.g., Azure Managed Redis) onboarded to APIM and an **embeddings backend**. Lookup goes in `inbound`, store in `outbound`. + +```xml + + + @(context.Subscription.Id) + +``` + +```xml + + +``` + +- Lower `score-threshold` = stricter match (fewer cache hits, higher fidelity). Tune per use case; start around `0.05`–`0.15`. +- Partition the cache per tenant/consumer with `` so users never receive another consumer's cached completion. + +## Content safety — `llm-content-safety` + +Screen prompts (and optionally responses) through **Azure AI Content Safety** before they reach the model. Configure a content-safety backend and set severity thresholds; `shield-prompt="true"` adds jailbreak/prompt-injection detection. + +```xml + + + + + + + +``` + +- Thresholds use 0–7 severity: a `threshold="4"` allows 0–3 and blocks 4–7. Raise to be more permissive, lower to be stricter. + +## Policy skeleton and ordering + +Keep AI gateway policies in the correct sections and preserve ``: + +```xml + + + + + + + @("Bearer " + (string)context.Variables["mi"]) + + + + + + + + + + + + + + + + + + + +``` + +## Foundry-native AI gateway + +Foundry has **built-in integration** with APIM: from the Foundry portal you can create or attach an APIM instance as an AI gateway and set per-deployment TPM limits and quotas from **Operate → Admin → AI Gateway → Token management** — these map directly to `llm-token-limit`. The gateway can also govern registered **agents** (running anywhere) and **MCP tools**, surfacing them in the Foundry inventory with policy enforcement and telemetry. When a repo uses this integration, keep custom policies compatible with the Foundry-managed configuration rather than overriding it. Prefer importing the API as a **Language Model API** or **Azure AI Foundry API** in APIM, which auto-creates the backend, `set-backend-service`, and optional token/caching/safety policies. + +## Grounding + +The AI gateway policy set evolves quickly. Before finalizing policy XML, verify element names, attributes, and ordering against Microsoft Learn (`genai-gateway-capabilities`, the per-policy reference pages) or the Microsoft Docs MCP server rather than relying on training data. From 6683218053580ba81615e54ddcfc8b5ca08e9c03 Mon Sep 17 00:00:00 2001 From: Konstantinos Passadis | Azure MVP | MCT Date: Sun, 9 Aug 2026 01:09:53 +0300 Subject: [PATCH 02/11] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- instructions/azure-apim-ai-gateway.instructions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/instructions/azure-apim-ai-gateway.instructions.md b/instructions/azure-apim-ai-gateway.instructions.md index bc9820a07..bdc224d6e 100644 --- a/instructions/azure-apim-ai-gateway.instructions.md +++ b/instructions/azure-apim-ai-gateway.instructions.md @@ -102,7 +102,7 @@ resource backend1 'Microsoft.ApiManagement/service/backends@2023-09-01-preview' failureCondition: { count: 3 interval: 'PT1H' - statusCodeRanges: [ { min: 500, max: 599 } ] + statusCodeRanges: [ { min: 429, max: 429 }, { min: 500, max: 599 } ] errorReasons: [ 'Server errors' ] } tripDuration: 'PT1H' From 376a9df2f7f03e198b41414d8faa998e3ff2c09e Mon Sep 17 00:00:00 2001 From: Konstantinos Passadis | Azure MVP | MCT Date: Sat, 8 Aug 2026 22:29:09 +0000 Subject: [PATCH 03/11] Address Copilot review feedback on APIM AI gateway instructions --- .../azure-apim-ai-gateway.instructions.md | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/instructions/azure-apim-ai-gateway.instructions.md b/instructions/azure-apim-ai-gateway.instructions.md index bdc224d6e..81c9337e9 100644 --- a/instructions/azure-apim-ai-gateway.instructions.md +++ b/instructions/azure-apim-ai-gateway.instructions.md @@ -15,7 +15,7 @@ Guidance for putting **Azure API Management (APIM)** in front of **Microsoft Fou - **Throttle by tokens, not by call count**, for LLM APIs. `rate-limit-by-key` counts requests and is blind to token cost; use `llm-token-limit`. - **Authenticate to Foundry with a managed identity**, never a stored key. Give APIM's identity the **Cognitive Services OpenAI User** role on the Foundry resource. - **Respect policy element order.** Set elements and child elements in the order documented for each policy, and keep `` in each section (`inbound`, `backend`, `outbound`, `on-error`). -- **The `llm-*` and `azure-openai-*` metric/token policies are not available on the Consumption tier.** Check tier support before recommending them. +- **Check tier support per policy — it varies.** `llm-token-limit` is not available on the Consumption tier; `llm-emit-token-metric`, `llm-semantic-cache-*`, and `llm-content-safety` apply to all tiers (including Consumption). Verify each policy's "Applies to" line rather than assuming. - Prefer configuring an APIM **backend** resource (with managed-identity credentials) over inline `authentication-managed-identity` + `set-header`; importing a Foundry API wires this up automatically. ## Token rate limiting and quotas — `llm-token-limit` @@ -52,7 +52,8 @@ Emit prompt/completion/total token metrics to **Application Insights** so you ca ``` - Requires an Application Insights logger wired to the APIM instance. Also enable LLM request logging to capture prompts/completions for auditing. -- Not available on the Consumption tier. +- Metrics come from the `usage` section of the model response. Some OpenAI models — **especially when streaming** — omit token counts unless the request sets `include_usage: true` (`stream_options`), and an interrupted stream yields inaccurate counts. Ensure clients enable usage reporting or the metric will be silently incomplete. +- Applies to all API Management tiers (including Consumption). A maximum of 5 custom dimensions per policy. ## Authentication — managed identity, not keys @@ -98,7 +99,7 @@ resource backend1 'Microsoft.ApiManagement/service/backends@2023-09-01-preview' circuitBreaker: { rules: [ { - name: 'trip-on-5xx' + name: 'trip-on-backend-failures' failureCondition: { count: 3 interval: 'PT1H' @@ -112,6 +113,14 @@ resource backend1 'Microsoft.ApiManagement/service/backends@2023-09-01-preview' } } } + +resource backend2 'Microsoft.ApiManagement/service/backends@2023-09-01-preview' = { + name: '${apimName}/foundry-paygo' + properties: { + url: 'https://.openai.azure.com/openai' + protocol: 'http' + } +} ``` ## Semantic caching — `llm-semantic-cache-lookup` / `-store` @@ -189,7 +198,7 @@ Keep AI gateway policies in the correct sections and preserve ``: ## Foundry-native AI gateway -Foundry has **built-in integration** with APIM: from the Foundry portal you can create or attach an APIM instance as an AI gateway and set per-deployment TPM limits and quotas from **Operate → Admin → AI Gateway → Token management** — these map directly to `llm-token-limit`. The gateway can also govern registered **agents** (running anywhere) and **MCP tools**, surfacing them in the Foundry inventory with policy enforcement and telemetry. When a repo uses this integration, keep custom policies compatible with the Foundry-managed configuration rather than overriding it. Prefer importing the API as a **Language Model API** or **Azure AI Foundry API** in APIM, which auto-creates the backend, `set-backend-service`, and optional token/caching/safety policies. +Foundry has **built-in integration** with APIM: from the Foundry portal you can create a new APIM instance or attach an existing one as an AI gateway, then set per-deployment TPM limits and quotas from **Operate → Admin → AI Gateway → Token management** — these map directly to `llm-token-limit`. Attaching an **existing** APIM instance has hard requirements: it must be in the **same Microsoft Entra tenant and subscription** as the Foundry resource, be a **v2 tier** (Basic v2 / Standard v2 / Premium v2), and you need at least the **API Management Service Contributor** (or Owner) role on it — otherwise it won't appear as selectable. The gateway can also govern registered **agents** (running anywhere) and, in **preview**, **MCP tools** (only new MCP tools that don't use managed OAuth are routed; policies are applied in the Azure portal, not the Foundry portal). When a repo uses this integration, keep custom policies compatible with the Foundry-managed configuration rather than overriding it. Prefer importing the API as a **Language Model API** or **Azure AI Foundry API** in APIM, which auto-creates the backend, `set-backend-service`, and optional token/caching/safety policies. ## Grounding From ad80db55ebbc92efa46888e0272610b4ca4a5854 Mon Sep 17 00:00:00 2001 From: Konstantinos Passadis | Azure MVP | MCT Date: Sat, 8 Aug 2026 22:59:40 +0000 Subject: [PATCH 04/11] Fix AI gateway auth scope and Consumption-tier note per review --- instructions/azure-apim-ai-gateway.instructions.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/instructions/azure-apim-ai-gateway.instructions.md b/instructions/azure-apim-ai-gateway.instructions.md index 81c9337e9..844419cdd 100644 --- a/instructions/azure-apim-ai-gateway.instructions.md +++ b/instructions/azure-apim-ai-gateway.instructions.md @@ -57,7 +57,12 @@ Emit prompt/completion/total token metrics to **Application Insights** so you ca ## Authentication — managed identity, not keys -Give APIM's managed identity the **Cognitive Services OpenAI User** role on the Foundry resource, then authenticate at the gateway. Inline form: +The RBAC role **and the token audience depend on the model type** — this trips people up because the OpenAI role and audience don't work for other Foundry models: + +- **Azure OpenAI deployments** → assign **Cognitive Services OpenAI User**; token audience `https://cognitiveservices.azure.com`. +- **Non-OpenAI Foundry Models** (DeepSeek, Llama, Grok, and other models sold by Azure) → assign **Cognitive Services User** (the OpenAI role does **not** grant access to these); token audience `https://ai.azure.com`. + +Assign the role to APIM's managed identity on the Foundry resource, then authenticate at the gateway. Inline form (Azure OpenAI shown — swap `resource` to `https://ai.azure.com` for non-OpenAI Foundry models): ```xml @@ -70,7 +75,7 @@ Give APIM's managed identity the **Cognitive Services OpenAI User** role on the ``` -Preferred form: configure a **backend** with managed-identity credentials to `https://cognitiveservices.azure.com/` and reference it with ``. This is what APIM sets up when you import a Foundry API directly. +Preferred form: configure a **backend** with managed-identity credentials to the matching audience (`https://cognitiveservices.azure.com/` for Azure OpenAI, `https://ai.azure.com/` for other Foundry models) and reference it with ``. This is what APIM sets up when you import a Foundry API directly. ## Resiliency — backend pools, load balancing, and circuit breakers @@ -103,7 +108,7 @@ resource backend1 'Microsoft.ApiManagement/service/backends@2023-09-01-preview' failureCondition: { count: 3 interval: 'PT1H' - statusCodeRanges: [ { min: 429, max: 429 }, { min: 500, max: 599 } ] + statusCodeRanges: [ { min: 500, max: 599 } ] errorReasons: [ 'Server errors' ] } tripDuration: 'PT1H' From 9cba8601f795ed3dc7557d5fd3caeb37cc979638 Mon Sep 17 00:00:00 2001 From: Konstantinos Passadis | Azure MVP | MCT Date: Sat, 8 Aug 2026 23:18:38 +0000 Subject: [PATCH 05/11] Fix circuit-breaker 429, auth-role contradiction, and malformed XML per review --- instructions/azure-apim-ai-gateway.instructions.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/instructions/azure-apim-ai-gateway.instructions.md b/instructions/azure-apim-ai-gateway.instructions.md index 844419cdd..8e873e317 100644 --- a/instructions/azure-apim-ai-gateway.instructions.md +++ b/instructions/azure-apim-ai-gateway.instructions.md @@ -13,7 +13,7 @@ Guidance for putting **Azure API Management (APIM)** in front of **Microsoft Fou - **Prefer the provider-agnostic `llm-*` policies** (`llm-token-limit`, `llm-emit-token-metric`, `llm-semantic-cache-lookup`/`-store`, `llm-content-safety`). They work across OpenAI Chat Completions/Responses, Anthropic Messages (v2 tiers), and Google Vertex AI. Only use the `azure-openai-*` variants when the API is exclusively Azure OpenAI and you have a reason to. - **Throttle by tokens, not by call count**, for LLM APIs. `rate-limit-by-key` counts requests and is blind to token cost; use `llm-token-limit`. -- **Authenticate to Foundry with a managed identity**, never a stored key. Give APIM's identity the **Cognitive Services OpenAI User** role on the Foundry resource. +- **Authenticate to Foundry with a managed identity**, never a stored key. The exact role and token audience depend on the model type (see [Authentication](#authentication--managed-identity-not-keys)) — Azure OpenAI uses **Cognitive Services OpenAI User** (`https://cognitiveservices.azure.com`); other Foundry models use **Cognitive Services User** (`https://ai.azure.com`). - **Respect policy element order.** Set elements and child elements in the order documented for each policy, and keep `` in each section (`inbound`, `backend`, `outbound`, `on-error`). - **Check tier support per policy — it varies.** `llm-token-limit` is not available on the Consumption tier; `llm-emit-token-metric`, `llm-semantic-cache-*`, and `llm-content-safety` apply to all tiers (including Consumption). Verify each policy's "Applies to" line rather than assuming. - Prefer configuring an APIM **backend** resource (with managed-identity credentials) over inline `authentication-managed-identity` + `set-header`; importing a Foundry API wires this up automatically. @@ -47,7 +47,7 @@ Emit prompt/completion/total token metrics to **Application Insights** so you ca - + ``` @@ -108,7 +108,8 @@ resource backend1 'Microsoft.ApiManagement/service/backends@2023-09-01-preview' failureCondition: { count: 3 interval: 'PT1H' - statusCodeRanges: [ { min: 500, max: 599 } ] + // 429 = PTU/TPM saturation (honors Retry-After), 5xx = backend failure + statusCodeRanges: [ { min: 429, max: 429 }, { min: 500, max: 599 } ] errorReasons: [ 'Server errors' ] } tripDuration: 'PT1H' From b452228a100350c8bb31cd35b17a5c43afc3c1de Mon Sep 17 00:00:00 2001 From: Konstantinos Passadis | Azure MVP | MCT Date: Sun, 9 Aug 2026 01:12:56 +0000 Subject: [PATCH 06/11] Harden semantic cache isolation, logging privacy, and policy order per review --- instructions/azure-apim-ai-gateway.instructions.md | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/instructions/azure-apim-ai-gateway.instructions.md b/instructions/azure-apim-ai-gateway.instructions.md index 8e873e317..5494e058d 100644 --- a/instructions/azure-apim-ai-gateway.instructions.md +++ b/instructions/azure-apim-ai-gateway.instructions.md @@ -51,7 +51,7 @@ Emit prompt/completion/total token metrics to **Application Insights** so you ca ``` -- Requires an Application Insights logger wired to the APIM instance. Also enable LLM request logging to capture prompts/completions for auditing. +- Requires an Application Insights logger wired to the APIM instance. Token metrics do **not** require logging message content. Full prompt/completion logging is a separate, **opt-in** step — enable it only with a clear need, because it can persist PII, secrets, and other sensitive content. If you do, apply field redaction, restrict who can read the logs, set a short retention window, and run it past your compliance/privacy review. - Metrics come from the `usage` section of the model response. Some OpenAI models — **especially when streaming** — omit token counts unless the request sets `include_usage: true` (`stream_options`), and an interrupted stream yields inaccurate counts. Ensure clients enable usage reporting or the metric will be silently incomplete. - Applies to all API Management tiers (including Consumption). A maximum of 5 custom dimensions per policy. @@ -141,7 +141,9 @@ Cache completions by vector proximity of the prompt to reduce token spend and la embeddings-backend-auth="system-assigned" ignore-system-messages="true" max-message-count="10"> - @(context.Subscription.Id) + + @(context.Principal?.Claims.GetValueOrDefault("oid", context.Subscription.Id)) ``` @@ -151,7 +153,7 @@ Cache completions by vector proximity of the prompt to reduce token spend and la ``` - Lower `score-threshold` = stricter match (fewer cache hits, higher fidelity). Tune per use case; start around `0.05`–`0.15`. -- Partition the cache per tenant/consumer with `` so users never receive another consumer's cached completion. +- Partition the cache on the **actual confidentiality boundary** with ``. Keying only on the APIM subscription id means every user sharing that subscription shares one cache partition and can receive each other's cached completions — a data-exposure risk. When responses are user-specific, add an authenticated user/subject identifier (for example a JWT `sub`/`oid` claim via `context.Principal`) to `` so per-user isolation is enforced. ## Content safety — `llm-content-safety` @@ -187,11 +189,14 @@ Keep AI gateway policies in the correct sections and preserve ``: + + - From 076b6b0a44b29d489c35bde49a5b3eaa8ca63011 Mon Sep 17 00:00:00 2001 From: Konstantinos Passadis | Azure MVP | MCT Date: Sun, 9 Aug 2026 01:25:56 +0000 Subject: [PATCH 07/11] Complete emit-token-metric prerequisites and Foundry private-networking requirement --- instructions/azure-apim-ai-gateway.instructions.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/instructions/azure-apim-ai-gateway.instructions.md b/instructions/azure-apim-ai-gateway.instructions.md index 5494e058d..f95311818 100644 --- a/instructions/azure-apim-ai-gateway.instructions.md +++ b/instructions/azure-apim-ai-gateway.instructions.md @@ -51,7 +51,8 @@ Emit prompt/completion/total token metrics to **Application Insights** so you ca ``` -- Requires an Application Insights logger wired to the APIM instance. Token metrics do **not** require logging message content. Full prompt/completion logging is a separate, **opt-in** step — enable it only with a clear need, because it can persist PII, secrets, and other sensitive content. If you do, apply field redaction, restrict who can read the logs, set a short retention window, and run it past your compliance/privacy review. +- Prerequisites for the metric — all three are required or the policy emits nothing usable: an Application Insights logger connected to the APIM instance; **Application Insights logging enabled for the LLM API**; and **custom metrics with dimensions enabled in Application Insights**. +- Emitting token metrics does **not** require logging message content. Full prompt/completion logging is a separate, **opt-in** step — enable it only with a clear need, because it can persist PII, secrets, and other sensitive content. If you do, apply field redaction, restrict who can read the logs, set a short retention window, and run it past your compliance/privacy review. - Metrics come from the `usage` section of the model response. Some OpenAI models — **especially when streaming** — omit token counts unless the request sets `include_usage: true` (`stream_options`), and an interrupted stream yields inaccurate counts. Ensure clients enable usage reporting or the metric will be silently incomplete. - Applies to all API Management tiers (including Consumption). A maximum of 5 custom dimensions per policy. @@ -209,7 +210,7 @@ Keep AI gateway policies in the correct sections and preserve ``: ## Foundry-native AI gateway -Foundry has **built-in integration** with APIM: from the Foundry portal you can create a new APIM instance or attach an existing one as an AI gateway, then set per-deployment TPM limits and quotas from **Operate → Admin → AI Gateway → Token management** — these map directly to `llm-token-limit`. Attaching an **existing** APIM instance has hard requirements: it must be in the **same Microsoft Entra tenant and subscription** as the Foundry resource, be a **v2 tier** (Basic v2 / Standard v2 / Premium v2), and you need at least the **API Management Service Contributor** (or Owner) role on it — otherwise it won't appear as selectable. The gateway can also govern registered **agents** (running anywhere) and, in **preview**, **MCP tools** (only new MCP tools that don't use managed OAuth are routed; policies are applied in the Azure portal, not the Foundry portal). When a repo uses this integration, keep custom policies compatible with the Foundry-managed configuration rather than overriding it. Prefer importing the API as a **Language Model API** or **Azure AI Foundry API** in APIM, which auto-creates the backend, `set-backend-service`, and optional token/caching/safety policies. +Foundry has **built-in integration** with APIM: from the Foundry portal you can create a new APIM instance or attach an existing one as an AI gateway, then set per-deployment TPM limits and quotas from **Operate → Admin → AI Gateway → Token management** — these map directly to `llm-token-limit`. Attaching an **existing** APIM instance has hard requirements: it must be in the **same Microsoft Entra tenant and subscription** as the Foundry resource, be a **v2 tier** (Basic v2 / Standard v2 / Premium v2), and you need at least the **API Management Service Contributor** (or Owner) role on it — otherwise it won't appear as selectable. If the Foundry resource has **public network access disabled**, the APIM instance must also be privately reachable — use Standard v2 or Premium v2 with a private endpoint, or Premium v2 injected into a virtual network. The gateway can also govern registered **agents** (running anywhere) and, in **preview**, **MCP tools** (only new MCP tools that don't use managed OAuth are routed; policies are applied in the Azure portal, not the Foundry portal). When a repo uses this integration, keep custom policies compatible with the Foundry-managed configuration rather than overriding it. Prefer importing the API as a **Language Model API** or **Azure AI Foundry API** in APIM, which auto-creates the backend, `set-backend-service`, and optional token/caching/safety policies. ## Grounding From 801cd45d78c89b5eb3ef521ddba8788f5a16f98c Mon Sep 17 00:00:00 2001 From: Konstantinos Passadis | Azure MVP | MCT Date: Sun, 9 Aug 2026 01:38:34 +0000 Subject: [PATCH 08/11] Fix Bicep array syntax and use supported JWT accessor per review --- .../azure-apim-ai-gateway.instructions.md | 36 +++++++++++++++---- 1 file changed, 29 insertions(+), 7 deletions(-) diff --git a/instructions/azure-apim-ai-gateway.instructions.md b/instructions/azure-apim-ai-gateway.instructions.md index f95311818..322e13516 100644 --- a/instructions/azure-apim-ai-gateway.instructions.md +++ b/instructions/azure-apim-ai-gateway.instructions.md @@ -90,8 +90,18 @@ resource pool 'Microsoft.ApiManagement/service/backends@2023-09-01-preview' = { type: 'Pool' pool: { services: [ - { id: backend1.id, priority: 1, weight: 1 } // PTU — preferred - { id: backend2.id, priority: 2, weight: 1 } // PayGo — fallback + { + // PTU — preferred + id: backend1.id + priority: 1 + weight: 1 + } + { + // PayGo — fallback + id: backend2.id + priority: 2 + weight: 1 + } ] } } @@ -110,8 +120,19 @@ resource backend1 'Microsoft.ApiManagement/service/backends@2023-09-01-preview' count: 3 interval: 'PT1H' // 429 = PTU/TPM saturation (honors Retry-After), 5xx = backend failure - statusCodeRanges: [ { min: 429, max: 429 }, { min: 500, max: 599 } ] - errorReasons: [ 'Server errors' ] + statusCodeRanges: [ + { + min: 429 + max: 429 + } + { + min: 500 + max: 599 + } + ] + errorReasons: [ + 'Server errors' + ] } tripDuration: 'PT1H' acceptRetryAfter: true @@ -143,8 +164,9 @@ Cache completions by vector proximity of the prompt to reduce token spend and la ignore-system-messages="true" max-message-count="10"> - @(context.Principal?.Claims.GetValueOrDefault("oid", context.Subscription.Id)) + For user-specific responses, vary by the authenticated caller's subject. Reads the + JWT from the Authorization header (authenticate it first with ): --> + @(context.Request.Headers.GetValueOrDefault("Authorization","").AsJwt()?.Subject ?? context.Subscription.Id) ``` @@ -154,7 +176,7 @@ Cache completions by vector proximity of the prompt to reduce token spend and la ``` - Lower `score-threshold` = stricter match (fewer cache hits, higher fidelity). Tune per use case; start around `0.05`–`0.15`. -- Partition the cache on the **actual confidentiality boundary** with ``. Keying only on the APIM subscription id means every user sharing that subscription shares one cache partition and can receive each other's cached completions — a data-exposure risk. When responses are user-specific, add an authenticated user/subject identifier (for example a JWT `sub`/`oid` claim via `context.Principal`) to `` so per-user isolation is enforced. +- Partition the cache on the **actual confidentiality boundary** with ``. Keying only on the APIM subscription id means every user sharing that subscription shares one cache partition and can receive each other's cached completions — a data-exposure risk. When responses are user-specific, add the authenticated caller's subject to `` — read the validated token with `context.Request.Headers.GetValueOrDefault("Authorization","").AsJwt()?.Subject` (or a specific claim via `.AsJwt()?.Claims.GetValueOrDefault("oid","")`), authenticated first with ``, so per-user isolation is enforced. ## Content safety — `llm-content-safety` From 42ffa1eef100ef712b9adeb6b6b62909bccfc635 Mon Sep 17 00:00:00 2001 From: Konstantinos Passadis | Azure MVP | MCT Date: Sun, 9 Aug 2026 01:47:02 +0000 Subject: [PATCH 09/11] Correct content-safety tier support and response-screening default per review --- instructions/azure-apim-ai-gateway.instructions.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/instructions/azure-apim-ai-gateway.instructions.md b/instructions/azure-apim-ai-gateway.instructions.md index 322e13516..d122869dd 100644 --- a/instructions/azure-apim-ai-gateway.instructions.md +++ b/instructions/azure-apim-ai-gateway.instructions.md @@ -15,7 +15,7 @@ Guidance for putting **Azure API Management (APIM)** in front of **Microsoft Fou - **Throttle by tokens, not by call count**, for LLM APIs. `rate-limit-by-key` counts requests and is blind to token cost; use `llm-token-limit`. - **Authenticate to Foundry with a managed identity**, never a stored key. The exact role and token audience depend on the model type (see [Authentication](#authentication--managed-identity-not-keys)) — Azure OpenAI uses **Cognitive Services OpenAI User** (`https://cognitiveservices.azure.com`); other Foundry models use **Cognitive Services User** (`https://ai.azure.com`). - **Respect policy element order.** Set elements and child elements in the order documented for each policy, and keep `` in each section (`inbound`, `backend`, `outbound`, `on-error`). -- **Check tier support per policy — it varies.** `llm-token-limit` is not available on the Consumption tier; `llm-emit-token-metric`, `llm-semantic-cache-*`, and `llm-content-safety` apply to all tiers (including Consumption). Verify each policy's "Applies to" line rather than assuming. +- **Check tier support per policy — it varies.** `llm-token-limit` and `llm-content-safety` are not available on the Consumption tier; `llm-emit-token-metric` and `llm-semantic-cache-*` apply to all tiers (including Consumption). Verify each policy's "Applies to" line rather than assuming. - Prefer configuring an APIM **backend** resource (with managed-identity credentials) over inline `authentication-managed-identity` + `set-header`; importing a Foundry API wires this up automatically. ## Token rate limiting and quotas — `llm-token-limit` @@ -180,7 +180,7 @@ Cache completions by vector proximity of the prompt to reduce token spend and la ## Content safety — `llm-content-safety` -Screen prompts (and optionally responses) through **Azure AI Content Safety** before they reach the model. Configure a content-safety backend and set severity thresholds; `shield-prompt="true"` adds jailbreak/prompt-injection detection. +Screen prompts (and optionally responses) through **Azure AI Content Safety** before they reach the model. Configure a content-safety backend and set severity thresholds. Mind the defaults: both `shield-prompt` (jailbreak/prompt-injection detection) and `enforce-on-completions` default to `false`, so an inbound policy screens **prompts only** — set `enforce-on-completions="true"` to also screen the model's completions (or place the policy in the `outbound` section to screen responses). ```xml From 2e813a7376762135cc63d85a1c1fdc9e718f9ef2 Mon Sep 17 00:00:00 2001 From: Konstantinos Passadis | Azure MVP | MCT Date: Sun, 9 Aug 2026 02:07:18 +0000 Subject: [PATCH 10/11] Fix cache-partition identity, skeleton isolation, and metric cardinality per review --- .../azure-apim-ai-gateway.instructions.md | 35 +++++++++++++------ 1 file changed, 25 insertions(+), 10 deletions(-) diff --git a/instructions/azure-apim-ai-gateway.instructions.md b/instructions/azure-apim-ai-gateway.instructions.md index d122869dd..2688866a0 100644 --- a/instructions/azure-apim-ai-gateway.instructions.md +++ b/instructions/azure-apim-ai-gateway.instructions.md @@ -45,16 +45,16 @@ Emit prompt/completion/total token metrics to **Application Insights** so you ca ```xml - - - + + ``` - Prerequisites for the metric — all three are required or the policy emits nothing usable: an Application Insights logger connected to the APIM instance; **Application Insights logging enabled for the LLM API**; and **custom metrics with dimensions enabled in Application Insights**. - Emitting token metrics does **not** require logging message content. Full prompt/completion logging is a separate, **opt-in** step — enable it only with a clear need, because it can persist PII, secrets, and other sensitive content. If you do, apply field redaction, restrict who can read the logs, set a short retention window, and run it past your compliance/privacy review. - Metrics come from the `usage` section of the model response. Some OpenAI models — **especially when streaming** — omit token counts unless the request sets `include_usage: true` (`stream_options`), and an interrupted stream yields inaccurate counts. Ensure clients enable usage reporting or the metric will be silently incomplete. -- Applies to all API Management tiers (including Consumption). A maximum of 5 custom dimensions per policy. +- Applies to all API Management tiers (including Consumption). Max 5 custom dimensions per policy. +- **Keep dimension cardinality low.** Azure Monitor caps custom metrics at 50,000 active time series per region per subscription (12-hour window), and the series count is the product of each dimension's distinct values. Avoid high-cardinality dimensions like client IP or per-user IDs — at normal user scale they blow past the cap and metrics get dropped. Prefer stable built-in dimensions (e.g. `API ID`, `Subscription ID`); use per-user attribution through logs/queries instead. ## Authentication — managed identity, not keys @@ -78,6 +78,8 @@ Assign the role to APIM's managed identity on the Foundry resource, then authent Preferred form: configure a **backend** with managed-identity credentials to the matching audience (`https://cognitiveservices.azure.com/` for Azure OpenAI, `https://ai.azure.com/` for other Foundry models) and reference it with ``. This is what APIM sets up when you import a Foundry API directly. +> **Important — the `Authorization` header gets overwritten.** The `set-header` above (and the backend form) replaces the caller's `Authorization` with APIM's managed-identity token *before* the request reaches the backend. If you need the caller's identity downstream (e.g. to partition the semantic cache per user), capture it **first** with `` and read it from that variable — after the overwrite, `Authorization` holds the backend identity, not the caller's. + ## Resiliency — backend pools, load balancing, and circuit breakers Do **not** hand-roll retry/failover across multiple Foundry endpoints in application code. Define an APIM **backend pool** and let the gateway load-balance (round-robin, weighted, priority, session-aware) with a **circuit breaker** that honors the backend `Retry-After` header. Use `priority` to prefer PTU endpoints and fall back to pay-as-you-go. @@ -164,9 +166,11 @@ Cache completions by vector proximity of the prompt to reduce token spend and la ignore-system-messages="true" max-message-count="10"> - @(context.Request.Headers.GetValueOrDefault("Authorization","").AsJwt()?.Subject ?? context.Subscription.Id) + For user-specific responses, vary by the authenticated caller's subject. Key off the + caller JWT captured by earlier + in inbound — do NOT read Authorization here, it has been overwritten with APIM's + managed-identity token before lookup: --> + @(context.Variables.GetValueOrDefault("callerJwt")?.Subject ?? context.Subscription.Id) ``` @@ -176,7 +180,7 @@ Cache completions by vector proximity of the prompt to reduce token spend and la ``` - Lower `score-threshold` = stricter match (fewer cache hits, higher fidelity). Tune per use case; start around `0.05`–`0.15`. -- Partition the cache on the **actual confidentiality boundary** with ``. Keying only on the APIM subscription id means every user sharing that subscription shares one cache partition and can receive each other's cached completions — a data-exposure risk. When responses are user-specific, add the authenticated caller's subject to `` — read the validated token with `context.Request.Headers.GetValueOrDefault("Authorization","").AsJwt()?.Subject` (or a specific claim via `.AsJwt()?.Claims.GetValueOrDefault("oid","")`), authenticated first with ``, so per-user isolation is enforced. +- Partition the cache on the **actual confidentiality boundary** with ``. Keying only on the APIM subscription id means every user sharing that subscription shares one cache partition and can receive each other's cached completions — a data-exposure risk. When responses are user-specific, add the authenticated caller's subject to ``. Capture the caller's token early in `inbound` with `` (before the managed-identity step overwrites `Authorization`), then key off the saved variable: `context.Variables.GetValueOrDefault("callerJwt")?.Subject` (or a specific claim via `...?.Claims.GetValueOrDefault("oid","")`). Do not re-read `Authorization` at lookup time — by then it holds APIM's backend token, which would collapse all callers into one partition. ## Content safety — `llm-content-safety` @@ -202,6 +206,14 @@ Keep AI gateway policies in the correct sections and preserve ``: + + + + + api://your-api-client-id + + @@ -215,8 +227,11 @@ Keep AI gateway policies in the correct sections and preserve ``: - - + + + @(context.Variables.GetValueOrDefault("callerJwt")?.Subject ?? context.Subscription.Id) + + From aee6c56f498e97743bc3e3e2789132b74a390886 Mon Sep 17 00:00:00 2001 From: Konstantinos Passadis | Azure MVP | MCT Date: Sun, 9 Aug 2026 07:05:51 +0000 Subject: [PATCH 11/11] Make policy expressions valid XML and clarify content-safety timing --- .../azure-apim-ai-gateway.instructions.md | 20 ++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/instructions/azure-apim-ai-gateway.instructions.md b/instructions/azure-apim-ai-gateway.instructions.md index 2688866a0..29cbb9fa9 100644 --- a/instructions/azure-apim-ai-gateway.instructions.md +++ b/instructions/azure-apim-ai-gateway.instructions.md @@ -167,10 +167,11 @@ Cache completions by vector proximity of the prompt to reduce token spend and la max-message-count="10"> - @(context.Variables.GetValueOrDefault("callerJwt")?.Subject ?? context.Subscription.Id) + caller JWT saved by validate-jwt (output-token-variable-name="callerJwt") earlier in + inbound — do NOT read Authorization here; it is overwritten with APIM's managed-identity + token before lookup. A cast is used because a generic (Jwt) would put a raw '<' in the + XML, which is invalid: --> + @(context.Variables.ContainsKey("callerJwt") ? ((Jwt)context.Variables["callerJwt"]).Subject : context.Subscription.Id) ``` @@ -180,11 +181,11 @@ Cache completions by vector proximity of the prompt to reduce token spend and la ``` - Lower `score-threshold` = stricter match (fewer cache hits, higher fidelity). Tune per use case; start around `0.05`–`0.15`. -- Partition the cache on the **actual confidentiality boundary** with ``. Keying only on the APIM subscription id means every user sharing that subscription shares one cache partition and can receive each other's cached completions — a data-exposure risk. When responses are user-specific, add the authenticated caller's subject to ``. Capture the caller's token early in `inbound` with `` (before the managed-identity step overwrites `Authorization`), then key off the saved variable: `context.Variables.GetValueOrDefault("callerJwt")?.Subject` (or a specific claim via `...?.Claims.GetValueOrDefault("oid","")`). Do not re-read `Authorization` at lookup time — by then it holds APIM's backend token, which would collapse all callers into one partition. +- Partition the cache on the **actual confidentiality boundary** with ``. Keying only on the APIM subscription id means every user sharing that subscription shares one cache partition and can receive each other's cached completions — a data-exposure risk. When responses are user-specific, add the authenticated caller's subject to ``. Capture the caller's token early in `inbound` with `` (before the managed-identity step overwrites `Authorization`), then key off the saved variable with a cast (a generic `` would put a raw `<` into the XML): `@(context.Variables.ContainsKey("callerJwt") ? ((Jwt)context.Variables["callerJwt"]).Subject : context.Subscription.Id)`. Do not re-read `Authorization` at lookup time — by then it holds APIM's backend token, which would collapse all callers into one partition. ## Content safety — `llm-content-safety` -Screen prompts (and optionally responses) through **Azure AI Content Safety** before they reach the model. Configure a content-safety backend and set severity thresholds. Mind the defaults: both `shield-prompt` (jailbreak/prompt-injection detection) and `enforce-on-completions` default to `false`, so an inbound policy screens **prompts only** — set `enforce-on-completions="true"` to also screen the model's completions (or place the policy in the `outbound` section to screen responses). +Screen prompts through **Azure AI Content Safety** **before they reach the model** and, optionally, screen the model's responses (completions) **after it replies**. Configure a content-safety backend and set severity thresholds. Mind the defaults: both `shield-prompt` (jailbreak/prompt-injection detection) and `enforce-on-completions` default to `false`, so an inbound policy screens **prompts only** — set `enforce-on-completions="true"` to also screen the model's completions (or place the policy in the `outbound` section to screen responses). ```xml @@ -228,10 +229,11 @@ Keep AI gateway policies in the correct sections and preserve ``: so a cached request must not consume the caller's TPM/quota. Content safety stays above the lookup so every prompt is still screened. --> - - @(context.Variables.GetValueOrDefault("callerJwt")?.Subject ?? context.Subscription.Id) + + @(context.Variables.ContainsKey("callerJwt") ? ((Jwt)context.Variables["callerJwt"]).Subject : context.Subscription.Id) - +