Add Azure API Management AI Gateway instructions 🤖🤖🤖 - #2595
Open
passadis wants to merge 11 commits into
Open
Conversation
APIM as a GenAI gateway for Microsoft Foundry and OpenAI-compatible LLM APIs: llm-* token-limit/token-metric policies, managed-identity auth, backend pools with circuit breakers, semantic caching, and content safety. Corrects the azure-openai-*/api-key/element-ordering defaults the model produces unprompted. Regenerates docs/README.instructions.md.
github-actions
Bot
added
instructions
Contributor
🔒 PR Risk Scan ResultsScanned 1 changed file(s).
✅ No matching risk patterns were detected in changed files.
|
Contributor
There was a problem hiding this comment.
Pull request overview
Adds Azure API Management AI gateway guidance for Microsoft Foundry and compatible LLM APIs.
Changes:
- Documents authentication, token governance, observability, resiliency, caching, and content safety.
- Adds a complete APIM policy skeleton and Bicep example.
- Registers the instruction in the documentation index.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 6 comments.
| File | Description |
|---|---|
instructions/azure-apim-ai-gateway.instructions.md |
Adds AI gateway implementation guidance. |
docs/README.instructions.md |
Lists the new instruction. |
Suppressed comments (2)
instructions/azure-apim-ai-gateway.instructions.md:55
- Current Microsoft Learn explicitly lists the Consumption gateway for
llm-emit-token-metric, so this limitation is incorrect.
- Not available on the Consumption tier.
instructions/azure-apim-ai-gateway.instructions.md:59
- This section presents Azure OpenAI's role and token audience as generic Foundry authentication. Non-OpenAI Foundry inference endpoints require
Cognitive Services Userand the Foundry audience (https://ai.azure.comforauthentication-managed-identity), so the shown setup will not authenticate those deployments covered by the document's scope.
Give APIM's managed identity the **Cognitive Services OpenAI User** role on the Foundry resource, then authenticate at the gateway. Inline form:
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| - **Throttle by tokens, not by call count**, for LLM APIs. `rate-limit-by-key` counts requests and is blind to token cost; use `llm-token-limit`. | ||
| - **Authenticate to Foundry with a managed identity**, never a stored key. Give APIM's identity the **Cognitive Services OpenAI User** role on the Foundry resource. | ||
| - **Respect policy element order.** Set elements and child elements in the order documented for each policy, and keep `<base />` in each section (`inbound`, `backend`, `outbound`, `on-error`). | ||
| - **The `llm-*` and `azure-openai-*` metric/token policies are not available on the Consumption tier.** Check tier support before recommending them. |
|
|
||
| - **Prefer the provider-agnostic `llm-*` policies** (`llm-token-limit`, `llm-emit-token-metric`, `llm-semantic-cache-lookup`/`-store`, `llm-content-safety`). They work across OpenAI Chat Completions/Responses, Anthropic Messages (v2 tiers), and Google Vertex AI. Only use the `azure-openai-*` variants when the API is exclusively Azure OpenAI and you have a reason to. | ||
| - **Throttle by tokens, not by call count**, for LLM APIs. `rate-limit-by-key` counts requests and is blind to token cost; use `llm-token-limit`. | ||
| - **Authenticate to Foundry with a managed identity**, never a stored key. Give APIM's identity the **Cognitive Services OpenAI User** role on the Foundry resource. |
Comment on lines
+48
to
+50
| <dimension name="Client IP" value="@(context.Request.IpAddress)" /> | ||
| <dimension name="API ID" value="@(context.Api.Id)" /> | ||
| <dimension name="User ID" value="@(context.Request.Headers.GetValueOrDefault("x-user-id", "N/A"))" /> |
| </llm-emit-token-metric> | ||
| ``` | ||
|
|
||
| - Requires an Application Insights logger wired to the APIM instance. Also enable LLM request logging to capture prompts/completions for auditing. |
| <llm-emit-token-metric namespace="llm-metrics"> | ||
| <dimension name="API ID" value="@(context.Api.Id)" /> | ||
| </llm-emit-token-metric> | ||
| <llm-semantic-cache-lookup score-threshold="0.1" embeddings-backend-id="embeddings-backend" embeddings-backend-auth="system-assigned" /> |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
Suppressed comments (6)
instructions/azure-apim-ai-gateway.instructions.md:18
- The current
llm-emit-token-metricreference says it applies to all API Management tiers and explicitly lists the Consumption gateway, so this blanket rule incorrectly excludes a supported policy. Restrict the exception tollm-token-limitand direct readers to check each policy individually.
- **The `llm-*` and `azure-openai-*` metric/token policies are not available on the Consumption tier.** Check tier support before recommending them.
instructions/azure-apim-ai-gateway.instructions.md:55
- This availability note is outdated: the current Microsoft Learn reference marks
llm-emit-token-metricas applying to all API Management tiers and lists the Consumption gateway. Leaving this note will steer Consumption users away from a supported observability policy.
- Not available on the Consumption tier.
instructions/azure-apim-ai-gateway.instructions.md:87
backend2.idreferences a symbol that is never declared in this snippet, so the example fails Bicep compilation and cannot demonstrate the advertised fallback pool. Define the second backend resource or clearly mark the block as incomplete pseudocode.
{ id: backend2.id, priority: 2, weight: 1 } // PayGo — fallback
instructions/azure-apim-ai-gateway.instructions.md:54
- The metric can be incomplete for a common streaming setup. Microsoft Learn notes that some OpenAI models do not return token counts by default—especially when streaming—unless the request enables
include_usage, and interrupted streams produce inaccurate counts. Add this caveat so the promised spend attribution is not silently wrong.
- Requires an Application Insights logger wired to the APIM instance. Also enable LLM request logging to capture prompts/completions for auditing.
instructions/azure-apim-ai-gateway.instructions.md:101
- The rule also trips on HTTP 429, so
trip-on-5xxmisstates its configured behavior and can confuse operators reviewing backend rules. Include both failure classes in the name.
name: 'trip-on-5xx'
instructions/azure-apim-ai-gateway.instructions.md:192
- This omits current setup and lifecycle constraints for the Foundry-native flow. Attaching an existing APIM instance requires the same tenant and subscription, sufficient APIM permissions, and a v2 tier; current Foundry documentation also labels MCP tool governance as preview. Without these qualifiers, users may plan around an ineligible gateway or assume preview tooling is generally available.
Foundry has **built-in integration** with APIM: from the Foundry portal you can create or attach an APIM instance as an AI gateway and set per-deployment TPM limits and quotas from **Operate → Admin → AI Gateway → Token management** — these map directly to `llm-token-limit`. The gateway can also govern registered **agents** (running anywhere) and **MCP tools**, surfacing them in the Foundry inventory with policy enforcement and telemetry. When a repo uses this integration, keep custom policies compatible with the Foundry-managed configuration rather than overriding it. Prefer importing the API as a **Language Model API** or **Azure AI Foundry API** in APIM, which auto-creates the backend, `set-backend-service`, and optional token/caching/safety policies.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
Suppressed comments (4)
instructions/azure-apim-ai-gateway.instructions.md:17
- There is no single documented ordering for peer AI-gateway policy statements: statements in each section execute sequentially, so their order depends on the desired behavior. Requiring every element to follow a fixed documented order is misleading; only the section structure and per-policy child-element schema impose validation order.
- **Respect policy element order.** Set elements and child elements in the order documented for each policy, and keep `<base />` in each section (`inbound`, `backend`, `outbound`, `on-error`).
instructions/azure-apim-ai-gateway.instructions.md:54
- The metric policy has an additional required setup step: custom metrics with dimensions must be enabled in Application Insights. Merely wiring the logger and enabling API logging is insufficient, so users following this prerequisite list can receive no dimensioned metrics.
- Requires an Application Insights logger wired to the APIM instance. Also enable LLM request logging to capture prompts/completions for auditing.
instructions/azure-apim-ai-gateway.instructions.md:10
- This says APIM rejects arbitrary ordering of peer AI-gateway policies, but policy statements within a section are valid in different sequences and execute sequentially. APIM's fixed section structure and each policy's child-element schema must be respected; peer-policy order should instead reflect the intended runtime behavior. The current wording teaches a nonexistent universal order.
This issue also appears on line 17 of the same file.
> **Field note (why this file exists):** In Copilot-assisted APIM work, the default behavior is to (1) reach for the older provider-specific `azure-openai-*` policies instead of the current provider-agnostic `llm-*` ones, (2) throttle LLM traffic with request-count policies (`rate-limit-by-key`) that don't understand tokens, (3) authenticate to the model backend with an `api-key` header pulled from a named value instead of a managed identity, and (4) emit policy elements in an arbitrary order — which APIM rejects, because **AI gateway policy elements are order-sensitive**. These instructions front-load those corrections so Copilot produces a valid, secure gateway policy on the first pass. When in doubt, ground against Microsoft Learn or the Microsoft Docs MCP server — the AI gateway policy surface changes frequently.
instructions/azure-apim-ai-gateway.instructions.md:18
- Microsoft Learn's current
llm-content-safetyreference excludes the Consumption tier, just likellm-token-limit. Claiming it applies to Consumption will lead users to deploy an unsupported policy.
- **Check tier support per policy — it varies.** `llm-token-limit` is not available on the Consumption tier; `llm-emit-token-metric`, `llm-semantic-cache-*`, and `llm-content-safety` apply to all tiers (including Consumption). Verify each policy's "Applies to" line rather than assuming.
| <llm-emit-token-metric namespace="llm-metrics"> | ||
| <dimension name="Client IP" value="@(context.Request.IpAddress)" /> | ||
| <dimension name="API ID" value="@(context.Api.Id)" /> | ||
| <dimension name="User ID" value="@(context.Request.Headers.GetValueOrDefault("x-user-id", "N/A"))" /> |
| ``` | ||
|
|
||
| - Lower `score-threshold` = stricter match (fewer cache hits, higher fidelity). Tune per use case; start around `0.05`–`0.15`. | ||
| - Partition the cache per tenant/consumer with `<vary-by>` so users never receive another consumer's cached completion. |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
Suppressed comments (3)
instructions/azure-apim-ai-gateway.instructions.md:111
- This breaker only counts 5xx responses. A saturated priority-1 PTU backend normally returns 429, so it never trips for the capacity condition that should activate the priority-2 PAYG fallback, and
acceptRetryAftercannot govern that response. Include 429 in the failure ranges.
statusCodeRanges: [ { min: 500, max: 599 } ]
instructions/azure-apim-ai-gateway.instructions.md:16
- This core rule gives the OpenAI-specific role for every Foundry model, contradicting the model-specific RBAC guidance on lines 60–63. A reader following the summary for a non-OpenAI deployment will receive authorization failures; point this rule to the role distinction below instead.
- **Authenticate to Foundry with a managed identity**, never a stored key. Give APIM's identity the **Cognitive Services OpenAI User** role on the Foundry resource.
instructions/azure-apim-ai-gateway.instructions.md:50
- The unescaped double quotes inside this double-quoted XML attribute terminate
valueearly, so the advertised policy snippet is not well-formed XML and cannot be deployed. Use a single-quoted XML attribute (or escape the inner quotes).
<dimension name="User ID" value="@(context.Request.Headers.GetValueOrDefault("x-user-id", "N/A"))" />
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
Suppressed comments (3)
instructions/azure-apim-ai-gateway.instructions.md:154
- The example only partitions by APIM subscription, so the claim that users cannot receive another user's cached completion is too strong: every user sharing that subscription shares the same semantic-cache partition. This can expose user-specific responses. Instruct consumers to key on the actual confidentiality boundary, adding an authenticated user/subject identifier when per-user isolation is required.
- Partition the cache per tenant/consumer with `<vary-by>` so users never receive another consumer's cached completion.
instructions/azure-apim-ai-gateway.instructions.md:54
- Full prompt/completion logging is optional and can persist PII, secrets, or other sensitive message content; it is not required for token metrics. Recommending it unconditionally as an auditing step creates avoidable data exposure. Make it opt-in and call out redaction, access control, retention, and compliance review.
- Requires an Application Insights logger wired to the APIM instance. Also enable LLM request logging to capture prompts/completions for auditing.
instructions/azure-apim-ai-gateway.instructions.md:194
- Move the semantic-cache lookup before the token-limit policy. A cache hit short-circuits the inbound pipeline, but in the current order
llm-token-limithas already estimated/accounted for the prompt, so cached requests can unnecessarily consume the caller's TPM/quota. The Microsoft semantic-caching pattern likewise places lookup before rate limiting.
<llm-semantic-cache-lookup score-threshold="0.1" embeddings-backend-id="embeddings-backend" embeddings-backend-auth="system-assigned" />
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
Suppressed comments (2)
instructions/azure-apim-ai-gateway.instructions.md:54
- The setup prerequisites are incomplete: the policy reference also requires Application Insights logging to be enabled for the LLM API and custom metrics with dimensions to be enabled in Application Insights. Merely wiring a logger can leave this policy without usable emitted metrics, so include both settings in the prerequisite.
- Requires an Application Insights logger wired to the APIM instance. Token metrics do **not** require logging message content. Full prompt/completion logging is a separate, **opt-in** step — enable it only with a clear need, because it can persist PII, secrets, and other sensitive content. If you do, apply field redaction, restrict who can read the logs, set a short retention window, and run it past your compliance/privacy review.
instructions/azure-apim-ai-gateway.instructions.md:212
- The claimed “hard requirements” omit the private-networking prerequisite. Microsoft Foundry also requires APIM to be privately accessible when the Foundry resource has public network access disabled (Standard v2 or Premium v2 with a private endpoint, or Premium v2 with VNet injection); otherwise an instance satisfying the listed tenant, subscription, tier, and role checks still cannot integrate.
Foundry has **built-in integration** with APIM: from the Foundry portal you can create a new APIM instance or attach an existing one as an AI gateway, then set per-deployment TPM limits and quotas from **Operate → Admin → AI Gateway → Token management** — these map directly to `llm-token-limit`. Attaching an **existing** APIM instance has hard requirements: it must be in the **same Microsoft Entra tenant and subscription** as the Foundry resource, be a **v2 tier** (Basic v2 / Standard v2 / Premium v2), and you need at least the **API Management Service Contributor** (or Owner) role on it — otherwise it won't appear as selectable. The gateway can also govern registered **agents** (running anywhere) and, in **preview**, **MCP tools** (only new MCP tools that don't use managed OAuth are routed; policies are applied in the Azure portal, not the Foundry portal). When a repo uses this integration, keep custom policies compatible with the Foundry-managed configuration rather than overriding it. Prefer importing the API as a **Language Model API** or **Azure AI Foundry API** in APIM, which auto-creates the backend, `set-backend-service`, and optional token/caching/safety policies.
| max-message-count="10"> | ||
| <!-- Subscription id alone shares one partition across all users on that subscription. | ||
| For user-specific responses, vary by an authenticated subject to isolate per user: --> | ||
| <vary-by>@(context.Principal?.Claims.GetValueOrDefault("oid", context.Subscription.Id))</vary-by> |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
Suppressed comments (2)
instructions/azure-apim-ai-gateway.instructions.md:113
- This
statusCodeRangesvalue uses JSON-style commas, which are invalid in Bicep both between object properties and array items. As written, the resiliency sample fails to compile.
statusCodeRanges: [ { min: 429, max: 429 }, { min: 500, max: 599 } ]
instructions/azure-apim-ai-gateway.instructions.md:157
- The prose repeats the unsupported
context.Principalguidance. APIM exposes validated claims through theJwtobject stored byvalidate-jwt, so following this instruction would lead Copilot to generate an undeployable policy.
- Partition the cache on the **actual confidentiality boundary** with `<vary-by>`. Keying only on the APIM subscription id means every user sharing that subscription shares one cache partition and can receive each other's cached completions — a data-exposure risk. When responses are user-specific, add an authenticated user/subject identifier (for example a JWT `sub`/`oid` claim via `context.Principal`) to `<vary-by>` so per-user isolation is enforced.
Comment on lines
+93
to
+94
| { id: backend1.id, priority: 1, weight: 1 } // PTU — preferred | ||
| { id: backend2.id, priority: 2, weight: 1 } // PayGo — fallback |
| max-message-count="10"> | ||
| <!-- Subscription id alone shares one partition across all users on that subscription. | ||
| For user-specific responses, vary by an authenticated subject to isolate per user: --> | ||
| <vary-by>@(context.Principal?.Claims.GetValueOrDefault("oid", context.Subscription.Id))</vary-by> |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
Suppressed comments (2)
instructions/azure-apim-ai-gateway.instructions.md:18
- The current
llm-content-safetyreference's APPLIES TO list excludes Consumption, so grouping it with the all-tier policies can lead Copilot to generate a policy that is unavailable on that tier. List it alongsidellm-token-limitinstead.
- **Check tier support per policy — it varies.** `llm-token-limit` is not available on the Consumption tier; `llm-emit-token-metric`, `llm-semantic-cache-*`, and `llm-content-safety` apply to all tiers (including Consumption). Verify each policy's "Applies to" line rather than assuming.
instructions/azure-apim-ai-gateway.instructions.md:183
- Response screening is disabled by default (
enforce-on-completionsdefaults tofalse). Since this section says responses can also be screened but never identifies the required setting, copied/generated policies will only inspect prompts. Name the attribute explicitly.
Screen prompts (and optionally responses) through **Azure AI Content Safety** before they reach the model. Configure a content-safety backend and set severity thresholds; `shield-prompt="true"` adds jailbreak/prompt-injection detection.
| <!-- Cache lookup BEFORE token-limit/metric: a cache hit short-circuits the pipeline, | ||
| so a cached request must not consume the caller's TPM/quota. Content safety stays | ||
| above the lookup so every prompt is still screened. --> | ||
| <llm-semantic-cache-lookup score-threshold="0.1" embeddings-backend-id="embeddings-backend" embeddings-backend-auth="system-assigned" /> |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
Suppressed comments (4)
instructions/azure-apim-ai-gateway.instructions.md:169
- Reading
Authorizationat lookup time does not reliably identify the caller: the canonical policy below overwrites this header with APIM's managed-identity token before cache lookup, so all callers can collapse into the backend identity's cache partition. Save the validated client JWT in a variable before that overwrite and key the cache from the saved token.
<!-- Subscription id alone shares one partition across all users on that subscription.
For user-specific responses, vary by the authenticated caller's subject. Reads the
JWT from the Authorization header (authenticate it first with <validate-jwt>): -->
<vary-by>@(context.Request.Headers.GetValueOrDefault("Authorization","").AsJwt()?.Subject ?? context.Subscription.Id)</vary-by>
instructions/azure-apim-ai-gateway.instructions.md:179
- This advice re-reads
Authorization, but the recommended backend-authentication sequence replaces that header with APIM's managed-identity token. That makes the purported per-user partition shared and can expose cached completions across callers. Reference the JWT object saved by the caller-token validation instead.
- Partition the cache on the **actual confidentiality boundary** with `<vary-by>`. Keying only on the APIM subscription id means every user sharing that subscription shares one cache partition and can receive each other's cached completions — a data-exposure risk. When responses are user-specific, add the authenticated caller's subject to `<vary-by>` — read the validated token with `context.Request.Headers.GetValueOrDefault("Authorization","").AsJwt()?.Subject` (or a specific claim via `.AsJwt()?.Claims.GetValueOrDefault("oid","")`), authenticated first with `<validate-jwt>`, so per-user isolation is enforced.
instructions/azure-apim-ai-gateway.instructions.md:218
- The canonical skeleton enables semantic caching without any
<vary-by>partition, despite the preceding warning. APIM therefore allows semantically similar requests from different callers to share cached completions. Add an authenticated tenant/user partition (captured beforeAuthorizationis replaced), or omit lookup/store from this generic skeleton until a confidentiality boundary is configured.
<llm-semantic-cache-lookup score-threshold="0.1" embeddings-backend-id="embeddings-backend" embeddings-backend-auth="system-assigned" />
instructions/azure-apim-ai-gateway.instructions.md:50
Client IPand arbitrary per-user values create a Cartesian product of metric time series. Azure Monitor caps these at 50,000 active series per region/subscription per 12 hours, so this example can cause token metrics to be dropped at normal user scale. Prefer stable, lower-cardinality built-in dimensions such as API and subscription IDs.
<dimension name="Client IP" value="@(context.Request.IpAddress)" />
<dimension name="API ID" value="@(context.Api.Id)" />
<dimension name="User ID" value='@(context.Request.Headers.GetValueOrDefault("x-user-id", "N/A"))' />
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
Suppressed comments (4)
instructions/azure-apim-ai-gateway.instructions.md:232
- The policy skeleton repeats the unescaped generic type argument, making the advertised complete XML policy malformed. Escape
<so this example can be pasted into APIM.
<vary-by>@(context.Variables.GetValueOrDefault<Jwt>("callerJwt")?.Subject ?? context.Subscription.Id)</vary-by>
instructions/azure-apim-ai-gateway.instructions.md:173
- This XML snippet is not well-formed because the raw
<inGetValueOrDefault<Jwt>is parsed as markup, so copying it into an APIM policy fails before policy validation. Escape the generic type delimiter in XML text.
This issue also appears on line 232 of the same file.
<vary-by>@(context.Variables.GetValueOrDefault<Jwt>("callerJwt")?.Subject ?? context.Subscription.Id)</vary-by>
instructions/azure-apim-ai-gateway.instructions.md:187
- This sentence incorrectly says responses are screened before reaching the model; completions only exist after the model responds. Distinguish request screening from response screening so the policy flow is unambiguous.
Screen prompts (and optionally responses) through **Azure AI Content Safety** before they reach the model. Configure a content-safety backend and set severity thresholds. Mind the defaults: both `shield-prompt` (jailbreak/prompt-injection detection) and `enforce-on-completions` default to `false`, so an inbound policy screens **prompts only** — set `enforce-on-completions="true"` to also screen the model's completions (or place the policy in the `outbound` section to screen responses).
instructions/azure-apim-ai-gateway.instructions.md:234
- This attribute is invalid XML in two ways: the raw
<starts markup and the inner double quotes terminatecounter-key. Escape both before presenting this as a deployable policy skeleton.
<llm-token-limit counter-key="@(context.Variables.GetValueOrDefault<Jwt>("callerJwt")?.Subject ?? context.Subscription.Id)" tokens-per-minute="500" estimate-prompt-tokens="true" />
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request Checklist
npm startand verified thatREADME.mdis up to date.mainbranch for this pull request.Description
Adds
instructions/azure-api-management-ai-gateway.instructions.md— guidance for using Azure API Management (APIM) as a GenAI / AI gateway in front of Microsoft Foundry model deployments (Azure OpenAI and other providers) and any OpenAI-compatible LLM API.Why it adds uplift beyond default model behavior: unprompted, Copilot produces APIM policy that looks plausible but is wrong or won't deploy. This file encodes the current, correct pattern: the provider-agnostic llm-* policies (llm-token-limit, llm-emit-token-metric, llm-semantic-cache-lookup/-store, llm-content-safety) with correct attributes and 429 (TPM) vs 403 (quota) semantics; managed-identity auth (authentication-managed-identity to https://cognitiveservices.azure.com, Cognitive Services OpenAI User role) instead of a stored api-key; backend pools with circuitBreaker / acceptRetryAfter priority failover instead of app-side retry loops; semantic caching's real dependency on an external RediSearch-compatible cache + embeddings backend; correct policy-section element ordering with <base /> preserved; and the Foundry-native AI gateway integration. Every snippet is grounded in official Microsoft Learn documentation.
Type of Contribution
Additional Notes
This file is grounded in a measured, repeatable failure mode. I ran an A/B test — 5 representative AI-gateway prompts, each sent twice to the same GPT model (temperature 0.2, identical text, instruction injected as context the way VS Code attaches it), objective criteria scored by deterministic detectors and policy-ordering by a GPT judge, with every flip hand-verified against the raw output:
11 criteria improved, 0 regressed. Unprompted, the model reached for the older/superseded
azure-openai-token-limit, an inventedazure-openai-token-quotapolicy that does not exist, andazure-openai-semantic-cache-*with<embedding-backend>child elements that aren't in the schema; built an Event Hub → Azure Function → Application Insights pipeline in place of the one-linellm-emit-token-metricpolicy; load-balanced two Foundry endpoints withnew System.Random()inside a policy expression while dropping<base />from<backend>; and offered a storedapi-keyfallback alongside managed identity. With the instruction attached, the same model produced currentllm-*policies, per-user metric dimensions, an APIM backend pool with circuit-breaker priority failover, and managed-identity-only auth. The one small delta (P2, +1) is the honest case where the baseline was already close. This is the "meaningful uplift beyond model defaults" the repo asks for rather than duplicating strengths the model already has. Verified against Microsoft Learn (genai-gateway-capabilitiesand the per-policy reference pages); rannpm startand confirmed the README tables are up to date.🤖 This PR was prepared with AI-agent assistance (disclosed per repo convention via the title marker).
By submitting this pull request, I confirm that my contribution abides by the Code of Conduct and will be licensed under the MIT License.