API-key auth is not model-scoped: any valid key works for every model on the shared gateway listener

[dcmcand]

Summary

Each LLMModel and PassthroughModel gets its own api-keys Secret and its own SecurityPolicy, which implies that a key minted for one model only works for that model. It does not. Every model on the cluster shares a single gateway listener (llm.<baseDomain>), and Envoy Gateway pools all apiKeyAuth credentials from the SecurityPolicies on that listener into one set with no per-route scoping. So any valid API key authenticates requests to every model on the gateway. This affects served models and passthrough models equally.

Access control by group still works at mint time (the key-manager only lets a user create a key for a model whose access.groups they belong to), but once a user holds any key, it works against all models on the listener. There is no request-time per-model enforcement on the API-key path.

Impact

• Per-model access control via API keys does not hold. A user authorized for a cheap model can call an expensive or restricted model with the same key by changing the model field.
• This is pack-wide, not passthrough-specific (routing.go and passthrough.go both attach to the shared listener).
• The internal/JWT endpoint is not affected the same way (see root cause).

Reproduction

On a cluster with three passthrough models (claude-sonnet-45, gemini-25-flash, llama-33-70b), each with its own api-keys Secret:

1. Put a key in only one model's Secret, leave the other two Secrets empty.
2. Call each model through the gateway with that one key.

key present only in claude's Secret; gemini and llama Secrets empty

claude key -> model anthropic/claude-sonnet-4.5       -> HTTP 200   (expected)
claude key -> model google/gemini-2.5-flash           -> HTTP 200   (should be 401; gemini Secret is empty)
claude key -> model meta-llama/llama-3.3-70b-instruct -> HTTP 200   (should be 401; llama Secret is empty)

invalid/empty key -> any model                        -> HTTP 401   (auth is active)
undeclared model (catchAll: false)                    -> HTTP 404   (routing is per-model)

Models with empty Secrets accept a key that exists only in another model's Secret. Auth is on (bad keys are rejected) and routing is per-model (undeclared models 404), so this is specifically credential pooling across the SecurityPolicies on the shared listener.

Root cause

The shared-listener design routes every model through one hostname and distinguishes models by the x-ai-eg-model header (reconcilers/common.go SharedExternalHostname, reconcilers/routing.go, reconcilers/passthrough.go; the design note in routing.go explains it replaced per-model hostnames to avoid needing a TLS SAN per model).

The deeper issue is that isolation is being expected from authentication, but it belongs in authorization:

• apiKeyAuth (authentication) is pooled per listener. Envoy's api_key_auth filter supports a per-route allowed_clients list, but Envoy Gateway never sets it, and SecurityPolicy.apiKeyAuth exposes no field for it. The running Envoy config confirms it: allowed_clients appears 0 times, and the merged credential set is replicated to every route on the listener.
• SecurityPolicy authorization rules (defaultAction: Deny + per-principal rules) are applied per-targeted-route and do not pool. This is what the internal endpoint already uses (buildJWTSecurityPolicy), so the JWT path gives genuine per-model control. It needs a real OIDC token, so it is not usable from a plain API-key client.

The external API-key path has no authorization layer, only pooled authentication. The link that is missing is carrying the authenticated key's identity into a per-route authorization rule. The code already anticipates this in reconcilers/auth.go:

// TODO: Add sanitize:true and forwardClientIDHeader when minimum
// Envoy Gateway version is bumped to v1.7+ (these fields are not
// in v1.3 SecurityPolicy CRD)

Validation: per-listener isolation works today

A second SNI listener (claude2.llm.local, its own cert) was added with one model's route + SecurityPolicy, leaving the others on llm.local:

c2 key -> claude2.llm.local (its own listener)   -> 200
c2 key -> llm.local (claude model)               -> 401   no leak
c2 key -> llm.local (gemini model)               -> 401   no leak
shared-listener key -> llm.local (gemini)        -> 200   (pool, within one listener)
shared-listener key -> claude2.llm.local         -> 401   no leak

A distinct hostname + cert gives a separate Envoy filter chain with its own credential pool. Isolation is by SNI filter chain, so it holds whether each listener has its own cert or they share one wildcard cert.

Options

Approach	Single listener	Needs newer EG	Notes
Per-route authorization + apiKeyAuth.forwardClientIDHeader	yes	EG v1.7+	The path auth.go already plans. Forward the client id, gate per model in authz. Keeps one cert/listener.
External authorization (ext_authz)	yes	no	Native extension; a service maps key -> allowed models. Most flexible.
Per-model SNI listener (per-host HTTP-01 cert, or one shared DNS-01 wildcard cert)	no	no	Proven above. Works on current EG. Cost: per-model listener/cert lifecycle.
JWT + group authorization (internal endpoint)	yes	no	Already works per-model, but requires an OIDC token, not an API key.

Recommendation: prefer the authorization-layer fixes (forwardClientIDHeader once EG v1.7+ is the floor, or ext_authz) since they keep the single-listener/single-cert design. Per-model listeners are a viable workaround on the current stack if a fix is needed before the EG bump.

References

• Envoy AI Gateway security capabilities: https://aigateway.envoyproxy.io/docs/capabilities/security/
• EG API Key Authentication: https://gateway.envoyproxy.io/docs/tasks/security/apikey-auth/
• EG JWT Claim-Based Authorization (per-route authz): https://gateway.envoyproxy.io/docs/tasks/security/jwt-claim-authorization/
• Code: operator/internal/controller/reconcilers/auth.go (apiKeyAuth + forwardClientIDHeader TODO), routing.go, passthrough.go, common.go (shared listener/hostname)

Definition of done

• Decision on the approach (authorization-layer vs per-model listener).
• Per-model API-key requests are authorized per model at request time, with a test that a key for model A is rejected (401/403) for model B.
• Docs updated to state the current behavior until the fix lands (API keys are gateway-scoped, not model-scoped).