Skip to content

fix(operator): make per-model API-key auth work end-to-end on AI Gateway v0.5 (#116)#117

Open
dcmcand wants to merge 21 commits into
mainfrom
fix/116-model-scoped-api-key-auth
Open

fix(operator): make per-model API-key auth work end-to-end on AI Gateway v0.5 (#116)#117
dcmcand wants to merge 21 commits into
mainfrom
fix/116-model-scoped-api-key-auth

Conversation

@dcmcand

@dcmcand dcmcand commented Jun 26, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What this fixes

Closes #116. API keys were gateway-scoped, not model-scoped: any valid key worked for every model on the shared external listener (llm.<baseDomain>). Closing that gap end-to-end on the current stack (Envoy Gateway v1.6.7 / AI Gateway v0.5) turned out to need four changes, three of which surfaced during live validation.

The four changes

1. Per-model authorization (the #116 core). Each model's external SecurityPolicy gets a deny-by-default authorization block whose Allow rule lists only the client IDs in that model's own api-keys Secret, matched on apiKeyAuth.forwardClientIDHeader: x-llm-client-id (with sanitize: true). Authentication is shared across the listener; authorization is per-route, so a key forwards its client ID and is allowed only for the model it was minted for.

2. Pooled authentication. EG's api_key_auth filter runs before the AI Gateway ext_proc resolves the model, so authentication happens on whichever catch-all route wins, not the model's own route. With per-route credentials, only one model's keys could authenticate and every other model's keys got 401. Each external SecurityPolicy now pools every model's api-keys Secret into credentialRefs (all same-namespace, per #59), so any valid key authenticates on the shared listener and the per-route authorization (change 1) does the scoping. This is the design #116 assumed; it just is not the default on EG v1.6.7. The controllers gather the full Secret set and re-reconcile all models when an api-keys Secret is added or removed.

3. Routing fix for AI Gateway v0.5. The AI Gateway v0.5 controller does not register a model whose AIGatewayRoute match carries any header beyond x-ai-eg-model. The operator added a Host matcher (defence-in-depth on top of sectionName), which made every request 404 "model not configured in the Gateway" and blocked all model serving. Removed it; sectionName already scopes each route to its listener.

4. EPP RBAC. The llm-d inference scheduler (EPP) v0.8.x watches InferenceObjective and InferenceModelRewrite (inference.networking.x-k8s.io). The per-model EPP Role lacked RBAC for them, so the informers never synced and the EPP crash-looped, leaving served models with no healthy upstream. Added get/list/watch.

Also: the dev mock-vllm now answers POST /v1/chat/completions so the dev stack can exercise the full served path.

Validation

Verified end-to-end on a kind cluster (EG v1.6.7 + AI Gateway v0.5), operator-driven, for both served (mock-vllm) and passthrough (real OpenRouter) models:

  • a key minted for model A returns 200 for A and 403 for every other model, in both directions
  • invalid key: 401; undeclared model (catchAll: false): 404; a model with no keys: 403 for any key
  • revoking a key (key-manager) returns 401 immediately; a freshly minted key works within ~4s

Automated: reconciler unit tests cover the rendered SecurityPolicy (forwardClientIDHeader, sanitize, deny-by-default authz, the per-model allow-list, and pooled credentialRefs) for both the served and passthrough paths.

Not yet validated against a live system: the internal/JWT endpoint, which needs a real Keycloak OIDC issuer (not available on a bare kind cluster). It is covered by unit tests and is not touched by these changes.

Blocked by #115

Stacked on #115 (the EG v1.6.7 dev-stack alignment). Based on feat/local-dev-passthrough-and-ui-devmode so the diff shows only this change; merges after #115. GitHub will retarget to main once #115 lands.

@dcmcand dcmcand added the status: blocked ⛔️ Another task is blocking this item label Jun 26, 2026
@dcmcand dcmcand changed the title fix(operator): scope API keys per-model via per-route authorization (#116) fix(operator): make per-model API-key auth work end-to-end on AI Gateway v0.5 (#116) Jun 29, 2026
This was referenced Jun 30, 2026
Open
Open
Open
Open
@dcmcand

dcmcand commented Jun 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Tested this live on a fresh AWS Nebari cluster (Envoy Gateway v1.6.2 + AI Gateway v0.5, GitOps via ArgoCD, operator pinned to the branch build), with one served LLMModel and OpenRouter PassthroughModels. Surfaced four problems. Three are fixed on this branch; the fourth is filed for follow-up.

What works

  • The operator generates the per-model external SecurityPolicy correctly: pooled credentialRefs across every model's api-keys Secret, forwardClientIDHeader: x-llm-client-id with sanitize: true, and a deny-by-default authorization block.
  • Missing/invalid key returns 401. Deny-by-default holds (a valid pooled key whose client ID is not in a model's allow-list gets 403, not 401).
  • Routing works: models match on x-ai-eg-model, no "model not configured" 404s, both llm-https and llm-internal-https listeners program.

Problems found

  1. EPP Role RBAC: operator ClusterRole missing inference.networking.x-k8s.io grants (served models fail on Helm install) #121 EPP Role RBAC: change 4 grants inference.networking.x-k8s.io to the EPP Role but not to the operator's own ClusterRole, so privilege-escalation prevention blocks the operator from creating it. Every served model fails to reconcile on a Helm install. Fixed: 0640fb2.
  2. Per-model API-key scoping broken when one user has keys for multiple models (colliding client IDs) #122 Per-model scoping breaks when one user has keys for multiple models. The key-manager assigns the same client ID (user-<name>-1) to the first key of every model; pooled into each SecurityPolicy's credentialRefs, the duplicate ID collides, so one model's key authenticates and authorizes for another and the user's other keys 401. Live: a user with keys for three models had only one key authenticate, and it returned 200 on a model it was not minted for. Fixed: d9f5a15 (client ID scoped by model + regression test).
  3. vLLM container crashloops on llm-d-cuda:v0.7.0 (no command set, NVIDIA wrapper entrypoint) #123 vLLM crashloops on llm-d-cuda:v0.7.0: the operator sets Args but no Command, and that image's entrypoint is the NVIDIA wrapper with no default CMD, so it exec's the flags directly (exec: --: invalid option). Served models never start. Fixed: c512e37.
  4. Served models on tainted GPU nodes can't get Longhorn storage #124 Served models on tainted GPU nodes can't get Longhorn storage (longhorn-manager isn't scheduled there). Worked around per-model with a gp2 storageClassName; filed for a proper fix.

All three code fixes are stacked on this branch. Re-validating end to end now with a clean teardown and reinstall.

@dcmcand dcmcand force-pushed the fix/116-model-scoped-api-key-auth branch from eb62722 to d9f5a15 Compare June 30, 2026 13:35
@dcmcand dcmcand deleted the branch main July 7, 2026 12:19
@dcmcand dcmcand closed this Jul 7, 2026
dcmcand added 14 commits July 7, 2026 14:29
@dcmcand
feat(operator): add ClientIDsFromSecret helper for api-key client IDs
cf189e9
@dcmcand
feat(operator): scope API keys per-model via per-route authorization
f6ecc79 
Add apiKeyAuth.forwardClientIDHeader + sanitize and a deny-by-default
authorization block listing each model's own client IDs to the external
SecurityPolicy. A key valid on the shared listener now gets 403 against a
model it was not minted for. Covers LLMModel and PassthroughModel.

Closes #116
@dcmcand
feat(operator): re-render api-key authorization when keys change
9986ab2 
Watch managed api-keys Secrets and enqueue the owning model so minting or
revoking a key promptly updates the per-route authorization allow-list.
@dcmcand
docs: describe per-model API-key authorization on the external endpoint
f9b8ac6
@dcmcand
docs: fold redundant forwardClientIDHeader bullet into the authorizat…
d6aa9ac 
…ion paragraph
@dcmcand
fix(operator): drop Host matcher from AIGatewayRoute so models route …
2f68a9f 
…on AI Gateway v0.5

The AI Gateway v0.5 controller does not register a model whose match rule
carries any header beyond x-ai-eg-model. The operator added a Host matcher
(defence-in-depth on top of sectionName) to every model route, which made every
request 404 "model not configured in the Gateway", blocking all model serving
on EG v1.6.7 / AI Gateway v0.5. Remove it from the served (buildAIGatewayRoute)
and passthrough declared-model rules; sectionName already scopes each route to
its listener. The passthrough catch-all rule is unchanged (opt-in, no
x-ai-eg-model).

Verified end-to-end on kind for served and passthrough models.
@dcmcand
fix(operator): grant EPP RBAC for the inference scheduler's GIE watches
67d45a2 
The llm-d inference scheduler (EPP) v0.8.x watches InferenceObjective and
InferenceModelRewrite (inference.networking.x-k8s.io). Without RBAC the
informers never sync and the EPP crash-loops ("failed waiting for
*v1alpha2.InferenceObjective Informer to sync"), so served models never get a
healthy upstream. Add get/list/watch on the x-k8s.io inference resources to the
per-model EPP Role.
@dcmcand
feat(operator): pool api-key authentication across the shared listener
ecd1b96 
EG's api_key_auth filter runs before the AI Gateway ext_proc resolves the
model, so authentication happens on whichever catch-all route wins, not the
model's own route. With per-route credentials only the winning model's keys
authenticated; every other model's keys got 401. Pool every model's api-keys
Secret into each external SecurityPolicy's credentialRefs so any valid key
authenticates on the shared listener, while the per-route authorization block
(deny-by-default + the model's own client IDs) still scopes each request to its
model. All Secrets are same-namespace (#59). The controllers gather the full
Secret set and re-reconcile all models when an api-keys Secret changes.

Completes per-model API-key auth for #116: a key minted for one model returns
200 for that model and 403 for every other, verified end-to-end on kind for
served and passthrough models.
@dcmcand
chore(dev): mock-vllm answers chat completions for end-to-end testing
71fecfe 
The dev mock-vllm only served /health and /v1/models; add a 200
OpenAI-compatible response for POST /v1/chat/completions (and /v1/completions)
so the dev stack can exercise the full served-model request path through the
gateway.
@dcmcand
fix(operator): grant the operator the inference.networking.x-k8s.io p…
fa745b7 
…erms the EPP Role needs

The per-model EPP Role grants get/list/watch on inferenceobjectives,
inferencemodelrewrites, inferencepools and inferencepoolimports
(inference.networking.x-k8s.io). Kubernetes privilege-escalation
prevention blocks the operator from creating a Role that grants
permissions the operator does not itself hold, so on a Helm/ArgoCD
install every served LLMModel failed to reconcile with "attempting to
grant RBAC permissions not currently held" - the EPP Role, and with it
the model's SecurityPolicy and AIGatewayRoute, were never created.

Add the matching get/list/watch rule to the operator ClusterRole so it
can create the EPP Role.
@dcmcand
fix(operator): set explicit vLLM command for the CUDA serving image
91b864f 
llm-d-cuda images use the NVIDIA CUDA wrapper as their entrypoint and ship
no default CMD, so the served pod must supply the command itself. The vllm
container set only Args (the vLLM flags); the wrapper exec'd them as the
command ("exec: --: invalid option") and the pod crash-looped. Set the
standard vLLM OpenAI-server command so the flags are its arguments.
@dcmcand
fix(key-manager): scope API-key client IDs by model
6914ac1 
The client-id sequence was counted per-model, so the first key a user
minted for any model was always "user-<name>-1". The operator pools every
model's api-keys Secret into each model's SecurityPolicy credentialRefs
(model-scoped auth) and uses the matched data key as the x-llm-client-id
for per-model authorization. Identical client IDs across models collide in
that pooled set: one model's key authenticates and authorizes for another,
and a user's other keys fail to authenticate at all (only one value
survives per duplicated key).

Include the model in the client ID (user-<name>-<model>-<n>) so each is
globally unique. Add a regression test asserting a user's keys for two
models get distinct client IDs.
@dcmcand
style(key-manager): gofmt handler_test.go struct field alignment
32ae2e4
@dcmcand
chore(docs): exclude architecture.md from content-parity
5b12f07 
PR #117 (#116) documents the now-enabled per-model API-key authorization in
architecture.md, so its body intentionally diverges from the frozen Hugo
source. Matches the existing local-development.md/installation.md exclusions.
@dcmcand dcmcand reopened this Jul 7, 2026
@dcmcand dcmcand changed the base branch from feat/local-dev-passthrough-and-ui-devmode to main July 7, 2026 12:54
@dcmcand dcmcand force-pushed the fix/116-model-scoped-api-key-auth branch from 11dcd11 to 5b12f07 Compare July 7, 2026 12:54
@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown

📄 Docs preview for fix/116-model-scoped-api-key-auth:
https://fix-116-model-scoped-api-key.llm-serving-pack.pages.dev

@dcmcand
fix(operator): use constants for authz actions and client-ID header
183ef1f 
golangci-lint (goconst) on current main flagged the repeated "Deny"/"Allow"
authorization-action literals and the "x-llm-client-id" header literal, which
already has the apiKeyClientIDHeader constant. Add authzActionAllow/authzActionDeny
constants in auth.go and reference all three constants from auth.go and the
reconciler tests. No behavior change - the rendered SecurityPolicy is identical.
@dcmcand dcmcand removed the status: blocked ⛔️ Another task is blocking this item label Jul 7, 2026
@dcmcand dcmcand requested a review from tylerpotts July 7, 2026 13:13
dcmcand added 6 commits July 9, 2026 13:31
@dcmcand
fix(key-manager): make client IDs collision-proof and never reuse liv…
0fdf5f7 
…e IDs

Two defects in the client-ID scheme, both regression-tested:

- "user-<name>-<model>-<n>" with hyphen separators is ambiguous: the
  pairs (mary, jane-chat) and (mary-jane, chat) composed the same
  clientID in two different Secrets, recreating the #122
  pooled-credential collision. An FNV-1a hash of the raw
  (username, model) pair is now part of every clientID, so distinct
  pairs always mint distinct IDs.
- The sequence number came from counting existing keys, so minting
  after a mid-sequence revocation reused a still-live clientID and
  silently overwrote that key in the Secret and its ConfigMap
  metadata. The sequence now probes upward until the ID is free in
  both.
@dcmcand
feat(operator): allow overriding the vLLM container command per model
2690f35 
Adds spec.serving.command to the LLMModel CRD. The default stays the
explicit vLLM OpenAI-server entrypoint required by the llm-d-cuda image
(its NVIDIA wrapper entrypoint ships no default CMD); the override lets
a custom serving.image use its own launcher.

The dev mock-vllm model needs it: forcing the vLLM command broke the
kind dev served path, since the mock image has no vllm module and
crash-looped with ModuleNotFoundError. dev/manifests/test-model.yaml
now sets command: ["python", "/server.py"].

configuration.md documents the field and joins the content-parity
exclusion list, following the same pattern as architecture.md.
@dcmcand
fix(dev): grant the operator the GIE x-k8s.io RBAC in the dev manifests
c10290d 
The #121 fix added inference.networking.x-k8s.io get/list/watch to the
Helm chart ClusterRole but not to dev/manifests/operator.yaml, so on
the kind dev stack RBAC escalation prevention still blocked the
per-model EPP Role and every served model failed to reconcile
("attempting to grant RBAC permissions not currently held"). Mirrors
the chart rule.
@dcmcand
refactor(operator): consolidate the api-keys Secret watch plumbing
fed7131 
The LLMModel and PassthroughModel controllers carried near-verbatim
copies of the api-keys Secret helpers (client-ID read, pooled Secret
listing, watch predicate, map function). They now share
internal/controller/apikeys.go, the "-api-keys" suffix is
single-sourced next to reconcilers.APIKeySecretName, and a failed List
in the map function is logged instead of silently dropping the
re-render.
@dcmcand
docs: record the validated AI Gateway v0.5 routing and client-ID cont…
e4bb9a1 
…ract

architecture.md still described the removed Host matcher, the old
two-header precedence rationale, and the pre-#117 client-ID format.
It now documents the behavior validated live on EG v1.6.7 /
AI Gateway v0.5:

- dispatch is decided by the ext_proc model registry, so served models
  win regardless of route age (verified against an older
  catchAll: true route)
- catchAll is currently inert: unregistered model ids 404 at the
  ext_proc before route matching runs
- key activation is split into its authentication (Secret sync,
  immediate) and authorization (operator re-render) halves
- the data-model examples use the pair-hashed client-ID format

The buildPassthroughRoute comments state the same validated behavior.
@dcmcand
docs: finish aligning prose with pooled authentication
2c8f010 
Follow-up drift sweep across docs/src/content/docs against the code at
this branch's head:

- architecture.md: explain why the pooled apiKeyAuth credentialRefs live
  on each per-route SecurityPolicy instead of one gateway-level policy
  (route-level policies take precedence over gateway-level ones rather
  than merging, and each route needs its own policy for the per-model
  authorization anyway), and note the deliberate exception to the
  cluster-singleton rule; fix the resources-table row that still said
  the external policy references only the per-model Secret; the
  PassthroughModel setup bullet now mentions the api-keys Secret watch.
- installation.md: the wrong-model-key troubleshooting entry sat under
  401, but pooled authentication makes that case a 403; split into 401
  (key in no Secret) and 403 (key valid, not on this model's allow-list,
  or allow-list not yet re-rendered).
- local-development.md and the dev passthrough manifest: note the
  transient 403 window between patching a key into the Secret and the
  operator re-rendering the allow-list.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tylerpotts tylerpotts Awaiting requested review from tylerpotts

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

API-key auth is not model-scoped: any valid key works for every model on the shared gateway listener

1 participant

@dcmcand