feat(dev): align local dev stack with AI Gateway v0.5 + key-manager UI dev mode

[dcmcand]

What and why

Makes the local kind dev path actually run an external-provider PassthroughModel end to end, and lets the key-manager UI run without Keycloak.

Closes #113
Closes #114

#113 - dev stack version mismatch

dev/Makefile installed Envoy Gateway v1.3.0 and Gateway API v1.2.1 alongside Envoy AI Gateway v0.5.0, which needs Envoy Gateway v1.6.x and Gateway API v1.4.0 (compatibility matrix). On v1.3.0 a PassthroughModel reconciled to Ready but the operator's BackendTLSPolicy was never translated into an upstream TLS socket, so Envoy dialed the provider in plaintext and inference returned 503 UC.

• Pin the dependency versions at the top of dev/Makefile and move them as a set; bump to Envoy Gateway v1.6.7 and Gateway API v1.4.0.
• Install Envoy Gateway with the AI Gateway ext_proc wiring (dev/eg-extension-values.yaml) and bring the AI Gateway up first so the extension server exists.
• Apply the PassthroughModel CRD in make setup.
• Extend dev/manifests/ so the operator can serve passthrough: RBAC for passthroughmodels, aiservicebackends/backendsecuritypolicies/backends/backendtlspolicies, and the shared-TLS reconciler (certificates, gateways); the passthrough validating webhook; and shared-TLS issuance through the existing local selfsigned-issuer (LLM_CLUSTER_ISSUER_NAME), so no ACME or hand-made cert is needed.
• New make targets: create-openrouter-secret, apply-passthrough-model, ui.
• Operator: emit the PassthroughModel BackendTLSPolicy as gateway.networking.k8s.io/v1 instead of v1alpha3. Gateway API v1.4.0 graduates the policy to v1 and no longer serves v1alpha3, so on the version-aligned stack the old apiVersion failed to apply and the upstream never got a TLS socket. This raises the effective floor for passthrough to Gateway API v1.4 / Envoy Gateway v1.6, which is what AI Gateway v0.5 already requires.

#114 - key-manager UI dev mode

The UI could not run locally: the gateway enforces OIDC before forwarding, and the key-manager 401s without a JWT.

• LLM_DEV_MODE (off by default) makes the auth middleware skip token handling and inject a fixed identity (LLM_DEV_USER, LLM_DEV_GROUPS). The production path is unchanged when unset, and a warning is logged when it is on.
• Exposed as keyManager.devMode in the Helm chart (default disabled) and enabled in the dev manifest. make ui port-forwards the Service so the gateway OIDC layer is bypassed too.

One-command UI dev environment (make run-dev)

For frontend work on the key-manager UI, make run-dev is the whole setup: drop an OpenRouter key in dev/.env and run it. It idempotently brings up the cluster, operator, dev-mode key-manager, and three passthrough models, then port-forwards the key-manager and starts a hot-reloading UI dev server.

• dev/uidev/ is a zero-dependency (stdlib-only) Go dev server: it serves the UI static files from disk, proxies /api/* to the port-forwarded key-manager, and live-reloads the browser on edits. The UI is plain static files, so there is no build step or npm.
• dev/manifests/dev-models.yaml gives the UI a populated three-model list.
• docs/ui-development.md documents the workflow; dev/.env is gitignored.

Verification

• key-manager: go vet clean, full go test ./... passes, including a new table-driven TestAuthMiddlewareDevMode.
• Chart lints; renders the dev-mode env when devMode.enabled=true and omits it by default.
• Operator: go build, go vet, and the Passthrough reconciler tests pass with the v1 assertion.
• Clean end-to-end on a fresh cluster: make teardown && make setup && make build-images && make load-images && make deploy && make create-openrouter-secret && make apply-passthrough-model brings the PassthroughModel to Ready, the operator patches the llm-https listener, and a chat completion through the gateway returns a real OpenRouter response (200). The key-manager logs the dev-mode warning and GET /api/me returns the injected dev identity (200, not 401).

Notes

• The chart references a ClusterIssuer/selfsigned-issuer it does not create; the dev path supplies one via cert-manager-config.yaml. Whether the chart should ship a dev issuer is out of scope here.
• The internal endpoint (llm-internal.<domain>) still requires a real Keycloak JWT even when access is public, so only the external endpoint is reachable on kind.

Review feedback (Helm 4 run by @jbouder)

Fixes for the issues found running make run-dev on Helm 4. Verified on a fresh
kind cluster, including a Helm 4.2.2 reproduction of the CRD conflict and its fix.

• Gateway API CRD ownership conflict on Helm 4. make setup applies the
  standalone Gateway API CRDs with kubectl apply --server-side, and the eg
  chart install passes --force-conflicts so it co-owns the CRDs instead of
  erroring. The flag is gated on the Helm major version (HELM_FORCE_CONFLICTS):
  Helm 4+ gets it, Helm 3 does not (no server-side apply, skips already-present
  crds/, and rejects the flag).
• make setup resumability. kind create cluster is guarded so a setup that
  dies midway re-runs without a teardown.
• Webhook startup race. operator.yaml gains a readinessProbe on the webhook
  port (tcpSocket: 9443) so the make deploy rollout waits for the webhook to
  serve. run-dev.sh also retries the dev-models.yaml apply: the probe narrows
  the race, but Service endpoint propagation still lags pod-Ready by about a second,
  so the retry is what guarantees reliability.
• Stale port-forward log. run-dev.sh uses a per-run mktemp file instead of a
  fixed /tmp path.

[jbouder]

Reviewed by actually running make run-dev on a fresh kind cluster (Helm 4). Two blockers stopped the bring-up (Gateway API CRD conflict; operator webhook startup race), plus a few smaller things. Details inline.

[jbouder]

🔴 Blocker on Helm 4: Gateway API CRD ownership conflict.

On Helm 4 (server-side apply is the default), the eg install at L39 aborts because the chart re-applies these same Gateway API CRDs via SSA and collides with this client-side kubectl apply:

Error: failed to install CRD crds/gatewayapi-crds.yaml: conflict occurred while applying object
... conflicts with "kubectl-client-side-apply"

This blocks make setup / make run-dev on a fresh cluster for anyone on Helm 4. Both install the same pinned version, so taking ownership is safe — switch this to server-side apply:

-	kubectl apply -f https://.../$(GATEWAY_API_VERSION)/standard-install.yaml
+	kubectl apply --server-side --force-conflicts -f https://.../$(GATEWAY_API_VERSION)/standard-install.yaml

and add --force-conflicts to the eg helm install (see comment on L39).

[jbouder]

Part of the Helm 4 CRD fix (see L27): add --force-conflicts so the chart's bundled Gateway API CRDs cleanly take ownership instead of erroring.

-	helm upgrade -i eg oci://.../gateway-helm --version $(ENVOY_GATEWAY_VERSION) -n envoy-gateway-system --create-namespace -f eg-extension-values.yaml
+	helm upgrade -i eg oci://.../gateway-helm --version $(ENVOY_GATEWAY_VERSION) -n envoy-gateway-system --create-namespace --force-conflicts -f eg-extension-values.yaml

[jbouder]

🟡 make setup isn't resumable after a partial failure. kind create cluster errors hard (node(s) already exist for a cluster with the name ...) when the cluster exists, aborting the whole target. Since run-dev.sh only calls make setup when the cluster is absent, a setup that dies midway (e.g. the CRD conflict above) can't be recovered with make setup or make run-dev — you have to make teardown first. Guarding the create makes it idempotent:

@kind get clusters | grep -qx $(CLUSTER_NAME) || kind create cluster --name $(CLUSTER_NAME)

[jbouder]

🔴 Webhook startup race — this apply fails intermittently and set -euo pipefail kills the run. The operator's validating webhook gates PassthroughModel creates, but it isn't serving yet right after make deploy:

Error from server (InternalError): ... failed calling webhook "vpassthroughmodel-v1alpha1.kb.io":
Post "https://llm-operator-webhook-service.../validate-...": dial tcp ...:443: connect: connection refused

Root cause is in operator.yaml (no readiness probe — see that comment); a bounded retry here makes it reliable regardless:

Suggested change

	kubectl apply -f manifests/dev-models.yaml >/dev/null
	# The operator's validating webhook gates PassthroughModel creates, and isn't
	# serving the instant `make deploy`'s rollout returns. Retry until it accepts.
	for attempt in $(seq 1 30); do
	kubectl apply -f manifests/dev-models.yaml >/dev/null 2>&1 && break
	if [[ $attempt -eq 30 ]]; then
	echo "ERROR: operator webhook never became ready" >&2
	kubectl apply -f manifests/dev-models.yaml >&2 || true
	exit 1
	fi
	echo "==> operator webhook not ready yet, retrying ($attempt)..."
	sleep 2
	done

[jbouder]

🟢 Nit: /tmp/km-portforward.log is a fixed path. A stale file from a crashed prior run can make the grep -q "Forwarding from" readiness check below pass instantly against old output. mktemp would avoid that.

[dcmcand]

@jbouder your Helm 4 run caught two things I couldn't see on Helm 3.18. All six items are fixed and pushed:

• Gateway API CRD conflict: make setup now applies the CRDs with --server-side, and the eg install passes --force-conflicts, gated on the Helm major version (Helm 3 has no server-side apply and rejects the flag; Helm 4 needs it). I reproduced both the conflict and the fix on Helm 4.2.2.
• make setup resumability: guarded kind create cluster.
• Webhook race: added the readinessProbe on 9443, plus the bounded retry in run-dev.sh. One correction worth flagging: the probe narrows the race but does not close it. It gates pod readiness, but the Service endpoint propagates about a second after the pod goes Ready, so a create fired the instant make deploy returns can still hit connection refused. The retry is what makes it reliable, so I kept both.
• Port-forward log: switched to mktemp.

Verified end to end on a fresh kind cluster. Ready for another look.

[jbouder]

Works great!

[github-actions]

📄 Docs preview for feat/local-dev-passthrough-and-ui-devmode:
https://feat-local-dev-passthrough-a.llm-serving-pack.pages.dev