Problem
Large OpenAI-compatible /v1/chat/completions requests can exceed Envoy Gateway's default downstream connection buffer limit before the request reaches Envoy AI Gateway or vLLM. Envoy AI Gateway examples raise ClientTrafficPolicy.spec.connection.bufferLimit to 50Mi for AI workloads.
In a Nebari deployment where nebari-llm-serving is configured to patch llm-https / llm-internal-https listeners onto a shared platform Gateway, the practical workaround may be to attach that larger buffer policy to the entire shared Gateway. That fixes LLM requests, but it also changes downstream buffering behavior for unrelated services behind the same Gateway, such as landing pages, auth services, admin UIs, or other routes.
Why this matters
This is acceptable as a short-term deployment workaround, but it is not ideal as the long-term pattern. Buffer limits are connection/listener-level Envoy behavior, so deployments that need larger LLM request bodies should be able to isolate that behavior from non-LLM traffic.
Desired outcome
Provide or document a first-class deployment mode where LLM traffic uses a dedicated Gateway / Envoy entrypoint, so operators can safely apply LLM-specific client traffic policies such as:
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
spec:
targetRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: <dedicated-llm-gateway>
connection:
bufferLimit: 50Miwithout affecting unrelated traffic on the shared platform Gateway.
Notes
The chart already exposes platform.gateway.external and platform.gateway.internal, and the operator reads those values via LLM_EXTERNAL_GATEWAY_* / LLM_INTERNAL_GATEWAY_*. The missing piece is a clear supported pattern, docs, or optional chart-managed resources for creating and wiring a dedicated LLM Gateway, including DNS/certificate expectations.
Problem
Large OpenAI-compatible
/v1/chat/completionsrequests can exceed Envoy Gateway's default downstream connection buffer limit before the request reaches Envoy AI Gateway or vLLM. Envoy AI Gateway examples raiseClientTrafficPolicy.spec.connection.bufferLimitto50Mifor AI workloads.In a Nebari deployment where
nebari-llm-servingis configured to patchllm-https/llm-internal-httpslisteners onto a shared platform Gateway, the practical workaround may be to attach that larger buffer policy to the entire shared Gateway. That fixes LLM requests, but it also changes downstream buffering behavior for unrelated services behind the same Gateway, such as landing pages, auth services, admin UIs, or other routes.Why this matters
This is acceptable as a short-term deployment workaround, but it is not ideal as the long-term pattern. Buffer limits are connection/listener-level Envoy behavior, so deployments that need larger LLM request bodies should be able to isolate that behavior from non-LLM traffic.
Desired outcome
Provide or document a first-class deployment mode where LLM traffic uses a dedicated Gateway / Envoy entrypoint, so operators can safely apply LLM-specific client traffic policies such as:
without affecting unrelated traffic on the shared platform Gateway.
Notes
The chart already exposes
platform.gateway.externalandplatform.gateway.internal, and the operator reads those values viaLLM_EXTERNAL_GATEWAY_*/LLM_INTERNAL_GATEWAY_*. The missing piece is a clear supported pattern, docs, or optional chart-managed resources for creating and wiring a dedicated LLM Gateway, including DNS/certificate expectations.