feat(routing): make AI gateway route request timeout configurable#125
Open
Adam-D-Lewis wants to merge 1 commit into
Open
feat(routing): make AI gateway route request timeout configurable#125Adam-D-Lewis wants to merge 1 commit into
Adam-D-Lewis wants to merge 1 commit into
Conversation
The operator generates an AIGatewayRoute per model endpoint but set no request timeout, so the Envoy AI Gateway applied its built-in 60s default to the rendered HTTPRoute. That is too short for many LLM generations (long or reasoning outputs, large prompts, cold model loads) and cannot be raised by a BackendTrafficPolicy, since an explicit route timeout always takes precedence - the route itself is the only effective lever. Add a chart-wide defaults.routing.requestTimeout value, plumbed through the LLM_ROUTE_REQUEST_TIMEOUT env var and OperatorConfig into both the external and internal AIGatewayRoutes as spec.rules[].timeouts.request. It defaults to empty, which leaves the field unset and preserves the gateway's 60s default on upgrade; set it to a duration such as "600s" to raise the timeout. The value is validated against the Gateway API duration format at operator startup. PassthroughModel routes keep their own fixed 120s timeout and are unaffected. A per-model LLMModel.spec override is left for a follow-up.
b2d8e2b to
8b16b33
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The operator generates an
AIGatewayRoutefor each model's external(
llm.<baseDomain>) and internal (llm-internal.<baseDomain>) endpoint, butit never set a request timeout. The Envoy AI Gateway therefore applies its
built-in 60s default (
defaultRequestTimeout) to theHTTPRouteitrenders from each route. That is too short for many real LLM requests — long
or reasoning-heavy generations, large prompts, and cold model loads routinely
run past 60s and get cut off with a 504 / stream reset.
The route's own timeout is the only effective place to raise this:
HTTPRouterequest timeout takes precedenceover a
BackendTrafficPolicy(the policy timeout is appliedsetIfNil), soa policy attachment cannot override the 60s.
without fighting the operator's reconcile.
Change
Add a chart-wide Helm value
defaults.routing.requestTimeout, plumbedthrough:
values.yaml→LLM_ROUTE_REQUEST_TIMEOUTenv (operator Deployment) →config.OperatorConfig.RouteRequestTimeout→spec.rules[].timeouts.requeston both the external and internal
AIGatewayRoutes.It defaults to empty, which leaves the field unset and preserves the
gateway's existing 60s behavior on upgrade. Set it to a Gateway API duration
string to raise the timeout for all of a cluster's model routes:
Testing
go test ./internal/config/... ./internal/controller/reconcilers/...— newcases assert both routes carry
timeouts.requestwhen set and omit theblock when empty; the config loader parses the new env var.
go build ./...,gofmt,go vetclean.helm lint+helm templateverified: no timeout rendered by default, theconfigured duration when set.
Follow-up
A per-model override on
LLMModel.spec(so individual models can set theirown timeout, overriding this chart-wide default) is left for a separate PR.