Skip to content

feat(routing): make AI gateway route request timeout configurable#125

Open
Adam-D-Lewis wants to merge 1 commit into
mainfrom
feat/configurable-route-request-timeout
Open

feat(routing): make AI gateway route request timeout configurable#125
Adam-D-Lewis wants to merge 1 commit into
mainfrom
feat/configurable-route-request-timeout

Conversation

@Adam-D-Lewis

Copy link
Copy Markdown
Member

Problem

The operator generates an AIGatewayRoute for each model's external
(llm.<baseDomain>) and internal (llm-internal.<baseDomain>) endpoint, but
it never set a request timeout. The Envoy AI Gateway therefore applies its
built-in 60s default (defaultRequestTimeout) to the HTTPRoute it
renders from each route. That is too short for many real LLM requests — long
or reasoning-heavy generations, large prompts, and cold model loads routinely
run past 60s and get cut off with a 504 / stream reset.

The route's own timeout is the only effective place to raise this:

  • In Envoy Gateway an explicit HTTPRoute request timeout takes precedence
    over a BackendTrafficPolicy (the policy timeout is applied setIfNil), so
    a policy attachment cannot override the 60s.
  • The route is controller-generated, so it cannot be patched from GitOps
    without fighting the operator's reconcile.

Change

Add a chart-wide Helm value defaults.routing.requestTimeout, plumbed
through:

values.yamlLLM_ROUTE_REQUEST_TIMEOUT env (operator Deployment) →
config.OperatorConfig.RouteRequestTimeoutspec.rules[].timeouts.request
on both the external and internal AIGatewayRoutes.

It defaults to empty, which leaves the field unset and preserves the
gateway's existing 60s behavior on upgrade. Set it to a Gateway API duration
string to raise the timeout for all of a cluster's model routes:

defaults:
  routing:
    requestTimeout: "600s"

Testing

  • go test ./internal/config/... ./internal/controller/reconcilers/... — new
    cases assert both routes carry timeouts.request when set and omit the
    block when empty; the config loader parses the new env var.
  • go build ./..., gofmt, go vet clean.
  • helm lint + helm template verified: no timeout rendered by default, the
    configured duration when set.

Follow-up

A per-model override on LLMModel.spec (so individual models can set their
own timeout, overriding this chart-wide default) is left for a separate PR.

@Adam-D-Lewis Adam-D-Lewis requested a review from tylerpotts July 1, 2026 22:25
The operator generates an AIGatewayRoute per model endpoint but set no
request timeout, so the Envoy AI Gateway applied its built-in 60s default
to the rendered HTTPRoute. That is too short for many LLM generations
(long or reasoning outputs, large prompts, cold model loads) and cannot
be raised by a BackendTrafficPolicy, since an explicit route timeout
always takes precedence - the route itself is the only effective lever.

Add a chart-wide defaults.routing.requestTimeout value, plumbed through
the LLM_ROUTE_REQUEST_TIMEOUT env var and OperatorConfig into both the
external and internal AIGatewayRoutes as spec.rules[].timeouts.request.
It defaults to empty, which leaves the field unset and preserves the
gateway's 60s default on upgrade; set it to a duration such as "600s" to
raise the timeout. The value is validated against the Gateway API
duration format at operator startup.

PassthroughModel routes keep their own fixed 120s timeout and are
unaffected. A per-model LLMModel.spec override is left for a follow-up.
@Adam-D-Lewis Adam-D-Lewis force-pushed the feat/configurable-route-request-timeout branch from b2d8e2b to 8b16b33 Compare July 1, 2026 22:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants