From 780221d6b91721934b0df4ce69a083c7c890f726 Mon Sep 17 00:00:00 2001 From: kayx23 Date: Fri, 17 Jul 2026 17:33:29 +0800 Subject: [PATCH] docs: document AI proxy multi stream flushing --- docs/en/latest/plugins/ai-proxy-multi.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/en/latest/plugins/ai-proxy-multi.md b/docs/en/latest/plugins/ai-proxy-multi.md index a91b0f43b160..7549670ffb53 100644 --- a/docs/en/latest/plugins/ai-proxy-multi.md +++ b/docs/en/latest/plugins/ai-proxy-multi.md @@ -126,6 +126,7 @@ When an instance's `provider` is set to `bedrock`, the Plugin expects requests i | max_req_body_size | integer | False | 67108864 | greater than or equal to 1 | Maximum request body size in bytes that the plugin reads into memory. Requests whose body exceeds this limit are rejected with `413`. Prevents unbounded memory buffering of large request bodies. | | max_stream_duration_ms | integer | False | | greater than or equal to 1 | Maximum wall-clock duration (in milliseconds) for a streaming AI response. If the upstream keeps sending data past this deadline, the gateway closes the connection. Unset means no cap. Use this to protect the gateway from upstream bugs that produce tokens indefinitely. When the limit is hit mid-stream, the downstream SSE stream is truncated (no protocol-specific terminator such as `[DONE]`, `message_stop`, or `response.completed`); well-behaved clients should treat a missing terminator as an incomplete response. | | max_response_bytes | integer | False | | greater than or equal to 1 | Maximum total bytes read from the upstream for a single AI response (streaming or non-streaming). If exceeded, the gateway closes the connection. For non-streaming responses with `Content-Length`, the check is performed before reading the body; for chunked (no-`Content-Length`) non-streaming responses and for streaming responses, the cap is enforced incrementally as bytes are received. Unset means no cap. | +| streaming_flush_interval_ms | integer | False | 10 | greater than or equal to 0 | Interval in milliseconds for the background flush thread. When greater than 0 (default: 10), a background timer calls `ngx.flush(false)` every N milliseconds, batching output for bursty upstreams. When 0, the background thread is disabled and each chunk is flushed synchronously through `ngx.flush(true)`, guaranteeing immediate client delivery. | | keepalive | boolean | False | true | | If true, keep the connection alive when requesting the LLM service. | | keepalive_timeout | integer | False | 60000 | greater than or equal to 1000 | Request timeout in milliseconds when requesting the LLM service. | | keepalive_pool | integer | False | 30 | | Keepalive pool size for when connecting with the LLM service. |