stream network progress to SSE clients - #15
Merged
Conversation
The /v1/chat/completions SSE path waited for the full T4T round-trip (provider selection, postJob, PSS notify, ack, deliver) before flushing any chunks. During that 5-30s window the HTTP connection sits idle and Open WebUI / fetch-based agents commonly hit their read timeout — the user sees no feedback either. Wire a ProgressEvent callback through handleChat and emit at each milestone, including the ack/deliver events that fire from the PSS subscriber. The SSE endpoint flushes headers immediately, renders each event as a markdown-blockquote line, sends a `: keepalive` comment every 10s, and only then streams the actual completion content. Errors are surfaced as a final blockquote chunk instead of a dropped socket. https://claude.ai/code/session_01U3z9KZi7mfiQvsMSWH7RKS
Previous patch streamed progress as markdown blockquote lines inside the assistant message, which works but renders as a single assistant bubble with status text at the top — not the visually distinct "system message streaming, then AI message" UX requested. OpenAI's chat-completion streaming protocol is single-message-per- request: clients latch the role from the first chunk and concatenate everything else into one bubble, so a literal mid-stream `role: 'system'` chunk doesn't render as a separate message. Switch to the established pattern: open a `<details type="status" done="false"><summary>t4t network</summary>` block at the head of the assistant turn, write progress as bullet lines inside it, close the block before the model output. Open WebUI specifically renders this as a collapsed status pill above the assistant bubble — the user perceives two distinct messages. Errors now land as a bullet inside the status block (then the block closes) so half-open markdown can't leak into the chat. https://claude.ai/code/session_01U3z9KZi7mfiQvsMSWH7RKS
The response arrives from Swarm as one finalized blob in `job_deliver` — there's nothing to actually token-stream. The previous 32-char chunkText loop emitted N tiny chunks back-to-back within microseconds, which doesn't pace the rendering but does delay when the bubble first paints (clients wait for the role primer + a few chunks). Emit the whole content as a single delta so the assistant bubble pops in atomically the instant delivery lands. Test pins the one-chunk behavior so we don't regress to cosmetic chunking later. https://claude.ai/code/session_01U3z9KZi7mfiQvsMSWH7RKS
The <details type="status"> status block at the head of the assistant
message is great for Open WebUI / LibreChat, but it's HTML noise for
headless agents (Cline, OpenCode, Aider, raw OpenAI SDK with JSON mode):
the assistant content no longer starts with `{`, so JSON.parse breaks,
and tool-call extractors get a markdown preamble they didn't ask for.
Detect structured output (response_format.type !== 'text', or non-empty
tools array) and skip the <details> wrapper entirely on those requests.
SSE keepalive comments still flow so the connection survives the
round-trip, but the assistant `delta.content` is exactly what the
provider produced. Errors in structured mode set finish_reason='error'
on the terminal chunk instead of injecting a markdown bullet, so SDKs
surface the failure cleanly.
https://claude.ai/code/session_01U3z9KZi7mfiQvsMSWH7RKS
Open WebUI's <details type="status"> rendering only works in Open WebUI.
Switching to <think> uses the de-facto reasoning-model convention
(DeepSeek R1, o1, QwQ, …), which every modern chat client — Open WebUI,
LibreChat, Continue, Cline, Cursor, Big-AGI — already renders as a
collapsed reasoning section, so users always see progress regardless of
client.
When the model's own response leads with <think>…</think> (reasoning
backends), splice that block into the gateway's thinking panel so the
user sees one unified collapsed section ("network status + model
reasoning") rather than two stacked ones, and strip the duplicate
<think> from the visible answer.
Structured-output requests (response_format / tools) still suppress the
wrapper entirely so JSON.parse / tool-call extraction stays clean.
https://claude.ai/code/session_01U3z9KZi7mfiQvsMSWH7RKS
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The /v1/chat/completions SSE path waited for the full T4T round-trip
(provider selection, postJob, PSS notify, ack, deliver) before flushing
any chunks. During that 5-30s window the HTTP connection sits idle and
Open WebUI / fetch-based agents commonly hit their read timeout — the
user sees no feedback either.
Wire a ProgressEvent callback through handleChat and emit at each
milestone, including the ack/deliver events that fire from the PSS
subscriber. The SSE endpoint flushes headers immediately, renders each
event as a markdown-blockquote line, sends a
: keepalivecomment every10s, and only then streams the actual completion content. Errors are
surfaced as a final blockquote chunk instead of a dropped socket.
https://claude.ai/code/session_01U3z9KZi7mfiQvsMSWH7RKS