Summary
The BraintrustStream streaming aggregator merges all choice indices into a single OutputChoice at index 0. When a Chat Completions request uses n > 1 to generate multiple parallel completions, the streamed chunks from different choice indices are concatenated into one string, producing a corrupted single output instead of preserving each choice separately.
What is missing
OpenAI Chat Completions streaming with n > 1 sends chunks tagged with a choices[].index field to distinguish parallel completions:
{"choices": [{"index": 0, "delta": {"role": "assistant", "content": "Hello"}}]}
{"choices": [{"index": 1, "delta": {"role": "assistant", "content": "Hi"}}]}
{"choices": [{"index": 0, "delta": {"content": " world"}}]}
{"choices": [{"index": 1, "delta": {"content": " there"}}]}The expected aggregated output should be two choices:
- Choice 0:
"Hello world"
- Choice 1:
"Hi there"
Currently in src/stream.rs:727-807, the aggregate() function:
- Ignores the choice
index — loops over all chunk.choices (line 771) but accumulates into a single aggregated_content: String and single role: Option<String> regardless of which choice index the delta belongs to
- Hardcodes a single output — creates one
OutputChoice at index 0 (line 804) and wraps it in a single-element vector (line 806)
- Corrupts content — text from choice 0 and choice 1 are interleaved into one string (
"Hello Hi world there")
- Loses finish reasons — only the last
finish_reason seen across all choices is kept (line 773-777), discarding per-choice finish reasons
The fix would involve:
- Tracking per-index state (content, role, tool_calls, finish_reason) using a
HashMap<usize, ChoiceState> or similar
- Building one
OutputChoice per distinct index seen in the stream
- Returning all choices in the
FinalizedStream::output vector
Braintrust docs status
supported — Braintrust's streaming documentation states that streaming responses are fully supported and automatically aggregated. The OpenAI integration docs do not specifically mention n > 1 handling. Status: unclear for multi-choice streaming specifically.
Upstream sources
Relationship to existing issues
Local files inspected
src/stream.rs:727-807 — aggregate() uses single aggregated_content, role, finish_reason variables for all choices; creates one OutputChoice at index 0 on line 804src/stream.rs:658-664 — StreamChoice struct does have an implicit index from the array position but no explicit index field parsed from the JSONsrc/stream.rs:354-395 — OutputChoice struct supports an index field, so the output type can represent multiple choicessrc/stream.rs:530-540 — FinalizedStream stores output: Vec<OutputChoice>, capable of holding multiple choices
Summary
The
BraintrustStreamstreaming aggregator merges all choice indices into a singleOutputChoiceat index 0. When a Chat Completions request usesn > 1to generate multiple parallel completions, the streamed chunks from different choice indices are concatenated into one string, producing a corrupted single output instead of preserving each choice separately.What is missing
OpenAI Chat Completions streaming with
n > 1sends chunks tagged with achoices[].indexfield to distinguish parallel completions:{"choices": [{"index": 0, "delta": {"role": "assistant", "content": "Hello"}}]} {"choices": [{"index": 1, "delta": {"role": "assistant", "content": "Hi"}}]} {"choices": [{"index": 0, "delta": {"content": " world"}}]} {"choices": [{"index": 1, "delta": {"content": " there"}}]}The expected aggregated output should be two choices:
"Hello world""Hi there"Currently in
src/stream.rs:727-807, theaggregate()function:index— loops over allchunk.choices(line 771) but accumulates into a singleaggregated_content: Stringand singlerole: Option<String>regardless of which choice index the delta belongs toOutputChoiceat index 0 (line 804) and wraps it in a single-element vector (line 806)"Hello Hi world there")finish_reasonseen across all choices is kept (line 773-777), discarding per-choice finish reasonsThe fix would involve:
HashMap<usize, ChoiceState>or similarOutputChoiceper distinct index seen in the streamFinalizedStream::outputvectorBraintrust docs status
supported — Braintrust's streaming documentation states that streaming responses are fully supported and automatically aggregated. The OpenAI integration docs do not specifically mention
n > 1handling. Status: unclear for multi-choice streaming specifically.Upstream sources
nparameter: https://platform.openai.com/docs/api-reference/chat/create — "n: How many chat completion choices to generate for each input message"choices[].indexto distinguish parallel completions: https://platform.openai.com/docs/api-reference/chat/streamingindexfield is present on every streamed choice deltaRelationship to existing issues
tool_callsfield inStreamDelta. This issue is about the aggregation logic ignoring the choiceindex, which affects all content (text, tool_calls, finish_reason) across multiple choices.StreamUsage. This issue is about structural aggregation logic.input_tokensto wrong field, dropsoutput_tokens#41 (StreamUsage serde alias bug): [BOT ISSUE] StreamUsage serde alias mapsinput_tokensto wrong field, dropsoutput_tokens#41 is about field name aliasing. This is about multi-choice index tracking.Local files inspected
src/stream.rs:727-807—aggregate()uses singleaggregated_content,role,finish_reasonvariables for all choices; creates oneOutputChoiceat index 0 on line 804src/stream.rs:658-664—StreamChoicestruct does have an implicit index from the array position but no explicitindexfield parsed from the JSONsrc/stream.rs:354-395—OutputChoicestruct supports anindexfield, so the output type can represent multiple choicessrc/stream.rs:530-540—FinalizedStreamstoresoutput: Vec<OutputChoice>, capable of holding multiple choices