Skip to content

Fix streamed message model_id latching onto empty string (nil costs on Azure OpenAI)#830

Merged
crmne merged 2 commits into
crmne:mainfrom
bdegomme:fix-stream-accumulator-empty-model-id
Jul 3, 2026
Merged

Fix streamed message model_id latching onto empty string (nil costs on Azure OpenAI)#830
crmne merged 2 commits into
crmne:mainfrom
bdegomme:fix-stream-accumulator-empty-model-id

Conversation

@bdegomme

@bdegomme bdegomme commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

What this does

Fixes streamed messages losing their model_id — and therefore all cost information — when the provider's first stream chunk carries an empty model field.

Azure OpenAI's first SSE chunk (the prompt filter chunk) has "model": "", while subsequent chunks carry the real id:

chunk model_ids: ["", "gpt-5.4-2026-03-05", "gpt-5.4-2026-03-05", ...]

StreamAccumulator#add does @model_id ||= chunk.model_id, and since "" is truthy, the empty string latches for the whole stream. The assembled Message then has model_id: "", Message#model_info raises ModelNotFoundError internally, and message.cost returns nil for every component (total, input, output, …) even though token usage is fully populated. Non-streamed requests against the same endpoint price correctly.

The fix keeps the first non-empty model_id instead:

@model_id = chunk.model_id if @model_id.to_s.empty?

Verified against a live Azure OpenAI deployment: before the change a streamed chat.ask returned cost.total => nil; after, it resolves the dated model and prices identically to the non-streaming path.

Type of change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation
  • Performance improvement

Scope check

  • I read the Contributing Guide
  • This aligns with RubyLLM's focus on LLM communication
  • This isn't application-specific logic that belongs in user code
  • This benefits most users, not just my specific use case

Quality check

  • I tested my changes thoroughly
    • All tests pass: bundle exec rspec (stream accumulator, streaming, and protocol specs)
  • I updated documentation if needed
  • I didn't modify auto-generated files manually (models.json, aliases.json)

AI-generated code

  • I used AI tools to help write this code
  • I have reviewed and understand all generated code (required if above is checked)

API changes

  • Breaking change
  • New public methods/classes
  • Changed method signatures
  • No API changes

bdegomme and others added 2 commits July 2, 2026 11:03
Azure OpenAI's first stream chunk (the prompt filter chunk) carries
"model": "". StreamAccumulator's `@model_id ||= chunk.model_id` treats
the empty string as set, so the assembled message keeps an empty
model_id, model lookup fails, and message.cost returns nil for every
component despite valid token usage.

Keep the first non-empty model_id instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@codecov

codecov Bot commented Jul 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.18%. Comparing base (5e252ea) to head (0d149ee).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #830   +/-   ##
=======================================
  Coverage   82.18%   82.18%           
=======================================
  Files         170      170           
  Lines        8029     8029           
  Branches     1347     1348    +1     
=======================================
  Hits         6599     6599           
  Misses        890      890           
  Partials      540      540           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@crmne crmne merged commit 69c7dc2 into crmne:main Jul 3, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants