support file-based ASR for nemotron streaming model by rui-ren · Pull Request #885 · microsoft/foundry-local

rui-ren · 2026-07-15T17:13:43Z

Add file-based Nemotron ASR transcription support in C++ SDK v2

Summary

This PR completes file-based audio transcription support for Nemotron streaming ASR models in sdk_v2/cpp by finishing the C++ AudioSession implementation and aligning behavior with the existing OpenAI-JSON transcription flow.

For more details, please refer to github/copilot-cli#4024 (comment)

What changed

• Implemented/finished Nemotron file transcription path in:

 sdk_v2/cpp/src/inferencing/generative/audio/audio_session.cc 
 sdk_v2/cpp/src/inferencing/generative/audio/audio_session.h

• Added WAV file parsing for file-based transcription input:
• RIFF/WAVE validation
• fmt / data chunk handling with bounds checks
• 16kHz enforcement
• PCM16 and float32 decoding
• Added Nemotron-specific language runtime option mapping ( lang_id ) and temperature handling.
• Fixed decode flow to keep a single generator across incremental audio processing (100ms chunks + flush), which resolves truncated transcription output for file-based runs.

Behavior

• OpenAI-JSON file transcription for Nemotron models now returns full transcript text from WAV input in SDK v2.
• Existing streaming and non-Nemotron paths remain unchanged.

End-to-end scenario covered

• Model alias: nemotron-3.5-asr-streaming-0.6b
• Input: sample-speech-1m-16k.wav
• Path exercised: JS SDK v2 AudioClient.transcribe(...) → C++ SDK v2 audio transcription pipeline

vercel · 2026-07-15T17:13:50Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
foundry-local	Ready	Preview, Comment	Jul 15, 2026 5:13pm

Copilot

Pull request overview

Adds file-based WAV transcription for Nemotron streaming ASR models in the C++ SDK.

Changes:

Routes Nemotron file requests through the streaming processor.
Parses 16 kHz PCM16/float32 WAV files with channel downmixing.
Adds language mapping, temperature handling, and OpenAI JSON output.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
`audio_session.h`	Declares Nemotron transcription and WAV parsing helpers.
`audio_session.cc`	Implements Nemotron inference, decoding, language mapping, and WAV parsing.

+  int completion_tokens = 0;
+
+  auto decode_all_tokens = [&]() {
+    while (!generator->IsDone() && !original_request.canceled) {


+    size_t count = std::min(kNemotronSamplesPerChunk, samples.size() - offset);
+    run_one_pass(processor->Process(samples.data() + offset, count));
+  }
+  run_one_pass(processor->Flush());


+    }
+  }
+
+  auto samples = LoadPcmWavAsFloatSamples(req.filename);


nenad1002 · 2026-07-15T17:42:28Z

Why? Parakeet can execute faster and more accurate file-based transcription. I wouldn't want customers to rely on streaming models for non-streaming scenarios, and we're encouraging them with this.

rui-ren · 2026-07-15T17:48:21Z

Why? Parakeet can execute faster and more accurate file-based transcription. I wouldn't want customers to rely on streaming models for non-streaming scenarios, and we're encouraging them with this.

This is not a way to encourage them to do file-base transcription with RNNT streaming model, as you can see here,

github/copilot-cli#4024 (comment)

they are using streaming model, however they do file-based transcription and get error raised. IMO, we better to cover this edge case for them, instead of switch models?

What do you think? @sylvanc

nenad1002 · 2026-07-15T18:09:28Z

Why? Parakeet can execute faster and more accurate file-based transcription. I wouldn't want customers to rely on streaming models for non-streaming scenarios, and we're encouraging them with this.

This is not a way to encourage them to do file-base transcription with RNNT streaming model, as you can see here,

github/copilot-cli#4024 (comment)

they are using streaming model, however they do file-based transcription and get error raised. IMO, we better to cover this edge case for them, instead of switch models?

What do you think? @sylvanc

Hmm, I haven't realized they're using file-based transcription. Ok, but there are better models for this going forward.

rui-ren · 2026-07-20T16:45:45Z

@skottmckay can you please review this PR ? Thanks.

kunal-vaishnavi · 2026-07-22T18:45:11Z

+  text.reserve(512);
+  int completion_tokens = 0;
+
+  auto decode_all_tokens = [&]() {


Can we put this in a named helper function rather than use a lambda function?

kunal-vaishnavi · 2026-07-22T18:45:27Z

+    }
+  };
+
+  auto run_one_pass = [&](std::unique_ptr<OgaNamedTensors> tensors) {


Same comment here

kunal-vaishnavi · 2026-07-22T18:46:20Z


+  // If the model is a Nemotron speech model, process the transcription using the Nemotron-specific method.
+  // e.g. RNNT models require Nemotron-specific transcription handling.
+  if (IsNemotronSpeechModel()) {


I presume that the v2 SDK already supported Nemotron when it was merged into main. Why do we now need an early return to support it?

kunal-vaishnavi · 2026-07-22T18:50:34Z

+  auto tokenizer_stream = OgaTokenizerStream::Create(*tokenizer);
+  auto generator_params = OgaGeneratorParams::Create(oga_model);
+  if (temperature.has_value()) {
+    generator_params->SetSearchOption("temperature", *temperature);


Can we simplify this if-else branching to the following?

generator_params->SetSearchOption("temperature", temperature.value_or(0.0f));

kunal-vaishnavi · 2026-07-22T18:51:55Z

+                                                      FOUNDRY_LOCAL_TEXT_ITEM_TYPE_OPENAI_JSON));
+}
+
+std::vector<float> AudioSession::LoadPcmWavAsFloatSamples(const std::string& audio_file_path) {


From Copilot:

Good addition overall — the Nemotron transcription path is clean, the WAV parser is thorough, and keeping a single generator across chunks is the right fix for truncated output. A few things worth addressing before merging:

🔴 Blocking issues

WAV chunk bounds check is subtly wrong (LoadPcmWavAsFloatSamples, chunk-walking loop)
The bounds guard chunk_start + chunk_size > file_size - (chunk_size % 2 == 1 ? 1 : 0) mixes the padding-byte adjustment into the main bounds check, which can undercount available space and let a crafted file slip through. The padding byte lives after the chunk data, so it shouldn't reduce the window used to validate the chunk itself. Suggested fix:

// Before: chunk_start + chunk_size > file_size - (chunk_size % 2 == 1 ? 1 : 0) // After: if (chunk_start < 0 || chunk_start + static_cast<std::streamoff>(chunk_size) > file_size) { FL_THROW(FOUNDRY_LOCAL_ERROR_INVALID_USAGE, "Corrupted WAV chunk."); }

The odd-byte padding seek that follows handles alignment correctly as-is.

Format validation happens after full data load (LoadPcmWavAsFloatSamples, data chunk branch)
data.resize(chunk_size) allocates the full audio buffer before sample rate and format have been validated. For a large or malformed file, this could allocate hundreds of MB only to throw immediately afterward. Consider moving the sample_rate, audio_format, and bits_per_sample checks to right after the fmt chunk is parsed, before the data chunk is even encountered.

🟡 Suggestions

Gap at language ID 5 in NemotronLanguageIdMap — IDs jump from 4 (zh-cn) to 6 (hi). If this is intentional (reserved/unsupported language), a brief comment would prevent future contributors from assuming it's a typo.

prompt_tokens hardcoded to 0 in ProcessNemotronFileTranscription — the existing path computes this from the processor. If it's not meaningful for Nemotron, a // TODO explaining why would help maintain parity expectations with the OpenAI usage spec.

decode_all_tokens lambda and thread-safety — the lambda captures generator by reference and is safe as used today (same-thread, same scope). A short comment noting it must not be moved off the creation thread would protect against future refactors.

Rui Ren and others added 2 commits July 14, 2026 10:45

add file based support for streaming model

77170bd

update the cpp code for streaming model

d5eafd6

Copilot AI review requested due to automatic review settings July 15, 2026 17:13

rui-ren changed the title ~~Ruiren/add file based support nemotron streaming~~ support file-based ASR for nemotron streaming model Jul 15, 2026

rui-ren requested a review from skottmckay July 15, 2026 17:14

Copilot started reviewing on behalf of rui-ren July 15, 2026 17:14 View session

rui-ren requested review from baijumeswani, kunal-vaishnavi and nenad1002 July 15, 2026 17:14

Copilot AI reviewed Jul 15, 2026

View reviewed changes

kunal-vaishnavi reviewed Jul 22, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

support file-based ASR for nemotron streaming model#885

support file-based ASR for nemotron streaming model#885
rui-ren wants to merge 2 commits into
mainfrom
ruiren/add-file-based-support-nemotron-streaming

rui-ren commented Jul 15, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Jul 15, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

nenad1002 commented Jul 15, 2026 •

edited

Loading

Uh oh!

rui-ren commented Jul 15, 2026 •

edited

Loading

Uh oh!

nenad1002 commented Jul 15, 2026

Uh oh!

rui-ren commented Jul 20, 2026

Uh oh!

kunal-vaishnavi Jul 22, 2026

Uh oh!

kunal-vaishnavi Jul 22, 2026

Uh oh!

kunal-vaishnavi Jul 22, 2026

Uh oh!

kunal-vaishnavi Jul 22, 2026

Uh oh!

kunal-vaishnavi Jul 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

rui-ren commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Add file-based Nemotron ASR transcription support in C++ SDK v2

Summary

What changed

Behavior

End-to-end scenario covered

Uh oh!

vercel Bot commented Jul 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

nenad1002 commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rui-ren commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nenad1002 commented Jul 15, 2026

Uh oh!

rui-ren commented Jul 20, 2026

Uh oh!

kunal-vaishnavi Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

kunal-vaishnavi Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

kunal-vaishnavi Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

kunal-vaishnavi Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

kunal-vaishnavi Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rui-ren commented Jul 15, 2026 •

edited

Loading

nenad1002 commented Jul 15, 2026 •

edited

Loading

rui-ren commented Jul 15, 2026 •

edited

Loading