[speechlm2] Add streaming inference pipeline for NemotronVoiceChat

[erastorgueva-nv]

Important

The Update branch button must only be pressed in very rare occassions.
An outdated branch is never blocking the merge of a PR.
Please reach out to the automation team before pressing that button.

What does this PR do ?

Add a streaming (real-time, chunk-by-chunk) inference pipeline for NemotronVoiceChat,
following the same architecture as the NeMo ASR Inference Pipelines.

Collection: speechlm2

Changelog

• Add StreamingS2SPipeline with generate_step() API for both batch file processing and server integration
• Add NemotronVoicechatInferenceWrapper with infer_one_step() implementing perception → LLM → TTS → codec decode
• Add perception cache with optional CUDA graph support for cache-aware streaming encoding
• Add S2SPipelineBuilder factory and Hydra config (s2s_streaming.yaml) for easy setup
• Add state management: slot-based S2SContextManager for decode state lifecycle, S2SStreamingState for output accumulation
• Add s2s_streaming_infer.py entry script for batch inference on files/manifests
• Extend DuplexSTTModel: KV cache support for Nemotron hybrid Mamba/Attention (with monkey-patches for upstream HF bugs), save_pretrained with tokenizer export, function head, ASR logit boosts, cache_position forwarding
• Speed up model loading: meta device initialization, skip codec silence token computation when codec has random weights
• Fix sampling: nan/inf check before top-p filtering, vectorized repetition penalty
• Fix byte-level BPE decoding and BOS/EOS preservation in text output
• Refactor tests: shared conftest.py fixtures, offline-vs-streaming parity test, no-crash config sweep
• Add docs: streaming_inference.rst with architecture, config reference, and server integration guide

Modifications to more general code - FYI @kevinhu-nv @Edresson

• Extend NemotronVoiceChat: from_pretrained supports loading from HF-format checkpoint with llm_artifacts/
• Extend EarTTSModel: vectorized depth-sum, precomputed RVQ schedule, optional torch.compile, subword cache
• Patches: _patch_nemotron_cache_bugs and _patch_nemotron_block_forward methods in DuplexSTTModel are patching bugs in the HF Nemotron model code so we can get the KV caching to work. The patches seem to work for me, though I wonder if we can use more up-to-date code that doesn't have the patches.

Usage

python examples/speechlm2/nemo_inference_pipelines/s2s_streaming_infer.py \
    audio_file=/path/to/audio.wav \
    s2s.model_path=/path/to/checkpoint \
    s2s.speaker_name="<name>" \
    s2s.engine_type="native" \
    streaming.chunk_size_in_secs=0.08 \
    streaming.buffer_size_in_secs=1.68

from nemo.collections.speechlm2.inference import S2SPipelineBuilder

pipeline = S2SPipelineBuilder.build_pipeline(cfg)
output = pipeline.run(audio_filepaths, options=options)

GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you add or update any necessary documentation?
• Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

• New Feature
• Bugfix
• Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

• Related to # (issue)

[pzelasko]

Partial review up to streaming_s2s_pipeline.py at line 511 (note to self where to pick up later)

[pzelasko]

Does this assume a single-turn evaluation? Or the audio file can have multiple turns and the agent is expected to handle that correctly? Let's clarify this in the docs.

[erastorgueva-nv]

Not sure what you mean - it's full-duplex, so it just generates one frame of output for every frame of audio input. Audio input can contain single-turn, muti-turn, whatever.

Or if you're asking about "evaluation" - the code doesn't support detailed "evaluation". We just generate text & audio for the full audio file (plus with an option to add silence padding at the end, so the agent can finish speaking). The one bit of "evaluation" we have is WER

[pzelasko]

it just generates one frame of output for every frame of audio input. Audio input can contain single-turn, muti-turn, whatever.

We just generate text & audio for the full audio file (plus with an option to add silence padding at the end, so the agent can finish speaking).

Let's write these in here - it's not obvious for outside reader what characterizes the inputs and outputs in this API.

[pzelasko]

Both examples here showcase audio_file. We need to mention how to perform live streaming inference (using mic or other streaming audio input connector) if it is supported by this API; or that it is not supported.

[pzelasko]

Is there another entry-point with a streaming input connector (mic)? We should mention.

[pzelasko]

Can we show how streamer is constructed? You'd normally refer the user to ASR pipelines documentation but it doesn't exist yet in main IIRC, so we need to describe at least basic concepts / APIs.

[pzelasko]

It looks like this method should live in DuplexSTT class? It's exposing inner workings of input construction to a high-level inference API.

If we build DuplexSTTv2 which does it completely differently, we don't want to re-write this wrapper - we should just call stt_model.build_input_embedding()

[pzelasko]

This method should be split to two and live in native / vllm LLM class

[pzelasko]

Should this (partially) live in ASR collection? Could we re-use your work here to accelerate streaming models like nemotron-speech-asr?

[pzelasko]

what's magical about 3?

[pzelasko]

Finalized my first review pass :)

[pzelasko]

Is it possible to support different top_p/temperature/repetition_penalty in different examples in a batch without using a for loop over them?

Is there a strong motivation to support that? Or could we expect the user to define session-level decoding parameters and have easier batching?

[pzelasko]

Need a docstring with overview of what's this doing

[pzelasko]

How is a "slot" defined here?

[pzelasko]

with (torch.device('meta') if use_meta_device else nullcontext()):

[pzelasko]

Is this compatible with transformers v5 or v4? Or both? We already bumped NeMo to v5

[pzelasko]

Since so much logic is dedicated to Nemotron v2 in VoiceChat, shouldn't we test against Nemotron v2 as well? Or are you concerned it will take very long time to load in CI?

[pzelasko]

These fixtures are only really shared between nemotron voicechat tests, not all speechlm2 tests, so how about creating a new tests/collections/speechlm2/voicechat directory and moving that conftest + related tests there?