Add runtime audio processing modes for local tracks#247
Open
hiroshihorie wants to merge 52 commits into
Open
Conversation
Member
Author
There was a problem hiding this comment.
Pull request overview
This PR adds runtime, track-level audio processing controls (AEC/NS/AGC/HPF) and introduces a per-component processing mode (automatic / platform / software) that is resolved when applying options in the voice engine. The change propagates updates via existing track/sender observer plumbing so options can be updated without restarting capture.
Changes:
- Adds new audio-processing mode fields to
AudioOptions, plus parsing support via legacyMediaConstraints. - Adds runtime
SetAudioProcessingOptionsAPIs for local audio tracks (ObjC + Android + C++), and propagates updates throughAudioRtpSender. - Centralizes platform-vs-software decision logic in a new
audio_processing_controller, with new/expanded unit test coverage.
Reviewed changes
Copilot reviewed 27 out of 27 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| sdk/objc/api/peerconnection/RTCAudioTrack.h | Adds ObjC API types for audio processing options and modes, plus track method to apply them. |
| sdk/objc/api/peerconnection/RTCAudioTrack.mm | Implements ObjC options object and applies options to the native track on the signaling thread. |
| sdk/media_constraints.h | Adds new constraint keys for per-component processing modes. |
| sdk/media_constraints.cc | Parses mode strings and copies them into AudioOptions. |
| sdk/android/src/jni/pc/rtp_sender.cc | Removes an unused include. |
| sdk/android/src/jni/pc/audio_track.cc | Adds JNI entrypoint for setting audio processing options on a native local track. |
| sdk/android/src/java/org/webrtc/audio/WebRtcAudioEffects.java | Enables toggling Android platform AEC/NS while recording via new toggle helpers. |
| sdk/android/api/org/webrtc/AudioTrack.java | Adds Java-facing audio processing options/modes and forwards them to JNI. |
| pc/rtp_sender.h | Adds caching for track source audio options so senders can reapply updated processing config. |
| pc/rtp_sender.cc | Reapplies audio options on track change notifications without restarting capture. |
| pc/rtp_sender_receiver_unittest.cc | Adds a unit test validating propagation/reapplication of updated audio processing options. |
| pc/media_stream_track_proxy.h | Proxies the new SetAudioProcessingOptions API. |
| pc/local_audio_source.h | Adds SetOptions() so the local source can store updated AudioOptions. |
| pc/audio_track.h | Declares track-level SetAudioProcessingOptions() override. |
| pc/audio_track.cc | Implements runtime option updates on the local source and notifies observers. |
| modules/audio_device/audio_engine_device.mm | Adds coupled AEC/NS topology reporting and makes built-in effect enable/disable stateful. |
| modules/audio_device/audio_engine_device.h | Extends engine state to track desired built-in AEC/NS state and topology support. |
| media/engine/webrtc_voice_engine.cc | Switches to centralized application via ApplyAudioProcessingOptions(). |
| media/engine/webrtc_voice_engine_unittest.cc | Adds unit tests for platform/software resolution behavior, including coupled AEC/NS topology. |
| media/engine/audio_processing_controller.h | New helper API for resolving/applying platform vs software audio processing. |
| media/engine/audio_processing_controller.cc | Implements resolution logic and applies APM config based on effective platform state. |
| media/BUILD.gn | Adds the new controller sources to the media build target. |
| api/media_stream_interface.h | Adds AudioSourceInterface::SetOptions() and AudioTrackInterface::SetAudioProcessingOptions(). |
| api/media_stream_interface.cc | Provides default no-op implementation for the new interface method. |
| api/audio/audio_device.h | Adds built-in audio processing topology concept to ADM API. |
| api/audio_options.h | Adds AudioProcessingMode and per-component mode fields to AudioOptions. |
| api/audio_options.cc | Updates AudioOptions equality/serialization and SetAll() to include mode fields. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+72
to
+78
| const bool platform_enabled = | ||
| EnablePlatformEffect(adm, is_available, enable); | ||
| if (!platform_enabled) { | ||
| LogPlatformOnlyRequestDisabled( | ||
| component, "platform processing could not be enabled"); | ||
| } | ||
| return false; |
…own crash) WebRtcVoiceEngine::Terminate calls RegisterAudioCallback(nullptr) to deregister, but a RTC_DCHECK(audioCallback != nullptr) crashed debug builds on disconnect. AudioDeviceBuffer::RegisterAudioCallback already accepts nullptr; drop the DCHECK.
cloudwebrtc
approved these changes
Jun 4, 2026
Member
Author
Member
Author
Member
Author
ResolveCoupledAudioProcessingPath took the AEC/NS platform-path active state as a bool, so the controller invoked CoupledEchoNoisePlatformPathIsActive() — an ADM state read — on every apply even when AEC or NS was present and the result was discarded. Take a FunctionView<bool()> instead and call it only in the AGC-only branch, where the resolver actually consults it. The iOS seed path and resolver unit tests pass equivalent thunks.
CoupledAudioProcessingPathResolution carried both has_echo_or_noise_option and should_update_echo_noise_platform_path, but the resolver set them to the same value unconditionally. Drop should_update_echo_noise_platform_path and have callers gate the platform-path update on has_echo_or_noise_option directly. No behavior change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds runtime audio-processing controls for local audio tracks.
The API keeps the existing per-component enabled flags and adds an optional per-component processing mode:
automatic: use platform processing when available, otherwise use WebRTC software APM.platform: use platform processing only. If platform processing is unavailable or cannot be enabled, software fallback is intentionally disabled.software: disable platform processing for that component and use WebRTC software APM.Covered components:
Propagation
Runtime updates are stored on the local audio source and propagated through the existing sender path:
AudioTrack::SetAudioProcessingOptions->LocalAudioSource::SetOptions->FireOnChanged->AudioRtpSender::OnChanged->SetSend->WebRtcVoiceEngine::ApplyOptionsThis avoids restarting capture. Options set before a sender exists are stored and applied when the track starts sending. If the track is disabled, options are stored and applied when the track is enabled again.
Platform Behavior
Adds a shared
audio_processing_controllerthat resolves platform vs software processing in one place.Android uses the independent topology:
AudioEffects.automaticfalls back to software AGC andplatformresolves disabled.Apple
AudioEngineDevicereports a coupled AEC/NS topology:automaticorplatformmode can activate the shared platform path for both AEC and NS.voiceProcessingAGCEnabled, but only has effect while the shared AEC/NS VPIO path is active.automaticfalls back to software andplatformresolves disabled.softwaremode does not callsetVoiceProcessingEnabled(false). It only bypasses platform effects and enables WebRTC APM.HPF has no platform implementation today, so
highPassFilter=true + platformresolves disabled with a warning.Compatibility
automatic.bypassVoiceProcessingis preserved as a deprecated coarse preset.voiceProcessingBypassedandSetVoiceProcessingBypassedremain low-level Apple ADM toggles.SetVoiceProcessingEnabled(false)make platform AEC/NS/AGC unavailable. In that state,automaticresolves to software andplatformresolves disabled.Validation
Added C++ coverage for:
automaticplatform selection and software fallback.platformwithout fallback.softwareforcing platform processing off.AudioRtpSender.Local compile validation:
audio_processing_controller.obuilds.webrtc_voice_engine_unittest.obuilds.rtc_media_unittestsbuild was not completed in this local checkout because the current Linux build environment is missingpipewire/pipewire.h.Android device smoke test:
AcousticEchoCanceler.setEnabled()successfully toggles whileAudioRecordis recording.NoiseSuppressor.setEnabled()successfully toggles whileAudioRecordis recording.AutomaticGainControl.isAvailable()is false on that device, so platform AGC was not testable there.Notes
This PR intentionally does not reject
platformrequests when platform processing is unavailable, because track options can be set before the final ADM/topology is known. The effective result is resolved when the voice engine applies options. Warnings are logged for platform-only requests that cannot be honored.