Record outbound phone calls + 2-channel socket #4416

mdmohsin7 · 2026-01-27T18:39:34Z

External.Device-trimmed.mp4

gemini-code-assist

Code Review

This pull request introduces a new multi-channel audio transcription WebSocket endpoint, along with supporting utilities for conversation management, Pusher integration, and translation. The changes include creating new files for streaming utilities and modifying the backend router to handle the new endpoint. A critical issue related to error handling and fallback strategy has been identified and needs to be addressed.

backend/routers/multichannel.py

mdmohsin7 · 2026-01-27T19:05:45Z

/gemini summary

gemini-code-assist · 2026-01-27T19:08:19Z

Summary of Changes

This pull request adds phone call functionality to the application, leveraging the Twilio Voice SDK for call management and real-time transcription. It incorporates native platform features like CallKit on iOS for a seamless user experience and introduces a new multi-channel audio processing pipeline. The changes include backend endpoints for token generation and phone number verification, as well as Flutter plugins for handling call initiation, control, and audio streaming.

Highlights

Twilio Voice SDK Integration: This PR introduces the Twilio Voice SDK to enable outbound phone call functionality within the app.
Real-time Transcription: The implementation includes real-time transcription of phone calls, providing users with live transcripts of their conversations.
CallKit Integration (iOS): For iOS, the PR integrates with CallKit to provide a native call experience, including handling incoming and outgoing calls through the system's call interface.
Multi-Channel Audio Support: The PR implements multi-channel audio support, allowing for the capture and processing of both the user's microphone input and the remote audio stream.
Phone Number Verification: The PR includes phone number verification to ensure that users can only make calls from verified numbers.

Changelog

app/android/app/build.gradle
- Added Twilio Voice SDK dependency.
app/android/app/src/main/kotlin/com/example/my_project/MainActivity.kt
- Registered the PhoneCallsPlugin with the Flutter engine.
app/android/app/src/main/kotlin/com/friend/ios/PhoneCallsPlugin.kt
- Added a new Flutter plugin for handling phone call functionality using the Twilio Voice SDK on Android.
- Implements method channel communication for call initialization, making calls, ending calls, muting, and toggling the speaker.
- Uses an event channel to stream call state changes to Flutter.
app/ios/Podfile
- Added TwilioVoice pod dependency.
app/ios/Podfile.lock
- Updated to include TwilioVoice and pasteboard dependencies, along with associated checksums and specifications.
app/ios/Runner/AppDelegate.swift
- Registered the PhoneCallsPlugin with the Flutter engine.
app/ios/Runner/Info.plist
- Added voip to the list of UIBackgroundModes.
app/ios/Runner/OmiRecordingAudioDevice.swift
- Added a custom audio device to capture both local and remote audio streams using Twilio Voice SDK and Core Audio.
- Implements audio unit setup, rendering, and capturing for real-time audio processing.
app/ios/Runner/PhoneCallsPlugin.swift
- Added a Flutter plugin for handling phone call functionality using the Twilio Voice SDK on iOS.
- Integrates with CallKit for native call management and user interface.
- Implements method channel communication for call initialization, making calls, ending calls, muting, and toggling the speaker.
- Uses an event channel to stream call state changes and audio data to Flutter.
app/lib/backend/http/api/phone_calls.dart
- Added API functions for phone number verification, checking verification status, retrieving verified phone numbers, deleting verified phone numbers, and generating Twilio access tokens.
app/lib/backend/schema/phone_call.dart
- Added data models for phone call-related data, including PhoneTranscriptSegment, VerifiedPhoneNumber, and PhoneCallToken.
app/lib/backend/schema/schema.dart
- Exported the new phone_call.dart schema.
app/lib/main.dart
- Added PhoneCallProvider to the list of ChangeNotifierProviders.
app/lib/pages/home/page.dart
- Added a floating action button to navigate to the PhoneCallsPage or PhoneSetupIntroPage based on the user's verification status.
app/lib/pages/phone_calls/active_call_page.dart
- Added a new page for displaying the active phone call UI, including call info, live transcript, and call controls.
app/lib/pages/phone_calls/phone_calls_page.dart
- Added a new page for initiating phone calls, with tabs for contacts and a dialpad.
app/lib/pages/phone_calls/phone_setup_intro_page.dart
- Added a new page to guide users through the phone number verification process.
app/lib/pages/phone_calls/phone_setup_number_page.dart
- Added a new page for entering and verifying the user's phone number.
app/lib/pages/phone_calls/phone_setup_verify_page.dart
- Added a new page for verifying the user's phone number using a validation code.
app/lib/pages/settings/phone_call_settings_page.dart
- Added a new page for managing verified phone numbers.
app/lib/pages/settings/settings_drawer.dart
- Added a navigation item for phone call settings.
app/lib/providers/phone_call_provider.dart
- Added a new provider to manage phone call state, Twilio SDK initialization, call control, and transcription.
app/lib/services/phone_call_service.dart
- Added a new service for handling native method channel calls related to phone call functionality.
backend/database/phone_calls.py
- Added database functions for managing verified phone numbers, including upsert, retrieval, and deletion.
backend/main.py
- Included the phone_calls router in the main FastAPI application.
backend/requirements.txt
- Added the twilio package as a dependency.
backend/routers/multichannel.py
- Added a new router for handling multi-channel audio streams, including phone calls and desktop audio.
- Implements WebSocket endpoints for receiving audio, processing transcriptions, and managing conversation state.
backend/utils/streaming/init.py
- Added streaming utility modules.
backend/utils/streaming/conversation_manager.py
- Added conversation management utilities.
backend/utils/streaming/pusher_handler.py
- Added Pusher handler utilities.
backend/utils/streaming/translator.py
- Added translation utilities.
backend/utils/streaming/usage_tracker.py
- Added usage tracking utilities.
backend/utils/twilio_service.py
- Added a new module for interacting with the Twilio API, including token generation, caller ID verification, and call management.

Activity

mdmohsin7 requested a Gemini summary.
gemini-code-assist[bot] suggested a more informative error message or a more intelligent fallback strategy for the Deepgram STT service in backend/routers/multichannel.py.

beastoin · 2026-01-28T04:12:47Z

@mdmohsin7 The TwiML webhook in phone_calls.py:200 lacks Twilio signature validation so anyone can POST and trigger calls, phone verification is account-level not user-bound in phone_calls.py:120 which lets users claim/delete each other's numbers, /verify/check has no rate limiting (1s polling hits Twilio API), multichannel WS accepts unbounded channels in multichannel.py:117 (DoS risk), TwiML To is not validated to E.164 in phone_calls.py:209, PII is logged in plaintext in phone_calls.py:214, and iOS OmiRecordingAudioDevice has a buffer alignment issue with potential leaks; plus there are no tests for the new endpoints or mobile plugins, though the separation of streaming utils (pusher_handler/usage_tracker/conversation_manager) is solid—can you address these and add coverage, and let me know when it's ready for re-review?

Reviewed by @kenji

mdmohsin7 · 2026-01-28T17:50:35Z

/gemini summary

gemini-code-assist · 2026-01-28T17:56:55Z

Summary of Changes

This pull request introduces phone call functionality to the application, leveraging the Twilio Voice SDK for call management and real-time transcription. It incorporates native platform features like CallKit on iOS for a seamless user experience and introduces a new multi-channel audio processing pipeline. The changes include backend endpoints for token generation and phone number verification, as well as Flutter plugins for handling call initiation, control, and audio streaming.

Highlights

Twilio Voice SDK Integration: This PR introduces the Twilio Voice SDK to enable outbound phone call functionality within the app.
Real-time Transcription: The implementation includes real-time transcription of phone calls, providing users with live transcripts of their conversations.
CallKit Integration (iOS): For iOS, the PR integrates with CallKit to provide a native call experience, including handling incoming and outgoing calls through the system's call interface.
Multi-Channel Audio Support: The PR implements multi-channel audio support, allowing for the capture and processing of both the user's microphone input and the remote audio stream.
Phone Number Verification: The PR includes phone number verification to ensure that users can only make calls from verified numbers.

Changelog

app/android/app/build.gradle
- Added Twilio Voice SDK dependency.
app/android/app/src/main/kotlin/com/example/my_project/MainActivity.kt
- Registered the PhoneCallsPlugin with the Flutter engine.
app/android/app/src/main/kotlin/com/friend/ios/PhoneCallsPlugin.kt
- Added a new Flutter plugin for handling phone call functionality using the Twilio Voice SDK on Android.
- Implements method channel communication for call initialization, making calls, ending calls, muting, and toggling the speaker.
- Uses an event channel to stream call state changes to Flutter.
app/ios/Podfile
- Added TwilioVoice pod dependency.
app/ios/Podfile.lock
- Updated to include TwilioVoice and pasteboard dependencies, along with associated checksums and specifications.
app/ios/Runner/AppDelegate.swift
- Registered the PhoneCallsPlugin with the Flutter engine.
app/ios/Runner/Info.plist
- Added voip to the list of UIBackgroundModes.
app/ios/Runner/OmiRecordingAudioDevice.swift
- Added a custom audio device to capture both local and remote audio streams using Twilio Voice SDK and Core Audio.
- Implements audio unit setup, rendering, and capturing for real-time audio processing.
app/ios/Runner/PhoneCallsPlugin.swift
- Added a Flutter plugin for handling phone call functionality using the Twilio Voice SDK on iOS.
- Integrates with CallKit for native call management and user interface.
- Implements method channel communication for call initialization, making calls, ending calls, muting, and toggling the speaker.
- Uses an event channel to stream call state changes and audio data to Flutter.
app/lib/backend/http/api/phone_calls.dart
- Added API functions for phone number verification, checking verification status, retrieving verified phone numbers, deleting verified phone numbers, and generating Twilio access tokens.
app/lib/backend/schema/phone_call.dart
- Added data models for phone call-related data, including PhoneTranscriptSegment, VerifiedPhoneNumber, and PhoneCallToken.
app/lib/backend/schema/schema.dart
- Exported the new phone_call.dart schema.
app/lib/main.dart
- Added PhoneCallProvider to the list of ChangeNotifierProviders.
app/lib/pages/home/page.dart
- Added a floating action button to navigate to the PhoneCallsPage or PhoneSetupIntroPage based on the user's verification status.
app/lib/pages/phone_calls/active_call_page.dart
- Added a new page for displaying the active phone call UI, including call info, live transcript, and call controls.
app/lib/pages/phone_calls/phone_calls_page.dart
- Added a new page for initiating phone calls, with tabs for contacts and a dialpad.
app/lib/pages/phone_calls/phone_setup_intro_page.dart
- Added a new page to guide users through the phone number verification process.
app/lib/pages/phone_calls/phone_setup_number_page.dart
- Added a new page for entering and verifying the user's phone number.
app/lib/pages/phone_calls/phone_setup_verify_page.dart
- Added a new page for verifying the user's phone number using a validation code.
app/lib/pages/settings/phone_call_settings_page.dart
- Added a new page for managing verified phone numbers.
app/lib/pages/settings/settings_drawer.dart
- Added a navigation item for phone call settings.
app/lib/providers/phone_call_provider.dart
- Added a new provider to manage phone call state, Twilio SDK initialization, call control, and transcription.
app/lib/services/phone_call_service.dart
- Added a new service for handling native method channel calls related to phone call functionality.
backend/database/phone_calls.py
- Added database functions for managing verified phone numbers, including upsert, retrieval, and deletion.
backend/main.py
- Included the phone_calls router in the main FastAPI application.
backend/requirements.txt
- Added the twilio package as a dependency.
backend/routers/multichannel.py
- Added a new router for handling multi-channel audio streams, including phone calls and desktop audio.
- Implements WebSocket endpoints for receiving audio, processing transcriptions, and managing conversation state.
backend/utils/streaming/init.py
- Added streaming utility modules.
backend/utils/streaming/conversation_manager.py
- Added conversation management utilities.
backend/utils/streaming/pusher_handler.py
- Added Pusher handler utilities.
backend/utils/streaming/translator.py
- Added translation utilities.
backend/utils/streaming/usage_tracker.py
- Added usage tracking utilities.
backend/utils/twilio_service.py
- Added a new module for interacting with the Twilio API, including token generation, caller ID verification, and call management.

Activity

mdmohsin7 requested a Gemini summary.
gemini-code-assist[bot] suggested a more informative error message or a more intelligent fallback strategy for the Deepgram STT service in backend/routers/multichannel.py.
beastoin noted several potential issues, including lack of Twilio signature validation, account-level phone verification, missing rate limiting, unbounded channels, lack of E.164 validation, PII logging, and a buffer alignment issue, and requested that these be addressed and that tests be added.
mdmohsin7 requested a Gemini summary.

mdmohsin7 · 2026-01-28T18:06:20Z

@mdmohsin7 The TwiML webhook in phone_calls.py:200 lacks Twilio signature validation so anyone can POST and trigger calls, phone verification is account-level not user-bound in phone_calls.py:120 which lets users claim/delete each other's numbers, /verify/check has no rate limiting (1s polling hits Twilio API), multichannel WS accepts unbounded channels in multichannel.py:117 (DoS risk), TwiML To is not validated to E.164 in phone_calls.py:209, PII is logged in plaintext in phone_calls.py:214, and iOS OmiRecordingAudioDevice has a buffer alignment issue with potential leaks; plus there are no tests for the new endpoints or mobile plugins, though the separation of streaming utils (pusher_handler/usage_tracker/conversation_manager) is solid—can you address these and add coverage, and let me know when it's ready for re-review?

Reviewed by @kenji

Done, pls check now @beastoin

mdmohsin7 added 2 commits January 28, 2026 00:06

create utils from transcribe

1ba62d6

n-channel socket

a4a2c17

gemini-code-assist bot reviewed Jan 27, 2026

View reviewed changes

backend/routers/multichannel.py Outdated Show resolved Hide resolved

mdmohsin7 added 3 commits January 28, 2026 00:23

twilio setup

ed1c010

phone calls on app side (swift and kotlin)

e48a428

misc

30851ad

mdmohsin7 marked this pull request as ready for review January 27, 2026 19:56

mdmohsin7 requested a review from beastoin January 27, 2026 19:56

mdmohsin7 and others added 4 commits January 28, 2026 23:04

android + bug fixes

44baf39

wbhook validation + fix channel + other fixes

544ee72

Merge branch 'main' into phone-calls

9a3e2b3

add tests

fe54a5d

mdmohsin7 changed the title ~~Record outbound phone calls + n-channel socket~~ Record outbound phone calls + 2-channel socket Jan 28, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Record outbound phone calls + 2-channel socket #4416

Record outbound phone calls + 2-channel socket #4416

mdmohsin7 commented Jan 27, 2026 •

edited

Loading

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

Uh oh!

mdmohsin7 commented Jan 27, 2026

Uh oh!

gemini-code-assist bot commented Jan 27, 2026

Uh oh!

beastoin commented Jan 28, 2026

Uh oh!

mdmohsin7 commented Jan 28, 2026

Uh oh!

gemini-code-assist bot commented Jan 28, 2026

Uh oh!

mdmohsin7 commented Jan 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Record outbound phone calls + 2-channel socket #4416

Are you sure you want to change the base?

Record outbound phone calls + 2-channel socket #4416

Conversation

mdmohsin7 commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

mdmohsin7 commented Jan 27, 2026

Uh oh!

gemini-code-assist bot commented Jan 27, 2026

Summary of Changes

Highlights

Uh oh!

beastoin commented Jan 28, 2026

Uh oh!

mdmohsin7 commented Jan 28, 2026

Uh oh!

gemini-code-assist bot commented Jan 28, 2026

Summary of Changes

Highlights

Uh oh!

mdmohsin7 commented Jan 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mdmohsin7 commented Jan 27, 2026 •

edited

Loading