From 085b8f4a747d8847fc25f7da064db761e210cbf1 Mon Sep 17 00:00:00 2001 From: Zoltan Ortutay Date: Fri, 27 Feb 2026 09:22:48 +0100 Subject: [PATCH] update-in-app-calling --- skills/in-app-calling/SKILL.md | 361 +---------- skills/in-app-calling/assets/SinchJWT.swift | 156 +++++ .../jwt-helper-andorid/FakeJWTFetcher.kt | 16 + .../assets/jwt-helper-andorid/Hmac.java | 36 ++ .../assets/jwt-helper-andorid/JWT.java | 73 +++ .../assets/jwt-helper-andorid/JWTFetcher.kt | 16 + skills/in-app-calling/assets/jwt-helper.js | 140 ++++ skills/in-app-calling/references/android.md | 494 ++++++++++++++ skills/in-app-calling/references/ios.md | 611 ++++++++++++++++++ skills/in-app-calling/references/js.md | 388 +++++++++++ 10 files changed, 1965 insertions(+), 326 deletions(-) create mode 100644 skills/in-app-calling/assets/SinchJWT.swift create mode 100644 skills/in-app-calling/assets/jwt-helper-andorid/FakeJWTFetcher.kt create mode 100644 skills/in-app-calling/assets/jwt-helper-andorid/Hmac.java create mode 100644 skills/in-app-calling/assets/jwt-helper-andorid/JWT.java create mode 100644 skills/in-app-calling/assets/jwt-helper-andorid/JWTFetcher.kt create mode 100644 skills/in-app-calling/assets/jwt-helper.js create mode 100644 skills/in-app-calling/references/android.md create mode 100644 skills/in-app-calling/references/ios.md create mode 100644 skills/in-app-calling/references/js.md diff --git a/skills/in-app-calling/SKILL.md b/skills/in-app-calling/SKILL.md index 0c20da8..fa5ce81 100644 --- a/skills/in-app-calling/SKILL.md +++ b/skills/in-app-calling/SKILL.md @@ -1,344 +1,53 @@ --- name: sinch-in-app-calling -description: Build in-app voice and video calling with Sinch RTC SDKs. Use for app-to-app, app-to-phone, video calls, and push notifications on iOS, Android, and Web. +description: Helps users integrate the Sinch In-App Voice SDK into their app. Use whenever a user mentions In-App Voice SDK, or asks about integrating Sinch In-App Voice SDK into an Android, iOS, or JavaScript web application project, or asks to create a WebRTC application using Sinch. --- # Sinch In-App Calling -## Overview +## What is the Sinch In-app Calling SDK? +The Sinch In-app Calling SDK enables real-time voice and video communication inside mobile and web apps. It connects to Sinch's cloud backend for signaling and routing. +Available for **Android**, **iOS**, and **JavaScript (Web)**. -The Sinch In-App Calling SDK enables real-time voice and video communication directly within your mobile or web application. Built on WebRTC, it supports app-to-app audio/video calls, app-to-phone (PSTN) calls, app-to-SIP calls, and conference calling across iOS, Android, and JavaScript platforms. +### Supported call types +- App-to-App (VoIP/WebRTC between users in the same app) +- App-to-Phone (call PSTN landlines/mobiles from the app) +- App-to-SIP (connect to PBXs, contact centers, AI agents) +- App-to-Conference (multi-party calls across channels) +- Phone-to-App and SIP-to-App (inbound calls into the app) -Key capabilities: -- **App-to-app voice and video calling** between users of your application -- **App-to-phone (PSTN)** calling from your app to any phone number -- **App-to-SIP** calling to SIP endpoints -- **Conference calling** with multiple participants -- **Push notifications** for incoming calls when the app is in the background -- **Screen sharing** on supported platforms +### Getting started requires +1. A Sinch account with an application key and secret from the Sinch Build Dashboard. +2. The SDK for the target platform. -See the [Sinch pricing page](https://sinch.com/pricing/) for current in-app calling rates. +## Platform Detection +Determine the user's platform from their project files, language, or question: +- Android (Kotlin/Java, Gradle, Android Studio) → Read `references/android.md` +- iOS (Swift/ObjC, Xcode, CocoaPods/SPM) → Read `references/ios.md` +- JavaScript/Web (npm, package.json, React, etc.) → Read `references/js.md` -## Getting Started +If the platform is unclear, ask the user which platform they're targeting. -### Authentication +## Public endpoints +The In-App Calling API uses different endpoints depending on your region. +When creating a Sinch client, choose the regional endpoint used to communicate with the Sinch platform by setting the `environmentHost` parameter. The following regional endpoints are available: -See the [sinch-authentication](../authentication/SKILL.md) skill for full auth setup, SDK initialization, and dashboard links. +| Endpoint (hostname) | Description | +| --- | --- | +| `https://ocra.api.sinch.com` | Global - redirected by Sinch to the closest region | +| `https://ocra-euc1.api.sinch.com` | Europe | +| `https://ocra-use1.api.sinch.com` | North America | +| `https://ocra-sae1.api.sinch.com` | South America | +| `https://ocra-apse1.api.sinch.com` | South East Asia 1 | +| `https://ocra-apse2.api.sinch.com` | South East Asia 2 | -In-App Calling uses **Application Key + Application Secret** from the Sinch Dashboard. Authentication requires a **JWT (JSON Web Token)** signed with a key derived from your Application Secret. - -**JWT signing key derivation:** -``` -signingKey = HMAC256(BASE64_DECODE(applicationSecret), UTF8_ENCODE(YYYYMMDD)) -``` - -The JWT must include: -- `iss` (issuer): Your Application Key -- `sub` (subject): The user ID -- `iat` (issued at): Current timestamp -- `exp` (expiration): Token expiry (TTL must be at least 1 minute) -- `nonce`: A unique nonce value - -**IMPORTANT:** In production, generate JWTs on your backend server. Never embed the Application Secret in client-side code. - -### Platform SDKs - -| Platform | SDK | Distribution | -|----------|-----|-------------| -| iOS | Sinch RTC SDK | CocoaPods / manual framework | -| Android | Sinch RTC SDK | Maven / manual AAR | -| JavaScript (Web) | Sinch RTC JS SDK | npm / CDN | - -Server-side SDKs for managing Voice resources: - -| Language | Package | Install | -|----------|---------|---------| -| Node.js | `@sinch/sdk-core` | `npm install @sinch/sdk-core` | -| Java | `com.sinch.sdk:sinch-sdk-java` | Maven dependency | -| Python | `sinch` | `pip install sinch` | -| .NET | `Sinch` | `dotnet add package Sinch` | - -### First Integration: JavaScript (Web) Voice Call - -**Step 1: Initialize the SinchClient** - -```javascript -const sinchClient = Sinch.getSinchClientBuilder() - .applicationKey('YOUR_APPLICATION_KEY') - .environmentHost('ocra.api.sinch.com') - .userId('alice') - .build(); -``` - -**Step 2: Add lifecycle and credential listeners** - -```javascript -const sinchClientListener = { - onCredentialsRequired: (sinchClient, clientRegistration) => { - // In production: fetch JWT from your backend - fetch('/api/sinch-token?userId=' + sinchClient.userId) - .then(res => res.json()) - .then(data => clientRegistration.register(data.token)) - .catch(() => clientRegistration.registerFailed()); - }, - onClientStarted: (sinchClient) => { - console.log('Sinch client started successfully'); - }, - onClientFailed: (sinchClient, error) => { - console.error('Sinch client failed to start:', error); - }, -}; - -sinchClient.addListener(sinchClientListener); -sinchClient.start(); -``` - -**Step 3: Make a voice call** - -```javascript -// Get the call client -const callClient = sinchClient.callClient; - -// Add call listener -callClient.addListener({ - onIncomingCall: (callClient, call) => { - console.log('Incoming call from:', call.remoteUserId); - // call.answer() to accept - // call.hangup() to reject - }, -}); - -// Make an app-to-app call -const call = callClient.callUser('bob'); - -call.addListener({ - onCallProgressing: (call) => { - console.log('Ringing...'); - }, - onCallEstablished: (call) => { - console.log('Call connected'); - }, - onCallEnded: (call) => { - console.log('Call ended'); - }, -}); -``` - -**Step 4: Make a video call** - -```javascript -const call = callClient.callUserVideo('bob'); - -call.addListener({ - onCallEstablished: (call) => { - // Attach local and remote video streams - document.getElementById('localVideo').srcObject = call.localStream; - document.getElementById('remoteVideo').srcObject = call.remoteStream; - }, -}); -``` - -### Backend: Generate JWT Token (Node.js) - -Your backend must generate JWT tokens for client authentication: - -```javascript -const crypto = require('crypto'); -const jwt = require('jsonwebtoken'); - -function generateSinchJWT(userId, applicationKey, applicationSecret) { - const now = new Date(); - const dateStr = now.toISOString().slice(0, 10).replace(/-/g, ''); - - // Derive signing key: HMAC-SHA256(base64decode(secret), dateString) - const decodedSecret = Buffer.from(applicationSecret, 'base64'); - const signingKey = crypto - .createHmac('sha256', decodedSecret) - .update(dateStr) - .digest(); - - const token = jwt.sign( - { - iss: applicationKey, - sub: userId, - iat: Math.floor(Date.now() / 1000), - exp: Math.floor(Date.now() / 1000) + 600, // 10 minute expiry - nonce: crypto.randomBytes(16).toString('hex'), - }, - signingKey, - { algorithm: 'HS256' } - ); - - return token; -} -``` - -### curl: Place an Outbound Call via REST API - -You can also initiate calls server-side using the Voice REST API: - -```bash -curl -X POST "https://calling.api.sinch.com/calling/v1/callouts" \ - -H "Content-Type: application/json" \ - -u "YOUR_APPLICATION_KEY:YOUR_APPLICATION_SECRET" \ - -d '{ - "method": "customCallout", - "customCallout": { - "destination": { - "type": "username", - "endpoint": "bob" - }, - "cli": "alice", - "ice": "{\"action\":{\"name\":\"connectMxp\",\"destination\":{\"type\":\"username\",\"endpoint\":\"bob\"}}}" - } - }' -``` - -## Key Concepts - -### Call Types - -| Type | Description | Method | -|------|-------------|--------| -| App-to-App (Audio) | Voice call between two app users | `callClient.callUser(userId)` | -| App-to-App (Video) | Video call between two app users | `callClient.callUserVideo(userId)` | -| App-to-Phone (PSTN) | Call from app to a phone number | `callClient.callPhoneNumber(number)` | -| App-to-SIP | Call from app to a SIP endpoint | `callClient.callSip(sipAddress)` | -| Conference | Multi-party call via conference ID | `callClient.callConference(conferenceId)` | - -### SinchClient Lifecycle - -1. **Build**: Create client with `SinchClientBuilder`, providing applicationKey, environmentHost, and userId. -2. **Register listener**: Add `SinchClientListener` with `onCredentialsRequired`, `onClientStarted`, and `onClientFailed`. -3. **Start**: Call `sinchClient.start()`. The SDK triggers `onCredentialsRequired` to obtain a JWT. -4. **Use**: Access `callClient` for making/receiving calls. -5. **Stop**: Call `sinchClient.terminate()` when done. - -### Push Notifications - -Push notifications enable incoming call alerts when the app is backgrounded or closed. - -| Platform | Push Service | Configuration | -|----------|-------------|---------------| -| iOS | Apple Push Notification service (APNs) | Upload APNs certificate/key in Sinch Dashboard | -| Android | Firebase Cloud Messaging (FCM) | Add FCM server key in Sinch Dashboard | -| Web | Not applicable | Use `setSupportManagedPush()` for browser notifications | - -Enable managed push in the client builder: -```javascript -const sinchClient = Sinch.getSinchClientBuilder() - .applicationKey('YOUR_APPLICATION_KEY') - .environmentHost('ocra.api.sinch.com') - .userId('alice') - .setSupportManagedPush() - .build(); -``` - -### Video Codecs - -Sinch supports standard WebRTC video codecs: -- **VP8** (default, widely supported) -- **VP9** (better compression) -- **H.264** (hardware acceleration on mobile) - -### Environment Hosts - -| Environment | Host | -|-------------|------| -| Production | `ocra.api.sinch.com` | - -## Common Patterns - -### Incoming Call Handling with UI - -```javascript -callClient.addListener({ - onIncomingCall: (callClient, call) => { - // Show incoming call UI - showIncomingCallUI(call.remoteUserId); - - call.addListener({ - onCallEstablished: (call) => { - showActiveCallUI(); - }, - onCallEnded: (call) => { - hideCallUI(); - }, - }); - - // User actions: - document.getElementById('answerBtn').onclick = () => call.answer(); - document.getElementById('declineBtn').onclick = () => call.hangup(); - }, -}); -``` - -### Mute/Unmute and Speaker Toggle - -```javascript -// Mute microphone -call.mute(); - -// Unmute microphone -call.unmute(); - -// Toggle video (pause/resume) -call.pauseVideo(); -call.resumeVideo(); -``` - -### Video Call with Stream Management - -```javascript -const call = callClient.callUserVideo('bob'); - -call.addListener({ - onCallEstablished: (call) => { - const localVideo = document.getElementById('localVideo'); - const remoteVideo = document.getElementById('remoteVideo'); - localVideo.srcObject = call.localStream; - remoteVideo.srcObject = call.remoteStream; - }, - onCallEnded: (call) => { - document.getElementById('localVideo').srcObject = null; - document.getElementById('remoteVideo').srcObject = null; - }, -}); -``` - -## Gotchas and Best Practices - -1. **Never embed Application Secret in client code.** Always generate JWT tokens on your backend. The client SDK calls `onCredentialsRequired` where you should fetch a token from your server. - -2. **Push notification certificate management is critical.** On iOS, you must upload a valid APNs certificate (or key) to the Sinch Dashboard. Expired certificates cause silent failures. On Android, ensure the FCM server key is current. - -3. **NAT traversal is handled by Sinch.** The SDK uses STUN/TURN servers managed by Sinch. You do not need to configure ICE servers, but ensure your network allows UDP traffic on ports used by WebRTC. - -4. **Token TTL must be at least 1 minute.** The Sinch SDK rejects tokens with very short expiration times. Set `exp` to at least 60 seconds from `iat`. - -5. **The SDK automatically re-requests credentials.** When the registration TTL nears expiry, `onCredentialsRequired` is called again. Your backend must be able to issue fresh tokens on demand. - -6. **SinchClient is a singleton per user session.** Create one instance and retain it for the entire app lifecycle. Do not create multiple instances. - -7. **Call `terminate()` on cleanup.** Failing to terminate the SinchClient can cause resource leaks and unexpected behavior on subsequent starts. - -8. **Browser compatibility for Web SDK.** WebRTC support is required. Chrome, Firefox, Safari, and Edge are supported. Ensure HTTPS -- WebRTC requires a secure context. - -9. **Video bandwidth adapts automatically.** The SDK adjusts video quality based on available bandwidth. You do not need to configure bitrate settings manually. - -10. **PSTN calls require credit.** App-to-phone calls consume your Sinch account balance. App-to-app calls cost $0.003/min/leg. Ensure your account is funded for production use. ## Links - [In-App Calling Overview](https://developers.sinch.com/docs/in-app-calling.md) -- [JavaScript SDK Guide](https://developers.sinch.com/docs/in-app-calling/js-cloud.md) -- [JavaScript Voice Calling](https://developers.sinch.com/docs/in-app-calling/js/calling.md) -- [Authentication & Authorization (JS)](https://developers.sinch.com/docs/in-app-calling/js/application-authentication.md) -- [Getting Started: Create App](https://developers.sinch.com/docs/in-app-calling/getting-started/javascript/create-app.md) -- [Getting Started: Make a Call](https://developers.sinch.com/docs/in-app-calling/getting-started/javascript/make-call.md) -- [SDK Downloads](https://developers.sinch.com/docs/in-app-calling/sdk-downloads.md) +- [SDK Downloads](https://developers.sinch.com/docs/in-app-calling/sdk-downloads.md) - [Reference Applications (GitHub)](https://github.com/sinch/rtc-reference-applications) -- [SinchClient JS API Reference](https://download.sinch.com/docs/javascript/latest/reference/classes/SinchClient.html) -- [In-App Calling API Reference](https://developers.sinch.com/docs/in-app-calling.md) -- [npm: @sinch/sdk-core](https://www.npmjs.com/package/@sinch/sdk-core) -- [LLMs.txt (full docs index)](https://developers.sinch.com/llms.txt) +- [Android SDK Online reference docs](https://download.sinch.com/android/latest/reference/index.html) +- [iOS SDK Online reference docs](https://download.sinch.com/ios/latest/reference/index.html) +- [JavaScript SDK Online reference docs](https://download.sinch.com/js/latest/reference/index.html) + diff --git a/skills/in-app-calling/assets/SinchJWT.swift b/skills/in-app-calling/assets/SinchJWT.swift new file mode 100644 index 0000000..28f1c32 --- /dev/null +++ b/skills/in-app-calling/assets/SinchJWT.swift @@ -0,0 +1,156 @@ +import CommonCrypto +import Foundation + +/** + * Helper class showing how to create a signed JWT (JSON Web Token) + * for User registration (@see SinchClientRegistration). + * + * IMPORTANT: This implementation here in the sample app is only here + * to quickly get you started with the samples apps and this + * implementation is NOT meant to be used in production. When + * deploying your application to production, the Application Secret + * should be kept secure on your backend and not be embedded in the + * your apps. + */ +enum JWTError: Error { + + case base64CreateFailed + case stringFromBase64Failed +} + +enum SinchJWT { + + static func sinchJWTForUserRegistration(withApplicationKey key: String, + applicationSecret secret: String, + userId: String) throws -> String { + let now = Date() + + return try sinchJWTForUserRegistration(withApplicationKey: key, + applicationSecret: secret, + userId: userId, + nonce: UUID().uuidString, + issuedAt: now, + expireAt: now.addingTimeInterval(600)) + } + + static func sinchJWTForUserRegistration(withApplicationKey key: String, + applicationSecret secret: String, + userId: String, + nonce: String, + issuedAt: Date, + expireAt: Date) throws -> String { + return try jwtWith(withApplicationKey: key, + applicationSecret: secret, + userId: userId, + nonce: nonce, + issuedAt: issuedAt, + expireAt: expireAt, + instanceExpireAt: nil) + } + + static func sinchJWTForUserRegistration(withApplicationKey key: String, + applicationSecret secret: String, + userId: String, + nonce: String, + issuedAt: Date, + expireAt: Date, + instanceExpireAt: Date) throws -> String { + return try jwtWith(withApplicationKey: key, + applicationSecret: secret, + userId: userId, + nonce: nonce, + issuedAt: issuedAt, + expireAt: expireAt, + instanceExpireAt: instanceExpireAt) + } + + private static func jwtWith(withApplicationKey key: String, + applicationSecret secret: String, + userId: String, + nonce: String, + issuedAt: Date, + expireAt: Date, + instanceExpireAt: Date?) throws -> String { + let header = ["alg": "HS256", + "typ": "JWT", + "kid": "hkdfv1-" + JWTFormatDate(issuedAt)] + + var payload: [String: Any] = ["iss": "//rtc.sinch.com/applications/" + key, + "sub": "//rtc.sinch.com/applications/" + key + "/users/" + userId, + "iat": NSNumber(value: Int(issuedAt.timeIntervalSince1970)), + "exp": NSNumber(value: Int(expireAt.timeIntervalSince1970)), + "nonce": nonce] + + if let expTime = instanceExpireAt { + payload["sinch:rtc:instance:exp"] = NSNumber(value: Int(expTime.timeIntervalSince1970)) + } + + let signingKey = try JWTDeriveSigningKey(secret, issuedAt) + return try makeJWT(withHeaders: header, payload: payload, signingKey: signingKey) + } + + // MARK: - Implementation + + private static func JWTFormatDate(_ date: Date) -> String { + let formatter = DateFormatter() + formatter.locale = Locale(identifier: "en_US_POSIX") + formatter.dateFormat = "yyyyMMdd" + formatter.timeZone = TimeZone(secondsFromGMT: 0) + return formatter.string(from: date) + } + + private static func JWTDeriveSigningKey(_ secret: String, _ issuedAt: Date) throws -> Data { + let data = Data(base64Encoded: secret) + + guard data != nil else { + throw JWTError.base64CreateFailed + } + + return makeHMAC_SHA256(data!, JWTFormatDate(issuedAt)) + } + + private static func makeJWT(withHeaders headers: [String: Any], + payload: [String: Any], + signingKey key: Data) throws -> String { + let arr = [try JWTBase64Encode(data: JSONSerialize(of: headers)), + try JWTBase64Encode(data: JSONSerialize(of: payload))] + let headerDotPayload = arr.joined(separator: ".") + let signature = try JWTBase64Encode(data: makeHMAC_SHA256(key, headerDotPayload)) + + return [headerDotPayload, signature].joined(separator: ".") + } + + private static func JWTBase64Encode(data: Data) throws -> String { + // JWT RFC mandates URL-safe base64-encoding without padding. + let base64 = String(data: data.base64EncodedData(), encoding: .utf8) + guard base64 != nil else { + throw JWTError.stringFromBase64Failed + } + return base64!.replacingOccurrences(of: "+", with: "-") + .replacingOccurrences(of: "/", with: "_") + .replacingOccurrences(of: "=", with: "") + } + + private static func JSONSerialize(of payload: [String: Any]) throws -> Data { + let json = try JSONSerialization.data(withJSONObject: payload, options: JSONSerialization.WritingOptions(rawValue: 0)) + let jsonStr = String(data: json, encoding: .utf8) + return (jsonStr?.replacingOccurrences(of: "\\/", with: "/").data(using: .utf8))! + } + + private static func makeHMAC_SHA256(_ key: Data, _ message: String) -> Data { + return makeInternalHMAC_SHA256(key, message.data(using: .utf8)!) + } + + private static func makeInternalHMAC_SHA256(_ key: Data, _ message: Data) -> Data { + let out = UnsafeMutablePointer.allocate(capacity: Int(CC_SHA256_DIGEST_LENGTH)) + defer { out.deallocate() } + + key.withUnsafeBytes { (rbpKey: UnsafeRawBufferPointer) in + message.withUnsafeBytes { (rpbMsg: UnsafeRawBufferPointer) in + CCHmac(CCHmacAlgorithm(kCCHmacAlgSHA256), rbpKey.baseAddress, rbpKey.count, rpbMsg.baseAddress, rpbMsg.count, out) + } + } + + return Data(bytes: out, count: Int(CC_SHA256_DIGEST_LENGTH)) + } +} diff --git a/skills/in-app-calling/assets/jwt-helper-andorid/FakeJWTFetcher.kt b/skills/in-app-calling/assets/jwt-helper-andorid/FakeJWTFetcher.kt new file mode 100644 index 0000000..939c6f1 --- /dev/null +++ b/skills/in-app-calling/assets/jwt-helper-andorid/FakeJWTFetcher.kt @@ -0,0 +1,16 @@ +package com.sinch.rtc.vvc.reference.app.utils.jwt + +import com.sinch.rtc.vvc.reference.app.storage.prefs.SharedPrefsManager + +/** + * DO NOT use this fetcher in your production application, instead implement here an async callback to your backend. + * It might be tempting to re-use this class and store the APPLICATION_SECRET in your app, but that would + * greatly compromise security. + */ +class FakeJWTFetcher(private val prefsManager: SharedPrefsManager) : JWTFetcher { + + override fun acquireJWT(applicationKey: String, userId: String, callback: (String) -> Unit) { + callback(JWT.create(applicationKey, prefsManager.usedConfig.appSecret, userId)) + } + +} \ No newline at end of file diff --git a/skills/in-app-calling/assets/jwt-helper-andorid/Hmac.java b/skills/in-app-calling/assets/jwt-helper-andorid/Hmac.java new file mode 100644 index 0000000..d809afa --- /dev/null +++ b/skills/in-app-calling/assets/jwt-helper-andorid/Hmac.java @@ -0,0 +1,36 @@ +package com.sinch.rtc.vvc.reference.app.utils.jwt; + +import java.io.UnsupportedEncodingException; +import java.security.InvalidKeyException; +import java.security.NoSuchAlgorithmException; + +import javax.crypto.Mac; +import javax.crypto.spec.SecretKeySpec; + +final class Hmac { + + private Hmac() { + } + + public static byte[] hmacSha256(byte[] key, String message) { + if (null == key || key.length == 0) + throw new IllegalArgumentException("Invaid input key to HMAC-256"); + + if (null == message) + throw new IllegalArgumentException("Input message to HMAC-256 must not be null"); + + try { + Mac mac = Mac.getInstance("HmacSHA256"); + SecretKeySpec keySpec = new SecretKeySpec(key, "HmacSHA256"); + mac.init(keySpec); + return mac.doFinal(message.getBytes("UTF-8")); + } catch (NoSuchAlgorithmException e) { + throw new RuntimeException(e); + } catch (InvalidKeyException e) { + throw new RuntimeException(e); + } catch (UnsupportedEncodingException e) { + throw new RuntimeException(e); + } + } +} + diff --git a/skills/in-app-calling/assets/jwt-helper-andorid/JWT.java b/skills/in-app-calling/assets/jwt-helper-andorid/JWT.java new file mode 100644 index 0000000..5c38374 --- /dev/null +++ b/skills/in-app-calling/assets/jwt-helper-andorid/JWT.java @@ -0,0 +1,73 @@ +package com.sinch.rtc.vvc.reference.app.utils.jwt; + +import android.util.Base64; + +import org.json.JSONException; +import org.json.JSONObject; + +import java.text.SimpleDateFormat; +import java.util.Date; +import java.util.Locale; +import java.util.TimeZone; +import java.util.UUID; + +public class JWT { + + // ********************************************** IMPORTANT ********************************************** + // + // The JWT class serves as an example of how to produce and sign the registration token that you need to + // initiate SinchClient, or UserController. Read more in the documentation online. + // + // DO NOT use this class in your application, instead implement the same functionality on your backend. + // It might be tempting to re-use this class and store the APPLICATION_SECRET in your app, but that would + // greatly compromise security. + + public static String create(String appKey, String appSecret, String userId) { + JSONObject header = new JSONObject(); + JSONObject payload = new JSONObject(); + final long issuedAt = System.currentTimeMillis() / 1000; + String kid = "hkdfv1-" + formatDate(issuedAt); + try { + header.put("alg", "HS256"); + header.put("typ", "JWT"); + header.put("kid", kid); + payload.put("iss", "//rtc.sinch.com/applications/" + appKey); + payload.put("sub", "//rtc.sinch.com/applications/" + appKey + "/users/" + userId); + payload.put("iat", issuedAt); + payload.put("exp", issuedAt + 600); + payload.put("nonce", UUID.randomUUID()); + } catch (JSONException e) { + throw new RuntimeException(e.getMessage(), e.getCause()); + } + + String headerStr = header.toString().trim().replace("\\/", "/"); + String payloadStr = payload.toString().trim().replace("\\/", "/"); + String headerBase64 = Base64.encodeToString(headerStr.getBytes(), Base64.NO_PADDING | Base64.NO_WRAP | Base64.URL_SAFE); + String payloadBase64 = Base64.encodeToString(payloadStr.getBytes(), Base64.NO_PADDING | Base64.NO_WRAP | Base64.URL_SAFE); + + String jwtToSign = headerBase64 + "." + payloadBase64; + + String jwtSignature; + try { + byte[] origKey = Base64.decode(appSecret, Base64.DEFAULT); + byte[] signingKey = deriveSigningKey(origKey, issuedAt); + final byte[] macData = Hmac.hmacSha256(signingKey, jwtToSign); + String signature = Base64.encodeToString(macData, Base64.NO_PADDING | Base64.NO_WRAP | Base64.URL_SAFE); + jwtSignature = jwtToSign + "." + signature; + } catch (Exception e) { + throw new RuntimeException(e.getMessage(), e.getCause()); + } + return jwtSignature; + } + + private static String formatDate(long time) { + String format = "yyyyMMdd"; + SimpleDateFormat sdf = new SimpleDateFormat(format, Locale.getDefault()); + sdf.setTimeZone(TimeZone.getDefault()); + return sdf.format(new Date(time * 1000)); + } + + private static byte[] deriveSigningKey(byte[] key, long issuedAt) { + return Hmac.hmacSha256(key, formatDate(issuedAt)); + } +} diff --git a/skills/in-app-calling/assets/jwt-helper-andorid/JWTFetcher.kt b/skills/in-app-calling/assets/jwt-helper-andorid/JWTFetcher.kt new file mode 100644 index 0000000..136ea6f --- /dev/null +++ b/skills/in-app-calling/assets/jwt-helper-andorid/JWTFetcher.kt @@ -0,0 +1,16 @@ +package com.sinch.rtc.vvc.reference.app.utils.jwt + +/** + * JWTFetcher is responsible of creating JWT tokens based on application key and user id. + */ +interface JWTFetcher { + + /** + * Creates JWT token asynchronously based on application key and userId. + * @param applicationKey Application key copied from Sinch Portal dashboard. + * @param userId Id of the logged in user. + * @param callback Callback invoked after generation of the JWT token (String parameter) + */ + fun acquireJWT(applicationKey: String, userId: String, callback: (String) -> Unit) + +} \ No newline at end of file diff --git a/skills/in-app-calling/assets/jwt-helper.js b/skills/in-app-calling/assets/jwt-helper.js new file mode 100644 index 0000000..3ac19f5 --- /dev/null +++ b/skills/in-app-calling/assets/jwt-helper.js @@ -0,0 +1,140 @@ +export default class Crypto { + static algorithm = { name: "HMAC", hash: { name: "SHA-256" } }; + + static convertUTF8ToArrayBuffer(data) { + return new TextEncoder().encode(data); + } + + static getKey(secret) { + return window.crypto.subtle.importKey("raw", secret, this.algorithm, true, [ + "sign", + "verify", + ]); + } + + static async hmacSHA256(data, secret) { + const key = await this.getKey(secret); + return crypto.subtle.sign( + this.algorithm, + key, + this.convertUTF8ToArrayBuffer(data), + ); + } + + static getRandomValues(length) { + return window.crypto.getRandomValues(new Uint8Array(length)); + } + + static convertArrayBufferToHex(data) { + return Array.from(new Uint8Array(data), (byte) => + // eslint-disable-next-line no-bitwise + `0${(byte & 0xff).toString(16)}`.slice(-2), + ).join(""); + } + + static toBase64(data) { + return btoa(String.fromCharCode.apply(null, data)); + } + + static fromBase64(data) { + const padding = "=".repeat((4 - (data.length % 4)) % 4); + const base64 = (data + padding).replace(/-/g, "+").replace(/_/g, "/"); + + return new Uint8Array(Array.from(atob(base64)).map((c) => c.charCodeAt(0))); + } +} + +export default class JWT { + constructor(key, secret, username) { + this.key = key; + this.username = username; + this.iat = new Date(); + this.base64Secret = secret; + } + + // JWT header parameter kid as "kid": "hkdfv1-", e.g. "kid": "hkdfv1-20181206" + // The prefix hkdfv1 comes from that the key derivation function used is a form of HKDF RFC-5869 https://tools.ietf.org/html/rfc5869 and will be used by the Authorization Server to select the same key derivation function and input(date). + deriveApplicationKeyId(issuedAt) { + return `hkdfv1-${this.formatDate(issuedAt)}`; + } + + // Format date as YYYYMMDD + formatDate(date) { + return date.toISOString().replaceAll("-", "").substring(0, 8); + } + + // JWT Headers + headers() { + return { + alg: "HS256", + type: "JWT", + kid: this.deriveApplicationKeyId(this.iat), + }; + } + + // JWT Payload + payload() { + const nonce = Crypto.convertArrayBufferToHex(Crypto.getRandomValues(16)); + return { + iss: `//rtc.sinch.com/applications/${this.key}`, + sub: `//rtc.sinch.com/applications/${this.key}/users/${this.username}`, + iat: this.convertToSeconds(this.iat), + exp: this.convertToSeconds(this.iat) + 600000, + nonce, + }; + } + + sortObject(object) { + const sorted = {}; + Object.keys(object) + .sort() + .forEach((key) => { + sorted[key] = object[key]; + }); + return sorted; + } + + // Create signature from headers and payload + async signToken(headers, payload, signingKey) { + const signature = await Crypto.hmacSHA256( + `${headers}.${payload}`, + signingKey, + ); + return this.makeURLSafe(Crypto.toBase64(new Uint8Array(signature))); + } + + async toJwt() { + const date = this.formatDate(this.iat); + const signingKey = await Crypto.hmacSHA256( + date, + Crypto.fromBase64(this.base64Secret), + ); + const encodedHeaders = this.convertObjectToBase64( + this.sortObject(this.headers()), + ); + const encodedPayload = this.convertObjectToBase64( + this.sortObject(this.payload()), + ); + const signature = await this.signToken( + encodedHeaders, + encodedPayload, + signingKey, + ); + return `${encodedHeaders}.${encodedPayload}.${signature}`; + } + + convertObjectToBase64(str) { + const bytes = Crypto.convertUTF8ToArrayBuffer(JSON.stringify(str)); + const base64 = Crypto.toBase64(bytes); + return this.makeURLSafe(base64); + } + + makeURLSafe(u) { + // JWT RFC 7515 specifies that base64 encoding without padding should be used. + return u.replace(/\+/g, "-").replace(/\//g, "_").replace(/=+$/, ""); + } + + convertToSeconds(date) { + return Math.round(date.getTime() / 1000); + } +} diff --git a/skills/in-app-calling/references/android.md b/skills/in-app-calling/references/android.md new file mode 100644 index 0000000..51f7a20 --- /dev/null +++ b/skills/in-app-calling/references/android.md @@ -0,0 +1,494 @@ + +# Sinch Android Voice and Video SDK + +## Add the Sinch library + +You can include the Sinch SDK in your Android project in two ways. + +### AAR from SDK download +Copy the `.aar` file to your `libs` folder and update `build.gradle`: +```gradle +repositories { + flatDir { + dirs 'libs' + } +} + +dependencies { + implementation(name:'sinch-android-rtc', version:'+', ext:'aar') +} +``` + +### Maven Central +Consume the SDK directly from Maven Central. See the [SDK Downloads](https://developers.sinch.com/docs/in-app-calling/sdk-downloads#android-sdk-on-maven-central) page for the Gradle coordinates. + +## Integration steps + +Every integration must follow these steps in order. Do not skip or reorder them. + +1. **Declare permissions in AndroidManifest.xml** - Add `INTERNET`, `ACCESS_NETWORK_STATE`, `RECORD_AUDIO`, `MODIFY_AUDIO_SETTINGS`, `READ_PHONE_STATE` . And add `CAMERA` if the implementation uses video calls. +2. **Request runtime permissions** - Before starting a call, request `RECORD_AUDIO` (and `CAMERA` for video calls) from the user at runtime using the standard Android permission flow. The SDK will not function correctly without these grants. +3. **Set up FCM** - Add `google-services.json` to your project, include the `com.google.gms.google-services` Gradle plugin, and add `com.google.firebase:firebase-messaging` as a dependency. Obtain the FCM sender ID and registration token. +4. **Build a PushConfiguration** - Create an `FcmPushConfiguration` with the sender ID and registration token obtained in step 3. +5. **Create a SinchClient** - Build the client with application key, environment host, user ID, Android context, and the push configuration from step 4. +6. **Add a SinchClientListener** - Attach a `SinchClientListener` to handle `onClientStarted`, `onClientFailed`, and `onCredentialsRequired`. +7. **Authorize the client** - Implement `onCredentialsRequired` to supply a signed JWT. Decide whether to sign on-device (prototyping only) or fetch from a backend. +8. **Start the client** - Call `sinchClient.start()`. Wait for `onClientStarted` before proceeding. +9. **Add a CallControllerListener** - Attach a listener to `sinchClient.callController` to detect incoming calls via `onIncomingCall`. +10. **Add a CallListener to each call** - On every outgoing or incoming call, attach a `CallListener` (or `VideoCallListener` for video) to handle `onCallProgressing`, `onCallRinging`, `onCallAnswered`, `onCallEstablished`, and `onCallEnded`. + +When helping a user integrate, walk through these steps one at a time. Confirm each step is in place before moving to the next. + + +## Permissions + +Add the following to your `AndroidManifest.xml`: +```xml + + + + + + +``` + +**Runtime permissions (Android 6.0+):** +`RECORD_AUDIO` and `CAMERA` are dangerous permissions. You must request them at runtime before placing or answering a call. + +Ask the user whether they have implemented the runtime permission request. If not, guide them through the standard `ActivityCompat.requestPermissions` flow for `RECORD_AUDIO` (required for all calls) and `CAMERA` (required for video calls). + +```kotlin +// Example: request audio and camera permissions before a video call +val permissions = arrayOf( + Manifest.permission.RECORD_AUDIO, + Manifest.permission.CAMERA +) +ActivityCompat.requestPermissions(activity, permissions, REQUEST_CODE) +``` + +After the user grants permissions, verify with `SinchClient.checkManifest()` before starting the client. This method checks that all permissions required by the currently enabled features are granted. + +Note: By default the SDK hangs up Sinch calls when a native phone call is active. This requires `READ_PHONE_STATE`. To disable this behavior, call `sinchClient.callController.setRespectNativeCalls(false)` and remove the permission. + + +## Sinch Client +The *SinchClient* is the Sinch SDK entry point. It configures user and device capabilities and provides access to the *CallController*, *AudioController*, and *VideoController*. + +### Create a *SinchClient* +```kotlin +val sinchClient = SinchClient.builder() + .context(applicationContext) + .applicationKey("") + .environmentHost("ocra.api.sinch.com") + .userId("") + .pushConfiguration(pushConfiguration) // see Push Notifications section + .build() +``` +- The *Application Key* is obtained from the [Sinch Developer Dashboard - Apps](https://dashboard.sinch.com/voice/apps). +- The *User ID* should uniquely identify the user on the device. +- The term *Ocra* in the hostname `ocra.api.sinch.com` is the name of the Sinch API that SDK clients target. +- The *Push Configuration* registers the device for incoming call notifications. See [Push Notifications](#push-notifications-fcm). + +**Listener threading:** All listener callbacks are invoked on the same thread that called `SinchClientBuilder.build`. If that is not the main thread, it must have an associated `Looper`. + +### Start the Sinch client +Before starting, add a client listener (see [SinchClientListener reference](https://download.sinch.com/android/latest/reference/sinch-rtc/com.sinch.android.rtc/-sinch-client-listener/index.html)): + +```kotlin +sinchClient.addSinchClientListener(object : SinchClientListener() { + + override fun onClientStarted(client: SinchClient) { + // Sinch client started successfully + } + + override fun onClientFailed(client: SinchClient, error: SinchError) { + // Sinch client start failed + } + + override fun onCredentialsRequired(clientRegistration: ClientRegistration) { + // Registration required, get JWT token for user and register + } + + override fun onLogMessage(level: Int, area: String, message: String) { } +}) + +sinchClient.start() +``` + +### Authentication & authorization +When `SinchClient` starts with a given user ID, you must provide an authorization token (JWT) to register with Sinch. +Implement `SinchClientListener.onCredentialsRequired()` and supply a JWT signed with the Application Secret. + +Read https://developers.sinch.com/docs/in-app-calling/android/application-authentication.md + +For production application it is recommended to generate and sign the JWT token on the backend server, then send it over a secure channel to the application and Sinch client running on the device. +For development or test purposes it is fine to have it embeded. + +Ask the user whether it can be embedded. +If it can be embedded: +A sample code for jwt generation is provided at `assets/jwt-helper-andorid` folder. Read and adapt it. +```kotlin +override fun onCredentialsRequired(clientRegistration: ClientRegistration) { + try { + val jwt = JWT.create(APP_KEY, APP_SECRET, userId) + clientRegistration.register(jwt) + } catch (e: Exception) { + Log.e(TAG, "Authentication failed: ${e.message}") + clientRegistration.registerFailed() + } +} +``` +If it cannot be embedded: +Implement the required functionality on your backend and fetch a signed registration token when required. + +```kotlin +override fun onCredentialsRequired(clientRegistration: ClientRegistration) { + yourAuthServer.getRegistrationToken(userId, object : AuthCallback { + fun onSuccess(token: String) { + clientRegistration.register(token) + } + fun onFailure() { + clientRegistration.registerFailed() + } + }) +} +``` + +### Lifecycle management +Keep the `SinchClient` instance alive and started for the lifetime of the running application. Avoid unnecessary stop/restart cycles. + +Stopping or disposing of a `SinchClient` does not prevent receiving incoming calls if the user was previously registered. When an incoming call push arrives, create a new `SinchClient` and forward the push payload to it. + +To fully dispose: +```kotlin +sinchClient?.terminateGracefully() +sinchClient = null +``` + + +## Push Notifications (FCM) + +Push notifications let your app receive incoming calls even when backgrounded or closed. This section covers the FCM flow only. + +### Step 1. Add Firebase to your project + +1. Register your app in the [Firebase Console](https://console.firebase.google.com/). +2. Download `google-services.json` and place it in your app module's root folder (e.g. `app/google-services.json`). +3. Add the Google Services Gradle plugin in your project-level `build.gradle`: +```gradle +plugins { + id 'com.google.gms.google-services' version '' apply false +} +``` +4. Apply it in your app-level `build.gradle`: +```gradle +plugins { + id 'com.google.gms.google-services' +} +``` +5. Add the Firebase Messaging dependency: +```gradle +dependencies { + implementation 'com.google.firebase:firebase-messaging:' +} +``` + +### Step 2. Configure FCM v1 OAuth in the Sinch Dashboard + +In the [Sinch Dashboard](https://dashboard.sinch.com/voice/apps), select your app and go to the "In-app Voice & Video SDKs" tab. Under "Google FCM Identification", provide your OAuth endpoints so Sinch can send push messages on your behalf. This requires implementing a backend OAuth flow. See the [FCM v1 OAuth2.0 documentation](https://developers.sinch.com/docs/in-app-calling/android/push-notifications#fcm-v1-oauth20-flow) for details. + +### Step 3. Acquire sender ID and registration token + +```kotlin +// Sender ID (stable, bundled in google-services.json) +val senderId: String = FirebaseApp.getInstance().options.gcmSenderId.orEmpty() + +// Registration token (async, can change) +FirebaseMessaging.getInstance().token.addOnCompleteListener { task -> + if (task.isSuccessful) { + val registrationToken = task.result + // Use this token to build push configuration + } +} +``` + +Track token changes by overriding `onNewToken` in your `FirebaseMessagingService`. When the token changes, recreate `SinchClient` with the new token. + +### Step 4. Build PushConfiguration and pass to SinchClient + +```kotlin +val pushConfiguration = PushConfiguration.fcmPushConfigurationBuilder() + .senderID(senderId) + .registrationToken(registrationToken) + .build() + +val sinchClient = SinchClient.builder() + .context(applicationContext) + .applicationKey("") + .userId("") + .environmentHost("ocra.api.sinch.com") + .pushConfiguration(pushConfiguration) + .build() + +sinchClient.addSinchClientListener(listener) +sinchClient.start() +``` + +### Step 5. Implement the FCM Listening Service + +Create a service extending `FirebaseMessagingService` to receive and forward Sinch push payloads: + +```kotlin +class FcmListenerService : FirebaseMessagingService() { + + override fun onMessageReceived(remoteMessage: RemoteMessage) { + if (SinchPush.isSinchPushPayload(remoteMessage.data)) { + val result = SinchPush.queryPushNotificationPayload( + applicationContext, remoteMessage.data + ) + // result contains caller info, video flag, timeout status + // Forward to your running SinchClient instance: + sinchClient.relayRemotePushNotification(result) + } else { + // Not a Sinch message, handle your own push logic + } + } + + override fun onNewToken(token: String) { + // Token changed: recreate SinchClient with new token + } +} +``` + +### Unregistering a device +If the user logs out, call `sinchClient.unregisterPushToken()` so the device stops receiving incoming call notifications. This is critical if your app supports switching between users on the same device. + +Note: +For use cases requiring only outgoing App-to-Phone, App-to-SIP, or App-to-Conference calls, providing `PushConfiguration` is not required. You can place these calls directly once the Sinch client is started. + + +## Voice Calling +The Sinch SDK supports four types of calls: *app-to-app (audio or video)*, *app-to-phone*, *app-to-sip* and *conference* calls. The *CallController* is the entry point for calling functionality. +Calls are placed through the *CallController* and events are received using the *CallControllerListener*. The call controller is owned by the SinchClient and accessed using `sinchClient.callController`. + +### Set up an *App-to-App* call +Use the [CallController.callUser()](https://download.sinch.com/android/latest/reference/sinch-rtc/com.sinch.android.rtc.calling/-call-controller/call-user.html) method so Sinch can connect the call to the callee. + +```kotlin +val callController = sinchClient.callController +val call = callController.callUser("", MediaConstraints(false)) +// Video call: +// val call = callController.callUser("", MediaConstraints(true)) +call.addCallListener(...) +``` +The returned call object includes participant details, start time, state, and possible errors. + +Assuming the callee's device is available, +[CallListener.onCallProgressing()](https://download.sinch.com/android/latest/reference/sinch-rtc/com.sinch.android.rtc.calling/-call-listener/on-call-progressing.html) +is invoked. If you play a progress tone, start it here. +When the callee receives the call, +[CallListener.onCallRinging()](https://download.sinch.com/android/latest/reference/sinch-rtc/com.sinch.android.rtc.calling/-call-listener/on-call-ringing.html) +fires. This indicates the callee's device is ringing. +When the other party answers, +[CallListener.onCallAnswered()](https://download.sinch.com/android/latest/reference/sinch-rtc/com.sinch.android.rtc.calling/-call-listener/on-call-answered.html) +is called. Stop any progress tone. +Once full audio connectivity is established, +[CallListener.onCallEstablished()](https://download.sinch.com/android/latest/reference/sinch-rtc/com.sinch.android.rtc.calling/-call-listener/on-call-established.html) +is emitted. Users can now talk. + +Typically, connectivity is already established when the call is answered, so +`onCallEstablished` may follow immediately after `onCallAnswered`. +On poor networks, it can take longer. Consider showing a "connecting" indicator. + +**Important:** +For *App-to-App* calls, you must provide FCM `PushConfiguration` to `SinchClientBuilder` to receive push messages. See the full setup in [Push Notifications](#push-notifications-fcm). + +### Set up an *App-to-Phone* call + +An *app-to-phone* call targets the regular telephone network. Use [CallController.callPhoneNumber()](https://download.sinch.com/android/latest/reference/sinch-rtc/com.sinch.android.rtc.calling/-call-controller/call-phone-number.html) with an E.164 formatted number prefixed with `+`. For example, to call the US number 415 555 0101, use `+14155550101`. + +#### Presenting a number to the destination you are calling + +Mandatory step! +You must provide a CLI (Calling Line Identifier) or your call will fail. +You need a number from Sinch so you can provide a valid CLI to the handset you are calling. + +Note: When your account is in trial mode, you can only call your [verified numbers](https://dashboard.sinch.com/numbers/verified-numbers). If you want to call any number, you need to upgrade your account! + +```kotlin +val callController = sinchClient.callController +val destinationNumber = "" +val cli = "" +val call = callController.callPhoneNumber(destinationNumber, cli) +``` + +### Set up an *App-to-sip* call + +Use [CallController.callSip()](https://download.sinch.com/android/latest/reference/sinch-rtc/com.sinch.android.rtc.calling/-call-controller/call-sip.html). The SIP identity should be in the form `user@server`. Custom SIP headers should be prefixed with `x-`. If the SIP server reports errors, `CallDetails` will provide an error with the `SIP` error type. + +### Set up a *Conference* call +A *conference* call connects a user to a room where multiple users can participate simultaneously. The conference room identifier may not exceed 64 characters. + +```kotlin +val callController = sinchClient.callController +val call = callController.callConference("") +call.addCallListener(...) +``` + +### Handle incoming calls + +Add a [CallControllerListener](https://download.sinch.com/android/latest/reference/sinch-rtc/com.sinch.android.rtc.calling/-call-controller-listener/index.html) to `CallController` to act on incoming calls. When a call arrives, [CallControllerListener.onIncomingCall()](https://download.sinch.com/android/latest/reference/sinch-rtc/com.sinch.android.rtc.calling/-call-controller-listener/index.html) is executed. + +```kotlin +val callController = sinchClient.callController +callController.addCallControllerListener(...) +``` + +When the incoming call callback fires, the call can be connected automatically or wait for user interaction. If waiting, play a ringtone to notify the user. + +```kotlin +override fun onIncomingCall(callController: CallController, call: Call) { + // Start playing ringing tone + ... + + // Add call listener + call.addCallListener(...) +} +``` + +#### Receiving Calls from PSTN or SIP (Phone-to-App / SIP-to-App) + +The Sinch SDK supports receiving incoming calls from the PSTN or SIP endpoints. When a call arrives at a Sinch voice number or via SIP origination, the Sinch platform triggers an **Incoming Call Event (ICE)** callback to your backend. Your platform can then route this call to an in-app user by responding with the `connectMxp` SVAML action. + +##### Prerequisites + +1. Rent a voice number from the [Sinch Build Dashboard](https://dashboard.sinch.com/numbers/overview) and assign it to your application, OR configure SIP origination for your application. +2. Configure a callback URL in your app's Voice settings where Sinch will send call-related events. +3. Implement the ICE callback handler in your backend to route calls to the appropriate app user. + +##### Backend Implementation + +When a PSTN or SIP call comes in, respond to the ICE callback with a `connectMxp` action: + +```json +{ + "action": { + "name": "connectMxp", + "destination": { + "type": "username", + "endpoint": "target-user-id" + } + } +} +``` + +#### Incoming video call + +When an incoming call is a video call, `onIncomingCall` is also executed and `call.details.isVideoOffered` returns `true`. See [Video Calling](#video-calling) for how to handle video views. + +#### Answer incoming call + +Use [Call.answer()](https://download.sinch.com/android/latest/reference/sinch-rtc/com.sinch.android.rtc.calling/-call/answer.html) to accept. Stop any ringtone. + +```kotlin +// User answers the call +call.answer() + +// Stop playing ringing tone +``` + +#### Decline incoming call + +Use [Call.hangup()](https://download.sinch.com/android/latest/reference/sinch-rtc/com.sinch.android.rtc.calling/-call/hangup.html) to decline. The caller is notified. Stop any ringtone. + +```kotlin +// User doesn't want to answer +call.hangup() + +// Stop playing ringing tone +``` + +### Disconnecting a Call +Use [Call.hangup()](https://download.sinch.com/android/latest/reference/sinch-rtc/com.sinch.android.rtc.calling/-call/hangup.html). Either party can disconnect. + +```kotlin +call.hangup() +``` + +When either party disconnects, [CallListener.onCallEnded()](https://download.sinch.com/android/latest/reference/sinch-rtc/com.sinch.android.rtc.calling/-call-listener/on-call-ended.html) fires, allowing you to update the UI or play an alert tone. + +A call can be disconnected before it is fully established: +```kotlin +val call = callController.callUser("", MediaConstraints(false)) +// User changed their mind +call.hangup() +``` + +### Volume control + +Call `setVolumeControlStream(AudioManager.STREAM_VOICE_CALL)` on the `Activity` handling the call so hardware volume buttons control the call volume. Reset it when the call ends: + +```kotlin +override fun onCallEnded(call: Call) { + setVolumeControlStream(AudioManager.USE_DEFAULT_STREAM_TYPE) +} +``` + +### Audio routing + +Use [AudioController](https://download.sinch.com/android/latest/reference/sinch-rtc/com.sinch.android.rtc/-audio-controller/index.html) via `sinchClient.audioController` to control mute, speaker, and automatic audio routing. + +Key methods: `mute()`, `unmute()`, `enableSpeaker()`, `disableSpeaker()`, `enableAutomaticAudioRouting(config)`, `disableAutomaticAudioRouting()`. + +Automatic audio routing priorities: Bluetooth (if enabled and `BLUETOOTH` permission granted) > Wired headset > Default (speakerphone or earpiece based on `useSpeakerphone` setting or proximity sensor in `AUTO` mode). + + +## Video Calling + +Video calls follow the same flow as audio calls but use `MediaConstraints(true)` and a `VideoCallListener`. + +### Setting up a video call + +```kotlin +val call = sinchClient.callController.callUser("", MediaConstraints(true)) +call.addCallListener(myVideoCallListener) +``` + +**Remember:** Request `CAMERA` permission at runtime before placing or answering a video call. + +### Showing the video streams + +Implement [VideoCallListener](https://download.sinch.com/android/latest/reference/sinch-rtc/com.sinch.android.rtc.video/-video-call-listener/index.html) and use `onVideoTrackAdded` to get video views: + +```kotlin +override fun onVideoTrackAdded(call: Call) { + val videoController = sinchClient.videoController + val localView = videoController.localView + val remoteView = videoController.remoteView + + // Add localView and remoteView to your view hierarchy +} +``` + +Remove the views from your hierarchy when the call ends. + +#### Pausing and resuming a video stream + +Use [Call.pauseVideo()](https://download.sinch.com/android/latest/reference/sinch-rtc/com.sinch.android.rtc.calling/-call/pause-video.html) and [Call.resumeVideo()](https://download.sinch.com/android/latest/reference/sinch-rtc/com.sinch.android.rtc.calling/-call/resume-video.html). Audio continues unless muted separately. + +Listeners are notified via `VideoCallListener.onVideoTrackPaused()` and `onVideoTrackResumed()` to update UI accordingly. + +### Video scaling + +Use `VideoController.setResizeBehaviour(VideoScalingType)` and `setLocalResizeBehaviour(VideoScalingType)` to control scaling. Options: `ASPECT_FIT`, `ASPECT_FILL`, `ASPECT_BALANCED`. + +### Switching camera + +Toggle front/back camera using [VideoController.toggleCaptureDevicePosition()](https://download.sinch.com/android/latest/reference/sinch-rtc/com.sinch.android.rtc.video/-video-controller/toggle-capture-device-position.html). + +### Accessing raw video frames + +Implement [RemoteVideoFrameListener](https://download.sinch.com/android/latest/reference/sinch-rtc/com.sinch.android.rtc.video/-remote-video-frame-listener/index.html) and register it with `videoController.setRemoteVideoFrameListener(handler)` to receive raw I420 frames for processing (filters, screenshots, etc.). A similar `setLocalVideoFrameListener` exists for local frames. + +Use `VideoUtils.I420toNV21Frame(videoFrame)` to convert to NV21 format for saving as an image on Android. \ No newline at end of file diff --git a/skills/in-app-calling/references/ios.md b/skills/in-app-calling/references/ios.md new file mode 100644 index 0000000..be0d4ee --- /dev/null +++ b/skills/in-app-calling/references/ios.md @@ -0,0 +1,611 @@ + +# Sinch iOS Voice and Video SDK (Swift) + +## Add the Sinch library +You can include the Sinch SDK in several ways. Pick the one that fits your setup. + +### Swift Package Manager (recommended for Swift) +1. In Xcode, go to **File > Add Package Dependencies...** +2. Repository URL: +``` +https://github.com/sinch/sinch-ios-sdk-spm +``` +3. Choose a dependency rule: + - Branch `dynamic` (or `main`) for the latest dynamic xcframework + - Branch `static` for the latest static xcframework + - **Exact Version** `x.y.z` for a pinned dynamic release +4. Click **Add Package**. + +For `dynamic`/`main`: in your app target > General > Frameworks, Libraries, and Embedded Content, set SinchRTC to **Embed & Sign**. +For `static`: set SinchRTC to **Do Not Embed**. + +### CocoaPods (Objective-C only) +```ruby +platform :ios, '12.0' + +target 'YourApp' do + pod 'SinchRTC', 'x.y.z' + workspace './YourApp.xcworkspace' +end +``` +Then run `pod install`. + +### Manual integration +Drag `SinchRTC.xcframework` (Swift) or `Sinch.xcframework` (Objective-C) into the Frameworks section in Xcode Project Navigator and set it to **Embed & Sign**. + +If you integrate manually, link these system frameworks/libraries: `libc++.tbd`, `libz.tbd`, `Security.framework`, `AVFoundation.framework`, `AudioToolbox.framework`, `VideoToolbox.framework`, `CoreMedia.framework`, `CoreVideo.framework`, `CoreImage.framework`, `GLKit.framework`, `OpenGLES.framework`, `QuartzCore.framework`, `Metal.framework`, `MetalKit.framework`, `PushKit.framework`, `SystemConfiguration.framework`. + +## Capabilities and Info.plist + +Before writing any code, configure your Xcode project. iOS handles permission prompts automatically at runtime, but you must declare the required entries or your app will crash or be rejected. + +### Capabilities +Enable **Push Notifications** in your app target's Signing & Capabilities. This adds `aps-environment` to your entitlements. + +### Info.plist entries + +Add the following keys to your `Info.plist`: + +**Required background modes** (`UIBackgroundModes`): +- `audio` - App plays audio or streams audio/video using AirPlay +- `voip` - App provides Voice over IP services + +**Privacy - Microphone Usage Description** (`NSMicrophoneUsageDescription`): +A string explaining why the app needs microphone access. Example: `"Application wants to use your microphone to be able to capture your voice in a call."` + +**Privacy - Camera Usage Description** (`NSCameraUsageDescription`) - only if you enable video: +A string explaining why the app needs camera access. Example: `"Application wants to use your camera to be able to make a video call."` + +Note: iOS will present a system permission dialog to the user the first time the microphone or camera is activated. You do not request these permissions manually through Sinch, but you must declare the usage descriptions or iOS will terminate your app. + +## Integration steps + +Every integration must follow these steps in order. Do not skip or reorder them. + +1. **Configure Info.plist and Capabilities** - Add background modes (`audio`, `voip`), microphone/camera usage descriptions, and enable Push Notifications capability. +2. **Create a SinchClient** - Instantiate with application key, environment host, and user ID. +3. **Set the client delegate** - Assign a `SinchClientDelegate` to handle `clientDidStart`, `clientDidFail`, and `clientRequiresRegistrationCredentials`. +4. **Authorize the client** - Implement `clientRequiresRegistrationCredentials` to supply a signed JWT. Decide whether to sign locally (prototyping only) or fetch from a backend. +5. **Enable managed push notifications** - Call `sinchClient.enableManagedPushNotifications()` before `start()`. Create `SinchManagedPush` early in your app lifecycle (typically in `AppDelegate`). Required for app-to-app calls, even as the caller. +6. **Set up CallKit (or LiveCommunicationKit)** - Integrate with Apple's system calling UI. You must report every incoming VoIP push as a call to CallKit/LiveCommunicationKit before the push delegate returns, or iOS will terminate your app. Also report outgoing calls so audio works when the app is backgrounded or the device is locked. You do not provide any token manually when building the client for this; the integration between Sinch and CallKit happens through push payload handling and `CXProvider`/`ConversationManager`. +7. **Start the client** - Call `sinchClient.start()`. Wait for `clientDidStart` before proceeding. +8. **Add a SinchCallClientDelegate** - Assign a delegate to `sinchClient.callClient` to detect incoming calls via `client(_:didReceiveIncomingCall:)`. +9. **Add a SinchCallDelegate to each call** - On every outgoing or incoming call, assign a `SinchCallDelegate` to handle `callDidProgress`, `callDidAnswer`, `callDidEstablish`, and `callDidEnd`. + +When helping a user integrate, walk through these steps one at a time. Confirm each step is in place before moving to the next. + + +## Sinch Client +The `SinchClient` is the Sinch SDK entry point. It manages the client lifecycle and capabilities, and exposes feature APIs such as `callClient` (calling), `audioController` (audio), and `videoController` (video). + +### Create a SinchClient +```swift +import SinchRTC + +// Keep a strong reference to sinchClient +private(set) var sinchClient: SinchClient? + +do { + self.sinchClient = try SinchRTC.client(withApplicationKey: "", + environmentHost: "ocra.api.sinch.com", + userId: "") +} catch { + // Handle error +} +``` +- The *Application Key* is obtained from the [Sinch Developer Dashboard - Apps](https://dashboard.sinch.com/voice/apps). +- The *User ID* should uniquely identify the user on the current device. +- The term *Ocra* in the hostname `ocra.api.sinch.com` is the name of the Sinch API that SDK clients target. + +### Start the Sinch client +Before starting, assign a delegate conforming to `SinchClientDelegate`: + +```swift +sinchClient.delegate = self +sinchClient.start() +``` + +Delegate methods: + +```swift +// SinchClientDelegate + +func clientDidStart(_ client: SinchRTC.SinchClient) { + // Sinch client started successfully +} + +func clientDidFail(_ client: SinchRTC.SinchClient, error: Error) { + // Sinch client start failed +} + +func clientRequiresRegistrationCredentials(_ client: SinchRTC.SinchClient, + withCallback callback: SinchRTC.SinchClientRegistration) { + // Registration required, get JWT token for user and register +} +``` + +### Authentication & authorization +When `SinchClient` starts with a given user ID, you must provide an authorization token (JWT) to register with Sinch. +Implement `clientRequiresRegistrationCredentials(_:withCallback:)` and supply a JWT signed with a key derived from the Application Secret. + +Read https://developers.sinch.com/docs/in-app-calling/ios/auth.md + +In general it is not suggested to embed the Application Secret in a production application. + +Ask the user whether it can be embedded. +If it can be embedded: +The Swift reference application on [GitHub](https://github.com/sinch/rtc-reference-applications/tree/master/ios) includes a `SinchJWT.swift` helper that demonstrates how to create and sign the JWT locally. I can be also found in `assets/SinchJWT.swift` Read and adapt it. + +```swift +func clientRequiresRegistrationCredentials(_ client: SinchRTC.SinchClient, + withCallback callback: SinchRTC.SinchClientRegistration) { + do { + // WARNING: Development example only. In production, fetch a JWT from your backend. + let jwt = try SinchJWT.sinchJWTForUserRegistration(withApplicationKey: "", + applicationSecret: "", + userId: client.userId) + callback.register(withJWT: jwt) + } catch { + callback.registerDidFail(error: error) + } +} +``` +If it can not be embedded: +Implement the required functionality on your backend and fetch a signed registration token when required. + +```swift +func clientRequiresRegistrationCredentials(_ client: SinchRTC.SinchClient, + withCallback callback: SinchRTC.SinchClientRegistration) { + authServer.fetchRegistrationToken(for: client.userId) { result in + switch result { + case .success(let token): + callback.register(withJWT: token) + case .failure(let error): + callback.registerDidFail(error: error) + } + } +} +``` + +### Life cycle management +Create and start a single `SinchClient` and keep it alive for the lifetime of your application. Retain a strong reference. The client uses little memory once started. + +To temporarily stop incoming calls without disposing the client: +```swift +sinchClient.unregisterPushNotificationDeviceToken() +``` + +To completely stop and dispose of the client: +```swift +sinchClient.terminateGracefully() +sinchClient = nil +``` + +### Push notifications +To receive incoming calls via Apple VoIP push notifications, enable managed push on the client and set up `SinchManagedPush`. + +**Enable managed push on the client:** +```swift +sinchClient.enableManagedPushNotifications() +``` + +**Create SinchManagedPush early in the app lifecycle** (typically in `AppDelegate.application(_:didFinishLaunchingWithOptions:)`): + +```swift +class AppDelegate: UIResponder, UIApplicationDelegate { + + private var sinchPush: SinchManagedPush? + + func application(_ application: UIApplication, + didFinishLaunchingWithOptions launchOptions: + [UIApplication.LaunchOptionsKey: Any]?) -> Bool { + sinchPush = SinchRTC.managedPush(forAPSEnvironment: .development) + sinchPush?.delegate = self + sinchPush?.setDesiredPushType(SinchManagedPush.TypeVoIP) + return true + } +} +``` + +The APS environment you pass (`.development` or `.production`) must match your app's provisioning profile. A Debug build signed with a Development profile uses `.development`; a Release build signed with a Distribution profile uses `.production`. + +**Upload your APNs Signing Key:** +1. Create an APNs Key in your [Apple Developer Account](https://developer.apple.com/). +2. Upload the `.p8` key file to your [Sinch Developer Account](https://dashboard.sinch.com/voice/apps). + +`SinchManagedPush` is lightweight and can live independently of a `SinchClient`. It acquires the push device token via PushKit and automatically registers it with any `SinchClient` created later. + +Note: +For use cases requiring only outgoing App-to-Phone, App-to-SIP, or Conference calls, calling `sinchClient.enableManagedPushNotifications()` is not required. You can place these calls directly once the Sinch client is started. + +### CallKit integration + +Apple requires that every incoming VoIP push notification is reported to CallKit (or LiveCommunicationKit on iOS 17.4+) before your push delegate returns. Failing to do so will cause iOS to terminate your app, and repeated failures may stop VoIP push delivery entirely. + +**Handling incoming pushes with CallKit:** + +```swift +func managedPush(_ managedPush: SinchRTC.SinchManagedPush, + didReceiveIncomingPushWithPayload payload: [AnyHashable: Any], + for type: String) { + let notification = queryPushNotificationPayload(payload) + + guard notification.isCall, notification.isValid else { return } + + let callNotification = notification.callResult + + let uuid = // Get or create a UUID mapped to callNotification.callId + + let update = CXCallUpdate() + update.remoteHandle = CXHandle(type: .generic, value: callNotification.remoteUserId) + + self.provider.reportNewIncomingCall(with: uuid, update: update) { error in + // Handle error and hangup call if needed + } +} +``` + +If you do not relay the payload to a `SinchClient`, call `SinchManagedPush.didCompleteProcessingPushPayload(_:)` so PushKit's completion handler is invoked. + +**Reporting outgoing calls to CallKit:** +While not strictly required by Apple for outgoing calls, reporting them to CallKit is necessary for audio to work when the caller app is in the background or the device is locked. + +```swift +func call(userId: String, uuid: UUID, with completion: @escaping (Error?) -> Void) { + let handle = CXHandle(type: .generic, value: userId) + let startCallAction = CXStartCallAction(call: uuid, handle: handle) + let transaction = CXTransaction(action: startCallAction) + self.callController.request(transaction, completion: completion) +} +``` + +Implement `CXProviderDelegate` to start the Sinch call when CallKit requests it: + +```swift +func provider(_ provider: CXProvider, perform action: CXStartCallAction) { + let recipientIdentifier = action.handle.value + let callResult = callClient.callUser(withId: recipientIdentifier) + + switch callResult { + case .success(let call): + call.delegate = self + action.fulfill() + case .failure(let error): + action.fail() + } +} +``` + +**Audio session with CallKit:** +When using CallKit, the system manages audio session activation. Forward these events to the SDK: + +```swift +// In your CXProviderDelegate +func provider(_ provider: CXProvider, didActivate audioSession: AVAudioSession) { + sinchClient?.callClient.didActivate(audioSession: audioSession) +} + +func provider(_ provider: CXProvider, didDeactivate audioSession: AVAudioSession) { + sinchClient?.callClient.didDeactivate(audioSession: audioSession) +} +``` + +See the full reference app for a complete CallKit implementation: [Swift reference application on GitHub](https://github.com/sinch/rtc-reference-applications/tree/master/ios). + +### LiveCommunicationKit (iOS 17.4+) + +LiveCommunicationKit is an alternative to CallKit. The same rules apply: report every incoming VoIP push before your delegate returns. + +```swift +func managedPush(_ managedPush: SinchRTC.SinchManagedPush, + didReceiveIncomingPushWithPayload payload: [AnyHashable: Any], + for type: String) { + let notification = queryPushNotificationPayload(payload) + + guard notification.isCall, notification.isValid else { return } + + let callNotification = notification.callResult + let uuid = // Get or create a UUID mapped to callNotification.callId + + let localHandle = Handle(type: .generic, value: "localUserId") + let remoteHandle = Handle(type: .generic, value: callNotification.remoteUserId) + + let update = Conversation.Update(localMember: localHandle, + activeRemoteMembers: [remoteHandle], + capabilities: nil) + + Task { + do { + try await self.conversationManager.reportNewIncomingConversation(uuid: uuid, update: update) + } catch { + // Handle error and hangup call if needed + } + } +} +``` + +## Voice calling +The Sinch SDK supports four types of calls: *app-to-app (audio or video)*, *app-to-phone*, *app-to-sip*, and *conference* calls. The `SinchCallClient` is the entry point for calling functionality. +Calls are placed through `SinchCallClient` and events are received via `SinchCallClientDelegate`. The call client is owned by `SinchClient` and accessed using `sinchClient.callClient`. + +### Set up an *App-to-App* call +```swift +guard let callClient = sinchClient?.callClient else { return } + +let callResult = callClient.callUser(withId: "") + +switch callResult { +case .success(let call): + call.delegate = self +case .failure(let error): + // Handle error +} +``` + +The returned call object includes participant details, start time, state, and possible errors. + +Assuming the callee's device is available, `callDidProgress(_:)` is invoked. If you play a progress tone, start it here. +When the callee answers, `callDidAnswer(_:)` fires. Stop any progress tone. +When full audio connectivity is established, `callDidEstablish(_:)` is called. Users can now talk. + +Typically, connectivity is already established when the call is answered, so `callDidEstablish` may follow immediately after `callDidAnswer`. On poor networks, it can take longer; consider showing a "connecting" indicator. + +**Important:** +For *App-to-App* calls, you must enable managed push by calling `sinchClient.enableManagedPushNotifications()`, even if you are the caller. See the full setup in [Push notifications](/docs/in-app-calling/ios/push-notifications). + +### Set up an *App-to-Phone* call + +An *app-to-phone* call is a call made to a phone on the regular telephone network. Use `callPhoneNumber(_:)` with an E.164-formatted number prefixed with `+`. + +```swift +guard let callClient = sinchClient?.callClient else { return } + +let callResult = callClient.callPhoneNumber("+14155550101") + +switch callResult { +case .success(let call): + call.delegate = self +case .failure(let error): + // Handle error +} +``` + +#### Presenting a number to the destination you are calling + +Mandatory step! +You must provide a CLI (Calling Line Identifier) or your call will fail. +You need a number from Sinch so you can provide a valid CLI to the handset you are calling. +Specify your CLI when creating the SinchClient: + +```swift +do { + self.sinchClient = try SinchRTC.client(withApplicationKey: "", + environmentHost: "ocra.api.sinch.com", + userId: "", + cli: "") +} catch { + // Handle error +} +``` + +Note: When your account is in trial mode, you can only call your [verified numbers](https://dashboard.sinch.com/numbers/verified-numbers). If you want to call any number, you need to upgrade your account! + +### Set up an *App-to-SIP* call + +An *app-to-sip* call is made to a SIP server. Use `callSIP(_:)` or `callSIP(_:headers:)`. The SIP identity should be in the form `user@server`. When passing custom headers, prefix them with `x-`. + +```swift +guard let callClient = sinchClient?.callClient else { return } + +let callResult = callClient.callSIP("") + +switch callResult { +case .success(let call): + call.delegate = self +case .failure(let error): + // Handle error +} +``` + +### Set up a *Conference* call +A conference call connects a user to a room where multiple users can participate. The identifier may not be longer than 64 characters. + +```swift +guard let callClient = sinchClient?.callClient else { return } + +let callResult = callClient.callConference(withId: "") + +switch callResult { +case .success(let call): + call.delegate = self +case .failure(let error): + // Handle error +} +``` + +### Handle incoming calls + +Add a `SinchCallClientDelegate` to `SinchCallClient` to act on incoming calls. When a call arrives, `client(_:didReceiveIncomingCall:)` is executed. + +**With CallKit/LiveCommunicationKit** (typical production setup): +Use `client(_:didReceiveIncomingCall:)` primarily to associate the `SinchCall` with the system call. Keep a mapping between system UUIDs and `callId`. + +```swift +extension SinchClientMediator: SinchCallClientDelegate { + func client(_ client: SinchRTC.SinchCallClient, + didReceiveIncomingCall call: SinchRTC.SinchCall) { + call.delegate = self + // Store/match call.callId with your CallKit UUID mapping + } +} +``` + +**Without CallKit** (e.g. testing or custom UI): +```swift +extension SinchClientMediator: SinchCallClientDelegate { + func client(_ client: SinchRTC.SinchCallClient, + didReceiveIncomingCall call: SinchRTC.SinchCall) { + call.delegate = self + // Present UI for call + } +} +``` + +#### Receiving Calls from PSTN or SIP (Phone-to-App / SIP-to-App) + +The Sinch SDK supports receiving incoming calls that originate from the PSTN (regular phone network) or from SIP endpoints. +When a call arrives at a Sinch voice number or via SIP origination, the Sinch platform triggers an **Incoming Call Event (ICE)** callback to your backend. +The platform can then route this call to an in-app user by responding with the `connectMxp` SVAML action. + +##### Prerequisites + +1. Rent a voice number from the [Sinch Build Dashboard](https://dashboard.sinch.com/numbers/overview) and assign it to your application, OR configure SIP origination for your application +2. Configure a callback URL in your app's Voice settings where Sinch will send call-related events +3. Implement the ICE callback handler in your backend to route calls to the appropriate app user + +##### Backend Implementation + +When a PSTN or SIP call comes in, respond to the ICE callback with a `connectMxp` action to route the call to an app user: + +```json +{ + "action": { + "name": "connectMxp", + "destination": { + "type": "username", + "endpoint": "target-user-id" + } + } +} +``` + +#### Answer incoming call + +To answer the call, use `call.answer()`: + +```swift +call.answer() +``` + +#### Decline incoming call + +If the call shouldn't be answered, use `call.hangup()` to decline. The caller is notified that the incoming call was denied. + +```swift +call.hangup() +``` + +### Disconnecting a call +When the user wants to disconnect an ongoing call, use `call.hangup()`. Either party can disconnect. + +```swift +call.hangup() +``` + +When either party disconnects, `callDidEnd(_:)` is called on the delegate: + +```swift +func callDidEnd(_ call: SinchCall) { + // Update UI, e.g., dismiss the call screen +} +``` + +A call can be disconnected before it has been completely established. + +## Video Calling + +Video calls follow the same flow as audio calls. Use `callClient.callUserVideo(withId:)` to start a video call. You receive the same callbacks: `callDidProgress`, `callDidAnswer`, `callDidEstablish`. + +### Showing the video streams + +Assuming a view controller with two `UIView` outlets for remote and local video: + +**Local video preview:** +```swift +override func viewDidLoad() { + super.viewDidLoad() + + guard let videoController = sinchClient?.videoController else { return } + localVideoView.addSubview(videoController.localView) +} +``` + +**Remote video stream** (attach when the remote track arrives): +```swift +func callDidAddVideoTrack(_ call: SinchCall) { + guard let videoController = sinchClient?.videoController else { return } + remoteVideoView.addSubview(videoController.remoteView) +} +``` + +#### Pausing and resuming a video stream + +Use `call.pauseVideo()` to temporarily stop sending local video and `call.resumeVideo()` to resume. Audio continues unless you mute it separately. + +```swift +call.pauseVideo() +call.resumeVideo() +``` + +The call delegate is notified via `callDidPauseVideoTrack(_:)` and `callDidResumeVideoTrack(_:)`. + +### Switching camera (front/back) +```swift +guard let videoController = sinchClient?.videoController else { return } +videoController.captureDevicePosition.toggle() +``` + +### Video content fitting +Control how rendered video fits a view via `UIView.contentMode`. Only `.scaleAspectFit` and `.scaleAspectFill` are respected. + +```swift +videoController.remoteView.contentMode = .scaleAspectFill +``` + +### Full screen mode +The SDK provides `UIView` extension methods for fullscreen transitions: + +```swift +if view.sinIsFullscreen() { + view.contentMode = .scaleAspectFit + view.sinDisableFullscreen(true) +} else { + view.contentMode = .scaleAspectFill + view.sinEnableFullscreen(true) +} +``` + +### Incoming video call +An incoming video call triggers `client(_:didReceiveIncomingCall:)` like a voice call. Check `call.details.isVideoOffered` to determine if video is included. + +## Miscellaneous + +### Minimum requirements +iOS **12.0** minimum deployment target. + +### User ID restrictions +User IDs must not be longer than 255 bytes and must only contain URL-safe characters: `ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghjiklmnopqrstuvwxyz0123456789-_=`. Consider base64url-encoding IDs that contain other characters. + +### Call details +The `SinchCallDetails` class holds metadata about a call including timestamps (`startedTime`, `progressedTime`, `rungTime`, `answeredTime`, `establishedTime`, `endedTime`), end cause (`.timeout`, `.denied`, `.noAnswer`, `.error`, `.hungUp`, `.canceled`, `.otherDeviceAnswered`, etc.), and error information. + +```swift +func callDidEnd(_ call: SinchCall) { + if call.details.endCause == .error { + if let error = call.details.error { + print("Call failed: \(error.localizedDescription)") + } + } +} +``` + +### Audio session +During calls, the SDK manages `AVAudioSession`. It sets the category to `.playAndRecord` with mode `.voiceChat` at the start of a call and restores original settings when the call ends. + +If using CallKit/LiveCommunicationKit, forward audio session activation events to the SDK (see CallKit integration section above). + +To override audio session category options: +```swift +sinchClient?.audioController.setAudioSessionCategoryOptions([.allowBluetooth, .allowBluetoothA2DP, .defaultToSpeaker]) +``` \ No newline at end of file diff --git a/skills/in-app-calling/references/js.md b/skills/in-app-calling/references/js.md new file mode 100644 index 0000000..3cea710 --- /dev/null +++ b/skills/in-app-calling/references/js.md @@ -0,0 +1,388 @@ + +# Sinch Javascript Voice and Video SDK + +## Add the Sinch library +You can include Sinch in several ways. Pick the one that fits your setup. + +### Host with your web app +Host the library with your site and include it: +```html + +``` + +### Load from Sinch CDN +```html + +``` + +## Integration steps + +Every integration must follow these steps in order. Do not skip or reorder them. + +1. **Create a SinchClient** - Build the client with application key, environment host, and user ID. +2. **Add a client listener** - Attach a `SinchClientListener` to handle `onClientStarted`, `onClientFailed`, and `onCredentialsRequired`. +3. **Authorize the client** - Implement `onCredentialsRequired` to supply a signed JWT. Decide whether to sign client-side (prototyping only) or fetch from a backend. +4. **Enable push notifications** - Call `sinchClient.setSupportManagedPush()` before `start()`. Provide a service worker (`sw.js`) that forwards push payloads to open tabs. Required for app-to-app calls, even as the caller. +5. **Start the client** - Call `sinchClient.start()`. Wait for `onClientStarted` before proceeding. +6. **Add a CallClientListener** - Attach a listener to `sinchClient.callClient` to detect incoming calls via `onIncomingCall`. +7. **Add a CallListener to each call** - On every outgoing or incoming call, attach a `CallListener` to handle `onCallProgressing`, `onCallRinging`, `onCallAnswered`, `onCallEstablished`, and `onCallEnded`. + +When helping a user integrate, walk through these steps one at a time. Confirm each step is in place before moving to the next. + + + +## Sinch Client +The *SinchClient* is the Sinch SDK entry point. It's used to configure the user’s and device’s capabilities, as well as to provide access to feature classes such as the *CallController*, *AudioController*, and *VideoController*. + +### Create a *SinchClient* +```javascript +const sinchClient = Sinch.getSinchClientBuilder() + .applicationKey("") + .environmentHost("ocra.api.sinch.com") + .userId("") + .build(); +``` +- The *Application Key* is obtained from the [Sinch Developer Dashboard - Apps](https://dashboard.sinch.com/voice/apps). +- The *User ID* should uniquely identify the user. +- The term *Ocra* in the hostname `ocra.api.sinch.com` is the name of the Sinch API that SDK clients target. + + +### Start the Sinch client +Before starting, add a client listener (see [SinchClientListener reference](https://download.sinch.com/js/latest/reference/interfaces/SinchClientListener.html)): + +```javascript +sinchClient.addListener({ + onClientStarted: (client) => { + // sinch client started successfully + }, + onClientFailed: (client, error) => { + // sinch client start failed + }, + onCredentialsRequired: (client, clientRegistration) => { + // registration required, get jwt token for user and register + } +}); +sinchClient.start(); +``` + +### Authentication & authorization +When `SinchClient` starts with a given user ID, you must provide an authorization token (JWT) to register with Sinch. +Implement `SinchClientListener.onCredentialsRequired()` and supply a JWT signed with the Application Secret. + +Read https://developers.sinch.com/docs/in-app-calling/js/application-authentication.md + +For production application it is recommended to generate and sign the JWT token on the backend server, then send it over a secure channel to the application and Sinch client running on the device. + +Ask the user whether it can be embedded. +If it can be embedded: +A sample JWT helper is provided at `assets/jwt-helper.js`. Read and adapt it. +```javascript +onCredentialsRequired(sinchClient, clientRegistration) { + try { + const jwt = new JWT(CONFIG.applicationKey, CONFIG.applicationSecret, this.userId); + const token = await jwt.toJwt(); + clientRegistration.register(token); + } catch (error) { + console.log('Authentication failed: ' + error.message); + clientRegistration.registerFailed(); + } + } +``` +If it can not be embedded: +Implement the required functionality on your backend and fetch a signed registration token when required. + +```javascript +onCredentialsRequired: (sinchClient, clientRegistration) => { + yourAuthServer + .getRegistrationToken(sinchClient.userId) + .then((token) => { + clientRegistration.register(token); + }) + .catch(() => { + clientRegistration.registerFailed(); + }); +}; +``` + +### Push notifications +To receive incoming calls in the browser via W3C Web Push API, enable managed push: + +```javascript +// Enable managed push with the default Service Worker path (sw.js) +sinchClient.setSupportManagedPush(); +``` + +By default, Sinch expects a Service Worker file at `sw.js` in your current directory. +If you use a custom path or filename, pass it explicitly: + +```javascript +sinchClient.setSupportManagedPush(""); +``` + +Add a basic Service Worker that forwards push payloads to open tabs. +This lets `SinchClient` process the payload and trigger `CallClientListener.onIncomingCall` when a new call arrives. + +```javascript +// sw.js (basic example) +self.addEventListener("push", (event) => { + const body = event.data?.json?.() ?? null; + event.waitUntil( + self.clients + .matchAll({ includeUncontrolled: true, type: "window" }) + .then((clients) => { + clients.forEach((client) => { + client.postMessage({ + visible: client.visibilityState === "visible", + data: body, + }); + }); + }) + ); +}); +``` + +For a more advanced `sw.js` that handles mobile browser limitations, see the reference app: +[voicecall/sw.js](https://github.com/sinch/rtc-reference-applications/blob/master/javascript/samples/voicecall/sw.js) + +When managed push is enabled and your Service Worker forwards messages as above, incoming calls will be delivered to `onIncomingCall` in [CallClientListener](https://download.sinch.com/js/latest/reference/interfaces/CallClientListener.html#onincomingcall-1). + +Note: +For use cases requiring only outgoing App-to-Phone, App-to-SIP, or App-to-Conference calls, calling `sinchClient.setSupportManagedPush` is not required. You can place these calls directly once the Sinch client is started. + +## Voice calling +The Sinch SDK supports four types of calls: *app-to-app (audio or video)*, *app-to-phone*, *app-to-sip* and *conference* calls. The *CallClient* is the entry point for the calling functionality of the Sinch SDK. +Calls are placed through the *CallClient* and events are received using the *CallClientListener*. The call client is owned by the SinchClient and accessed using `sinchClient.callClient`. + +### Set up an *App-to-App* call +Use the [CallClient.callUser()](https://download.sinch.com/js/latest/reference/interfaces/CallClient.html#calluser) method so Sinch can connect the call to the callee. + +```javascript +var callClient = sinchClient.callClient; +var call = callClient.callUser(''); +// Video call: +// var call = callClient.callUserVideo(''); +call.addListener(...); +``` +The returned call object includes participant details, start time, state,and possible errors. + +Assuming the callee’s device is available, +[CallListener.onCallProgressing()](https://download.sinch.com/js/latest/reference/interfaces/CallListener.html#oncallprogressing) +is invoked. If you play a progress tone, start it here. +When the callee receives the call, +[CallListener.onCallRinging()](https://download.sinch.com/js/latest/reference/interfaces/CallListener.html#oncallringing) +fires. This indicates the callee’s device is ringing. +When the other party answers, +[CallListener.onCallAnswered()](https://download.sinch.com/js/latest/reference/interfaces/CallListener.html#oncallanswered) +is called. Stop any progress tone. +Once full audio connectivity is established, +[CallListener.onCallEstablished()](https://download.sinch.com/js/latest/reference/interfaces/CallListener.html#oncallestablished) +is emitted. Users can now talk. + +Typically, connectivity is already established when the call is answered, so +`onCallEstablished` may follow immediately after `onCallAnswered`. +On poor networks, it can take longer—consider showing a “connecting” +indicator. + +**Important:** +For *App-to-App* calls, you must enable managed push by calling `sinchClient.setSupportManagedPush()`, even if you are the caller. See the full setup in [Push notifications](/docs/in-app-calling/js/push-notifications). + +### Set up an *App-to-Phone* call + +An *app-to-phone* call is a call that's made to a phone on the regular telephone network. Setting up an *app-to-phone* call is not much different than setting up an *app-to-app* call. Instead of invoking the `callUser` method, invoke the [CallClient.callPhoneNumber()](https://download.sinch.com/js/latest/reference/interfaces/CallClient.html#callphonenumber) method and pass an E.164 number prefixed with `+`. For example, to call the US phone number 415 555 0101, specify `+14155550101`. The `+` is the required prefix with the US country code `1` prepended to the local subscriber number. + +#### Presenting a number to the destination you are calling + +Mandatory step! +You must provide a CLI (Calling Line Identifier) or your call will fail. +You need a number from Sinch so you can provide a valid CLI to the handset you are calling. +Specify your CLI during building SinchClient by using [callerIdentifier](https://download.sinch.com/js/latest/reference/interfaces/SinchClientBuilder.html#calleridentifier-1) method. + +Note: When your account is in trial mode, you can only call your [verified numbers](https://dashboard.sinch.com/numbers/verified-numbers). If you want to call any number, you need to upgrade your account! + +### Set up an *App-to-sip* call + +An *app-to-sip* call is a call that's made to a SIP server. Setting up an *app-to-sip* call is not much different from setting up an *app-to-app* call. Instead of invoking the `callUser` method, invoke [CallClient.callSip()](https://download.sinch.com/js/latest/reference/interfaces/CallClient.html#callsip). The SIP identity should be in the form `user@server`. By convention, when passing custom headers in the SIP call, the headers should be prefixed with `x-`. If the SIP server reports any errors, the `CallDetails` object will provide an error with the `SIP` error type. + +### Set up a *Conference* call +A *conference* call can be made to connect a user to a conference room where multiple users can be connected at the same time. The identifier for a conference room may not be longer than 64 characters. + +```javascript +var callClient = sinchClient.callClient; +var call = callClient.callConference(''); +call.addListener(...); +``` + +### Selecting audio input device + +To select a specific audio input device, set the `deviceId` constraint via +[CallClient.setAudioTrackConstraints()](https://download.sinch.com/js/latest/reference/interfaces/CallClient.html#setaudiotrackconstraints). + + +```javascript +sinchClient.callClient.setAudioTrackConstraints({ + deviceId: { exact: }, +}); +``` + +For a full sample of input/output device selection, see the +[reference app](https://github.com/sinch/rtc-reference-applications/tree/master/javascript/samples). + +### Handle incoming calls + +To answer calls, the application must be notified when the user receives an incoming call. + +Add a [CallClientListener](https://download.sinch.com/js/latest/reference/interfaces/CallClientListener.html) to `CallClient` to act on incoming calls. When a call arrives, [CallClientListener.onIncomingCall()](https://download.sinch.com/js/latest/reference/interfaces/CallClientListener.html#onincomingcall) is executed. + + +```javascript +var callClient = sinchClient.callClient; +callClient.addListener(...); +``` + +When the incoming call method is executed, the call can either be connected automatically without any user action, or it can wait for the user to press the answer or the hangup button. If the call is set up to wait for a user response, we recommended that a ringtone is played to notify the user that there is an incoming call. + + +```javascript +onIncomingCall = (callClient, call) => { + // Start playing ringing tone + ... + + // Add call listener + call.addListener(...); +} +``` + +To get events related to the call, add a call listener. The call object contains details about participants, start time, potential error codes, and error messages. + +#### Receiving Calls from PSTN or SIP (Phone-to-App / SIP-to-App) + +The Sinch SDK supports receiving incoming calls that originate from the PSTN (regular phone network) or from SIP endpoints. +When a call arrives at a Sinch voice number or via SIP origination, the Sinch platform triggers an **Incoming Call Event (ICE)** callback to your backend. +Our platform can then route this call to an in-app user by responding with the `connectMxp` SVAML action. + +##### Prerequisites + +1. Rent a voice number from the [Sinch Build Dashboard](https://dashboard.sinch.com/numbers/overview) and assign it to your application, OR configure SIP origination for your application +2. Configure a callback URL in your app's Voice settings where Sinch will send call-related events +3. Implement the ICE callback handler in your backend to route calls to the appropriate app user + + +##### Backend Implementation + +When a PSTN or SIP call comes in, respond to the ICE callback with a `connectMxp` action to route the call to an app user: + +```json +{ + "action": { + "name": "connectMxp", + "destination": { + "type": "username", + "endpoint": "target-user-id" + } + } +} +``` + +#### Incoming video call + +When an incoming call is a video call, the [CallClientListener.onIncomingCall()](https://download.sinch.com/js/latest/reference/interfaces/CallClientListener.html) section for details on how to add video views. + +#### Answer incoming call + +To answer the call, use the +[Call.answer()](https://download.sinch.com/js/latest/reference/interfaces/Call.html#answer) +method on the call to accept it. If a ringtone was previously played, it +should be stopped now. + +User presses the answer button: + + +```javascript +// User answers the call +call.answer(); + +// Stop playing ringing tone +``` + +#### Decline incoming call + +If the call shouldn't be answered, use +[Call.hangup()](https://download.sinch.com/js/latest/reference/interfaces/Call.html#hangup) +on the call to decline. The caller is notified that the incoming call was +denied. If a ringtone was previously played, it should be stopped now. + +User presses the hangup button: + +```javascript +// User doesn't want to answer +call.hangup(); + +// Stop playing ringing tone +... +``` + +### Disconnecting a Call +When the user wants to disconnect an ongoing call, use [Call.hangup()](https://download.sinch.com/js/latest/reference/interfaces/Call.html#hangup) method. Either user taking part in a call can disconnect it. +Hanging up a call: + +```javascript +call.hangup(); +``` + +When either party disconnects a call, the application is notified using the +call listener method +[CallListener.onCallEnded()](https://download.sinch.com/js/latest/reference/interfaces/CallListener.html#oncallended). +This allows the user interface to be updated, an alert tone to be played, or +similar actions to occur. + +A call can be disconnected before it has been completely established. + +Hanging up a connecting call: + +```javascript +// Starts a call asynchronously +var call = callClient.callUser(''); + +// User changed his/her mind, let’s hangup +call.hangup(); +``` + +## Video Calling + +Set up video calls with the Sinch JavaScript Voice and Video SDK. + +### Setting up a video call + +Video calls follow the same flow as audio calls. +Use [CallClient.callUserVideo()](https://download.sinch.com/js/latest/reference/interfaces/CallClient.html#calluservideo) to start a video call with a specific user. +You’ll receive the same callbacks as for audio: progressing, ringing, answered, and established. For lifecycle details, see [Audio calling](/docs/in-app-calling/js/calling). + +### Showing the video streams +Once you have a `Call`, you can access two MediaStreams: +- `incomingStream` — remote stream sent to your app +- `outgoingStream` — your local stream sent to the other party + +Access them via [Call.incomingStream](https://download.sinch.com/js/latest/reference/interfaces/Call.html#incomingstream) and [Call.outgoingStream](https://download.sinch.com/js/latest/reference/interfaces/Call.html#outgoingstream). +Bind the appropriate stream to a `