Skip to content

native-bridge-roadmap.md

Latest commit

 

History

History
312 lines (237 loc) · 10.5 KB

native-bridge-roadmap.md

File metadata and controls

312 lines (237 loc) · 10.5 KB

Native Bridge Roadmap

OpenStudio Protocol (OSP) - A high-performance, low-latency audio networking layer inspired by AOO (Audio over OSC) but extended with rich control messages for DAW collaboration.

Protocol Overview

OSP is NOT a fork of SonoBus. It is a custom protocol that:

  • Is inspired by AOO (Audio over OSC) - the protocol SonoBus uses
  • Extends with DAW collaboration features (effects sync, multi-track, beat sync)
  • Adds Cloudflare MoQ relay for scalability beyond P2P mesh limits

Key Improvements Over SonoBus/AOO

  • Multi-track support (up to 8 tracks per user)
  • Full effects parameter synchronization
  • Master clock with sub-ms beat sync
  • Automatic P2P <-> relay switching
  • Cloudflare MoQ integration for scalability

Current Implementation Status

Component Status Notes
Audio Engine Core ✅ Done ASIO/CoreAudio device access, ringbuffers
AudioBridgeHandle ✅ Done Thread-safe handle for network ops
Opus Codec ✅ Done Full encode/decode with FEC
Jitter Buffer ✅ Done Adaptive for live jamming
Clock Sync ✅ Done NTP-style sub-ms sync
QUIC/MoQ Relay Client ✅ Done Connects to relay servers
P2P Network ✅ Done UDP receive/heartbeat loops implemented
Effects Chain ✅ Done 35 effects fully implemented (Phase 1 + 2)
Audio-Network Bridge ✅ Done Started from server on room join
Transport Forwarding ✅ Done Clock sync, tempo, transport events forwarded
Network Reconnection ✅ Done Automatic reconnection with exponential backoff
Connection Health Monitor ✅ Done Monitors relay/P2P health, auto-recovers

Effects Implementation (Phase 1 - Complete)

All 29 effects implemented with custom DSP primitives:

Base Effects (15)

  • ✅ Wah (manual/auto/envelope modes)
  • ✅ Overdrive (asymmetric soft clipping)
  • ✅ Distortion (5 types: classic, hard, fuzz, asymmetric, rectifier)
  • ✅ Amp simulation (6 types: clean, crunch, highgain, british, american, modern)
  • ✅ Cabinet simulation (EQ-based speaker/mic modeling)
  • ✅ Noise gate (state machine with attack/hold/release)
  • ✅ EQ (multi-band parametric)
  • ✅ Compressor (soft-knee dynamics)
  • ✅ Chorus (3-voice modulated delay)
  • ✅ Flanger (short modulated delay with feedback)
  • ✅ Phaser (up to 12-stage allpass cascade)
  • ✅ Delay (digital/analog/tape/pingpong types)
  • ✅ Tremolo (LFO amplitude modulation)
  • ✅ Reverb (Freeverb algorithm: 8 combs + 4 allpasses)
  • ✅ Limiter (brickwall with lookahead)

Extended Effects (14)

  • ✅ Vocal Doubler (multi-voice ADT)
  • ✅ De-esser (sidechain dynamics)
  • ✅ Bitcrusher (sample rate/bit depth reduction)
  • ✅ Ring Modulator (carrier frequency modulation)
  • ✅ Auto Pan (LFO stereo panning)
  • ✅ Multi-Filter (LP/HP/BP/Notch with modulation)
  • ✅ Vibrato (pitch modulation via variable delay)
  • ✅ Transient Shaper (attack/sustain control)
  • ✅ Stereo Imager (M/S width control)
  • ✅ Exciter (harmonic enhancement)
  • ✅ Multiband Compressor (3-band with LR4 crossovers)
  • ✅ Stereo Delay (independent L/R delays)
  • ✅ Room Simulator (early reflections + reverb)
  • ✅ Shimmer Reverb (pitch-shifted feedback)

Pitch-Based Effects (Phase 2 - Complete)

  • ✅ Pitch Correction (autocorrelation + granular shifting)
  • ✅ Harmonizer (3-voice musical interval shifting)
  • ✅ Formant Shifter (16-band vocoder-style filterbank)
  • ✅ Frequency Shifter (Bode-style Hilbert transform)
  • ✅ Granular Delay (8-grain texture/cloud generator)
  • ✅ Rotary Speaker (Leslie horn/drum simulation)

Completed Tasks Checklist

CRITICAL - Blocking Production ✅ COMPLETE

  • 1. Fix unimplemented!() Panic

    • Added new_shared() method for Arc-wrapped creation
    • Added init_self_ref() for post-Arc initialization
    • File: src/network/manager.rs

  • 2. Spawn P2P Receive/Send Loops

    • Implemented spawn_receive_loop() for UDP packet processing
    • Implemented spawn_heartbeat_loop() for peer keepalive/timeout
    • File: src/network/p2p.rs

  • 3. Start AudioNetworkBridge from Protocol Server

    • Bridge starts when user joins room via JoinRoom message
    • Bridge stops and cleans up on LeaveRoom
    • Files: src/protocol/server.rs, src/main.rs

  • 4. Wire Up Transport/Metronome Forwarding

    • Added TransportEvent enum for clock sync, tempo, transport state
    • Bridge emits transport events via broadcast channel
    • File: src/network/bridge.rs

  • 5. Expose send_packet for P2P Control Broadcast

    • Added public broadcast_control() method to P2P network
    • Manager now uses broadcast_control() for control/clock sync messages
    • Files: src/network/p2p.rs, src/network/manager.rs

HIGH PRIORITY ✅ COMPLETE

  • 6. Implement Audio Effects

    • All 29 effects fully implemented with custom DSP primitives
    • Core DSP: Biquad filters, delay lines, LFOs, envelope followers
    • Files: src/effects/*.rs (15+ files)

  • 7. Hook Mixer to Effects Chain

    • EffectsChain processes audio in correct order
    • All effects properly wired with settings and metering
    • File: src/effects/chain.rs

  • 8. Add Reconnection Logic for Relay

    • Exponential backoff reconnection implemented
    • Connection health monitoring loop
    • File: src/network/manager.rs

  • 9. Implement Error Recovery for Network Failures

    • Automatic reconnection with configurable max attempts
    • P2P quality monitoring (packet loss, RTT warnings)
    • Graceful degradation notifications
    • File: src/network/manager.rs

MEDIUM PRIORITY - Production Quality

10. Adaptive Bitrate Based on Network Conditions

  • Monitor packet loss and RTT
  • Adjust Opus bitrate dynamically

11. Packet Loss Concealment (PLC)

  • Use Opus FEC for packet loss recovery
  • Implement interpolation for missing frames

12. Master Election for Clock Sync

  • Implement MasterElection message type (OSP 0x0303)
  • Elect lowest-latency peer as clock master

13. WebSocket Heartbeat/Keepalive

  • Detect stale connections
  • Clean up resources for disconnected clients

14. Metrics/Telemetry for Latency Monitoring

  • Track RTT, jitter, packet loss per peer
  • Expose via API for UI display

LOW PRIORITY - Polish & Features

15. Effect Parameter Sync to Remote Users

  • When local user changes effect, broadcast to room

16. Chat Message Handling

  • Implement OSP message type 0x0207

17. Permission System for Track Control

  • Allow/deny remote control of tracks

18. Lyria AI State Sync

  • Sync AI-generated content state

19. Loop State Synchronization

  • Sync loop start/end/enabled state

20. Platform-Specific Audio Optimizations

  • Windows: ASIO exclusive mode
  • macOS: CoreAudio low-latency settings
  • Linux: JACK integration

TESTING - Required Before Ship

21. Integration Tests: P2P Audio Path

  • Two clients on same machine
  • Verify audio round-trip

22. Integration Tests: Relay Audio Path

  • Connect through actual MoQ relay
  • Verify audio quality

23. Stress Test: 8 Simultaneous Users

  • Maximum performer count
  • Monitor CPU, memory, latency

24. Latency Benchmarks

  • Target: <15ms RTT on LAN
  • Measure encode + network + decode + buffer

25. Network Failure Simulation

  • Packet loss, high latency, disconnects
  • Verify graceful handling

26. Cross-Platform Tests

  • Windows with ASIO
  • macOS with CoreAudio
  • Linux with ALSA/JACK

DSP Architecture

Core Primitives (src/effects/dsp.rs)

  • Biquad - Direct Form II Transposed filter
    • All standard types: LP, HP, BP, Notch, Peak, LowShelf, HighShelf, Allpass
  • DelayLine - Circular buffer with linear interpolation
  • LFO - Waveforms: sine, triangle, square, sawtooth
  • EnvelopeFollower - Attack/release dynamics
  • DcBlocker - High-pass at ~10Hz

Effect Processing Order

Wah → Overdrive → Distortion → Amp → Cabinet →
NoiseGate → DeEsser → TransientShaper →
EQ → Exciter → MultiFilter →
Compressor → MultibandCompressor →
Bitcrusher → RingMod → Chorus → Flanger → Phaser → Vibrato → Tremolo → AutoPan → VocalDoubler →
Delay → StereoDelay →
RoomSimulator → Reverb → ShimmerReverb →
StereoImager → Limiter

Effort Estimates (Updated)

Category Tasks Status
Critical blockers 5 ✅ Complete
High priority 4 ✅ Complete
Effects (Phase 1) 29 effects ✅ Complete
Effects (Phase 2) 6 pitch effects ✅ Complete
Room context integration 3 tasks ✅ Complete
Medium priority 5 In progress
Testing & polish 6 Not started

Phase 1 + 2 Complete: All 35 effects fully functional, including pitch-based effects.

Next Steps:

  • Production testing
  • Platform-specific optimizations
  • Effect parameter sync to remote users

Room State Synchronization

Architecture Overview

Room state is synchronized using Supabase Realtime (PostgreSQL NOTIFY/LISTEN), NOT the native OSP protocol. OSP handles P2P/relay audio streaming only.

What Gets Synced via Supabase Realtime

State Table/Channel Notes
Room members rooms Presence, join/leave
Musical key/scale syncedAnalysis Master broadcasts detected key
BPM syncedAnalysis Master broadcasts tempo
User tracks user_tracks Track settings, stems
Effect presets saved_tracks Saved presets (not live state)
Chat messages room_chat Text chat
World positions room_store 3D positioning

How Effects Get Room Context

  1. Frontend detects key/BPM changes (via audio analysis or user input)
  2. Master user broadcasts via broadcastAnalysis() → Supabase Realtime
  3. All clients receive update in room-store.ts → setSyncedAnalysis()
  4. Frontend sends SetRoomContext message to native bridge WebSocket
  5. Native bridge updates EffectsChain.set_room_context():
    • Pitch Correction uses key/scale for note snapping
    • Harmonizer uses key/scale for harmony intervals
    • Delay/AutoPan/StereoDelay use BPM for tempo sync

Why Separate from OSP?

  • Latency tolerance: Room state can handle 100-500ms latency
  • Reliability: Supabase provides persistence and retries
  • Scalability: Works for 100+ observers (not just 8 performers)
  • OSP focus: OSP stays lean for <15ms audio latency

OSP Protocol Messages (Audio/Timing Only)

Message Purpose
AudioFrame Opus-encoded audio data
ClockSync Sub-ms beat synchronization
Ping/Pong RTT measurement
Handshake P2P peer discovery