Home

Creating a low-latency Microphone plugin in Unity3d for IOS.

Unity3d is a great cross-platform gaming development platform, used by some of the bigger gaming companies to create advanced 2D and 3D games. Unfortunately, it’s native support for microphone streaming is, in a word, non-existent. Streaming data from the microphone in Unity entails recording to a file, waiting until the microphone “write” position has advanced far enough that playback can start, and then starting playback of that same file. Playing back the file will start calling your own “DSP Plugin” callbacks which can be used to obtain and process the data from the microphone.

The initial version of the pitch game in Band-Blast was developed using this cross-platform method of “streaming,” but we found that, on IOS, it resulted in constant drops of audio from the microphone which became worse as the game was played. By playing around, we found that these dropouts were reduced when longer microphone clips were used, but we’d then instead get huge latency build-ups between input and output. The top suspect for these audio drops is the memory leaks in Unity 4.6’s microphone support, although it could have also been some unfixable synchronization problem caused by the multitudes of threads that Unity spawns while audio clips are being recorded (after several minutes of “recording,” XCode was showing CPU usage for Mysterious Audio Thread 120, after starting recording on Mysterious Audio Thread 32)

After trying, and failing, to resolve these problems by tweaking Unity3D’s microphone support, we finally decided to abandon it and implement, at least for IOS, using a native plugin. As for Android, performance is more reasonable (the microphone doesn’t “die” as it does on IOS) so a native microphone plugin for Android may, or may not, be forthcoming.

What You Need To Know

Plugin Building:

When building from within Unity on iOS, Unity generates an XCode project. That XCode project is then opened in XCode and built to a native IOS app. Native iOS Plugins required for the project, including source code (.m, .mm, .c, .cpp, .h, etc.) and libraries (.a) are placed in /Assets/Plugins/iOS/ and moved, when you build in Unity, to the /Libraries subdirectory. “Build Directory” is where you decide to place your build in Unity, with Unity also setting up the XCode project to build the source code and link in the libraries. All this is somewhat magical in that it just works without much setup.

Low-latency iOS audio:

Every audio developer and his dog knows that to get the lowest latency audio on iOS, you need to use the bare-metal “Audio Units” API. AudioQueue will also get you access to the microphone, but at the expense that buffers passed to the app can be upwards of 500ms of audio at a time. When using Audio Units, we want to hook an event callback up to the microphone “hardware,” and intercept the samples directly from the hardware. This callback will run in a very high priority thread, so we do not want to spend lots of time any kind of heavy processing in this callback.

Unity Threads:

Unity code is generally run under a single thread, so no synchronization is needed between different classes or UI elements. Unity3d’s low-level access for audio input and output is, howeer, handled in separate threads that Unity spawns, but we won’t be needing those. We want our analysis routines at the same priority as the Unity GUI thread, because we want them running lock-step with one another. If we run analysis at a higher priority, we could end up “overanalyzing” the incoming audio and producing more updates than the graphics can display, and if the graphics get priority, they could steal processor from the audio analysis, re-rendering the same scene with the same “input” and causing overflows in the incoming audio.

Producer Consumer Circular Buffer:

Our microphone callback is a producer, and our main Unity thread is the consumer. For super-awesome responsiveness, we used a rather small circular buffer between these two, with enough space for some extra data for between frames, but not very much. With a target frame rate of 30Hz, Unity’s default microphone sample rate of 24kHz (we don’t want to change the microphone settings in case it affects the other parts of the app that also use the microphone) and the iOS Audio Unit callback producing 256 frames per sample, the buffer needs to be at least MAX(24000/30, 256), or MAX(800, 256), or 800 frames large. However, no system is perfect, so we should probably aim for a refresh rate of 10Hz just to handle a bit of occasional lag, meaning our circular buffer should be about (24000frames/s)/10Hz = 2400 frames large.

Assuring Responsiveness by “Force Writing”

During times of extreme stress, the consumer may lag behind the producer since the producer is a high-priority audio callback called by the low-level iOS audio routines, and the consumer is a lowly graphics Update() callback. If this happens, the “typical” circular buffer implementation will just refuse to write new data since the buffer is full - but this is of course not we want since it prioritizes old audio over newer audio. Instead, we want to “force write” the audio data, moving both the read AND write pointers when writing the data, keeping the buffer “full” and effectively throwing away older data in lieu of newer data.

Consumer - Analysis

Our consumer runs from within Update(), which is called by Unity’s main thread when a new frame is being drawn to the screen - all our Update() has to do is try to read from the circular buffer, as the buffer will return the number of frames successfully read. If enough data has been read from the buffer to do another pitch analysis, then we forward the data that we just read to our native pitch detection routines, the same ones used in DaTuner and atHandTuner, and these routines, in turn, tell us which notes have been played.

chris.fogelklou@acorntechnology.se