turn-taker — C++ conversational turn-taking engine (no models)

A single small program that holds a real two-way spoken conversation with you using no language model, no ASR, no neural VAD — nothing learned anywhere in the loop. It decides when you have finished speaking from the sound of your voice alone (six prosodic cues feeding a Bayesian cost policy), takes the floor, speaks, hears you interrupt it over its own speaker echo, stops, and remembers exactly how much of its sentence you actually heard.

This is the C++ implementation of the design in Voice Activity Research/04 The One True Pipeline.md, carrying every fix the (since-retired) Python reference build discovered on real hardware. PLAN.md is the spec; its §4 table maps each of those fixes to the code.

Build

make            # -> build/turn-taker and build/tests
make test       # runs the 110 regression pins

Needs only the Xcode command-line tools — no package manager, no dependencies to fetch. CMake is supported too, and is the better choice if you want to consume the engine as a library:

cmake -B build -S . && cmake --build build && ctest --test-dir build

Both paths are verified. First run triggers the macOS microphone permission dialog — your terminal app needs mic permission (System Settings → Privacy & Security → Microphone).

Run

./build/turn-taker                   # just talk to it
./build/turn-taker --gain 0.3        # quieter, so it is easier to interrupt
./build/turn-taker --listen-only     # never takes the floor; pure analyser
./build/turn-taker --replay f.wav    # run the engine over a recording
./build/turn-taker --analyse DIR     # why barge-in is or isn't working
./build/turn-taker --help            # all options

On start it pre-renders the agent's lines (macOS say), then plays a short chirp and measures the speaker→mic round trip — this is what lets the echo canceller work at all on laptop speakers. Then talk. Things to try, in order:

1. Say something and trail off — it answers in ~200–500 ms.
2. Pause mid-sentence (flat, loud voice) — it waits through the pause. No fixed silence threshold can do both: within-speaker pauses (470–640 ms) are longer than between-speaker gaps (110–318 ms).
3. Talk over it — within ~60 ms it ducks to a quarter volume rather than stopping, which is both a natural signal that it heard you and the thing that makes the next decision possible (it can hear you far better once it is 12 dB quieter). If you keep going it stops at ~420 ms and reports how many milliseconds of its sentence you actually heard. If you don't, it comes back up to full volume and carries on — a false alarm costs a dip, never an utterance.
4. Say "mm-hm" while it talks — it keeps the floor and keeps going.

It replies like an attentive listener: a two-second remark gets an acknowledgement, a long or clearly-finished turn gets a longer answer (which is what you need in order to practise interrupting), and cutting it off gets you a short apology worded by how much of the sentence you actually heard. It cannot respond to what you said — there is no ASR anywhere in this program, so no line it speaks assumes knowledge of your words. The timing instrumentation lives in the console (MY TURN waited 469ms (4 cues, p0 0.61)), not in the voice.

Quit with q or Ctrl-C. It writes sessions/live/:

• session.wav — stereo: left = you, right = it. Overlaps are legible.
• session_labels.txt — Audacity label track (File → Import → Labels over the WAV lays the whole decision history on the waveform).
• events.jsonl — every decision, machine-readable.
• timeline.tsv — one row per 10 ms: your activity (estimated), its own activity (exact), and the floor state. The raw material for the numbers printed at quit.
• session.rttm — the same segmentation in NIST Rich Transcription format, so diarization tooling (dscore, md-eval) can read it.
• session.json — the acoustic conditions the session ran under: the measured speaker→mic delay, ERLE, and what the room taught it about its own echo band by band. A recording that cannot describe its own conditions is one you cannot diagnose later.

Who has the floor

While it runs, three lanes scroll under the status line:

  you  |####    ..###########      ###|
  me   |    ========                  |
  floor|UUUUUSSSSSSSSSS!!!!!UUUUUUUUUU|

Your lane is estimated — shaded by how confident the VAD is, because it is inference from a microphone that also carries the agent's echo. Its lane is exact: it generated that audio and counted the samples out of the speaker. The floor lane is what the decision layer believes, and it is deliberately allowed to disagree with the other two. A mid-sentence pause shows both lanes idle while the floor still reads U, because you have not finished; a backchannel shows both lanes active while the floor stays S, because saying "mm-hm" is not a bid for the floor. Neither is a bug — that gap between what is observed and what is meant is the thing this system exists to model.

While the agent talks, the status line shows evidence rather than a level: how much of what is in the microphone its own output cannot account for. That is the quantity the interruption decision is made on, and it is the one that still means something when your voice is quieter than the echo.

At quit it prints the derived structure: occupancy (ITU-T P.59's mutual silence / single talk / double talk), IPU and overlap counts, and the two distributions that matter — gaps (silence where the other party took over) versus pauses (silence where the same party resumed). Once there are enough of them it fits mu_pause and the dead-air cost to your speech, replacing the published Swedish/English medians the policy ships with.

Practical notes

• Headphones make barge-in crisp (no acoustic echo path at all). On open speakers the interruption is decided on echo evidence rather than level, because on real hardware your voice arrives at the microphone several dB below the agent's own residual echo — measured, and the reason a level threshold cannot work there at all. Run --analyse sessions/live after a session and read the rightmost column: positive in some band means barge-in is possible in your room, negative everywhere means turn the agent down (--gain 0.3) or put headphones on.
• A noisy room with continuous music or TV reads as a user who never stops talking — correctly. Turn it off.
• Strictly full duplex. The microphone is captured unconditionally, including while the agent is speaking — there is no mute path and no half-duplex mode. Muting the mic during playback is what makes barge-in structurally impossible, and it reappearing on systems that nominally have echo cancellation is the usual reason barge-in "stops working". Overlap is resolved by cancelling the echo, never by refusing to listen.

Layout

The engine is a header-only library under include/vt/, grouped by the pipeline layers the research defines (SENSE → DESCRIBE → DECIDE → ACT). The terminal UI is not part of it — it lives with the app that uses it.

include/vt/                  the engine (header-only; include "vt/...")
├── engine.h                 pure facade: audio in, events out (replayable)
├── session.h                the live loop tying everything together
├── timeline.h               two activity lanes + the floor; invariant checks
├── analysis.h               IPUs / gaps / pauses / overlaps; RTTM export
├── core/
│   ├── config.h             every constant, with provenance
│   ├── event.h              the one currency between layers
│   └── fft.h                in-tree radix-2 real FFT
├── echo_diag.h              offline: is the canceller bad, or the room?
├── sense/
│   ├── biquad.h             the mandatory 80 Hz high-pass
│   ├── features.h           energy / ZCR / ACF / YIN-lite F0 per frame
│   ├── noise_floor.h        running-min floor, median cold-start, dual clamps
│   ├── frame_vad.h          3 room rules + Sohn HMM; evidence path while we talk
│   ├── segmenter.h          pre-roll / hangover / min-utterance state machine
│   ├── aec.h                MDF echo canceller, Geigel + NCC, delay cal
│   ├── echo_bands.h         per-band residual model: evidence + suppression
│   └── bargein.h            fusing the two detectors by measured reliability
├── describe/
│   ├── prosody.h            6 Gravano & Hirschberg yield cues, running z-scores
│   └── backchannel.h        Ward & Tsukahara rule
├── decide/
│   ├── policy.h             the cost policy (the actual brain)
│   └── fsttm.h              six-state conversational floor
├── act/
│   ├── agent.h              the rule-based responder
│   └── tts.h                macOS say, cached
└── io/
    ├── device.h             miniaudio duplex, rings, exact-truncation playback
    ├── wav.h                RIFF read/write
    └── recorder.h           Audacity label track + events.jsonl

apps/turn-taker/             the executable
├── main.cpp                 CLI, live loop driver, --replay
└── ui.h                     ANSI status line (app-only: the library never
                             depends on it)

src/miniaudio_impl.cpp       the single compiled TU (third-party impl)
tests/                       one suite per module; 110 regression pins, each
                             pinning a bug this design has already paid for
third_party/miniaudio/       vendored, v0.11.25
docs/PLAN.md                 the spec, the §4 requirements table, deviations
docs/TIMELINE.md             the voice activity timeline build plan
docs/BARGE-IN.md             why barge-in is deaf on open speakers, and the plan