Automate matched real-call RTCP experiments

README.md

@@ -244,28 +244,58 @@ speaker-to-microphone echo. The bridge records local queue disposal and provider
 chunk correlation, but `remote_playout_observed: false` means it cannot claim
 exactly what the caller heard without a caller-side recording.
 
-### 7. Collect RTCP during the next call
+### 7. Run the matched real-call experiment with RTCP
 
 An empty **RTP quality** panel does not prove the network was clean. It means no
-normalized RTP/RTCP evidence reached that run. Start the collector in terminal 4
-while the real call is active, using the new run ID:
+normalized RTP/RTCP evidence reached that run. For a matched experiment, run one
+condition at a time. The bridge automatically binds the read-only AMI collector
+to the run ID created for each call.
+
+First use a headset and let Gemini finish without speaking over it:
+
+```fish
+./scripts/asterisk-local gemini \
+  --collect-rtcp \
+  --experiment-condition no-interruption
+```
+
+Call `7000`, keep the call active for 20–30 seconds, hang up, and stop the
+bridge. Then repeat with one deliberate interruption at a repeatable point:
+
+```fish
+./scripts/asterisk-local gemini \
+  --collect-rtcp \
+  --experiment-condition intentional-barge-in
+```
+
+The local helper supplies the loopback-only AMI account from the Asterisk
+development configuration. When invoking the CLI directly, keep credentials in
+environment variables:
 
 ```fish
 set -gx VOXBENCH_AMI_USERNAME voxbench-rtcp
 set -gx VOXBENCH_AMI_SECRET voxbench-ami-local-only
 
-voxbench asterisk-ami-rtcp \
-  --run-id '<run-id>' \
+voxbench audiosocket-realtime \
+  --provider gemini-live \
   --control-plane-url http://127.0.0.1:8001 \
-  --host 127.0.0.1 \
-  --port 5038 \
-  --clock-rate-hz 8000
+  --collect-rtcp \
+  --ami-host 127.0.0.1 \
+  --ami-port 5038 \
+  --ami-clock-rate-hz 8000 \
+  --experiment-condition no-interruption
 ```
 
-Keep the call active for 20–30 seconds so Asterisk has time to emit RTCP
-reports. Aggregate RTCP can show loss, jitter, and RTT; it cannot identify an
-exact missing RTP sequence number. Use the library packet-observation adapter
-when packet-level proof is required.
+Keep only one test call active. AMI reports aggregate Asterisk RTCP events and
+does not retain the Channel identifier, so concurrent calls cannot be safely
+attributed to separate runs. Aggregate RTCP can show loss, jitter, and RTT; it
+cannot identify an exact missing RTP sequence number. Use the library
+packet-observation adapter when packet-level proof is required.
+
+For an audible-quality conclusion, retain a caller-side reference recording.
+The simplest independent setup is a second device recording the headset output;
+label it with the run ID printed by the bridge. VoxBench stage WAVs prove the
+local processing path, but not what the remote softphone ultimately played.
 
 ## Local demo troubleshooting
 
@@ -282,7 +312,7 @@ when packet-level proof is required.
 | **Fetch** or the circular refresh button appears to do nothing | Refreshing the same completed run does not create new evidence or change immutable recordings. Select a different run, keep an active call running, or create a new call. |
 | **Readiness** shows unchecked/incomplete items | Readiness is an evidence checklist, not an automatic failure count. A live bridge can complete while deployment-specific checklist fields remain unknown. |
 | Audio is choppy or clicks | Compare all four stage WAVs, click the barge-in incidents, repeat once with a headset and no overlap, then repeat with RTCP collection. Do not attribute the symptom to packet loss until transport evidence exists. |
-| **RTP quality** is empty | Start `asterisk-ami-rtcp` during the active call and keep the call long enough for RTCP. No points means unobserved transport quality, not confirmed zero loss. |
+| **RTP quality** is empty | Restart the Gemini bridge with `--collect-rtcp`, keep exactly one call active, and keep it connected long enough for RTCP. The standalone `asterisk-ami-rtcp` command remains available for manual collection. No points means unobserved transport quality, not confirmed zero loss. |
 
 Detailed operator and library integration references are available in
 [`docs/demo-live-softphone.md`](docs/demo-live-softphone.md) and

docs/demo-live-softphone.md

@@ -296,25 +296,65 @@ export VOXBENCH_AMI_USERNAME=voxbench-rtcp
 export VOXBENCH_AMI_SECRET='REPLACE_WITH_LOCAL_SECRET'
 ```
 
-Start the AudioSocket bridge and place a call. Copy the run ID printed by the
-bridge, then start the collector in a third terminal while that run is active:
+The recommended local workflow attaches the collector automatically when each
+AudioSocket run is created:
 
-```bash
-voxbench asterisk-ami-rtcp \
-  --run-id '<run_id>' \
-  --host 127.0.0.1 \
-  --port 5038 \
-  --clock-rate-hz 8000
+```fish
+./scripts/asterisk-local gemini \
+  --collect-rtcp \
+  --experiment-condition no-interruption
 ```
 
-`--clock-rate-hz` is the RTP codec clock, not necessarily an audio output sample
-rate. Use 8000 for PCMU/PCMA and configure the actual clock for other codecs. Stop
-the collector when the call/run ends. The Web RTP quality panel shows direction,
-jitter, loss, RTT, MOS when separately supplied, and each point's relative time.
-Live preview also shows an RTP collector block: `connected` after AMI login,
-`collecting` after the first normalized RTCP event, and `failed` after a safe
-collector failure observation. Its event count is additive across collector
-restarts for the same run.
+The helper supplies the development AMI credentials through environment
+variables. The bridge starts one collector after provider connection succeeds,
+binds it to the new run ID, and cancels it before that run is completed. Collector
+failure is reported as operational evidence but does not fail an otherwise valid
+voice call.
+
+To invoke the bridge directly, set `VOXBENCH_AMI_USERNAME` and
+`VOXBENCH_AMI_SECRET`, then pass `--collect-rtcp`, `--ami-host`, `--ami-port`, and
+`--ami-clock-rate-hz`. The clock rate is the RTP codec clock, not necessarily an
+audio output sample rate. Use 8000 for PCMU/PCMA and configure the actual clock
+for other codecs.
+
+The standalone `voxbench asterisk-ami-rtcp --run-id ...` command remains
+available for integrations that manage run and collector lifecycles separately.
+
+The Web RTP quality panel shows direction, jitter, loss, RTT, MOS when separately
+supplied, and each point's relative time. Live preview also shows an RTP collector
+block: `connected` after AMI login, `collecting` after the first normalized RTCP
+event, and `failed` after a safe collector failure observation. Its event count
+is additive across collector restarts for the same run.
+
+Keep only one call active while automatic collection is enabled. The privacy-safe
+collector intentionally drops Asterisk Channel, address, and SSRC fields, so AMI
+events from concurrent calls cannot be attributed safely.
+
+## Matched Choppy-Audio Experiment
+
+Use identical provider model, codec, route, gain, noise floor, caller phrase, and
+call duration for both conditions.
+
+1. Use a headset to prevent speaker audio from re-entering the microphone.
+2. Start `no-interruption` with automatic RTCP collection, call `7000`, speak one
+   fixed prompt, and remain silent until Gemini finishes.
+3. Keep the call active for 20–30 seconds, hang up, and retain the printed run ID.
+4. Restart the bridge with `--experiment-condition intentional-barge-in`.
+5. Repeat the same prompt, but interrupt Gemini once at the same relative phrase.
+6. Retain an independent caller-side recording for each call and name it with the
+   corresponding run ID.
+7. Select the no-interruption run as **Primary** and the interruption run as
+   **Compare**.
+
+Confirm all of the following before attributing choppy audio:
+
+- **RTP quality** contains received or sent points rather than remaining empty.
+- The experiment condition is present in the run environment tags/operator note.
+- The same provider model appears on both runs.
+- Stage WAVs identify the first local stage containing the audible defect.
+- Barge-in evidence explains any queued audio disposal.
+- The caller-side recording confirms that the candidate instant was actually
+  audible after Asterisk RTP and softphone playout.
 
 Relevant Asterisk references:
 

scripts/asterisk-local

@@ -133,6 +133,7 @@ case "$command" in
     shift
     gemini_model="${VOXBENCH_GEMINI_MODEL:-gemini-3.1-flash-live-preview}"
     previous_arg=""
+    collect_rtcp=0
     gemini_args=()
     for arg in "$@"; do
       if [[ "$previous_arg" == "--model" ]]; then
@@ -147,6 +148,10 @@ case "$command" in
         --model=*)
           gemini_model="${arg#--model=}"
           ;;
+        --collect-rtcp)
+          collect_rtcp=1
+          gemini_args+=("$arg")
+          ;;
         *)
           gemini_args+=("$arg")
           ;;
@@ -204,6 +209,12 @@ EOF
     echo "Starting Gemini Live bridge on 127.0.0.1:9019."
     echo "Model: $gemini_model"
     echo "Control Plane: $CONTROL_PLANE_URL"
+    if [[ "$collect_rtcp" == "1" ]]; then
+      export VOXBENCH_AMI_USERNAME="${VOXBENCH_AMI_USERNAME:-voxbench-rtcp}"
+      export VOXBENCH_AMI_SECRET="${VOXBENCH_AMI_SECRET:-$AMI_PASSWORD}"
+      echo "Asterisk RTCP auto-collection: enabled on 127.0.0.1:$AMI_PORT"
+      echo "Keep only one test call active so aggregate AMI events stay attributable."
+    fi
     echo "Call 7000 after the listening message appears. Press Ctrl+C to stop."
     exec "$ROOT/.venv/bin/voxbench" audiosocket-realtime \
       --control-plane-url "$CONTROL_PLANE_URL" \

src/voxbench/cli/main.py

@@ -15,6 +15,7 @@
 from voxbench.live_demo.observed_run import (
     LiveDemoProvider,
     build_audiosocket_observed_run_payload,
+    normalize_experiment_condition,
 )
 from voxbench.observability import HttpObservationTransport, VoxBenchObserver
 from voxbench.realtime_providers import (
@@ -163,6 +164,34 @@ def audiosocket_realtime(
         float,
         typer.Option("--connect-backoff-seconds", min=0.0, max=30.0),
     ] = 0.5,
+    experiment_condition: Annotated[
+        str | None,
+        typer.Option(
+            "--experiment-condition",
+            help="Safe alias recorded on each run, such as no-interruption.",
+        ),
+    ] = None,
+    collect_rtcp: Annotated[
+        bool,
+        typer.Option(
+            "--collect-rtcp",
+            help="Attach aggregate Asterisk AMI RTCP observations to each call run.",
+        ),
+    ] = False,
+    ami_host: Annotated[str, typer.Option("--ami-host")] = "127.0.0.1",
+    ami_port: Annotated[int, typer.Option("--ami-port", min=1, max=65535)] = 5038,
+    ami_clock_rate_hz: Annotated[
+        int,
+        typer.Option("--ami-clock-rate-hz", min=1, max=384_000),
+    ] = 8_000,
+    ami_username_env: Annotated[
+        str,
+        typer.Option("--ami-username-env"),
+    ] = "VOXBENCH_AMI_USERNAME",
+    ami_secret_env: Annotated[
+        str,
+        typer.Option("--ami-secret-env"),
+    ] = "VOXBENCH_AMI_SECRET",
 ) -> None:
     """Bridge Asterisk AudioSocket PCM to a realtime AI provider."""
 
@@ -178,6 +207,30 @@ def audiosocket_realtime(
         detail = f"set {env_vars} and install the live extra: pip install -e '.[live]'"
         raise typer.BadParameter(detail)
 
+    if experiment_condition is not None:
+        try:
+            experiment_condition = normalize_experiment_condition(experiment_condition)
+        except ValueError as exc:
+            raise typer.BadParameter(str(exc)) from None
+
+    ami_username: str | None = None
+    ami_secret: str | None = None
+    if collect_rtcp:
+        ami_username = os.environ.get(ami_username_env)
+        ami_secret = os.environ.get(ami_secret_env)
+        missing = [
+            name
+            for name, value in (
+                (ami_username_env, ami_username),
+                (ami_secret_env, ami_secret),
+            )
+            if not value
+        ]
+        if missing:
+            raise typer.BadParameter(
+                f"set required environment variable(s): {', '.join(missing)}"
+            )
+
     transport = HttpObservationTransport(control_plane_url)
 
     async def create_session(call_uuid: UUID) -> RealtimeCallSession:
@@ -190,6 +243,7 @@ async def create_session(call_uuid: UUID) -> RealtimeCallSession:
             noise_floor=noise_floor,
             mode="provider",
             model=selected_model,
+            experiment_condition=experiment_condition,
         )
         run = await asyncio.to_thread(transport.start_run, payload)
         observer = VoxBenchObserver(run["run_id"], transport)
@@ -245,6 +299,40 @@ def report_retry(failed_attempt: int, delay: float) -> None:
             f"AudioSocket call {call_id} -> {provider}/{selected_model} "
             f"-> run {run['run_id']}"
         )
+        if experiment_condition is not None:
+            typer.echo(f"Experiment condition: {experiment_condition}")
+        background_tasks: tuple[asyncio.Task[object], ...] = ()
+        if collect_rtcp:
+            assert ami_username is not None
+            assert ami_secret is not None
+            rtcp_observer = VoxBenchObserver(run["run_id"], transport)
+            collector = AmiRtcpCollector(
+                host=ami_host,
+                port=ami_port,
+                username=ami_username,
+                secret=ami_secret,
+                clock_rate_hz=ami_clock_rate_hz,
+            )
+
+            async def collect_call_rtcp() -> None:
+                try:
+                    await collector.collect(rtcp_observer)
+                except asyncio.CancelledError:
+                    raise
+                except AmiError as exc:
+                    rtcp_observer.observe_metric("asterisk_ami_rtcp_failures", 1.0)
+                    with suppress(Exception):
+                        await asyncio.to_thread(rtcp_observer.flush)
+                    typer.echo(
+                        f"Asterisk RTCP collection failed for run {run['run_id']}: {exc}",
+                        err=True,
+                    )
+
+            background_tasks = (asyncio.create_task(collect_call_rtcp()),)
+            typer.echo(
+                f"Asterisk RTCP collection attached to run {run['run_id']} "
+                f"on {ami_host}:{ami_port}"
+            )
         return RealtimeCallSession(
             call_id=call_id,
             observer=observer,
@@ -257,6 +345,7 @@ def report_retry(failed_attempt: int, delay: float) -> None:
             target_rms=target_rms,
             max_gain=max_gain,
             noise_floor=noise_floor,
+            background_tasks=background_tasks,
         )
 
     server = AudioSocketRealtimeServer(

src/voxbench/live_demo/observed_run.py

@@ -3,6 +3,7 @@
 from __future__ import annotations
 
 import json
+import re
 from copy import deepcopy
 from pathlib import Path
 from typing import Any, Literal
@@ -18,6 +19,19 @@
     "manifests/processor/limiter.json",
     "manifests/processor/serializer.json",
 )
+EXPERIMENT_CONDITION_PATTERN = re.compile(r"[a-z0-9][a-z0-9._-]{0,63}")
+
+
+def normalize_experiment_condition(value: str) -> str:
+    """Return a safe, stable alias for a matched live-call condition."""
+
+    normalized = value.strip().lower()
+    if EXPERIMENT_CONDITION_PATTERN.fullmatch(normalized) is None:
+        raise ValueError(
+            "experiment_condition must be a 1-64 character lowercase alias using "
+            "letters, digits, dot, underscore, or dash"
+        )
+    return normalized
 
 
 def build_audiosocket_observed_run_payload(
@@ -29,6 +43,7 @@ def build_audiosocket_observed_run_payload(
     noise_floor: float,
     mode: AudioSocketDemoMode = "loopback",
     model: str | None = None,
+    experiment_condition: str | None = None,
 ) -> dict[str, Any]:
     config = deepcopy(_load_json(f"configs/live-demo-{provider}.json"))
     selected_model = model or str(config["spec"]["ai"]["model"])
@@ -57,6 +72,13 @@ def build_audiosocket_observed_run_payload(
         else "Bidirectional AudioSocket media connected to the selected provider."
     )
     provider_note = f"{provider_note} Provider model: {selected_model}."
+    tags = ["live-demo", "audiosocket", mode, provider]
+    if experiment_condition is not None:
+        experiment_condition = normalize_experiment_condition(experiment_condition)
+        provider_note = (
+            f"{provider_note} Experiment condition: {experiment_condition}."
+        )
+        tags.append(f"experiment-{experiment_condition}")
     return {
         "config_name": config["meta"]["name"],
         "configs": [config],
@@ -71,7 +93,7 @@ def build_audiosocket_observed_run_payload(
             "integration_target_alias": f"{provider}:{selected_model}:{mode}",
             "started_from": f"voxbench-audiosocket-{mode}",
             "operator_note": provider_note,
-            "tags": ["live-demo", "audiosocket", mode, provider],
+            "tags": tags,
             "secret_ref_names": (
                 ["OPENAI_API_KEY"] if provider == "openai-realtime" else ["GOOGLE_API_KEY"]
             )