feat(eot): add audio models AGT-2919

.changeset/audio-eou.md

@@ -0,0 +1,17 @@
+---
+"@livekit/agents": patch
+"@livekit/agents-plugin-silero": patch
+"@livekit/agents-plugins-livekit": patch
+---
+
+feat(core): audio end-of-turn detection with cloud → local fallback (AGT-2520)
+
+- New `inference.TurnDetector`: WebSocket cloud EOT transport (`version: 'v1'`, model name `turn-detector-v1`) with automatic fallback to the local native model (`version: 'v1-mini'`, model name `turn-detector-v1-mini`) via `@livekit/local-inference`. Auto-selects `'v1'` when `LIVEKIT_REMOTE_EOT_URL` is set, `'v1-mini'` otherwise. The `version` is the constructor knob; telemetry/billing report the full model name via `detector.model`.
+- The local EOT model runs in the shared inference process (the same `InferenceProcExecutor` the text turn detector uses), loaded once per worker host (~138 MB) instead of in every job worker. The runner is registered by default when the native binding is available, so the inference process spawns on worker startup; on platforms where the binding can't load, local EOT degrades to a positive-default prediction and the worker still starts. (This is a JS-specific divergence from Python, which keeps EOT in-process and relies on forkserver COW sharing.)
+- No prewarm helpers: EOT auto-warms in the inference process; the in-process silero VAD lazy-loads on first stream. (The `inference.prewarm*` helpers added during development were removed before release.)
+- New `inference.VAD` (local-only streaming VAD via `@livekit/local-inference`).
+- `AgentSession` now auto-provisions a bundled silero VAD when `vad` is omitted (`isDefault=true`). Pass `vad: null` to opt out.
+- `livekit-plugins-silero` is deprecated; pass `vad: null` to opt out of the bundled default, or use `inference.VAD({ model: 'silero', ... })` to customise.
+- `livekit-plugins-livekit` turn detector is deprecated in favor of `inference.TurnDetector`.
+- New `EOTInferenceMetrics` and `EOTModelUsage`; new telemetry span attributes (`lk.eou.source`, `lk.eou.from_cache`, `lk.eou.detection_delay`); new `eot_prediction` event forwarded over remote sessions.
+- Requires `@livekit/protocol` >= 1.46.5 (exposes the `AgentInference` message namespace used by the cloud transport, including the server-provided `SessionCreated` default thresholds).

MODEL_LICENSE

@@ -0,0 +1,113 @@
+LIVEKIT MODEL LICENSE AGREEMENT
+
+1. Introduction
+
+   LiveKit Incorporated ("LiveKit") is making available its proprietary models for
+   use pursuant to the terms and conditions of this Agreement. As further
+   described below, you may use these LiveKit models freely but can only use them
+   together with the LiveKit Agents framework. You cannot use the LiveKit models
+   on a standalone basis or with any other frameworks.
+
+   BY CLICKING "I ACCEPT," OR BY DOWNLOADING, INSTALLING, OR OTHERWISE ACCESSING
+   OR USING THE LIVEKIT MATERIALS, YOU AGREE THAT YOU HAVE READ AND UNDERSTOOD,
+   AND, AS A CONDITION TO YOUR USE OF THE LIVEKIT MATERIALS, YOU AGREE TO BE
+   BOUND BY, THE FOLLOWING TERMS AND CONDITIONS.
+
+2. Definitions
+
+   "Agreement" means this LiveKit Model License Agreement.
+
+   "Documentation" means the specifications, manuals, and documentation
+   accompanying any LiveKit Model and distributed by LiveKit.
+
+   "Licensee" or "you" means the individual or entity agreeing to be bound by
+   this Agreement.
+
+   "LiveKit Agents" means the proprietary LiveKit software framework for building
+   real-time multimodal AI applications with programmable backend participants.
+
+   "LiveKit Materials" means, collectively, the LiveKit Models and Documentation.
+
+   "LiveKit Model" means any of LiveKit's proprietary software models or
+   algorithms, including machine-learning software code, model weights,
+   inference-enabling software code, training-enabling software code, and
+   fine-tuning enabling software code. Any derivative works of a LiveKit Model,
+   whether developed by LiveKit, you, or any third party, will be deemed the
+   "LiveKit Model" for the purposes of this Agreement.
+
+3. License Rights
+
+   Right to Use LiveKit Materials. Subject to the terms and conditions of this
+   Agreement, including the requirements of Section 3.b, LiveKit grants you a
+   nonexclusive, nontransferable, worldwide, royalty-free license under LiveKit's
+   intellectual property rights to use, reproduce, distribute, copy, and create
+   derivative works of the LiveKit Materials.
+
+   Limitation on Use. As a condition to your use of the LiveKit Materials, you
+   agree: (i) not to use any LiveKit Models on a standalone basis or with any
+   frameworks other than LiveKit Agents; (ii) not to use any LiveKit Materials or
+   any output from, or results of using, LiveKit Models (including any derivative
+   works thereof) to improve or otherwise develop any other models that are not
+   LiveKit Models; or (iii) distribute or otherwise make available the LiveKit
+   Materials (including any derivative works thereof) except (x) pursuant to the
+   terms of this Agreement, and (y) you reproduce the above copyright notice.
+
+4. Intellectual Property
+
+   The LiveKit Materials are owned by LiveKit and its licensors. Except for the
+   rights granted to you under this Agreement, all rights are reserved and no
+   other express or implied rights are granted.
+
+   You will own any derivative works that you created from the LiveKit Materials,
+   subject to the terms of this Agreement.
+
+5. Disclaimer
+
+   UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING, LIVEKIT PROVIDES
+   THE LIVEKIT MATERIALS, AND ANY OUTPUT OR RESULTS THEREFROM, ON AN "AS IS"
+   BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED,
+   INCLUDING, WITHOUT LIMITATION, ANY WARRANTIES OR CONDITIONS OF TITLE,
+   NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. YOU
+   ARE SOLELY RESPONSIBLE FOR DETERMINING THE APPROPRIATENESS OF USING OR
+   REDISTRIBUTING THE LIVEKIT MATERIALS AND ASSUME ANY RISKS ASSOCIATED WITH YOUR
+   USE OF THE LIVEKIT MATERIALS AND ANY OUTPUT AND RESULTS.
+
+6. Limitation of Liability
+
+   IN NO EVENT AND UNDER NO LEGAL THEORY, WHETHER IN TORT (INCLUDING NEGLIGENCE),
+   CONTRACT, OR OTHERWISE, UNLESS REQUIRED BY APPLICABLE LAW (SUCH AS DELIBERATE
+   AND GROSSLY NEGLIGENT ACTS) OR AGREED TO IN WRITING, WILL LIVEKIT BE LIABLE TO
+   YOU FOR INDIRECT DAMAGES, INCLUDING ANY SPECIAL, INCIDENTAL, OR CONSEQUENTIAL
+   DAMAGES OF ANY CHARACTER ARISING AS A RESULT OF THIS AGREEMENT OR OUT OF THE
+   USE OR INABILITY TO USE THE LIVEKIT MATERIALS OR ANY OUTPUT OR RESULTS
+   THEREFROM (INCLUDING BUT NOT LIMITED TO DAMAGES FOR LOSS OF GOODWILL, WORK
+   STOPPAGE, COMPUTER FAILURE OR MALFUNCTION, OR ANY AND ALL OTHER COMMERCIAL
+   DAMAGES OR LOSSES), EVEN IF LIVEKIT HAS BEEN ADVISED OF THE POSSIBILITY OF
+   SUCH DAMAGES.
+
+7. Trademarks
+
+   This Agreement does not grant permission to use the trade names, trademarks,
+   service marks, or product names of LiveKit, except as required for reasonable
+   and customary use in describing the origin of the LiveKit Materials.
+
+8. Term and Termination
+
+   The term of this Agreement commences upon your acceptance of this Agreement
+   and continues in effect until you cease using the LiveKit Materials or it is
+   terminated by either party (on immediate written notice to the other party).
+   This Agreement will automatically terminate if you breach any of its terms.
+   Upon termination, you must immediately cease all use of the LiveKit Materials.
+   Sections 4, 5, 6, and 9 will survive termination.
+
+9. Governing Law and Venue
+
+   This Agreement is subject to the laws of the State of California, without
+   regard to its conflict of laws principles. The UN Convention on Contracts for
+   the International Sale of Goods does not apply to this Agreement. The courts
+   located in San Francisco, California, have exclusive jurisdiction for any
+   dispute arising out of this Agreement.
+
++ + + +
+
+Last Updated: November 25, 2024

README.md

@@ -368,6 +368,8 @@ To connect and talk to your agent:
 This project is licensed under `Apache-2.0`, and is [REUSE-3.2](https://reuse.software) compliant.
 Refer to [the license](LICENSES/Apache-2.0.txt) for details.
 
+The LiveKit turn detection models are licensed under the [LiveKit Model License](MODEL_LICENSE).
+
 <!--BEGIN_REPO_NAV-->
 <br/><table>
 <thead><tr><th colspan="2">LiveKit Ecosystem</th></tr></thead>

REUSE.toml

@@ -7,6 +7,12 @@ SPDX-PackageName = "agents-js"
 SPDX-PackageSupplier = "LiveKit, Inc. <https://livekit.io>"
 SPDX-PackageDownloadLocation = "https://github.com/livekit/agents-js"
 
+# model license
+[[annotations]]
+path = ["MODEL_LICENSE"]
+SPDX-FileCopyrightText = "2024 LiveKit, Inc."
+SPDX-License-Identifier = "Apache-2.0"
+
 # trivial files
 [[annotations]]
 path = [".gitignore", "flake.lock", ".envrc", "packages/livekit-rtc/.gitignore", ".changeset/**", "**/CHANGELOG.md", "NOTICE", ".github/**"]

agents/package.json

@@ -52,8 +52,9 @@
   "dependencies": {
     "@bufbuild/protobuf": "^1.10.0",
     "@ffmpeg-installer/ffmpeg": "^1.1.0",
+    "@livekit/local-inference": "^0.2.5",
     "@livekit/mutex": "^1.1.1",
-    "@livekit/protocol": "^1.46.4",
+    "@livekit/protocol": "^1.46.5",
     "@livekit/throws-transformer": "0.1.8",
     "@livekit/typed-emitter": "^3.0.0",
     "@opentelemetry/api": "^1.9.0",

agents/src/inference/_warmup.ts

@@ -0,0 +1,45 @@
+// SPDX-FileCopyrightText: 2026 LiveKit, Inc.
+//
+// SPDX-License-Identifier: Apache-2.0
+
+/**
+ * Loader for the bundled `@livekit/local-inference` native binding.
+ *
+ * Memory model (measured ~138 MB for the EOT model, ~2 MB for VAD): Node has
+ * no forkserver/COW, so anything loaded in a job worker is private to that
+ * worker. To avoid paying ~138 MB per worker, the EOT model is NOT loaded in
+ * job workers — it runs in the shared `InferenceProcExecutor` (see
+ * `inference/eot/runner.ts`), loaded once per host. The VAD stays in-process
+ * (it's small and runs continuously) and is reached via this loader.
+ *
+ * There are intentionally no public `prewarm*` helpers: EOT auto-warms via
+ * the inference runner's `initialize()` at proc startup, and the VAD lazy-
+ * loads on first stream.
+ */
+import { createRequire } from 'node:module';
+import { log } from '../log.js';
+
+const cjsRequire = createRequire(import.meta.url);
+
+let nativeMod: typeof import('@livekit/local-inference') | undefined;
+let triedLoad = false;
+
+function getNative(): typeof import('@livekit/local-inference') | undefined {
+  if (triedLoad) return nativeMod;
+  triedLoad = true;
+  try {
+    nativeMod = cjsRequire('@livekit/local-inference') as typeof import('@livekit/local-inference');
+    return nativeMod;
+  } catch (err) {
+    log().warn(
+      { err: err instanceof Error ? err.message : String(err) },
+      '@livekit/local-inference native binding not loadable; local VAD/EOT paths disabled',
+    );
+    return undefined;
+  }
+}
+
+/** @internal Returns the loaded native module, or `undefined` if unavailable. */
+export function _getLocalInferenceModule(): typeof import('@livekit/local-inference') | undefined {
+  return getNative();
+}