docs: measure what int8 costs; drop the platform ONNX Runtime dropped

.github/workflows/ci.yml

@@ -75,8 +75,11 @@ jobs:
   # `cargo clippy --workspace --all-targets` in the `check` job already type-checks every crate on
   # every push, so what the matrix below adds is *linking per platform* — a packaging question, and
   # packaging is what `master` and a tag are for. On a pull request that was five extra platforms,
-  # one of which (macOS Intel) queues for hours because those runners are scarce, holding up review
-  # of a one-line change behind a question that will be asked again on merge anyway.
+  # holding up review of a one-line change behind a question that will be asked again on merge
+  # anyway.
+  #
+  # `x86_64-apple-darwin` is not in this list: ONNX Runtime publishes no build for Intel macOS any
+  # more, so it cannot link. The reasoning is in `release.yml`, next to the matrix it was cut from.
   plan:
     name: which platforms
     runs-on: ubuntu-latest
@@ -88,7 +91,6 @@ jobs:
           full='[{"os":"ubuntu-latest","target":"x86_64-unknown-linux-gnu"},
                  {"os":"ubuntu-24.04-arm","target":"aarch64-unknown-linux-gnu"},
                  {"os":"macos-latest","target":"aarch64-apple-darwin"},
-                 {"os":"macos-15-intel","target":"x86_64-apple-darwin"},
                  {"os":"windows-latest","target":"x86_64-pc-windows-msvc"}]'
           one='[{"os":"ubuntu-latest","target":"x86_64-unknown-linux-gnu"}]'
           if [ "${{ github.event_name }}" = "pull_request" ]; then

.github/workflows/release.yml

@@ -30,16 +30,23 @@ jobs:
             target: aarch64-unknown-linux-gnu
           - os: macos-latest
             target: aarch64-apple-darwin
-          # `macos-15-intel`, not `macos-13`: the 13 image is gone from GitHub's fleet, and a job
-          # asking for a label nobody offers does not fail — it sits in `queued` with an empty
-          # runner name until the run is cancelled. Four platforms went green in eleven minutes
-          # while this one waited half an hour for a machine that no longer exists.
-          - os: macos-15-intel
-            target: x86_64-apple-darwin
           - os: windows-latest
             target: x86_64-pc-windows-msvc
           # `aarch64-pc-windows-msvc` is absent on purpose — see the note in `ci.yml`. Opus has no
           # ARM64 Windows build in `audiopus_sys`, so the target cannot be produced at all yet.
+          #
+          # `x86_64-apple-darwin` is absent for a different reason, and not one we chose: ONNX
+          # Runtime no longer ships a build for it. Microsoft's 1.28 and 1.29 releases carry
+          # `onnxruntime-osx-arm64` and nothing else, and `ort-sys` 2.0.0-rc.13's prebuilt table
+          # lists nine targets of which Intel macOS is not one. So the release job for it failed
+          # with `no prebuilt binaries available for target x86_64-apple-darwin`.
+          #
+          # Pinning back to `ort` 2.0.0-rc.10, which did have Intel macOS binaries, does not work
+          # either — that version has no `tls-rustls` feature, which `summo-vad` uses. The routes
+          # left are building ONNX Runtime from source for one legacy platform, or dropping the
+          # `models` feature there and shipping a build that cannot detect speech. Neither is worth
+          # doing before somebody asks: Apple stopped selling Intel Macs in 2023, and an Intel Mac
+          # user can still build from source, where cargo will tell them the same thing.
 
     runs-on: ${{ matrix.os }}
     steps:

README.md

@@ -41,7 +41,7 @@ command shown; see [`docs/benchmarks.md`](docs/benchmarks.md) and
 
 | Claim | Measured | Source |
 |---|---|---|
-| Vietnamese recognition accuracy | 2.4 % WER, 1.7 % CER (`gipformer-65M`, Fleurs VI test set) | `cargo run --release -p summo-bench --features asr -- asr` |
+| Vietnamese recognition accuracy | 8.5 % WER, 6.7 % CER (`gipformer-65M`, 100 FLEURS VI clips, 21.3 min; 5.3 % on the 84 clips whose reference contains no digits) | `cargo run --release -p summo-bench --features asr -- asr` |
 | Live pipeline speed | RTF 0.107, roughly 9× faster than realtime (raw mic capture) | `docs/benchmarks.md`, end-to-end pipeline section — two short single-mic captures, not yet WER-scored |
 | Voice activity detection | Silero v5, F1 0.940 (precision 0.925, recall 0.956) | `cargo run --release -p summo-bench --features silero -- vad --sweep` |
 | Finding a meeting without an index | ~30 ms across 1,000 meetings (8-thread scan), which is why there is no database | `cargo run --release -p summo-bench -- vault --sizes 100,1000,5000` |
@@ -57,6 +57,11 @@ no web server to start and no directory of static files to keep in step.
 tar -xzf dist/summo-*.tar.gz && cd summo-* && ./summo serve
 ```
 
+Releases carry four builds: Linux x64 and arm64, macOS on Apple silicon, and Windows x64. Intel
+macOS is missing because ONNX Runtime no longer publishes a build for it — Microsoft's 1.28 and 1.29
+releases ship `osx-arm64` only. Building from source on an Intel Mac hits the same wall, with a
+clearer message.
+
 Or building from source, without packaging:
 
 ```bash

README.vi.md

@@ -43,7 +43,7 @@ bằng đúng lệnh ghi kèm; xem đầy đủ phương pháp và các lưu ý
 
 | Nhận định | Đo được | Nguồn |
 |---|---|---|
-| Độ chính xác nhận dạng tiếng Việt | 2,4 % WER, 1,7 % CER (`gipformer-65M`, bộ test Fleurs VI) | `cargo run --release -p summo-bench --features asr -- asr` |
+| Độ chính xác nhận dạng tiếng Việt | 8,5 % WER, 6,7 % CER (`gipformer-65M`, 100 clip FLEURS VI, 21,3 phút; còn 5,3 % nếu bỏ các clip mà bản tham chiếu viết số bằng chữ số) | `cargo run --release -p summo-bench --features asr -- asr` |
 | Tốc độ pipeline chạy live | RTF 0,107, tức nhanh hơn thời gian thực khoảng 9 lần (ghi bằng mic thô) | `docs/benchmarks.md`, mục pipeline đầu-cuối — mới đo trên hai đoạn ghi ngắn từ một mic, chưa tính WER |
 | Voice activity detection (VAD) | Silero v5, F1 0,940 (precision 0,925, recall 0,956) | `cargo run --release -p summo-bench --features silero -- vad --sweep` |
 | Tìm một cuộc họp mà không cần index | ~30 ms trên 1.000 cuộc họp (scan 8 luồng) — đây cũng là lý do không có database | `cargo run --release -p summo-bench -- vault --sizes 100,1000,5000` |
@@ -59,6 +59,11 @@ nên không có web server nào cần khởi động, cũng không có thư mụ
 tar -xzf dist/summo-*.tar.gz && cd summo-* && ./summo serve
 ```
 
+Mỗi bản phát hành có bốn bản build: Linux x64 và arm64, macOS trên chip Apple, và Windows x64. Không
+có macOS Intel, vì ONNX Runtime không còn phát hành bản build cho nền tảng đó — bản 1.28 và 1.29 của
+Microsoft chỉ có `osx-arm64`. Build từ source trên máy Mac Intel cũng dừng ở đúng chỗ này, chỉ là
+báo lỗi rõ ràng hơn.
+
 Hoặc build từ source, không đóng gói:
 
 ```bash

bench/encoder_precision.py

@@ -0,0 +1,129 @@
+#!/usr/bin/env python3
+"""Time one encoder pass at fp32 and at int8, in both CPU runtimes.
+
+    pip install onnxruntime openvino
+    taskset -c 0-3 python bench/encoder_precision.py --fp32 fp32/tiny-encoder.onnx \
+                                                     --int8 int8/tiny-encoder.int8.onnx
+
+## Why this exists separately from `summo-bench asr`
+
+`summo-bench asr` measures the thing users feel: audio in, text out. It found that int8 whisper-tiny
+is exactly as fast as fp32 end to end, which is a surprising enough result to be worth explaining
+rather than just recording — a quantised model is *supposed* to be faster.
+
+This script isolates the encoder, which is the half of whisper that looks like the matrix
+multiplication quantisation is good at: one pass over a fixed 30 s window, large matrices. It turns
+out int8 wins there by about 1.4× in ONNX Runtime and loses the rest back in the autoregressive
+decoder, whose matrices are small enough that the quantise/dequantise around each one costs more
+than the cheaper multiply saves.
+
+It runs OpenVINO alongside ONNX Runtime to answer the other suspicion — that the machine, not the
+model, is bad at int8. A runtime tuned by Intel for Intel silicon disagreeing with ORT about *which*
+precision is faster is the evidence that this is a property of the graph.
+
+## Why `taskset`
+
+Without pinning, a thread pool asked for four threads on an idle 64-core machine does not behave
+like four cores on a laptop: the OS spreads it across sockets and the memory-bound parts get more
+bandwidth than any real machine would give them. Pinning to four hardware threads on one socket is
+the closest this hardware gets to the machine the answer is for.
+"""
+
+from __future__ import annotations
+
+import argparse
+import statistics
+import time
+
+import numpy as np
+
+
+def bench_ort(path: str, mel: np.ndarray, threads: int, runs: int) -> list[float]:
+    import onnxruntime as ort
+
+    options = ort.SessionOptions()
+    options.intra_op_num_threads = threads
+    # The default already, set explicitly so the comparison cannot be blamed on it.
+    options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
+    session = ort.InferenceSession(path, options, providers=["CPUExecutionProvider"])
+    feed = {session.get_inputs()[0].name: mel}
+
+    for _ in range(3):  # warm up: first passes allocate arenas and pick kernels
+        session.run(None, feed)
+    timings = []
+    for _ in range(runs):
+        start = time.perf_counter()
+        session.run(None, feed)
+        timings.append(time.perf_counter() - start)
+    return timings
+
+
+def bench_openvino(path: str, mel: np.ndarray, threads: int, runs: int) -> list[float]:
+    import openvino as ov
+
+    core = ov.Core()
+    model = core.read_model(path)
+    # The exported graph has a dynamic batch and frame count; a fixed shape is what a runtime can
+    # actually optimise for, and it is what the app feeds.
+    model.reshape({model.inputs[0].get_any_name(): ov.PartialShape(list(mel.shape))})
+    compiled = core.compile_model(
+        model, "CPU", {"INFERENCE_NUM_THREADS": threads, "PERFORMANCE_HINT": "LATENCY"}
+    )
+    request = compiled.create_infer_request()
+    feed = {compiled.inputs[0].get_any_name(): mel}
+
+    for _ in range(3):
+        request.infer(feed)
+    timings = []
+    for _ in range(runs):
+        start = time.perf_counter()
+        request.infer(feed)
+        timings.append(time.perf_counter() - start)
+    return timings
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(
+        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
+    )
+    parser.add_argument("--fp32", required=True, help="full-precision encoder .onnx")
+    parser.add_argument("--int8", required=True, help="quantised encoder .onnx")
+    parser.add_argument("--threads", type=int, default=4)
+    parser.add_argument("--runs", type=int, default=15)
+    parser.add_argument(
+        "--frames",
+        type=int,
+        default=3000,
+        help="mel frames; 3000 is whisper's fixed 30 s window",
+    )
+    args = parser.parse_args()
+
+    # Random mel rather than real audio: the encoder does the same work whatever the values are, and
+    # a fixed shape keeps the two precisions comparable.
+    mel = np.random.randn(1, 80, args.frames).astype(np.float32)
+
+    runners = [("onnxruntime", bench_ort)]
+    try:
+        import openvino  # noqa: F401
+
+        runners.append(("openvino", bench_openvino))
+    except ImportError:
+        print("openvino not installed; reporting ONNX Runtime only\n")
+
+    print(f"{'runtime':<14}{'precision':<10}{'median ms':>10}{'min ms':>9}")
+    for name, run in runners:
+        results = {}
+        for precision, path in (("fp32", args.fp32), ("int8", args.int8)):
+            timings = run(path, mel, args.threads, args.runs)
+            results[precision] = statistics.median(timings)
+            print(
+                f"{name:<14}{precision:<10}"
+                f"{statistics.median(timings) * 1000:10.1f}{min(timings) * 1000:9.1f}"
+            )
+        ratio = results["fp32"] / results["int8"]
+        verdict = "faster" if ratio > 1 else "slower"
+        print(f"{'':14}int8 is {abs(ratio if ratio > 1 else 1 / ratio):.2f}× {verdict}\n")
+
+
+if __name__ == "__main__":
+    main()