diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 3411917..e4e3f87 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -75,8 +75,11 @@ jobs: # `cargo clippy --workspace --all-targets` in the `check` job already type-checks every crate on # every push, so what the matrix below adds is *linking per platform* — a packaging question, and # packaging is what `master` and a tag are for. On a pull request that was five extra platforms, - # one of which (macOS Intel) queues for hours because those runners are scarce, holding up review - # of a one-line change behind a question that will be asked again on merge anyway. + # holding up review of a one-line change behind a question that will be asked again on merge + # anyway. + # + # `x86_64-apple-darwin` is not in this list: ONNX Runtime publishes no build for Intel macOS any + # more, so it cannot link. The reasoning is in `release.yml`, next to the matrix it was cut from. plan: name: which platforms runs-on: ubuntu-latest @@ -88,7 +91,6 @@ jobs: full='[{"os":"ubuntu-latest","target":"x86_64-unknown-linux-gnu"}, {"os":"ubuntu-24.04-arm","target":"aarch64-unknown-linux-gnu"}, {"os":"macos-latest","target":"aarch64-apple-darwin"}, - {"os":"macos-15-intel","target":"x86_64-apple-darwin"}, {"os":"windows-latest","target":"x86_64-pc-windows-msvc"}]' one='[{"os":"ubuntu-latest","target":"x86_64-unknown-linux-gnu"}]' if [ "${{ github.event_name }}" = "pull_request" ]; then diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index ff1bfcd..0272fe7 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -30,16 +30,23 @@ jobs: target: aarch64-unknown-linux-gnu - os: macos-latest target: aarch64-apple-darwin - # `macos-15-intel`, not `macos-13`: the 13 image is gone from GitHub's fleet, and a job - # asking for a label nobody offers does not fail — it sits in `queued` with an empty - # runner name until the run is cancelled. Four platforms went green in eleven minutes - # while this one waited half an hour for a machine that no longer exists. - - os: macos-15-intel - target: x86_64-apple-darwin - os: windows-latest target: x86_64-pc-windows-msvc # `aarch64-pc-windows-msvc` is absent on purpose — see the note in `ci.yml`. Opus has no # ARM64 Windows build in `audiopus_sys`, so the target cannot be produced at all yet. + # + # `x86_64-apple-darwin` is absent for a different reason, and not one we chose: ONNX + # Runtime no longer ships a build for it. Microsoft's 1.28 and 1.29 releases carry + # `onnxruntime-osx-arm64` and nothing else, and `ort-sys` 2.0.0-rc.13's prebuilt table + # lists nine targets of which Intel macOS is not one. So the release job for it failed + # with `no prebuilt binaries available for target x86_64-apple-darwin`. + # + # Pinning back to `ort` 2.0.0-rc.10, which did have Intel macOS binaries, does not work + # either — that version has no `tls-rustls` feature, which `summo-vad` uses. The routes + # left are building ONNX Runtime from source for one legacy platform, or dropping the + # `models` feature there and shipping a build that cannot detect speech. Neither is worth + # doing before somebody asks: Apple stopped selling Intel Macs in 2023, and an Intel Mac + # user can still build from source, where cargo will tell them the same thing. runs-on: ${{ matrix.os }} steps: diff --git a/README.md b/README.md index 9376cf0..2e6563d 100644 --- a/README.md +++ b/README.md @@ -41,7 +41,7 @@ command shown; see [`docs/benchmarks.md`](docs/benchmarks.md) and | Claim | Measured | Source | |---|---|---| -| Vietnamese recognition accuracy | 2.4 % WER, 1.7 % CER (`gipformer-65M`, Fleurs VI test set) | `cargo run --release -p summo-bench --features asr -- asr` | +| Vietnamese recognition accuracy | 8.5 % WER, 6.7 % CER (`gipformer-65M`, 100 FLEURS VI clips, 21.3 min; 5.3 % on the 84 clips whose reference contains no digits) | `cargo run --release -p summo-bench --features asr -- asr` | | Live pipeline speed | RTF 0.107, roughly 9× faster than realtime (raw mic capture) | `docs/benchmarks.md`, end-to-end pipeline section — two short single-mic captures, not yet WER-scored | | Voice activity detection | Silero v5, F1 0.940 (precision 0.925, recall 0.956) | `cargo run --release -p summo-bench --features silero -- vad --sweep` | | Finding a meeting without an index | ~30 ms across 1,000 meetings (8-thread scan), which is why there is no database | `cargo run --release -p summo-bench -- vault --sizes 100,1000,5000` | @@ -57,6 +57,11 @@ no web server to start and no directory of static files to keep in step. tar -xzf dist/summo-*.tar.gz && cd summo-* && ./summo serve ``` +Releases carry four builds: Linux x64 and arm64, macOS on Apple silicon, and Windows x64. Intel +macOS is missing because ONNX Runtime no longer publishes a build for it — Microsoft's 1.28 and 1.29 +releases ship `osx-arm64` only. Building from source on an Intel Mac hits the same wall, with a +clearer message. + Or building from source, without packaging: ```bash diff --git a/README.vi.md b/README.vi.md index 2e796a2..ad27996 100644 --- a/README.vi.md +++ b/README.vi.md @@ -43,7 +43,7 @@ bằng đúng lệnh ghi kèm; xem đầy đủ phương pháp và các lưu ý | Nhận định | Đo được | Nguồn | |---|---|---| -| Độ chính xác nhận dạng tiếng Việt | 2,4 % WER, 1,7 % CER (`gipformer-65M`, bộ test Fleurs VI) | `cargo run --release -p summo-bench --features asr -- asr` | +| Độ chính xác nhận dạng tiếng Việt | 8,5 % WER, 6,7 % CER (`gipformer-65M`, 100 clip FLEURS VI, 21,3 phút; còn 5,3 % nếu bỏ các clip mà bản tham chiếu viết số bằng chữ số) | `cargo run --release -p summo-bench --features asr -- asr` | | Tốc độ pipeline chạy live | RTF 0,107, tức nhanh hơn thời gian thực khoảng 9 lần (ghi bằng mic thô) | `docs/benchmarks.md`, mục pipeline đầu-cuối — mới đo trên hai đoạn ghi ngắn từ một mic, chưa tính WER | | Voice activity detection (VAD) | Silero v5, F1 0,940 (precision 0,925, recall 0,956) | `cargo run --release -p summo-bench --features silero -- vad --sweep` | | Tìm một cuộc họp mà không cần index | ~30 ms trên 1.000 cuộc họp (scan 8 luồng) — đây cũng là lý do không có database | `cargo run --release -p summo-bench -- vault --sizes 100,1000,5000` | @@ -59,6 +59,11 @@ nên không có web server nào cần khởi động, cũng không có thư mụ tar -xzf dist/summo-*.tar.gz && cd summo-* && ./summo serve ``` +Mỗi bản phát hành có bốn bản build: Linux x64 và arm64, macOS trên chip Apple, và Windows x64. Không +có macOS Intel, vì ONNX Runtime không còn phát hành bản build cho nền tảng đó — bản 1.28 và 1.29 của +Microsoft chỉ có `osx-arm64`. Build từ source trên máy Mac Intel cũng dừng ở đúng chỗ này, chỉ là +báo lỗi rõ ràng hơn. + Hoặc build từ source, không đóng gói: ```bash diff --git a/bench/encoder_precision.py b/bench/encoder_precision.py new file mode 100644 index 0000000..11e05c6 --- /dev/null +++ b/bench/encoder_precision.py @@ -0,0 +1,129 @@ +#!/usr/bin/env python3 +"""Time one encoder pass at fp32 and at int8, in both CPU runtimes. + + pip install onnxruntime openvino + taskset -c 0-3 python bench/encoder_precision.py --fp32 fp32/tiny-encoder.onnx \ + --int8 int8/tiny-encoder.int8.onnx + +## Why this exists separately from `summo-bench asr` + +`summo-bench asr` measures the thing users feel: audio in, text out. It found that int8 whisper-tiny +is exactly as fast as fp32 end to end, which is a surprising enough result to be worth explaining +rather than just recording — a quantised model is *supposed* to be faster. + +This script isolates the encoder, which is the half of whisper that looks like the matrix +multiplication quantisation is good at: one pass over a fixed 30 s window, large matrices. It turns +out int8 wins there by about 1.4× in ONNX Runtime and loses the rest back in the autoregressive +decoder, whose matrices are small enough that the quantise/dequantise around each one costs more +than the cheaper multiply saves. + +It runs OpenVINO alongside ONNX Runtime to answer the other suspicion — that the machine, not the +model, is bad at int8. A runtime tuned by Intel for Intel silicon disagreeing with ORT about *which* +precision is faster is the evidence that this is a property of the graph. + +## Why `taskset` + +Without pinning, a thread pool asked for four threads on an idle 64-core machine does not behave +like four cores on a laptop: the OS spreads it across sockets and the memory-bound parts get more +bandwidth than any real machine would give them. Pinning to four hardware threads on one socket is +the closest this hardware gets to the machine the answer is for. +""" + +from __future__ import annotations + +import argparse +import statistics +import time + +import numpy as np + + +def bench_ort(path: str, mel: np.ndarray, threads: int, runs: int) -> list[float]: + import onnxruntime as ort + + options = ort.SessionOptions() + options.intra_op_num_threads = threads + # The default already, set explicitly so the comparison cannot be blamed on it. + options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL + session = ort.InferenceSession(path, options, providers=["CPUExecutionProvider"]) + feed = {session.get_inputs()[0].name: mel} + + for _ in range(3): # warm up: first passes allocate arenas and pick kernels + session.run(None, feed) + timings = [] + for _ in range(runs): + start = time.perf_counter() + session.run(None, feed) + timings.append(time.perf_counter() - start) + return timings + + +def bench_openvino(path: str, mel: np.ndarray, threads: int, runs: int) -> list[float]: + import openvino as ov + + core = ov.Core() + model = core.read_model(path) + # The exported graph has a dynamic batch and frame count; a fixed shape is what a runtime can + # actually optimise for, and it is what the app feeds. + model.reshape({model.inputs[0].get_any_name(): ov.PartialShape(list(mel.shape))}) + compiled = core.compile_model( + model, "CPU", {"INFERENCE_NUM_THREADS": threads, "PERFORMANCE_HINT": "LATENCY"} + ) + request = compiled.create_infer_request() + feed = {compiled.inputs[0].get_any_name(): mel} + + for _ in range(3): + request.infer(feed) + timings = [] + for _ in range(runs): + start = time.perf_counter() + request.infer(feed) + timings.append(time.perf_counter() - start) + return timings + + +def main() -> None: + parser = argparse.ArgumentParser( + description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter + ) + parser.add_argument("--fp32", required=True, help="full-precision encoder .onnx") + parser.add_argument("--int8", required=True, help="quantised encoder .onnx") + parser.add_argument("--threads", type=int, default=4) + parser.add_argument("--runs", type=int, default=15) + parser.add_argument( + "--frames", + type=int, + default=3000, + help="mel frames; 3000 is whisper's fixed 30 s window", + ) + args = parser.parse_args() + + # Random mel rather than real audio: the encoder does the same work whatever the values are, and + # a fixed shape keeps the two precisions comparable. + mel = np.random.randn(1, 80, args.frames).astype(np.float32) + + runners = [("onnxruntime", bench_ort)] + try: + import openvino # noqa: F401 + + runners.append(("openvino", bench_openvino)) + except ImportError: + print("openvino not installed; reporting ONNX Runtime only\n") + + print(f"{'runtime':<14}{'precision':<10}{'median ms':>10}{'min ms':>9}") + for name, run in runners: + results = {} + for precision, path in (("fp32", args.fp32), ("int8", args.int8)): + timings = run(path, mel, args.threads, args.runs) + results[precision] = statistics.median(timings) + print( + f"{name:<14}{precision:<10}" + f"{statistics.median(timings) * 1000:10.1f}{min(timings) * 1000:9.1f}" + ) + ratio = results["fp32"] / results["int8"] + verdict = "faster" if ratio > 1 else "slower" + print(f"{'':14}int8 is {abs(ratio if ratio > 1 else 1 / ratio):.2f}× {verdict}\n") + + +if __name__ == "__main__": + main() diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 94c9330..772aea4 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -57,29 +57,41 @@ cargo run --release -p summo-bench --features silero -- vad \ Single-pass decode per utterance, scored against reference transcripts. This measures the model; the session's re-decode multiplier is measured separately below. -**Dataset:** Fleurs VI test, 15 clips, 146.6 s. **Model:** `gipformer-65M` (Zipformer RNN-T, INT8 -ONNX, 73 MB) via sherpa-onnx, 4 threads. - -| Model | Dataset | Threads | WER | CER | RTF | Audio | Empty | -|---|---|---:|---:|---:|---:|---:|---:| -| gipformer-65M | fleurs_vi (Vietnamese) | 4 | **2.4 %** | 1.7 % | 0.021 | 146.6 s | 0 | -| whisper-tiny | fleurs_vi (Vietnamese) | 4 | 65.5 % | 47.8 % | 0.116 | 146.6 s | 0 | -| whisper-tiny | whisper test set (English) | 4 | **4.5 %** | 0.3 % | 0.107 | 23.3 s | 0 | - -Those three rows are the argument for a flat registry rather than a "basic / better / best" ladder. +**Dataset:** FLEURS `vi_vn` test, **100 clips, 1279.8 s (21.3 min)** — the first 100 by filename, so +the selection is reproducible and not chosen after seeing a score. 16 kHz mono, converted from +FLEURS' float WAVs. Two runs per row, median reported. Xeon Gold 6226R, sherpa-onnx. + +| Model | Dataset | Threads | WER | CER | RTF | +|---|---|---:|---:|---:|---:| +| gipformer-65M (int8) | fleurs_vi | 4 | **8.5 %** | 6.7 % | 0.023 | +| gipformer-65M (int8) | fleurs_vi | 8 | 8.6 % | 6.8 % | 0.019 | +| whisper-tiny (fp32) | fleurs_vi | 4 | 67.6 % | 45.1 % | 0.137 | +| whisper-tiny (int8) | fleurs_vi | 4 | 81.3 % | 60.0 % | 0.138 | +| whisper-tiny (fp32) | fleurs_vi | 8 | 67.6 % | 45.1 % | 0.116 | +| whisper-tiny (int8) | fleurs_vi | 8 | 81.0 % | 59.7 % | 0.120 | +| whisper-tiny (fp32) | whisper test set (English) | 4 | **4.5 %** | 0.3 % | 0.107 | + +Those rows are the argument for a flat registry rather than a "basic / better / best" ladder. Whisper-tiny is not a bad model — it scores 4.5 % on English. It is a bad model *for Vietnamese*, -where it is 27× worse than a 73 MB transducer that also runs five times faster. No single ordering of -models is correct across languages, so Summo does not impose one: each manifest states which -languages it was measured on, and the app recommends from that. +where it is eight times worse than a 73 MB transducer that also runs six times faster. No single +ordering of models is correct across languages, so Summo does not impose one: each manifest states +which languages it was measured on, and the app recommends from that. The English figure comes from two clips and should be read as "the model works", not as a WER measurement. A real English number needs LibriSpeech or Common Voice. -**2.4 % WER is exactly what the Python prototype measured on the same model and dataset.** That -agreement is the point of running this: it confirms the Rust port feeds the model the same audio and -reads back the same text, rather than being a plausible-looking reimplementation that quietly -differs. RTF is higher than the prototype's 0.017 because that run used 16 threads against this one's -4, not because the decode changed. +**These numbers replace a 15-clip run that reported 2.4 % for gipformer.** 146 s of read speech is +not a measurement, it is an anecdote: it happened to contain no clip with a number in it. Sixteen of +these hundred clips do, and FLEURS writes them as digits (`5 trận`) while every speech model spells +them (`năm trận`), so each one scores as several substitutions. Removing those sixteen clips and +rescoring the remaining 84 gives **5.3 %** for gipformer and 65.7 % for whisper-tiny fp32 — the gap +between 8.5 % and 5.3 % is the dataset's number formatting, not the model. The published figure is +the full hundred, because scoring every model the same way matters more than the flattering subset, +and because a user's meeting will also contain numbers. + +The old figure agreed with the Python prototype on the same 15 clips, which was the point of running +it: it confirmed the Rust port feeds the model the same audio and reads back the same text. That +agreement still holds; only the sample it was measured on was too small to publish. Text is normalised before scoring — lowercased, punctuation stripped, whitespace collapsed. A transducer emits uppercase without punctuation while the reference has both, and counting that as @@ -95,7 +107,91 @@ cargo run --release -p summo-bench --features asr -- asr \ ``` The dataset directory needs a `transcripts.json` of `{wav, text, duration_s}` entries alongside the -16 kHz mono WAVs it names. +16 kHz mono WAVs it names. To rebuild the FLEURS set used above: + +```bash +python -c " +from huggingface_hub import hf_hub_download +hf_hub_download('google/fleurs', 'data/vi_vn/audio/test.tar.gz', repo_type='dataset', local_dir='fleurs') +hf_hub_download('google/fleurs', 'data/vi_vn/test.tsv', repo_type='dataset', local_dir='fleurs')" +tar xzf fleurs/data/vi_vn/audio/test.tar.gz -C fleurs +# FLEURS ships float WAVs; hound reads 16-bit PCM. +for f in $(ls fleurs/test | sort | head -100); do sox "fleurs/test/$f" -r 16000 -c 1 -b 16 -e signed-integer "fleurs/wav/$f"; done +# transcripts.json: column 2 of test.tsv is the filename, column 3 the reference. +``` + +## Quantisation: does int8 pay on a CPU? + +The question behind every model we ship: Summo runs on a laptop CPU, and int8 halves the download +and the resident memory. Does it also make inference faster, the way it does on a GPU with INT8 +tensor cores? + +**On the whole model, no — and for Vietnamese it costs a lot of accuracy.** Same 100-clip FLEURS +set, same runtime, the two builds whisper-tiny publishes: + +| | WER | RTF @4t | RTF @8t | Size | +|---|---:|---:|---:|---:| +| fp32 | **67.6 %** | **0.137** | **0.116** | 146 MB | +| int8 | 81.3 % | 0.138 | 0.120 | 99 MB | + +Identical speed, **13.7 points worse**. Everything int8 buys here is on disk. + +That is not because quantisation does nothing. It is because whisper is an encoder plus an +*autoregressive* decoder, and the two react in opposite directions. Encoder only, one 30 s window, +pinned to four cores, 15 runs, median: + +| Runtime | fp32 | int8 | | +|---|---:|---:|---| +| ONNX Runtime 1.28 | 155 ms | **126 ms** | int8 **1.24× faster** | +| OpenVINO 2026.3 | **115 ms** | 182 ms | int8 1.59× *slower* | + +So int8 does win where the matrices are large and the pass happens once — the encoder, in the +runtime we ship. It loses in the decoder, which runs a few hundred times per clip on matrices small +enough that the quantise/dequantise around each one costs more than the cheaper multiply saves. +Arithmetic on the numbers above: an average 12.8 s clip costs ~1.75 s at 4 threads, of which the +encoder is ~0.16 s; int8 saves ~0.03 s there and hands it straight back in the decoder. The two +measurements are not pinned the same way — end-to-end runs use the whole machine, the encoder rows +are pinned to four hardware threads — so read the ratios, not the absolute milliseconds. + +The OpenVINO column answers the other suspicion — *maybe this CPU is bad at int8*. This is a +Cascade Lake Xeon with AVX-512 VNNI, the instruction set built for exactly this, and a second +runtime tuned by Intel for Intel reaches the same conclusion by a different route: it is fastest of +all on fp32 and worst of all on this int8 graph. The limit is the shape of the model, not the chip. + +**Decision:** prefer fp32 and fall back to int8 only when memory forces it, which is what +`variant::rank` already does. Publish both builds — a machine with 2 GB free needs the smaller one, +and 81 % WER is still better than no transcription — but never pick int8 to go faster. + +**Why not ship OpenVINO, or oneDNN, or MKL:** the ONNX Runtime inside our release exports exactly +one execution provider — + +```console +$ nm -D libonnxruntime.so | grep AppendExecutionProvider +000000000036d9c0 T OrtSessionOptionsAppendExecutionProvider_CPU@@VERS_1.17.1 +``` + +— so its kernels are MLAS, which already has AVX-512 and VNNI paths. Adding oneDNN or OpenVINO means +building ONNX Runtime ourselves for five platforms and carrying that build forever, and the fp32 +result above (169 ms vs 265 ms) is the whole prize. It is real, and it is not worth a bespoke +toolchain for a decode that is already 6× faster than real time. Revisit if a model ever lands whose +cost is dominated by one big encoder pass. + +Graph optimisation is already at maximum everywhere: `summo-mt` and `summo-vad` set +`GraphOptimizationLevel::Level3`, and sherpa-onnx leaves ONNX Runtime's default, which is +`ORT_ENABLE_ALL` — its `SetGraphOptimizationLevel` line is commented out for that reason. + +### Reproduce + +```bash +# End-to-end, both builds of the same model, on the same dataset. +cargo run --release -p summo-bench --features asr -- asr \ + --dataset fleurs-vi --model whisper:fp32 --model whisper:int8 --lang vi --threads 4 + +# Encoder only, pinned so the thread pool cannot borrow idle cores. +pip install onnxruntime openvino +taskset -c 0-3 python bench/encoder_precision.py \ + --fp32 fp32/tiny-encoder.onnx --int8 int8/tiny-encoder.int8.onnx +``` ## Speech recognition — end-to-end pipeline