docs: measure what int8 costs; drop the platform ONNX Runtime dropped - #6
Merged
Merged
Conversation
…ime dropped Three things, all from running the thing rather than reasoning about it. **Quantisation.** int8 is supposed to be the fast build, so the app's rule — prefer fp32, fall back to int8 only when memory forces it — deserved a check. On 100 FLEURS clips whisper-tiny int8 is exactly as fast as fp32 (RTF 0.138 vs 0.137 at four threads) and 13.7 WER points worse. The reason is not the chip: on the encoder alone int8 *is* 1.24× faster in ONNX Runtime, and the win is handed straight back by the autoregressive decoder, whose matrices are small enough that the quantise/dequantise around each one costs more than the cheaper multiply saves. `bench/encoder_ precision.py` is added so that decomposition can be re-run rather than believed. OpenVINO is in that script for the suspicion it settles: that this Xeon is bad at int8. It has AVX-512 VNNI, and a runtime tuned by Intel for Intel reaches the same conclusion by a different route — fastest of all on fp32, slowest of all on this int8 graph. The limit is the model's shape. Also written down, because it keeps being asked: the ONNX Runtime we ship exports one execution provider, `CPU`, so its kernels are MLAS, which already has AVX-512 and VNNI paths. oneDNN, MKL and OpenVINO would each mean building ONNX Runtime ourselves for every platform and carrying that build forever. Graph optimisation is already maximal — `Level3` in our own sessions, and sherpa-onnx leaves ORT's default, which is `ORT_ENABLE_ALL`. **Accuracy.** The published 2.4 % Vietnamese WER came from 15 clips and 146 seconds. On 100 clips it is 8.5 %, of which about three points are FLEURS writing numbers as digits while every speech model spells them out. Both READMEs and `docs/benchmarks.md` now carry the larger measurement and say what is in it. **Intel macOS.** `bundle (x86_64-apple-darwin)` fails with `no prebuilt binaries available`, and it is not our packaging: Microsoft's ONNX Runtime 1.28 and 1.29 ship `osx-arm64` only, and `ort-sys` 2.0.0-rc.13 lists nine prebuilt targets without Intel macOS among them. Pinning back to rc.10, which had them, breaks on a feature `summo-vad` needs. The target is removed from both workflows with the evidence in a comment, and both READMEs now say which four builds a release carries and why the fifth is absent — better than a download page with a gap in it that nobody explains.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
int8 does not make this faster, and for Vietnamese it costs 13.7 WER points
100 FLEURS clips (21.3 min), whisper-tiny's own two builds, same runtime:
Not the chip's fault. Encoder alone, pinned to four cores, 15 runs:
int8 wins on the encoder — one pass, large matrices — and hands it back in the autoregressive
decoder. OpenVINO is there to settle "maybe this Xeon is bad at int8": it has AVX-512 VNNI, and
Intel's own runtime agrees about the graph while disagreeing about everything else.
Confirms the rule already in
variant::rank: prefer fp32, fall back to int8 only when memoryforces it. New
bench/encoder_precision.pyso the decomposition can be re-run.MKL / oneDNN / OpenVINO as an EP: the ONNX Runtime in our release exports one provider —
nm -D libonnxruntime.so | grep AppendExecutionProviderreturns..._CPUand nothing else — so thekernels are MLAS, which already has AVX-512/VNNI paths. Anything else means building ONNX Runtime
ourselves for five platforms forever, for the 155 → 115 ms in the table. Graph optimisation is
already maximal:
Level3in our sessions,ORT_ENABLE_ALL(ORT's default) in sherpa-onnx.Vietnamese WER: 2.4 % → 8.5 %
The old figure came from 15 clips / 146 s. About three of the added points are FLEURS writing
numbers as digits while models spell them out (5.3 % on the 84 clips with no digits). Registry PR:
Techainer/summo-registry#2.
Intel macOS is dropped, and not by us
Microsoft ships
onnxruntime-osx-arm64only in 1.28 and 1.29;ort-sysrc.13 lists nine prebuilttargets, none Intel macOS. Pinning back to rc.10 (which had them) fails on a missing
tls-rustlsfeature that
summo-vaduses. Removed from both workflows with the evidence in a comment; bothREADMEs now name the four builds a release carries and why the fifth is absent.
Along the way:
macos-13no longer exists as a runner label, and a job asking for one does not fail— it queues forever with an empty runner name.
🤖 Generated with Claude Code