measure: rescore on 100 FLEURS clips instead of 15 - #2
Merged
Conversation
The published Vietnamese WER for gipformer-65m was 2.41 %, measured on 15 clips and 146 seconds. On 100 clips (21.3 minutes, the first hundred by filename so the selection cannot have been made after seeing a score) the same model on the same runtime measures 8.5 %. Most of that gap is the dataset, not the model: sixteen of the hundred clips contain a number, and FLEURS writes numbers as digits while every speech model spells them out, so each one scores as several substitutions. Rescoring the 84 clips with no digits gives 5.3 %. The full hundred is what is published here, because every model has to be scored the same way and because a real meeting contains numbers too. RTF rows are added for the 4-thread bucket and corrected for the 8-thread one; both are medians of two runs. Whisper's row is its int8 build, which is the slower of the two it publishes — a model that turns out slower than its manifest promised cannot keep up with live audio, while one that turns out faster only surprises somebody pleasantly. Whisper-tiny's Vietnamese WER moves 65.5 % → 67.6 % on the larger sample, which changes nothing about the recommendation: it remains the wrong model for this language by a factor of eight.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The published Vietnamese WER for
gipformer-65mwas 2.41 %, from 15 clips / 146 s. On 100clips (21.3 min) the same model, same runtime, same machine measures 8.5 %.
Sixteen of the hundred clips contain a number. FLEURS writes them as digits (
5 trận), every speechmodel spells them (
năm trận), and each one scores as several substitutions — that is the 8.5 → 5.3difference, and it is the dataset's formatting rather than the model's error. The full hundred is
published because every model must be scored the same way, and because a real meeting has numbers in
it too.
RTF gains a
cpu_x86_avx512vnni_4trow and the 8-thread row is corrected. Both are medians of tworuns on a Xeon Gold 6226R. Whisper's figure is its int8 build — the slower of the two it
publishes — because a model that turns out slower than its manifest promised cannot keep up with
live audio, while one that turns out faster only surprises somebody pleasantly.
Selection is the first 100 files by name, so it could not have been picked after seeing a score.
Method and rebuild instructions:
docs/benchmarks.mdinsummo-app.summo registry check: 8 manifests valid.🤖 Generated with Claude Code