From 8cc2080ebc59da5fa38626d4ec34d12b8b27956f Mon Sep 17 00:00:00 2001
From: Viet Nguyen <drake@techainer.com>
Date: Thu, 13 Aug 2026 12:37:40 +0000
Subject: [PATCH] measure: rescore gipformer and whisper-tiny on 100 FLEURS
 clips, not 15
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The published Vietnamese WER for gipformer-65m was 2.41 %, measured on 15 clips and 146 seconds.
On 100 clips (21.3 minutes, the first hundred by filename so the selection cannot have been made
after seeing a score) the same model on the same runtime measures 8.5 %.

Most of that gap is the dataset, not the model: sixteen of the hundred clips contain a number, and
FLEURS writes numbers as digits while every speech model spells them out, so each one scores as
several substitutions. Rescoring the 84 clips with no digits gives 5.3 %. The full hundred is what
is published here, because every model has to be scored the same way and because a real meeting
contains numbers too.

RTF rows are added for the 4-thread bucket and corrected for the 8-thread one; both are medians of
two runs. Whisper's row is its int8 build, which is the slower of the two it publishes — a model
that turns out slower than its manifest promised cannot keep up with live audio, while one that
turns out faster only surprises somebody pleasantly.

Whisper-tiny's Vietnamese WER moves 65.5 % → 67.6 % on the larger sample, which changes nothing
about the recommendation: it remains the wrong model for this language by a factor of eight.
---
 models/gipformer-65m.json | 9 +++++----
 models/whisper-tiny.json  | 8 +++++---
 2 files changed, 10 insertions(+), 7 deletions(-)

diff --git a/models/gipformer-65m.json b/models/gipformer-65m.json
index 5b988e4..77210bc 100644
--- a/models/gipformer-65m.json
+++ b/models/gipformer-65m.json
@@ -20,8 +20,9 @@
     },
     "min_ram_mb": 1024,
     "rtf": {
-      "cpu_x86_avx512vnni_8t": 0.024,
-      "cpu_x86_avx2_4t": 0.06
+      "cpu_x86_avx2_4t": 0.06,
+      "cpu_x86_avx512vnni_4t": 0.023,
+      "cpu_x86_avx512vnni_8t": 0.019
     },
     "latency_ms": {
       "first_partial": 200,
@@ -29,8 +30,8 @@
       "finalize_p95": 700
     },
     "quality": {
-      "wer_fleurs_vi": 0.0241,
-      "cer_fleurs_vi": 0.0167
+      "wer_fleurs_vi": 0.085,
+      "cer_fleurs_vi": 0.067
     },
     "accel": [
       "cpu",
diff --git a/models/whisper-tiny.json b/models/whisper-tiny.json
index 315ec2e..b935f81 100644
--- a/models/whisper-tiny.json
+++ b/models/whisper-tiny.json
@@ -20,8 +20,9 @@
     },
     "min_ram_mb": 1024,
     "rtf": {
-      "cpu_x86_avx512vnni_8t": 0.107,
-      "cpu_x86_avx2_4t": 0.3
+      "cpu_x86_avx2_4t": 0.3,
+      "cpu_x86_avx512vnni_4t": 0.138,
+      "cpu_x86_avx512vnni_8t": 0.12
     },
     "latency_ms": {
       "first_partial": 0,
@@ -30,7 +31,8 @@
     },
     "quality": {
       "wer_whisper_testset_en": 0.045,
-      "wer_fleurs_vi": 0.655
+      "wer_fleurs_vi": 0.676,
+      "cer_fleurs_vi": 0.451
     },
     "accel": [
       "cpu",