[pull] main from CrispStrobe:main - #10
Open
pull[bot] wants to merge 3840 commits into
Open
Conversation
whisper_vad_build_lstm_layer fed a transpose VIEW straight into a matmul:
ggml_tensor* x_t = ggml_transpose(ctx0, cur); // non-contiguous
ggml_tensor* inp_gate = ggml_mul_mat(ctx0, model.lstm_ih_weight, x_t);
ggml_transpose swaps nb[0]/nb[1], so x_t's row stride becomes sizeof(float) and
llamafile_sgemm's `ldb` collapses to 1 while k is lstm_hidden_size (128) —
tripping its `ldb >= k` precondition (ggml-cpu/llamafile/sgemm.cpp:3700). Fixed
with ggml_cont, which materialises the transpose; that is the idiom the rest of
this codebase already uses (src/audioseal.cpp, five sites). The VAD was the
outlier.
WHAT MADE IT INVISIBLE, which is the part worth remembering. Release defines
NDEBUG, so the assert is compiled out and the matmul RAN ANYWAY with a violated
stride precondition rather than stopping — producing plausible-looking segments.
Only a Debug build aborted, and the Debug leg had never run: CI executed 1 of 162
unit tests until 2026-07-29. Turning the unit tier on is what surfaced this.
I initially misread it as a brittle test (hard-coded "344 probs / 4 segments"
flipping between -O0 and -O2). The assertion text settled it: the test was right,
the code was wrong.
Verified on Kaggle before pushing, A/B on ONE tree, both compilers, 8/8 checks:
before gcc/clang Debug rc=134, assertion `ldb >= k`, 0 segments
after gcc/clang Debug rc=0, 4 segments
before gcc/clang Release rc=0, 4 segments
after gcc/clang Release rc=0, 4 segments (BYTE-IDENTICAL)
The Release half matters as much as the Debug half: a precondition fix must not
smuggle in a behaviour change, so "the abort is gone" and "the output did not
move" are checked as separate claims. tools/kaggle/vad-sgemm-fix/ carries the
harness.
Two follow-ups from running the unit tier for the first time. **python3.** The sanitizer legs run inside a container whose apt line was `build-essential cmake git curl`. test-release-workflow is a python test, so the moment the tier started running there it failed with "/usr/bin/env: 'python3': No such file or directory". Not a code bug — a missing dependency that only became visible once the tests actually ran. (The runner-hosted jobs have python3, which is why they pass it.) **GGML_NATIVE=OFF on the x86 gcc and clang jobs.** These built -march=native, so their numerics depended on whichever CPU the runner drew. The arm64 job in this same file already pinned it, and release.yml pins it for every shipped binary; the x86 legs were the outliers. Being straight about what this does and does not do: it is a REPRODUCIBILITY fix, not a fix for the core_adaln failure. That failure remains OPEN and un-root-caused — see PLAN.md for the five hypotheses eliminated by measurement (clang in general, the clang version, -march=native on a real AVX-512 box, llamafile_sgemm, and the sanitizers' view). It is not reproducible on the hardware available to me because GitHub's runners expose VNNI/BF16 that box does not, and ggml has paths for both. Shipped artifacts were never affected.
… trigger Pinning the ISA turned both clang legs green, which is positive evidence rather than an absence: the trigger is native codegen on the runner's CPU. My earlier 'NOT -march=native' conclusion was a false negative — the box I reproduced on has avx512f/bw/cd/dq/vl but no VNNI/BF16, which GitHub's runners have and ggml has kernels for. Compilable there, not executable. Root cause is an upstream ggml CPU kernel, not core/adaln.h.
The failure was never x86. `ubuntu-22-clang`'s matrix has `include:` entries that do not key on `build:`, so GitHub merges them and the LAST wins — every "ubuntu-22-clang" job actually ran on ubuntu-22.04-arm. Its cmake output says so outright: -- Adding CPU backend variant ggml-cpu: -mcpu=native+dotprod+i8mm+sve+nosme aarch64 with SVE. That explains why no amount of AVX-512 flag-twisting on an x86 box reproduced it, and why GGML_NATIVE=OFF cures it — that drops the +sve. It also explains an existing oddity: ubuntu-22-gcc-arm64 in the same workflow already pins -DGGML_NATIVE=OFF -DGGML_CPU_ARM_ARCH=armv8-a, i.e. someone hit ARM trouble there before and worked around it without writing down why. GitHub's ARM runners are the only SVE hardware available here, so the bisect runs there: native (+sve) to reproduce, an explicit arch WITHOUT +sve to test whether SVE alone is responsible, and GGML_NATIVE=OFF as the control. Plus a per-stage dump — the six modulation views ARE the projection output, so divergence there implicates modulate6's mul_mat while a clean set with a bad `out` implicates the norm chain. Dispatch-only; nothing runs it automatically. test-core-adaln links just Catch2 + ggml, so each config is a small build.
… ARM)
Bumps the ggml pin bfe8ea22 -> 392ac397, which cherry-picks upstream
6aab1bcb "ggml-cpu: fix SVE leftover path in ggml_vec_dot_f32 (llama/24699)"
onto crispstrobe-ops with authorship intact (Tarek Dakhran, 2026-06-26).
THE BUG. ggml_vec_dot_f32's SVE tail was:
sum1 = svmad_f32_m(pg, ax1, ay1, sum1);
svmad_..._m computes a*b + c but MERGES ON THE FIRST OPERAND, so inactive lanes
take ax1 — which the predicated load had zeroed. That wipes the lanes the
preceding leftover loop had already accumulated. Upstream's fix, which is exactly
what this analysis arrived at independently:
sum1 = svmla_f32_m(pg, sum1, ax1, ay1); // merges on sum1, lanes preserved
It only bites when n >= epr && n % epr != 0, which is why normal LLM dims (multiples
of 32) never hit it. Upstream found it via 2D convolutions with kernel size 9; we
hit it with core_adaln's dim=6 (epr=4 at VL=128: 4 lanes accumulated, tail zeroes 2).
HOW WE GOT HERE, because the trail was misleading. test-core-adaln failed only in
build.yml's `ubuntu-22-clang` legs. That job name is wrong: its matrix `include:`
entries do not key on `build:`, so GitHub merges them and the LAST wins — every
"ubuntu-22-clang" job actually ran on ubuntu-22.04-arm. Its cmake line said so:
-- Adding CPU backend variant ggml-cpu: -mcpu=native+dotprod+i8mm+sve+nosme
aarch64 + SVE, never x86 — which is why no amount of AVX-512 flag-twisting on an
x86 box reproduced it. A bisect on real SVE2 hardware then isolated it exactly:
native(+sve) diverged on ALL SIX modulation views (0.21-1.32) plus out (0.66/1.96),
while the same arch WITHOUT +sve and GGML_NATIVE=OFF were 0.000000 everywhere. The
views ARE the projection output, so the fault was in modulate6's mul_mat, not the
norm chain — which is what pointed at ggml_vec_dot_f32.
So the answer to "what do we already fix in our fork" is the opposite here: we were
BEHIND upstream, not ahead. No PR to file.
STILL OPEN: the F16 tail at vec.cpp:334 is `svmad_f16_x(pg, hx, hy, sum1)`. `_x`
leaves inactive lanes UNSPECIFIED after an accumulating loop — correct only by
codegen luck. Upstream has since reworked that path (ggml_sve_f16_fma_widened);
not cherry-picked here because it is a larger change and F16 dots are not what this
bug report covers.
Added while core_adaln was unexplained; keeping it now would permanently blind CI to ISA-specific kernel bugs, which is exactly the class it just caught. The real fix is in the ggml pin (fb7972a): upstream's svmla_f32_m leftover-lane fix. Verified on real SVE2 hardware that native(+sve) is 0.000000 on every stage after the pin bump, so restoring native is safe and restores the coverage. ubuntu-22-gcc-arm64 keeps its own -DGGML_NATIVE=OFF -DGGML_CPU_ARM_ARCH=armv8-a; that predates this work and is not mine to churn.
**ggml pin 392ac397 -> 52165e4c** cherry-picks upstream f69bdbb3 "ggml: fixed Arm SVE usage bug in vec.h, vec.cpp (llama/22841)" (Martin Klacer + Milos Puzovic, Arm, 2026-05-28) with authorship intact. This is the F16 half of the SVE problem, and "upstream reworked that path" deserves spelling out: they did NOT swap svmad_f16_x for a merging variant. They changed the accumulation STRATEGY — F16 accumulators became paired F32 (sum_lo/sum_hi) and every multiply-accumulate now goes through ggml_sve_f16_fma_widened(), which widens F16->F32 before accumulating. The tail uses that same helper, so there is no predicated FMA left to get wrong: the predicated load's zeroed lanes just contribute 0*0. It is a precision fix (F32 accumulation) that removes the predication hazard structurally instead of patching it. Checked, NOT changed: vec.h:396 and :513 still use svmad_f32_m / svmad_f16_x, but both are immediately followed by a PREDICATED STORE (svst1_*(pg, ...)), so inactive lanes are never written back. Upstream carries them verbatim. Not bugs. **The clang matrix.** `ubuntu-22-clang` declared arch via two bare `include:` entries sharing no key with `build:`. GitHub merges those into every combination, so the LAST won and every such job ran on ubuntu-22.04-arm: clang-on-x86 has never been tested, and the arm64 runs were labelled as if they were x86. That mislabelling is what sent the core_adaln investigation chasing "a clang bug" and then AVX-512 on x86 for three rounds when the cause was aarch64 SVE. `arch` is now a real matrix dimension and `include:` only maps arch -> runner, which is what it is for: 4 jobs instead of 2, and two of them are coverage that never existed.
…matrix F16: upstream f69bdbb3 (Arm) replaces F16 accumulators with paired F32 via ggml_sve_f16_fma_widened, removing the predicated FMA entirely rather than swapping the intrinsic. vec.h:396/:513 checked and left alone — predicated stores mean inactive lanes are never written back, and upstream carries them verbatim. Matrix: arch is now a real dimension, so clang-on-x86 is tested for the first time and arm64 runs are no longer mislabelled as x86.
…take MASTER-AUDIT.md tracks what we send upstream. Nothing tracked the other direction, which is why core_adaln failed on ARM for weeks: our vendored ggml predated TWO upstream SVE fixes and nobody was looking. crispstrobe-ops is 11 ahead / 435 behind, base v0.10.2 vs upstream v0.17.0. Of the 11, two are today's cherry-picks, so nine are genuinely ours. Priorities are ranked by what CrispASR actually runs, not by upstream's own severity. Priority 1 is conv1d/im2col on Metal (b40b6928 says "audio models" in its subject; M1 is the primary dev box), a simd_gemm TAIL-COLUMN indexing fix which is the same class of bug as the SVE one we just fixed, quantized concat, and rms_norm_back under in-place aliasing. Then Vulkan (we ship it on Windows and have four standing Vulkan-only TTS failures) and CUDA. Two collisions found that must be settled before any sync: * upstream now has its own col2im_1d (fda9d536). We carry ours (PR #160) behind a "MUST RE-APPLY after ggml bump" marker in ggml-metal-device.m. They will collide. upstream-prs/20 is therefore SUPERSEDED — adopt upstream's, drop ours. * conv_transpose_1d arrives twice: a056a26f is our OWN merged PR (ggml#1477) and we also carry it locally. Not a gap, but it will present as a conflict. Recommendation: take Priority 1 now, then do a real sync rather than more cherry-picks. Seven minor versions of drift plus manual re-apply markers is what produced the SVE miss; each further cherry-pick applies against more foreign context. MASTER-AUDIT.md is itself 435 commits stale and wants re-running.
Russian ASR, 220 M params. One runtime serves all four ASR revisions —
`ctc` / `rnnt` (33 Cyrillic chars, lowercase) and `e2e_ctc` / `e2e_rnnt`
(SentencePiece, punctuation + casing + ITN) — the GGUF carries the head
type and tokenizer kind.
Per-stage parity vs a PyTorch reference dumped from the upstream
`modeling_gigaam.py`, on GigaAM's own example.wav: F16 is cos 1.000000 at
mel, pre-encode, all 16 conformer layers, the encoder output and every
head stage, with a byte-identical transcript. Q8_0 holds the transcript
byte-identical on all four variants (encoder cos 0.997-0.999), so it is
the registry default; Q4_K stays exact on the charwise variants and costs
a few capital letters on the SentencePiece ones.
Three blueprint details that the shapes do not catch:
* RoPE is applied to the block INPUT before the Q/K/V projections, so
Q = Wq*RoPE(x), K = Wk*RoPE(x) and V = Wv*x — V is unrotated.
* The rotary base is pos_emb_max_len = 5000, not 10000, because
RotaryPositionalEmbedding's second positional arg is `base`.
* conv.batch_norm is an nn.LayerNorm (conv_norm_type='layer_norm') —
affine only, no running stats to fold.
Two integration bugs the diff harness cannot see (it ends at the logits):
* the charwise revisions auto-enabled FireRedPunc, a Chinese/English
restorer, which injected full-width CJK punctuation into Russian
("надеждой， сладкой ... зеленый。"). CAP_PUNCTUATION_NATIVE is now
declared for every revision; an explicit --punc-model still applies.
* `-l auto` ran a whisper-tiny LID pass on a Russian-only model (#227);
sole_language() == "ru" skips it (21.3x -> 43.4x realtime on M1).
The FFN / conv / macaron halves reuse core_conformer::BlockWeights; only
the attention differs, so the block builder is local rather than a rel-pos
core_conformer::build_block call. The quantizer keeps encoder.pre.* (the
mel is un-normalized log-mel) and joint.* / decoder.* / head.ctc.* at
source precision.
Weights: https://huggingface.co/cstr/gigaam-v3-GGUF
Notes + follow-ups: docs/gigaam/PLAN.md
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… equivalent
An earlier draft said 'adopt upstream's and delete ours'. Comparing the two
headers during the merge shows that is wrong and would have been silent:
upstream scatter-add, p0 = crop BOTH sides, T_out = ... - 2*p0
ours #160 gather, p0 = LEFT offset, T_out = ... - p0
Same name, same signature, different semantics. src/core/conv.h:166 passes
crop_left as p0 and trims crop_right with a view, i.e. it relies on ours.
Swapping in upstream's compiles and links and changes ConvTranspose1d output
length in every TTS decoder — wrong audio, no error.
Adopting upstream's requires rewriting convt1d_decomp (p0=0 + views for both
crops) and validating against TTS audio before any pin bump.
Attempted the full merge: crispstrobe-ops <- ggml-org/ggml master. 435 commits, 19 conflicted files, 53 hunks. Resolved 3 and stopped deliberately. Resolved: gguf.cpp (upstream implemented our empty-key rejection independently, so 1dc4cb93 is superseded), test-quantize-fns.cpp (upstream superset), and ggml-metal-device.m (kept BOTH — our pipeline cache and upstream's new device_id_parse are adjacent additions, not an overlap). Two blockers make the rest unsafe to finish blind, and both fail SILENTLY: 1. GGML_OP_* is an ORDERED enum with parallel name/symbol tables in ggml.c. Ours asserts COUNT==99, upstream ==101; both sides inserted ops at different positions. Merged wrong, ops dispatch to the WRONG KERNEL with no compile error. The enum and both tables have to be re-derived entry-by-entry. 2. col2im_1d is TWO DIFFERENT OPS sharing a name and signature (scatter-add with symmetric crop vs our gather with left offset). src/core/conv.h depends on ours; taking upstream's changes ConvTranspose1d output length in every TTS decoder, compiling cleanly the whole way. It drives 4 of the conflicted files. The inventory records per-file guidance for the remaining 16 so this does not have to be re-derived: which hunks are pure additions of ours to keep (the 588-line CUDA peer/VMM block, the im2col grid.y clamp), which are genuine overlaps needing judgement (fattn RDNA4 gate, cpy.cu GH#65 vs upstream's rework, the issue-#38 im2col_type selection), and which are blocked on blocker 2. CrispASR's pin stays at 52165e4c — it has both SVE fixes and is the known-good tree. Bumping it from a partially-resolved merge is exactly the wrong move.
`tools/check-backend-wiring.py` reported PASS, but walking docs/contributing.md by hand found four things it does not check: * the session C ABI never forwarded the decode knob. The CLI adapter called gigaam_set_max_symbols(), crispasr_c_api.cpp did not — so --max-new-tokens worked on the CLI and silently no-op'd for every binding and the server. This is the multi-surface trap the guide calls the #1 recurring bug (#292/#315): a fix in one surface never reaches the others. * docs/environment-variables.md listed all five CRISPASR_GIGAAM_* gates nowhere; every other backend's are enumerated there. * tools/reference_envs/gigaam/requirements.txt was missing, so `tools/bootstrap_ref_env.sh gigaam` could not build the dump env. * docs/diff-harness-coverage.md is generated by audit_diff_coverage.py and had never been regenerated after the backend landed. The audit script checks factory / c_api dispatch / available_backends / feature-matrix / registry and is worth running, but it does not read contributing.md's per-section requirements. Both passes are needed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
First half of the FoxNoseTech/diarize port: the 256-dim speaker
embedder its clustering consumes. Clean-room from the architecture —
the upstream (wenet-e2e/wespeaker, Apache-2.0) was read to establish
the spec and is imported by the reference dumper as an ORACLE, but no
expression from it is reproduced, so the repo stays MIT.
Per-stage parity vs that oracle on samples/jfk.wav:
fbank cos_mean 0.999999
stem/layer1-4 cos_mean 0.99997 - 0.999995
stats cos_mean 0.999999
embedding cos_mean 0.999997, cosine(emb, ref) = 0.99999747
Details that decide correctness, all traced to the driving code rather
than assumed:
* int16-SCALE waveform — wavform_norm defaults to False upstream, so
kaldi.fbank sees a +/-32768 signal; hamming window, per-utterance
CMN.
* the 2-D map is HEIGHT=freq, WIDTH=time. TSTP reduces over time then
flattens (channel, freq) with freq fastest, and seg_1's 5120 columns
are in that order — transposing the map silently permutes the stats
vector against its weights.
* TSTP std uses torch's UNBIASED (n-1) variance, +1e-7 inside sqrt.
* output is seg_1(stats) raw: no ReLU, no BN, no L2 norm.
* BN folded into every conv at convert time (219 -> 74 tensors); the
ArcMargin `projection` head is training-only and dropped
(11.25M -> 6.6M params).
Two ggml findings worth recording:
* F16 conv kernels trip GGML_ASSERT(src1->type == GGML_TYPE_F32) in
the CPU conv_2d_direct path. Conv kernels are pinned to F32 by the
converter; F16 on the 2-D linear is fine and gives an identical
embedding (0.99999744).
* ggml_add1(ggml_new_f32(...)) is invalid in a no_alloc graph context
— ggml_new_f32 writes to tensor->data at build time, which is null.
ggml_scale_bias folds the constant into the op instead.
cos_min on the post-ReLU maps is a brittle statistic (36% of stem_out's
rows are entirely dead); the harness now prints the dead-row counts so
a cos_min of 0.000000 is interpretable rather than alarming. Judge this
backend on cos_mean and on `embedding`.
Live tests cover the property the diarizer actually rests on: two
windows of the same speaker score cos 0.595 against 0.100 for a
different speaker. ⚠ samples/multispeaker.wav opens with all 11 s of
samples/jfk.wav verbatim, so a cross-speaker window has to be taken
past that boundary — the first draft of the test scored cos = 1.000000
and was measuring jfk against itself.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Clustering half of the FoxNose diarization recipe: PCA(8) -> full-covariance GMM BIC sweep -> cosine-affinity spectral clustering -> silhouette refinement -> spherical centroid refinement. Clean-room, so the repo stays MIT. There was nothing to translate: the upstream clustering.py is a sequence of scikit-learn CALLS, and C++ has no sklearn, so the numerics are written from the published algorithms. What comes from upstream is the recipe and the tuned constants (p10 >= 0.16, 0.04*log k, PCA=8, the k-window, n_init/max_iter) — parameters and facts, not copyrightable expression. Bit-exact sklearn parity is unachievable here and always would be: its k-means++ seeding, GMM init and ARPACK eigensolver all ride its own RNG stream. So the gate is known-answer unit tests plus DER, not label equality. Determinism across runs IS guaranteed — everything is explicitly seeded. Two measured findings, both recorded as tests rather than folklore: * The upstream ceiling of n/2+1 components is far too loose for a FULL covariance. A d x d covariance is not estimable from fewer than d+1 points, so past n/(d+1) every extra component fits a near-singular Gaussian whose density diverges, BIC falls monotonically and the sweep runs away to max_k. Measured on 3 blobs (n=60, pca_dim=8): BIC dropped 881 -> 138 straight through k=10. reg_covar does not save this — sklearn's 1e-6 default is negligible against PCA noise-component variances of ~0.03. Bounded by n/(d+1); that alone fixed true k=2. * The [k-2, k+3] silhouette window recovers a BIC UNDER-count (true 4/5/6 anchored at 2/2/3, all recovered exactly) but not a large OVER-count (true 3 anchored at 8 leaves the window at [6,10]). Silhouette itself is the reliable half — it scored the true k at 1.0390 against 0.68/0.79 for its neighbours. CRISPASR_DIARIZE_FULL_K_SEARCH=1 scores the full range instead: 5/5 exact vs 4/5 for the window. The full search is GATED OFF and the upstream window stays the default until the DER harness has a say — gate the new path, do not delete the working one. The unit test pins BOTH arms so neither regresses unnoticed. 23 cases / 256 assertions, all hermetic: no model, no audio, no network. Synthetic blob centres point in random directions rather than along coordinate axes, because axis-aligned centres make the between-blob signal rank k-1 and hand the GMM six dimensions of near-degenerate noise — which is what surfaced the ceiling bug in the first place. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Second half of the clustering stack. Clustering labels each embedding
window independently, so its output flickers — a window straddling a
pause, or one noisy embedding, flips the label mid-utterance. This
imposes temporal continuity:
majority_label / smooth_window_labels 3-window majority, ties keep the
original label
speaker_centroids L2-normalised centroid per label,
mean of NORMALISED members
viterbi_smooth max total score with a 0.18
penalty per label change
smooth_segment_temporal score windows against centroids,
anchor the original label by 0.02,
Viterbi-decode
collapse_short_islands absorb an A-B-A island <= 1.2 s
restore_sustained_runs put back original runs > 1.2 s that
Viterbi flattened
Clean-room on the same basis as core/spectral_diarize: Viterbi decoding
and majority filtering are textbook; what comes from upstream is the
order of operations and the constants.
Three behaviours that are easy to get subtly wrong and are pinned by
tests rather than left to comments:
* a TIE is not a majority. Returning a winner would let the 3-window
filter manufacture a label exactly at a speaker boundary — the one
place the sequence is genuinely ambiguous.
* centroids average the NORMALISED members. Averaging raw vectors lets
a loud window dominate its speaker's centroid; the test uses members
with 5x magnitude differences so the two disagree.
* only an A-B-A sandwich is flicker. A-B-C is a genuine three-way
change and must survive however short B is.
17 cases / 30 assertions, hermetic. The switch penalty is bracketed: a
0.10 score margin does NOT flip the label at the shipped 0.18 penalty
but DOES at penalty 0, so retuning the constant fails the test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ties the pieces together — speech regions -> sliding embedding windows -> clustering -> temporal smoothing -> merged turns — and adds the metric that judges the result. DER is the right gate here because nothing else fits: there is no transcript to compare as in ASR, and no per-stage reference to diff as in a model port (the clustering is weight-free and sklearn parity is unachievable). core/der.h follows the NIST/dscore conventions: a collar around reference boundaries, an optimal 1:1 speaker mapping, and total reference speech as the denominator. The mapping is EXACT by enumeration up to 8 speakers and greedy beyond, and says which it used rather than letting a caller mistake an approximation for a guarantee. The pipeline takes its embedder as a FUNCTION POINTER rather than linking one in. That keeps core/ free of model dependencies, lets the wiring layer pick WeSpeaker or TitaNet — and above all makes the orchestration testable with a synthetic embedder whose speakers are known by construction. A model-driven test cannot separate "the pipeline is wrong" from "the embedder is weak"; this one can. End-to-end on a synthetic 8-turn, 2-speaker timeline: DER 0.0. Two things the tests pinned down: * The synthetic speaker centres are ORTHOGONALISED, and that is faithfulness rather than convenience. Two random 32-D directions have cosine std ~= 1/sqrt(32) ~= 0.18, so a draw can exceed kSingleSpeakerSimP10 = 0.16 and trip the single-speaker veto on a genuinely two-speaker timeline — the first version of this test did exactly that and reported 1 speaker at DER 0.5. Real WeSpeaker embeddings measure ~0.10 cross-speaker, well under the threshold. Worth knowing: the veto is one global threshold, so two similar-sounding speakers can still be merged. * "Total confusion scores 1" is wrong as usually stated — with a 1:1 mapping, swapping the labels of a symmetric 2-speaker reference is FREE. The test forces real confusion by collapsing the hypothesis onto one speaker instead. 11 DER/pipeline cases, all hermetic. Whole new suite: 375 assertions across 57 cases (der, spectral, smooth, wespeaker-params). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
--diarize-method foxnose --diarize-embedder <wespeaker.gguf>, with --diarize-max-speakers / --diarize-num-speakers. Plugs into the existing crispasr_diarize_segments contract: the caller's segments ARE the speech regions (they come from ASR/VAD upstream), so this method deliberately runs no VAD of its own — re-segmenting would duplicate work and desynchronise labels from the segments they attach to. --diarize-embedder was already claimed by the TitaNet remap at THREE call sites (cli.cpp, crispasr_run.cpp, crispasr_server.cpp), which tried to load the WeSpeaker GGUF as TitaNet and failed with a confusing "block_repeats/kernels array size mismatch". All three now ask one shared predicate, params.diarize_embedder_is_foxnose(), so a fourth cannot drift. Gating only cli.cpp was not enough — the unified crispasr_run path is the one actually taken, and only an end-to-end run surfaced that. MEASURED on samples/multispeaker.wav (31.5 s, opens with all 11 s of samples/jfk.wav then changes speaker): num_speakers=2 pinned 2 speakers, boundary at 10.5 s vs true 11 s auto, max_speakers=4 2 speakers, same correct turns auto, max_speakers=8 8 speakers, heavy flicker auto, max=8 + FULL_K_SEARCH 8 speakers, gate does not help So the embedder, clustering and smoothing are right; AUTOMATIC SPEAKER COUNTING is the open problem. Silhouette saturates and rises monotonically to the ceiling on real embeddings: within-speaker cosine is ~0.595 against ~0.100 cross-speaker, so splitting a speaker cuts the intra term sharply while the inter term barely moves, and the +0.04*log(k) bonus then pushes to the top of the range. Upstream defaults max_speakers to 20, which is why their README concedes it struggles past 8 speakers. Settling that needs DER on labelled audio, of which the repo has none — see docs/foxnose-diarize/PLAN.md. Until then the honest advice is a conservative --diarize-max-speakers (4-6) or pinning the count. Known limitation, documented: labels are attributed at SEGMENT granularity, so an ASR emitting one 26 s segment across several speakers gets a single label however good the turns are. Splitting at turn boundaries, as the pyannote path does, is the follow-up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…-speakers THIRD_PARTY_NOTICES.txt gains its first CC-BY-4.0 entry. The WeSpeaker weights carry an ATTRIBUTION REQUIREMENT, and several downstream projects (including the issue's own reference) describe them as Apache-2.0 — the wenet-e2e/wespeaker CODE is Apache-2.0, the published WEIGHTS are not. The notice also records that no upstream source is incorporated: the converter, runtime and clustering are independent implementations, and the upstream packages are used only as measurement oracles by the reference dumper. --diarize-max-speakers now defaults to 4 for this method rather than upstream's 20, because the measurement says a loose bound is actively harmful: on samples/multispeaker.wav a bound of 8 yields 8 speakers with heavy flicker while 4 yields the correct 2. An explicit flag always wins, via a diarize_max_speakers_explicit flag following the same contract as max_new_tokens_explicit (#292) — a per-method default must never override a value the user chose. docs/architecture.md#foxnose-diarize documents the pipeline, the licensing position, why sklearn parity is unachievable, and BOTH known weaknesses: speaker counting, and segment-granularity label attribution. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Without this the method was of limited use: labels attached at the CALLER's segment granularity, so an ASR emitting one 26 s segment across several speakers received a single label however good the derived turns were. Measured on samples/multispeaker.wav, both ASR segments collapsed to "(speaker 0)". crispasr_diarize_segments now returns the turns it derived through a new optional out_turns parameter (FoxNose only — the other methods label caller segments directly and leave it empty), and the CLI splits any multi-speaker segment at word-aligned boundaries. Only the per-word labelling is new: a turn-interval lookup replaces pyannote's posterior scoring, and everything after it reuses the existing group_words_into_speaker_runs grouping and sub-segment emission. After: the first slice reads 0 -> 1 -> 0 across the ~11 s boundary, which matches the whole-file pipeline's turns. ⚠ Documented, not fixed: labels are still not consistent ACROSS slices. Long audio is cut into slices and each is diarized independently, restarting its numbering at 0 — so the final turn reads "speaker 0" locally where the whole-file pipeline says SPEAKER_01. The pyannote path solves this with a global cache computed once over the full audio (#107); FoxNose needs the same. Until then this method is trustworthy only on audio short enough to form a single slice. Recorded in docs/architecture.md#foxnose-diarize and docs/foxnose-diarize/PLAN.md rather than left to be rediscovered. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Per-slice diarization cannot give consistent speaker identities: each slice clusters independently and restarts numbering at 0, so "speaker 0" in one slice is a different person from "speaker 0" in the next. On samples/multispeaker.wav (2 slices) the final turn came out speaker 0 where the whole-file pipeline says SPEAKER_01. FoxNose now runs ONCE over the whole audio after transcription (crispasr_apply_foxnose_global), using the final segment list as its speech regions. The pyannote path solves the same problem with a posterior cache computed BEFORE transcription (#107); foxnose cannot do that, because it needs the segments as its speech regions and they do not exist yet — so it hooks the existing global speaker stage instead. The per-slice path stands down via params.diarize_foxnose_global, which also means the embedder is loaded once instead of once per slice. CLI output now matches the whole-file pipeline's turns exactly: 0.28-10.84 speaker 0 (truth 0.00-10.50 SPEAKER_00) 11.64-15.88 speaker 1 (truth 10.50-15.90 SPEAKER_01) 17.04-26.80 speaker 0 (truth 15.90-26.70 SPEAKER_00) 26.52-31.52 speaker 1 (truth 26.70-31.50 SPEAKER_01) The last row is the fix — it read speaker 0 before. Scope note: this is the unified crispasr_run path. The legacy cli.cpp whisper path still diarizes per slice, the same fallback situation the pyannote cache has there. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The upstream estimator does not work on real speaker embeddings. Silhouette saturates and climbs monotonically to whatever ceiling it is given: within-speaker cosine is only ~0.595 while cross-speaker is ~0.100, so splitting a real speaker keeps cutting the intra-cluster term while the inter-cluster term barely moves. On samples/multispeaker.wav with max_speakers=8 it reports 7-8 speakers where the truth is 2. Replaced with the eigengap of the normalised Laplacian, the standard estimator for spectral diarization: it reads cluster structure off the spectrum instead of scoring partitions, so saturation cannot arise. A naive eigengap does NOT work here, and the reason is worth recording: the cosine affinity (cos+1)/2 is DENSE — around 0.5 even for unrelated windows — so the graph is nearly complete, one eigenvalue dominates, and the largest gap always falls at k=1. First implementation duly reported one speaker for all five synthetic cases. Row-wise thresholding fixes it: keep each row's strongest 15%, ATTENUATE the rest by 0.01 rather than deleting them so the graph stays connected, then symmetrise with an elementwise max. configuration synthetic (5 true-k) real, max=8 BIC + silhouette (upstream) 4/5 exact 7-8 speakers eigengap, no thresholding 0/5 (always k=1) — eigengap + row thresholding 5/5 exact 2, correct turns It also costs less — one eigendecomposition instead of a GMM sweep plus max_k spectral runs. Winning on quality AND speed is what justifies flipping the default; CRISPASR_DIARIZE_COUNT=bic keeps the upstream path selectable, gated rather than deleted. Because eigengap is robust to a loose bound, --diarize-max-speakers goes back from the defensive 4 to 8, so a genuine 5-6 speaker meeting is reachable again. ⚠ Caught by rebuilding: the existing test asserted the DEFAULT gets k=3 wrong, so flipping the default inverted its meaning — and it kept passing against a stale binary until the test target was rebuilt explicitly (HARD RULE #8's sibling trap). It is now split in two: one test pins the default reaching the truth via eigengap, the other pins the gated BIC path still needing FULL_K_SEARCH to get there. Evidence limits, stated plainly: five synthetic configurations and one 31.5 s real clip. No DER on labelled audio yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…sured
BLUEPRINT PARITY — the check that was missing. The upstream Python
pipeline (pip install diarize==0.1.2) was run on the same
samples/multispeaker.wav and compared against this port.
With the speaker count pinned to 2 on both sides the speaker assignment
is IDENTICAL and boundaries agree within ~1 s. Scored with the DER
harness (0.25 s collar, optimal 1:1 mapping), upstream as reference:
missed 0.00 s
false alarm 1.05 s
confusion 0.00 s
DER 3.93 %
Zero speaker confusion: wherever both assign a speaker, they agree. The
whole residual is false alarm and it is explained — upstream runs its own
Silero VAD and drops silence gaps, while this port tiles the caller's
speech regions contiguously. Different segmentation SOURCE, not a
diarization disagreement.
On automatic counting this port is BETTER than upstream: on the same clip
upstream emits 11 speakers across 25 segments at its default
max_speakers=20, this port emits 2. The gated CRISPASR_DIARIZE_COUNT=bic
path reproduces upstream's failure mode (7-8), which is what shows the
port is faithful — the improvement comes from the eigengap switch, not
from a divergence in the shared parts.
Session C ABI: crispasr_diarize_opts_abi gains foxnose fields, method
range 0..4. The struct is laid out BY HAND in bindings/go's cgo preamble,
so fields are append-only and the Go layout is updated in the same commit
— a missing field there means the C side reads past the end of a
Go-allocated struct. Go gets DiarizeSegmentsFoxNose + FoxNoseOpts;
DiarizeSegments keeps its old signature.
Registry: cstr/wespeaker-resnet34-lm-GGUF, so --diarize-embedder auto
--auto-download works end to end (verified from a cleared cache).
Uploaded files verified by SHA256, not size:
wespeaker-resnet34-lm.gguf 2461a2b9…8d26
wespeaker-resnet34-lm-f32.gguf edd96a69…3499
Card carries license: cc-by-4.0 with attribution, as CC-BY requires.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
I made eigengap the default on five synthetic blob configurations (5/5
exact vs BIC's 4/5) and one 31.5 s clip. Both were unrepresentative —
samples/multispeaker.wav is the same speech re-read by different
speakers, an unusually easy case — and a real benchmark contradicts it.
8 VoxConverse dev files, HUMAN labels, 0.25 s collar, optimal 1:1
mapping, pooled over 1109 s of scored reference speech:
upstream Python diarize 0.1.2 miss 0.6 fa 4.7 conf 29.0 3.1 %
this port, bic miss 0.0 fa 26.1 conf 32.7 5.3 %
this port, eigengap miss 0.0 fa 26.1 conf 101.3 11.4 %
Eigengap systematically UNDER-counts on real speech —
reference 4 7 2 5 5 4 4 5
eigengap 3 5 2 3 3 2 2 3
bic 4 6 3 5 4 3 2 5
— and the confusion term triples. Default is back to bic;
CRISPASR_DIARIZE_COUNT=eigengap keeps it selectable, since it IS better on
well-separated data and costs less. The unit test now pins the default so
a synthetic win cannot quietly flip it again.
On the remaining 5.3 % vs 3.1 %: our false alarm is 26.1 s against 4.7 s
because the benchmark driver hands the pipeline WHOLE FILES as one speech
region while upstream runs Silero VAD first. Substituting upstream's false
alarm gives 3.4 %, i.e. parity within ~0.3 points. The real CLI path has no
such handicap — it takes the caller's ASR/VAD segments. Adding a VAD stage
to the benchmark driver is the obvious next step.
This is what the earlier "no DER on labelled audio" caveat was protecting
against, and it caught a wrong call. Synthetic known-answer tests are
necessary but they are not an accuracy benchmark.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous benchmark handed our pipeline WHOLE FILES as one speech
region while upstream ran Silero VAD first, charging us 26.1 s of false
alarm the real CLI path never incurs (it takes the caller's ASR/VAD
segments). Re-run with the same Silero VAD and upstream's parameters
(threshold 0.45, min speech 200 ms, min silence 50 ms, pad 20 ms), over 8
VoxConverse dev files against HUMAN labels:
upstream Python diarize 0.1.2 miss 0.6 fa 4.7 conf 29.0 3.07 %
this port, bic + Silero VAD miss 0.0 fa 9.0 conf 26.5 3.18 %
this port, bic, no VAD miss 0.0 fa 26.1 conf 32.7 5.27 %
0.11 points apart, and our speaker CONFUSION is lower (26.5 vs 29.0) —
the residual is false alarm, not diarization. Estimated speaker counts now
differ on one file of eight:
reference 4 7 2 5 5 4 4 5
this port 4 6 2 5 4 4 2 5
upstream 4 7 2 5 4 4 2 5
This confirms the earlier arithmetic estimate of 3.4 % (actual 3.18 %) and
closes the acceptance question the port has carried since the start: the
port is faithful, and the gap was the measurement setup.
tools/der_score.py carries the scorer (the Python twin of src/core/der.h)
so the benchmark is reproducible for pipelines that are not CrispASR; the
VoxConverse recipe is in docs/foxnose-diarize/PLAN.md.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ing.md `tools/check-backend-wiring.py` reported PASS and every test binary I had been running was green, but walking the checklist by hand found four things neither covers: * bindings/go/whisper.go cgo LDFLAGS were STALE. wespeaker is a new add_library reachable from crispasr-lib, so the hand-maintained -l list needed regenerating — this is the CI-enforced `cgo-ldflags-drift` check and it would have gone red. (The drift checker itself failed to run at first against a stale temp graphviz dir from an earlier port; clearing it surfaced the real answer.) * tests/env-live-tests.sh had no CRISPASR_MODEL_WESPEAKER, so the live tests could not run from ctest at all. * docs/environment-variables.md documented none of the new gates (CRISPASR_DIARIZE_COUNT / _FULL_K_SEARCH, CRISPASR_WESPEAKER_*). * docs/diff-harness-coverage.md is generated by audit_diff_coverage.py and had not been regenerated since the wespeaker reference backend landed. Also adds the README row for foxnose diarization, with its measured DER. Verified after: 1213/1213 unit tests pass (the earlier "12 failed" was 12 NOT-RUN binaries I had never built, not failures); wespeaker live 11/11 and gigaam live 12/12 through ctest; LDFLAGS drift check clean; wiring audit PASS. wespeaker is deliberately NOT a --backend: it is a component consumed by crispasr_diarize.cpp, exactly like pyannote-seg, so the 12-point backend checklist applies only in part. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four commits, one of which matters to #324: 00285218 cpu: build conv_2d/conv_3d im2col patches in vec_dot_type (fixes F16 conv_2d) 6572b47f ci: hosted-runner CI + carried-patch guard on crispstrobe-ops 6397bc5c test: fix heap overflow in test-quantize-fns scratch buffers 14b4971d cuda: fix batched indexing in our k-quant GET_ROWS patch 00285218 removes the GGML_ASSERT(src1->type == GGML_TYPE_F32) that made convert-wespeaker-to-gguf.py pin every 4-D conv kernel to F32. F16 conv now runs and is numerically fine (cosine 0.99999724 vs the oracle, and the GGUF drops 23.9 MB -> 13.3 MB), but it is 2.2x SLOWER on CPU (297 vs 133 ms per 1.2 s window, same loop, back to back), so the F32 pin STAYS. The comment in the converter now says why it is a speed choice rather than a crash workaround. ggml/ci/check-crispasr-patches.sh reports 24 patches present. 1213/1213 unit tests pass against the new pointer.
…round The previous commit's message said this had been written; it had not been. Both the converter comment and the model card still described the F32 pin as a way around GGML_ASSERT(src1->type == GGML_TYPE_F32), and pointed at a note in wespeaker.cpp that does not exist. The fork's 00285218 removed that assert, so the old reason is gone and the real one needs recording: F16 conv kernels work and halve the file, they are just 2.2x slower on ggml's CPU conv path (297 vs 133 ms per window, same loop, back to back, M1).
AGENTS.md is the only agent-instruction file in this repo that reaches a clone: CLAUDE.md is gitignored (per-checkout), and the development guide lives BESIDE the repo and is not tracked anywhere. So the two documents that explain how to work here were invisible to anything but the maintainer's own machine, and the guide got read only when a human typed its path by hand. Adds a READ-FIRST pointer phrased for both audiences: on a maintainer checkout the sibling ../crispasr-crispembed-dev.md is there and must be read in full (explicitly NOT summarised here — it wins any conflict); on an outside clone it is absent, nothing is missing from the build, and the map below is enough. The map is here rather than duplicated per-checkout so there is one copy to drift. It also front-loads the two habits that keep costing time: PLAN/HISTORY prose goes stale (audit the code, not the note), and CLI auto-detection proves nothing about the bindings (#335). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ernel_im2col_flat on sync/upstream-v0.17 (89a2039d) The v0.17 sync dropped the OCC variant; the successor covers batch-1 AND the conv_2d_dw lowering (the class #23 missed), measured 2.3x on CrispEmbed PP-OCRv6 rec / 1.6x layout_detect, byte-identical. CrispASR activation needs a pin bump to >= 89a2039d plus fresh melotts/moonshine parity+timing runs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The per-candidate probe line used printf "%.60s", which cuts at 60 BYTES. For Greek, Arabic or CJK that lands inside a multi-byte character and puts a severed lead byte on stderr. Not cosmetic, and not theoretical: it killed the republish kernel outright. Python's subprocess decodes stderr as text, so the 14-way probe raised UnicodeDecodeError on 0xce — the Greek candidate's lead byte — and took the run down after its uploads had completed. Any consumer reading our stderr as text hits the same thing, and only ever for non-Latin languages, which is why local Latin testing never saw it. utf8_prefix() walks back to a codepoint boundary. The test loops every cut point of a Greek, an Arabic and a CJK string: naive truncation is invalid UTF-8 at 47 of 107 offsets for the Greek fixture, so the guard genuinely goes red on the old code. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…14-probe claim Two things, both found by finally RUNNING the converter end-to-end. 1. GGUFWriter(path, arch) already writes general.architecture, and the converter wrote it again. gguf-py versions that reject duplicates raise ValueError; older ones only warned and overwrote, which is how this survived — the converter had never been run on a strict version. It now completes: 2104 tensors, 4.14 GB, supported_languages=['en','ar'], max_clip_s=35, and the resulting model transcribes Arabic correctly and enforces its own whitelist. 2. Retracts the "accuracy degrades past ~4 candidates" claim I documented in 0b85808. Measured on the REAL 14-language base model (republished with its whitelist), the forced 14-way probe is CORRECT on both clips: jfk.wav -> en (228, p=0.169), an Arabic clip -> ar (292, p=0.254). The earlier "picks fr" came from forcing a 14-language list onto the TWO-language Arabic finetune, which translates when asked for a language it lacks; the real base model's fr probe code-switches instead ("Et so, my fellow Americans...", agreement 0.00, score 57) and loses. The <=4 ceiling stays, for the reason originally intended: cost. 14 probes take 37 s on an M1 against ~1 s for whisper-tiny. The scoring soft spot is real and stays documented — a fluent translation CAN outscore repetitive truth — it is just not what the candidate count controls. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…robe claim retracted Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… (cohere) Also: forcing a fake capability list does not simulate having the capability — it exercises a different failure mode. That is what made my first 14-candidate probe measurement wrong. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…1 im2col occupancy win the v0.17 sync dropped The sync removed CRISPASR_METAL_IM2COL_OCC (dispatch reworked upstream), silently un-fixing the melotts/piper P0-hifigan row. The re-derivation (kernel_im2col_flat, authored on the CrispEmbed PP-OCR profile where it is 2.3x) covers the old batch-1 case via N*KH*KW<128 plus the conv_2d_dw lowering (IC==1); CRISPASR_METAL_IM2COL_FLAT=0 restores the standard kernel. Validated on M1 (interleaved pairs): melotts hifigan_decode 2.02 s -> 1.10 s (1.85x), round-trip clean (WAV byte-compare is invalid for melotts — the VITS flow is stochastic, legacy-vs-legacy already differs); moonshine transcript byte-identical, RTF neutral+; paraformer transcript byte-identical, its 2 im2col nodes 8.5-9.1 -> 0.4-1.0 ms/node (wall pairs were load-noise, mechanism checked per-op). 1473/1473 unit tests. Evidence: PERFORMANCE.md top entry (2026-08-06). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…d pin (melotts 1.85x, moonshine/paraformer byte-identical) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MetadataDetails gained sub_type in newer gguf-py; the version installed here is (type, value, description) and raised TypeError, so the script could not run at all. Pass the kwarg only when the field exists — omitting it is safe, add_key_value infers the element type from the first element. Verified on a synthetic glm-asr GGUF: merges land as ARRAY/STRING with both the pair-form and string-form inputs normalised, and arch, vocab and tensors pass through unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Cohere Transcribe transcribes NON-SPEECH as text, and our chunker splits
long audio at RMS minima without ever DROPPING the quiet parts. Measured
on the Arabic q4_k-imatrix build:
10 s of pure digital silence -> "And I'm going to go ahead and do that."
jfk.wav + 20 s of silence -> that same sentence appended to an
otherwise perfect transcript
With VAD both disappear. (Low-level noise and a 440 Hz tone produce
nothing, so it is silence specifically.)
A/B'd on real speech before flipping, and VAD wins there too: on a 60 s
FLEURS clip the un-VAD'd run cut mid-sentence, garbled a clause ("how
acidic, basic, alkaline the cabbage juice is" for "...the chemical is")
and DROPPED an entire sentence that the VAD run recovers. So this is a
content-recovery win, not just a silence guard.
Uses the existing long-audio safeguard: arms only above 30 s and only
when no --vad / --vad-model / --chunk-seconds was given. Verified: 31 s
input auto-enables and comes back clean, 11 s does not arm, and an
explicit --chunk-seconds still wins.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The encoder output and cross-attention KV depend only on the AUDIO —
the language enters solely through the decoder prompt — and encode is
87% of a pass (881 ms vs 113 ms decode, M1 q4_k-imatrix on 11 s
jfk.wav). So probing N languages over one clip was paying N encodes for
one clip's worth of information.
Two `{` become `if (!reuse_enc) {` — no re-indentation, and the cross-KV
free+realloc lives INSIDE the second block, so skipping it keeps the
previous allocation live, which is exactly what gets reused.
A/B on 14 candidates, back-to-back on a quiet box, and repeated in
reverse order to rule out cold-start bias:
OLD 12 s, 12 s NEW 5 s, 4 s output byte-IDENTICAL
Normal transcription is untouched (the flag defaults false): short,
long-chunked and Arabic transcripts all match their pre-change
baselines. CRISPASR_COHERE_PROBE_REUSE_ENC=0 restores one encode per
candidate.
The invariant — reuse only across calls over the SAME samples — is
documented on the context field, and cohere_detect_language is the only
setter and clears it in the same function.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…e, merges tool Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Handed a span with no signal, this model does not return nothing — it invents. 10 s of zeros decodes to "And I'm going to go ahead and do that."; and since the chunk loop re-enters transcribe_ex per chunk, a long file whose trailing 30 s window is silent gets that sentence appended to an otherwise perfect transcript. prefers_vad (3dea0fe) hid this above 30 s, but VAD only auto-arms on long audio and costs a model download, so a plain 10 s silent file still produced a sentence. This gates it directly: no signal in, empty transcript out, at zero cost and with no extra model. The threshold is deliberately near-exact. 1e-5 sits BELOW one int16 LSB (3.05e-5), so a single non-zero sample disables the gate. Measured headroom against real audio: the quietest speech to hand (a FLEURS clip) peaks at 0.038 — ~3800x the threshold — and low-level noise at ~0.0018 does not provoke the model anyway. So the gate covers exactly the observed failure and declines to guess about anything else. Verified: 10 s silence -> empty; jfk, Arabic and the -28 dBFS FLEURS clip all unchanged; the 31 s chunked case is clean even with VAD bypassed via --chunk-seconds; CRISPASR_COHERE_SILENCE_GATE=0 restores the old behaviour on both. Predicate lives in audio_chunking.h with hermetic tests covering both failure directions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…not short clips Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The whitelist and probe-LID docs I wrote in 0b85808 only ever existed in hf_readmes/ — the cards on HuggingFace still said nothing about a restriction that now actively rewrites a user's -l. Uploaded to both repos. Uploading the in-repo copy verbatim would have DELETED the "Provenance and EU AI Act Art. 53 note" section: it exists on the published cards and not in hf_readmes/, so the repo copy had drifted BEHIND HF. The upload was rebased on the live card and adds only the new sections — verified purely additive (36 and 13 lines added, 0 removed, frontmatter byte-identical), and both live cards re-checked afterwards for the AI Act section and license: apache-2.0. This back-ports that section so hf_readmes/ stops trailing. NOT synced: the published cards still carry the `whisper.cpp` tag and a plain repo URL where hf_readmes/ has `crispasr` and a `/tree/ggml` link. One of those is a deliberate rebrand and the other looks like a stale branch link; deciding that is not part of this change, so both sides keep what they had. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…once/compute-many fix The fork commit makes repeat computes of one sched allocation safe (state-flagged src-rewire log; CrispEmbed O6 replay crash root cause) and is behavior-neutral for CrispASR's reset-per-step engines — verified: 1477/1477 unit tests, moonshine + paraformer transcripts byte-identical to the 89a2039d baselines, melotts TTS->ASR roundtrip reproduces the text exactly. Unblocks any future persistent-graph decode path here (the CrispEmbed PERFORMANCE.md 'sched alloc-once' entry has the contract). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ps, metal im2col batch-1 occupancy, conv_1d batch layout fix Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reported by toki1703 in a HuggingFace PR (cstr/cohere-transcribe-03-2026-GGUF discussion #2) on 2026-05-21, still open. Every step was wrong, and I verified each against the repo before changing it: git clone -b ggml ... -> that branch has 0 remote heads make -j$(nproc) cohere-main -> no Makefile build; no such target ./bin/cohere-main -> no such binary; it is `crispasr --backend cohere` Plus one the report predates: ggml is a SUBMODULE now, so a plain clone leaves ggml/CMakeLists.txt missing and cmake refuses to configure — the build system itself prints "re-clone with --recursive". Verified by running the corrected steps from a clean clone: recursive clone -> cmake configure -> build --target crispasr-cli -> ./bin/crispasr exists and parses `--backend cohere -m ... -f ... -l en`. Also fixes the stale /tree/ggml link in the prose, and the comparison table's `cohere-main` reference. I published an addition to this card yesterday and did not notice any of this, because I only diffed my own insertions against the live copy and never read what I was adding to. The Arabic card was already correct. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… surfaces SubtitleEdit's OmniVoice language menu did nothing. Three independent breaks: 1. The CLI adapter's synthesize() applied only tts_num_steps per call; omnivoice_set_language ran ONLY in init(). The server owns one backend instance per session, so after the first line the menu could never change anything — even though crispasr_server.cpp has parsed language/target_lang into rp.language since #249/#304. 2. The session C-ABI's omnivoice arm was a bare omnivoice_synthesize(), so set_target_language never reached it and bindings/Flutter/Android had no language knob by any route. #329's cosyvoice3 bug, one backend over. 3. The runtime dropped the string VERBATIM into <|lang_start|>…<|lang_end|>. The blueprint's _resolve_language() is ID-passthrough → lowercase-name lookup → None; we did none of it, so 'de-DE' or a typo conditioned the model on tokens it never saw in that slot while looking like it worked. core/omnivoice_lang.h mirrors the blueprint resolver over a generated 646-id table (tools/gen-omnivoice-lang-map.py <- upstream lang_map.py). The runtime resolves centrally in omnivoice_set_language so no surface can skip it, and warns with a did-you-mean before falling back to language-agnostic. Verified on CODES, not WAVs — output is watermarked, so cmp on audio measures the watermark. CRISPASR_OMNIVOICE_DUMP_CODES + --no-spoken-disclaimer, English jfk.wav reference -> German target: '-l de' vs none DIFFERENT; '-l German' vs '-l de' IDENTICAL; '-l de-DE' vs none IDENTICAL; session vs CLI IDENTICAL; server vs CLI IDENTICAL across three sequential requests on one process. NEGATIVE RESULT, recorded in docs/omnivoice/PLAN.md: this does NOT demonstrably fix the reported accent. whisper LID over 3 German sentences moved only on the first (0.927->0.998); sentences 2-3 were already 0.999 untagged and the tag moved neither. LID is accent-robust by design, so it is the wrong metric. OmniVoice also has no cross-lingual drop-ref path — create_voice_clone_prompt requires ref_text and _combine_text positions it before the reference audio — so #329's transcript-drop fix does not transfer. Adjacent fix: /v1/audio/speech advertises a per-request seed that the omnivoice adapter dropped, so re-rendering a line was not reproducible. Applied when non-zero; --seed 999 differs from the default and repeats byte-identically. Still open SE-side: OmniVoiceCrispAsr.Speak() accepts the language and never sends it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ually run it The generator fetched lang_map.py from GitHub, so `--check` needed the network and was left out of CI — which meant nothing caught a hand-edit of src/core/omnivoice_lang_table.h, and the ids in it are what the model is conditioned on (#13273). Vendor the source instead: third_party/omnivoice/lang_map.py is a byte-identical copy of upstream (blob ffcda10, commit c8c0a625), so --check renders from it with no network and is now a hard gate in lint.yml. Verified it goes red — a one-id hand-edit of the header exits 1. Upstream drift is a separate, best-effort question and gets its own advisory step: --check-upstream compares git blob hashes via the GitHub API, prints a NOTE and exits 0. It must not auto-fail, because taking a new revision is a judgement call — a NEW language is an addition we want, but a CHANGED id for an existing name silently alters what every caller of that name synthesises. --update-vendored does the refetch when that call is made. Apache-2.0 notice added for the vendored file; its upstream header is retained. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SubtitleEdit's OmniVoice language menu is still not wired to its request
payload, so every dubbed line arrives language-agnostic no matter what the user
picks. Rather than leave the fix blocked on someone else's release, omnivoice
now detects the language from the text it is about to speak when — and only
when — no language was requested. CRISPASR_OMNIVOICE_AUTO_LANG=0 restores the
old behaviour.
Guessing is normally the wrong instinct, so the justification is a measurement
of the HARM side, not of detection accuracy: German text deliberately mis-tagged
'-l en' gives a word-perfect ASR round-trip and whisper LID 'de' at 0.984 — the
same band as the correct tag (0.947) and no tag (0.998), i.e. no ordering at
all. A bad guess costs nothing detectable; the upside is the one upstream
documents ("performance is slightly better if you specify the language"). If a
later measurement shows a wrong tag DOES degrade output, flip this back to
opt-in — that asymmetry is the whole argument.
Verified on codes with --no-spoken-disclaimer: no -l at all == explicit -l de
IDENTICAL; AUTO_LANG=0 == the old untagged codes IDENTICAL; explicit -l en on
German text == the pre-existing -l en result IDENTICAL (the guess never
overrides); and an SE-shaped POST of {input, response_format} with no language
field == -l de IDENTICAL.
Two load-bearing details, both guarded:
- Detect over the TARGET text, never combined_text — the latter carries the
reference transcript, so an English reference clip would drag every German
subtitle's guess to English, silently and only when cloning.
- Per call into a local, never written back to ctx->language — the server
reuses one context, so a sticky guess would leak line N onto line N+1, the
same per-call-vs-init bug this change set exists to fix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The style prefix is where the language and instruct conditioning actually lives, and a BPE difference there is invisible in every downstream metric -- the graph still computes and the audio still sounds like speech. Printing the ids makes prompt-token parity against the HF tokenizer a one-command check (dev-guide step 0), which is how the #13273 language path was confirmed byte-identical to the blueprint for de/en/arb/zh, None, and the <|denoise|> clone variant. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…heet + real patch; prose stays human-authored
…sibling
Not from a report. Going back to diff our runtime against the blueprint properly
turned up _resolve_instruct sitting ten lines below _resolve_language, doing the
same job for the same prompt slot, mirrored no better.
The instruct is a CLOSED 48-item vocabulary (6 mutually-exclusive categories,
each item with an EN/ZH counterpart). Upstream lowercases it, repairs
half/full-width separators, unifies it to one language and RAISES on anything
else. We stored the raw string and dropped it into <|instruct_start|>:
'Male, British Accent' -> [151672, 36421, 11, 7855, 81809, 151673]
'male, british accent' -> [151672, 36476, 11, 93927, 29100, 151673]
Not one shared token id, for what a user considers the same request. It also
carried the SAME per-call bug as language — set only in init(), so the server's
per-request "instructions" field was dead after the first line — in the very
function just edited to fix that for language.
Mirrors upstream by rejecting rather than degrading (the CLI exits non-zero, the
server 400s with the offending item and a did-you-mean), because a voice-design
request that silently does nothing is the failure being fixed. Two-phase on
purpose: parse() is text-independent and runs at set time; render() needs the
text — a dialect forces Chinese, an accent forces English, otherwise it follows
whether the TARGET text is Chinese — so it runs per synthesis. Baking the
rendered string at set time would freeze one line's EN/ZH choice onto every
later line on a reused context.
Verified against the blueprint resolver AND the HF tokenizer: 'Male, British
Accent' on English text and 'Male, Elderly' on Chinese text both produce style
ids byte-identical to AutoTokenizer; 'britsh accent' → CLI rc=13 / server 400;
'male, female' → 400; three different instructs on one server process each
applied (codes DIFFERENT per request).
Along the way the language path itself is now PROVEN rather than argued: style
ids byte-identical for de/en/arb/zh, None, and the <|denoise|> clone variant.
Qwen2 has no BOS so the blueprint's add_special_tokens asymmetry is a no-op; the
uncond CFG arm is target-audio-only in both; normalize_text defaults off and the
official CLI never passes it. Correction: treating "auto" as cleared is not an
addition on top of the blueprint — demo.py:186 does exactly that.
The suggestion string is the one deliberate non-parity (LCS ratio vs difflib's
Ratcliff/Obershelp, same 0.6 cutoff); it changes no model input.
voice_design.py vendored byte-identically alongside lang_map.py so the new CI
gate is hermetic too; verified it goes red on a tampered table.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…CJK range Two gaps from my own audit of the #13273 work. 1. The resolvers had unit tests; the string they FEED did not. Prompt-token parity was verified against the real HF tokenizer by hand — 1.2 GB model, debug print, eyeball — which is exactly the check that gets done once and then rots, and is how the instruct defect survived the first review. Style assembly moves to core/omnivoice_prompt.h (weight-free) so the exact string is unit-testable, and every string asserted was verified byte-for-byte against AutoTokenizer, with the ids recorded in the test file. Verified the guard goes red: injecting one space into a tag fails 3 of 5 cases. 2. text_is_zh() sniffed the UTF-8 lead byte (0xE4..0xE9) where Python's _ZH_RE is exactly [U+4E00, U+9FFF]. Those look equivalent and are not — the shortcut over-matches U+4000..U+4DFF by 3584 codepoints (CJK Ext-A, Yijing hexagrams). That window decides whether an instruct renders "male, elderly" or "男，老年", so it now decodes the codepoint properly. Tested at both range edges, inside the over-match window, and across kana/hangul/emoji (the 4-byte case the decoder must step over, not into). Also asserts the auto-language fallback is env-gated and defaults ON — an accidental flip to false would be invisible, since output would just revert to language-agnostic. 1510/1510 unit tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…checks Closes the gaps my own coverage audit turned up. CORRECTION: I reported "there is no session instruct setter at all". crispasr_session_set_instruct has existed all along, dispatching to qwen3-tts and parler — omnivoice fell through to `return -3`, i.e. "this backend has no instruct contract". So it was the same wiring bug a THIRD time, not a missing feature. Now dispatched, which makes voice design reachable from every binding; the Python set_instruct needed no change beyond documenting that omnivoice takes a closed vocabulary rather than prose. Guarded by a source test scoped to the function body, watched red by unwiring the arm. Two things had been verified by hand only and are now runnable gates: - tests/test-omnivoice-style-tokens.sh — prompt-token parity against the real vocabulary. The unit tests pin the style STRING; this pins what it tokenizes to, which is what the model consumes and the one link no hermetic test sees. 9 cases, ids from the reference Qwen2Tokenizer. All pass; verified red (one wrong id → rc=1, checked without a pipe laundering the status). - tests/test-omnivoice-surface-parity.sh — CLI / server / session must agree. Source guards catch a MISSING call site; only this catches one present and wrong. Compares CODES, never the watermarked WAV, and every check includes an arm expected to be IDENTICAL. 10 arms pass including the session one. Writing the parity test caught a stale assumption of mine: with auto-detect shipped, "no language" on German text is no longer agnostic — it is `de`, so those arms are correctly identical. The discriminators now use two EXPLICIT languages, so the guesser cannot mask a dead knob by making every arm German; the agnostic baseline runs with AUTO_LANG=0; and the auto-detect contract is pinned separately on the SubtitleEdit-shaped request. Both SKIP cleanly without a model; env registered in tests/env-live-tests.sh. 1511/1511 unit tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…can name the side that moved The live style-token test pinned ids from a tokenizer run with no record of WHICH tokenizer — a mismatch could not say whether our side or upstream drifted, and closing that honestly seemed to mean vendoring the 7 MB tokenizer.json. Cheaper close: record the provenance instead. - Test header now pins k2-fsa/OmniVoice rev c5fdb5cc + tokenizer.json sha256 408f669b, with the one-command disambiguation (same sha -> our side moved, fix us; new sha -> re-derive pins AND regenerate the GGUF vocabulary, update both). - All 9 sequences re-derived byte-identical today from that exact file (tokenizers 0.23.1, independent of our code); upstream main unchanged since 2026-07-03, so the pins' lineage is a single revision. - Both live gates re-run green on a rebuilt binary (style-tokens 9/9, surface-parity all arms; session arm SKIP, binding not built here). - PLAN de-staled: the SE-side payload gap is no longer blocking since the auto-lang fallback (cf93079) — the old block still said the menu stays decoration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…he gate able to go red `crispasr-diff omnivoice` was a stub — loaded the model, printed the ref token count, compared nothing, exited 0 — while the real per-stage comparison (omnivoice_encode_diff) hid behind an env var in the CLI adapter. And run_encode_diff returned 0 unconditionally: with the corrupt tokenizer it printed acoustic_enc FAIL and FULL wav->codes 15.0% and still exited clean. - diff-main omnivoice branch now resolves the audio tokenizer (CRISPASR_OMNIVOICE_TOKENIZER_GGUF, else next-to-model candidates mirroring the CLI adapter) and runs the encode diff as a counted harness stage. - run_encode_diff counts main-chain stage failures and returns nonzero; RVQ gated at >=99.5% exact (measured 99.9%), FULL wav->codes at >=95% (measured 99.0%; residual is the documented resampler-vs-torchaudio Hann-sinc gap, corrupt-tokenizer mode sits at 15%). Verified red-first with the quarantined corrupt tokenizer GGUF: 2 stages FAILED, crispasr-diff rc=6, CLI env path rc=1. Green run: all stages cos_min=1.000000, RVQ 2197/2200, FULL 2178/2200, rc=0. Hermetic omnivoice tests still pass. Also refreshed docs/diff-harness-coverage.md (madlad row, dep drift). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )