feat(kokoro): real StyleTTS-2/iSTFTNet decoder + dequant-at-load + espeak G2P (#9588)#37
Merged
Merged
Conversation
…peak G2P (#9588)
Kokoro TTS never actually spoke: `kokoro_synthesize` ran a placeholder
spectrogram (envelope x 1/f timbre x random phase) and discarded the model,
and the scalar predictor read F16/quantized GGUF tensors as raw F32 (garbage),
collapsing BERT to a constant -> a fixed-pitch buzz. This implements the real
forward pass end-to-end, validated against the PyTorch reference and verified
intelligible by whisper on the SHIPPED Q4_K_M bundle (no GGUF republish needed).
Root causes fixed:
- DTYPE: kokoro_load_model now dequantizes every non-F32 tensor to a parallel
all-F32 context (ggml to_float traits: F16 + Q5_0/Q4_K/Q6_K). The
predictor/decoder read F32 only. This alone un-breaks the (already-correct)
prosody predictor — BERT now matches the reference to 3 decimals.
- SYNTHESIS WAS A STUB: implement the iSTFTNet decoder.
- kokoro-decoder-front.h: Decoder.forward up to the generator (F0/N conv,
encode, asr_res, 4 AdainResBlk1d decode blocks). Validated corr=1.0/stage.
- kokoro-generator.{h,cpp}: the Generator — harmonic source (SineGen+Source
ModuleHnNSF from F0), center=True STFT, 2 ConvTranspose upsamples, 6 Snake
AdaINResBlock1, noise convs/res, conv_post, center=True iSTFT. corr 0.993
vs reference; whisper transcribes the output.
- kokoro-decoder.{h,cpp}: binds the model's F32 tensors into the front +
generator weight structs and runs them; wired into kokoro_synthesize
(predictor -> transpose asr -> decoder), variable-length T.
- G2P: kokoro-phonemes now drives libespeak-ng (en-us IPA -> Kokoro vocab ids,
reproducing the reference token sequence) when linked; ASCII fallback only
when espeak is absent. CMake auto-detects libespeak-ng.
Validation (Apple M-series): full text->speech via kokoro-tts on BOTH the
all-F32 regen GGUF and the published Q4_K_M bundle, whisper-intelligible across
phrases ("Hello, this is a native Kokoro voice test.", "The quick brown fox
...", "Open the pod bay doors please."). Per-component standalone tests
(decoder front, Snake resblock, harmonic source+STFT, iSTFT) corr 1.0 vs the
PyTorch reference. New unit test test_kokoro_g2p_espeak; kokoro-stage-dump /
kokoro-decoder-test dev harnesses. The published bundle GGUF is unchanged.
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the actual Kokoro TTS forward pass — previously
kokoro_synthesizewas a placeholder spectrogram and the scalar predictor read F16/quantized tensors as raw F32 (→ a constant buzz).kokoro_load_modelbuilds a parallel all-F32 context (ggmlto_floattraits: F16 + Q5_0/Q4_K/Q6_K) so the predictor/decoder read F32 only.kokoro-decoder-front.h(Decoder.forward up to the generator) +kokoro-generator.{h,cpp}(harmonic source, center=True STFT, ConvTranspose upsamples, Snake AdaINResBlock1, conv_post, center=True iSTFT) +kokoro-decoder.{h,cpp}(binds tensors, runs front→generator). Wired intokokoro_synthesize.kokoro-phonemesdrives libespeak-ng (en-us IPA → vocab ids); CMake auto-detects it.Verified intelligible (whisper) end-to-end on the shipped
elizaos/eliza-1bundles/2bQ4_K_M GGUF — no model republish needed. Per-component standalone tests vs the PyTorch reference: decoder-front / Snake resblock / iSTFT corr 1.0, harmonic+STFT 0.9999, full generator 0.993. New unit testtest-kokoro-g2p-espeak.Counterpart eliza submodule pin: elizaOS/eliza#9738.
🤖 Generated with Claude Code