Add VoxCPM2 TTS model (2B params, 48kHz, voice cloning) - #175
Open
sb1992 wants to merge 8 commits into
Open
Conversation
Autoregressive MiniCPM backbone + CFM diffusion + AudioVAE decoder. Ported from mlx-audio Python voxcpm2 implementation with: - Config-driven generation constants (inferenceTimesteps, cfgScale) - Resilient Codable decoding with defaults for all config structs - 48kHz→16kHz resampling for encoder input - BOS/EOS stripping in tokenizer to match upstream Python behavior - Proper weight sanitization (snake_case→camelCase, RoPE buffers) - Smoke tests for generate and stream paths
- Qualify Tokenizer as Tokenizers.Tokenizer to resolve ambiguity with MLXLMCommon.Tokenizer (build blocker) - Remove unused variable (MiniCPM.swift), change var→let where not mutated (VoxAudioVAE.swift, VoxCPM2Model.swift) - Add README.md with usage examples, streaming behavior, output format, and known limitations
Matches the existing exclude pattern for all other model READMEs.
- Broaden eval() to include predFeat, breaking MLX lazy graph chains that accumulate across 2000 AR steps and cause OOM - Add Task.checkCancellation() in generation loop for cooperative cancel - Add reserveCapacity for predFeatSeq and KV cache arrays - Cache SinusoidalPosEmb freqs in init (was recomputed ~20k times/gen) - Make rope, specialToken immutable (let) where never mutated after init
- Make tokenizer a let property injected via init (eliminates data race) - Store Task handle in generateStream and hook onTermination for cancellation - Replace Array(repeating:) with MLX.broadcast in Euler solver (avoids CPU alloc per ODE step) - Add unmapped weight key logging in sanitize() (excludes quantization artifacts) - Promote NSRegularExpression to static let in VoxAudioVAE sanitize
Author
|
Pushed a small compatibility fix for the failed CI build. The failure was from Xcode 26.3 rejecting |
Resolve TTSModel.swift by keeping upstream's loader refactor and MOSS-TTS registration, then adding VoxCPM2 registration in the same load helper style.\n\nVerified with Xcode 26.3 build-for-testing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Swift port of VoxCPM2, based on the Python implementation in Blaizzy/mlx-audio#641.
Autoregressive MiniCPM backbone + CFM diffusion + AudioVAE decoder.
Supports zero-shot generation and reference-audio voice cloning across 30+ languages.
Supported model
Example
Smoke test results (M5 Max, macOS 26.3.2)
voxcpm2Generate()voxcpm2GenerateStream()Both produce valid mono Float32 WAV at 48kHz.
Known limitations
maxTokensconservativelygenerate()call rather than yielding incremental chunksChecklist
SpeechGenerationModelTTSModel.swiftswift buildclean (zero warnings)