mlx-speech

Note

This project wouldn't exist without the inspiration and generous support of the incredible community at linux.do.

Local speech synthesis, editing, and transcription on Apple Silicon, running pure MLX. No cloud, no PyTorch.

mlx-speech is an App Automaton project. The appautomaton org hosts the code on GitHub and the converted weights on Hugging Face.

Models

Pre-converted MLX weights are published under the App Automaton Hugging Face org, appautomaton, and download automatically on first use. Load by alias or by full repo id — tts.load("fish-s2-pro") and tts.load("appautomaton/fishaudio-s2-pro-8bit-mlx") are equivalent. Each model name links to a guide covering behavior, flags, and known limitations.

Text-to-speech

Alias	Model	Weights
fish-s2-pro	Fish S2 Pro — dual-AR TTS, voice cloning, emotion tags	int8
vibevoice	VibeVoice Large — hybrid LLM+diffusion TTS, voice cloning	int8
longcat	LongCat AudioDiT — flow-matching diffusion TTS	int8
moss-local	OpenMOSS TTS Local — local-attention multi-VQ TTS	int8
moss-ttsd	MOSS-TTSD — delay-pattern dialogue TTS	int8
moss-sound-effect	OpenMOSS Sound Effect — text-to-sound-effect generation	4-bit
step-audio	Step-Audio-EditX — voice cloning, audio editing	int8
dramabox	DramaBox — Resemble flow-matching diffusion TTS, 48 kHz stereo	bf16¹

Speech-to-text

Alias	Model	Weights
cohere-asr	Cohere Transcribe — multilingual ASR	int8
qwen3-asr-1.7b	Qwen3-ASR-1.7B — English, Chinese, and mixed Chinese/English ASR	bf16
—	IBM Granite Speech 4.0 1B — runs the original sharded checkpoint from a local path	local checkpoint

¹ tts.load("dramabox") also pulls the Gemma 3 12B backbone text encoder automatically. Output is 48 kHz stereo. For advanced controls (cfg, steps, voice reference) use scripts/generate_dramabox.py. See docs/dramabox.md.

Installation

Requires an Apple Silicon Mac (M1 or later) and Python 3.13+.

pip install mlx-speech

Quick Start

Python:

import mlx_speech

# Text-to-speech
model = mlx_speech.tts.load("fish-s2-pro")
result = model.generate("Hello from mlx-speech!")
# result.waveform: mx.array, result.sample_rate: int

# Voice cloning with emotion tags
result = model.generate(
    "[excited] This is amazing!",
    reference_audio="reference.wav",
    reference_text="Transcript of the reference audio.",
)

# Speech-to-text
asr = mlx_speech.asr.load("qwen3-asr-1.7b")
print(asr.generate("audio.wav").text)

# Local checkpoint paths work anywhere an alias does
granite = mlx_speech.asr.load("models/ibm/granite_4_0_1b_speech/original")
print(granite.generate("audio.wav").text)

# Discover models
mlx_speech.tts.list_models()
mlx_speech.asr.list_models()

CLI:

# Generate speech
mlx-speech tts --model fish-s2-pro --text "Hello!" -o output.wav

# Voice cloning with emotion tags
mlx-speech tts --model fish-s2-pro \
  --text "[whisper] Just between us..." \
  --reference-audio ref.wav \
  --reference-text "Transcript of reference." \
  -o cloned.wav

# Step Audio emotion editing
mlx-speech tts --model step-audio \
  --reference-audio input.wav \
  --reference-text "Transcript." \
  --edit-type emotion --edit-info happy \
  -o happy.wav

# Sound effect generation
mlx-speech tts --model moss-sound-effect \
  --text "rolling thunder with rainfall" \
  --duration-seconds 8 \
  -o thunder.wav

# Transcribe audio
mlx-speech asr --model cohere-asr --audio speech.wav
mlx-speech asr --model qwen3-asr-1.7b --audio speech.wav --language Chinese

# Local checkpoint paths work anywhere an alias does
mlx-speech tts --model models/fish_s2_pro/mlx-int8 --text "Hello!" -o output.wav
mlx-speech asr --model models/ibm/granite_4_0_1b_speech/original --audio speech.wav

# Discover models
mlx-speech tts --list-models
mlx-speech asr --list-models
mlx-speech --help

Note: The mlx-speech CLI covers the common path — basic generation, voice cloning, and editing. For advanced controls (sampling temperature, top-p/k, diffusion steps, batch JSONL, duration tuning, etc.) use the scripts in scripts/ directly. Each model family has a corresponding script with the full inference surface documented in docs/.

Conversion

To convert upstream source weights yourself:

python scripts/convert/fish_s2_pro.py
python scripts/convert/longcat_audiodit.py
python scripts/convert/vibevoice.py
python scripts/convert/moss_local.py
python scripts/convert/moss_ttsd.py
python scripts/convert/moss_sound_effect.py
python scripts/convert/step_audio_editx.py
python scripts/convert/cohere_asr.py
python scripts/convert/qwen3_asr.py

Development

git clone https://github.com/appautomaton/mlx-speech.git
cd mlx-speech
uv sync
uv run pytest tests/unit/
uv run ruff check .

mlx-speech/
  src/mlx_speech/    library code
  scripts/           conversion, generation, eval, and audit entry points
  models/            local checkpoints (not in git)
  tests/             unit, checkpoint, runtime, integration tests
  docs/              model-family behavior guides

License

MIT — see LICENSE

Built and maintained by App Automaton.