🎙️ A language-learning app where the tutor has moods. Pick Friendly, Toxic, Strict, or Cute — the AI character literally roasts you (or hugs you) based on how you speak.
MiMo Mood Tutor explores what happens when you wire a multimodal LLM directly into a real-time speech feedback loop. MiMo Omni transcribes the user's audio and scores fluency or single-word pronunciation; MiMo v2.5-Pro generates personality-driven feedback that doubles as a coaching note; and an animated SVG / chat UI delivers it with TTS.
This is a learning project — not a production substitute for Duolingo, Speechling, or Voca. The goal is to see how far a 7B-class multimodal model can carry a real-time consumer-app interaction across two complementary modes:
| Mode | What you do | Endpoint |
|---|---|---|
| 🎯 Word Drill | Speak a target word, get a 0-100 score and a mood-flavored reaction | POST /score |
| 💬 Conversation | Pick a topic, have a multi-turn back-and-forth with the AI tutor | POST /chat |
In both modes you choose one of four mood personas — same model, same topic, very different vibe.
Pick a topic from the sidebar (Cafe, Trip Planning, Job Interview, Casual Small Talk, Asking for Directions, Hobbies & Interests), choose a tutor mood, and have a free-form conversation. After each user turn the model:
- Transcribes the audio (MiMo Omni)
- Scores fluency 0-100 (intonation, pace, intelligibility)
- Detects one grammar / word-choice issue (or returns "none")
- Replies in character continuing the dialogue, weaving in the correction gently
- Speaks the reply via Edge TTS
Sample flow at the cafe with the Friendly persona — gentle correction without breaking the role-play:
The Toxic persona, mid-record during an interview role-play (mic pulses while MiMo Omni listens):
The Strict persona, formal British, quotes IPA when needed:
The Cute persona, uwu chaos but genuinely helpful with grammar:
┌──────────────────────────────────────────────────────────────┐
│ 1. Pick a target word "apple" │
│ 2. Tap the mic 🎙️ │
│ 3. Speak "epol" │
│ 4. MiMo Omni transcribes + scores: 42/100 │
│ 5. MiMo v2.5-Pro drafts a Toxic-mode reply │
│ 6. Character animates "smirk" + speaks via TTS │
│ 7. User retries or moves to next word │
└──────────────────────────────────────────────────────────────┘
| Mode | Tone | Wrong response example |
|---|---|---|
| Friendly | encouraging coach | "Almost there! Listen to the /æ/ sound and try once more 🌸" |
| Toxic | drill-sergeant snark | "What was THAT? Did your tongue file for unemployment? Again!" |
| Strict | formal teacher | "Incorrect. Target: /ˈæpəl/. Focus on the final consonant. Repeat." |
| Cute | uwu chaos | "ahh sayy bwoke da wowd 🥺 twy agen pwease~" |
Same model, same task, different system prompt. Mood is a runtime parameter, not a separate model.
┌────────────────┐ audio (mp3) ┌──────────────────┐
│ Next.js UI │ ─────────────────► │ FastAPI │
│ + SVG actor │ │ /score endpoint │
│ + Web Audio │ ◄───────────────── │ │
└────────────────┘ {score, reply, └────────┬─────────┘
mood, audio_url} │
│ MiMo Omni
▼
┌──────────────────┐
│ 9Router proxy │
│ → Token Plan │
│ SGP endpoint │
└────────┬─────────┘
│
│ mimo-v2-omni (audio understanding)
│ mimo-v2.5-pro (personality reply)
▼
Edge TTS → mp3 → frontend
Why two MiMo variants: Omni handles audio-in (we tested mp3 + wav, ogg is rejected). v2.5-Pro handles the creative-writing side of the persona prompts. Splitting keeps the audio call fast (~2s) and lets us use the cheaper text variant for the chatty bits.
cd backend
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env # add your MiMo Token Plan key
uvicorn main:app --reload --port 8000cd frontend
npm install
npm run dev # http://localhost:3000# backend/.env
MIMO_BASE_URL=https://token-plan-sgp.xiaomimimo.com/v1
MIMO_API_KEY=tp-xxxxxxxxxxxxxxxxxxxxxxxxx
OMNI_MODEL=mimo-v2-omni
PERSONA_MODEL=mimo-v2.5-pro
TTS_VOICE=en-US-AriaNeural # any edge-tts voiceSee docs/METRICS.md for full numbers. Quick snapshot:
| Stage | Model | Median latency | Notes |
|---|---|---|---|
| Audio transcribe | mimo-v2-omni | ~2.1 s | enable_thinking=False mandatory |
| Persona reply | mimo-v2.5-pro | ~0.9 s | streamed, ~80 tokens average |
| TTS synthesis | edge-tts (free) | ~0.4 s | local, no network round-trip per word |
| End-to-end | (combined) | ~3.4 s | acceptable for turn-based practice |
Cost per session of ~10 words: ≈ 4,000 tokens on the persona model + ≈ 2,500 on Omni.
- Pronunciation scoring is heuristic, not phoneme-accurate. We ask Omni to transcribe and rate similarity, then map to a 0–100 score. A real pronunciation engine (Kaldi, Whisper-PA) would be more rigorous.
- English source language only. Target language can be swapped in
vocab.yaml, but persona prompts are tuned for English speakers. - No long-term progress modelling. Sessions are stateless beyond a SQLite log of attempts.
- Toxic mode is a parlour trick. Effective for short demos; not what you'd ship to a 12-year-old.
mimo-mood-tutor/
├── backend/
│ ├── main.py FastAPI app, /score endpoint
│ ├── mimo_client.py MiMo Omni + v2.5-Pro wrapper (OpenAI-compatible)
│ ├── persona.py four mood system prompts
│ ├── pronunciation.py scoring heuristic (Omni → score)
│ ├── tts.py edge-tts wrapper
│ ├── vocab.yaml starter wordlist (50 EN words, IPA, hints)
│ └── requirements.txt
├── frontend/
│ ├── app/
│ │ ├── page.tsx main lesson screen
│ │ ├── components/
│ │ │ ├── Character.tsx SVG actor + state machine
│ │ │ ├── MicButton.tsx Web Audio recorder
│ │ │ ├── ModeSelect.tsx persona switcher
│ │ │ └── ScoreBar.tsx 0–100 visual
│ │ └── lib/
│ │ └── api.ts backend client
│ └── package.json
├── docs/
│ ├── METRICS.md latency/cost/accuracy tables
│ ├── PERSONA-DESIGN.md how the four mood prompts were built
│ └── ARCHITECTURE.md deeper diagram + decisions
├── examples/
│ ├── score_one_word.py minimal Python example
│ └── batch_score.py run vocab.yaml through the pipeline
└── tests/
└── test_persona_consistency.py
- MiMo — Xiaomi's multimodal LLM family. Built for the 100T Token Plan.
- 9Router — local proxy used during development.
- edge-tts — free Microsoft Edge TTS bridge.
License: MIT.