From 06ffb8d7254ec34dfc1f9366b397ffc86113c09c Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 25 Jul 2026 17:52:50 +0000 Subject: [PATCH 1/6] Improve transcription accuracy without a second speech model LocalVocal stays the transcriber; this adds the text-side layer around it, plus a way to measure how well it is actually doing on this streamer's mic. - src/speech.js: hallucination filtering (whisper's "Thank you." / "[Music]" / repetition loops on silence, matched whole-line so real sentences survive), a per-streamer correction map, word-level alignment, WER, and derivation of correction candidates from a known reference reading. - Mic check (Settings -> Voice): generates a short script from the cast names, channel and custom vocabulary, listens through the live LocalVocal path, and reports the real word error rate plus one-click fixes for the names it got wrong. It is a measurement, not training - no audio leaves the machine and no model is modified. - Guards so a sloppy reading cannot produce sloppy rules: mismatch segments past a few words are discarded, both sides must resemble each other, and a partly-mangled multi-word name widens to the whole name ("hollow night -> Hollow Knight", never "night -> Knight"). - Corrections apply live via HOT_PATHS, so accepting fixes mid-stream takes effect on the next line. The cast is also told which proper nouns matter, so it recovers near-misses the map has not learned yet. - Voice replies are suppressed while a mic check runs, and an abandoned check self-expires. - README: config reference rows plus LocalVocal tuning guidance (bigger .en model, longer buffer over caption latency, partials off, clean audio). Release-Bump: minor Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SK871Ay2eiEzxTPorRufLP --- README.md | 24 +++ config.example.json | 5 + public/settings.html | 168 ++++++++++++++++ src/brain.js | 11 +- src/server.js | 87 ++++++++- src/speech.js | 410 ++++++++++++++++++++++++++++++++++++++++ src/transcript.js | 32 +++- test/speech.test.js | 147 ++++++++++++++ test/transcript.test.js | 54 +++++- 9 files changed, 932 insertions(+), 6 deletions(-) create mode 100644 src/speech.js create mode 100644 test/speech.test.js diff --git a/README.md b/README.md index 7645109..53631a5 100644 --- a/README.md +++ b/README.md @@ -90,6 +90,27 @@ Quit Hype Ghost. out loud *is* replying. Everything it hears also shows up as faint ๐ŸŽ™ lines in the deck feed (toggle in Settings โ†’ Voice), so when the cast reacts oddly you can see exactly what the transcription thought you said. +- **Getting transcription accurate:** run **Settings โ†’ Voice โ†’ Mic check**. It hands you a + short script built from your ghosts' names, your channel, and anything in + `speech.vocabulary`; you read it out loud, and Hype Ghost lines it up against what + LocalVocal actually heard. You get your real word error rate and one-click **word fixes** + for the names it got wrong, applied to every transcript line from then on (live, no + restart). It is a measurement, not training โ€” nothing is uploaded and no model is + modified. Tuning that helps *before* you get there, in order of impact: + - **Use a bigger, English-only model.** LocalVocal defaults to `tiny.en`. Moving to + `small.en` (or `medium.en` on a strong PC) is the single biggest accuracy win, and at + equal size the `.en` models beat the multilingual ones for English. + - **Trade latency for accuracy.** Most LocalVocal guides tune for snappy on-screen + captions; that's the wrong target here. The cast speaks on a cadence of *minutes*, so a + longer processing buffer costs you nothing and gives Whisper more context per chunk. + - **Turn partial transcriptions off.** They write half-finished duplicate lines into the + file Hype Ghost tails. + - **Fix the audio first.** Noise suppression *before* the Transcription filter in the + chain, and no game or music audio bleeding into the mic track โ€” no model recovers from + a dirty input. + - Note the trade-off with the OBS crash below: the CPU backend is the safe one, and it's + also what limits how big a model you can run. Move up model sizes until OBS starts + struggling, then step back one. - **Party / co-op audio (second channel):** streaming alongside friends? Route their audio (Discord/TeamSpeak/party chat) to a separate OBS source, add a *second* LocalVocal Transcription filter to it, output to a **different** file/text source than your mic, and @@ -128,6 +149,9 @@ Quit Hype Ghost. | `transcript.mode` | `off`, `file` (tail LocalVocal's .txt/.srt output), or `textSource` (poll a text source over OBS WebSocket). | | `transcript.showInFeed` | Echo what voice awareness hears into the deck feed as faint ๐ŸŽ™ lines (party audio as ๐ŸŽง) so you can spot mishears the cast might be reacting to (default true). Deck only โ€” never in the cast's chat history, the recap, or the on-stream overlay. | | `transcript2.*` | *(Optional)* A **second** transcription channel for party/co-op audio (a separate audio device with its own LocalVocal filter). Same `mode`/`file`/`textSource`/`pollSeconds` as `transcript`, plus `label` โ€” how the cast refers to those people (e.g. "my co-op squad"). Treated as *other people*, never as the streamer. | +| `speech.dropHallucinations` | Discard speech-to-text filler invented on silence and music โ€” "Thank you.", "[Music]", "please subscribe", words stuck on repeat (default true). Matched whole-line only, so a real sentence containing those words survives. Applies to both channels. | +| `speech.corrections` | Word fixes applied to every transcript line before the cast sees it: `[{ "from": "bacon", "to": "Beacon" }]`. Whole words only, case-insensitive; a blank `to` deletes the phrase. Build this from **Settings โ†’ Voice โ†’ Mic check**, or by hand. Applies live on save. | +| `speech.vocabulary` | Extra proper nouns worth getting right โ€” the game, in-jokes, regulars' names. Your ghosts' names and `twitch.channel` are included automatically. Used to build the mic-check script, and told to the cast so it reads a near-miss as the real word. | | `cadence.soloSeconds` / `quietSeconds` | Average gap between messages when alone / when real viewers are present. | | `cadence.jitter`, `burstChance`, `lullChance` | Natural rhythm: ยฑjitter on normal gaps, occasional quick bursts (0.3โ€“0.6x) and long lulls (1.6โ€“3x). | | `cadence.minScreenshotGapSeconds` | Skip re-screenshotting on quick follow-ups (default 25). | diff --git a/config.example.json b/config.example.json index c50146c..0ee439d 100644 --- a/config.example.json +++ b/config.example.json @@ -104,6 +104,11 @@ "pollSeconds": 2, "label": "the people the streamer is playing with (co-op partners / a Discord or party call)" }, + "speech": { + "dropHallucinations": true, + "vocabulary": [], + "corrections": [] + }, "cadence": { "soloSeconds": 120, "quietSeconds": 480, diff --git a/public/settings.html b/public/settings.html index 8167212..d47d8d8 100644 --- a/public/settings.html +++ b/public/settings.html @@ -44,6 +44,18 @@ .arche { display: flex; gap: 7px; flex-wrap: wrap; margin-top: 8px; } #addGhost { margin-top: 2px; } + /* mic check */ + .miccheck { border: 1px solid var(--border); border-radius: var(--r-lg); padding: 14px 16px; background: var(--glass-2); margin-top: 12px; } + .miccheck ol { margin: 10px 0; padding-left: 22px; display: flex; flex-direction: column; gap: 7px; } + .miccheck li { line-height: 1.5; } + .miccheck .heard { color: var(--dim); font-size: var(--fs-sm); word-break: break-word; } + .wer { display: flex; gap: 22px; align-items: baseline; flex-wrap: wrap; margin-bottom: 4px; } + .wer b { font-size: var(--fs-lg); } + .fixrow { display: flex; align-items: center; gap: 10px; flex-wrap: wrap; margin-top: 9px; } + .fixrow .from { color: var(--bad); } + .fixrow .to { color: var(--good); } + .fixrow button { padding: 5px 11px; } + .accent-row { display: flex; gap: 10px; margin-top: 6px; } .accent-dot { width: 30px; height: 30px; border-radius: 50%; cursor: pointer; border: 2px solid transparent; } .accent-dot.on { border-color: #fff; box-shadow: 0 0 0 2px var(--bg), 0 0 0 4px currentColor; } @@ -216,6 +228,37 @@

๐ŸŽ™๏ธ Voice awareness

+
+ + +
+ +
+

๐ŸŽ™๏ธ Mic check & word fixes

+

Transcription garbles names worst โ€” and names are what this room runs on. Read a short script out loud and Hype Ghost lines it up against what LocalVocal actually heard: you get your real error rate on your mic, and one-click fixes for the words it got wrong. Nothing is uploaded and no model is trained โ€” this builds a find-and-replace list applied to the transcript before the cast ever sees it.

+
+ + +
+ + + + + +

One per line, what it hears = what you meant. Applied to every transcript line before the cast sees it โ€” whole words only, case-insensitive. Leave the right side blank to drop a phrase entirely. Applies live on save.

+ + + +

Extra proper nouns worth getting right: the game, in-jokes, regulars' names. Your ghosts' names and Twitch channel are already included. These go into the mic-check script, and the cast is told to read a near-miss of one as the real word. Changing these restarts the app.

+

๐ŸŽง Second channel โ€” party / co-op audio

@@ -555,6 +598,9 @@

Danger zone

if (data.openaiKeySaved) fields.find((f) => f.dataset.path === 'brain.openaiApiKey').placeholder = saved; if (data.twitchSecretSaved) fields.find((f) => f.dataset.path === 'twitch.clientSecret').placeholder = saved; $('talkingPoints').value = (cfg.talkingPoints || []).join('\n'); + const speech = cfg.speech || {}; + $('corrections').value = (speech.corrections || []).map((c) => `${c.from} = ${c.to}`).join('\n'); + $('vocabulary').value = (speech.vocabulary || []).join('\n'); renderCast(); // accent picker const acc = (cfg.theme && cfg.theme.accent) || 'violet'; @@ -568,6 +614,125 @@

Danger zone

initTts(); }); + // ---- mic check ---- + // The script is read into the *live* LocalVocal path, so the number it + // reports is the accuracy the cast actually gets โ€” not a lab measurement. + let micPoll = null; + const pct = (v) => Math.round((Number(v) || 0) * 100) + '%'; + + function micReset() { + clearInterval(micPoll); micPoll = null; + $('micPanel').style.display = 'none'; + $('micStart').disabled = false; + } + + function parseFixes(text) { + return text.split('\n').map((line) => { + const at = line.indexOf('='); + if (at < 0) return null; + const from = line.slice(0, at).trim(); + return from ? { from, to: line.slice(at + 1).trim() } : null; + }).filter(Boolean); + } + + function addFix(fix, btn) { + const existing = parseFixes($('corrections').value); + if (!existing.some((f) => f.from.toLowerCase() === fix.from.toLowerCase())) { + const box = $('corrections'); + box.value = (box.value.trim() ? box.value.replace(/\s*$/, '\n') : '') + `${fix.from} = ${fix.to}`; + } + if (btn) { btn.textContent = 'added โœ“'; btn.disabled = true; } + $('saveResult').className = 'result'; $('saveResult').textContent = 'press Save changes to apply'; + } + + $('micStart').onclick = async () => { + $('micResult').className = 'result'; $('micResult').textContent = 'startingโ€ฆ'; + $('micReport').style.display = 'none'; + const r = await fetch('/api/miccheck/start', { method: 'POST' }).then((x) => x.json()).catch((e) => ({ ok: false, error: e.message })); + if (!r.ok) { $('micResult').className = 'result err'; $('micResult').textContent = 'โœ— ' + r.error; return; } + $('micResult').textContent = 'listeningโ€ฆ'; + $('micStart').disabled = true; + const list = $('micScript'); + list.innerHTML = ''; + for (const line of r.script) { const li = document.createElement('li'); li.textContent = line; list.appendChild(li); } + $('micHeard').textContent = 'โ€ฆ'; + $('micPanel').style.display = ''; + micPoll = setInterval(async () => { + const p = await fetch('/api/miccheck/progress').then((x) => x.json()).catch(() => null); + if (!p) return; + if (!p.active) return micReset(); + $('micHeard').textContent = p.heard.length ? p.heard.join(' ') : 'โ€ฆ'; + }, 1200); + }; + + $('micCancel').onclick = async () => { + await fetch('/api/miccheck/cancel', { method: 'POST' }).catch(() => {}); + micReset(); + $('micResult').className = 'result'; $('micResult').textContent = ''; + }; + + $('micFinish').onclick = async () => { + const r = await fetch('/api/miccheck/finish', { method: 'POST' }).then((x) => x.json()).catch((e) => ({ ok: false, error: e.message })); + micReset(); + if (!r.ok) { $('micResult').className = 'result err'; $('micResult').textContent = 'โœ— ' + r.error; return; } + $('micResult').textContent = ''; + const box = $('micReport'); + box.innerHTML = ''; + const wer = document.createElement('div'); + wer.className = 'wer'; + const score = document.createElement('span'); + score.innerHTML = 'Word errors: ' + pct(r.wer) + ''; + wer.appendChild(score); + if (r.suggestions.length && r.projectedWer < r.wer) { + const after = document.createElement('span'); + after.className = 'result ok'; + after.textContent = 'โ†’ ' + pct(r.projectedWer) + ' with the fixes below'; + wer.appendChild(after); + } + box.appendChild(wer); + + const heard = document.createElement('p'); + heard.className = 'heard'; + heard.textContent = 'Heard: ' + r.heard; + box.appendChild(heard); + + const note = document.createElement('p'); + note.className = 'hint'; + note.textContent = r.suggestions.length + ? 'Fixes it found for the names it got wrong. Ordinary words are ignored on purpose โ€” a fumbled "the" costs nothing, a fumbled ghost name breaks a reply.' + : (r.wer > 0.35 + ? 'No name fixes to suggest, but the error rate is high โ€” try a bigger LocalVocal model, a longer buffer, and check for game audio bleeding into your mic (see the README).' + : 'Clean run โ€” it got every name right. Nothing to fix.'); + box.appendChild(note); + + for (const fix of r.suggestions) { + const row = document.createElement('div'); + row.className = 'fixrow'; + const from = document.createElement('span'); from.className = 'from'; from.textContent = fix.from; + const arrow = document.createElement('span'); arrow.textContent = 'โ†’'; + const to = document.createElement('span'); to.className = 'to'; to.textContent = fix.to; + const btn = document.createElement('button'); btn.textContent = 'Add fix'; + btn.onclick = () => addFix(fix, btn); + row.append(from, arrow, to, btn); + box.appendChild(row); + } + if (r.suggestions.length > 1) { + const all = document.createElement('div'); + all.className = 'fixrow'; + const btn = document.createElement('button'); + btn.className = 'primary'; + btn.textContent = 'Add all ' + r.suggestions.length; + btn.onclick = () => { + r.suggestions.forEach((f) => addFix(f, null)); + box.querySelectorAll('.fixrow button').forEach((b) => { b.disabled = true; }); + btn.textContent = 'added โœ“'; + }; + all.appendChild(btn); + box.appendChild(all); + } + box.style.display = ''; + }; + function toggleProvider() { const oai = $('providerSel').value === 'openai'; $('anthFields').style.display = oai ? 'none' : ''; @@ -743,6 +908,9 @@

Danger zone

setPath(cfg, f.dataset.path, v); } cfg.talkingPoints = $('talkingPoints').value.split('\n').map((s)=>s.trim()).filter(Boolean); + cfg.speech = { ...(cfg.speech || {}), + corrections: parseFixes($('corrections').value), + vocabulary: $('vocabulary').value.split('\n').map((s)=>s.trim()).filter(Boolean) }; // the cast (new source of truth) + mirror into bot/bot2 for back-compat const roster = collectCast(); cfg.cast = roster; diff --git a/src/brain.js b/src/brain.js index 7355f60..7655b03 100644 --- a/src/brain.js +++ b/src/brain.js @@ -64,10 +64,11 @@ export class Brain { * @param {Array<{name:string, personality:string}>} opts.personas 1โ€“4 entries * @param {string} opts.language */ - constructor({ brain, anthropic, personas, language }) { + constructor({ brain, anthropic, personas, language, vocabulary }) { this.provider = brain.provider === 'openai' ? 'openai' : 'anthropic'; this.personas = personas; this.language = language || 'English'; + this.vocabulary = Array.isArray(vocabulary) ? vocabulary : []; if (this.provider === 'anthropic') { const apiKey = anthropic.apiKey || process.env.ANTHROPIC_API_KEY || ''; this.hasKey = Boolean(apiKey); @@ -115,6 +116,14 @@ export class Brain { `You may be given an auto-generated transcript of what the streamer said out loud on mic.`, `Treat it as the streamer talking to chat: follow up naturally, never quote it verbatim`, `or correct its errors.`, + // Speech-to-text fails hardest on proper nouns, and proper nouns are what + // this room runs on. Naming them lets the model recover the intended word + // from a near-miss instead of reacting to nonsense. + ...(this.vocabulary.length + ? [`That transcript comes from speech-to-text and garbles names worst. These names matter`, + `here: ${this.vocabulary.join(', ')}. If a transcript word is a near-miss for one of them,`, + `assume that is what the streamer said โ€” silently, never announcing the fix.`] + : []), ``, `You may ALSO be given a separate "party audio" transcript: OTHER people the streamer`, `is playing with (co-op partners, a Discord or party call). These are DIFFERENT people`, diff --git a/src/server.js b/src/server.js index 2d39911..c33b65c 100644 --- a/src/server.js +++ b/src/server.js @@ -14,6 +14,7 @@ import { Brain } from './brain.js'; import { TwitchViewers, DecApi } from './twitch.js'; import { TwitchChat } from './twitchchat.js'; import { TranscriptFeed } from './transcript.js'; +import { collectTerms, buildMicCheckScript, deriveCorrections, wordErrorRate, compileCorrections, applyCorrections } from './speech.js'; import { SapiTts } from './tts.js'; import { GhostLoop } from './loop.js'; @@ -81,11 +82,22 @@ export function startServer(opts = {}) { // UI font scale (Settings โ†’ App). Never below 1: text can grow, not shrink โ€” // and the ๐Ÿค– AI badge inherits the same guarantee. const fontScale = () => Math.min(1.5, Math.max(1, Number(config.app.fontScale) || 1)); + // Proper nouns speech-to-text garbles worst โ€” cast names, the Twitch channel, + // plus anything the streamer added by hand. The mic check builds its reading + // script from these, and the brain is told to recover near-misses of them. + if (!isPlainObject(config.speech)) config.speech = { dropHallucinations: true, vocabulary: [], corrections: [] }; + const speechTerms = () => + collectTerms({ + cast: personas, + twitchChannel: config.twitch?.channel, + vocabulary: config.speech.vocabulary, + }); const brain = new Brain({ brain: config.brain, anthropic: config.anthropic, personas, language: config.bot.language, + vocabulary: speechTerms(), }); // "Brain is configured" gates the wizard redirect and preview mode: the // wizard can be finished without one (skip), and then the loop stays quiet. @@ -235,6 +247,10 @@ export function startServer(opts = {}) { 'energy', 'talkingPoints', 'theme.', 'overlay.', 'moments.', 'memory.', 'stream.', 'app.costMeter', 'app.uiLanguage', 'app.fontScale', 'app.autoPause', 'app.autoPauseMinutes', 'app.autoUpdate', 'app.ttsVoice', 'app.ttsRate', 'app.ttsOutputDevice', 'transcript.showInFeed', + // The feeds read config.speech live, so accepting mic-check corrections + // mid-stream applies to the very next line. (speech.vocabulary is not hot: + // it is baked into the brain's cached system prompt at startup.) + 'speech.corrections', 'speech.dropHallucinations', 'cadence.soloSeconds', 'cadence.quietSeconds', 'cadence.jitter', 'cadence.burstChance', 'cadence.lullChance', 'cadence.replyDelaySeconds', 'cadence.minVoiceReplyGapSeconds', 'cadence.minScreenshotGapSeconds', 'cadence.minPartyNudgeGapSeconds', @@ -502,6 +518,69 @@ export function startServer(opts = {}) { } }); + // ---------- mic check (Settings โ†’ Voice) ---------- + // Hand the streamer a short script, listen to what LocalVocal makes of it, + // and align the two. Because the reference text is known, the result is a + // measurement (word error rate) plus evidence-based correction candidates โ€” + // no model is trained or touched, and the audio path is the live one, so + // what it measures is exactly what the cast will hear all stream. + let micCheck = null; // {script, terms, heard[], startedAt} + const MIC_CHECK_MAX_MS = 5 * 60 * 1000; + + // Self-expiring: a check abandoned by closing the settings page must not keep + // suppressing voice replies (below) for the rest of the session. + function activeMicCheck() { + if (micCheck && Date.now() - micCheck.startedAt > MIC_CHECK_MAX_MS) micCheck = null; + return micCheck; + } + + app.post('/api/miccheck/start', (_req, res) => { + if (config.transcript.mode === 'off') { + return res.json({ ok: false, error: 'Turn on voice awareness above (and save) before running a mic check.' }); + } + const terms = speechTerms(); + const { lines, terms: covered } = buildMicCheckScript({ terms }); + micCheck = { script: lines, terms: covered, heard: [], startedAt: Date.now() }; + res.json({ ok: true, script: lines, terms: covered }); + }); + + app.get('/api/miccheck/progress', (_req, res) => { + const check = activeMicCheck(); + res.json({ active: Boolean(check), heard: check ? check.heard : [] }); + }); + + app.post('/api/miccheck/cancel', (_req, res) => { + micCheck = null; + res.json({ ok: true }); + }); + + app.post('/api/miccheck/finish', (_req, res) => { + const check = activeMicCheck(); + if (!check) return res.json({ ok: false, error: 'No mic check is running โ€” start one first.' }); + const { script, terms, heard } = check; + micCheck = null; + const reference = script.join(' '); + const transcribed = heard.join(' '); + if (!transcribed.trim()) { + return res.json({ + ok: false, + error: 'Nothing came through. Check that OBS is running with the LocalVocal filter on your mic, and that the file/text source above is the one it writes to.', + }); + } + const suggestions = deriveCorrections(reference, transcribed, terms); + // What the same reading would have scored with these corrections in place โ€” + // so the streamer can see whether accepting them is actually worth it. + const projected = applyCorrections(transcribed, compileCorrections(suggestions)); + res.json({ + ok: true, + reference, + heard: transcribed, + wer: wordErrorRate(reference, transcribed), + projectedWer: wordErrorRate(reference, projected), + suggestions, + }); + }); + const httpServer = createServer(app); const wss = new WebSocketServer({ server: httpServer }); @@ -553,8 +632,11 @@ export function startServer(opts = {}) { const transcriptFeed = new TranscriptFeed({ ...config.transcript, windowSeconds: config.cadence.transcriptWindowSeconds, + speech: config.speech, obs, onSpeech: (line) => { + const check = activeMicCheck(); + if (check) check.heard.push(line); state.lastHeard = { text: line, ts: Date.now() }; // Deck-only echo of what was transcribed, so mishears are visible the // moment they happen. Never enters `history` (the brain already gets @@ -564,7 +646,9 @@ export function startServer(opts = {}) { broadcast({ type: 'heard', heard: { channel: 'mic', text: line, ts: Date.now() } }); } broadcastState(); - loop.onSpeech(); + // Reading the mic-check script is talking *at* the app, not to chat โ€” + // replying to it would cost a generation and drown out the check. + if (!check) loop.onSpeech(); }, }); @@ -575,6 +659,7 @@ export function startServer(opts = {}) { const partyFeed = new TranscriptFeed({ ...config.transcript2, windowSeconds: config.cadence.transcriptWindowSeconds, + speech: config.speech, obs, onSpeech: (line) => { state.lastHeardParty = { text: line, ts: Date.now() }; diff --git a/src/speech.js b/src/speech.js new file mode 100644 index 0000000..ab27975 --- /dev/null +++ b/src/speech.js @@ -0,0 +1,410 @@ +/** + * Speech-to-text quality layer. + * + * LocalVocal (whisper.cpp inside OBS) is the transcriber and stays the + * transcriber โ€” this module is everything we can do to the *text* it produces + * without running a second speech model on a machine that is already sharing + * a GPU with OBS and a game: + * + * 1. Drop Whisper's well-known hallucinations on silence/music ("Thank you.", + * "[Music]", subtitle-credit lines, stuck repetition loops). Left alone, + * each one looks like the streamer speaking and can trigger a voice reply + * to something nobody said. + * 2. Apply a per-streamer correction map โ€” Whisper mangles proper nouns worst + * (ghost names, channel name, game titles), and those are exactly the words + * that matter here. + * 3. Build that map from evidence instead of guesswork: the "mic check" hands + * the streamer a short script to read, so we know the reference text and can + * align it against what actually came back. + * + * Note what this is NOT: it is not training or fine-tuning. A paragraph of + * audio is nowhere near enough to adapt an acoustic model, and it never + * touches the model. It is a measurement, and a lookup table built from it. + * + * Everything here is pure โ€” no I/O, no config, no clock โ€” so it is directly + * testable and safe to call per transcript line. + */ + +const escapeRe = (s) => s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); + +// Word boundaries that behave for unicode letters (\b does not). +const WORD_START = '(?= 6) return true; + } + // A 2โ€“3 word phrase tiled across the entire line ("no way no way no way โ€ฆ"). + for (const unit of [2, 3]) { + if (norms.length < unit * 4 || norms.length % unit !== 0) continue; + const head = norms.slice(0, unit).join(' '); + let tiled = true; + for (let i = unit; i < norms.length; i += unit) { + if (norms.slice(i, i + unit).join(' ') !== head) { tiled = false; break; } + } + if (tiled) return true; + } + return false; +} + +/** True when a line is almost certainly Whisper filler rather than speech. */ +export function isHallucination(text) { + const norm = normalizeForCompare(text); + if (!norm) return true; + if (isSoundEventOnly(text)) return true; + if (FILLER_LINES.has(norm)) return true; + if (FILLER_LINES.has(norm.replace(/\s+/g, ' '))) return true; + return isRepetitionLoop(norm.split(' ')); +} + +// --------------------------------------------------------------------------- +// 2. Correction map +// --------------------------------------------------------------------------- + +/** + * Turn [{from, to}] into compiled whole-word regexes. Compile once per + * corrections array (callers cache on array identity) โ€” this is hot code. + * A blank `to` deletes the phrase, which is the escape hatch for a filler + * line the built-in hallucination list doesn't know about. + */ +export function compileCorrections(corrections) { + const compiled = []; + const list = Array.isArray(corrections) ? corrections.slice(0, MAX_CORRECTIONS) : []; + for (const entry of list) { + const from = String(entry?.from ?? '').trim(); + if (!from || from.length > MAX_FROM_LEN) continue; + const to = String(entry?.to ?? '').trim(); + // Internal whitespace matches loosely so a two-word `from` still hits when + // the transcriber spaced it differently. + const body = from.split(/\s+/).map(escapeRe).join('\\s+'); + try { + compiled.push({ re: new RegExp(`${WORD_START}${body}${WORD_END}`, 'giu'), to }); + } catch { + // An unrepresentable `from` is skipped rather than breaking the feed. + } + } + return compiled; +} + +/** Apply compiled corrections; returns '' if the line was fully deleted. */ +export function applyCorrections(text, compiled) { + let out = String(text ?? ''); + for (const { re, to } of compiled ?? []) { + re.lastIndex = 0; + out = out.replace(re, to); + } + return out.replace(/\s+/g, ' ').trim(); +} + +// --------------------------------------------------------------------------- +// 3. Mic check โ€” reference script, alignment, suggestions +// --------------------------------------------------------------------------- + +/** + * The proper nouns this install cares about getting right. Ordinary words are + * deliberately excluded: a mishear of "the" costs nothing, a mishear of a + * ghost's name breaks a voice reply. + */ +export function collectTerms({ cast = [], twitchChannel = '', game = '', vocabulary = [] } = {}) { + const terms = []; + const seen = new Set(); + const add = (value) => { + const term = String(value ?? '').trim(); + if (!term || term.length > 40) return; + const key = term.toLowerCase(); + if (seen.has(key)) return; + seen.add(key); + terms.push(term); + }; + for (const member of cast) add(typeof member === 'string' ? member : member?.name); + add(twitchChannel); + add(game); + for (const word of Array.isArray(vocabulary) ? vocabulary : []) add(word); + return terms.slice(0, 10); +} + +// Carrier sentences, written the way a streamer actually talks so the reading +// voice matches the streaming voice. Each %s takes one term. +const ONE_TERM = [ + 'okay chat, %s is asking about the build again.', + 'shoutout to %s, thanks for hanging out tonight.', + 'hold on, %s just said something and i missed it.', + 'i swear %s says that every single stream.', + 'alright %s, you called it, that was rough.', +]; +const TWO_TERM = [ + '%s and %s, you two are menaces today.', + 'between %s and %s i have no idea who to believe.', +]; +// Always included: a term-free line, so a baseline error rate exists even for +// an install with no custom vocabulary at all. +const CONTROL_LINES = [ + 'let me check if this thing is actually hearing me at normal talking volume.', + 'that was a clutch play and i genuinely did not think it was going to work.', +]; + +/** + * A short script covering this install's proper nouns. Returns the lines to + * read plus the terms being measured. ~20 seconds of reading. + */ +export function buildMicCheckScript({ terms = [] } = {}) { + const pool = terms.slice(0, 6); + const lines = [CONTROL_LINES[0]]; + let i = 0; + let oneIdx = 0; + let twoIdx = 0; + while (i < pool.length) { + if (pool.length - i >= 2 && twoIdx < TWO_TERM.length) { + lines.push(TWO_TERM[twoIdx++].replace('%s', pool[i]).replace('%s', pool[i + 1])); + i += 2; + } else { + lines.push(ONE_TERM[oneIdx++ % ONE_TERM.length].replace('%s', pool[i])); + i += 1; + } + } + lines.push(CONTROL_LINES[1]); + return { lines, terms: pool }; +} + +/** + * Word-level alignment (Levenshtein with backtrace). Returns the edit script + * as ops: 'equal' | 'sub' | 'del' (in reference, missing from heard) | 'ins' + * (heard but not in reference). + */ +export function alignWords(refNorms, hypNorms) { + const a = refNorms.slice(0, MAX_ALIGN_WORDS); + const b = hypNorms.slice(0, MAX_ALIGN_WORDS); + const n = a.length; + const m = b.length; + const d = Array.from({ length: n + 1 }, () => new Int32Array(m + 1)); + for (let i = 0; i <= n; i++) d[i][0] = i; + for (let j = 0; j <= m; j++) d[0][j] = j; + for (let i = 1; i <= n; i++) { + for (let j = 1; j <= m; j++) { + const cost = a[i - 1] === b[j - 1] ? 0 : 1; + d[i][j] = Math.min(d[i - 1][j - 1] + cost, d[i - 1][j] + 1, d[i][j - 1] + 1); + } + } + const ops = []; + let i = n; + let j = m; + while (i > 0 || j > 0) { + if (i > 0 && j > 0 && d[i][j] === d[i - 1][j - 1] + (a[i - 1] === b[j - 1] ? 0 : 1)) { + ops.push({ op: a[i - 1] === b[j - 1] ? 'equal' : 'sub', ref: i - 1, hyp: j - 1 }); + i--; j--; + } else if (i > 0 && d[i][j] === d[i - 1][j] + 1) { + ops.push({ op: 'del', ref: i - 1, hyp: -1 }); + i--; + } else { + ops.push({ op: 'ins', ref: -1, hyp: j - 1 }); + j--; + } + } + ops.reverse(); + return ops; +} + +/** Character edit distance, for judging whether two phrases are plausibly the same one. */ +function editDistance(a, b) { + if (a === b) return 0; + let prev = Array.from({ length: b.length + 1 }, (_, j) => j); + for (let i = 1; i <= a.length; i++) { + const row = [i]; + for (let j = 1; j <= b.length; j++) { + row[j] = Math.min(prev[j - 1] + (a[i - 1] === b[j - 1] ? 0 : 1), prev[j] + 1, row[j - 1] + 1); + } + prev = row; + } + return prev[b.length]; +} + +/** 1 = identical, 0 = nothing in common. */ +export function similarity(a, b) { + const x = normalizeForCompare(a).replace(/\s+/g, ''); + const y = normalizeForCompare(b).replace(/\s+/g, ''); + if (!x || !y) return 0; + return 1 - editDistance(x, y) / Math.max(x.length, y.length); +} + +/** Standard word error rate: (substitutions + deletions + insertions) / reference words. */ +export function wordErrorRate(reference, heard) { + const ref = tokenize(reference).map((t) => t.norm); + const hyp = tokenize(heard).map((t) => t.norm); + if (!ref.length) return hyp.length ? 1 : 0; + const errors = alignWords(ref, hyp).filter((o) => o.op !== 'equal').length; + return Math.min(1, errors / ref.length); +} + +/** + * Derive correction candidates by aligning what was read against what came + * back. Only mismatches whose *reference* side contains a tracked term are + * offered โ€” the point is fixing "beacon โ†’ bacon", not rewriting every filler + * word the transcriber fumbled. A mismatch the transcriber simply dropped + * (nothing heard) yields no suggestion: there is no text to rewrite. + * + * Two guards keep a sloppy reading from producing sloppy rules. A streamer who + * skips a line, ad-libs, or reads out of order makes the alignment collapse + * into one enormous mismatch โ€” so segments past a few words are discarded, and + * both sides must actually resemble each other. A bad suggestion is worse than + * a missing one: it silently rewrites real speech for the rest of the stream. + */ +export function deriveCorrections(reference, heard, terms = []) { + const ref = tokenize(reference); + const hyp = tokenize(heard); + const refNorms = ref.map((t) => t.norm); + const ops = alignWords(refNorms, hyp.map((t) => t.norm)); + + const phrases = terms.map((t) => normalizeForCompare(t)).filter(Boolean); + // Individual words of multi-word terms count as hits too ("Knight" of + // "Hollow Knight"), minus the short connectives that would match anything. + const termWords = new Set(); + for (const phrase of phrases) { + for (const word of phrase.split(' ')) { + if (word.length >= 4 || phrase.split(' ').length === 1) termWords.add(word); + } + } + // Where a multi-word term sits in the reference, so a mangle of one of its + // words widens to the whole name: "night โ†’ Knight" would rewrite every + // "good night" all stream, while "hollow night โ†’ Hollow Knight" cannot. + const spans = []; + for (const phrase of phrases) { + const words = phrase.split(' '); + if (words.length < 2) continue; + for (let s = 0; s + words.length <= refNorms.length; s++) { + if (words.every((w, k) => refNorms[s + k] === w)) spans.push([s, s + words.length - 1]); + } + } + const refToHyp = new Map(); + for (const o of ops) if (o.ref >= 0 && o.hyp >= 0) refToHyp.set(o.ref, o.hyp); + const clean = (s) => s.replace(/[^\p{L}\p{N}\s'-]/gu, ' ').replace(/\s+/g, ' ').trim(); + + const suggestions = []; + const seen = new Set(); + let k = 0; + while (k < ops.length) { + if (ops[k].op === 'equal') { k++; continue; } + // Group the consecutive mismatch into one segment, so a two-word name + // heard as three words becomes a single replacement. + const refIdx = []; + const hypIdx = []; + while (k < ops.length && ops[k].op !== 'equal') { + if (ops[k].ref >= 0) refIdx.push(ops[k].ref); + if (ops[k].hyp >= 0) hypIdx.push(ops[k].hyp); + k++; + } + if (!refIdx.length || !hypIdx.length) continue; + if (refIdx.length > MAX_SEGMENT_WORDS || hypIdx.length > MAX_SEGMENT_WORDS) continue; + + let refStart = Math.min(...refIdx); + let refEnd = Math.max(...refIdx); + const span = spans.find(([s, e]) => refIdx.some((i) => i >= s && i <= e)); + if (span) { + refStart = Math.min(refStart, span[0]); + refEnd = Math.max(refEnd, span[1]); + } else if (!refIdx.some((i) => termWords.has(refNorms[i]))) { + continue; // nothing worth protecting in this mismatch + } + + // Widen the heard side to whatever aligned with the widened reference. + const hypRange = [...hypIdx]; + for (let i = refStart; i <= refEnd; i++) { + const j = refToHyp.get(i); + if (j !== undefined) hypRange.push(j); + } + const from = clean(hyp.slice(Math.min(...hypRange), Math.max(...hypRange) + 1).map((t) => t.raw).join(' ')); + const to = clean(ref.slice(refStart, refEnd + 1).map((t) => t.raw).join(' ')); + if (!from || !to || normalizeForCompare(from) === normalizeForCompare(to)) continue; + if (similarity(from, to) < MIN_SIMILARITY) continue; + const key = `${from.toLowerCase()}โ†’${to.toLowerCase()}`; + if (seen.has(key)) continue; + seen.add(key); + suggestions.push({ from, to }); + } + return suggestions.slice(0, 12); +} diff --git a/src/transcript.js b/src/transcript.js index 1a893ad..8324cfe 100644 --- a/src/transcript.js +++ b/src/transcript.js @@ -1,4 +1,5 @@ import { openSync, readSync, fstatSync, closeSync, existsSync, statSync } from 'node:fs'; +import { isHallucination, compileCorrections, applyCorrections } from './speech.js'; /** * LocalVocal can write plain text or SRT subtitles. In SRT, only every third @@ -25,9 +26,13 @@ const MAX_CHUNK = 256 * 1024; // safety cap on a single delta read * session reusing the path) are detected by comparing the first bytes of the * file, and reset the offset. A trailing line with no newline yet is held * back until it completes, so half-written words never enter the transcript. + * + * Every line then passes through the speech-quality layer (`src/speech.js`): + * Whisper's silence hallucinations are dropped, and the streamer's correction + * map is applied, before anything reaches the cast or the deck feed. */ export class TranscriptFeed { - constructor({ mode, file, textSource, pollSeconds, windowSeconds, obs, onSpeech }) { + constructor({ mode, file, textSource, pollSeconds, windowSeconds, obs, onSpeech, speech }) { this.mode = mode || 'off'; this.file = file; this.textSource = textSource; @@ -35,6 +40,11 @@ export class TranscriptFeed { this.windowMs = (windowSeconds ?? 120) * 1000; this.obs = obs; this.onSpeech = onSpeech || (() => {}); + // Live reference to config.speech โ€” hot config saves deep-assign into the + // same object, so corrections apply mid-stream without a relaunch. + this.speech = speech || {}; + this.compiled = null; // corrections compiled once, keyed on array identity + this.compiledFrom = null; this.entries = []; // {ts, text} this.lastHeardAt = null; // file-mode tail state @@ -76,9 +86,25 @@ export class TranscriptFeed { } } + /** Compiled corrections, rebuilt only when the config array is swapped out. */ + corrections() { + const list = this.speech?.corrections; + if (list !== this.compiledFrom) { + this.compiledFrom = list; + this.compiled = compileCorrections(list); + } + return this.compiled; + } + addLine(text) { - const cleaned = String(text).trim(); - if (!cleaned || isSrtMetadata(cleaned)) return; + const raw = String(text).trim(); + if (!raw || isSrtMetadata(raw)) return; + // Whisper filler on silence/music would otherwise read as the streamer + // speaking โ€” and a voice reply to something nobody said is worse than + // missing a line, so this is on by default. + if (this.speech?.dropHallucinations !== false && isHallucination(raw)) return; + const cleaned = applyCorrections(raw, this.corrections()); + if (!cleaned) return; // a correction mapping to "" deletes the line this.entries.push({ ts: Date.now(), text: cleaned }); this.lastHeardAt = Date.now(); this.prune(); diff --git a/test/speech.test.js b/test/speech.test.js new file mode 100644 index 0000000..b63351b --- /dev/null +++ b/test/speech.test.js @@ -0,0 +1,147 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { + isHallucination, compileCorrections, applyCorrections, wordErrorRate, + deriveCorrections, collectTerms, buildMicCheckScript, normalizeForCompare, similarity, +} from '../src/speech.js'; + +// ---- hallucination filtering ---- + +test('drops whisper filler that is the entire line', () => { + for (const line of ['Thank you.', 'thanks for watching!', 'Please subscribe to my channel', '[Music]', 'โ™ชโ™ชโ™ช', 'Bye.', '(upbeat music)']) { + assert.equal(isHallucination(line), true, `expected filler: ${line}`); + } +}); + +// The whole point of matching the entire line: streamers really do say these +// words, just inside sentences. Dropping those would be worse than the bug. +test('keeps real speech that merely contains a filler phrase', () => { + for (const line of ['thank you for the follow that is so nice', 'okay so the plan is we go left', 'bye bye little guy that was a rough end']) { + assert.equal(isHallucination(line), false, `expected speech: ${line}`); + } +}); + +test('drops decoder repetition loops but keeps ordinary repetition', () => { + assert.equal(isHallucination('yeah yeah yeah yeah yeah yeah yeah'), true); + assert.equal(isHallucination('no way no way no way no way'), true); + assert.equal(isHallucination('go go go we need to move'), false); +}); + +// ---- correction map ---- + +test('applies corrections on whole words only, case-insensitively', () => { + const compiled = compileCorrections([{ from: 'bacon', to: 'Beacon' }]); + assert.equal(applyCorrections('lol bacon is right', compiled), 'lol Beacon is right'); + assert.equal(applyCorrections('BACON said hi', compiled), 'Beacon said hi'); + // substring must not match, or "baconator" becomes "Beaconator" + assert.equal(applyCorrections('baconator burger', compiled), 'baconator burger'); +}); + +test('multi-word corrections tolerate different spacing', () => { + const compiled = compileCorrections([{ from: 'hyped host', to: 'Hype Ghost' }]); + assert.equal(applyCorrections('welcome to hyped host', compiled), 'welcome to Hype Ghost'); +}); + +test('a blank target deletes the phrase, and an emptied line collapses to ""', () => { + const compiled = compileCorrections([{ from: 'uh', to: '' }]); + assert.equal(applyCorrections('uh okay uh sure', compiled), 'okay sure'); + assert.equal(applyCorrections('uh', compiled), ''); +}); + +test('malformed correction entries are skipped, not thrown', () => { + const compiled = compileCorrections([null, { from: '', to: 'x' }, { from: 'a'.repeat(500), to: 'y' }, { from: 'ok', to: 'okay' }]); + assert.equal(compiled.length, 1); + assert.equal(applyCorrections('ok', compiled), 'okay'); + assert.deepEqual(compileCorrections(undefined), []); +}); + +test('regex metacharacters in a correction are matched literally', () => { + const compiled = compileCorrections([{ from: 'c++', to: 'C plus plus' }]); + assert.equal(applyCorrections('i write c++ daily', compiled), 'i write C plus plus daily'); +}); + +// ---- measurement ---- + +test('word error rate counts substitutions, deletions and insertions', () => { + assert.equal(wordErrorRate('hello there chat', 'hello there chat'), 0); + assert.equal(wordErrorRate('hello there chat', 'hello their chat'), 1 / 3); + assert.equal(wordErrorRate('hello there chat', 'hello chat'), 1 / 3); + assert.equal(wordErrorRate('hello there chat', 'hello there my chat'), 1 / 3); + // punctuation and casing are not errors + assert.equal(wordErrorRate('Hello, there chat!', 'hello there chat'), 0); +}); + +// ---- suggestion derivation ---- + +test('suggests a fix only for mismatches that hit a tracked term', () => { + const reference = 'okay chat, Beacon is asking about the build again.'; + const heard = 'okay chat bacon is asking about a build again'; + const fixes = deriveCorrections(reference, heard, ['Beacon']); + assert.deepEqual(fixes, [{ from: 'bacon', to: 'Beacon' }]); +}); + +test('groups a multi-word mangle into one replacement', () => { + const fixes = deriveCorrections('shoutout to Hype Ghost tonight', 'shoutout to hyped host tonight', ['Hype Ghost']); + assert.deepEqual(fixes, [{ from: 'hyped host', to: 'Hype Ghost' }]); +}); + +// Nothing was heard in place of the name, so there is no text a +// find-and-replace could rewrite โ€” suggesting one would be a no-op. +test('a dropped word yields no suggestion', () => { + assert.deepEqual(deriveCorrections('hey Wisp you there', 'hey you there', ['Wisp']), []); +}); + +// A streamer who skips a line or ad-libs collapses the alignment into one +// enormous mismatch. Turning that into a find-and-replace rule would silently +// rewrite real speech for the rest of the stream. +test('a wildly divergent reading produces no suggestions', () => { + const reference = 'okay chat, Beacon is asking about the build again. shoutout to emurray, thanks for hanging out tonight.'; + const heard = 'uh i totally lost my place there hang on let me find where i was on this thing'; + assert.deepEqual(deriveCorrections(reference, heard, ['Beacon', 'emurray']), []); +}); + +test('unrelated words dragged together by a misalignment are rejected', () => { + // "Wisp" and "emurray" look nothing alike โ€” pairing them is an alignment + // artifact, not a mishear. + assert.ok(similarity('whisp', 'emurray') < 0.45); + assert.ok(similarity('bacon', 'Beacon') > 0.7); + assert.ok(similarity('hollow night', 'Hollow Knight') > 0.7); +}); + +// "night โ†’ Knight" alone would rewrite every "good night" the streamer says. +test('a partly-mangled multi-word name widens to the whole name', () => { + const fixes = deriveCorrections('okay chat, Hollow Knight is fun', 'okay chat hollow night is fun', ['Hollow Knight']); + assert.deepEqual(fixes, [{ from: 'hollow night', to: 'Hollow Knight' }]); +}); + +test('a clean reading yields no suggestions', () => { + assert.deepEqual(deriveCorrections('hey Wisp you there', 'hey wisp you there', ['Wisp']), []); +}); + +test('derived fixes actually repair the transcript when applied', () => { + const reference = 'alright Beacon, you called it, that was rough.'; + const heard = 'alright bacon you called it that was rough'; + const fixes = deriveCorrections(reference, heard, ['Beacon']); + const repaired = applyCorrections(heard, compileCorrections(fixes)); + assert.ok(wordErrorRate(reference, repaired) < wordErrorRate(reference, heard)); +}); + +// ---- terms + script ---- + +test('collectTerms merges cast, channel and custom vocabulary without duplicates', () => { + const terms = collectTerms({ + cast: [{ name: 'Beacon' }, { name: 'Wisp' }], + twitchChannel: 'emurray', + vocabulary: ['Hollow Knight', 'wisp'], + }); + assert.deepEqual(terms, ['Beacon', 'Wisp', 'emurray', 'Hollow Knight']); +}); + +test('the mic-check script covers every term and works with none', () => { + const { lines, terms } = buildMicCheckScript({ terms: ['Beacon', 'Wisp', 'emurray'] }); + const script = normalizeForCompare(lines.join(' ')); + for (const term of terms) assert.ok(script.includes(normalizeForCompare(term)), `${term} missing from script`); + assert.ok(lines.length >= 3); + // No cast, no channel, no vocabulary โ€” still a readable baseline script. + assert.ok(buildMicCheckScript({ terms: [] }).lines.length >= 2); +}); diff --git a/test/transcript.test.js b/test/transcript.test.js index e6ba39a..a47cd59 100644 --- a/test/transcript.test.js +++ b/test/transcript.test.js @@ -9,13 +9,14 @@ const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); // Feeds are driven by calling pollFile() directly โ€” start() is never called, // so no intervals are created and tests control every tick. -function makeFeed(file) { +function makeFeed(file, speech) { const heard = []; const feed = new TranscriptFeed({ mode: 'file', file, pollSeconds: 999, windowSeconds: 120, + speech, onSpeech: (line) => heard.push(line), }); return { feed, heard }; @@ -89,6 +90,57 @@ test('getWindow prunes entries older than the window and honors sinceTs', (t) => assert.equal(feed.getWindow(Date.now() + 1000), ''); // nothing newer than the future }); +// The speech-quality layer sits between the tail and onSpeech, so a dropped +// hallucination never reaches the deck feed, a voice reply, or the window. +test('drops transcription hallucinations before they reach onSpeech', (t) => { + const file = withTmp(t); + const { feed, heard } = makeFeed(file, {}); + writeFileSync(file, 'Thank you.\n[Music]\nokay that boss is actually unfair\n'); + feed.pollFile(); + assert.deepEqual(heard, ['okay that boss is actually unfair']); + assert.equal(feed.getWindow(), 'okay that boss is actually unfair'); +}); + +test('hallucination filtering can be turned off', (t) => { + const file = withTmp(t); + const { feed, heard } = makeFeed(file, { dropHallucinations: false }); + writeFileSync(file, 'Thank you.\n'); + feed.pollFile(); + assert.deepEqual(heard, ['Thank you.']); +}); + +test('applies word fixes before the line is stored or announced', (t) => { + const file = withTmp(t); + const { feed, heard } = makeFeed(file, { corrections: [{ from: 'bacon', to: 'Beacon' }] }); + writeFileSync(file, 'lol bacon is right\n'); + feed.pollFile(); + assert.deepEqual(heard, ['lol Beacon is right']); + assert.equal(feed.getWindow(), 'lol Beacon is right'); +}); + +// Accepting mic-check fixes is a hot config save: the server deep-assigns into +// the same speech object the feed holds, so the next line must already use them +// without an app relaunch. +test('picks up corrections swapped in live, without reconstruction', (t) => { + const file = withTmp(t); + const speech = { corrections: [] }; + const { feed, heard } = makeFeed(file, speech); + writeFileSync(file, 'hey bacon\n'); + feed.pollFile(); + speech.corrections = [{ from: 'bacon', to: 'Beacon' }]; + appendFileSync(file, 'hey bacon again\n'); + feed.pollFile(); + assert.deepEqual(heard, ['hey bacon', 'hey Beacon again']); +}); + +test('a line deleted entirely by a fix never reaches onSpeech', (t) => { + const file = withTmp(t); + const { feed, heard } = makeFeed(file, { corrections: [{ from: 'subscribe now', to: '' }] }); + writeFileSync(file, 'subscribe now\nreal speech here\n'); + feed.pollFile(); + assert.deepEqual(heard, ['real speech here']); +}); + // textSource mode must drive the exact same onSpeech path as file mode โ€” the // deck's "heard" feed echo and voice replies hang off that callback, so both // transcription modes get identical behavior. From d1cd3c01ab1ff33554d721f21d7776297a915b91 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 25 Jul 2026 19:15:01 +0000 Subject: [PATCH 2/6] Reframe hallucination copy around mic silence, not music MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit LocalVocal only ever runs on the mic device, so there is no music in the audio. The filter still matters โ€” more so, if anything: whisper invents filler during near-silence, and a mic-only track is mostly near-silence between sentences. Documentation only; no behavior change. The bracketed sound-event case stays, since a line with no words in it is never speech. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SK871Ay2eiEzxTPorRufLP --- README.md | 2 +- public/settings.html | 2 +- src/speech.js | 18 ++++++++++++------ src/transcript.js | 6 +++--- 4 files changed, 17 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index 53631a5..27bc4ba 100644 --- a/README.md +++ b/README.md @@ -149,7 +149,7 @@ Quit Hype Ghost. | `transcript.mode` | `off`, `file` (tail LocalVocal's .txt/.srt output), or `textSource` (poll a text source over OBS WebSocket). | | `transcript.showInFeed` | Echo what voice awareness hears into the deck feed as faint ๐ŸŽ™ lines (party audio as ๐ŸŽง) so you can spot mishears the cast might be reacting to (default true). Deck only โ€” never in the cast's chat history, the recap, or the on-stream overlay. | | `transcript2.*` | *(Optional)* A **second** transcription channel for party/co-op audio (a separate audio device with its own LocalVocal filter). Same `mode`/`file`/`textSource`/`pollSeconds` as `transcript`, plus `label` โ€” how the cast refers to those people (e.g. "my co-op squad"). Treated as *other people*, never as the streamer. | -| `speech.dropHallucinations` | Discard speech-to-text filler invented on silence and music โ€” "Thank you.", "[Music]", "please subscribe", words stuck on repeat (default true). Matched whole-line only, so a real sentence containing those words survives. Applies to both channels. | +| `speech.dropHallucinations` | Discard speech-to-text filler โ€” "Thank you.", "Thanks for watching!", "please subscribe", words stuck on repeat (default true). Whisper invents these during near-silence, which is most of a mic-only track: the pauses between sentences, breaths, keyboard, room tone. Matched whole-line only, so a real sentence containing those words survives. Applies to both channels. | | `speech.corrections` | Word fixes applied to every transcript line before the cast sees it: `[{ "from": "bacon", "to": "Beacon" }]`. Whole words only, case-insensitive; a blank `to` deletes the phrase. Build this from **Settings โ†’ Voice โ†’ Mic check**, or by hand. Applies live on save. | | `speech.vocabulary` | Extra proper nouns worth getting right โ€” the game, in-jokes, regulars' names. Your ghosts' names and `twitch.channel` are included automatically. Used to build the mic-check script, and told to the cast so it reads a near-miss as the real word. | | `cadence.soloSeconds` / `quietSeconds` | Average gap between messages when alone / when real viewers are present. | diff --git a/public/settings.html b/public/settings.html index d47d8d8..63e4b5c 100644 --- a/public/settings.html +++ b/public/settings.html @@ -230,7 +230,7 @@

๐ŸŽ™๏ธ Voice awareness

- +
diff --git a/src/speech.js b/src/speech.js index ab27975..d071cac 100644 --- a/src/speech.js +++ b/src/speech.js @@ -6,10 +6,12 @@ * without running a second speech model on a machine that is already sharing * a GPU with OBS and a game: * - * 1. Drop Whisper's well-known hallucinations on silence/music ("Thank you.", - * "[Music]", subtitle-credit lines, stuck repetition loops). Left alone, - * each one looks like the streamer speaking and can trigger a voice reply - * to something nobody said. + * 1. Drop Whisper's well-known hallucinations ("Thank you.", "Thanks for + * watching!", subtitle-credit lines, stuck repetition loops). These are + * triggered by near-silence, and a mic-only feed is mostly near-silence โ€” + * pauses between sentences, breaths, keyboard clatter, fan and room tone. + * Left alone, each one looks like the streamer speaking and can trigger a + * voice reply to something nobody said. * 2. Apply a per-streamer correction map โ€” Whisper mangles proper nouns worst * (ghost names, channel name, game titles), and those are exactly the words * that matter here. @@ -67,7 +69,7 @@ export function tokenize(text) { // --------------------------------------------------------------------------- /** - * Whisper's greatest hits when fed silence, music, or mic noise. Matched + * Whisper's greatest hits when fed silence or mic noise. Matched * against the WHOLE normalized line only โ€” never as a substring โ€” because * every one of these is also something a streamer might genuinely say inside * a longer sentence ("thank you for the follow" must survive; a bare @@ -100,7 +102,11 @@ const FILLER_LINES = new Set([ 'all rights reserved', ]); -/** A line of only bracketed sound events / music glyphs: "[Music]", "โ™ชโ™ชโ™ช". */ +/** + * A line of only bracketed sound events: "[BLANK_AUDIO]", "[Music]", "โ™ชโ™ชโ™ช". + * Rarer on a mic-only feed than the filler phrases above, but it costs nothing + * to catch โ€” a line with no words in it is never speech. + */ function isSoundEventOnly(text) { const stripped = String(text ?? '') .replace(/[[(][^\])]*[\])]/g, ' ') diff --git a/src/transcript.js b/src/transcript.js index 8324cfe..f60a70e 100644 --- a/src/transcript.js +++ b/src/transcript.js @@ -99,9 +99,9 @@ export class TranscriptFeed { addLine(text) { const raw = String(text).trim(); if (!raw || isSrtMetadata(raw)) return; - // Whisper filler on silence/music would otherwise read as the streamer - // speaking โ€” and a voice reply to something nobody said is worse than - // missing a line, so this is on by default. + // Whisper filler invented during the near-silence between sentences would + // otherwise read as the streamer speaking โ€” and a voice reply to something + // nobody said is worse than missing a line, so this is on by default. if (this.speech?.dropHallucinations !== false && isHallucination(raw)) return; const cleaned = applyCorrections(raw, this.corrections()); if (!cleaned) return; // a correction mapping to "" deletes the line From 104360e957b9f8d78c37930688c668679007cde9 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 25 Jul 2026 20:00:56 +0000 Subject: [PATCH 3/6] Let advanced users pick any Claude model ID The Anthropic model dropdown was a closed list of three, so a model released after a build shipped - or an older one worth pinning - was unreachable without an app update. The OpenAI-compatible path already had a free-text model field; this gives the Claude path the same escape hatch. - Settings -> Brain -> "Custom model ID..." reveals a text field. The saved id stays the source of truth: one that isn't in the catalog shows in that field rather than the dropdown silently snapping to a different model. - The hint says plainly when the cost meter has no rates for the id, instead of implying a price it can't compute. Prefix-aware, so a dated snapshot of a known model is still recognised as priced. - "Test brain" uses the typed id, so it can be verified before saving; an empty custom id is rejected on save. - The setup wizard carries a custom id as its own option, so re-running it no longer blanks the model (an unmatched value left the select with no selection, and saved an empty string). - Catalog: add claude-opus-5, keep 4.8 so existing configs keep costing correctly, and fix the stale Opus hourly estimate (~$0.60 -> ~$0.33; it is 1.7x Sonnet at current rates, not 3x). Same correction in the README. - Tests cover isKnownModel and assert no catalog id shadows another by prefix, which would misprice a dated id. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SK871Ay2eiEzxTPorRufLP --- README.md | 2 +- public/settings.html | 51 ++++++++++++++++++++++++++++++++++++++++++-- public/setup.html | 7 ++++++ src/models.js | 8 ++++++- test/models.test.js | 22 ++++++++++++++++++- 5 files changed, 85 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 27bc4ba..c981265 100644 --- a/README.md +++ b/README.md @@ -139,7 +139,7 @@ Quit Hype Ghost. | `stream.gameFromObs` | Fallback: guess the game from an OBS **Game/Window Capture** source's captured window when Twitch has no category (default true). | | `talkingPoints` | Topics you want covered; the cast works one in occasionally when it fits naturally. | | `anthropic.apiKey` | Anthropic API key (console.anthropic.com), or set `ANTHROPIC_API_KEY` env var. | -| `anthropic.model` | `claude-sonnet-5` is the sweet spot; `claude-haiku-4-5` ~3x cheaper, `claude-opus-4-8` ~3x pricier and chattiest. | +| `anthropic.model` | `claude-sonnet-5` is the sweet spot; `claude-haiku-4-5` ~3x cheaper, `claude-opus-5` ~1.7x pricier and chattiest. Any other model ID your key can reach also works โ€” Settings โ†’ Brain โ†’ **Custom model IDโ€ฆ** (or just set it here), so a new release isn't gated behind an app update. It must be **vision-capable**, and the cost meter only prices the listed models: on anything else the deck shows a message count with no dollar figure. | | `bot.name` / `bot.personality` | Legacy single-ghost fields, kept in sync with `cast[0]` for backward compatibility. | | `obs.url` / `obs.password` | OBS WebSocket (default port 4455). | | `obs.screenshotWidth` | Screenshot width in px (default 800). Image tokens scale with pixels (wร—hรท750) โ€” JPEG quality has no effect on tokens. | diff --git a/public/settings.html b/public/settings.html index 63e4b5c..77c061b 100644 --- a/public/settings.html +++ b/public/settings.html @@ -130,6 +130,11 @@

๐Ÿง  Brain

Keys at console.anthropic.com.

+
@@ -454,6 +459,8 @@

Danger zone

const fields = [...document.querySelectorAll('[data-path]')]; let cfg = null, apiKeySaved = false, canRestart = false; let palette = {}, archetypes = [], castData = []; + const CUSTOM_MODEL = '__custom'; + let modelIds = []; // ids the cost meter has rates for let ACCENTS = {}; // accent hex values come from /api/config function applyAccent(name) { @@ -583,7 +590,12 @@

Danger zone

cfg = data.config; apiKeySaved = data.apiKeySaved; canRestart = data.canRestart; palette = data.palette || {}; archetypes = data.archetypes || []; castData = data.cast || []; ACCENTS = data.accents || {}; - $('modelSelect').innerHTML = data.models.map((m) => ``).join(''); + modelIds = data.models.map((m) => m.id); + // The catalog, plus an escape hatch: any model id the key can reach, so a + // release newer than this build (or an older one worth pinning) isn't + // gated behind an app update. + $('modelSelect').innerHTML = data.models.map((m) => ``).join('') + + ``; for (const f of fields) { const v = getPath(cfg, f.dataset.path); if (f.dataset.type === 'bool') f.checked = v !== false; @@ -610,6 +622,7 @@

Danger zone

$('accentField').value = d.dataset.acc; applyAccent(d.dataset.acc); $('accentRow').querySelectorAll('.accent-dot').forEach((x) => x.classList.toggle('on', x === d)); }); + syncModelChoice(); toggleProvider(); initTts(); }); @@ -733,6 +746,34 @@

Danger zone

box.style.display = ''; }; + // ---- model choice ---- + // The saved id is the source of truth. One that isn't in the catalog is a + // custom pick, not a broken one โ€” show it in the text field rather than + // silently snapping the dropdown to a different model. + const chosenModel = () => + $('modelSelect').value === CUSTOM_MODEL ? $('customModel').value.trim() : $('modelSelect').value; + + function syncModelChoice() { + const saved = (cfg.anthropic && cfg.anthropic.model) || ''; + const known = modelIds.includes(saved); + $('modelSelect').value = known ? saved : CUSTOM_MODEL; + if (!known && saved) $('customModel').value = saved; + toggleCustomModel(); + } + + function toggleCustomModel() { + const custom = $('modelSelect').value === CUSTOM_MODEL; + $('customModelRow').style.display = custom ? '' : 'none'; + if (!custom) return; + // Say so plainly when the meter can't price it, rather than showing $0.00. + const id = $('customModel').value.trim(); + $('customModelCost').textContent = !id || modelIds.some((known) => id.startsWith(known)) + ? '' + : 'No published rates for this one, so the deck shows a message count without a dollar figure.'; + } + $('modelSelect').onchange = toggleCustomModel; + $('customModel').oninput = toggleCustomModel; + function toggleProvider() { const oai = $('providerSel').value === 'openai'; $('anthFields').style.display = oai ? 'none' : ''; @@ -818,7 +859,7 @@

Danger zone

const oai = $('providerSel').value === 'openai'; const body = oai ? { provider: 'openai', baseUrl: val('brain.openaiBaseUrl').trim(), model: val('brain.openaiModel').trim(), apiKey: val('brain.openaiApiKey').trim() } - : { apiKey: val('anthropic.apiKey').trim(), model: val('anthropic.model') }; + : { apiKey: val('anthropic.apiKey').trim(), model: chosenModel() }; const r = await fetch('/api/test-key', { method:'POST', headers:{'Content-Type':'application/json'}, body: JSON.stringify(body) }).then((x)=>x.json()).catch((e)=>({ok:false,error:e.message})); el.className = 'result ' + (r.ok ? 'ok':'err'); el.textContent = r.ok ? 'โœ“ works' + (r.note ? ' โ€” '+r.note : '') : 'โœ— ' + (r.error || 'failed'); @@ -919,6 +960,12 @@

Danger zone

? { ...cfg.bot2, enabled: true, name: roster[1].name, personality: roster[1].personality } : { ...cfg.bot2, enabled: false }; + // The dropdown's own value is a sentinel when "Custom model IDโ€ฆ" is picked, + // so the real id has to be written after the generic field loop. + setPath(cfg, 'anthropic.model', chosenModel()); + if (cfg.brain.provider === 'anthropic' && !cfg.anthropic.model) { + $('saveResult').className = 'result err'; $('saveResult').textContent = 'enter a model ID, or pick one from the list (Brain tab)'; setTab('brain'); return; + } if (cfg.brain.provider === 'anthropic' && !cfg.anthropic.apiKey && !apiKeySaved) { $('saveResult').className = 'result err'; $('saveResult').textContent = 'an API key is required (Brain tab)'; setTab('brain'); return; } diff --git a/public/setup.html b/public/setup.html index 0b40e83..18d8472 100644 --- a/public/setup.html +++ b/public/setup.html @@ -185,6 +185,13 @@

Your room is ready!

$('apiKey').placeholder = apiKeySaved ? 'saved โœ“ โ€” leave blank to keep the current key' : 'sk-ant-โ€ฆ'; if (data.openaiKeySaved) $('oaiKey').placeholder = 'saved โœ“ โ€” leave blank to keep'; if (data.twitchSecretSaved) $('twSecret').placeholder = 'saved โœ“ โ€” leave blank to keep'; + // A model set in Settings โ†’ Brain โ†’ "Custom model IDโ€ฆ" isn't in the catalog. + // Carry it as its own option so re-running the wizard keeps it instead of + // blanking it (an unmatched value would leave the select with no selection). + if (cfg.anthropic.model && !data.models.some((m) => m.id === cfg.anthropic.model)) { + $('model').insertAdjacentHTML('beforeend', + ``); + } $('model').value = cfg.anthropic.model; $('provider').value = cfg.brain.provider; $('oaiBase').value = cfg.brain.openaiBaseUrl; $('oaiModel').value = cfg.brain.openaiModel; $('oaiKey').value = cfg.brain.openaiApiKey; toggleProvider(); diff --git a/src/models.js b/src/models.js index 1fc6661..71f1734 100644 --- a/src/models.js +++ b/src/models.js @@ -6,9 +6,15 @@ export const MODELS = [ { id: 'claude-sonnet-5', label: 'Sonnet โ€” recommended (~$0.20/hr)', inRate: 3, outRate: 15 }, { id: 'claude-haiku-4-5', label: 'Haiku โ€” budget (~$0.07/hr)', inRate: 1, outRate: 5 }, - { id: 'claude-opus-4-8', label: 'Opus โ€” premium (~$0.60/hr)', inRate: 5, outRate: 25 }, + { id: 'claude-opus-5', label: 'Opus โ€” premium (~$0.33/hr)', inRate: 5, outRate: 25 }, + { id: 'claude-opus-4-8', label: 'Opus 4.8 โ€” previous premium (~$0.33/hr)', inRate: 5, outRate: 25 }, ]; +/** True when the cost meter has published rates for this model id. */ +export function isKnownModel(modelId) { + return MODELS.some((m) => modelId && modelId.startsWith(m.id)); +} + /** * Cost of one API response in dollars, or null for models we have no rates * for. Cache reads bill at 10% of the input rate, 5-minute cache writes at diff --git a/test/models.test.js b/test/models.test.js index b78fb07..dd92ee6 100644 --- a/test/models.test.js +++ b/test/models.test.js @@ -1,6 +1,6 @@ import { test } from 'node:test'; import assert from 'node:assert/strict'; -import { MODELS, messageCost } from '../src/models.js'; +import { MODELS, messageCost, isKnownModel } from '../src/models.js'; test('plain input/output tokens bill at the listed rates', () => { // Sonnet: $3/MTok in, $15/MTok out @@ -32,3 +32,23 @@ test('every catalog entry has the fields the UI and cost meter need', () => { assert.equal(typeof m.outRate, 'number'); } }); + +// No catalog id may be a prefix of another, or messageCost's prefix match would +// bill a custom/dated id at the wrong model's rates. +test('catalog ids do not shadow each other by prefix', () => { + for (const a of MODELS) { + for (const b of MODELS) { + if (a !== b) assert.ok(!a.id.startsWith(b.id), `${a.id} shadowed by ${b.id}`); + } + } +}); + +// Drives the "Custom model IDโ€ฆ" hint: it warns the deck can't show a dollar +// figure only when the id really has no rates. +test('isKnownModel tracks what the cost meter can price', () => { + assert.equal(isKnownModel('claude-sonnet-5'), true); + assert.equal(isKnownModel('claude-opus-5-20260101'), true); // dated snapshot + assert.equal(isKnownModel('claude-opus-4-7'), false); + assert.equal(isKnownModel(''), false); + assert.equal(isKnownModel(undefined), false); +}); From 4f1e6a82735550ab2c86fa540b48b7492071643c Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 25 Jul 2026 21:13:35 +0000 Subject: [PATCH 4/6] Add a per-game screen guide, learn corrections live, warn on silence MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Five things, all aimed at the cast reacting to what is actually happening. Prompt cache TTL 5m -> 1h. The two breakpoints in brain.js sit under Sonnet's 1024-token minimum today, so they are inert; more importantly the default 5-minute entry expires between messages at `quietSeconds` (8 minutes), so every call would pay the write premium and never read. A 1h write costs 2x instead of 1.25x but survives the gaps. Screen guide (---GAMEINFO---). The cast learns how to READ each game - HUD layout, what a death or menu screen looks like, where the streamer's overlay sits - and reuses it on later streams of that game. Written as a tail section on generations that already happen (no extra API calls) and refined across a session, so it self-corrects instead of baking in whatever the first frame was. Lives in the cached context block, keyed by game in game-notes.json. Visible and editable at Settings -> Stream, including guides from past streams: it is treated as truth on every message, so a wrong line has to be fixable. Click-to-fix in the deck. Click a mishear in the ๐ŸŽ™ feed, retype it, press Enter; the words that changed become corrections and apply to the very next line. deriveCorrections gains options because the gates that keep the mic check honest are wrong here - the streamer is not guessing at what they said. Silent-transcript warning. Voice on, OBS connected, and nothing EVER heard for ten minutes is a wrong path or a missing filter, and it looks exactly like the cast ignoring you. Gated on "nothing ever", not "nothing lately", so a streamer who simply has not talked yet is never nagged. Mic check history. Past scores persist, so a LocalVocal change is judged by a number rather than a feeling. Also: game-notes.json and miccheck.json are added to DATA_FILES, or "Move data folder..." would silently leave them behind - with a test asserting dataFilePaths and DATA_FILES stay in step. New test/brain.test.js stubs the transport to cover tail parsing and prompt assembly. Release-Bump: minor Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SK871Ay2eiEzxTPorRufLP --- README.md | 14 ++++ config.example.json | 3 +- electron/main.js | 4 +- public/dashboard.html | 70 ++++++++++++++++++-- public/settings.html | 74 +++++++++++++++++++++ src/brain.js | 52 +++++++++++---- src/loop.js | 12 ++++ src/server.js | 150 +++++++++++++++++++++++++++++++++++++++++- src/speech.js | 10 ++- src/storage.js | 7 +- test/brain.test.js | 107 ++++++++++++++++++++++++++++++ test/speech.test.js | 18 +++++ test/storage.test.js | 14 +++- 13 files changed, 508 insertions(+), 27 deletions(-) create mode 100644 test/brain.test.js diff --git a/README.md b/README.md index c981265..ad3e96a 100644 --- a/README.md +++ b/README.md @@ -77,6 +77,13 @@ Quit Hype Ghost. - **Settings โ†’ About** shows the app version, the honesty/privacy rules at a glance, where your data lives, and **Reset to factory defaults** โ€” wipes all settings and memory, then re-runs the setup wizard. +- **Game screen guide:** the cast learns how to *read* each game it watches โ€” where the + health bar is, what a death or menu screen looks like, where your webcam and alerts sit โ€” + and reuses it on later streams of that game. It writes this itself from screenshots it is + already being shown, so it costs no extra API calls, and it sits in the cached part of the + prompt. The point is reacting to game **state** ("your health is gone") rather than just + appearance. It's plain text at **Settings โ†’ Stream โ†’ Screen guide**: read it, and edit it + if it gets something wrong, because it's treated as truth on every message. - **Data folder:** settings, session memory, the cross-stream profile, and the app log (`hype-ghost.log`, rotating) live in `%APPDATA%\Hype Ghost` by default. **Settings โ†’ About โ†’ Move data folderโ€ฆ** relocates all of it to any folder you like (another drive, a @@ -108,6 +115,12 @@ Quit Hype Ghost. - **Fix the audio first.** Noise suppression *before* the Transcription filter in the chain, and no game or music audio bleeding into the mic track โ€” no model recovers from a dirty input. + - **Fix mishears as they happen.** Click any ๐ŸŽ™ line in the deck feed, retype what you + actually said, press Enter โ€” the words that changed become permanent corrections and + apply to the very next line, no save or restart. This is where the correction map + really grows: the mic check covers names you can predict up front, this covers whatever + a real stream throws at it. Re-running the mic check shows your past scores, so a + LocalVocal change can be judged by a number instead of a feeling. - Note the trade-off with the OBS crash below: the CPU backend is the safe one, and it's also what limits how big a model you can run. Move up model sizes until OBS starts struggling, then step back one. @@ -151,6 +164,7 @@ Quit Hype Ghost. | `transcript2.*` | *(Optional)* A **second** transcription channel for party/co-op audio (a separate audio device with its own LocalVocal filter). Same `mode`/`file`/`textSource`/`pollSeconds` as `transcript`, plus `label` โ€” how the cast refers to those people (e.g. "my co-op squad"). Treated as *other people*, never as the streamer. | | `speech.dropHallucinations` | Discard speech-to-text filler โ€” "Thank you.", "Thanks for watching!", "please subscribe", words stuck on repeat (default true). Whisper invents these during near-silence, which is most of a mic-only track: the pauses between sentences, breaths, keyboard, room tone. Matched whole-line only, so a real sentence containing those words survives. Applies to both channels. | | `speech.corrections` | Word fixes applied to every transcript line before the cast sees it: `[{ "from": "bacon", "to": "Beacon" }]`. Whole words only, case-insensitive; a blank `to` deletes the phrase. Build this from **Settings โ†’ Voice โ†’ Mic check**, or by hand. Applies live on save. | +| `memory.gameInfoEvery` | How often (in cast messages) the cast refreshes its **screen guide** for the current game โ€” what the HUD means, what a death screen looks like, where your overlay sits (default 8). Written as a tail section on generations that are already happening, so it costs no extra API calls. Stored per game in `game-notes.json` and reused on later streams; view and edit it at Settings โ†’ Stream. | | `speech.vocabulary` | Extra proper nouns worth getting right โ€” the game, in-jokes, regulars' names. Your ghosts' names and `twitch.channel` are included automatically. Used to build the mic-check script, and told to the cast so it reads a near-miss as the real word. | | `cadence.soloSeconds` / `quietSeconds` | Average gap between messages when alone / when real viewers are present. | | `cadence.jitter`, `burstChance`, `lullChance` | Natural rhythm: ยฑjitter on normal gaps, occasional quick bursts (0.3โ€“0.6x) and long lulls (1.6โ€“3x). | diff --git a/config.example.json b/config.example.json index 0ee439d..9ebbcdb 100644 --- a/config.example.json +++ b/config.example.json @@ -81,7 +81,8 @@ "memory": { "enabled": true, "updateEvery": 4, - "profileEvery": 12 + "profileEvery": 12, + "gameInfoEvery": 8 }, "twitch": { "clientId": "", diff --git a/electron/main.js b/electron/main.js index fd234e0..ccd9884 100644 --- a/electron/main.js +++ b/electron/main.js @@ -49,7 +49,7 @@ if (!gotLock) { // Dev (npm start): the project folder, so hacking on it stays simple. const defaultDataDir = app.isPackaged ? app.getPath('userData') : packageRoot; const { dir: dataDir, custom: customDataDir } = resolveDataDir(defaultDataDir); -const { configPath, notesPath, sessionPath, profilePath, logPath, skipPath } = dataFilePaths(dataDir); +const { configPath, notesPath, sessionPath, profilePath, gameInfoPath, micCheckPath, logPath, skipPath } = dataFilePaths(dataDir); // A tray-resident app has no visible console โ€” keep a rotating file log next // to the data for bug reports. Packaged only: dev has a real terminal. @@ -182,6 +182,8 @@ if (gotLock) app.whenReady().then(() => { notesPath, sessionPath, profilePath, + gameInfoPath, + micCheckPath, publicDir: path.join(packageRoot, 'public'), // Settings โ†’ About โ†’ Storage: shows where the data lives and moves it. // Files are copied (originals stay as a fallback), the pointer is diff --git a/public/dashboard.html b/public/dashboard.html index 33640b9..75a8a3a 100644 --- a/public/dashboard.html +++ b/public/dashboard.html @@ -127,6 +127,14 @@ .heard.party { align-self: flex-start; } .heard .ic { font-style: normal; flex: none; } .heard .txt { overflow-wrap: anywhere; min-width: 0; } + /* Clickable because a mishear is fixable in place โ€” see editHeard(). */ + .heard { cursor: text; } + .heard:hover .txt { color: var(--dim); text-decoration: underline dotted; text-underline-offset: 3px; } + .heard .heardfix { + font: inherit; font-style: normal; flex: 1; min-width: 0; margin: 0; + padding: 2px 8px; border-radius: var(--r-sm); + border: 1px solid var(--accent-line); background: var(--glass-2); color: var(--text); + } .heard .tm { font-style: normal; font-size: var(--fs-xs); flex: none; } .moment-card { align-self: center; display: flex; align-items: center; gap: 10px; padding: 7px 15px; border-radius: var(--r-full); background: var(--glass); border: 1px solid var(--accent-line); font-size: var(--fs-sm); animation: pop .4s cubic-bezier(.2,1.3,.5,1); } @keyframes pop { from { opacity: 0; transform: scale(.85); } to { opacity: 1; transform: none; } } @@ -269,7 +277,8 @@ micOff:'off', micQuiet:'quiet', micWait:'waiting', party:'Party', partyQuiet:'quiet', partyWait:'waiting', chatQuiet:'quiet', chatActive:'active', lost:'Connection lost โ€” reconnectingโ€ฆ', msgs:'msgs', watching:'watching', idle:'idle', noMoments:'No moments yet โ€” the cast flags clip-worthy plays here.', previewBody:'Preview mode โ€” your cast has no brain yet, so the room stays quiet.', previewCta:'Connect one in Settings', - heardTip:'What voice awareness transcribed โ€” the cast may be reacting to this. If it misheard you, that explains their reaction. (Deck only โ€” never shown on stream.)' }, + heardTip:'What voice awareness transcribed โ€” the cast may be reacting to this. If it misheard you, that explains their reaction. Click to fix a mishear: the words you change are corrected from now on. (Deck only โ€” never shown on stream.)', + fixSaved:'Learned:', fixNone:'Nothing changed โ€” no correction saved.' }, es: { brain:'Cerebro', viewers:'Espectadores', mic:'Micro', chat:'Chat', cast:'El elenco', energy:'Energรญa', chill:'Tranqui', balanced:'Normal', hype:'Mรกx', room:'Sala', modeAuto:'Auto', modeSolo:'Solo', modeViewers:'Pรบblico', say:'Di algo', pause:'Pausar', resume:'Reanudar', moments:'Momentos', recap:'Exportar', send:'Enviar', you:'Tรบ', @@ -279,7 +288,8 @@ micOff:'apagado', micQuiet:'silencio', micWait:'esperando', party:'Grupo', partyQuiet:'silencio', partyWait:'esperando', chatQuiet:'tranquilo', chatActive:'activo', lost:'Conexiรณn perdida โ€” reconectandoโ€ฆ', msgs:'msjs', watching:'mirando', idle:'inactivo', noMoments:'Aรบn no hay momentos โ€” el elenco marcarรก jugadas destacadas aquรญ.', previewBody:'Modo vista previa โ€” tu elenco aรบn no tiene cerebro, asรญ que la sala sigue en silencio.', previewCta:'Conรฉctalo en Ajustes', - heardTip:'Lo que la escucha de voz transcribiรณ โ€” el elenco puede estar reaccionando a esto. Si te entendiรณ mal, eso explica su reacciรณn. (Solo en la consola โ€” nunca en el stream.)' }, + heardTip:'Lo que la escucha de voz transcribiรณ โ€” el elenco puede estar reaccionando a esto. Si te entendiรณ mal, eso explica su reacciรณn. Haz clic para corregir: las palabras que cambies se corregirรกn a partir de ahora. (Solo en la consola โ€” nunca en el stream.)', + fixSaved:'Aprendido:', fixNone:'Sin cambios โ€” no se guardรณ ninguna correcciรณn.' }, }; let L = STR.en; const T = (k) => (L[k] ?? STR.en[k] ?? k); @@ -416,16 +426,66 @@ div.title = T('heardTip'); const time = new Date(h.ts).toLocaleTimeString([], { hour: '2-digit', minute: '2-digit' }); div.innerHTML = `${h.channel === 'party' ? '๐ŸŽง' : '๐ŸŽ™๏ธ'}${time}`; - div.querySelector('.txt').textContent = (h.channel === 'party' && h.label ? h.label + ' ยท ' : '') + 'โ€œ' + h.text + 'โ€'; + setHeardText(div, h, h.text); + div.onclick = () => editHeard(div, h); feed.appendChild(div); feed.scrollTop = feed.scrollHeight; } - function addSystem(text) { + function setHeardText(div, h, text) { + const span = document.createElement('span'); + span.className = 'txt'; + span.textContent = (h.channel === 'party' && h.label ? h.label + ' ยท ' : '') + 'โ€œ' + text + 'โ€'; + (div.querySelector('.txt') || div.querySelector('.heardfix')).replaceWith(span); + } + + // Click a mishear, retype what you actually said, press Enter. The words that + // changed become permanent corrections and apply to the very next line โ€” no + // save, no restart. This is how the correction map grows during a real + // stream, where the mishears are ones no setup script could predict. + function editHeard(div, h) { + if (div.querySelector('.heardfix')) return; + const input = document.createElement('input'); + input.className = 'heardfix'; + input.value = h.text; + div.querySelector('.txt').replaceWith(input); + input.focus(); + input.select(); + + let settled = false; + const cancel = () => { if (!settled) { settled = true; setHeardText(div, h, h.text); } }; + const commit = async () => { + if (settled) return; + const corrected = input.value.trim(); + if (!corrected || corrected === h.text) return cancel(); + settled = true; + const heard = h.text; + h.text = corrected; + setHeardText(div, h, corrected); + const r = await fetch('/api/speech/correct', { + method: 'POST', headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ heard, corrected }), + }).then((x) => x.json()).catch((e) => ({ ok: false, error: e.message })); + if (!r.ok) addSystem(r.error); + else if (!r.added.length) addSystem(T('fixNone')); + else addSystem(T('fixSaved') + ' ' + r.added.map((f) => `โ€œ${f.from}โ€ โ†’ โ€œ${f.to}โ€`).join(', '), 'โœ“'); + }; + input.onkeydown = (e) => { + e.stopPropagation(); + if (e.key === 'Enter') commit(); + else if (e.key === 'Escape') cancel(); + }; + input.onblur = cancel; + input.onclick = (e) => e.stopPropagation(); + } + + // โš  is the warning icon (DESIGN.md); a confirmed action passes โœ“ instead so + // a success doesn't read as a problem. + function addSystem(text, icon = 'โš ') { clearEmpty(); const div = document.createElement('div'); div.className = 'sys'; - div.textContent = 'โš  ' + text; + div.textContent = icon + ' ' + text; feed.appendChild(div); feed.scrollTop = feed.scrollHeight; } diff --git a/public/settings.html b/public/settings.html index 77c061b..80cdd63 100644 --- a/public/settings.html +++ b/public/settings.html @@ -197,6 +197,17 @@

๐ŸŽฎ Stream info for the cast

+
+ + + +
+ + + +
+

How to read the screen in this game โ€” where the health bar is, what a death screen looks like, where your webcam and alerts sit. The cast writes this itself from screenshots it's already being shown (no extra API calls), and refines it as it watches. It's kept per game and reused on future streams. Edit it if it gets something wrong: it's treated as truth on every message, so a bad line is worse than a blank one. Saves immediately โ€” separate from the button below.

+
@@ -245,6 +256,7 @@

๐ŸŽ™๏ธ Mic check & word fixes

+