Skip to content

Prototype avatar popout local voice#123

Open
HashwanthVen wants to merge 1 commit into
ianphil:masterfrom
HashwanthVen:feat/avatar-popout-local-voice
Open

Prototype avatar popout local voice#123
HashwanthVen wants to merge 1 commit into
ianphil:masterfrom
HashwanthVen:feat/avatar-popout-local-voice

Conversation

@HashwanthVen

@HashwanthVen HashwanthVen commented Apr 25, 2026

Copy link
Copy Markdown
Contributor

Summary

Avatar-first popped-out agent window for the developer prototype.

  • Routes popped-out agent windows to a dedicated avatar experience instead of the full chat UI (in Electron desktop only — browser mode falls back to the chat view).
  • Adds a TalkingHead-based avatar stage with captions, microphone, and transcript exit controls.
  • Keeps the main/default chat interface unchanged.
  • Adds local developer voice support using Windows built-in speech recognition for STT and Edge TTS for speech output.

Updated for monorepo restructure

This PR was originally opened before master's monorepo restructure into apps/desktop, apps/web, apps/server, packages/services, packages/shared. The full prototype has been rebased onto the new layout:

Original (flat src/) New location
src/main/ipc/voice.ts apps/desktop/src/main/ipc/voice.ts
src/main.ts (wiring) apps/desktop/src/main.ts (+ setupVoiceIPC())
src/preload.ts (voice namespace) apps/desktop/src/preload.ts
src/main/ipc/mind.ts (popout window tweaks) apps/desktop/src/main/ipc/mind.ts (860×900 + backgroundThrottling: false)
src/shared/types.ts (voice types) packages/shared/src/types.ts
src/renderer/components/avatar/* apps/web/src/renderer/components/avatar/*
src/renderer/types/talkinghead.d.ts apps/web/src/renderer/types/talkinghead.d.ts
src/renderer/index.css (avatar stage CSS) apps/web/src/renderer/index.css
src/renderer/components/layout/AppShell.tsx apps/web/src/renderer/components/layout/AppShell.tsx (with browser-vs-Electron guard via window.electronAPI?.voice)
src/test/helpers.ts (mockElectronAPI voice) apps/web/src/test/helpers.ts
vite.renderer.config.ts (optimizeDeps.exclude) apps/web/vite.config.ts
src/main/services/sdk/sdkPaths.ts (dev-mode fix) No longer needed — master's packages/services/src/sdk/sdkPaths.ts already enforces repo-local SDK in dev mode (more strictly than the original PR fix).

A new apps/web/src/browserApi.ts voice stub returns "unavailable in browser mode" errors so the browser variant doesn't crash if a developer opens the popout URL there. The avatar stage activates only when window.electronAPI.voice is present (Electron desktop).

Validation

  • npm run lint — typecheck + eslint + dependency-cruiser clean (315 modules, 593 deps, 0 violations)
  • npm test — full vitest suite clean (1050/1050 passing)
  • Focused tests passing at new locations:
    • apps/web/src/renderer/components/avatar/PopoutAvatarWindow.test.tsx (4)
    • apps/web/src/renderer/components/avatar/AgentAvatarPanel.logic.test.ts (7)
    • packages/services/src/sdk/sdkPaths.test.ts (4)
    • packages/services/src/sdk/CopilotClientFactory.test.ts (8)
  • Manual local dev smoke (npm start): avatar visible, mic captures speech via Windows System.Speech, agent reply plays via Edge TTS, red ✕ returns to transcript, main chat unchanged.

Dependencies added

  • @met4citizen/talkinghead@^1.7.0 (MIT) → apps/web — avatar rendering
  • edge-tts-universal@^1.4.0 (AGPL-3.0) → apps/desktop — local TTS output
    • Note: AGPL-3.0 is copyleft; flagged here for visibility. Used only at runtime in the desktop main process; not redistributed as source.

Follow-ups (intentionally out of scope here)

  • Voice/avatar provider abstraction. Current implementation has flat voice IPC and a single TalkingHead path. Formalizing this behind a VoiceProvider interface should happen alongside any second provider (e.g. Foundry).
  • Foundry / Azure Speech avatar integration as a separate optional provider — being explored on a separate spike branch, not mixed into the local prototype.
  • Browser variant avatar/voice support — the popout currently degrades to the chat view in apps/web. Adding a Web Speech API + browser TTS shim could enable a browser-only path.

Add an avatar-first popout window with local Windows speech recognition and Edge TTS playback for the developer prototype.

Rebased PR ianphil#123 onto current master after the apps/desktop + apps/web + packages/services + packages/shared monorepo restructure. New file locations:

- apps/desktop/src/main/ipc/voice.ts (was src/main/ipc/voice.ts)

- apps/web/src/renderer/components/avatar/* (was src/renderer/components/avatar/*)

- apps/web/src/renderer/types/talkinghead.d.ts

- packages/shared/src/types.ts: + VoiceRecognitionResult, VoiceSynthesisResult, ElectronAPI.voice

- apps/web/vite.config.ts: + optimizeDeps.exclude for @met4citizen/talkinghead

- apps/web/src/renderer/components/layout/AppShell.tsx: render avatar in popout when window.electronAPI.voice is available; gracefully fall back to chat in browser mode

- apps/web/src/browserApi.ts: + voice stub returning 'unavailable in browser' errors

- apps/desktop/src/main/ipc/mind.ts: popout window 860x900 + backgroundThrottling: false

Original PR's sdkPaths.ts dev-mode fix is no longer needed: master already enforces repo-local SDK in dev (packages/services/src/sdk/sdkPaths.ts) more strictly.

Validation: typecheck + eslint + deps:check (315 modules, 593 deps, 0 violations); all 4 ported test files pass at new locations (23/23); full suite passes (1050/1050).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@HashwanthVen HashwanthVen force-pushed the feat/avatar-popout-local-voice branch from 4802533 to d1cccb7 Compare May 5, 2026 18:55
@HashwanthVen HashwanthVen marked this pull request as ready for review May 5, 2026 18:55
@HashwanthVen

Copy link
Copy Markdown
Contributor Author

Hey @ianphil - Rebased onto current master, out of draft, all checks green.

What changed in this revision

Ported the prototype onto the new monorepo layout (apps/desktop, apps/web, packages/services, packages/shared). Same TalkingHead avatar + Windows System.Speech STT + Edge TTS - just at the right paths and importing through @chamber/shared / @chamber/services.

A couple things worth flagging:

  • Added a small browser-mode voice stub in apps/web/browserApi.ts so the popout degrades gracefully (chat view) when window.electronAPI.voice isn't present.
  • Dropped the original sdkPaths.ts tweak - your packages/services/src/sdk/sdkPaths.ts already enforces repo-local SDK in dev, more strictly than my fix did.
  • 2 new deps: @met4citizen/talkinghead@^1.7.0 (MIT, apps/web) and edge-tts-universal@^1.4.0 (AGPL-3.0, apps/desktop - flagged in PR body, runtime-only in main process).

Validation

npm run lint clean, npm test 1050/1050 passing, CI green, local smoke tested (avatar + mic + TTS + interrupt + main chat all working).

Follow-ups (intentionally out of scope)

Foundry/Speech avatar will land as a separate optional provider per our chat. Voice provider abstraction deferred until the second provider exists. Also naturally absorbs #19 (Whisper Light onboarding voice) which you closed as superseded - happy to split that out if you'd prefer something narrower.

Let me know if you'd like me to break anything out, tighten scope, or address review comments - happy to iterate. 🎙️

@HashwanthVen

Copy link
Copy Markdown
Contributor Author

One more thing worth surfacing while we're on this topic, Ian - I have an Azure Foundry-based avatar/speech version sitting on a separate fork branch that's the natural next step beyond this local prototype. Posting it here in case you want to take a look so we can decide how (or whether) to bring it into master.

Branch: https://github.com/HashwanthVen/chamber/tree/voice-avatar-foundry
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant