draft: add ARKit52 NPC voice lip sync lane#319
Merged
Conversation
dd500da to
7777bd2
Compare
3047fd0 to
bb2ab0d
Compare
This was referenced May 29, 2026
…root # Conflicts: # docs/design/82-alpha-npc-voice-and-facial-animation-lane.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PrototypeFacialAnimationDriverand checks actualSkinnedMeshRendererblendshape weights.secondspawn_voice_audio_chunk_get, so chunked Gemini audio retrieval is tested instead of only voice-session metadata.origin/devinto this draft branch after resolving the voice lane doc conflict.Verification
dotnet build Unity\SecondSpawn.AI.csproj --nologo: passed, 0 warnings.dotnet build Unity\SecondSpawn.UI.csproj --nologo: passed, 0 warnings.dotnet build Unity\Assembly-CSharp-Editor.csproj --nologo: passed. Warnings are existing Photon/Fusion package warnings, not new SECOND SPAWN code.npm.cmd run build: passed.npm.cmd test: passed after build, including Gemini voice pool selection coverage.git diff --check: passed.python -m py_compile tools\lipsync\check_unity_lipsync_contract.py tools\lipsync\check_lipsync_provider_readiness.py tools\lipsync\run_arkit52_smoke.py tools\lipsync\wav2arkit_http_server.py: passed.python tools\lipsync\check_unity_lipsync_contract.py: passed 13/13 checks for ARKit52 sidecar channel shape, Nakama chunk smoke assertions, Unity DTO shape, presenter lip-sync forwarding, facial driver blendshape mapping, synthetic editor driver smoke hook, focused/free-mode voice gating, and portrait ARKit status reporting.python tools\lipsync\run_arkit52_smoke.py: passed, 99 frames, 52 channels, 72.11 ms wall in the isolated worktree.python tools\lipsync\run_arkit52_smoke.py --include-nakama --actor-id npc-scrap-warden-0441 --text "Xin chao. Day la spike test ARKit nam muoi hai voi giong nhan vat nam.": passed, providergemini_tts, transportnakama_audio_chunks, voiceAlgenib, male hint, 158 ARKit52 frames, 52 channels, 104 ms lip-sync latency, 6580.26 ms total RPC wall time, 3 audio chunks, 335360 base64 chars fetched.Unity.exe -batchmode -nographics -quit -projectPath .claude/worktrees/arkit52-lipsync-root/Unity -executeMethod SecondSpawn.EditorTools.SecondSpawnFacialBlendshapeReportUtility.RunArkit52DriverSmoke: Unity did not reach the smoke method because Package Manager failed startup withThe "path" argument must be of type string. Received undefined. This is recorded as a worktree batch runner blocker, not a failed driver smoke.npc-scrap-warden-0441returned stable voiceAlgenibtwice with male hint;npc-clinic-operator-0320returnedSulafatwith female hint. Both returned Gemini audio chunks plus ARKit52 frames.python tools\lipsync\check_lipsync_provider_readiness.py --scratch-root D:\Projects\Second-Spawn\.tmp-import\lipsync-spike: passed. Onlywav2arkit_cpuis testable now; SALSA/uLipSync/Convai/FaceSync are not installed in Unity, NeuroSync is blocked by brokentorchaudio, and NVIDIA A2F SDK is blocked by CUDA 13.3 plus missingTENSORRT_ROOT_DIR.python tools\unity\check_visual_prefab_catalog.py: passed and reported 50 entries, 29 missing generated prefabs, 50 missing source assets, 3 unresolved generated prefab source GUIDs.Still Missing Before Ready
no_unity_session; official Unity MCP tools were visible, and the Editor log showed 20 official MCP tools discovered, butUnity_ManageEditorandUnity_ReadConsoletimed out after 120 seconds from this agent session.dev; this branch is not merged intodev, so the feature is not yet applied to the root game workspace.Related: #139, #288