diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 0000000..57f4a13 --- /dev/null +++ b/.dockerignore @@ -0,0 +1,27 @@ +.git +.gitignore +.DS_Store +.env +.venv +.pixi +__pycache__ +*.py[cod] +.pytest_cache +.mypy_cache +.ruff_cache + +node_modules +frontend/node_modules +frontend/dist +dist +build + +checkpoints +.hf-cache +data +recordings +tmp +logs + +models/SoulX-LiveAct +models/**/checkpoints diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..819d3ef --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,70 @@ +# Project Guidelines + +1. All code comments must be written in English. +2. Frontend changes must consider internationalization and include both Chinese and English user-facing text where applicable. +3. When modifying code, follow the minimum-change principle and keep edits narrowly scoped. + + +Behavioral guidelines to reduce common LLM coding mistakes. Merge with project-specific instructions as needed. + +**Tradeoff:** These guidelines bias toward caution over speed. For trivial tasks, use judgment. + +## 1. Think Before Coding + +**Don't assume. Don't hide confusion. Surface tradeoffs.** + +Before implementing: +- State your assumptions explicitly. If uncertain, ask. +- If multiple interpretations exist, present them - don't pick silently. +- If a simpler approach exists, say so. Push back when warranted. +- If something is unclear, stop. Name what's confusing. Ask. + +## 2. Simplicity First + +**Minimum code that solves the problem. Nothing speculative.** + +- No features beyond what was asked. +- No abstractions for single-use code. +- No "flexibility" or "configurability" that wasn't requested. +- No error handling for impossible scenarios. +- If you write 200 lines and it could be 50, rewrite it. + +Ask yourself: "Would a senior engineer say this is overcomplicated?" If yes, simplify. + +## 3. Surgical Changes + +**Touch only what you must. Clean up only your own mess.** + +When editing existing code: +- Don't "improve" adjacent code, comments, or formatting. +- Don't refactor things that aren't broken. +- Match existing style, even if you'd do it differently. +- If you notice unrelated dead code, mention it - don't delete it. + +When your changes create orphans: +- Remove imports/variables/functions that YOUR changes made unused. +- Don't remove pre-existing dead code unless asked. + +The test: Every changed line should trace directly to the user's request. + +## 4. Goal-Driven Execution + +**Define success criteria. Loop until verified.** + +Transform tasks into verifiable goals: +- "Add validation" → "Write tests for invalid inputs, then make them pass" +- "Fix the bug" → "Write a test that reproduces it, then make it pass" +- "Refactor X" → "Ensure tests pass before and after" + +For multi-step tasks, state a brief plan: +``` +1. [Step] → verify: [check] +2. [Step] → verify: [check] +3. [Step] → verify: [check] +``` + +Strong success criteria let you loop independently. Weak criteria ("make it work") require constant clarification. + +--- + +**These guidelines are working if:** fewer unnecessary changes in diffs, fewer rewrites due to overcomplication, and clarifying questions come before implementation rather than after mistakes. \ No newline at end of file diff --git a/README.ja.md b/README.ja.md index 7f30226..99c2b03 100644 --- a/README.ja.md +++ b/README.ja.md @@ -111,6 +111,37 @@ git clone https://github.com/dsd2077/CyberVerse.git cd CyberVerse ``` +### 任意: Pixi で再現可能な開発環境を作成する + +CyberVerse には、Python / Go / フロントエンドの開発チェックを再現するための Pixi 環境が 2 つ用意されています。 + +- `macos-arm`: Apple Silicon Mac（`osx-arm64`） +- `linux-x86`: x86_64 Linux（`linux-64`） + +Pixi が未インストールの場合は、先に実行します。 + +```bash +curl -fsSL https://pixi.sh/install.sh | sh +``` + +Apple Silicon Mac: + +```bash +pixi install -e macos-arm +pixi run -e macos-arm python-test +pixi run -e macos-arm go-test +``` + +x86_64 Linux: + +```bash +pixi install -e linux-x86 +pixi run -e linux-x86 python-test +pixi run -e linux-x86 go-test +``` + +Pixi 環境には、Python 3.10、Go 1.25、Node 22、FFmpeg、`soxr`、`libopus`、`opusfile` などの開発・テスト依存関係が含まれます。GPU アバター推論には、引き続き下記の CUDA / PyTorch とモデル重みの設定が必要です。 + ### ステップ 2: Python 環境を作成する ```bash diff --git a/README.ko.md b/README.ko.md index a6bb052..64a6770 100644 --- a/README.ko.md +++ b/README.ko.md @@ -111,6 +111,37 @@ git clone https://github.com/dsd2077/CyberVerse.git cd CyberVerse ``` +### 선택 사항: Pixi로 재현 가능한 개발 환경 만들기 + +CyberVerse는 Python / Go / 프런트엔드 개발 검사를 재현하기 위한 두 가지 Pixi 환경을 제공합니다. + +- `macos-arm`: Apple Silicon Mac(`osx-arm64`) +- `linux-x86`: x86_64 Linux(`linux-64`) + +Pixi가 없다면 먼저 설치합니다. + +```bash +curl -fsSL https://pixi.sh/install.sh | sh +``` + +Apple Silicon Mac: + +```bash +pixi install -e macos-arm +pixi run -e macos-arm python-test +pixi run -e macos-arm go-test +``` + +x86_64 Linux: + +```bash +pixi install -e linux-x86 +pixi run -e linux-x86 python-test +pixi run -e linux-x86 go-test +``` + +Pixi 환경에는 Python 3.10, Go 1.25, Node 22, FFmpeg, `soxr`, `libopus`, `opusfile` 등 개발 및 테스트 의존성이 포함됩니다. GPU 아바타 추론에는 아래 CUDA / PyTorch 및 모델 가중치 설정이 여전히 필요합니다. + ### 2단계: Python 환경 만들기 ```bash diff --git a/README.md b/README.md index 4dbdbe4..2c140df 100644 --- a/README.md +++ b/README.md @@ -112,6 +112,45 @@ git clone https://github.com/dsd2077/CyberVerse.git cd CyberVerse ``` +### Optional: Pixi reproducible development environments + +CyberVerse provides two Pixi environments for reproducible Python / Go / frontend development checks: + +- `macos-arm`: macOS on Apple Silicon (`osx-arm64`) +- `linux-x86`: Linux on x86_64 (`linux-64`) + +Install Pixi if needed: + +```bash +curl -fsSL https://pixi.sh/install.sh | sh +``` + +On macOS Apple Silicon: + +```bash +pixi install -e macos-arm +pixi run -e macos-arm python-test +pixi run -e macos-arm go-test +``` + +After copying `infra/cyberverse_config.example.yaml` to `cyberverse_config.yaml` and configuring `.env`, you can start the local Mac-side services in one foreground process: + +```bash +pixi run -e macos-arm dev +``` + +`Ctrl+C` stops the inference process, Go server, and frontend dev server. For weak Mac hardware, keep GPU/avatar inference on the remote Ubuntu deployment and point the local config or frontend environment at that remote API. + +On Linux x86_64: + +```bash +pixi install -e linux-x86 +pixi run -e linux-x86 python-test +pixi run -e linux-x86 go-test +``` + +The Pixi environments cover development and test dependencies such as Python 3.10, Go 1.25, Node 22, FFmpeg, `soxr`, `libopus`, and `opusfile`. GPU avatar inference still requires the CUDA / PyTorch and model-weight setup below. + ### Step 2: Create Python environment ```bash @@ -141,10 +180,19 @@ Edit `.env`, fill in your API keys: ``` DOUBAO_ACCESS_TOKEN=your_doubao_access_token # ByteDance Doubao omni model DOUBAO_APP_ID=your_doubao_app_id +HINDSIGHT_API_KEY=your_hindsight_api_key # Optional PersonaAgent long-term memory +HINDSIGHT_BASE_URL=https://hindsight.jmsu.top +HINDSIGHT_BANK_ID_TEMPLATE=cv:user:{user_id}:character:{character_id} +HINDSIGHT_USER_ID=local-user # Stable cross-session memory namespace +HINDSIGHT_USER_TAG=local-user # Legacy/common recall tag ``` Doubao Voice: get **App ID** / **API Key** per [Volcengine quick start](https://www.volcengine.com/docs/6561/2119699?lang=zh) → `DOUBAO_APP_ID` / `DOUBAO_ACCESS_TOKEN`. +PersonaAgent can use Hindsight for cross-session memory when `HINDSIGHT_BASE_URL` and optional `HINDSIGHT_API_KEY` are set in your local `.env`. By default it writes to banks shaped like `cv:user:{user_id}:character:{character_id}`; set `HINDSIGHT_USER_ID` to keep one user's memories stable across sessions. Keep real Hindsight keys only in `.env`; do not commit them. + +For the PersonaAgent digital-human path, the required external interfaces are the realtime omni model credentials, the text LLM credentials used by local subagents, optional `ZHIHU_ACCESS_SECRET` for research tools, optional Hindsight memory credentials, and the local or remote Go/API + avatar inference endpoints. + After the stack is running, you can change these values (and other API keys / service endpoints) from the web UI at **`/settings`** instead of editing `.env` only. ### Step 4: Download model weights diff --git a/README.zh-CN.md b/README.zh-CN.md index 8cd11eb..2aa1fe7 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -113,6 +113,45 @@ git clone https://github.com/dsd2077/CyberVerse.git cd CyberVerse ``` +### 可选：使用 Pixi 创建可复现开发环境 + +CyberVerse 提供两个 Pixi 环境，用于复现 Python / Go / 前端开发检查： + +- `macos-arm`：Apple Silicon Mac（`osx-arm64`） +- `linux-x86`：x86_64 Linux（`linux-64`） + +如未安装 Pixi，先执行： + +```bash +curl -fsSL https://pixi.sh/install.sh | sh +``` + +Apple Silicon Mac： + +```bash +pixi install -e macos-arm +pixi run -e macos-arm python-test +pixi run -e macos-arm go-test +``` + +复制 `infra/cyberverse_config.example.yaml` 为 `cyberverse_config.yaml` 并配置 `.env` 后，可以用一个前台进程启动本地 Mac 侧服务： + +```bash +pixi run -e macos-arm dev +``` + +按 `Ctrl+C` 会停止 inference、Go server 和前端 dev server。Mac 性能较弱时，把 GPU/avatar 推理留在远程 Ubuntu 部署上，本地配置或前端环境指向远程 API 即可。 + +x86_64 Linux： + +```bash +pixi install -e linux-x86 +pixi run -e linux-x86 python-test +pixi run -e linux-x86 go-test +``` + +Pixi 环境会安装 Python 3.10、Go 1.25、Node 22、FFmpeg、`soxr`、`libopus`、`opusfile` 等开发和测试依赖。GPU 数字人推理仍需按下方步骤配置 CUDA / PyTorch 与模型权重。 + ### 第 2 步：创建 Python 环境 ```bash @@ -143,10 +182,19 @@ cp infra/.env.example .env ``` DOUBAO_ACCESS_TOKEN=your_doubao_access_token # ByteDance Doubao 语音 LLM DOUBAO_APP_ID=your_doubao_app_id +HINDSIGHT_API_KEY=your_hindsight_api_key # 可选：PersonaAgent 长期记忆 +HINDSIGHT_BASE_URL=https://hindsight.jmsu.top +HINDSIGHT_BANK_ID_TEMPLATE=cv:user:{user_id}:character:{character_id} +HINDSIGHT_USER_ID=local-user # 跨会话稳定记忆命名空间 +HINDSIGHT_USER_TAG=local-user # 兼容旧配置的通用 recall tag ``` 豆包语音：按 [火山引擎快速入门](https://www.volcengine.com/docs/6561/2119699?lang=zh) 获取 **App ID** / **API Key**，填入 `DOUBAO_APP_ID` / `DOUBAO_ACCESS_TOKEN`。 +PersonaAgent 在本地 `.env` 配置 `HINDSIGHT_BASE_URL` 和可选 `HINDSIGHT_API_KEY` 后，会使用 Hindsight 做跨会话长期记忆。默认 bank 形如 `cv:user:{user_id}:character:{character_id}`；设置 `HINDSIGHT_USER_ID` 可让同一用户跨会话稳定使用同一记忆库。真实 Hindsight key 只放本地 `.env`，不要提交到代码仓库。 + +PersonaAgent 数字人链路需要的外部接口包括：实时 omni 模型凭证、本地 subagent 使用的文本 LLM 凭证、可选的 `ZHIHU_ACCESS_SECRET` 调研工具凭证、可选的 Hindsight 记忆凭证，以及本地或远程 Go/API + avatar inference endpoint。 + 服务启动后，你也可以在 Web UI 的 **`/settings`** 页面修改这些值（以及其他 API Key / 服务端点），而不必只依赖编辑 `.env`。 ### 第 4 步：下载模型权重 diff --git a/deploy/gpu/.env.example b/deploy/gpu/.env.example new file mode 100644 index 0000000..a51de22 --- /dev/null +++ b/deploy/gpu/.env.example @@ -0,0 +1,31 @@ +DASHSCOPE_API_KEY=your_dashscope_api_key +DASHSCOPE_BASE_URL=https://dashscope-intl.aliyuncs.com/compatible-mode/v1 + +# Direct WebRTC TURN relay secret. Change this before opening the service. +TURN_PASSWORD=replace_with_a_long_random_password + +# Optional Go internal task API token. Leave empty for local development; set a +# nonempty value before exposing internal task endpoints in production. +AGENT_INTERNAL_TOKEN= + +# Optional PersonaAgent task/search tools. +ZHIHU_ACCESS_SECRET=your_zhihu_access_secret + +# Optional PersonaAgent long-term memory. Keep real keys only in .env. +HINDSIGHT_ENABLED=true +HINDSIGHT_BASE_URL=https://hindsight.jmsu.top +HINDSIGHT_API_KEY=your_hindsight_api_key +HINDSIGHT_BANK_ID_TEMPLATE=cv:user:{user_id}:character:{character_id} +HINDSIGHT_USER_ID=local-user +HINDSIGHT_USER_TAG=local-user +HINDSIGHT_TIMEOUT_SECONDS=30 +HINDSIGHT_RECALL_BUDGET=low +HINDSIGHT_RECALL_MAX_RESULTS=5 +HINDSIGHT_RECALL_MAX_TOKENS=4096 +HINDSIGHT_RETAIN_CONTEXT=cyberverse realtime conversation +HINDSIGHT_DOCUMENT_ID_TEMPLATE=session:{session_id}:turn:{turn_id} +HINDSIGHT_RETAIN_MAX_CHARS=6000 + +# Optional Hugging Face settings for weight download. +# HF_ENDPOINT=https://hf-mirror.com +# HF_TOKEN= diff --git a/deploy/gpu/Dockerfile.inference b/deploy/gpu/Dockerfile.inference new file mode 100644 index 0000000..a430978 --- /dev/null +++ b/deploy/gpu/Dockerfile.inference @@ -0,0 +1,52 @@ +FROM pytorch/pytorch:2.8.0-cuda12.8-cudnn9-devel + +ENV DEBIAN_FRONTEND=noninteractive \ + PIP_NO_CACHE_DIR=1 \ + PYTHONUNBUFFERED=1 \ + PYTHONPATH=/workspace:/workspace/inference/generated + +RUN set -eux; \ + echo 'Acquire::ForceIPv4 "true";' > /etc/apt/apt.conf.d/99force-ipv4; \ + rm -f /etc/apt/sources.list.d/cuda*.list /etc/apt/sources.list.d/nvidia*.list; \ + sed -i \ + -e 's|http://archive.ubuntu.com/ubuntu|http://mirrors.aliyun.com/ubuntu|g' \ + -e 's|http://security.ubuntu.com/ubuntu|http://mirrors.aliyun.com/ubuntu|g' \ + /etc/apt/sources.list; \ + apt-get update && apt-get install -y --no-install-recommends \ + build-essential \ + ca-certificates \ + curl \ + ffmpeg \ + git \ + libgl1 \ + libglib2.0-0 \ + libgomp1 \ + libsm6 \ + libxext6 \ + libxrender1 \ + pkg-config \ + && rm -rf /var/lib/apt/lists/* + +WORKDIR /workspace + +COPY pyproject.toml README.md ./ +COPY proto/ ./proto/ +COPY inference/ ./inference/ +COPY agent_runtime/ ./agent_runtime/ +COPY models/flash_head/ ./models/flash_head/ + +RUN python -m pip install --upgrade pip setuptools wheel \ + && python -m pip install -e ".[inference,agent,qwen,omni,tts,flash_head]" \ + && mkdir -p inference/generated \ + && python -m grpc_tools.protoc \ + -I proto/ \ + --python_out=inference/generated \ + --grpc_python_out=inference/generated \ + proto/*.proto + +HEALTHCHECK --interval=30s --timeout=5s --retries=20 --start-period=600s \ + CMD python -c "import grpc; ch = grpc.insecure_channel('127.0.0.1:50051'); grpc.channel_ready_future(ch).result(timeout=3)" || exit 1 + +EXPOSE 50051 + +CMD ["python", "-m", "inference.server", "--config", "/workspace/deploy/gpu/cyberverse_config.gpu.yaml"] diff --git a/deploy/gpu/Dockerfile.server b/deploy/gpu/Dockerfile.server new file mode 100644 index 0000000..efd8f5f --- /dev/null +++ b/deploy/gpu/Dockerfile.server @@ -0,0 +1,55 @@ +FROM golang:1.25-bookworm AS builder + +ENV GOPROXY=https://goproxy.cn,direct \ + GOSUMDB=sum.golang.google.cn + +WORKDIR /build + +RUN set -eux; \ + echo 'Acquire::ForceIPv4 "true";' > /etc/apt/apt.conf.d/99force-ipv4; \ + sed -i \ + -e 's|http://deb.debian.org/debian|http://mirrors.aliyun.com/debian|g' \ + -e 's|http://deb.debian.org/debian-security|http://mirrors.aliyun.com/debian-security|g' \ + /etc/apt/sources.list.d/debian.sources; \ + apt-get update && apt-get install -y --no-install-recommends \ + build-essential \ + ca-certificates \ + libopus-dev \ + libopusfile-dev \ + libsoxr-dev \ + pkg-config \ + && rm -rf /var/lib/apt/lists/* + +COPY server/go.mod server/go.sum ./ +RUN go mod download + +COPY server/ ./ +RUN CGO_ENABLED=1 go build -o cyberverse-server ./cmd/cyberverse-server + +FROM debian:bookworm-slim + +RUN set -eux; \ + echo 'Acquire::ForceIPv4 "true";' > /etc/apt/apt.conf.d/99force-ipv4; \ + sed -i \ + -e 's|http://deb.debian.org/debian|http://mirrors.aliyun.com/debian|g' \ + -e 's|http://deb.debian.org/debian-security|http://mirrors.aliyun.com/debian-security|g' \ + /etc/apt/sources.list.d/debian.sources; \ + apt-get update && apt-get install -y --no-install-recommends \ + ca-certificates \ + curl \ + ffmpeg \ + libopus0 \ + libopusfile0 \ + libsoxr0 \ + && rm -rf /var/lib/apt/lists/* + +WORKDIR /app + +COPY --from=builder /build/cyberverse-server ./cyberverse-server + +HEALTHCHECK --interval=10s --timeout=5s --retries=30 --start-period=30s \ + CMD curl -fsS http://127.0.0.1:8080/api/v1/health >/dev/null || exit 1 + +EXPOSE 8080 8443 + +CMD ["./cyberverse-server", "--config", "/app/cyberverse_config.yaml"] diff --git a/deploy/gpu/README.zh-CN.md b/deploy/gpu/README.zh-CN.md new file mode 100644 index 0000000..5fb7eea --- /dev/null +++ b/deploy/gpu/README.zh-CN.md @@ -0,0 +1,120 @@ +# CyberVerse 远程 GPU 部署说明 + +这个目录用于把数字人重模型部署到 Ubuntu 24.04 + RTX 5090 机器上。Mac 本地只运行前端或轻量开发服务，通过 HTTP/WebSocket/WebRTC 访问远程 API。 + +## 架构 + +```mermaid +flowchart LR + Mac["macOS Frontend / Browser"] -->|"HTTP / WebSocket / WebRTC"| API["Remote Go Server :8080"] + API -->|"gRPC"| INF["Python Inference :50051"] + INF --> PERSONA["PersonaAgent + LocalTaskRuntime + LangGraph subagent"] + INF -->|"CUDA logical cuda:0"| GPU["Host RTX 5090 physical GPU 1"] + INF -->|"DashScope realtime/text/TTS/ASR"| QWEN["Qwen / DashScope APIs"] + PERSONA -->|"recall / retain"| HINDSIGHT["Hindsight API"] + PERSONA -->|"optional tools"| ZHIHU["Zhihu API"] +``` + +## GPU 选择 + +`docker-compose.yml` 使用： + +```yaml +gpus: + - driver: nvidia + device_ids: ["1"] + capabilities: ["gpu"] +environment: + - NVIDIA_VISIBLE_DEVICES=1 +``` + +这会把宿主机物理 GPU 1 暴露给容器。容器内部通常会把这张卡重新编号成 `cuda:0`，所以 `cyberverse_config.gpu.yaml` 里写 `cuda_visible_devices: 0` 和 `device: "cuda:0"` 是正确的，不会占用宿主机物理 GPU 0。 + +## 首次启动 + +在远程机器上： + +```bash +cd ~/project/CyberVerse +cp deploy/gpu/.env.example .env +``` + +编辑 `.env`，至少填： + +```bash +DASHSCOPE_API_KEY=... +TURN_PASSWORD=... +HINDSIGHT_API_KEY=... +HINDSIGHT_BASE_URL=https://hindsight.jmsu.top +HINDSIGHT_BANK_ID_TEMPLATE=cv:user:{user_id}:character:{character_id} +HINDSIGHT_USER_ID=local-user +HINDSIGHT_USER_TAG=local-user +``` + +如果需要知乎任务工具，再填 `ZHIHU_ACCESS_SECRET`。真实 Hindsight / Zhihu key 只放远程 `.env`，不要提交。 + +下载 FlashHead 权重： + +```bash +docker compose -f deploy/gpu/docker-compose.yml --profile weights run --rm weights +``` + +如果 Hugging Face 访问慢，可以先设置： + +```bash +export HF_ENDPOINT=https://hf-mirror.com +``` + +启动服务： + +```bash +docker compose -f deploy/gpu/docker-compose.yml up --build +``` + +前台运行时按 `Ctrl+C` 会停止 compose 管理的服务。需要后台运行时使用： + +```bash +docker compose -f deploy/gpu/docker-compose.yml up -d --build +``` + +## Mac 访问远程服务 + +远程服务启动后，Mac 前端可指向： + +```bash +VITE_API_BASE=http://122.205.95.186:8080/api/v1 +VITE_WS_BASE=ws://122.205.95.186:8080 +``` + +远程机器需要开放 TCP `8080` 和 `8443`。`50051` 只在 docker compose 内部网络使用，不需要对公网开放。 + +连通性检查： + +```bash +curl http://122.205.95.186:8080/api/v1/health +curl http://122.205.95.186:8080/api/v1/components +``` + +跨公网访问时，部分网络会拦截 HTTP `DELETE`。前端关闭会话默认使用兼容接口： + +```bash +POST http://122.205.95.186:8080/api/v1/sessions/{session_id}/close +``` + +原有 `DELETE /api/v1/sessions/{session_id}` 仍保留，适合内网或未拦截 DELETE 的环境。 + +## 权重来源 + +当前默认使用 FlashHead Lite 单卡配置，权重目录为： + +```text +checkpoints/SoulX-FlashHead-1_3B +checkpoints/wav2vec2-base-960h +``` + +来源： + +- `Soul-AILab/SoulX-FlashHead-1_3B` +- `facebook/wav2vec2-base-960h` + +如果要改成 Pro 模型，把 `deploy/gpu/cyberverse_config.gpu.yaml` 里的 `model_type: "lite"` 改成 `model_type: "pro"`。单张 5090 可以尝试 Pro，但实时性通常不如 Lite；FlashHead 官方说明 Pro 实时体验更偏向双 5090 + SageAttention。 diff --git a/deploy/gpu/cyberverse_config.gpu.yaml b/deploy/gpu/cyberverse_config.gpu.yaml new file mode 100644 index 0000000..f3a37b1 --- /dev/null +++ b/deploy/gpu/cyberverse_config.gpu.yaml @@ -0,0 +1,156 @@ +server: + host: "0.0.0.0" + http_port: 8080 + grpc_port: 50051 + cors_origins: ["*"] + +livekit: + url: "ws://unused-livekit:7880" + api_key: "devkey" + api_secret: "secret" + +warmup: + enabled: true + distributed: + enabled: false + timeout_s: 30 + +inference: + avatar: + default: "flash_head" + runtime: + cuda_visible_devices: 0 + world_size: 1 + flash_head: + plugin_class: "inference.plugins.avatar.flash_head_plugin.FlashHeadAvatarPlugin" + models_dir: "/workspace/models" + checkpoint_dir: "/workspace/checkpoints/SoulX-FlashHead-1_3B" + wav2vec_dir: "/workspace/checkpoints/wav2vec2-base-960h" + model_type: "lite" + device: "cuda:0" + seed: 9999 + compile_model: true + compile_vae: true + dist_worker_main_thread: true + infer_params: + frame_num: 33 + motion_frames_latent_num: 2 + tgt_fps: 20 + sample_rate: 16000 + sample_shift: 5 + color_correction_strength: 1.0 + cached_audio_duration: 8 + num_heads: 12 + height: 512 + width: 512 + + omni: + default: "qwen_omni" + qwen_omni: + plugin_class: "inference.plugins.voice_llm.qwen_omni_realtime.QwenOmniRealtimePlugin" + api_key: "${DASHSCOPE_API_KEY}" + model: "qwen3.5-omni-flash-realtime" + voice: "Tina" + input_sample_rate: 16000 + output_sample_rate: 24000 + vad_type: "semantic_vad" + vad_threshold: 0.5 + vad_silence_duration_ms: 800 + + persona: + persona: + plugin_class: "inference.plugins.voice_llm.persona_agent.PersonaAgentPlugin" + model_provider: "qwen_omni" + checkpoint_db_path: "/workspace/data/tasks/langgraph_checkpoints.db" + llm: + provider: "qwen" + memory: + hindsight: + enabled: "${HINDSIGHT_ENABLED}" + base_url: "${HINDSIGHT_BASE_URL}" + api_key: "${HINDSIGHT_API_KEY}" + bank_id_template: "${HINDSIGHT_BANK_ID_TEMPLATE}" + user_id: "${HINDSIGHT_USER_ID}" + user_tag: "${HINDSIGHT_USER_TAG}" + timeout_seconds: "${HINDSIGHT_TIMEOUT_SECONDS}" + tags: + - "source:cyberverse" + recall: + budget: "${HINDSIGHT_RECALL_BUDGET}" + max_results: "${HINDSIGHT_RECALL_MAX_RESULTS}" + max_tokens: "${HINDSIGHT_RECALL_MAX_TOKENS}" + retain: + context: "${HINDSIGHT_RETAIN_CONTEXT}" + document_id_template: "${HINDSIGHT_DOCUMENT_ID_TEMPLATE}" + max_chars: "${HINDSIGHT_RETAIN_MAX_CHARS}" + tools: + zhihu: + access_secret: "${ZHIHU_ACCESS_SECRET}" + api_base: "https://developer.zhihu.com" + timeout_seconds: 30 + zhida_model: "zhida-fast-1p5" + max_agent_iterations: 100 + + llm: + default: "qwen" + qwen: + plugin_class: "inference.plugins.llm.qwen_plugin.QwenLLMPlugin" + api_key: "${DASHSCOPE_API_KEY}" + model: "qwen3.6-plus" + temperature: 0.7 + extra_body: + enable_thinking: false + + tts: + default: "qwen" + qwen: + plugin_class: "inference.plugins.tts.qwen_tts_plugin.QwenTTSPlugin" + api_key: "${DASHSCOPE_API_KEY}" + model: "qwen3-tts-flash-realtime" + voice: "Momo" + sample_rate: 24000 + target_sample_rate: 16000 + + asr: + default: "qwen" + qwen: + plugin_class: "inference.plugins.asr.qwen_asr_plugin.QwenASRPlugin" + api_key: "${DASHSCOPE_API_KEY}" + model: "qwen3-asr-flash-realtime" + language: "auto" + sample_rate: 16000 + vad_threshold: 0.8 + vad_silence_duration_ms: 600 + +session: + max_concurrent: 1 + idle_timeout_s: 300 + max_duration_s: 3600 + +pipeline: + default_mode: "omni" + streaming_mode: "direct" + visual_input: + enabled: true + frame_interval_ms: 1000 + max_width: 1280 + max_height: 720 + jpeg_quality: 0.78 + max_frame_bytes: 524288 + ws_max_message_bytes: 1048576 + max_recent_frames: 2 + frame_ttl_ms: 10000 + turn_enabled: true + turn_port: 8443 + turn_realm: "cyberverse" + turn_username: "cyberverse" + turn_password: "${TURN_PASSWORD}" + ice_public_ip: "122.205.95.186" + +recording: + enabled: true + output_dir: "/app/data/recordings" + crf: 23 + +inference_grpc: + addr: "cyberverse-inference:50051" diff --git a/deploy/gpu/docker-compose.yml b/deploy/gpu/docker-compose.yml new file mode 100644 index 0000000..ba2d43a --- /dev/null +++ b/deploy/gpu/docker-compose.yml @@ -0,0 +1,77 @@ +services: + weights: + image: python:3.11-slim + profiles: ["weights"] + working_dir: /workspace + volumes: + - ../../checkpoints:/workspace/checkpoints + environment: + - HF_ENDPOINT=${HF_ENDPOINT:-} + - HF_TOKEN=${HF_TOKEN:-} + command: + - /bin/sh + - -lc + - | + set -eu + python -m pip install --no-cache-dir -U "huggingface_hub[cli]" + hf download Soul-AILab/SoulX-FlashHead-1_3B --local-dir /workspace/checkpoints/SoulX-FlashHead-1_3B + hf download facebook/wav2vec2-base-960h --local-dir /workspace/checkpoints/wav2vec2-base-960h + + cyberverse-inference: + build: + context: ../.. + dockerfile: deploy/gpu/Dockerfile.inference + network: host + image: cyberverse-inference-rtx5090:cuda12.8 + container_name: cyberverse-inference-gpu1 + restart: unless-stopped + ipc: host + shm_size: "16gb" + gpus: + - driver: nvidia + device_ids: ["1"] + capabilities: ["gpu"] + env_file: + - path: ../../.env + required: false + environment: + - NVIDIA_VISIBLE_DEVICES=1 + - NVIDIA_DRIVER_CAPABILITIES=compute,utility,video + - CYBERVERSE_CONFIG_DIR=/workspace + - LANGGRAPH_CHECKPOINT_DB=/workspace/data/tasks/langgraph_checkpoints.db + - HF_HOME=/workspace/.hf-cache + - HF_HUB_ENABLE_HF_TRANSFER=0 + - PYTHONPATH=/workspace:/workspace/inference/generated + volumes: + - ./cyberverse_config.gpu.yaml:/workspace/deploy/gpu/cyberverse_config.gpu.yaml:ro + - ../../checkpoints:/workspace/checkpoints + - ../../data:/workspace/data + - ../../.hf-cache:/workspace/.hf-cache + expose: + - "50051" + ulimits: + memlock: -1 + stack: 67108864 + + cyberverse-server: + build: + context: ../.. + dockerfile: deploy/gpu/Dockerfile.server + network: host + image: cyberverse-server:gpu-remote + container_name: cyberverse-server-remote + restart: unless-stopped + env_file: + - path: ../../.env + required: false + environment: + - GRPC_INFERENCE_ADDR=cyberverse-inference:50051 + volumes: + - ./cyberverse_config.gpu.yaml:/app/cyberverse_config.yaml:ro + - ../../data:/app/data + ports: + - "8080:8080" + - "8443:8443" + depends_on: + cyberverse-inference: + condition: service_healthy diff --git a/deploy/gpu/download_flashhead_weights.sh b/deploy/gpu/download_flashhead_weights.sh new file mode 100755 index 0000000..3d8aa5e --- /dev/null +++ b/deploy/gpu/download_flashhead_weights.sh @@ -0,0 +1,17 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" +CHECKPOINT_DIR="$PROJECT_ROOT/checkpoints" + +mkdir -p "$CHECKPOINT_DIR" + +python3 -m pip install --user -U "huggingface_hub[cli]" +export PATH="$HOME/.local/bin:$PATH" + +hf download Soul-AILab/SoulX-FlashHead-1_3B \ + --local-dir "$CHECKPOINT_DIR/SoulX-FlashHead-1_3B" + +hf download facebook/wav2vec2-base-960h \ + --local-dir "$CHECKPOINT_DIR/wav2vec2-base-960h" diff --git a/docs/zh-CN/features/2026-05-11-langgraph-agent-development-plan.md b/docs/zh-CN/features/2026-05-11-langgraph-agent-development-plan.md new file mode 100644 index 0000000..497e5b2 --- /dev/null +++ b/docs/zh-CN/features/2026-05-11-langgraph-agent-development-plan.md @@ -0,0 +1,706 @@ + + + +# PersonaAgent + LocalTaskRuntime + LangGraph + Hindsight Development Plan + +> **For agentic workers:** REQUIRED: Use `superpowers:subagent-driven-development` if subagents are available, or `superpowers:executing-plans` to implement this plan. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Decision status:** 2026-05-13 已确认改为当前架构：`PersonaAgentPlugin + LocalTaskRuntime + LangGraph subagent + Hindsight v1`。本节是新的权威计划；下方 Appendix A 保留旧计划与 `/autoplan` 审查记录，仅作为历史背景，不应继续按旧 Agent Worker / `agent_runtime/server.py` 方向实现。 + +**Goal:** 把 Hindsight 作为 PersonaAgent 数字人长期记忆层，并围绕当前本地 `LocalTaskRuntime` 与 LangGraph subagent 路径，补齐可验证的数字人后台任务、记忆、部署和故障恢复能力。 + +**Architecture:** 当前 `features` 分支的真实执行边界是 Python 进程内 PersonaAgent：`PersonaAgentPlugin` 负责实时对话与隐藏工具，`HindsightMemoryClient` 负责长期记忆，`PersonaSupervisor` 与 `LocalTaskRuntime` 负责本地任务调度，`persona/subagents` 负责 LangGraph 长任务执行；Go `server` 保持 session、WebSocket、API 转发、task event 投影和历史落盘职责。`agent_runtime/*` 当前视为兼容 shim，不作为新功能主入口。 + +**Tech Stack:** Python 3.10+, httpx, LangGraph, OpenAI-compatible realtime/text LLM, Go server, Vue frontend, Pixi, Hindsight API, remote GPU/avatar inference endpoints. + +```mermaid +flowchart LR + U["User / Browser"] --> F["Vue frontend"] + F --> G["Go server: session, WebSocket, APIs"] + G --> P["Python inference: PersonaAgentPlugin"] + P --> M["HindsightMemoryClient"] + M --> H["Hindsight bank: cv:user:{user_id}:character:{character_id}"] + P --> S["PersonaSupervisor"] + S --> R["LocalTaskRuntime"] + R --> L["LangGraph subagent"] + L --> T["Task tools: Zhihu, report artifact"] + P --> A["Remote/avatar + realtime model endpoints"] + R --> G +``` + +--- + +## Current Understanding + +- `git pull --ff-only` 在 2026-05-13 因本地与远程各有提交而拒绝快进；已通过 `git fetch` 后 `git rebase origin/features` 接入远程最新 `origin/features`。 +- 当前 `features` 本地分支在 rebase 后仍领先 `origin/features` 2 个本地提交。 +- 恢复本地未提交改动时，`inference/plugins/voice_llm/persona_agent.py` 出现一次冲突；冲突解决原则是同时保留远程 task event 并发监听逻辑与本地 Hindsight source text 记忆采集逻辑。 +- Hindsight v1 代码已经在工作区实现： + - `inference/plugins/voice_llm/persona/memory.py` + - `inference/plugins/voice_llm/persona_agent.py` + - `tests/unit/test_hindsight_memory.py` + - `tests/unit/test_persona_agent_plugin.py` + - `infra/.env.example` + - `infra/cyberverse_config.example.yaml` + - `README.md` + - `README.zh-CN.md` +- Hindsight v1 的设计边界保持最小：不改 Go API，不改 gRPC，不做 QMD，不把真实 key 写进仓库。 +- LangGraph 后续工作应围绕 `inference/plugins/voice_llm/persona/subagents/` 的现有任务路径，而不是旧 `agent_runtime/server.py`。 + +## Development Assumptions + +- v1 优先接入 PersonaAgent 数字人路径；Go standard pipeline 也已接入同一套 Hindsight retain/recall 客户端，用于非数字人文本对话的长期记忆。 +- v1 使用 `HINDSIGHT_BANK_ID_TEMPLATE`、`HINDSIGHT_USER_ID` 和 `HINDSIGHT_USER_TAG` 做跨会话隔离，不引入登录态。 +- v1 每个完成回合保存简洁 `用户: ...\n助手: ...` 文本，不额外调用 LLM 做摘要或分类。 +- Hindsight 服务异常不能阻塞主对话；recall 失败返回空记忆，retain 失败只记录 warning。 +- 真实 Hindsight key 只放本地 `.env`，不得提交；由于 key 已在聊天上下文出现，生产/长期 key 应轮换一次。 +- Go server 继续承担对外 API/WebSocket/session 职责；Python `LocalTaskRuntime` 是当前活动任务运行时，生产级持久化作为后续单独决策。 +- 前端若涉及用户可见文本，必须同时维护中文和英文 i18n。 + +## File Map + +- Keep/extend: `inference/plugins/voice_llm/persona/memory.py` - Hindsight 配置、recall、retain、错误降级和结果兼容。 +- Modify if needed: `inference/plugins/voice_llm/persona_agent.py` - PersonaAgent 隐藏记忆工具、system prompt、final 后异步 retain、task event 并发输出。 +- Modify if needed: `inference/plugins/voice_llm/persona/runtime.py` - 当前本地任务状态、事件、artifact、取消和 shutdown 行为。 +- Modify if needed: `inference/plugins/voice_llm/persona/supervisor.py` - hidden tool 路由和任务启动 ACK 语义。 +- Modify if needed: `inference/plugins/voice_llm/persona/subagents/*.py` - LangGraph 长任务、工具调用、终态 artifact 和失败路径。 +- Modify if needed: `server/internal/api/tasks.go` - 仅在任务 API/auth 明确缺口时小范围修改。 +- Modify if needed: `frontend/src/composables/useChat.ts`, `frontend/src/components/TaskProgressCard.vue`, `frontend/src/i18n/messages.ts` - 任务可见性与 i18n。 +- Config/docs: `infra/.env.example`, `infra/cyberverse_config.example.yaml`, `README.md`, `README.zh-CN.md`. +- Tests: `tests/unit/test_hindsight_memory.py`, `tests/unit/test_persona_agent_plugin.py`, Go `server/internal/*` tests. + +## Chunk 0: Post-Pull Baseline + +### Task 0: Resolve pull/rebase state and verify no conflict markers + +- [x] **Step 1: Fetch and integrate latest `origin/features`** + +Completed on 2026-05-13 with `git rebase origin/features`. Fast-forward pull was not possible because local and remote branches had diverged. + +- [x] **Step 2: Resolve PersonaAgent conflict** + +Resolution keeps both: + +- upstream task event queue / concurrent model-event handling; +- local `remember_source_text` callback used by Hindsight retain. + +- [x] **Step 3: Verify conflict cleanup** + +Run: + +```bash +rg -n "<<<<<<<|=======|>>>>>>>" inference/plugins/voice_llm/persona_agent.py +git diff --check +``` + +Result on 2026-05-13: no conflict markers and no whitespace errors. + +## Chunk 1: Hindsight v1 Closure + +### Task 1: Verify PersonaAgent memory path after rebase + +**Files:** +- `inference/plugins/voice_llm/persona/memory.py` +- `inference/plugins/voice_llm/persona_agent.py` +- `tests/unit/test_hindsight_memory.py` +- `tests/unit/test_persona_agent_plugin.py` + +- [x] **Step 1: Keep Hindsight config optional** + +Required `.env` contract: + +```bash +HINDSIGHT_ENABLED=true +HINDSIGHT_BASE_URL=http://w4ce.jmsu.top:8888 +HINDSIGHT_API_KEY=your_hindsight_api_key +HINDSIGHT_BANK_ID_TEMPLATE=cv:user:{user_id}:character:{character_id} +HINDSIGHT_USER_ID=zly +HINDSIGHT_USER_TAG=zly +HINDSIGHT_TIMEOUT_SECONDS=30 +HINDSIGHT_RECALL_MAX_RESULTS=5 +HINDSIGHT_RECALL_MAX_TOKENS=4096 +HINDSIGHT_RETAIN_MAX_CHARS=6000 +``` + +Expected: missing or placeholder key disables memory without breaking conversation. + +- [x] **Step 2: Keep recall automatic before response** + +PersonaAgent does not expose `recall_memory` as an active model tool. It automatically recalls Hindsight memory after a complete user transcript and injects the formatted result into `response_instructions` before the model answers. + +- [x] **Step 3: Keep retain nonblocking** + +After assistant final response, PersonaAgent schedules retain in the background and drains/cancels tasks during shutdown. + +- [x] **Step 4: Re-run focused unit tests after pull** + +Run: + +```bash +pixi run -e macos-arm python -m pytest tests/unit/test_hindsight_memory.py tests/unit/test_persona_agent_plugin.py -q +``` + +Result on 2026-05-13: PASS with `16 passed, 1 warning`. + +- [ ] **Step 5: Manual Hindsight smoke** + +With local `.env` configured and without committing secrets: + +1. Say: `记住我喜欢用 Pixi 管理环境。` +2. Wait for final response. +3. In a new session ask: `我之前说过我喜欢怎么管理环境吗？` + +Expected: PersonaAgent recalls the Pixi preference naturally. If Hindsight is unavailable, normal conversation continues without memory. + +Status on 2026-05-13: Hindsight live recall smoke passed after adding local `.env`; client enabled and returned 5 recall results. Full digital-human conversation smoke still needs DashScope/Qwen and remote avatar/inference configuration. + +Status on 2026-05-14: CyberVerse remote smoke against `http://w4ce.jmsu.top:8888` verified retain writes documents into bank `cv:user:zly:character:c8af36f1-ed95-4b8e-8cda-0e1e156596fc`. The real PersonaAgent text turn stored `session:eec76193-512e-415b-b9cc-4c0720a0c696:turn:1` with the expected transcript, tags, and metadata. Blocking issue: Hindsight retain/reprocess operations complete successfully but produce `memory_unit_count=0`, `total_nodes=0`, `usage.total_tokens=0`, and recall returns no results. This points to the Hindsight backend extraction path or bank/global LLM configuration, not the CyberVerse retain payload. + +## Chunk 2: PersonaAgent + LocalTaskRuntime Reliability + +### Task 2: Make the active digital-human task path measurable + +**Files:** +- `inference/plugins/voice_llm/persona_agent.py` +- `inference/plugins/voice_llm/persona/runtime.py` +- `inference/plugins/voice_llm/persona/supervisor.py` +- `inference/plugins/voice_llm/persona/subagents/` +- `tests/unit/test_persona_agent_plugin.py` + +- [x] **Step 1: Preserve nonblocking ACK behavior** + +Acceptance: + +- `create_task` returns accepted tool result before task completion; +- assistant final ACK happens before background task start when applicable; +- task event queue can emit progress while model stream is still open. + +Result on 2026-05-13: `LocalTaskRuntime.create_task()` now only creates a queued task. PersonaAgent schedules `start_task()` after assistant ACK final, and tests assert ACK appears before `task.started`. + +- [x] **Step 2: Define terminal task behavior** + +Acceptance: + +- completed task emits `task.completed` and artifact payload if created; +- failed task emits `task.failed` with actionable reason; +- cancelled task emits `task.cancelled`; +- if model never calls terminal artifact tool, bounded iteration/timeout marks failure instead of hanging. + +Result on 2026-05-13: tests cover completed artifact flow, explicit failure, cancellation, and missing terminal report -> `task.failed`. + +- [x] **Step 3: Verify cancellation** + +Acceptance: + +- user cancellation targets latest active session task; +- background work stops or becomes terminal quickly; +- digital human acknowledges cancellation in normal language. + +Result on 2026-05-13: `test_local_task_runtime_cancels_active_task` verifies latest active task cancellation emits `task.cancelled`. + +- [x] **Step 4: Add focused tests only where gaps are found** + +Do not refactor runtime structure unless a test exposes an actual failure. + +Result on 2026-05-13: added focused tests in `tests/unit/test_persona_agent_plugin.py`; no broad runtime refactor. + +## Chunk 3: LangGraph Subagent Trust Path + +### Task 3: Prove one useful task before abstracting providers + +**Files:** +- `inference/plugins/voice_llm/persona/subagents/agent.py` +- `inference/plugins/voice_llm/persona/subagents/default_tools.py` +- related tests under `tests/unit/` + +- [x] **Step 1: Keep current tool path first** + +Use the existing `ZhihuToolExecutor` + `create_html_report` path for the first end-to-end task. Do not prioritize generic `SearchTool` factory until this path is trustworthy. + +Result on 2026-05-13: kept the existing Zhihu/default tool path and added tests around `create_html_report`. + +- [x] **Step 2: Add no-source and weak-source behavior** + +Acceptance: + +- no results produce a visible no-source explanation or small artifact; +- weak/conflicting results are not presented as strong facts; +- tool credential failures produce clear failed task events. + +Result on 2026-05-13: subagent prompt now requires a terminal report even on empty/tool-error results, with caveats instead of pretending weak evidence is fact. Tests verify no-source HTML artifact and clear missing `ZHIHU_ACCESS_SECRET` error. + +- [x] **Step 3: Artifact handoff** + +Acceptance: + +- generated HTML/markdown artifact has stable title, MIME type, and URL; +- frontend can open it from task event payload; +- digital human final message can reference completion without dumping the full artifact. + +Result on 2026-05-13: tests verify HTML artifact type, MIME type, source count, tool trace metadata, and `task.completed` payload with `artifact_id`. + +## Chunk 4: Frontend Task Visibility And i18n + +### Task 4: Make task state understandable in Chinese and English + +**Files:** +- `frontend/src/composables/useChat.ts` +- `frontend/src/components/TaskProgressCard.vue` +- `frontend/src/i18n/messages.ts` + +- [x] **Step 1: Audit hardcoded visible strings** + +Current task UI contains Chinese literals. If this plan touches task UI, move visible strings behind existing i18n. + +Result on 2026-05-13: task status, event titles, current step, timeline labels, artifact labels, and artifact actions now use the `task` i18n namespace in Chinese and English. + +- [x] **Step 2: Verify task card states** + +Acceptance: + +- queued/running/completed/failed/cancelled states are visible; +- artifact link is visible and stable; +- long task titles and messages do not overflow on mobile or desktop. + +Result on 2026-05-13: task card still exposes queued/running/completed/failed/cancelled states and artifact links; existing CSS keeps task titles and artifact titles constrained with ellipsis. + +- [x] **Step 3: Build only if frontend changes** + +Run: + +```bash +npm run --prefix frontend build +``` + +Result on 2026-05-13: PASS. + +## Chunk 5: Deployment, `.env`, And Remote GPU/Avatar Smoke + +### Task 5: Keep Mac development lightweight and remote inference explicit + +**Files:** +- `infra/.env.example` +- `infra/cyberverse_config.example.yaml` +- `README.md` +- `README.zh-CN.md` +- optional deployment docs + +- [x] **Step 1: Document required interfaces** + +The local Mac needs: + +- OpenAI-compatible LLM URL/key/model for PersonaAgent; +- Hindsight URL/key/bank/tag for memory; +- remote avatar/inference API URL(s) if the digital-human model runs on Ubuntu GPU; +- Go server/frontend local URLs; +- optional search/Zhihu credentials for task tools. + +Result on 2026-05-13: root README, Chinese README, `infra/.env.example`, `infra/cyberverse_config.example.yaml`, and `deploy/gpu/*` document PersonaAgent LLM, Hindsight, Zhihu, local API, and remote GPU/avatar endpoints. + +- [x] **Step 2: Keep secrets local** + +Expected: + +- `.env` is ignored by git; +- docs use placeholders only; +- exposed Hindsight development key is not copied into docs or source. + +Result on 2026-05-13: secret scan found no copied real Hindsight key fragments; docs and examples use placeholders only. + +- [x] **Step 3: One-command local startup** + +Use or maintain the Pixi dev command that starts local services in the foreground and terminates child processes on `Ctrl+C`. + +Expected: + +```bash +pixi run -e macos-arm dev +``` + +starts the local Mac-side CyberVerse services that call remote GPU/avatar endpoints through `.env`. + +Result on 2026-05-13: `pixi run -e macos-arm dev` is documented in README/README.zh-CN and maintained in `pixi.toml`; full service smoke still depends on local `.env` and remote GPU/avatar availability. + +## Chunk 6: Final Verification + +Run after implementation or conflict resolution: + +```bash +pixi run -e macos-arm python-agent-test +pixi run -e macos-arm go-test +git diff --check +``` + +Run frontend build only if frontend files were changed: + +```bash +npm run --prefix frontend build +``` + +Expected: all relevant commands pass. If an external service is unavailable, document the exact missing dependency and whether the failure blocks local development. + +Result on 2026-05-13: + +- `pixi run -e macos-arm python-agent-test`: PASS, `23 passed, 1 warning`. +- `pixi run -e macos-arm go-test`: PASS. +- `npm run --prefix frontend build`: PASS. +- `git diff --check`: PASS. +- YAML parse check for `infra/cyberverse_config.example.yaml` and `deploy/gpu/cyberverse_config.gpu.yaml`: PASS. +- Historical local Hindsight recall smoke: PASS after local `.env` was added. +- 2026-05-14 remote Hindsight smoke against `http://w4ce.jmsu.top:8888`: CyberVerse retain writes bank documents, but Hindsight extraction returns `memory_unit_count=0` and recall remains empty. This blocks end-to-end durable recall until the Hindsight backend extraction/LLM worker is fixed. +- Full digital-human conversation smoke with Qwen: PASS for the PersonaAgent response and retain document write; durable recall remains blocked by Hindsight backend extraction. + +## Future Decisions + +- Whether active task state should remain Python in-memory or move to Go as a durable source of truth. +- Whether QMD/local markdown memory adds value after Hindsight v1 is stable. +- Whether generic search provider abstraction is worth adding after the current Zhihu/report path is proven. +- Whether production deployment should split PersonaAgent, task runtime, and avatar inference into separate services. + +--- + +## Appendix A: Superseded Legacy Plan (Do Not Implement) + +The previous 2026-05-11 plan targeted the old Agent Worker and `agent_runtime/server.py` architecture. It has been intentionally replaced by the current PersonaAgent + LocalTaskRuntime + LangGraph subagent + Hindsight v1 plan above. Restore details are available from the two `/autoplan restore point` comments at the top of this file. + +## GSTACK Autoplan Review Report + +### Phase 1: CEO Premise Review + +**Scope detected** + +- UI scope: yes. Task progress and artifact visibility touch `frontend/src/composables/useChat.ts`, `frontend/src/components/TaskProgressCard.vue`, and i18n copy. +- DX scope: yes. The plan depends on API wiring, Pixi startup, `.env` configuration, and repeatable local/remote runtime setup. +- Review voices: Claude subagent completed; Codex CLI review timed out after partial code inspection, but its partial finding matched the main blocker below. + +**Primary blocker** + +This plan is stale against the current branch. It describes an older Agent Worker HTTP boundary and `agent_runtime/server.py`, but the current code has moved the agent runtime in-process: + +- `agent_runtime/graph.py` is now a compatibility re-export shim. +- `agent_runtime/tools.py` is now a compatibility re-export shim. +- `agent_runtime/server.py` does not exist. +- `inference/plugins/voice_llm/persona/runtime.py` owns `LocalTaskRuntime`. +- `inference/plugins/voice_llm/persona/supervisor.py` owns PersonaAgent supervisor routing. +- `inference/plugins/voice_llm/persona/subagents/agent.py` owns the LangGraph subagent path. +- `inference/plugins/voice_llm/persona/subagents/default_tools.py` owns the default task tools, including `create_html_report`. + +The plan should not add work to the old Worker boundary unless the product decision is to reintroduce that boundary intentionally. + +**Product premise challenge** + +The strongest product objective is not "build a generic LangGraph research agent." It is: + +```text +The digital human can acknowledge a user request immediately, continue speaking naturally, +run a bounded long task through LangGraph, surface progress and artifacts, and recover from +missing tools, missing credentials, cancellation, and remote runtime failures. +``` + +The avatar only matters if the user sees embodied task progress: quick acknowledgment, interruptible work, visible state, and a finished artifact that the digital human can reference. + +**Existing leverage map** + +| Area | Current code to preserve | Review note | +| --- | --- | --- | +| Persona turn handling | `PersonaAgentPlugin` | Keep voice/chat response fast; do not block avatar turns on long tasks. | +| Task runtime | `LocalTaskRuntime` | This is the current execution boundary; inject memory and Hindsight here first if needed. | +| Supervisor | `PersonaSupervisor` | Keep structured tool calls as the control path. | +| Subagent | `run_subagent`, `run_task_with_langgraph` | Add acceptance limits around iterations, terminal artifact creation, and failure paths. | +| Tools | `ZhihuToolExecutor`, `create_html_report` | Prove the existing Zhihu/report path before building a generic search provider factory. | +| Go task API | `server/internal/agenttask`, `server/internal/api/tasks.go` | Keep one task surface for frontend state. | +| Frontend | `useChat`, `TaskProgressCard.vue` | Needs i18n and remote API image URL handling if changed. | +| Environment | Pixi, `.env`, docs | Runtime keys and remote GPU endpoints must be documented as required startup inputs. | + +**Dream-state ladder** + +```text +CURRENT + Digital human can run locally wired PersonaAgent code, and remote GPU services can be reached. + Task execution exists, but the plan references older architecture and acceptance criteria are loose. + +THIS PLAN, FIXED + Plan targets current in-process PersonaAgent runtime. + One research task can be started, tracked, cancelled, completed, and displayed with an artifact. + Missing secrets, missing search results, and remote failures have explicit user-visible fallbacks. + +12-MONTH IDEAL + Digital human has durable user memory through Hindsight or a local memory layer. + Long tasks are resumable, auditable, and privacy-aware. + The avatar becomes the interaction surface for progress, interruption, and task handoff. +``` + +### Alternatives + +| Option | Decision | Reason | +| --- | --- | --- | +| Patch the old Agent Worker plan | Reject | It targets missing files and conflicts with current code. | +| Reframe around current PersonaAgent local runtime | Recommended | Lowest-risk path and matches the branch. | +| Reintroduce external Agent Worker now | Defer | Larger scope; only worth doing if deployment isolation becomes a hard requirement. | +| Build a generic search provider factory first | Defer | Current tool path already uses `ZhihuToolExecutor`; prove one real path before generalizing. | +| Add QMD local memory now | Defer | User marked QMD optional; Hindsight boundary should be specified first. | + +### Selective Expansion + +The plan should expand only in the areas that protect the user-visible digital-human workflow: + +- Update architecture sections to current PersonaAgent/LocalTaskRuntime code. +- Add explicit `.env` contract for LLM, Hindsight, Zhihu/search, remote inference, and avatar endpoints. +- Add acceptance criteria for acknowledgment latency, task event emission, artifact creation, cancellation, and no-source fallback. +- Add i18n requirements for every frontend string touched by task progress UI. +- Add a later memory-injection chunk that stores/retrieves through Hindsight at the PersonaAgent runtime boundary. + +Do not expand into a second orchestrator, a provider marketplace, or a local QMD memory system in this pass. + +### Temporal Interrogation + +| Horizon | What must be true | +| --- | --- | +| Hour 1 | Plan no longer references `agent_runtime/server.py`; all chunks point to current runtime files. | +| Hour 6 | One real PersonaAgent task can produce progress events and an artifact against configured services. | +| Day 2 | Frontend build passes with i18n task strings and clear failure/cancellation states. | +| Week 2 | Hindsight integration point is implemented or specified with privacy and deletion behavior. | +| Month 6 | Memory improves continuity without leaking private history into unrelated sessions. | + +### Dual-Voice Review + +| Voice | Result | +| --- | --- | +| Claude subagent | Strong objection: plan solves yesterday's architecture; recommended reframing around in-process PersonaAgent runtime. | +| Codex CLI | Timed out before final response; partial inspection independently found the same Agent Worker mismatch. | +| Consensus | The premise must change before detailed engineering review continues. | + +### Findings + +| Area | Risk | Finding | +| --- | --- | --- | +| Architecture | Critical | Plan targets old Agent Worker files and graph stages that are not present in the current branch. | +| Error rescue | High | Missing explicit fallbacks for missing keys, no search results, no terminal report tool call, and remote service failure. | +| Product | High | Generic research-agent framing underuses the digital human; progress, interruption, and artifact handoff are the defensible workflow. | +| Tests | High | Current plan needs tests for cancellation, failed task events, missing credentials, no terminal artifact, and frontend i18n build behavior. | +| Security | Medium | Hindsight keys, LLM keys, search credentials, and persisted memory need privacy boundaries before implementation. | +| Data flow | Medium | Artifact and character image URLs may need normalization when the Mac frontend talks to a remote API. | +| Observability | Medium | Need correlation IDs or task IDs in logs across PersonaAgent, task runtime, Go API, and frontend events. | +| Performance | Medium | `max_agent_iterations` exists, but plan should also define wall-clock bounds and first-ack latency. | +| UX | Medium | `TaskProgressCard.vue` currently contains Chinese literals; any touched visible text needs Chinese and English entries. | +| Deployment | Medium | Remote GPU service is separate from local Mac Pixi startup; `.env` must distinguish local UI/API from remote inference/avatar endpoints. | + +### Error And Rescue Registry + +| Failure | Required behavior | +| --- | --- | +| Missing LLM URL/key/model | Refuse task start with a visible configuration error; do not hang the avatar turn. | +| Missing Hindsight URL/key | Continue without durable memory if memory is optional; log degraded mode. | +| Missing Zhihu/search credentials | Return a no-tool fallback response and a failed task event with actionable reason. | +| No search results | Complete with a short no-source artifact or user-visible explanation. | +| Model never calls terminal artifact tool | Stop after bounded iterations and mark task failed with a clear reason. | +| User cancels active task | Emit `task.cancelled`, stop background work, and let the digital human acknowledge cancellation. | +| Remote GPU/avatar endpoint unavailable | Keep frontend/API alive and show connection failure without crashing task UI. | + +### Failure Modes To Test + +| Test target | Case | +| --- | --- | +| PersonaAgent runtime | Task start returns quickly while background task continues. | +| LangGraph subagent | Fake model calls tools then `create_html_report`. | +| LangGraph subagent | Fake model exceeds iteration cap without terminal report. | +| Task runtime | Exception emits `task.failed`. | +| Task runtime | Cancellation emits `task.cancelled`. | +| Frontend | Task progress and artifact UI build in Chinese and English. | +| Remote Mac-to-Ubuntu flow | Mac frontend/API can reach remote inference and avatar URLs from `.env`. | + +### Not In Scope For This Plan + +- Reintroducing a separate Agent Worker HTTP service. +- Adding a second task orchestrator beside the current Go task API and `LocalTaskRuntime`. +- Building a generic search provider marketplace. +- Implementing QMD local memory in this pass. +- Replacing PersonaAgent tool-call routing with visible JSON parsing. +- Expanding Standard mode task behavior unless required by the current PersonaAgent path. + +### Decision Audit Trail + +| Decision | Status | Reason | +| --- | --- | --- | +| Use Selective Expansion | Proposed | Existing core runtime exists, but the plan must be corrected. | +| Reframe to current in-process PersonaAgent runtime | Proposed | Matches current code and avoids resurrecting missing Worker files. | +| Prove existing Zhihu/report tool path first | Proposed | It is already wired; generic search abstraction can wait. | +| Require i18n before touching task UI | Proposed | Project guidelines require Chinese and English user-facing text. | +| Defer QMD memory | Proposed | Hindsight integration is the minimal durable-memory path requested by the user. | + +### Premise Gate + +Before Phase 2 engineering review continues, confirm the core premise: + +> Should this plan be rewritten around the current `PersonaAgentPlugin` + `LocalTaskRuntime` + LangGraph subagent architecture, instead of the old Agent Worker / `agent_runtime/server.py` architecture? + +## GSTACK Autoplan Review Report - 2026-05-13 Hindsight Implementation Pass + +### Scope + +- Review mode: Phase 1 CEO / founder premise review. +- UI scope detected: yes. The task card and chat stream are part of the visible product path. +- DX scope detected: yes. Pixi, `.env`, backend services, remote GPU/avatar URLs, and Hindsight configuration affect first-run reliability. +- Current implementation status: Hindsight v1 memory code has been added to the PersonaAgent path and unit/regression tests passed locally before this review. + +### 0A. Premises Under Review + +| Premise | Verdict | Reason | +| --- | --- | --- | +| `agent_runtime/graph.py` owns the research graph | False | Current `agent_runtime/graph.py` is a compatibility re-export shim. The active runtime is under `inference/plugins/voice_llm/persona/`. | +| `agent_runtime/server.py` is the worker/API boundary | False | That file does not exist in this branch. Continuing to target it would recreate an old boundary. | +| Search provider factory is the next highest-leverage work | Weak | The active path uses `ZhihuToolExecutor` and `create_html_report`; generic provider selection is not the current blocker. | +| Go TaskService owns active task state | Unresolved | The visible Go API exists, but `LocalTaskRuntime` currently stores task state/events/artifacts in memory. | +| Hindsight memory should be PersonaAgent-only in v1 | Strong | This matches the implemented digital-human path and avoids changing Go/gRPC protocol for the first durable-memory pass. | + +### 0B. Existing Code Leverage Map + +| Area | Current asset | How to leverage | +| --- | --- | --- | +| Digital-human brain | `inference/plugins/voice_llm/persona_agent.py` | Keep memory/tool orchestration in the active PersonaAgent path. | +| Long-term memory | `inference/plugins/voice_llm/persona/memory.py` | Use Hindsight recall before response and nonblocking retain after final response. | +| Task orchestration | `inference/plugins/voice_llm/persona/runtime.py` and `supervisor.py` | Treat `LocalTaskRuntime` as the current source of active task execution until persistence is deliberately redesigned. | +| LangGraph task work | `inference/plugins/voice_llm/persona/subagents/` | Prove the existing Zhihu-to-HTML-report workflow before abstracting providers. | +| Go/API layer | `server/internal/api/tasks.go` | Keep it as session/WebSocket/task projection layer unless a durability project is opened. | +| Frontend task UI | `frontend/src/components/TaskProgressCard.vue`, `frontend/src/composables/useChat.ts` | Make progress, cancellation, failure, and artifacts bilingual before shipping task UI changes. | + +### 0C. Dream State Delta + +```mermaid +flowchart LR + A["Current: PersonaAgent + LocalTaskRuntime + Hindsight v1 code exists, but plan text still targets old worker files"] --> B["This Plan: rewrite around current PersonaAgent runtime, smoke real Hindsight recall/retain, define task reliability gates"] + B --> C["12-Month Ideal: durable private memory, resumable long tasks, reliable avatar progress/cancel/artifact flow, clear privacy controls"] +``` + +### 0C-bis. Alternatives + +| Option | Decision | Why | +| --- | --- | --- | +| Rewrite around current PersonaAgent runtime | Recommended | Matches the branch and the implemented Hindsight work. | +| Preserve old Agent Worker architecture | Reject | It points at missing/obsolete files and adds avoidable architecture. | +| Move task source of truth into Go now | Defer | Important for production durability, but larger than Hindsight v1. | +| Build generic search provider factory now | Defer | The immediate product risk is task trust/reliability, not provider variety. | + +### 0D. Mode Selection + +Mode: Selective Expansion. + +Approved expansion: +- Correct the plan to current PersonaAgent runtime files. +- Keep Hindsight memory v1 in PersonaAgent only. +- Add real-service Hindsight smoke verification. +- Add explicit failure/cancel/no-source acceptance criteria. +- Treat task UI i18n and internal API auth posture as ship gates. + +Deferred: +- Separate Agent Worker service. +- QMD local memory. +- Generic search provider marketplace. +- Full Go-owned durable task persistence. + +### 0E. Temporal Interrogation + +| Horizon | What must be true | +| --- | --- | +| Hour 1 | The plan no longer targets `agent_runtime/server.py`; all chunks name the current PersonaAgent files. | +| Hour 6 | Hindsight recall/retain can be smoke-tested against configured `.env` without committing secrets. | +| Day 2 | One end-to-end digital-human task can show progress, cancellation/failure, and artifact handoff. | +| Week 2 | Memory privacy behavior, user tag isolation, and deletion/rotation guidance are documented. | +| Month 6 | Users trust the avatar because it remembers relevant context without leaking unrelated private history. | + +### 0F. Premise Gate + +Phase 2 should not continue until the user confirms the plan should be rewritten around `PersonaAgentPlugin + LocalTaskRuntime + LangGraph subagent`, with Hindsight memory v1 as the current shipped path. + +### Dual-Voice Review + +| Voice | Result | +| --- | --- | +| Claude subagent | Critical objection: the plan solves yesterday's architecture and should reframe around the in-process PersonaAgent runtime. | +| Codex CLI | Premise gate failed: `agent_runtime/graph.py` and `tools.py` are shims, `agent_runtime/server.py` is absent, and the active runtime is `persona/runtime.py`. | +| Consensus | Do not implement the old plan as written. Rewrite the plan around the current branch and prove one real digital-human task/memory flow. | + +### Consensus Dimensions + +| Dimension | Result | +| --- | --- | +| Architecture | Current plan is stale; active orchestration is in-process PersonaAgent, not old Worker files. | +| Product | The differentiated product is embodied long-running work with memory, progress, interruption, and artifact handoff. | +| Scope | Hindsight v1 and task reliability are appropriate; search-provider abstraction is premature. | +| Risk | Persistence, auth, privacy, and weak/no-source task outcomes need explicit treatment. | +| UX | Task UI must be bilingual and measurable on first ACK, progress, cancel, failure, and artifact visibility. | +| DX | `.env` examples and Pixi commands are useful, but live smoke commands and remote endpoint assumptions need to be explicit. | + +### Required Review Sections + +1. Architecture: rewrite around `PersonaAgentPlugin`, `HindsightMemoryClient`, `LocalTaskRuntime`, `PersonaSupervisor`, and `run_task_with_langgraph`. Treat `agent_runtime/*` as compatibility surface unless proven otherwise. +2. Error/rescue: Hindsight failure is correctly designed as nonblocking, but task flow still needs acceptance behavior for missing credentials, no sources, model never producing an artifact, cancellation, and remote avatar/GPU outage. +3. Product: a generic research agent is not enough. The product promise is a digital human that remembers, acknowledges quickly, works in the background, and returns a trustworthy artifact. +4. Scope: keep v1 narrow: PersonaAgent memory plus one reliable task path. Defer QMD, new Go protocol, and provider marketplace. +5. Alternatives: do not add a SearchTool factory before proving the current Zhihu/report tool path and source-quality behavior. +6. Security/privacy: never commit Hindsight keys; rotate the exposed development key; document user tag isolation, retention, and production `AGENT_INTERNAL_TOKEN` expectations. +7. Data flow: user turn -> Hindsight recall -> realtime model/tool loop -> final assistant text -> async retain; task events/artifacts flow from Python runtime to Go/WebSocket/frontend. +8. Observability: add task IDs and memory operation warnings to logs; later add metrics for recall latency, retain failure, task terminal status, and artifact creation. +9. Performance: recall adds latency before response. Keep timeout/budget bounded and only force recall on complete user intent, not partial speech. +10. Deployment/DX: `.env` should distinguish local Mac services, remote GPU/avatar services, LLM endpoint, and Hindsight endpoint. Provide a smoke path that does not expose keys. +11. Design/UI: hardcoded task UI text must move behind existing i18n before task UI changes are considered complete. + +### Error And Rescue Registry + +| Failure | Required behavior | +| --- | --- | +| Hindsight URL/key missing | Disable memory and continue normal PersonaAgent conversation. | +| Hindsight recall timeout/error | Log warning, return empty memory, continue response. | +| Hindsight retain timeout/error | Log warning, do not block final response or leak background tasks. | +| Missing LLM/remote avatar config | Surface a clear configuration failure instead of hanging the avatar turn. | +| Search credentials missing | Emit failed task state with actionable reason. | +| No useful sources | Complete with transparent no-source explanation or artifact. | +| Terminal artifact tool never called | Stop after bounded iterations and mark the task failed. | +| User cancels | Emit `task.cancelled`, stop background work, and let the digital human acknowledge. | +| Internal task API unauthenticated in production | Require a nonempty token or document dev-only exposure. | + +### Failure Modes To Test + +| Target | Case | +| --- | --- | +| Hindsight client | recall payload includes query, world/experience types, mid budget, max tokens, tag, and `tags_match=any`. | +| Hindsight client | retain payload includes one `{content, context, tags}` item and `async=true`. | +| PersonaAgent | memory enabled exposes hidden `recall_memory`; disabled mode does not. | +| PersonaAgent | fake realtime model calls recall and receives memory tool result. | +| PersonaAgent | assistant final schedules retain and shutdown drains/cancels background tasks. | +| Task runtime | start returns quickly while background work continues. | +| Task runtime | failure and cancellation produce terminal events. | +| Frontend | task progress/artifact/failure labels are bilingual. | +| Remote deployment | Mac startup can reach remote inference/avatar endpoints from `.env`. | + +### Not In Scope + +- Recreating `agent_runtime/server.py`. +- Changing Go API or gRPC protocol for Hindsight v1. +- Building QMD local memory. +- Adding login/user identity beyond fixed `HINDSIGHT_USER_TAG`. +- Making generic search-provider abstraction the first follow-up. +- Promising durable task recovery across process restart before a source-of-truth decision. + +### What Already Exists + +- PersonaAgent-side Hindsight config/client and hidden recall tool. +- Async retain after assistant final response. +- `.env` and config example entries for Hindsight. +- Unit tests for Hindsight payloads/error tolerance and PersonaAgent memory wiring. +- Regression tests passed locally: Python agent tests, Go tests, and targeted unit tests. + +### Decision Audit Trail + +| Decision | Classification | Principle | Status | +| --- | --- | --- | --- | +| Selective Expansion | Mechanical | P1/P2: preserve current working runtime and expand only where needed | Proposed | +| Reframe to current PersonaAgent runtime | User Challenge | Old plan targets missing files and would waste engineering effort | Requires user confirmation | +| Keep Hindsight v1 PersonaAgent-only | Mechanical | P3/P5: minimum-change integration with no protocol churn | Proposed | +| Defer SearchTool factory | Mechanical | P4/P5: prove current product path before abstraction | Proposed | +| Defer Go-owned durable task state | Taste/Strategy | Production durability matters, but it is a larger separate design | Proposed | +| Require i18n/security smoke gates | Mechanical | P1: visible product and production safety should not regress | Proposed | + +### Phase 1 Completion Summary + +Phase 1 found a blocking premise mismatch. Both review voices agree that the plan should be rewritten before Phase 2 engineering review continues. The correct next decision is whether to make the current `PersonaAgentPlugin + LocalTaskRuntime + LangGraph subagent + Hindsight v1` path the official plan baseline. diff --git a/frontend/src/components/AvatarUpload.vue b/frontend/src/components/AvatarUpload.vue index 87998dc..d15e803 100644 --- a/frontend/src/components/AvatarUpload.vue +++ b/frontend/src/components/AvatarUpload.vue @@ -2,6 +2,7 @@ import { ref, computed, watch, nextTick, onBeforeUnmount } from 'vue' import { useI18n } from 'vue-i18n' import type { ImageInfo } from '../types' +import { resolveApiUrl } from '../services/api' interface DisplayImage { key: string // unique key for v-for @@ -109,9 +110,9 @@ const displayImages = computed(() => { for (const img of orderedImages) { list.push({ key: 'srv-' + img.filename, - src: img.url || (props.characterId + src: resolveApiUrl(img.url || (props.characterId ? `/api/v1/characters/${props.characterId}/images/${img.filename}` - : ''), + : '')), filename: img.filename, }) } diff --git a/frontend/src/components/CharacterCard.vue b/frontend/src/components/CharacterCard.vue index 255782d..e74451c 100644 --- a/frontend/src/components/CharacterCard.vue +++ b/frontend/src/components/CharacterCard.vue @@ -3,6 +3,7 @@ import { computed } from 'vue' import { useRouter } from 'vue-router' import { useI18n } from 'vue-i18n' import type { Character } from '../types' +import { resolveApiUrl } from '../services/api' import { formatVoiceTypeDisplay } from '../utils/voice' const router = useRouter() @@ -10,9 +11,9 @@ const { t, locale } = useI18n() const props = defineProps<{ character: Character }>() const emit = defineEmits<{ delete: [id: string] }>() const coverImage = computed(() => - props.character.active_image + resolveApiUrl(props.character.active_image ? `/api/v1/characters/${props.character.id}/images/${encodeURIComponent(props.character.active_image)}` - : props.character.avatar_image + : props.character.avatar_image) ) // Generate a gradient from character name hash diff --git a/frontend/src/components/TaskProgressCard.vue b/frontend/src/components/TaskProgressCard.vue index e18f7d1..8373658 100644 --- a/frontend/src/components/TaskProgressCard.vue +++ b/frontend/src/components/TaskProgressCard.vue @@ -1,11 +1,13 @@