feat: OpenViking Active Daemon - Automatic Knowledge Extraction from Claude Code Logs

docs/daemon/README.md

@@ -0,0 +1,61 @@
+# OpenViking Active Daemon
+
+自动监听 Claude Code 会话日志，提取知识并写入 OpenViking 知识库。
+
+## 快速开始
+
+### 启用 Daemon
+
+设置环境变量：
+
+```bash
+export OV_DAEMON_ENABLED=true
+export OV_DAEMON_WATCH_DIR=~/.claude/projects    # 监听目录（可选）
+export OV_DAEMON_BATCH_LINES=50                   # 批处理触发行数（可选）
+export OV_DAEMON_BATCH_SECONDS=300                # 批处理触发秒数（可选）
+```
+
+### 启动 OpenViking Server
+
+```bash
+openviking serve
+```
+
+Daemon 会在服务器启动时自动运行（如果 `OV_DAEMON_ENABLED=true`）。
+
+## 工作原理
+
+1. **文件监听** — 监控 `~/.claude/projects/` 下的 `.jsonl` 文件变化
+2. **增量读取** — 文件游标技术，只处理新增内容
+3. **批量处理** — 累积 50 行或 5 分钟后触发 ETL 管道
+4. **知识提取** — 调用 LLM 过滤噪声，提取有价值的知识
+5. **自动存储** — 写入 `viking://skills/`、`viking://memories/`、`viking://resources/`
+
+## 知识分类
+
+| 类型 | 目标路径 | 说明 |
+|------|---------|------|
+| Skills | `viking://skills/claude_code/<title>.md` | 可复用的操作指南 |
+| Memories (有项目) | `viking://memories/projects/<project>/decisions.md` | 项目决策日志 |
+| Memories (无项目) | `viking://memories/global/<title>.md` | 全局记忆 |
+| Resources | `viking://resources/<tech>/<title>.md` | 参考资源 |
+
+## 架构
+
+```
+Claude Code JSONL → File Watcher → Batch Buffer → Filter → Reconstruct
+    → LLM Extract → Deduplicate → Route → viking:// Storage
+```
+
+## 故障排查
+
+### Daemon 未启动
+检查日志中是否有 `Active Daemon is disabled` 消息，确认 `OV_DAEMON_ENABLED=true`。
+
+### 没有提取到知识
+- 确认 Claude Code 正在写入 JSONL 文件（`~/.claude/projects/` 下有 `.jsonl` 文件）
+- 对话内容可能不够有价值（简单问答会被过滤）
+- 查看日志中的 ETL 处理信息
+
+### 知识写入失败
+检查 OpenViking ResourceService 是否正常运行。

docs/daemon/configuration.md

@@ -0,0 +1,50 @@
+# Active Daemon 配置指南
+
+## 环境变量
+
+| 变量 | 说明 | 默认值 |
+|------|------|--------|
+| `OV_DAEMON_ENABLED` | 启用 Daemon | `false` |
+| `OV_DAEMON_WATCH_DIR` | 监听目录 | `~/.claude/projects` |
+| `OV_DAEMON_DB_PATH` | 游标数据库路径 | `~/.qoderworkcn/openviking/daemon_cursors.db` |
+| `OV_DAEMON_BATCH_LINES` | 批处理触发行数 | `50` |
+| `OV_DAEMON_BATCH_SECONDS` | 批处理触发秒数 | `300` |
+
+## YAML 配置 (ov.conf)
+
+```json
+{
+  "daemon": {
+    "enabled": true,
+    "watch_dir": "~/.claude/projects",
+    "batch_trigger_lines": 50,
+    "batch_trigger_seconds": 300
+  }
+}
+```
+
+## Docker 部署
+
+```yaml
+services:
+  openviking:
+    image: openviking:latest
+    environment:
+      - OV_DAEMON_ENABLED=true
+      - OV_DAEMON_WATCH_DIR=/data/claude-projects
+      - OV_DAEMON_DB_PATH=/data/daemon.db
+    volumes:
+      - ./claude-projects:/data/claude-projects
+      - ./daemon-data:/data
+    ports:
+      - "1933:1933"
+```
+
+## 日志
+
+Daemon 使用 OpenViking 标准日志系统。关键日志：
+
+- `Claude Code watcher started on ...` — 监听器启动
+- `Flushing batch with N events` — 批处理触发
+- `Extracted N knowledge items` — 知识提取完成
+- `Knowledge ingested: viking://...` — 知识写入成功

examples/daemon/docker-compose-daemon.yml

@@ -0,0 +1,14 @@
+version: '3.8'
+
+services:
+  openviking:
+    image: openviking:latest
+    environment:
+      - OV_DAEMON_ENABLED=true
+      - OV_DAEMON_WATCH_DIR=/data/claude-projects
+      - OV_DAEMON_DB_PATH=/data/daemon.db
+    volumes:
+      - ./claude-projects:/data/claude-projects
+      - ./daemon-data:/data
+    ports:
+      - "1933:1933"

examples/daemon/start-daemon.bat

@@ -0,0 +1,10 @@
+@echo off
+REM Start OpenViking with Active Daemon enabled on Windows
+
+set OV_DAEMON_ENABLED=true
+set OV_DAEMON_WATCH_DIR=%USERPROFILE%\.claude\projects
+set OV_DAEMON_BATCH_LINES=50
+set OV_DAEMON_BATCH_SECONDS=300
+
+echo Starting OpenViking with Active Daemon...
+openviking serve %*

examples/daemon/start-daemon.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+# Start OpenViking with Active Daemon enabled
+set -e
+
+echo "Starting OpenViking with Active Daemon..."
+
+export OV_DAEMON_ENABLED=true
+export OV_DAEMON_WATCH_DIR="${OV_DAEMON_WATCH_DIR:-$HOME/.claude/projects}"
+export OV_DAEMON_BATCH_LINES="${OV_DAEMON_BATCH_LINES:-50}"
+export OV_DAEMON_BATCH_SECONDS="${OV_DAEMON_BATCH_SECONDS:-300}"
+
+openviking serve "$@"

openviking/daemon/__init__.py

@@ -0,0 +1,7 @@
+"""
+OpenViking Active Daemon package.
+Monitors AI tool logs and automatically extracts knowledge into viking:// storage.
+"""
+from openviking.daemon.service import DaemonService
+
+__all__ = ["DaemonService"]

openviking/daemon/conversation_reconstructor.py

@@ -0,0 +1,53 @@
+"""
+Reconstruct conversation turns from flat event lists.
+Pairs user prompts with assistant responses into structured ConversationTurn objects.
+"""
+from typing import Dict, List
+
+from openviking.daemon.models import ConversationTurn
+from openviking_cli.utils.logger import get_logger
+
+logger = get_logger(__name__)
+
+
+class ConversationReconstructor:
+    """Reconstructs paired conversation turns from chronological events."""
+
+    def reconstruct(self, events: List[Dict]) -> List[ConversationTurn]:
+        """
+        Pair user prompts with assistant responses.
+        Events are sorted by timestamp. An assistant response is paired
+        with the most recent unpaired user prompt.
+        """
+        turns: List[ConversationTurn] = []
+        current_user_prompt = None
+        current_metadata: Dict = {}
+
+        sorted_events = sorted(events, key=lambda e: e.get("timestamp", ""))
+
+        for event in sorted_events:
+            role = event.get("role")
+            content = event.get("content", "")
+
+            if role == "user":
+                current_user_prompt = content
+                current_metadata = {
+                    "session_id": event.get("session_id"),
+                    "project_name": event.get("project_name"),
+                    "timestamp": event.get("timestamp"),
+                }
+            elif role == "assistant" and current_user_prompt:
+                turns.append(
+                    ConversationTurn(
+                        user_prompt=current_user_prompt,
+                        assistant_response=content,
+                        session_id=current_metadata.get("session_id"),
+                        project_name=current_metadata.get("project_name"),
+                        timestamp=current_metadata.get("timestamp"),
+                    )
+                )
+                current_user_prompt = None
+                current_metadata = {}
+
+        logger.info("Reconstructed %d conversation turns from %d events", len(turns), len(events))
+        return turns

openviking/daemon/cursor_manager.py

@@ -0,0 +1,92 @@
+"""
+Cursor Manager for tracking file read positions.
+Persists state in SQLite so Daemon can resume after restart.
+"""
+import sqlite3
+import time
+from pathlib import Path
+from typing import Dict
+
+from openviking.daemon.models import FileCursor
+from openviking_cli.utils.logger import get_logger
+
+logger = get_logger(__name__)
+
+
+class CursorManager:
+    """Manages file cursor state with SQLite persistence."""
+
+    def __init__(self, db_path: str):
+        self.db_path = db_path
+        self._init_db()
+
+    def _init_db(self):
+        """Initialize the SQLite database and schema."""
+        Path(self.db_path).parent.mkdir(parents=True, exist_ok=True)
+
+        conn = sqlite3.connect(self.db_path)
+        try:
+            conn.execute("""
+                CREATE TABLE IF NOT EXISTS file_cursors (
+                    file_path TEXT PRIMARY KEY,
+                    last_position INTEGER NOT NULL DEFAULT 0,
+                    last_read_time REAL NOT NULL DEFAULT 0.0,
+                    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+                )
+            """)
+            conn.commit()
+        finally:
+            conn.close()
+
+    def get_cursor(self, file_path: str) -> FileCursor:
+        """Get the cursor state for a file. Returns zero-position cursor if not found."""
+        conn = sqlite3.connect(self.db_path)
+        try:
+            row = conn.execute(
+                "SELECT last_position, last_read_time FROM file_cursors WHERE file_path = ?",
+                (file_path,),
+            ).fetchone()
+
+            if row:
+                return FileCursor(
+                    file_path=file_path,
+                    last_position=row[0],
+                    last_read_time=row[1],
+                )
+            return FileCursor(file_path=file_path)
+        finally:
+            conn.close()
+
+    def update_cursor(self, file_path: str, position: int):
+        """Update the cursor position for a file."""
+        conn = sqlite3.connect(self.db_path)
+        try:
+            conn.execute(
+                """
+                INSERT OR REPLACE INTO file_cursors (file_path, last_position, last_read_time)
+                VALUES (?, ?, ?)
+                """,
+                (file_path, position, time.time()),
+            )
+            conn.commit()
+        finally:
+            conn.close()
+
+    def get_all_cursors(self) -> Dict[str, FileCursor]:
+        """Get all tracked cursor states."""
+        conn = sqlite3.connect(self.db_path)
+        try:
+            rows = conn.execute(
+                "SELECT file_path, last_position, last_read_time FROM file_cursors"
+            ).fetchall()
+
+            return {
+                row[0]: FileCursor(
+                    file_path=row[0],
+                    last_position=row[1],
+                    last_read_time=row[2],
+                )
+                for row in rows
+            }
+        finally:
+            conn.close()

openviking/daemon/deduplicator.py

@@ -0,0 +1,35 @@
+"""
+Knowledge deduplication based on content hashing.
+Prevents duplicate knowledge from being written to viking:// storage.
+"""
+import hashlib
+from typing import Set
+
+from openviking.daemon.models import ExtractedKnowledge
+from openviking_cli.utils.logger import get_logger
+
+logger = get_logger(__name__)
+
+
+class KnowledgeDeduplicator:
+    """Deduplicates knowledge items based on MD5 content hash."""
+
+    def __init__(self):
+        self.seen_hashes: Set[str] = set()
+
+    def is_duplicate(self, knowledge: ExtractedKnowledge) -> bool:
+        """Check if this knowledge is a duplicate of something already seen."""
+        content_hash = hashlib.md5(
+            knowledge.content.encode("utf-8")
+        ).hexdigest()
+
+        if content_hash in self.seen_hashes:
+            logger.debug("Duplicate knowledge skipped: %s", knowledge.title)
+            return True
+
+        self.seen_hashes.add(content_hash)
+        return False
+
+    def clear(self):
+        """Clear the dedup cache."""
+        self.seen_hashes.clear()