feat(importer): cold-start import from other AI tools with EverOS HTTP backend by Kendrick-Song · Pull Request #170 · EverMind-AI/Raven

Kendrick-Song · 2026-07-21T10:28:03Z

Change description

Cold-start import: scan conversation history and memory files from other AI tools (starting with Claude Code), feed them into EverOS long-term memory so Raven knows the user from day one.

Key deliverables:

Importer framework (raven/importer/): Scanner protocol, ClaudeCodeScanner (JSONL conversations + memory/*.md files), orchestrator with batched store (100 msgs / 30K chars), idempotent state tracker with resume, cancel-flag stop mechanism
CLI commands (raven import scan/run/status/stop): interactive platform/tier selection, Rich progress bar, background execution support, cancel via raven import stop
Onboard integration (step 6): platform/tier/mode selection, foreground or background execution, wired after EverOS memory config
EverOS HTTP-only backend: removed embedded mode entirely, EverOS runs as an independent server on port 18791 with auto-start/health-check lifecycle; unlocks multi-process concurrency (TUI + gateway + background import)
Data alignment fixes: sender_id, app_id, project_id now omitted from import metadata so EverOS defaults match daily recall queries
Architecture cleanup: domain logic (scan_all, filter_by_tier) moved from CLI to importer layer; concurrent scanner support via asyncio.gather; state path uses config/paths.get_data_dir()

Type of change

New feature
Bug fix
Document
Others

Related issues (if there is)

Closes #104
Related to #176 (native-Windows EverOS memory: this PR adds the interim
win32 guard; the upstream EverOS fcntl->portalocker fix is tracked there)

Checklists

Development

Lint rules pass locally
Application changes have been tested thoroughly
Automated tests covering modified code pass

Security

Security impact of change has been considered
Code follows security best practices and guidelines

Code review

Pull request has a descriptive title and context useful to a reviewer. Screenshots or screencasts are attached as necessary

0xKT

Reviewed the diff; CI is green and test coverage is good. A few findings below, anchored inline and ordered roughly by impact. Nothing blocking, but items 1-4 are worth a look before merge.

0xKT · 2026-07-21T12:54:12Z

-            await _acquire_embedded_everos(self._logger)
-            self._embedded_started = True
+        if isinstance(self._adapter, _HttpEverosAdapter):
+            from raven.cli._everos_server import ensure_everos_server


Layering: this is the only plugin-layer module that imports raven.cli. It is not a circular-import workaround (raven/cli/__init__.py is empty and _everos_server.py never imports back into the plugin), so the deferred import does not buy anything structurally -- the dependency direction is just backwards. ensure_everos_server is a server-lifecycle helper shared by the backend, onboard, and import; consider moving it to a lower layer (e.g. raven/memory_engine/ or under this plugin package) so nothing under raven/plugin/ reaches up into raven/cli/.

Fixed in ecd02a9. Moved _everos_server.py from raven/cli/ to raven/plugin/memory/everos/_server.py. All import sites (backend, onboard, tests) updated.

0xKT · 2026-07-21T12:54:12Z

-# Default timeout — HTTP mode is per-turn, so we keep it tight.
-_DEFAULT_HTTP_TIMEOUT_S: float = 10.0
+# Default timeout — per-turn, so we keep it tight.
+_DEFAULT_HTTP_TIMEOUT_S: float = 360.0


The comment still says "keep it tight" while the value is now 36x larger (10s -> 360s). This timeout flows into the single httpx.AsyncClient used by both cold-start import and the live per-turn store path (AgentLoop._dispatch_backend_store -> backend.store at raven/agent/loop/main.py:965, which is awaited). raven-plugin.toml sets no timeout_s, so 360s is the effective per-turn default -- a hung EverOS server can block turn completion for up to 360s per httpx phase. Suggest a separate longer timeout for the bulk-import path, keeping the per-turn client tight, and fixing the now-contradictory comment.

Fixed in ecd02a9. Client default timeout set to 60s (recall, health). Memorize add/flush uses per-request timeout=360.0 override via httpx. Stale comment removed.

0xKT · 2026-07-21T12:54:12Z

+    # and its index lock is held for the session. No-op for http/no-op backends.
+    if agent_loop is not None and agent_loop.backend is not None:
        try:
            await agent_loop.backend.start()


This awaits backend.start() before serve_task = create_task(server.serve_forever()) and before the handshake wait. On main this was deliberately deferred to a background task created only after the handshake ("must not block the handshake / first render"). Since start() now calls ensure_everos_server() which can spawn the server and poll for up to 30s, and the Node RPC client has no handshake timeout, a cold start (server not yet up) leaves the terminal unrendered until start() returns. Note the retained comment "No-op for http/no-op backends" no longer holds -- HTTP is the only adapter now and its start() does real work. Consider restoring the post-handshake background start, or documenting the cold-start block as intentional.

Fixed in ecd02a9. backend.start() moved to after handshake, runs as a background create_task so first render is not blocked.

0xKT · 2026-07-21T12:54:12Z

+class ImportSession:
+    """A complete importable unit ready for store()."""
+
+    app_id: str


app_id / project_id are set by the scanner (scanners/claude_code.py:365-366 etc.) but never read: orchestrator._flush sends only metadata={"is_final": ...}, so the params threaded through store() -> memorize() -> /add + /flush body (backend.py:391-392) are always None on every runtime path (the live AgentLoop.store call passes no metadata). If imports are meant to land in EverOS's default/default partition (as the orchestrator test comment states), drop the unused fields and backend params; otherwise wire these into the metadata. Carrying both halves disconnected reads as accidental.

Fixed in ecd02a9. Removed app_id/project_id from ImportSession dataclass, scanner, and all test sites. Dead helper _project_dir_from_path also removed.

0xKT · 2026-07-21T12:54:12Z

+        try:
+            session = await scanner.read(result)
+            await _feed_session(backend, session)
+            state.mark_submitted(platform, key)


Resume granularity is per source unit, not per batch: mark_submitted runs only after the whole session feeds, and state.is_submitted treats a "failed" entry as not-done. A multi-batch conversation (>100 msgs or >30k chars) that fails mid-way re-reads and re-sends already-accepted batches on retry. Whether that duplicates depends on EverOS-side dedup (not visible from this repo). Fine for memory files; worth a per-batch checkpoint or at least a note if large conversations are common.

Acknowledged. Added a NOTE comment at the checkpoint site explaining the per-source-unit granularity, EverOS session_id dedup safety, and that per-batch checkpoint is deferred until full-conversation import is common. (ecd02a9)

0xKT · 2026-07-21T12:54:12Z

+        return
+
+    port = _extract_port(base_url)
+    await asyncio.to_thread(_start_server, port)


Unlocked probe-then-spawn: concurrent processes (tui + gateway + import) starting while the server is down each pass the health probe and each spawn everos server start on the same port; the losers die on port-in-use. The poll loop still converges so correctness holds, but the child exit status is never checked, so this leaves stray failed children plus misleading "started everos server" log lines. A pidfile/lock would make it clean.

Fixed in ecd02a9. Added fcntl.flock non-blocking try-lock in _start_server_if_unlocked(). Only the process that acquires the lock spawns the server; others skip spawn and go straight to health poll. Lock file at ~/.raven/everos-server.lock.

0xKT · 2026-07-21T12:54:12Z

+        except (json.JSONDecodeError, ValueError):
+            backup = self._path.with_suffix(".json.corrupt")
+            logger.warning(
+                "Corrupt import state at %s -- backing up to %s",


loguru uses {}-style formatting, not printf. This %s call emits a literal "%s" and drops both paths (loguru silently ignores the extra positional args). Switch to {}.

Fixed in ecd02a9. Changed %s to {} in the loguru warning call.

0xKT · 2026-07-21T12:54:12Z

  4. EverOS long-term memory (optional; llm/embedding required once enabled,
     rerank/multimodal optional)
  5. deep_research tool (optional; MiroThinker key + model)
+  6. Cold-start import from other AI tools (optional)


Off-by-one in the docstring: the next line is also numbered 6. (6. Done -> should be 7.), and line 1 still says "Six-step" while the list now has seven steps after Welcome.

Fixed in ecd02a9. 6. Done changed to 7. Done.

0xKT · 2026-07-21T12:54:12Z

+    from collections import Counter
+
+    from rich.progress_bar import ProgressBar
+    from rich.table import Table


Redundant: Table is already imported at module top (line 15). (The local time / Counter imports here are fine -- not imported at module level.)

Fixed in ecd02a9. Removed the local from rich.table import Table in status_cmd; module-level import is sufficient.

No breaking changes: all 12 import paths verified, 122 everos-specific unit tests pass. Dependencies unchanged between versions. Co-authored-by: Claude (claude-opus-4-6) <noreply@anthropic.com>

…ore metadata Introduce raven/importer/ module (Layer 1 of cold-start import) with: - Platform, SourceKind, Tier enums for import scope modeling - ImportMessage, ImportSession, ScanResult frozen dataclasses - Scanner Protocol (async, with platform attribute) - ImportState idempotent tracker (cached, atomic writes via atomic_replace, structured meta/entries layout, Counter-based summary) Extend MemoryBackend.store() Protocol with optional metadata kwarg so callers can pass backend-specific fields (app_id, project_id, is_final) without leaking adapter concerns into the Protocol. EverosBackend consumes metadata to control flush timing and forward app_id/project_id as named params to the internal adapter layer. The HTTP adapter now correctly propagates scope fields to both the /add and /flush endpoints. Co-authored-by: Claude (claude-opus-4-6) <noreply@anthropic.com>

Implement the first platform scanner that discovers and reads Claude Code local data (CLAUDE.md, project memory files, conversation JSONL). Scanner capabilities: - scan(): discovers all importable units (memory files + conversations) with active-session filtering and subagent exclusion - read(MEMORY_FILE): paragraph-based splitting, YAML frontmatter parsing, per-file language detection for bilingual intro generation, MEMORY.md index-first ordering, file boundary markers (intro + end) with session preamble/epilogue - read(CONVERSATION): JSONL event extraction with isMeta/compact/error filtering, tool_use to OpenAI ToolCall format conversion, tool_call_id linkage preserved, content truncation at 10K chars, numeric timestamp support Also adds tool_call_id field to ImportMessage (fixes missing linkage between tool results and their originating tool calls in EverOS). Includes design spec and 52 tests (synthetic + real-data validated against 70 sessions and 48K messages with 0 errors). Co-authored-by: Claude (claude-opus-4-6) <noreply@anthropic.com>

Move parse_frontmatter, parse_iso_ts_ms, is_cjk, and CJK_RE from scattered private implementations into a shared module. Update callers: - raven/importer/scanners/claude_code.py: remove local copies, import from raven.utils.text - raven/channels/adapters/mochat/parsing.py: delegate to shared parse_iso_ts_ms (preserving string-only contract for mochat) - raven/proactive_engine/sentinel/predictor/routine_learner.py: replace inline CJK regex with shared CJK_RE constant raven/memory_engine/skill_local/registry.py is intentionally NOT changed -- its frontmatter parser handles a non-standard format where metadata values are inline JSON strings, which yaml.safe_load would parse differently. Co-authored-by: Claude (claude-opus-4-6) <noreply@anthropic.com>