feat(weave): prototype atomic historical turn ingestion

[ash0ts]

Why

I want HiveMind chat history to appear in Weave Agents without risking partial turns, duplicate retries, or silent data loss. The existing log_turn/OTLP path is useful for ordinary telemetry, but it does not provide the atomic, durable idempotency contract a large historical backfill needs.

This draft establishes the API and storage boundary that the HiveMind importer can depend on. It is deliberately fail-closed and is not ready for hosted deployment yet.

What this draft implements

• Python SDK entry points for prepare_turn, upsert_turn, get_turn_status, and capability discovery.
• Deterministic project-scoped logical keys, trace/root/child IDs, canonical wire digests, and root certificate validation.
• Historical-turn request/response models and remote HTTP bindings for:
  • GET /agents/v1/historical-turns/capabilities
  • PUT /agents/v1/historical-turns/{logical_key}
  • GET /agents/v1/historical-turns/{logical_key}
• Exact handling for committed, replayed, in-progress, conflict, capability-mismatch, size, and schema outcomes.
• A local SQLite reference implementation with a durable compare-and-set commit journal and one immutable complete turn row.
• Crash convergence when storage completed but journal adoption or the HTTP response did not.
• Concurrency coverage for identical and conflicting requests.
• Explicit refusal in the current hosted adapters, ClickHouse path, and in-memory server where durable atomic guarantees are unavailable.

Why it is still a draft

The open-source vertical slice currently uses canonical JSON identity encoding and a local SQLite backend. Before this can serve real imports, the hosted service still needs:

• trusted route-level project ownership and authentication;
• the production strongly consistent commit store and immutable historical-turn table;
• deterministic gzipped protobuf transport with compressed/decompressed limits;
• staged immutable authenticated text references and scanner-policy evidence;
• conversation, agent, span, and message-search query integration;
• tombstone retention, failure injection, generated-client support, and UI expansion for referenced text.

The advertised prototype capability values intentionally cause the HiveMind importer to reject it. There is no OTLP or log_turn fallback.

Related importer

The dependent importer work is in the draft HiveMind PR. That PR implements sealed date-based plans, whole-session micro-cohorts, exact per-turn reconciliation, canary selection, private scheduler state, Keychain-backed scheduled credentials, and content-free reporting.

No real HiveMind chat content was uploaded while developing this change.

Validation

• 55 focused historical-turn SDK/server/HTTP tests passed.
• Ruff formatting and targeted lint passed.
• git diff --check passed.

[codecov]

Codecov Report

❌ Patch coverage is 78.53470% with 167 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
weave/trace_server/historical_turns.py	83.43%	30 Missing and 25 partials ⚠️
.../trace_server/agents/historical_turn_validation.py	60.80%	33 Missing and 16 partials ⚠️
weave/conversation/historical_turn.py	83.57%	13 Missing and 10 partials ⚠️
weave/trace_server/agents/types.py	82.75%	10 Missing and 10 partials ⚠️
...ver_bindings/stainless_remote_http_trace_server.py	0.00%	7 Missing ⚠️
.../trace_server_bindings/remote_http_trace_server.py	88.88%	2 Missing and 2 partials ⚠️
...ve/trace_server/clickhouse_trace_server_batched.py	57.14%	3 Missing ⚠️
...erver/external_to_internal_trace_server_adapter.py	57.14%	3 Missing ⚠️
weave/trace_server/in_memory_trace_server.py	57.14%	3 Missing ⚠️

📢 Thoughts on this report? Let us know!

[ash0ts]

Review-mirror compatibility update

The HiveMind review mirror now pins this companion branch at 0b58f67e1539bfaa2c705e35bed2d9896a319c6a.

That commit adds the typed-message PII redaction fix from #7699 on top of this branch. The review mirror needs both capabilities present here:

• weave.init(..., ensure_project_exists=False), so an importer cannot create a mistyped destination; and
• typed-safe message redaction, so Presidio cannot corrupt a UriPart discriminator before a referenced root is emitted.

The importer has a real-package regression for this exact combination. Its full suite passes (614 passed, 1 opt-in live test deselected). No hosted import was resumed as part of this update.