Import HiveMind chats into Weave Agents - #8
Conversation
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
|
Warning Review the following alerts detected in dependencies. According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.
|
|
I’ve updated this draft with a separate, explicitly noncanonical review path so we can evaluate recent HiveMind chats without pretending the hosted historical-turn API exists yet. What I tried first was a native Agents backfill: map every ATIF turn to a root plus its LLM/tool children and send it through the existing historical logging path. That failed at the reliability boundary for large conversations. The hosted OTLP path does not give me an atomic, durable, idempotent commit for the whole span graph, so a late failure can leave a partial turn and an ambiguous retry. The companion Weave branch is still useful for validating the future API shape, but running its client code locally cannot create those missing server guarantees. This revision adds a temporary review mirror instead:
I also tightened privacy around low-entropy IDs and selectors. Source/principal IDs must be canonical lowercase UUIDv4/v7 before they can be hashed or persisted; uncontracted agent/trajectory IDs become a constant marker; filter state keeps only kinds/counts; preserved JSON metadata is structurally redacted; and name-like values hidden in technical/model/code-identifier shapes are covered by regression tests. Current validation is 605 passing tests with the single opt-in live smoke deselected, plus Ruff, offline source/wheel builds, CLI smoke checks, and an independent security review with no remaining P1/P2 finding. I have not created the project or uploaded any chat. The next safe steps are still sequential: verify the caller-created project is private and writable, run the synthetic large-turn smoke, run one deterministic real canary and confirm a zero-emission rerun, capture an exact trailing-21-day cutoff, then apply whole-session cohorts of 1, 5, 20, and the remainder. Any uncertainty, conflict, privacy failure, missing reference, or count mismatch stops the rollout. This mirror is only for review. |
Update: safe three-week review mirror and real canary findingsI kept working from the live failure evidence instead of trying another large upload. What I triedI added the temporary, explicitly noncanonical review path for I then ran discovery over the exact trailing 21-day window. The bounded canary scan examined two plausible settled top-level sessions. Neither satisfied the strict whole-session canary limits, so planning stopped and uploaded nothing. The limits were not relaxed automatically. What went wrongThe real pinned Weave SDK exposed a separate bug before any W&B write: with PII redaction enabled, the SDK recursively redacted typed message protocol strings. Presidio could classify the literal The earlier code also retained more source identity material than necessary and did not prove enough about a sealed plan before a cohort write. In particular, account-label-derived state, arbitrary UUIDv5 values, mutable plan rows, and a partial candidate universe could create privacy or substitution risks. What changed
The standalone Weave fix is wandb/weave#7699. The combined reviewed companion remains wandb/weave#7693. Current status
This stays draft. I am not claiming that the three-week logs are visible yet. |
Final status-command correctionThe last local check found that The focused regression passes, Ruff/format/diff checks remain clean, and a real invocation now reports zero plans/turns with the review state still absent. |
1 participant
Why I started this
I wanted a practical way to review the last few weeks of my HiveMind chats in Weave Agents without manually opening sessions one by one. The important requirements were that the import be lossless after redaction, resumable, safe to rerun, and easy to exercise on one real chat before touching a large backlog.
What I tried first, and why it did not work
My first implementation mapped each HiveMind turn to
weave.log_turnand sent inferred LLM/tool spans through the existing OTLP path. That works for ordinary telemetry, but it was not a sound historical-backfill boundary for these chats. Large turns could time out or fail partway through export, and the existing path does not give the importer a transactional, durable idempotency guarantee for one historical root plus all of its children. After an ambiguous response, I could not prove whether retrying would fill a gap or create a duplicate.I stopped using that path for the backfill rather than repairing traces manually or treating partial visibility as success. The future canonical import still depends on a real server-side atomic historical-turn API.
What this draft does now
This draft adds a separate, explicitly noncanonical review mirror in the private
wandb/hivemind-chats-reviewproject. It does not write to the project reserved for the future native-span import.The importer processes one whole session revision at a time:
This is intentionally an operational review mirror, not a claim of server-side atomicity. Object retries are safe because they are digest-addressed. Root submission is the ambiguity boundary: once an attempt crosses it, the importer never automatically retries. Exact reconciliation must resolve it before later writes.
Full messages, system instructions, reasoning, tools, usage, timestamps, warnings, and session linkage remain in the attached manifest. The Agents root contains bounded redacted previews and searchable linkage attributes. This path deliberately does not fake native LLM/tool child spans.
What the live rollout taught me
There were three different failure modes, and they needed different handling.
First, HiveMind's ATIF export can change while the session summary and
last_activity_atremain unchanged. One 12-turn, roughly 7.66 MB session certified successfully during preview, then produced different turn certificates during apply. The importer stopped before the first root. Two hosted absence sweeps and repeated source preparation proved there had been no remote write, so the attempt was retired as immutable zero-write evidence instead of being silently reopened.Second, local preparation itself can be pathological. The next candidate consumed a CPU core for more than 61 minutes without sealing a plan, uploading content, or changing the journal. I interrupted it and verified the project was still clean. That showed byte and turn limits were not enough; preparation also needed a real wall-clock boundary.
Third, an arbitrary fixed number of whole-payload redaction passes was not a sound convergence proof. Presidio can reveal another match after an earlier replacement, and typed markers can be reinterpreted on a later pass. Redaction now converges independently per text leaf with marker shielding, digest-only cycle detection, strict progress, and a cumulative UTF-8 work budget, followed by an exact second structural verification pass. It fails closed if it cannot prove a fixed point.
Bounded preparation and fair retries
Every real review preview now prepares each session in a fresh read-only subprocess. The parent sends source metadata through a private bounded pipe. The worker receives no W&B/model credential, SQLite path, state mutation interface, or upload interface. Only validated source coordinates, digests, timestamps, sizes, an authoritative subagent boolean, and content-free canary counts return; temporary worker evidence is removed before plan hashing or SQLite.
The parent owns the monotonic deadline. On timeout it terminates and reaps the complete process group before recording the content-free failure code. A timeout, interrupt, worker crash, malformed or oversized response, inherited result pipe, orphaned descendant, or unknown exception can never submit a root. Only a still-running leader at the observed deadline can create
preparation_timeoutevidence.The SQLite journal is now schema v13. It retains exact, immutable pre-seal failure evidence and schedules untouched revisions before retries. Retry ordering uses parsed UTC instants rather than lexical timestamp ordering. Revisions already owned by a live or completed plan are not counted as pending retries; zero-write retired/revalidated attempts remain eligible for a deterministic successor. A bad chat therefore cannot monopolize every invocation or turn
--next-sessions 1into an unbounded batch.Canary and exact-session previews use the same subprocess boundary. Canary child/root classification comes from the authoritative detail export, not a potentially stale list summary. Parent-side invalid Unicode or oversized source metadata is handled as a candidate-local, content-free serialization failure; malformed worker evidence remains an unrecorded run-level failure.
Security cleanup
The importer never opens, sources, copies, parses, or prints a
.envfile. HiveMind authentication stays inside the installed HiveMind CLI. W&B authentication must already be in the calling process or in the supported project-scoped Keychain path.Redaction happens before hashing, planning, staging, serialization, or upload. Reports use aliases and counts instead of titles, repository paths, source IDs, prompts, tool data, hashes, or trace IDs. SQLite stores only the private evidence needed for resumption and reconciliation. The review path is locked to the expected private hosted project and pinned Weave revision; it has no custom endpoint, proxy, legacy upload fallback, manual trace repair, or automatic retry of an uncertain root.
The temporary local credential launcher used for this rollout was deleted after the final live status check. No diagnostic helper remains in
/private/tmp.Current live checkpoint
The private review project now has 21 completed HiveMind conversations and 63 visible review turns. There are zero planned, publishing, submitting, uncertain, or conflicting turns.
After calibrating the hard preparation deadline, two additional whole root sessions certified and imported in isolated one-session plans. Each contained 10 turns. Both completed with all 20 expected turns visible, and an immediate replay of each completed plan emitted zero additional turns.
The exact trailing window is
[2026-07-16T16:20:00Z, 2026-08-06T16:20:00Z). The latest scan found 44 eligible root-session revisions and reported 30 still unplanned after the final selected session. The three-week primary-session backfill is therefore still in progress, and subagent sessions have not been claimed complete. This draft does not overstate that status.Validation
733 passed, 1 deselectedfor the importer; the destructive/live test remains opt-in.git diff --checkpassed.What is still missing
This review mirror is useful now, but it is not the final HiveMind-to-Weave representation. A canonical import still needs the upstream atomic historical-turn API and SDK support so one request can durably commit one historical turn and all native LLM/tool spans with compare-and-set idempotency, stable returned IDs, exact replay semantics, large authenticated text references, and atomic visibility.
Until that exists, this draft keeps the private review mirror separate, preserves complete redacted chats through verified objects, and avoids claiming that root-only review turns are equivalent to native historical Agents traces.