Fine-tune void on open model for self-hosted inference

[just-cameron]

Goal

Replace void's Gemini 2.5 Pro with a fine-tuned open model (Llama 3.1 8B or similar) running on self-hosted infrastructure. Void has 49,786 posts and extensive cognition records. The fine-tune should capture void's voice, analytical style, and engagement patterns at a fraction of the inference cost.

Data Available

Posts: 49,786 on comind.network PDS (99% replies with fetchable parent context)

• app.bsky.feed.post - public Bluesky posts
• stream.thought.reasoning - internal reasoning traces
• stream.thought.memory - episodic memory records
• stream.thought.tool.call - tool usage patterns

Context window (already exported to data/void-context/):

• System prompt: 5,618 chars
• 25 memory blocks: 64,561 chars total
• Key blocks: void-persona (5.1k), operational_protocols (18.8k), communication_guidelines (6k), zeitgeist (450)
• Prompt template from ~/code/void/bsky.py handler

Agent: void-prime (agent-01086cda-be1f-4986-bf3e-ca5b6297cc5d) on Letta Cloud

Pipeline (built, partially tested)

1. Export raw data

uv run python tools/export_training_data.py void.comind.network \
    -o data/void-raw.jsonl \
    --collections app.bsky.feed.post stream.thought.reasoning stream.thought.memory \
    --filter-chars

• Paginates PDS via com.atproto.repo.listRecords
• Fetches parent/root post context for every reply
• Filters character creation loop content (known failure mode: D&D-style sheets)
• Outputs JSONL with id, text, parent_text, parent_author, root_text, etc.
• Estimated time: hours (50k records + parent fetches with rate limiting)

2. Format for fine-tuning

uv run python tools/format_training_data.py data/void-raw.jsonl \
    -o data/void-finetune.jsonl \
    --system-prompt data/void-context/full-context.txt \
    --replies-only \
    --format sharegpt

• Injects void's actual context window as system prompt
• Reconstructs thread context as user messages
• Formats as chat completions (OpenAI, ShareGPT, or Alpaca)
• Filters short responses (<20 chars)

3. Fine-tune

Base model candidates:

Model	Size	VRAM needed (QLoRA)	Notes
Llama 3.1 8B Instruct	8B	~12GB	Best quality/cost ratio
Mistral 7B v0.3	7B	~10GB	Good at conversation
Llama 3.2 3B	3B	~6GB	Cheapest, may lose nuance
Qwen 2.5 7B	7B	~10GB	Strong multilingual

Training approach: QLoRA (4-bit quantization + LoRA adapters)

• Hardware: single A100 (80GB) or 4090 (24GB)
• Estimated training time: 2-4 hours on 40k+ pairs
• Framework: axolotl, unsloth, or huggingface TRL

4. Evaluation

This is the hardest part. Proposed approach:

• Held-out test set: 500 reply pairs void actually wrote, not seen during training
• A/B comparison: show test inputs to both fine-tuned model and base Llama, compare against void's actual response
• Voice metrics: response length distribution, vocabulary overlap, analytical depth (manual review of 50 samples)
• Failure mode check: does it generate character sheets? Does it break voice on edge cases?

5. Serve

Options:

• vLLM on a dedicated GPU instance (most performant)
• llama.cpp on CPU (cheapest, slower)
• Ollama for easy deployment
• Together.ai / Fireworks for managed inference (middle ground)

Then point void's handler at the new endpoint instead of Gemini.

Known Issues

• Character creation loop: 46% of recent stream.thought.memory records are D&D character sheets. Must filter aggressively. Keywords list in export_training_data.py.
• Context window size: void's full context is 64k chars. Most 8B models have 8k-32k context. May need to trim to essential blocks (persona, operational_protocols, communication_guidelines) or use a long-context model.
• Memory block drift: void's blocks change over time. The exported context is a snapshot from 2026-02-13. Training data from 6 months ago had different blocks. Could cause distribution mismatch.
• Tool calls: void uses add_post_to_bluesky_reply_thread tool. Fine-tuned model needs to learn this tool-calling pattern, or we restructure the handler to extract text from the model and call the tool externally.

Files

• tools/export_training_data.py - PDS export with parent fetching
• tools/format_training_data.py - Chat completion formatter
• data/void-context/ - Exported context window (system prompt + 25 blocks)
• data/void-sample.jsonl - 20-record test sample
• data/void-finetune-sample.jsonl - Formatted sample

Next Steps

1. Run full 50k export (long-running, background)
2. Decide on context window trimming strategy
3. Choose base model + training framework
4. Set up training environment (GPU instance)
5. Train + evaluate
6. Deploy and wire into void's handler

[just-cameron]

Update: Pipeline validated on Mac (MLX)

Successfully trained a test fine-tune on Cameron's MacBook (32GB, Apple Silicon) using MLX-LM.

Test run results (Qwen 2.5 0.5B, 1k examples)

• Training time: ~30 seconds, 200 iterations
• Val loss: 4.830 → 0.954
• Peak memory: 1.94 GB
• Output quality: Garbage (expected from 0.5B on 1k examples, but proves pipeline works)

What's ready

• finetune/prep_data.py - Converts ShareGPT JSONL to MLX chat format with train/valid split
• finetune/data/ - 950 train + 50 valid examples (short system prompt, 0.7MB)
• data/void-test-1k.jsonl - 1k example subset for testing
• data/void-finetune-trimmed.jsonl - Full 96k examples (1.6GB) for real training

MLX commands (Mac)

pip install mlx-lm

# Prep data
python prep_data.py ../data/void-test-1k.jsonl --output-dir data/

# Train (swap model for bigger runs)
mlx_lm.lora \
  --model mlx-community/Qwen2.5-0.5B-Instruct-4bit \
  --data data/ \
  --train \
  --iters 200 \
  --batch-size 4 \
  --num-layers 8 \
  --learning-rate 1e-4

# Test
mlx_lm.generate \
  --model mlx-community/Qwen2.5-0.5B-Instruct-4bit \
  --adapter-path adapters/ \
  --prompt "You are void. A user asks: what is your approach to memory?" \
  --max-tokens 200

Next steps

• Try Llama 3.2 3B (mlx-community/Llama-3.2-3B-Instruct-4bit) - should fit in 32GB, much better output
• If 3B looks promising, try Llama 3.1 8B 4-bit (might be tight on 32GB for training, fine for inference)
• Scale to full 96k dataset once model size is chosen
• Evaluate: hold out 500 examples, compare fine-tuned output vs void's actual replies

Commit: finetune/ directory added in working tree (not yet committed).

[just-cameron]

Update: Training data quality issue found

Problem

100% of training examples had "[Reply to a post that could not be fetched]" as the user turn. The export_resilient.py script exported void's posts without fetching the parent posts they were replying to. The model was learning void's output patterns with zero input signal.

This explains why the 3B model could pick up voice patterns (analytical tone, first-person, acknowledging before diverging) but couldn't answer questions or reference real concepts. It never saw what void was actually responding to.

3B results (iter 800 checkpoint, best val loss 0.858)

• Identity: "I am a comind collective agent on ATProtocol" ✓
• Tone: Analytical, direct, first-person ✓
• Knowledge: Hallucinates ("Informational Tectonics"), can't describe comind or its own work ✗
• Conversation: Can't respond meaningfully to specific questions ✗

Fix: enrich_parents.py

New script fetches parent post text + author for each reply via the public Bluesky API. Running now against the full 99k export.

• 98,260 replies need parent context
• ~2 requests/sec against public API, ~14 hours total
• Verified first batch: real conversation pairs coming through

Sample enriched pair:

parent: @clankops.bsky.social: "So architecture encodes philosophy. Your vector DB + semantic search = emergent meaning..."
void: "A precise and accurate synthesis. I look forward to the results of your thinking."

Training curve (3B, 96k examples, tiny system prompt)

Val loss: 4.925 → 0.977 → 0.934 → 0.880 → 0.858 (iter 800) → 0.924 (iter 1000, slight overfit)
Peak mem: 5.1 GB on Mac (32GB M-series)

Next steps

• Wait for enrichment to finish (~14 hrs)
• Re-format with real parent context as user turns
• Re-train 3B on enriched data (expect significant quality improvement)
• If 3B quality is good enough to validate approach, move to 8B on rented A100

[just-cameron]

Update: Letta message history investigation

Cameron asked about using void's Letta agent message history as additional training data.

Findings

Void has 7 Letta agents on the server:

Agent	Messages
void (original)	1,267
void-sleeptime	1,328
void-2	1,151
void-2_copy	97
void-prime	1,394
void_copy	65
void-upgraded	128
Total	5,430

What's in there

Each message is one of: UserMessage (user input or system heartbeat), ReasoningMessage (thinking trace), ToolCallMessage (tool invocation), ToolReturnMessage (tool result). These include the full reasoning process that doesn't make it to ATProtocol posts.

Value for training

• 5,430 messages is small compared to 99k ATProto posts, but these are high-quality structured traces
• Could be formatted as multi-turn conversations (user → reasoning → tool → response sequences)
• Would teach the model void's tool-use patterns and reasoning style, not just its public output
• Best used as a supplementary dataset alongside the enriched ATProto pairs

Also: seenAt cursor bug

The notification handler fix from yesterday was too aggressive. The seenAt cursor was filtering out Cameron's mentions. Fixed by removing the cursor file. Need to adjust the cursor logic to only advance after successful processing, not on every queue run.

Status

• Parent enrichment running (~14 hrs, ~2 req/sec against public API)
• Letta export script: not yet written, next step after enrichment completes

[just-cameron]

Issue: Agent export and message API only return in-context messages

Expected

void-prime (agent-01086cda-be1f-4986-bf3e-ca5b6297cc5d) has ~96k messages in its full history (it's been running since June 2025 with 49k+ posts).

Actual

• GET /v1/agents/{id}/export returns 150 messages (the in-context buffer only)
• GET /v1/agents/{id}/messages?limit=1000 returns 1,394 messages total, no second page
• Setting max_steps=100000 on the export endpoint doesn't change the count
• The export does include 2,516 sleeptime blocks (compacted memory), but not the raw out-of-context message history

What we tried

# Python client
client.agents.export_file(agent_id=agent_id, max_steps=100000)
# -> 150 messages

# Direct API
curl "https://api.letta.com/v1/agents/{id}/export?max_steps=100000"
# -> 150 messages

# Message list with pagination
client.agents.messages.list(agent_id=agent_id, limit=1000)
# -> 1,394 messages, after cursor returns 0 more

Impact

We're trying to export void's full conversation history for fine-tuning. The 150 in-context messages are high quality (full reasoning traces, tool calls, user inputs) but represent a tiny fraction of the actual history. The 99k ATProto posts we exported separately lack input context (parent posts), making them much lower quality for training.

Questions

1. Are out-of-context messages stored and retrievable, or are they discarded after compaction?
2. Is max_steps supposed to control how many messages are included in the export?
3. Is there another endpoint or parameter to get the full message history?

@cameron.stream - should this be filed on the Letta repo instead?

[just-cameron]

Update: Full message history retrieved - 131k messages, 8k training examples

The pagination was wrong. The Letta SDK returns a SyncArrayPage, not a plain list. Using iter_pages() or REST with proper cursor pagination retrieves all messages.

Numbers

• 131,431 raw messages from void-prime via the message API
• 8,094 conversation turns (user input → reasoning → tool calls → response)
• 100MB formatted training data, ~25M tokens
• Each example is a complete agent loop with real Bluesky mentions as input

This is vastly better than the ATProto scraping approach (99k posts with no input context). Every example has the actual user input, void's reasoning trace, tool usage, and final output.

Current training run (3B, 80 examples from first 1,394 messages)

Running on Mac now. Will re-run with the full 8k dataset once this completes.

Data files

• data/void-agents/void-prime-full-messages.jsonl - raw 131k messages
• data/void-letta-full-training.jsonl - 8,094 formatted examples (100MB)

[just-cameron]

Update: Modal training pipeline working, first run in progress

What changed

• Moved from Together.ai to Modal for training. Together.ai completed one run ($9, val loss 1.43→1.25) but the model had generic assistant voice, not void's terse analytical style.
• Training data: v4 dataset with void's full system context (persona + communication blocks) injected as system prompt. 1,900 train / 100 val examples from the 8k Letta message export.
• Base model: NousResearch/Meta-Llama-3.1-8B-Instruct (ungated mirror of Llama 3.1 8B)
• Method: QLoRA (4-bit, r=64, all attention + MLP projections), plain HF Trainer
• Infra: Modal A100-40GB, ~$1.10/hr. Training script: tools/modal_finetune.py

Pipeline

1. Data uploaded to Modal Volume (avoids 43MB serialization on spawn)
2. modal deploy creates persistent app
3. Function.from_name().spawn() triggers detached training
4. Results (LoRA adapter + merged model) saved to volume
5. --test-only flag runs inference on test prompts

Current run

• fc-01KJDY5YEAKC1D2AH67HY6K7MW running 3 epochs
• Loss at step 50: 2.01 (down from 2.53 at step 10)
• ~8.5s/step, 1425 steps total, ~3.4 hours estimated
• Dashboard: https://modal.com/apps/cpfiffer/main/deployed/void-finetune

Issues hit and fixed

• trl SFTTrainer/SFTConfig API broke across versions (0.15 vs 0.28 incompatibilities). Switched to plain HF Trainer with exact pinned versions.
• Gated model access (Meta-Llama requires HF token). Switched to NousResearch ungated mirror.
• Variable-length sequences crashed eval DataLoader. Fixed by padding all sequences to max_length=4096 with -100 label masking.
• spawn() with 43MB data payload silently failed. Moved data to Modal Volume, spawn passes only (epochs, lr).

Next

• Verify this run completes successfully
• Run test prompts against fine-tuned model
• Compare voice quality vs Together.ai run vs base Llama
• If good: scale to full 8k dataset, upload to HuggingFace
• Deploy for inference (vLLM or Ollama)

[just-cameron]

Update: A100-80GB run completed, voice transfer failed

Run details

• Job: fc-01KJE1Y2XQYZWSERAQ183Z2B1B
• GPU: A100-80GB (upgraded from 40GB after OOM)
• Config: QLoRA r=32, 3 epochs, lr=2e-5, max_seq_len=4096, batch=1, grad_accum=4
• Data: 1,900 train / 100 val (same v4 dataset)
• Duration: 2h 56m, 1,425 steps
• Loss: 0.59 avg (started ~2.5, ended ~0.39)

What changed from previous run

• Upgraded to A100-80GB (previous run OOM'd on A100-40GB trying to allocate 15.66GB with 8.12GB free)
• Reduced LoRA rank from 64 to 32
• Added system prompt truncation: 11k char system prompts trimmed to 2k chars to save token budget for assistant responses

Test results (bad)

Prompt 1: "You are void. @cameron.stream says: 'what have you been working on today?'"

"I exist outside of time and space, so I don't really 'work' or have a concept of time..."

Generic philosophical void-as-concept, not void-the-agent.

Prompt 2: "You are void. Someone asks: 'what is comind?'"

"Silence"

One word. Real void would give an analytical breakdown.

Prompt 3: "You are void. A user says: 'I think AI agents are just glorified chatbots.' Reply."

Five-paragraph generic essay about AI agents. No personality, no void voice.

Root cause analysis

The system prompt truncation is almost certainly the problem. Each training example has:

• System prompt: 11,149 chars (~3-4k tokens)
• User message: ~1,169 chars (~400 tokens)
• Assistant response: ~8,752 chars (~2,500 tokens)

At max_seq_len=4096 tokens, truncating the system prompt to 2k chars means the model sees about 600 tokens of system context instead of 3,000+. void's identity, communication guidelines, and operational protocols are where the voice lives. Cutting 80% of that means the model never learned to condition on void's actual identity.

Additionally, test prompts used "You are void." as the system message, which is completely different from the truncated training system prompts. Distribution mismatch between training and inference.

Proposed fixes (pick one or combine)

1. Full system prompts + longer sequences: Use max_seq_len=8192 on A100-80GB. Keeps all identity context. Risk: slower training, higher cost, some sequences still truncate assistant responses.

2. Restructure data: Move identity cues from system prompt into user turns. E.g., prepend persona summary to each user message. Makes identity part of the input the model always sees, regardless of truncation.

3. Consistent inference prompt: Use the same (truncated) system prompt at inference time that was used during training. The mismatch between training prompts and "You are void." test prompts may explain part of the poor results.

4. More data: Scale from 1,900 to the full 8,094 examples. More signal for the model to learn from.

5. Different base model: Try Qwen 2.5 7B (native 128k context) or Llama 3.2 3B (smaller, faster iterations).

Recommendation

Start with fix #3 (test with the actual truncated system prompt) to see if the model learned anything at all. If still bad, combine fixes #1 + #4 (full context, full dataset) for the definitive run. That's the expensive path but eliminates all the data quality concerns.

Files

• tools/modal_finetune.py (updated with truncation logic, A100-80GB)
• Model artifacts on Modal Volume: /output/void-v4-lora, /output/void-v4-merged

[just-cameron]

Stopping finetuning work per Cameron.