harness-farmer

Turn your everyday Claude Code and Codex work into a private, curated dataset for local models.
Collect the history you already own. Remove the noise and sensitive data. Keep the useful conversations.

Quickstart · How it works · Curation council · Privacy · CLI reference

The short version: harness-farmer is a local-first data pipeline for harvesting CLI-agent conversations and turning them into clean, PII-scrubbed, origin-aware training data. It prepares exports for your fine-tuning stack; it does not send your history to a hosted service or train a model itself.

Why this exists

Your best prompts and answers are already hidden in terminal sessions: debugging strategies, architecture decisions, implementation patterns, and the exact context that makes an agent useful. Those conversations are valuable training material, but raw session logs also contain tool chatter, injected harness context, credentials, personal paths, binary blobs, and low-signal acknowledgements.

Harness-farmer is the missing data layer between using an agent and improving a local model:

Your CLI history
       │
       ▼
  harvest → clean → scrub → store → judge → report / export
                                              │
                                              ▼
                               your local fine-tuning workflow

Everything runs on your machine. PostgreSQL is the primary store; SQLite is available for a zero-setup start, development, and portable exports.

What it does

Stage	Responsibility	Result
Harvest	Parse Claude Code and Codex JSONL session files	Normalized conversations with origin preserved
Clean	Remove harness blocks, tool traffic, ANSI escapes, blobs, trivial turns, and short replies	Substantive user → assistant examples
Scrub	Redact PII, credentials, tokens, paths, IPs, phones, and card numbers before storage	Safer text with stable placeholders
Store	Persist sessions, examples, file hashes, and judgment state	Incremental, deduplicated local dataset
Curate	Run an origin-blind persona council over offline batches	Keep/reject verdicts and 0–10 usefulness grades
Analyze	Compare source quality and dataset composition	CLI statistics or a self-contained SVG/HTML report
Export	Write JSONL chat data, CSV, or standalone SQLite	Files ready for analysis or SFT/LoRA tooling

How it works

1. Harvest the conversations you already have

The built-in parsers read the history directories used by the two supported CLI tools:

• Claude Code: ~/.claude/projects/*/*.jsonl
• Codex: ~/.codex/sessions/**/rollout-*.jsonl

They keep real user and assistant messages, attach claude-code or codex as the origin, and drop tool results, thinking blocks, injected context, sidechains, and non-human traffic. Adding another source is intentionally small: implement the common harvest(...) interface and register it in sources/__init__.py.

Harvesting is on demand, incremental, and idempotent. Each source file is hashed and recorded only after its examples have committed. Unchanged files are skipped before JSON parsing; changed or newly grown files are re-read, while content hashes prevent duplicate examples. Resumed or forked sessions may share a session id, so file count and session count are intentionally different concepts.

2. Convert noisy turns into useful pairs

The cleaning stage turns normalized turns into supervised examples:

raw turns
  → strip <system-reminder>, <environment_context>, command blocks, ANSI, and blobs
  → merge consecutive turns from the same role
  → pair substantive user prompts with the assistant response that follows
  → discard trivial prompts and responses shorter than 40 characters
  → cap individual text fields at 32,000 characters

This keeps the signal that teaches a model how to solve a task, rather than the machinery surrounding the task.

3. Scrub before anything is stored

Every prompt, response, project name, and source path passes through the privacy scrubber in the CLI before it reaches the database. Exporters never receive an unscrubbed row. Replacements are stable and scrubbing is idempotent, which makes repeat harvests predictable.

See the full privacy guarantees below for the supported leak classes and the honest caveats.

4. Curate without leaking origin bias

The optional curation loop is deliberately offline. judge dump emits only an example id, prompt, and response; it does not expose whether the row came from Claude Code or Codex. Persona judges therefore assess the material rather than their expectation of its source.

The final verdict is applied locally. Kept rows remain available for export; rejected text is deleted and only its content hash plus origin survive in a rejection ledger. If a changed session file is harvested again, the rejected content cannot re-enter.

5. Measure what you are about to train

report summarizes filtered and kept data, grades, and per-origin outcomes. report --html creates a standalone visual report with inline SVG, light/dark styling, response-length profiles, grade distributions, harvest volumes, and deterministic analyst insights such as grade cuts, sampling weights, multi-turn opportunities, and near-duplicate cleanup.

Quickstart

The core package uses only the Python standard library (Python 3.10+). Start with SQLite if you want to try the pipeline immediately:

git clone <your-clone-url> harness-farmer
cd harness-farmer

pip install -e .
export HARNESS_FARMER_DB="sqlite:///farm.db"

harness-farmer init-db
harness-farmer harvest
harness-farmer stats

Export a chat-format dataset:

harness-farmer export --format jsonl -o dataset.jsonl

The package can also be run directly from a checkout on a machine without pip:

PYTHONPATH=src python3 -m harness_farmer.cli --db sqlite:///farm.db harvest

PostgreSQL

Install the optional driver and point the store at PostgreSQL:

pip install -e ".[postgres]"
export HARNESS_FARMER_DB="postgresql://harness:harness@127.0.0.1:5433/harness_farmer"
harness-farmer init-db

For a disposable local container:

docker run -d --name harness-farmer-pg --restart unless-stopped \
  -e POSTGRES_PASSWORD=harness \
  -e POSTGRES_USER=harness \
  -e POSTGRES_DB=harness_farmer \
  -p 127.0.0.1:5433:5432 \
  -v harness_farmer_pgdata:/var/lib/postgresql/data \
  postgres:17

A complete curation loop

Harvest first, then judge in reviewable batches:

# 1. Create an origin-blind batch
harness-farmer judge dump -o batch.jsonl --limit 50

# 2. Have the council personas review batch.jsonl and produce verdicts.jsonl

# 3. Apply {id, grade, verdict, reasons} records
harness-farmer judge apply verdicts.jsonl

# 4. Inspect the result
harness-farmer report
harness-farmer report --html report.html

# 5. Export only the strongest judged examples
harness-farmer export --format jsonl --min-grade 7 -o top.jsonl

--min-grade excludes unjudged examples as well as rows below the requested 0–10 floor, making it suitable for a deliberate high-quality training slice.

The curation council

Four portable Markdown personas make different kinds of evidence visible:

• Data miner — signal density, prompt/response coupling, label correctness, and distribution health.
• Business analyst — recurring real-world capability, actionability, and usefulness beyond one narrow task.
• Safety officer — harmful content and residual personal data; holds an absolute veto.
• Counselor — chairs the debate, resolves disagreement, and emits the machine-readable verdict.

The council is intentionally not a single opaque score. The judges use different value systems, the counselor rules on the debate, and the report reconnects the final outcomes to their original source for an honest Claude Code vs. Codex comparison.

Dataset formats

JSONL — chat format

Compatible with common supervised fine-tuning workflows such as Axolotl, Unsloth, LLaMA-Factory, and MLX-LM. The top-level source field preserves origin for sampling and ablation; trainers can ignore the extra metadata.

{"source":"codex","messages":[
  {"role":"user","content":"How should this parser handle resumed sessions?"},
  {"role":"assistant","content":"Treat the file hash and session id as separate identities..."}
]}

Use --multi-turn to group all cleaned pairs from a session into one messages array.

CSV

Flat rows with source, session_id, project, turn_index, prompt, and response columns. Useful for spreadsheets, notebooks, and quick audits.

SQLite

A standalone examples table with the same fields, useful for ad-hoc SQL analysis without opening the primary store.

Privacy by design

Your source files are read-only. The harvester never edits or deletes anything under ~/.claude, ~/.codex, or an overridden --root; destructive operations apply only to the dataset store. A regression test checks source bytes and timestamps after harvesting.

Leak class	Replacement
Email addresses	[EMAIL]
API keys, tokens, JWTs, private keys, bearer headers, and secret assignments	[SECRET]
Credentials embedded in URLs	[CREDENTIALS]@
Public IPv4 addresses (loopback preserved)	[IP]
International phone numbers	[PHONE]
Luhn-valid payment card numbers	[CARD]
Home directories such as /home/alice or /Users/alice	/home/USER
The local account name in paths, prompts, and prose	USER

Two caveats matter:

1. Changing a scrub rule does not rewrite rows already in a store. Re-harvest into a fresh store to validate a new rule.
2. Scrubbing errs toward over-redaction. An identifier beginning with a sensitive keyword can lose nearby text; the safety judge should treat resulting corruption as a quality issue.

CLI reference

Global options go before the subcommand:

Option	Meaning
--db DSN	postgresql://... or sqlite:///path.db; defaults to $HARNESS_FARMER_DB, then postgresql://localhost/harness_farmer
--version	Print the installed version

Command	Purpose
init-db	Create or migrate sessions, examples, files, and rejected_hashes
harvest	Parse, clean, scrub, and store session history
stats	Print JSON counts, including per-source totals and average response length
judge dump	Write unjudged, origin-blind JSONL batches
judge apply	Validate and apply council verdicts
report	Show filtering, grades, and per-origin comparison
report --html PATH	Write the self-contained visual analyst report
export	Write jsonl, csv, or sqlite output

Harvest options:

--source {claude-code,codex,all}   default: all
--root PATH                        override a source history directory
--since YYYY-MM-DD                 process files modified on/after this date

Export options:

--format {csv,jsonl,sqlite}        required output format
-o, --output PATH                  required destination
--min-response-chars N             omit shorter responses
--min-grade 0-10                   keep only judged examples at or above grade
--multi-turn                       JSONL only; group pairs by session

Project layout

src/harness_farmer/
├── cli.py                 command-line orchestration
├── sources/               Claude Code and Codex parsers
├── cleaning.py            turn normalization and example extraction
├── privacy.py             pre-storage PII scrubbing
├── db.py                  SQLite/PostgreSQL store and migrations
├── export.py              JSONL, CSV, and SQLite exporters
├── analytics.py           deterministic dataset analysis
└── html_report.py         self-contained SVG/HTML report
personas/                  portable analyst, scientist, reviewer, and council roles
tests/                     synthetic fixtures and regression suite
branding/                  README artwork

The runtime pipeline is deliberately linear and easy to audit:

sources/ → cleaning.py → privacy.py → db.py → export.py
                                      ↘ analytics.py → html_report.py

Development

No third-party dependencies are required for the core test suite:

python3 -m unittest discover -s tests

To exercise a disposable PostgreSQL database as well:

HARNESS_FARMER_TEST_PG_DSN="postgresql://harness:harness@127.0.0.1:5433/harness_farmer_test" \
  python3 -m unittest discover -s tests

Contributor constraints and architecture notes live in CLAUDE.md. In particular: keep source files read-only, keep the core stdlib-only, preserve source through every stage, and run the project reviewer persona after changes.

License

Apache 2.0