Skip to content

Overhaul ACE data pipeline: performance reports, project registry, vault removal #175

@akashgit

Description

@akashgit

Factory experiment 53. Implements issue #173.

What to Build

A 5-phase implementation that closes all 4 data gaps in the ACE self-improvement pipeline:

Phase 1 — Core Data Infrastructure

  • Add 5 new Pydantic models to factory/models.py: AgentVerdict, Observation, PerformanceReport, ProjectEntry, ProjectRegistry
  • New factory/registry.py: atomic load/save of ~/.factory/registry.json, idempotent register_project(), update_project_stats(), get_project_paths() replacing discover_projects() with fallback
  • New factory/report.py: parse_ceo_verdicts(), build_performance_report(), observation append, file I/O

Phase 2 — Store & CLI Integration

  • store.py: begin() calls register_project(), finalize() calls update_project_stats() (both try/except non-blocking)
  • cli.py: new report-update and registry-list commands
  • Update cmd_ace(), cmd_insights(), cmd_study(): make --projects-dir optional, prefer registry

Phase 3 — Reflector Enhancement

  • reflector.py: _load_from_reports() with TSV fallback, _verdict_bullets(), _observation_bullets(), use get_project_paths()

Phase 4 — Prompt Updates (Vault Removal)

  • archivist.md: remove ALL Obsidian references, .factory/archive/ only, add factory report-update responsibility
  • ceo.md: remove --projects-dir, update archivist task strings, meta mode vault removal
  • researcher.md: replace vault references with .factory/archive/ paths

Phase 5 — Tests & Migration

  • tests/test_registry.py, tests/test_report.py, tests/test_reflector_enhanced.py
  • Update existing tests that mock discover_projects
  • Migration: auto-populate registry from directory scan on first use

Acceptance Criteria

  • factory report-update /path generates .factory/performance_report.json with experiments, verdicts, observations
  • factory registry-list shows registered projects
  • factory ace /path works without --projects-dir (uses registry)
  • factory ace /path reads performance reports for qualitative signals
  • Archivist prompt has zero Obsidian/vault references
  • CEO prompt archivist tasks reference factory report-update instead of vault
  • store.begin() and store.finalize() auto-update the registry
  • Reflector generates bullets from CEO verdict patterns and archivist observations
  • All existing tests pass
  • New tests for registry, report, and updated reflector
  • ruff check . clean, mypy factory/ clean

Constraints

  • Read CLAUDE.md before starting
  • Do NOT touch files outside declared scope
  • Do NOT modify eval/score.py or .factory/ data files
  • All registry/report calls in store.py must be try/except non-blocking
  • All new models use ConfigDict(strict=True, extra="forbid")
  • Maintain backward compatibility — TSV fallback, discover_projects() fallback

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions