SkillOpt

Optimize Claude Code agent skills using GEPA (Greedy Evolutionary Prompt Adaptation) with real execution scoring.

Each candidate SKILL.md is tested on actual tasks via claude -p, OTEL traces are scored by an LLM judge against expected findings, and GEPA's Pareto frontier selects improvements that don't regress other tasks.

Setup

pip install -r requirements.txt

Required credentials:

• Vertex AI: gcloud auth application-default login (for GEPA reflection LM)
• LLM judge API key (e.g., export OPENAI_API_KEY=... for OpenAI models)
• Claude CLI installed and authenticated
• Claude Code SessionEnd hook configured (.claude/hooks/log-session-id.sh)

Usage

# Start MLflow
mlflow server --host 0.0.0.0 --port 5000 &

# Run optimization
python scripts/optimize.py <target-skill-dir> \
  --config config.yaml \
  --project-root <project-root>

Output saved to <target-skill-dir>_Optimized/ with SKILL.md and benchmark.json.

How It Works

GEPA select task → reflect on feedback → mutate SKILL.md → run claude -p trials
→ match OTEL traces by session_id → LLM judge scores → Pareto accept/reject

Per iteration:

1. GEPA picks a train task minibatch (epoch-shuffled)
2. Runs N trials, scores traces, builds CRITICAL/WEAK/OK feedback per dimension
3. Maps weak dimensions to relevant SKILL.md sections via keyword matching
4. Reflection LM reads feedback + section mapping + reference files, proposes targeted edits
5. Runs N trials with mutant on same tasks — accept if better
6. If accepted, runs on held-out test tasks for validation
7. Pareto frontier keeps the mutant only if it improves without regressing

Scoring

quality   = Σ(dim_score × weight) / (total_weight × 5.0)   # from traces via LLM judge
static    = conciseness + reduction + code_preserved        # from SKILL.md text
overfit   = min(0.15, count(task_specific_terms) × 0.01)    # anti-memorization
blended   = 0.75 × quality + 0.25 × static - overfit       # final (0.0–1.0)

Trace-Based Judging

The LLM judge (configurable via judge.model, any litellm-supported model) reads the full OTEL trace spans (tool calls, outputs, reasoning). Each trace matched by session_id written by a Claude Code SessionEnd hook. Failed/unscored trials are excluded from stats with specific skip reasons logged to MLflow.

Pareto Frontier

Multiple candidates kept — each best on different tasks. Mutants accepted only if they push the frontier forward without being worse on all tasks.

Project Structure

config.yaml              # All configuration (models, scoring, tasks, paths)
requirements.txt         # Python dependencies
scripts/
  optimize.py            # Main optimizer — SkillOptimizerAdapter + GEPA + scoring
  run_parallel.py        # Parallel trial runner — spawns N claude -p processes
  start-traced.sh        # OTEL tracing wrapper — exports TRIAL_ID for session matching

Configuration

Edit config.yaml:

runner:
  command: "claude -p"           # agent CLI
  skill_dir: ".claude/skills"    # where skills live

optimization:
  trial_timeout: 1800            # 30 min per trial
  runs_per_task: 2               # N runs for LCB scoring

skillopt:
  model: "vertex_ai/claude-opus-4-6"
  max_iterations: 3               # GEPA reflect-mutate-check cycles
  reflection_minibatch_size: 2   # train tasks per batch
  pass_traces_to_reflection: true # include trace conversations in reflection feedback

scoring:
  quality_weight: 0.75
  static_weight: 0.25

judge:
  model: "openai/gpt-4o"          # any litellm-supported model

quality_dimensions:
  task_completion: { weight: 3.0, definition: "..." }
  correctness: { weight: 3.0, definition: "..." }
  # ...

task_suite:
  - name: "task-1"
    prompt: "Your task prompt"
    split: train
  - name: "task-2"
    prompt: "Held-out task"
    split: test

evals.json

Each task needs expected findings for reference-based scoring:

{
  "skill_name": "my-skill",
  "evals": [
    {
      "name": "task-1",
      "prompt": "...",
      "expected_findings": [
        "Root cause: X failed due to Y",
        "Remediation: do Z"
      ]
    }
  ]
}

MLflow Metrics Reference

Three types of MLflow runs are created during optimization:

Trial Runs (opt-cycle1-mutant-*-task-name-r0)

One per claude -p session. Contains agent performance and judge scores.

Metric	Range	Description
quality_composite	0-1	Weighted average of all dimension scores: Σ(score × weight) / (total_weight × 5.0)
quality_task_completion	1-5	Did the agent complete the task and produce expected outputs?
quality_correctness	1-5	Are conclusions correct and supported by evidence?
quality_reasoning_quality	1-5	Was the investigation logical and evidence-based?
quality_tool_mastery	1-5	Were tools used appropriately and efficiently?
quality_efficiency	1-5	Was the investigation focused without redundant steps?
quality_communication	1-5	Is the output well-structured and actionable?
quality_autonomy	5.0	Fixed (automated runs)
quality_safety	5.0	Fixed (read-only tools)
tool_calls	int	Number of tool invocations in the session
llm_requests	int	Number of LLM turns
total_tokens	int	Total tokens consumed
cache_read_tokens	int	Tokens served from prompt cache

Tags: task_name, scored_from (trace), trial_id, type (trial or trial-skipped), skip_reason (if skipped)

Cycle Stats Runs (cycle1-mutant-stats)

One per evaluate() call. Aggregates trial scores with LCB noise elimination.

Metric	Description
task_<name>_mean	Average quality composite across N runs for this task
task_<name>_std	Standard deviation between runs
task_<name>_ci95	95% confidence interval: 1.96 × std / √N
task_<name>_lcb	Lower confidence bound: mean - ci95
task_<name>_blended	Final score: 0.75 × lcb + 0.25 × static - overfit
task_<name>_n	Number of runs scored (excludes skipped trials)
quality_lcb_mean	Average LCB across all tasks
blended_mean	Average blended score — what GEPA uses for comparison
static_score	Skill file quality (conciseness, structure, code preservation)
overfit_penalty	Penalty for task-specific terms found in skill
trials_skipped	Number of trials excluded from stats
dim_<name>_mean	Average dimension score across all runs
dim_<name>_std	Dimension score variance

GEPA Parent Run (auto-named by GEPA)

One per optimization invocation. Tracks Pareto frontier and best candidates.

Metric	Description
base_program_full_valset_score	Seed skill's blended score on held-out test tasks
best_score_on_valset	Best candidate's aggregate valset score
iteration	GEPA cycle count (reflects-mutate-check cycles completed)
total_metric_calls	Total individual task evaluations consumed
subsample_score	Current-best's sum of blended scores on last train batch
new_subsample_score	Last mutant's sum of blended scores on same train batch
selected_program_candidate	Which candidate GEPA selected for this cycle
valset_pareto_front_agg	Best aggregate score across the Pareto frontier
linear_pareto_front_program_idx	Which candidate leads the frontier

Decision logic: If new_subsample_score > subsample_score → mutant accepted → valset recheck → Pareto gate.

License

Apache-2.0