research: DGM vs HyperAgents — what should devswarm's self-improvement architecture look like?

[justrach]

Context

Two papers define the space for self-improving agent systems. We need to decide which architecture to adopt for devswarm's evolutionary grid (#274), prompt evolution (#353), and eval framework (#354) — and whether the cost is practical for production use.

Darwin Gödel Machine (DGM) — open-ended evolution of coding agents
HyperAgents (DGM-H) (arXiv:2603.19461) — extends DGM with editable meta-agents

The core difference

	DGM	HyperAgents (DGM-H)
What evolves	Task agent (coding agent)	Task agent + meta agent (both editable)
Meta-level	Fixed, handcrafted scaffold	Editable — agent can improve how it improves
Domain assumption	Coding skill ≈ self-improvement skill (aligned)	No alignment needed — meta agent is domain-agnostic
Archive	Stepping stones of agent variants	Same, but hyperagent variants (task + meta)
Transfer	Weak — improvements are coding-specific	Strong — meta-improvements transfer across domains

What DGM does well

1. Crisp formulation. Mutate coding agents, evaluate empirically, keep good branches, repeat. Clean bridge between Gödel-machine theory and running systems.
2. Archive/open-ended search is the real contribution. Keeping many lineages matters because useful stepping stones may only pay off much later.
3. Strong empirical results in its home domain. 20% → 50% on SWE-bench, 14.2% → 30.7% on Polyglot.

Critique of DGM

1. "Self-improvement" is only partly self-improvement. The open-ended exploration process (archive maintenance, parent selection) is fixed and non-modifiable. The instruction-generation mechanism is handcrafted. DGM is really: self-editing agent inside a fixed evolutionary scaffold.
2. Domain-specific alignment assumption. Coding performance as proxy for self-improvement ability is plausible in coding, weak outside it.
3. May be too benchmark-coupled. Better at the benchmark distribution ≠ more general self-reflection.

What HyperAgents does well

1. Attacks the real bottleneck. Makes the meta-level procedure editable — the strongest conceptual advance over DGM.
2. Generalizes beyond coding. Evaluated on paper review, robotics reward design, Olympiad math grading.
3. Transfer of meta-level improvements. Learned mechanisms (persistent memory, performance tracking) transfer across domains and accumulate across runs. DGM-H goes from 0.0 → 0.710 on paper review; DGM gets 0.0.
4. Stays competitive on coding. 0.140 → 0.340 on Polyglot (comparable to DGM, without being handcrafted for coding).

Critique of HyperAgents

1. Experimentally messier. Broader claim surface (coding, review, robotics, math, transfer, compounding) gives more room for hidden evaluator dependence and benchmark fragility.
2. "Improves how it improves" is shallow-ish. Evidence includes memory and tracking — real meta-improvement, but still high-level software-engineering scaffolding, not deep algorithmic self-redesign.
3. Non-coding tasks are less ground-truth hard. Paper review and grading can partially mirror the generator's style. "Works on any computable task" is a research direction, not a proven result.
4. Still inherits fixed outer-loop choices. Human-defined evaluation tasks, archive rules, staged evaluation, compute budgets. More self-referential than DGM, but not fully unconstrained.

The production cost problem

Both approaches are expensive to run continuously in production software:

• DGM/DGM-H run in Docker containers, generating full code diffs, evaluating over 50-80 iterations
• Each generation costs real API calls (orchestrator + workers + evaluation)
• Our swarm already costs significant API spend at 4 agents per run
• Running an outer evolution loop ON TOP of the swarm is prohibitive for daily use

Proposed compromise: periodic tuning cadence

Instead of continuous evolution, devswarm should adopt a periodic tuning schedule:

Cadence	What evolves	Cost
Per-run	Nothing — use current grid + prompts as-is	$0 extra
Weekly	Light: prompt variants via QD search on accumulated telemetry	Low — reuse existing swarm runs as eval signal
Bi-weekly	Medium: grid mapping (role→model) + prompt text	Moderate — dedicated eval runs
Monthly	Heavy: full DGM-H style — meta agent + task agent + strategies	High — but amortized over a month of usage

Telemetry from every run_swarm call (#342, #346) feeds into the archive passively. The evolution loop runs offline, not in the hot path.

What this means for devswarm's architecture

Current state (what we have)

[Orchestrator] → [Workers] → [Synthesizer]
     ↑ fixed prompt    ↑ fixed roles    ↑ fixed strategy

#353 as filed (prompt-only evolution) — this is the "DGM-H w/o self-improve" baseline

[Orchestrator] → [Workers] → [Synthesizer]
     ↑ fixed         ↑ EVOLVED prompts    ↑ fixed

The HyperAgents paper shows this is the weaker approach.

What HyperAgents suggests (full hyperagent evolution)

[Orchestrator] → [Workers] → [Synthesizer]
     ↑ EVOLVED       ↑ EVOLVED prompts    ↑ EVOLVED
     decomposition    + tool chains         synthesis strategy
     strategy         + meta strategies

The meta agent (orchestrator + synthesis) must also be evolvable.

Practical middle ground for devswarm

Monthly evolution loop:
  1. Collect telemetry from all swarm runs (passive)
  2. Evolve prompt variants (QD search) — cheap
  3. Evolve decomposition strategies (orchestrator prompt variants) — moderate
  4. Evaluate on accumulated task history — use real past runs as benchmark
  5. Archive best variants in .devswarm/evolved/
  6. Grid auto-updates to use winners

Per-run: just use the current best from the archive. Zero overhead.

Decision needed

1. Do we go DGM-style (evolve task agents only, fixed meta) or DGM-H-style (evolve everything)?

   • DGM is simpler, cheaper, proven for coding
   • DGM-H is more general, but costlier and less proven in production

2. What's the tuning cadence?

   • Weekly light / bi-weekly medium / monthly heavy?
   • Or triggered by performance regression (adaptive)?

3. What's the "genome" that evolves?

   • Prompt text only? (cheapest, weakest)
   • Prompt + grid mapping? (moderate)
   • Prompt + grid + decomposition + synthesis strategies? (DGM-H level)

Verdict

DGM is the stronger paper qua experiment. HyperAgents is the stronger paper qua research agenda.

For devswarm: start with DGM-style (evolve prompts + grid, fixed meta) because it's cheaper and we have production constraints. But architect the system so the meta-level (orchestrator, synthesis) CAN be swapped in as evolvable later — don't paint ourselves into a corner.

Blocks

• feat: evolutionary grid tuning — DGM outer loop tunes role→model per codebase #274 (evolutionary grid tuning) — this issue defines the architecture
• feat: evolutionary prompt optimization — Rainbow Teaming QD loop for role system prompts #353 (prompt evolution) — scope depends on this decision
• research: multi-agent eval framework — what metrics select the best role prompts? #354 (eval framework) — needs score_child_prop parent selection from DGM-H

References

• DGM paper — Darwin Gödel Machine
• HyperAgents paper — DGM-H
• Rainbow Teaming — QD search for prompt diversity
• HyperAgents code

[justrach]

Baselines & success criteria

What "baseline" means here

A baseline is the control group — the version of devswarm with no evolution, so we can measure whether evolution actually helps. Without baselines, we can't distinguish "evolution improved things" from "we just spent money on API calls."

Baseline definitions

Baseline	Description	Purpose
B0: Static hand-tuned	Current roles.zig prompts + grid.zig mapping, no evolution. This is what we ship in #352.	The "does evolution beat a human?" test. If evolution can't beat hand-tuned prompts, it's not worth running.
B1: Random search	Generate N random prompt variants, pick the best by eval score. No archive, no stepping stones.	The "does structured search beat random?" test. If QD/DGM doesn't beat random, the archive machinery is waste.
B2: DGM-H w/o self-improve	Evolve task agent prompts only, keep meta agent (orchestrator/synthesis) fixed. Use archive + stepping stones.	The "does meta-evolution matter?" test. This is what #353 currently proposes.
B3: DGM-H w/o open-ended exploration	Evolve everything (task + meta), but no archive — each new variant replaces its parent.	The "does the archive matter?" test. Isolates whether keeping lineages helps.
B4: Full DGM-H	Evolve task + meta agents, with archive, stepping stones, and parent selection.	The full system. Should beat all above if the theory is right.

How to ensure it works — success criteria

Per-role metrics (from #354)

Role	Primary metric	Secondary	Baseline bar
finder	Recall@k (found all relevant files?)	Precision (false positive rate)	B0 hand-tuned finder finds ≥70% of relevant files
reviewer	True positive rate on seeded bugs	False alarm rate	B0 catches ≥60% of seeded bugs
fixer	Compile rate × test pass rate	Diff size (smaller = better)	B0 fix compiles 90%+, passes tests 70%+
safety_auditor	Detection rate on seeded UAF/double-free	False alarm rate	B0 detects ≥50% of seeded memory bugs
orchestrator	Child performance (do its decompositions produce better worker output?)	Decomposition quality score	B0 decomposition leads to ≥3/5 successful workers
synthesizer	Human preference rating vs concatenated output	Information preservation	B0 synthesis preferred ≥60% over raw concat

System-level success criteria

1. Evolution must beat B0 (hand-tuned) by ≥10% on primary metrics within 20 generations. If it can't, the cost isn't justified.
2. Evolution must beat B1 (random search) by ≥15%. If structured search barely beats random, the archive/QD machinery is over-engineering.
3. No regression on any role. If evolving the fixer makes it better but the reviewer gets worse, that's a failure. Pareto improvement required.
4. Cost ceiling: evolution budget ≤ 20% of monthly swarm API spend. If tuning costs more than using, the economics don't work.
5. Convergence check: improvement must be statistically significant (p < 0.05) over 5 repeated runs. One lucky run doesn't count.

How to detect if it's NOT working

Red flag	What it means	Action
Evolved prompts are longer but not better	The LLM mutation is adding verbosity, not insight	Add a length penalty to fitness
Scores improve on train, flat on held-out	Overfitting to the benchmark task suite	Expand held-out set, add diversity pressure
B2 ≈ B4 (meta-evolution doesn't help)	The orchestrator/synthesis is already good enough, or the search space is too constrained	Stick with B2 (cheaper), revisit when more domains
B1 ≈ B2 (random ≈ structured search)	Archive/stepping stones aren't helping — the fitness landscape is smooth enough that random search works	Simplify to random search, save the engineering effort
Costs exceed 20% ceiling	Evolution is too expensive for the gains	Increase tuning cadence interval (monthly → quarterly) or reduce population size

Staged rollout

Don't go from B0 → B4 in one jump. Staged validation:

Week 1-2:  Implement B1 (random search) + eval harness from #354
           → Establishes that eval works, gives a random baseline number

Week 3-4:  Implement B2 (prompt-only evolution with archive)
           → If B2 > B1 by ≥15%, archive is justified
           → If B2 ≈ B1, stop here, random search is enough

Week 5-6:  Implement B3 (full evolution, no archive)
           → If B3 > B2, meta-evolution matters
           → If B3 ≈ B2, meta is not worth the cost

Week 7-8:  Implement B4 (full DGM-H) only if B3 > B2
           → If B4 > B3, archive + meta is the winning combo
           → Ship B4 as the monthly tuning loop

Each stage has a go/no-go gate. We don't build B4 unless B2 and B3 justify it.

This way we spend engineering effort proportional to proven value, and we have clear numbers at each stage to decide whether to continue or stop.

[justrach]

Update: skill-compatible mode changes the evolution story

#356 proposes a skill-compatible mode where roles are defined as Markdown files in .devswarm/skills/. This has a major implication for the DGM vs HyperAgents decision:

Markdown skill files are trivially evolvable. Instead of evolving compiled Zig (dangerous, complex), the evolution loop mutates Markdown prompts (safe, inspectable, revertable). The "agent program" that DGM-H wants to evolve becomes a collection of .md files — which is exactly the right abstraction level for production self-improvement.

This makes the B2 baseline (prompt-only evolution with archive) much more practical, and potentially sufficient without needing B4 (full DGM-H). The meta-level (orchestrator strategy, pipeline order) can also be expressed as skill metadata, keeping everything in the evolvable Markdown layer.