Skip to content

feat(orchestrator): decouple summarizer model from design#286

Open
kalantar wants to merge 1 commit into
AI-native-Systems-Research:mainfrom
kalantar:orchestrator/decouple-summarizer-model
Open

feat(orchestrator): decouple summarizer model from design#286
kalantar wants to merge 1 commit into
AI-native-Systems-Research:mainfrom
kalantar:orchestrator/decouple-summarizer-model

Conversation

@kalantar

Copy link
Copy Markdown
Contributor

Summary

  • Per-iteration gate summaries previously read models.design, but the two phases use different transports: DESIGN goes through claude -p (which can carry the [1m] 1M-context tag) while gate summaries go through the LLM API via an OpenAI-compatible proxy that may not serve the same tag.
  • Add a dedicated models.summarizer knob (default claude-opus-4-6), expose it in campaign.schema.yaml, and have run_iteration resolve the summarizer model via _model_for("summarizer") instead of "design".
  • DESIGN and EXECUTE_ANALYZE behavior is unchanged. The existing summarizer role plumbing in dispatch.py, campaign.py, and llm_dispatch.py already expected this — the previous models.design lookup was a leftover from when the two phases shared a model.

Files changed

  • orchestrator/defaults.yaml — new models.summarizer default.
  • orchestrator/schemas/campaign.schema.yaml — schema entry so users can override per-campaign.
  • orchestrator/iteration.py — one-line swap: _model_for("design")_model_for("summarizer") for the summarizer's LLMDispatcher.

Ripples considered

No graphify-out/graph.json is present in the repo, so no automated ripple detection. Manual sweep across orchestrator/ and docs/ for models.summarizer / models.design / summarizer shows no other call sites that need updating.

Test plan

  • On a campaign that does NOT set models.summarizer: confirm gate summaries run with claude-opus-4-6 (the new default).
  • On a campaign that sets models.summarizer: <some-proxy-served-tag> in campaign.yaml: confirm gate summaries use that override and DESIGN keeps its own models.design.
  • On a campaign whose models.design is a [1m] variant the proxy doesn't serve: confirm DESIGN still works and summaries are no longer broken by the proxy mismatch.

Per-iteration gate summaries shared `models.design`, but the two phases
take different transports. DESIGN runs through `claude -p`, which can
carry the `[1m]` 1M-context model tag. Gate summarization runs through
the LLM API (OpenAI-compatible proxy), which may not serve that tag.

Add `models.summarizer` (default `claude-opus-4-6`), expose it in the
campaign schema, and have `run_iteration` resolve the summarizer model
independently. DESIGN is unchanged.
@mtoslalibu

Copy link
Copy Markdown
Collaborator

Review — Small fix + docs needed

Good fix for a real problem (proxy can't serve [1m] tags). Config/schema/defaults look good.

The issue

The llm_dispatcher at line 384 is shared — it's the fallback for DESIGN/EXECUTE when there's no CLI. This PR changes its model, which accidentally changes the fallback model too.

The fix — do what the reporter does

The reporter in campaign.py (line 94) already solves this correctly:

resolved = _resolve_model(campaign, "report", model)
dispatcher = LLMDispatcher(work_dir=work_dir, campaign=campaign, model=resolved)
dispatcher.dispatch("extractor", "report", ...)

It creates its own fresh LLMDispatcher with its own model. Nobody else shares it.

The summarizer should do the same. Instead of changing the model on the shared llm_dispatcher, create a dedicated instance:

# Line 384 — leave this alone (DESIGN/EXECUTE fallback)
llm_dispatcher = LLMDispatcher(work_dir=work_dir, campaign=campaign, model=_model_for("design"))

# New — dedicated instance for gate summaries, same pattern as reporter
summary_dispatcher = LLMDispatcher(work_dir=work_dir, campaign=campaign, model=_model_for("summarizer"))

Then pass summary_dispatcher to _generate_gate_summary() at lines 439 and 548.

Both reporter and summarizer use LLMDispatcher — the only difference is the reporter creates its own instance while the summarizer currently piggybacks on a shared one. This fix makes them consistent.

Documentation needed

Please add a section to the campaign configuration docs (or README) that explains the model configuration for users:

1. What models exist and their defaults:

Role Default Transport Purpose
models.design claude-opus-4-6 claude -p (CLI) Plans experiments, reads codebase
models.execute_analyze claude-sonnet-4-6 claude -p (CLI) Implements and runs experiments
models.summarizer claude-opus-4-6 API proxy Generates per-iteration gate summaries
models.report claude-sonnet-4-6 API proxy Generates final campaign report

2. How to override in campaign.yaml:

models:
  design: "claude-opus-4-6[1m]"       # CLI — supports [1m] context tags
  execute_analyze: "claude-sonnet-4-6"
  summarizer: "claude-opus-4-6"        # API proxy — must be proxy-served
  report: "claude-sonnet-4-6"          # API proxy — must be proxy-served

3. The key distinction users need to understand:

  • Agents (design, execute_analyze) use claude -p CLI — support [1m] tags, tool use, deep context
  • LLMs (summarizer, reporter) use the OpenAI-compatible API proxy — must use tags your proxy actually serves. Don't use [1m] variants here.

This helps users understand why models are configured separately and prevents the exact proxy mismatch bug this PR fixes.

Summary

What Status
Config + schema + defaults Good
iteration.py wiring Needs fix — create a separate summary_dispatcher
User-facing documentation Needed — model table, override example, agent vs LLM explanation

With those two additions, good to merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants