Skip to content

feat: close plugin-vs-CLI gaps (#199)#201

Merged
xukai92 merged 4 commits into
akashgit:mainfrom
xukai92:fix/plugin-cli-gaps
May 9, 2026
Merged

feat: close plugin-vs-CLI gaps (#199)#201
xukai92 merged 4 commits into
akashgit:mainfrom
xukai92:fix/plugin-cli-gaps

Conversation

@xukai92
Copy link
Copy Markdown
Collaborator

@xukai92 xukai92 commented May 8, 2026

Closes #199.

Summary

Closes #199. Four fixes to bring the plugin path closer to CLI observability:

  • Event capture: add factory emit CLI subcommand so pipeline skills can log agent.started/agent.completed events to .factory/events.jsonl
  • Playbook injection: generate_agent_content() now calls load_playbook() + inject_playbook() — plugin agents get the same behavioral playbooks as CLI agents
  • Model flexibility: omit model from generated frontmatter so plugin subagents inherit the parent session's model instead of hardcoding sonnet/opus
  • Review persistence: pipeline-subagents skill now instructs writing agent output to .factory/reviews/<role>-latest.md, matching CLI artifacts

Test plan

  • uv run pytest -x -q — 1516 passed
  • uv run ruff check . — clean
  • 3 new tests for factory emit (basic event, with JSON data, without agent)
  • Existing plugin agent tests updated for model omission and playbook injection

🤖 Generated with Claude Code

Four fixes to bring the plugin path closer to CLI observability:

1. Event capture: add `factory emit` CLI subcommand so pipeline
   skills can log agent.started/completed events to events.jsonl
2. Playbook injection: generate_agent_content() now calls
   load_playbook() + inject_playbook() — plugin agents get the
   same behavioral playbooks as CLI agents
3. Model flexibility: omit model from generated frontmatter so
   plugin subagents inherit the parent session's model
4. Review persistence: pipeline-subagents skill now instructs
   writing output to .factory/reviews/<role>-latest.md

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@codecov
Copy link
Copy Markdown

codecov Bot commented May 8, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.01%. Comparing base (5a62439) to head (7e6f1ed).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #201      +/-   ##
==========================================
+ Coverage   86.88%   87.01%   +0.13%     
==========================================
  Files          50       51       +1     
  Lines        7127     7300     +173     
==========================================
+ Hits         6192     6352     +160     
- Misses        935      948      +13     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@xukai92 xukai92 requested a review from akashgit May 8, 2026 17:52
Covers the argparse definition lines to fix codecov/patch.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown
Owner

@akashgit akashgit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review

Clean, well-scoped PR. Four issues to fix before merging:

1. Redundant local import in cmd_emit (factory/cli.py)

json is already imported at module level (line 7). The local import json as _json is unnecessary — just use json.loads(args.data).

2. No error handling for malformed --data JSON (factory/cli.py)

factory emit agent.started --data "not json" produces a raw json.JSONDecodeError traceback. Wrap in try/except with a friendly error message. Add a test for the invalid-JSON case.

3. Playbook injection makes check_agents_in_sync environment-dependent (factory/agents/plugin.py)

generate_agent_content() now calls load_playbook(), which resolves via ~/.factory/playbooks/<role>.md (user-local) before falling back to factory defaults. This means:

  • check_agents_in_sync() returns different results depending on the developer's local ~/.factory/playbooks/
  • scripts/sync_agents.py produces different output on different machines
  • CI and local dev may diverge

The CLI path injects playbooks at runtime in runner.py. Consider doing the same for the plugin path — inject at invocation time rather than baking into the static generated files. Alternatively, generate_agent_content could explicitly only use factory defaults (skip the user-local tier) so sync output is deterministic.

4. AgentMeta.model is now dead code (factory/agents/plugin.py)

The model field is still read from agents.yml into AgentMeta (line 37) but is no longer used anywhere after removing it from frontmatter generation. Drop it from the dataclass or document why it's retained.

xukai92 and others added 2 commits May 8, 2026 18:50
The Agent tool doesn't go through the factory runner, so events
aren't emitted automatically. The skill now calls factory emit
before/after each agent invocation for observability parity.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove redundant local json import in cmd_emit, use module-level json
- Add error handling for malformed --data JSON with friendly message
- Only inject factory-default playbooks in generate_agent_content()
  (skip user-local ~/.factory/playbooks/) for deterministic sync output
- Document why AgentMeta.model is retained despite not being in frontmatter
- Add test for invalid JSON rejection

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown
Owner

@akashgit akashgit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review — all prior feedback addressed

Issue 1 (redundant import): Fixed — cmd_emit uses the module-level json import.

Issue 2 (JSON error handling): Fixed — try/except with friendly stderr message and return 1. New test test_cmd_emit_rejects_invalid_json covers it.

Issue 3 (environment-dependent sync): Fixed — generate_agent_content() uses DEFAULTS_DIR (factory-shipped playbooks at factory/agents/playbooks/) directly, bypassing load_playbook() which would resolve user-local ~/.factory/playbooks/ first. Sync output is now deterministic across machines.

Issue 4 (AgentMeta.model "dead code"): Not actually dead — meta.model is still validated in test_ceo_uses_opus and test_non_ceo_agents_use_sonnet. The field represents the agents.yml schema; only its emission into frontmatter was removed. Comment documents this clearly.

CI all green (lint + tests on 3.11/3.12/3.13 + codecov). LGTM.

@xukai92 xukai92 merged commit ad298ec into akashgit:main May 9, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Plugin vs CLI agent gaps: observability, prompt enforcement, playbook injection

2 participants