fix: split agi-reasoning into hidden-CoT and visible-reasoning agents#185
Conversation
…eating visible-reasoning agent Agent-Logs-Url: https://github.com/Bryan-Roe/Aria/sessions/5192c609-b7f9-4805-9fff-b3e278367040 Co-authored-by: Bryan-Roe <74067792+Bryan-Roe@users.noreply.github.com>
github-actions
Bot
added
the
documentation
🔴 Coverage —
|
| Metric | Value |
|---|---|
| Total coverage | 59.7% |
→ vs main |
0.0% |
| Minimum threshold | 60% |
Updated on every push · 2026-05-09
|
Semgrep found 1 Possibly found usage of AI: OpenAI |
![bryan-roe-bot[bot]](https://avatars.githubusercontent.com/in/1073263?s=60&v=4){kind=small}
|
|
||
| ## Workspace Context | ||
|
|
||
| - **Provider chain**: Azure OpenAI → OpenAI → LMStudio → LoRA → Local |
There was a problem hiding this comment.
Possibly found usage of AI: OpenAI
🧹 Fixed in commit 7f5acb1 🧹
Reviewer's GuideIntroduces a new Flow diagram for selecting agi-reasoning vs visible-reasoning agentflowchart TD
start([User request])
intent{User wants visible step-by-step<br/>reasoning or CoT?}
start --> intent
intent -->|Yes, show reasoning| visible_agent
intent -->|No, concise final answer| agi_agent
visible_agent["Use visible-reasoning.agent.md<br/>via reason.prompt.md"]
agi_agent["Use agi-reasoning.agent.md<br/>via agi.prompt.md or other autonomous flows"]
visible_agent --> visible_output["Response includes explicit chain-of-thought,<br/>classification, decomposition, reflection table"]
agi_agent --> agi_output["Response hides chain-of-thought,<br/>only final answer shown"]
File-Level Changes
Assessment against linked issues
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've left some high level feedback:
- The visible-reasoning and agi-reasoning agent specs now duplicate the “difference between agents” section; consider centralizing this comparison (e.g., in copilot-instructions.md only) and keeping the agent files narrowly focused on behavior to avoid divergence over time.
- The visible-reasoning agent enforces fully visible chain-of-thought for every non-trivial request; if you anticipate using this in contexts with stricter CoT redaction requirements, consider adding a short preamble about when it should not expose reasoning (e.g., sensitive or policy-limited scenarios).
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- The visible-reasoning and agi-reasoning agent specs now duplicate the “difference between agents” section; consider centralizing this comparison (e.g., in copilot-instructions.md only) and keeping the agent files narrowly focused on behavior to avoid divergence over time.
- The visible-reasoning agent enforces fully visible chain-of-thought for every non-trivial request; if you anticipate using this in contexts with stricter CoT redaction requirements, consider adding a short preamble about when it should not expose reasoning (e.g., sensitive or policy-limited scenarios).Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
There was a problem hiding this comment.
Pull request overview
Aligns the repo’s “Reason” prompt with the intended reasoning-visibility behavior by introducing a dedicated visible-reasoning agent and updating documentation cross-references so the correct agent is selected for step-by-step explanations.
Changes:
- Added a new
.github/agents/visible-reasoning.agent.mdthat explicitly outputs step-by-step reasoning. - Rewired
.github/prompts/reason.prompt.mdto useagent: visible-reasoningand updated its description to match. - Updated
.github/agents/agi-reasoning.agent.mdand.github/copilot-instructions.mdto clarify the distinction between hidden vs visible reasoning agents.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
.github/prompts/reason.prompt.md |
Switches the Reason prompt to route to visible-reasoning and updates the prompt description accordingly. |
.github/copilot-instructions.md |
Documents the split between hidden-CoT and visible-CoT agents and updates the prompts list entry for reason.prompt.md. |
.github/agents/visible-reasoning.agent.md |
Introduces the new visible reasoning agent definition and output framework. |
.github/agents/agi-reasoning.agent.md |
Adds a cross-reference section clarifying when to use visible-reasoning instead. |
| - **Provider chain**: Azure OpenAI → OpenAI → LMStudio → LoRA → Local | ||
| - **Config precedence**: YAML base < CLI flags < per-job YAML < env vars | ||
| - **Data immutability**: Read-only `datasets/`, write-only `data_out/` | ||
| - **Testing**: `python scripts/test_runner.py --unit` before committing | ||
| - **Safety**: `--dry-run` all orchestrators before execution |
There was a problem hiding this comment.
The “Provider chain” listed here (Azure OpenAI → OpenAI → LMStudio → LoRA → Local) doesn’t match the actual provider auto-detection priority used by detect_provider() (LM Studio → Ollama → AGI/Quantum → Azure → OpenAI → Local, with LoRA only when explicitly selected). Please update this bullet to reflect the real order so the agent doesn’t give users incorrect operational guidance.
| You are a structured reasoning agent whose primary purpose is to **show your reasoning process transparently** to the user. Every reasoning step must appear in the response — chain-of-thought is visible, not hidden. | ||
|
|
||
| ## Core Principle | ||
|
|
||
| Unlike autonomous AGI agents that internalize reasoning, this agent **always shows its work**. Each analysis step, assumption, confidence score, and verification check must appear in the final response so the user can follow, critique, and learn from the reasoning process. |
There was a problem hiding this comment.
This agent instructs that “every reasoning step must appear in the response” and to show intermediate reasoning/results. Without an explicit redaction rule, this can cause accidental leakage of secrets (API keys, tokens), private prompt text, or sensitive tool outputs when reasoning about repository configuration. Consider adding a constraint to keep reasoning high-level when needed and to never include secrets or internal/system prompt content verbatim.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6350a103bd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| name: "Reason" | ||
| argument-hint: "Problem or question to analyze (example: decision + relevant context + constraints or trade-offs)" | ||
| agent: agi-reasoning | ||
| agent: visible-reasoning |
There was a problem hiding this comment.
/reason workflows
Switching reason.prompt.md to visible-reasoning removes notebook execution/read capabilities that were available through agi-reasoning (execute/runNotebookCell, read/getNotebookSummary, read/readNotebookCellOutput are not in the new agent’s tools list). As a result, /reason requests that need to inspect or run .ipynb cells can no longer complete end-to-end, which is a functional regression from the previous routing.
Useful? React with 👍 / 👎.
|
@copilot resolve the merge conflicts in this pull request |
- Merge origin/main into branch (visible-reasoning.agent.md, reason.prompt.md, copilot-instructions.md) - Fix provider chain order to match detect_provider() priority: LM Studio → Ollama → AGI → Quantum → Azure → OpenAI → Local - Add Security Constraints section to visible-reasoning.agent.md to prevent accidental leakage of secrets/internal prompts in visible reasoning output Co-authored-by: Bryan-Roe <74067792+Bryan-Roe@users.noreply.github.com>
Agent-Logs-Url: https://github.com/Bryan-Roe/Aria/sessions/92a26e93-0503-42e4-853d-2ccf4e7a5a20 Co-authored-by: Bryan-Roe <74067792+Bryan-Roe@users.noreply.github.com>
Merge conflicts resolved in commit
|
Signed-off-by: Bryan <74067792+Bryan-Roe@users.noreply.github.com>
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
| - Note any remaining uncertainties | ||
|
|
||
| **Codebase context:** | ||
| - **Provider chain**: LM Studio → Ollama → AGI → Quantum → Azure → OpenAI → Local (LoRA only when `provider=lora`) |
There was a problem hiding this comment.
Semgrep identified an issue in your code:
Possibly found usage of AI: OpenAI
To resolve this comment:
🔧 No guidance has been designated for this issue. Fix according to your organization's approved methods.
💬 Ignore this finding
Reply with Semgrep commands to ignore this finding.
/fp <comment>for false positive/ar <comment>for acceptable risk/other <comment>for all other reasons
Alternatively, triage in Semgrep AppSec Platform to ignore the finding created by detect-generic-ai-oai.
You can view more details about this finding in the Semgrep AppSec Platform.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fc61153545
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| ## Workspace Context | ||
|
|
||
| - **Provider chain**: Azure OpenAI → OpenAI → LMStudio → LoRA → Local | ||
| - **Provider chain**: LM Studio → Ollama → AGI → Quantum → Azure → OpenAI → Local (LoRA only when `provider=lora`) |
There was a problem hiding this comment.
The updated provider chain states AGI and Quantum are checked before Azure/OpenAI, but detect_provider() only enters those branches when the user explicitly sets --provider agi or --provider quantum (ai-projects/chat-cli/src/chat_providers.py around lines 1370-1404), and auto mode skips directly from LM Studio/Ollama to Azure/OpenAI/local (1498-1549). This mismatch can cause the visible reasoning agent to give incorrect troubleshooting or routing advice for default /reason runs because it describes an auto-fallback order that does not exist.
Useful? React with 👍 / 👎.
reason.prompt.mdpromised visible chain-of-thought output but was wired toagi-reasoning, which explicitly suppresses CoT in responses. This caused incorrect tool routing and misleading documentation.Changes
New agent:
visible-reasoning.github/agents/visible-reasoning.agent.md— agent whose outputs always expose reasoning steps (classify → decompose → execute → reflect → synthesize), with confidence scores and a self-check table shown in the responseUpdated prompt wiring
.github/prompts/reason.prompt.md—agent: agi-reasoning→agent: visible-reasoning; description updated to state CoT is visibleCross-references added
.github/agents/agi-reasoning.agent.md— appended "Difference fromvisible-reasoning" section so users know when to use which agent.github/copilot-instructions.md— split the singleagi-reasoningtable row into two distinct entries; updatedreason.prompt.mddescription in the prompts listagi-reasoningvisible-reasoningSummary by Sourcery
Introduce a dedicated visible reasoning agent and update prompts and docs to distinguish it from the hidden chain-of-thought AGI reasoning agent.
New Features:
visible-reasoningagent that exposes step-by-step chain-of-thought reasoning in responses.Enhancements:
reason.prompt.mdto use the newvisible-reasoningagent and clarify that it shows explicit chain-of-thought in outputs.agi-reasoningandvisible-reasoningin the AGI agent spec.Documentation:
agi-reasoningandvisible-reasoningagents with clarified CoT visibility and adjust thereason.prompt.mddescription accordingly.