Codex/agent targeting llm options#44
Draft
Zbl1007 wants to merge 2 commits into
Draft
Conversation
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds configurable LLM interaction diagnostics (request/response lifecycle + structured parse/repair events) with support for “summary” vs “raw” modes, wired through config/init/doctor flows and covered by new tests.
Changes:
- Introduces
LLMInteractionReporterinfrastructure with buffered/no-op implementations and a factory. - Extends
LLMServiceto emit interaction lifecycle events and adds config surface (showInteraction,interactionMode) across orchestrators. - Adds test coverage for default config behavior, CLI config setters, init prompt defaults, doctor diagnostics, and interaction event emission.
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| test/state-services.test.ts | Asserts default config includes new LLM interaction fields. |
| test/llm.test.ts | Adds interaction lifecycle tests for streaming/non-streaming and structured output repair. |
| test/llm-interaction-reporter.test.ts | New tests for buffered reporter summary/raw output and factory behavior. |
| test/init-orchestrator.test.ts | Verifies init prompt defaults interaction output to false. |
| test/doctor.test.ts | Ensures doctor diagnostics show interaction output state/mode. |
| test/config-orchestrator.test.ts | Tests dotted-key config set + validation for new fields. |
| src/types/config.types.ts | Adds LLMInteractionMode and new LLMConfig fields. |
| src/services/llm.ts | Emits reporter events; threads stage/context through calls; adds custom validation content handling. |
| src/orchestration/provider.ts | Passes interaction reporter + mode into validation initialization. |
| src/orchestration/init.ts | Prompts and persists showInteraction; initializes LLM with reporter/mode. |
| src/orchestration/doctor.ts | Extends LLM config detail with interaction output state/mode. |
| src/orchestration/config.ts | Displays/sets/validates llm.showInteraction + llm.interactionMode. |
| src/orchestration/analyze.ts | Initializes LLM with interaction reporting from config. |
| src/orchestration/agent.ts | Initializes LLM with interaction reporting from config. |
| src/infra/llm-interaction-reporter.ts | New reporter types + buffered implementation + factory. |
| src/infra/index.ts | Re-exports reporter types and factory. |
| src/infra/config.ts | Adds defaults + normalization for new config fields. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+60
to
+62
| expect(output).toContain('Prompt: 1,200 chars'); | ||
| expect(output).toContain('Context: acme/demo'); | ||
| expect(output).toContain('Assistant response received: 5,000 chars'); |
Comment on lines
+445
to
+472
| const interactionEvent = this.createInteractionEvent(input.stage, input.prompt, input.context); | ||
| const content = await this.chat(input.prompt, { | ||
| temperature: input.temperature, | ||
| stage: input.stage, | ||
| context: input.context, | ||
| }); | ||
|
|
||
| try { | ||
| return input.parser(content); | ||
| const parsed = input.parser(content); | ||
| this.emitParseComplete(interactionEvent, parsed); | ||
| return parsed; | ||
| } catch (error) { | ||
| if (!input.repairPrompt) { | ||
| throw error; | ||
| } | ||
|
|
||
| logger.debug('Structured output parsing failed, attempting repair', error); | ||
| this.emitRepairStart({ | ||
| ...interactionEvent, | ||
| error: error instanceof Error ? error.message : String(error), | ||
| }); | ||
| const repairedContent = await this.chat(fillPrompt(input.repairPrompt, { | ||
| invalidResponse: content.slice(0, 12000), | ||
| }), { temperature: 0 }); | ||
| }), { temperature: 0, stage: input.stage, context: input.context }); | ||
|
|
||
| return input.parser(repairedContent); | ||
| const parsed = input.parser(repairedContent); | ||
| this.emitParseComplete(interactionEvent, parsed); | ||
| return parsed; |
Comment on lines
+241
to
+243
| private normalizeInteractionMode(value: unknown): AppConfig['llm']['interactionMode'] { | ||
| return value === 'raw' ? 'raw' : 'summary'; | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.