Skip to content

feat(codex): implement F103 NDJSON output parity for Codex provider#364

Merged
pocky merged 1 commit into
mainfrom
feature/F103-codex-provider-jsonl-output-parity
Jun 4, 2026
Merged

feat(codex): implement F103 NDJSON output parity for Codex provider#364
pocky merged 1 commit into
mainfrom
feature/F103-codex-provider-jsonl-output-parity

Conversation

@pocky
Copy link
Copy Markdown
Contributor

@pocky pocky commented Jun 3, 2026

Summary

  • Closes the F103 output parity gap for Codex: state.Output and ConversationResult.Output now contain clean aggregated assistant text extracted from NDJSON streams instead of raw JSON envelopes, regardless of output_format value — matching the behavior Claude, Gemini, and OpenCode already had since 0.7.1.
  • Introduces presence-aware overwrite logic: An empty assistant_message event (hadText=true, text="") is correctly distinguished from non-NDJSON plain-text fallback (hadText=false), preventing inflated token estimates and stale turn Content when the model responds with empty output.
  • Populates state.Response for Codex via tryParseJSONResponse on the extracted assistant text, giving Codex the same automatic structured-output heuristic already present in other providers and enabling {{.states.step.Response.field}} in downstream workflow steps.
  • Docs updated to reflect that state.Output stores extracted text (not raw NDJSON) for all CLI providers and that state.Response is now populated heuristically across all six providers.

Changes

Infrastructure — Codex Provider

  • internal/infrastructure/agents/codex_provider.go: Wire extractCodexTextContent hook into cliProviderHooks; add extractCodexAssistantText (presence-aware NDJSON aggregator returning (string, bool)); add extractCodexTextContent adapter for the base hook signature; override Execute and ExecuteConversation with F103 presence-aware overwrite of Output, Response, token counts, and trailing turn Content; simplify NewCodexProvider to delegate to NewCodexProviderWithOptions; document NUL-byte preservation divergence from Claude provider.
  • internal/infrastructure/agents/helpers.go: Migrate findFirstNDJSONEvent and findLastNDJSONEvent from strings.Split to strings.SplitSeq (allocation-free iterator).

Tests

  • internal/infrastructure/agents/codex_provider_unit_test.go: Add ~450 lines of new unit tests covering all F103 paths: presence-aware overwrite, output_format-invariant extraction, Response population, empty assistant_message, multi-event aggregation, NUL byte preservation, non-JSON plain text, and token recount; remove superseded timestamp/provider-name duplicate tests and stub-era expectedTokens hardcoding.
  • internal/infrastructure/agents/codex_provider_delegation_test.go: Remove all stub-era conditional if err != nil { return } guards; harden assertions with require.NoError; add TestCodexProvider_ExecuteConversation_OutputExtraction, HookWired, PlainTextFallback, and NDJSONOutput delegation tests; assert errors.Is(err, context.Canceled) on cancellation paths; remove no-op TestCodexProvider_Validate_BinaryNotFound.
  • internal/infrastructure/agents/codex_provider_test_helpers_test.go: New file — shared ndjsonLine helper that JSON-escapes arbitrary text into a valid Codex item.completed NDJSON envelope.

Documentation

  • docs/reference/interpolation.md: Expand Response section to cover agent steps with examples showing {{.states.step.Response.field}}; clarify JSON vs Response semantics and when each is populated.
  • docs/user-guide/agent-steps.md: Update state.Output description to reflect extracted-text semantics for CLI providers; add F103 Output Parity note to the display-filtering table; correct "raw NDJSON" references in silent-mode note.
  • README.md: Update output-formatting feature bullet to mention automatic state.Response population.
  • CHANGELOG.md: Add [Unreleased] entry for F103 with full parity description.

ZPM Knowledge Base

  • .zpm/kb/pr_feature_f103_codex_provider_jsonl_output_parity/journal.wal: PR tracking journal.
  • .zpm/kb/pr_feature_f103_codex_provider_jsonl_output_parity/knowledge.pl: PR Prolog schema for blocking-issue tracking.

Test plan

  • Run make test — all unit and integration tests pass with zero failures and no data races (make test-race)
  • Run make lint — zero violations; verify nolint:errcheck directives carry explanatory comments
  • Execute a Codex agent step in a workflow that pipes {{.states.codex_step.Output}} into a downstream step and confirm clean text (not raw NDJSON) is interpolated
  • Execute a Codex agent step with a prompt that returns JSON and verify {{.states.codex_step.Response.field}} resolves correctly without requiring output_format: json

Closes #363


Generated with awf commit workflow

@pocky pocky force-pushed the feature/F103-codex-provider-jsonl-output-parity branch from b88543f to 238087f Compare June 4, 2026 09:24
- `.zpm/kb/pr_feature_f103_codex_provider_jsonl_output_parity/journal.wal`: Add ZPM PR tracking journal
- `.zpm/kb/pr_feature_f103_codex_provider_jsonl_output_parity/knowledge.pl`: Add ZPM PR knowledge base schema
- `CHANGELOG.md`: Document F103 presence-aware output extraction and Response population
- `README.md`: Update output formatting feature description to include Response auto-population
- `docs/reference/interpolation.md`: Expand Response section with agent step examples and JSON/Response comparison
- `docs/user-guide/agent-steps.md`: Update Output/Response field descriptions and output_format table for F103
- `internal/infrastructure/agents/codex_provider.go`: Add extractCodexAssistantText/extractCodexTextContent helpers; wire extractTextContent hook; implement presence-aware Output overwrite and Response population in Execute and ExecuteConversation
- `internal/infrastructure/agents/codex_provider_delegation_test.go`: Remove stub-era conditionals; add OutputExtraction, HookWired, PlainTextFallback, NDJSONOutput delegation tests
- `internal/infrastructure/agents/codex_provider_test_helpers_test.go`: Add ndjsonLine test helper
- `internal/infrastructure/agents/codex_provider_unit_test.go`: Expand unit test coverage for F103 extraction paths; remove duplicated ProviderName/TimestampOrdering tests
- `internal/infrastructure/agents/helpers.go`: Remove stale comment from appendCodexOptions

Closes #363
@pocky pocky force-pushed the feature/F103-codex-provider-jsonl-output-parity branch from 238087f to e6de4ad Compare June 4, 2026 09:41
@pocky pocky marked this pull request as ready for review June 4, 2026 10:05
@pocky pocky merged commit 9014ee7 into main Jun 4, 2026
5 checks passed
@pocky pocky deleted the feature/F103-codex-provider-jsonl-output-parity branch June 4, 2026 10:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

F103: Codex provider JSONL output parity

1 participant