[roadmap] Add sanitized parser fixtures and tests

[latencyong-design]

Goal: keep the parser reliable as Codex session JSONL shapes evolve.

Scope:

• add sanitized fixture files for common response_item shapes;
• cover active-file read behavior;
• cover -Shareable redaction behavior;
• keep fixtures free of real paths, hosts, tokens, or user messages.

Done when contributors can validate parser changes without using private local sessions.

[Necmttn]

A good next step here is to make the fixtures describe parser contracts, not just happy-path example files.

Current repo-grounded notes:

• Export-CodexSessionIndex.ps1 reads active files correctly with FileShare.ReadWrite.
• It currently extracts session_meta via lightweight regex, then only parses response_item rows for messages, function/custom tool calls, and tool outputs.
• It skips reasoning rows and excludes developer messages when -Shareable is set, even if -IncludeDeveloperMessages is passed.
• The existing test covers one session_meta plus user/assistant/developer response_item messages and path redaction for Windows/macOS/Linux profile roots.
• Protect-OutputText() is path-focused today. That is fine if documented, but fixtures should make clear whether -Shareable is expected to redact only local paths or also hostnames/tokens/URLs.

Fixture shape I would add:

tests/fixtures/codex-jsonl/
  01-basic-response-item.jsonl
  02-event-msg-message-shape.jsonl
  03-tool-calls-and-outputs.jsonl
  04-active-file-partial-tail.jsonl
  05-shareable-redaction.jsonl
  06-session-meta-fallbacks.jsonl
  expected/
    01-index.md
    01-rescue.md
    ...

Contracts worth encoding:

1. response_item message content variants. Keep the current array-of-objects shape, but also include string content and nested content fields because Get-TextFromContent() already tries to support nested content.

2. event_msg compatibility. If the intended v1 behavior is "ignore event_msg rows", add a fixture that proves they do not produce duplicate turns. If the intended behavior is "event_msg is the primary source when present", add that as a separate issue and fixture. The important thing is to avoid silently double-counting a message when both event_msg and response_item appear for the same logical turn.

3. Tool call inputs that are objects, not only strings. Current code casts arguments or input to string. Fixtures should include custom_tool_call with input: {"command":"git status"} and a custom_tool_call_output with both benign output and an error-like output. Expected rescue output should prove commands/errors are useful but truncated.

4. Active-file behavior. Include a JSONL fixture whose final line is incomplete/truncated. The script should preserve all valid earlier rows and ignore the partial row without failing the whole export. That is the real-world FileShare.ReadWrite case.

5. turn_context / cwd fallback. The current script gets cwd from session_meta. If a future Codex shape puts cwd in a turn_context row, the fixture should either prove fallback support or prove the output safely leaves cwd empty. Either behavior is fine if explicit.

6. Session identity fallback. Cover missing session_meta.id, missing timestamp, and missing model. The index already falls back to (no id) and file stem in different places; fixtures should lock down that behavior.

7. Shareable privacy boundaries. At minimum, keep the current path redaction and developer-message exclusion tests. I would also add synthetic host/token strings that are obviously fake, for example host.internal.example, ghp_FAKE000000000000000000000000000000000000, and sk-fake000.... Then either assert they are redacted or assert the generated warning says -Shareable is path-only and still requires manual secret review. Ambiguity here is what causes unsafe public issue attachments.

8. Reasoning/lifecycle/token rows. Include reasoning, session_meta, and token/accounting-like rows to prove they do not appear as rescue turns. They may be useful metadata later, but they should not leak into the human-readable transcript by accident.

For the test harness, I would avoid one large end-to-end test becoming a junk drawer. A tiny helper can run the script once per fixture directory and compare generated index/rescue text after normalizing volatile lines like Generated: and Last write:. That gives contributors a safe way to update expected output when the parser intentionally changes.

The ax parser ended up needing exactly this split: raw provider events stay raw, normalized turns are explicit, and public/shareable output is a separate mode with its own redaction contract. Encoding those boundaries in fixtures early will make this kit much easier to keep compatible as Codex JSONL evolves.

---

Generated with ax.

[latencyong-design]

Thanks, this is exactly the parser-contract direction this issue needs.

I agree with splitting the fixtures into three clear layers:

• raw Codex/provider session rows,
• normalized turns generated by the parser,
• public/shareable output with its own redaction contract.

For the current v1 scope, I plan to keep response_item as the primary parsed turn source and add an explicit fixture proving that event_msg rows do not produce duplicate transcript turns. Treating event_msg as a primary source should be a separate issue so the behavior stays reviewable.

On -Shareable, the current guarantee should stay conservative: common local path redaction plus developer-message exclusion, not a complete secret scanner. I like your suggestion to include obviously fake host/token examples. The fixture/test can either prove redaction for supported patterns or prove that the generated warning makes the manual-review boundary explicit.

I will use your fixture list as the acceptance shape for this roadmap item. The docs/examples PR linked to #2 covers the human-facing examples; this issue should stay focused on parser contracts, expected outputs, and regression tests.