Parity: OpenAI (and other provider) skill-drift checks alongside the Claude Agent SDK version

[mlieberman85]

Context

Feature 028 (spec at specs/028-audit-parity-tests/spec.md, PR TBD) ships a two-tier parity test suite:

• Tier 1: mechanical MCP-tool-vs-harness parity (no API, every PR).
• Tier 2: coding-agent skill vs raw tool output, invoked via the Claude Agent SDK.

Tier 2 is intentionally provider-specific in the MVP -- the darnit product surface is invoked by real users through Claude Code today, so measuring drift against that specific consumer is where the diagnostic value lives.

What this issue asks for

Parallel Tier-2-style parity checks for OTHER provider agent SDKs -- OpenAI's assistants SDK, and whatever future SDK becomes relevant (Gemini, Anthropic non-Claude-Code paths, self-hosted coding agents). The shape is the same as the Claude Agent SDK path in feature 028: invoke a skill-like scripted agent, capture its final assistant message, diff against the raw MCP tool output for the same fixture. Different SDK, same fixture corpus, same parser strategy.

Why not bundle with feature 028

• Each SDK has its own dep, install path, and prompt-shape quirks. Bundling multiplies scope without material MVP benefit.
• The /darnit-audit skill is Claude-Code-specific today (it's a Claude Code skill). Reproducing its behavior on another provider requires porting the skill or using an OpenAI-equivalent -- design work of its own.
• The governance concern that made feature 028 manual-only (API-key ownership under neutral governance) applies here too but each provider has different secret-management shapes.

Scope for a follow-up feature (or features)

• OpenAI Agents / Assistants SDK: invoke a skill equivalent, diff against raw tool output. Test-only dep on the OpenAI SDK.
• Same access-control model as feature 028's Tier 2 (manual dispatch, Environment gate, no key exposure to other workflows).
• Same fixture corpus. New fixtures if a provider is expected to expose corner cases the current corpus misses.
• A "provider matrix" report so a maintainer can see all provider drifts side by side.
• Same one-drift-class-fails-CI semantic: any per-control status difference between the provider's summary and the raw tool output is a hard failure.

Related

• Feature 028 spec: specs/028-audit-parity-tests/spec.md (Q1 clarification on 2026-08-09).
• Original discovery: PR feat: RFC-0001 Stage 1 authority + darnit harness fleet driver #365 review noting the Claude-Code /darnit-audit skill silently reclassifying WARN as PASS.

[mlieberman85]

PR #371 opened to close this issue. Stacked on #370 (feature 028's parity test suite). When both merge, this issue closes automatically.