test(parity): add OpenAI Tier 2 backend + SkillInvocationBackend Protocol (Fixes #368) [stacked on #370]

[mlieberman85]

STACKED ON #370. Do not merge before #370. This branch is based on 028-audit-parity-tests. Until #370 lands, this PR's diff visually includes all of #370's changes plus feature 029. After #370 merges, rebase onto main and the diff collapses to feature 029 only.

Summary

Adds a second Tier 2 provider adapter to feature 028's parity test suite. Introduces a shared SkillInvocationBackend Protocol so future adapters (Gemini, xAI, self-hosted) slot in without touching the shared runner, differ, parser, or artifact writer.

Motivated by issue #368: users of darnit through OpenAI-backed coding assistants (Cursor with GPT-4, ChatGPT-in-IDE, custom OpenAI agents) deserve the same drift-detection surface feature 028 gave Claude Code users. Without this, a silent WARN-to-PASS reclassification by an OpenAI-based agent would go undetected while the Claude path stays honest.

Detailed rationale in the commit body.

What ships

• tests/darnit/parity/tier2/backends/ -- new package: Protocol (base.py), Claude adapter refactored from feature 028 (claude_agent_sdk.py), OpenAI Chat Completions adapter (openai_backend.py), test-only NoopBackend, BACKEND_REGISTRY
• tests/darnit/parity/tier2/claude_agent_sdk_client.py -- backwards-compat shim; feature 028's tests keep working unchanged
• Runner extensions: --backend, --model, --max-turns; new turn_cap_exhausted outcome (exit code 5); main(argv, backends=) for test-side backend injection
• .github/workflows/parity-tier2-openai.yml -- manual dispatch, environment: parity-tier2-openai gated with required reviewers, OPENAI_API_KEY at Environment scope only, pinned gpt-4o-2024-08-06 default
• Zero product package changes -- inherited from feature 028's guard

Constitution + governance notes

• II (Conservative-by-Default) reinforced across a second provider: strengthens the "WARN counts as FAIL" invariant against OpenAI-side silent reinterpretation.
• Two-Environment governance: parity-tier2 (Claude, test(parity): two-tier audit parity test suite (Fixes #366) [stacked on #365] #370) and parity-tier2-openai (OpenAI, this PR) each hold their own reviewer list and secret. A reviewer authorized for OpenAI spend need not be trusted for Anthropic spend, and vice versa. Enforced at the GitHub Actions Environment layer + mechanically verified by a workflow-config test that iterates every .github/workflows/*.yml and asserts key exclusivity.
• SC-010 pinned versioned model: a regex-based workflow-config test rejects moving aliases (gpt-4o alone) as the default. Bumping the pin requires a PR edit -- explicit, reviewable, correlatable with any subsequent test-result changes. Reproducibility is load-bearing for a diagnostic.

Load-bearing safety properties (test-enforced)

• SC-005 Protocol conformance: test_backend_protocol_conformance.py parametrizes over BACKEND_REGISTRY + NoopBackend; all pass isinstance check.
• SC-007 extensibility: test_backend_extensibility.py injects an inline backend via main(backends=...) and confirms invocation without any edit to shared modules.
• SC-011 turn_cap_exhausted: adversarial test with a mocked SDK that always returns tool_calls (never text); asserts backend returns turn_cap_exhausted=True and runner exits 5.
• B-17 malicious local_path: adversarial test proves _dispatch_tool_call overrides the model's local_path argument with the fixture directory. A rogue model cannot make the audit wander outside the fixture.
• SC-002 key exclusivity: pure-Python file iteration verifies OPENAI_API_KEY appears only in parity-tier2-openai.yml.
• OW-8/OW-9 cross-contamination guard: Claude workflow has no OPENAI_API_KEY, OpenAI workflow has no ANTHROPIC_API_KEY. Two explicit tests.
• FR-14 (MC2 shared parser): an OpenAI-shaped Markdown response is parseable by feature 028's shared SkillReport.parse(). Guards against silent parser fork.

Test plan

• uv run pytest tests/ -q -- expect 2621 passed, 15 skipped, 0 failures
• uv run pytest tests/darnit/parity/tier2/ -q -- expect 53 passed
• uv run ruff check . -- clean
• Manual dry-run: uv run python -m tests.darnit.parity.tier2.run --backend openai --fixture-glob 'all_pass_repo' --dry-run --artifact-dir /tmp/parity-openai-dryrun -- writes openai_final_message.md, mcp_tool_result.json, diff_report.md, metadata.json
• Manual grep: grep -r 'OPENAI_API_KEY' .github/workflows/ -- returns matches ONLY in parity-tier2-openai.yml
• Manual (post-merge, in the parity-tier2-openai Environment): dispatch the workflow with default fixture_glob='*' and default model='gpt-4o-2024-08-06'; verify the reviewer-approval gate fires; run to completion; download parity-artifacts-openai bundle

Before-merge maintainer actions (manual GitHub UI)

Automated tests verify the workflow YAML shape but cannot verify the Environment's UI-side configuration. Before dispatching Tier 2 OpenAI in production:

• Create GitHub Environment parity-tier2-openai under Settings -> Environments in darnitdevorg/darnit
• Add required reviewers (authorized maintainers only)
• Add OPENAI_API_KEY at the ENVIRONMENT level (NOT repo level)
• Confirm no other Environment on the repo also holds OPENAI_API_KEY

Reviewer checklist for the governance-critical YAML

• on: block contains only workflow_dispatch
• Job declares environment: parity-tier2-openai
• permissions.contents: read and no write scopes anywhere
• OPENAI_API_KEY referenced only in the SDK-invocation step's env: block, sourced from secrets.OPENAI_API_KEY (Environment secret)
• Preflight audit step logs actor + SHA + fixture_glob + model BEFORE the SDK step
• Artifact upload step has if: always()
• Model input default matches the version-suffixed pattern (gpt-4o-YYYY-MM-DD or equivalent)
• No api_key or similar workflow input

Follow-ups

• Parity Tier 2: add scheduled cadence + governance-appropriate API-key sourcing #369 (scheduled cadence + governance-appropriate key sourcing) -- now applies to BOTH the Claude and OpenAI workflows; a single follow-up covers both
• Future features for additional providers (Gemini, xAI, self-hosted) each reuse the Protocol without touching feature 029's code

Rebase plan

• When test(parity): two-tier audit parity test suite (Fixes #366) [stacked on #365] #370 merges: git fetch upstream && git rebase upstream/main on this branch; git push --force-with-lease. Diff collapses to feature 029 only.

[kusari-inspector]

Kusari Analysis Results:

✅ No Flagged Issues Detected
All values appear to be within acceptable risk parameters.

Both independent analyses return clean results. The dependency analysis found no pinned version dependency changes, presenting no dependency-related risk. The code analysis flagged a low-confidence, medium-severity observation about missing persist-credentials: false on actions/checkout steps in two workflow files (parity-tier2-openai.yml and parity-tier2.yml), but confirmed it is not exploitable given the following compensating controls: (1) workflows are triggered exclusively via workflow_dispatch with no untrusted fork or PR trigger; (2) both workflows are gated behind named GitHub Environments requiring human reviewer approval before any secrets are exposed; (3) all actions are pinned to full 40-character commit SHAs, eliminating supply-chain tag-repointing risk; (4) permissions are scoped to contents: read only; (5) the artifact upload path (parity-artifacts/) does not include the .git directory where a persisted credential would reside, making the artipacked attack vector non-applicable. No injection vulnerabilities, secret leaks, or other code issues were identified. Adding persist-credentials: false remains a best-practice hardening improvement and is recommended as a low-priority follow-up, but does not represent a meaningful security gap given the controls in place.

Note

View full detailed analysis result for more information on the output and the checks that were run.

---

@kusari-inspector rerun - Trigger a re-analysis of this PR
@kusari-inspector feedback [your message] - Send feedback to our AI and team
See Kusari's documentation for setup and configuration.
Commit: f007869, performed at: 2026-08-12T17:57:56Z

Found this helpful? Give it a 👍 or 👎 reaction!

[kusari-inspector]

Issue: Move all github context and workflow inputs to env: variables before using them in the run: block to prevent shell injection. Replace direct ${{ }} interpolation with shell environment variable references.

Recommended Code Changes:

- name: Preflight audit log (OW-10)
  env:
    ACTOR: ${{ github.actor }}
    SHA: ${{ github.sha }}
    FIXTURE_GLOB: ${{ inputs.fixture_glob }}
    MODEL: ${{ inputs.model }}
  run: |
    {
      echo "## Tier 2 (OpenAI) Preflight"
      echo ""
      echo "- actor: $ACTOR"
      echo "- sha: $SHA"
      echo "- fixture_glob: $FIXTURE_GLOB"
      echo "- model: $MODEL"
      echo "- timestamp: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
    } >> "$GITHUB_STEP_SUMMARY"

[kusari-inspector]

Issue: The run: block at this step has OPENAI_API_KEY in scope and directly interpolates inputs.model and inputs.fixture_glob. Move inputs to env: variables to prevent injection-based secret exfiltration.

Recommended Code Changes:

- name: Run Tier 2 parity check (OpenAI)
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
    MODEL: ${{ inputs.model }}
    FIXTURE_GLOB: ${{ inputs.fixture_glob }}
  run: |
    uv run python -m tests.darnit.parity.tier2.run \
      --backend openai \
      --model "$MODEL" \
      --fixture-glob "$FIXTURE_GLOB"

[kusari-inspector]

Issue: Move all github context and workflow inputs to env: variables before using them in the run: block to prevent shell injection.

Recommended Code Changes:

- name: Preflight audit log (T2-7/T2-8)
  env:
    ACTOR: ${{ github.actor }}
    SHA: ${{ github.sha }}
    FIXTURE_GLOB: ${{ inputs.fixture_glob }}
  run: |
    {
      echo "## Tier 2 Preflight"
      echo ""
      echo "- actor: $ACTOR"
      echo "- sha: $SHA"
      echo "- fixture_glob: $FIXTURE_GLOB"
      echo "- timestamp: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
    } >> "$GITHUB_STEP_SUMMARY"

[kusari-inspector]

Issue: The run: block at this step has ANTHROPIC_API_KEY in scope and directly interpolates inputs.fixture_glob. Move the input to an env: variable to prevent injection-based secret exfiltration.

Recommended Code Changes:

- name: Run Tier 2 parity check
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
    FIXTURE_GLOB: ${{ inputs.fixture_glob }}
  run: |
    uv run python -m tests.darnit.parity.tier2.run \
      --fixture-glob "$FIXTURE_GLOB"

[kusari-inspector]

All action references in this workflow use mutable version tags (actions/checkout@v4, actions/setup-python@v5, astral-sh/setup-uv@v6, actions/upload-artifact@v4). Pin each to a full 40-character commit SHA to prevent supply-chain attacks if an action owner's account is compromised and the tag is repointed.

[kusari-inspector]

All action references in this workflow use mutable version tags (actions/checkout@v4, actions/setup-python@v5, astral-sh/setup-uv@v6, actions/upload-artifact@v4). Pin each to a full 40-character commit SHA to prevent supply-chain attacks if an action owner's account is compromised and the tag is repointed.

[mlieberman85]

Added T022 (US3 aggregate cross-provider diff script) as a follow-on commit c09bfea. Local maintainer tool at tests/darnit/parity/tier2/scripts/aggregate_provider_diff.py; 10 unit tests. Completes US3, so PR now covers all three user stories from the spec.

[kusari-inspector]

Kusari PR Analysis rerun based on - c09bfea performed at: 2026-08-11T20:24:48Z - link to updated analysis

[pxp928]

@kusari-inspector rerun

[kusari-inspector]

🔄 Run triggered at 15:30:26 UTC. Starting fresh analysis...

[kusari-inspector]

Kusari PR Analysis rerun based on - c09bfea performed at: 2026-08-12T15:32:30Z - link to updated analysis

[pxp928]

@kusari-inspector rerun

[kusari-inspector]

🔄 Run triggered at 16:14:59 UTC. Starting fresh analysis...

[kusari-inspector]

Kusari PR Analysis rerun based on - c09bfea performed at: 2026-08-12T16:16:29Z - link to updated analysis

[kusari-inspector]

Kusari PR Analysis rerun based on - f007869 performed at: 2026-08-12T17:58:30Z - link to updated analysis

[pxp928]

Approving on the condition that the findings below are addressed before merge. The SkillInvocationBackend Protocol is a clean extraction and the OpenAI backend is a reasonable implementation of it — this is the most straightforward PR of the four. But it currently ships two red tests, inherited from the fixture-discovery bug in #370, so it can't merge before that one is fixed.

Reviewed the incremental diff pr-370..pr-371 (commits a57a34b, a2a87b7, f007869). I ran the new tests and ruff locally. Ruff and format are clean.

---

Blocker — ships red

test_backend_extensibility.py:52 — both SC-007 tests fail. Verified: 2 failed, 1 passed.

run._discover_fixtures requires .baseline.toml in each fixture dir, but that filename is in .gitignore:32, so no fixture is ever discovered and main() returns 3 instead of exercising the injected backend. The third test passes vacuously for the same reason — exit 3 either way, so it asserts nothing.

The discovery filter itself predates this PR (feature 028 / #370), but these tests depend on it, so this PR can't go in until #370's fixture markers are committed.

---

Should fix

• test_openai_backend_adversarial.py:116 — bare import openai hard-fails with ModuleNotFoundError when the dev extra isn't synced (reproduced). Needs pytest.importorskip. Separately, line 118 does openai.AsyncOpenAI = lambda: mock_client and never restores it, leaking a broken stub into the rest of the pytest session — use monkeypatch.setattr.
• backends/openai_backend.py:89 — _dispatch_tool_call pins only local_path; level and output_format are setdefault-ed. A model that chooses level: 1 makes the OpenAI-side audit narrower than _run_mcp_tool's level 3, producing a spurious counts_disagree / exit 1 — the harness reports a parity failure that is really a parameter mismatch. Unknown hallucinated keys reach audit_openssf_baseline(**args) → TypeError → the whole run exits 3. Pin all comparison-relevant params and drop unrecognised keys.
• test_openai_backend_adversarial.py (and the rest of the new tests) — none of the ~40 new tests carry a unit or integration marker. CI runs only -m unit / -m integration, so pytest tests/darnit/parity/ -m unit gives 95 deselected / 0 selected. The SC-002 key-exclusivity and SC-010 pinned-model governance tests never run — which is exactly the kind of check you want running unattended.
• scripts/aggregate_provider_diff.py:126 — a present-but-unparseable provider message yields an empty control map; all rows show - and the section reports "0 disagreements" with no warning. Cross-provider drift gets silently reported as agreement. An unparseable message should be a loud error, not an empty map.
• artifact_writer.py:38 — only the final-message file is provider-namespaced. diff_report.md, metadata.json, and mcp_tool_result.json clobber each other across providers, contradicting both the docstring and the aggregate script's documented single-root mode.
• diff.py:133 — diff reports hardcode skill_final_message.md, a filename that doesn't exist in non-Claude artifact bundles.

---

Nice touch on f007869 pinning the workflow actions to SHAs and moving context expressions into env: vars — that's the right instinct for a workflow that handles provider API keys.