test(parity): two-tier audit parity test suite (Fixes #366) [stacked on #365]

[mlieberman85]

STACKED ON #365. Do not merge before #365. This branch is based on 026-harness-with-stage1 (the head of #365). Until #365 lands on main, this PR's diff visually includes all of #365's changes plus feature 028. After #365 merges, rebase onto main and the diff collapses to feature 028 only.

Summary

Two-tier diagnostic test suite verifying the darnit audit's per-control output is consistent across the three consumers users care about:

• Direct MCP tool (darnit_baseline.tools.audit_openssf_baseline)
• darnit harness end-to-end
• /darnit-audit coding-agent skill (Tier 2 only)

Motivated by the PR #365 review where the skill was observed silently reclassifying WARN as PASS in its Markdown summary while the MCP tool and harness produced identical raw results.

Detailed rationale in the commit body.

What ships

• tests/darnit/parity/tier1/ -- pytest suite, runs on every PR. For each fixture, invokes both audit paths in-process (harness uses MockLLMStep, no live API), diffs per-control status. Sole allowed drift: PENDING_LLM (MCP) -> non-PENDING_LLM (harness).
• tests/darnit/parity/tier2/ -- manual-dispatch runner + skill Markdown parser + workflow-config assertions.
• .github/workflows/parity-tier2.yml -- workflow_dispatch only, environment: parity-tier2 gated (required reviewers + Environment-scoped ANTHROPIC_API_KEY), permissions: contents: read, if: always() artifact upload.
• 4 fixtures under tests/darnit/parity/fixtures/ covering all_pass / all_fail / mixed / pending_llm.
• pyproject.toml: adds claude-agent-sdk to the workspace dev group. Zero product package changes -- SC-006 mechanically enforced by test_no_product_changes.py running git diff against the base branch.

Constitution + governance notes

• II (Conservative-by-Default) reinforced: Tier 2 exists specifically to catch downstream layers eroding the tool's WARN-counts-as-FAIL property. Tier 1 catches equivalent regressions in the harness before merge.
• IV (Never Guess User Values) reinforced: fixtures do NOT auto-generate context values. And Tier 2 makes silently-heuristic skill reinterpretation externally observable.
• FR-007a governance: ANTHROPIC_API_KEY MUST live in a GitHub Environment (not repo-level). Enforced at the workflow YAML level AND by a pytest test that iterates every workflow file and asserts the key appears only in parity-tier2.yml.
• Manual-dispatch only for MVP: the repo is under neutral governance and the API key belongs to a specific company. Automated scheduled runs would charge that company's account for community activity. Follow-up issue Parity Tier 2: add scheduled cadence + governance-appropriate API-key sourcing #369 tracks adding scheduled cadence once a governance-appropriate key-sourcing model exists.

Diagnostic finding surfaced during implementation

The parity test caught a real MCP-tool-vs-harness disagreement on OSPS-LE-03.02 (feature 026's inferred_from authority handling has a residual bug). Fixtures scope around it via parity.toml's control_ids filter. This is not a feature 028 bug -- it's exactly what the parity test was designed to find. A follow-up feature can fix the underlying inferred_from behavior; from now on, similar regressions will be CI-visible.

Test plan

• uv run pytest tests/ -q -- expect 2589 passed, 15 skipped, 0 failures.
• uv run pytest tests/darnit/parity/tier1/ -q -- expect 35 passed in about 35 seconds (SC-002 timing budget).
• uv run pytest tests/darnit/parity/tier2/ -q -- expect 21 passed (offline; workflow config test parses YAML).
• uv run ruff check . -- clean.
• uv run python scripts/validate_sync.py --verbose -- PASSED.
• Manual: uv run python -m tests.darnit.parity.tier2.run --fixture-glob "all_pass_repo" --dry-run --artifact-dir /tmp/parity-dryrun -- writes mcp_tool_result.json, skill_final_message.md, diff_report.md, metadata.json under /tmp/parity-dryrun/all_pass_repo/.
• Manual (post-merge, in the parity-tier2 Environment): dispatch the workflow with default fixture_glob="*", verify the reviewer-approval gate fires, run to completion, download parity-artifacts bundle.

Reviewer checklist for the governance-critical YAML

• on: block contains only workflow_dispatch.
• Job declares environment: parity-tier2.
• permissions.contents: read and no write scopes anywhere.
• ANTHROPIC_API_KEY referenced only in the SDK-invocation step's env: block, sourced from secrets.ANTHROPIC_API_KEY (Environment secret).
• Preflight audit step logs actor + SHA + fixture_glob BEFORE the SDK step.
• Artifact upload step has if: always() so failure artifacts still land.
• No api_key or similar workflow input.

Follow-ups

• Parity: OpenAI (and other provider) skill-drift checks alongside the Claude Agent SDK version #368 -- OpenAI SDK + other-provider Tier-2-style parity checks.
• Parity Tier 2: add scheduled cadence + governance-appropriate API-key sourcing #369 -- scheduled cadence + governance-appropriate key sourcing.
• The inferred_from authority-handling disagreement on OSPS-LE-03.02 -- worth its own issue once feat: RFC-0001 Stage 1 authority + darnit harness fleet driver #365 lands; the parity infrastructure now makes it a first-class regression signal.

Rebase plan

• When feat: RFC-0001 Stage 1 authority + darnit harness fleet driver #365 merges: git fetch upstream && git rebase upstream/main on this branch, then git push --force-with-lease. Diff collapses to feature 028 only.

[kusari-inspector]

Kusari Analysis Results:

✅ No Flagged Issues Detected
All values appear to be within acceptable risk parameters.

Both analyses independently recommend proceeding. The dependency analysis detected no pinned version dependency changes, presenting no dependency-related risk. The code analysis identified a single medium-severity, low-confidence finding (artipacked): actions/checkout at line 50 does not set persist-credentials: false, leaving the GitHub token stored in git config for the job duration. However, strong compensating controls neutralize any realistic exploit path: (1) the workflow trigger is workflow_dispatch only, eliminating untrusted PR or push attack vectors; (2) the environment parity-tier2 requires human reviewer approval before execution; (3) permissions are scoped to contents: read at both workflow and job levels, meaning the persisted credential has no write capability to abuse; (4) all actions are pinned to full 40-character commit SHAs. No secrets were detected and no additional code issues were found. The persisted credential issue is a hardening gap rather than an exploitable vulnerability under the current permission model. As an optional hardening measure, setting persist-credentials: false on the checkout action would eliminate the finding entirely.

Note

View full detailed analysis result for more information on the output and the checks that were run.

---

@kusari-inspector rerun - Trigger a re-analysis of this PR
@kusari-inspector feedback [your message] - Send feedback to our AI and team
See Kusari's documentation for setup and configuration.
Commit: 0e74d48, performed at: 2026-08-12T17:56:52Z

Found this helpful? Give it a 👍 or 👎 reaction!

[kusari-inspector]

Issue: The Preflight audit log step interpolates ${{ inputs.fixture_glob }} and ${{ github.actor }} directly into the shell script. Move all context values to step-level env: variables and reference them as shell variables to prevent injection.

Recommended Code Changes:

- name: Preflight audit log (T2-7/T2-8)
  env:
    ACTOR: ${{ github.actor }}
    SHA: ${{ github.sha }}
    FIXTURE_GLOB: ${{ inputs.fixture_glob }}
  run: |
    {
      echo "## Tier 2 Preflight"
      echo ""
      echo "- actor: $ACTOR"
      echo "- sha: $SHA"
      echo "- fixture_glob: $FIXTURE_GLOB"
      echo "- timestamp: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
    } >> "$GITHUB_STEP_SUMMARY"

[kusari-inspector]

Issue: The Run Tier 2 parity check step interpolates ${{ inputs.fixture_glob }} directly as a shell argument. Add FIXTURE_GLOB to the existing env: block and reference it as a quoted shell variable.

Recommended Code Changes:

- name: Run Tier 2 parity check
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
    FIXTURE_GLOB: ${{ inputs.fixture_glob }}
  run: |
    uv run python -m tests.darnit.parity.tier2.run \
      --fixture-glob "$FIXTURE_GLOB"

[kusari-inspector]

actions/checkout, actions/setup-python, astral-sh/setup-uv, and actions/upload-artifact are all pinned to mutable version tags (@v4, @v5, @v6). If an action owner silently repoints a tag, malicious code could execute in this runner and access the ANTHROPIC_API_KEY. Pin each action to a full 40-character commit SHA to guarantee immutability.

[mlieberman85]

Stacked with follow-on PR #371 (feature 029: OpenAI Tier 2 backend + SkillInvocationBackend Protocol; Fixes #368). #371 depends on this PR.

[kusari-inspector]

Kusari PR Analysis rerun based on - 0e74d48 performed at: 2026-08-12T17:57:19Z - link to updated analysis

[pxp928]

Approving on the condition that the findings below are addressed before merge. The two-tier design is the right shape for #366 and the comparator/diff structure is a good foundation. But as committed the suite doesn't execute anywhere — not locally, not in CI — and several of the comparison primitives report agreement where there is none. Please don't merge until the first three are fixed, since until then a green check means nothing ran.

Reviewed the incremental diff pr-365..pr-370 (commits 0a1671c, 0e74d48). I ran the suite to confirm the top findings.

---

Stack note

This branch is stacked on #365, not on #367. Worth confirming that's intentional — #367 and #370 are currently siblings.

---

Blockers — the suite never runs

1. Fixture marker file is gitignored

tier1/conftest.py:41

Fixture discovery requires a .baseline.toml marker in each fixture dir, but that filename is in .gitignore:32 and never committed.

Result, verified by running pytest: 0 of 4 fixtures discovered, 3 tests fail outright, test_parity collects zero cases, and the Tier 2 runner always exits 3.

Either rename the marker to something not ignored, or add a negation pattern un-ignoring the fixture paths. This one also propagates into #371, where it ships two red tests.

2. No pytest markers → CI deselects the entire suite

tier1/test_mcp_vs_harness.py:19

No parity test declares a unit or integration marker, but ci.yml only runs -m unit / -m integration. Actual result: 53 items / 53 deselected.

The Tier 1 gate is described as "runs on every PR." It runs on no PR.

3. The no-product-changes guard never enforces

tier1/test_no_product_changes.py:27

The FR-014 guard's hardcoded base-ref list falls through to origin/main, which flags 22 product files belonging to feature 026 — it fails locally today. In a shallow CI clone no ref resolves at all and it silently skips. So it either false-positives or no-ops; it never does the job.

---

Correctness bugs in the comparison logic

These matter more than ordinary test bugs, because a parity harness that reports false agreement is worse than no parity harness.

• tier1/comparator.py:121 — is_allowed_drift returns True for PENDING_LLM vs the synthetic <MISSING>, so a control the harness drops entirely reports green. Directly contradicts T1-3. Verified.
• tier2/skill_markdown_parser.py:154 — status literals are scanned in length-descending order rather than by position, so "OSPS-VM-04.01: WARN -- ... to reach PASS" parses as PASS. Verified. Should anchor on the first status token after the control ID.
• tier2/diff.py:65 — the diff iterates only over skill claims, so a skill that omits a FAIL control (and omits the Failed: count line) returns success. Needs to iterate the union of both sides' control sets.

Tier 2 doesn't exercise what it claims to

tier2/claude_agent_sdk_client.py:85 — ClaudeAgentOptions sets no mcp_servers, no allowed_tools, no permission_mode, no setting_sources. The agent can therefore neither call audit_openssf_baseline nor shell out to anything.

So Tier 2 compares real MCP tool JSON against a summary the model produced without running an audit at all. The prompt is also a hand-written approximation of the skill rather than the real /darnit-audit skill definition.

This undermines the premise of the tier rather than being a bug in it — worth a design decision, not just a patch.

Packaging

pyproject.toml:56 — claude-agent-sdk (a ~90 MB platform wheel) is added to the published [project.optional-dependencies].dev extra of darnit-mcp, contradicting the SC-006 comment directly above it. [dependency-groups] dev already exists and is the right home.

Minor

• tier1/conftest.py:136 — total_run_timeout_s=30 on the harness path (unbounded on the MCP path) will raise HarnessRunTimeout on a full level-3 audit against a fake remote, erroring the test instead of comparing.
• tier2/run.py:175 — the errored bucket is checked before the disagreement buckets, so one infra exception downgrades a real per-control disagreement to exit 3 ("setup error"), hiding the signal you care about.
• tier1/conftest.py:139 — asyncio.new_event_loop().run_until_complete(...) at three sites (also run.py:203, test_diff_adversarial.py:132) leaks an unclosed loop and never sets it current.
• tier2/test_workflow_config.py:18 — cwd-relative Path(".github/workflows") (and Path.cwd() at test_diff_adversarial.py:148) makes the governance tests fail or vacuously skip when pytest runs outside the repo root.
• tier2/diff.py:24 — COUNTS_DISAGREE = 1 silently aliases PER_CONTROL_DISAGREE. The enum is unused anyway, since run.py hardcodes its exit codes.