feat: RFC-0001 Stage 1 authority + `darnit harness` fleet driver by mlieberman85 · Pull Request #365 · darnitdevorg/darnit

mlieberman85 · 2026-08-06T20:26:27Z

Summary

Two stacked features shipped in one PR so the substrate and the deliverable can be reviewed together. Split into two commits; each commit has its own scope and rationale.

Commit 1 -- feat(rfc-0001-stage1): authority model, LLMStep protocol, ActionPlan public protocol, per-phase Check termination rule. This is the substrate the harness sits on. Adds pydantic-ai-slim[anthropic] as a required runtime dep.

Commit 2 -- feat(harness): darnit harness <path> async driver that runs one repo end to end. Fleet-driver surface -- not the primary product path.

Detailed rationale in each commit body.

What ships as user-visible

New CLI subcommand: darnit harness <path> [--level N] [--framework X] [--answers file] [--output file] [--format md|json]
Two new MCP tools: run_next_action, submit_action_result (client-owned state)
Baseline attestation predicate gains a per-result authority field (additive within v1)

What's substrate (not user-visible)

authority on every sieve step + result -- LLM-only output can no longer manufacture a PASS
AnswerSource protocol + AnswerResolver for pluggable question resolution
HarnessRun async driver with fail-fast credential check, per-call + total-run timeouts, redaction

Constitution notes

IV (Never Guess User Values): reinforced. The reference control STAGE1-REF-SECURITY-01 demonstrates the two-step pattern -- a dispositive observation must accompany a suggestive LLM step for the control to conclude PASS.
LLM SDK is a required dep, not optional. There is no deterministic-only product tier.

Test plan

uv run pytest tests/ -q -- expect 2528 passed
uv run ruff check .
Manual smoke: uv run darnit harness /path/to/some/repo --level 1 --format json | jq .summary
Manual smoke: unset ANTHROPIC_API_KEY && uv run darnit harness /path/to/some/repo -- expect exit 2 in <2s with a setup_error stderr summary
Manual smoke: run against a real repo you're familiar with; sanity-check the report's status distribution against your mental model. Note: the report reflects what run_sieve_audit produces -- see Test parity between coding-agent (MCP) audit path and harness audit path #366 for the parity work between harness and coding-agent paths.

Follow-up

Test parity between coding-agent (MCP) audit path and harness audit path #366: parity testing between the coding-agent path (/darnit-audit skill invoking MCP tools) and the harness path. Intentionally scoped out of this PR.

…tocol RFC-0001 Stage 1 substrate. Adds `authority` (`dispositive` | `suggestive` | `asserted`) as a first-class attribute of every sieve step and result; enforces a per-phase Check execution rule so only dispositive or asserted results conclude a control. LLM-only output can no longer manufacture a PASS: a suggestive `llm_extract` step must be paired with a dispositive observation (typically `file_exists`) for the control to conclude PASS. Adds the pluggable `LLMStep` Protocol with `PydanticAILLMStep` (real, via pydantic-ai-slim[anthropic]) and `MockLLMStep` (test). Makes `pydantic-ai-slim[anthropic]` a required runtime dep -- there is no deterministic-only tier. Introduces `darnit.core.action_plan` as the public typed protocol (`next_action` / `submit_result`) for advancing the audit graph; the existing `agent.graph.route()` becomes a thin adapter. Exposes the two protocol calls via new `run_next_action` / `submit_action_result` MCP tools so clients own the state. Baseline attestation predicate gains a per-result `authority` field (additive within v1). Adds `STAGE1-REF-SECURITY-01` -- a reference control demonstrating the two-step dispositive-plus-suggestive pattern. Sieve orchestrator: `_apply_cel_expr` and `verify_with_llm_response` now propagate `authority` through PASS/FAIL/INCONCLUSIVE/WARN rebuilds. `_check_inferred_from` inherits the source control's authority so an inferred PASS is never `unknown`. Tests: 20+ new tests covering the authority Literal domain, per-phase Check rule, LLMStep Protocol conformance, PydanticAI construction without an API key, the STAGE1-REF-SECURITY-01 dispositive/suggestive pairing, action-plan/graph equivalence, and the MCP loop tools.

Adds a deliverable fleet-driver surface on top of the Stage 1 substrate. `darnit harness <path>` runs one repo end to end: initial sieve audit with stop_on_llm=True, batched LLM continuation via the injected `LLMStep`, unanswered-question collection through a pluggable `AnswerSource` chain, and a single `HarnessReport` (Markdown or JSON) that the caller can pipe into CI. Four-class exit codes distinguish audit outcomes from setup and internal errors: SUCCESS=0, AUDIT_FAILURES=1, SETUP_ERROR=2, INTERNAL_ERROR=3. Setup validation fails fast (<2s) when `ANTHROPIC_API_KEY` is absent or `pydantic_ai` is not importable, before any control runs. Per-call and total-run timeouts bound LLM calls via `asyncio.wait_for`; an LLM outage collapses to INCONCLUSIVE and the affected control resolves to WARN (never a fabricated PASS -- SC-008). Progress lines `[N/M] <control_id> <phase-verb>` emit through stdlib logging on the `darnit.harness` logger for grep-able CI dashboards. API key never appears in logs or the report. Pluggable answer sources (`ProjectYamlAnswerSource`, `FileAnswerSource`) compose via `AnswerResolver` with last-wins precedence: `--answers` overrides `.project/project.yaml`. Unanswered questions are captured in the report but MVP does not re-audit after collect; that policy is enforced by a driver-internal invariant test so any future auto-reaudit change is a deliberate contract update. Tests: 42 new tests (T007..T042 + T045b) covering the four exit classes, fail-fast bound, progress-line format, answer-source composition and precedence, LLM-suggestive-cannot-conclude-PASS (SC-008), API-key redaction, and no-re-audit-after-collect invariant. Note: the CLI is a dev/test/fleet-driver surface, not the primary product path. The product is MCP tools + coding-agent invocation.

mlieberman85 · 2026-08-09T15:30:35Z

Stacked with follow-on PR #367 (feature 027: interactive question resolvers). #367 depends on this PR; when this merges, #367 will rebase and its diff will collapse to only feature-027 changes.

mlieberman85 · 2026-08-10T22:18:11Z

Stacked with follow-on PR #370 (feature 028: two-tier audit parity test suite; Fixes #366). #370 depends on this PR. When this merges, #370 will rebase and its diff will collapse to only feature-028 changes.

Note: #370 also depends on #367 semantically (both are stacked on this PR's branch), but they're siblings once #365 lands -- #367 and #370 can merge in any order after #365.

pxp928

Approving on the condition that the two blockers below are fixed before merge. The architecture is right and the substrate is worth building on — but as it stands this PR silently regresses every plugin-defined control from PASS to WARN, and the reference control it ships can never PASS. Please don't merge until those two are addressed.

Reviewed as the base of a 4-PR stack (#365 → #367 / #370 → #371). Findings verified by running the code at 559dd1e against the merge-base 9974caf.

Blockers

1. Plugin handlers all default to `suggestive` → silent PASS→WARN regression

handler_registry.py:175 — register() defaults default_authority="suggestive".

The built-in handlers were all updated to pass it explicitly (builtin_handlers.py:965-1046). No plugin was. All eight plugin sieve handlers register without it:

Package	Handlers
`darnit-gittuf`	`gittuf_verify_policy`, `gittuf_commits_signed`
`darnit-reproducibility`	`repro_deps_pinned`, `repro_build_env_declared`, `repro_hermetic_build`, `repro_provenance_exists`, `repro_bit_for_bit`
`darnit-baseline`	`generate_threat_model`

Full chain: suggestive + last step → TERMINATE_INCONCLUSIVE (orchestrator.py:72) → WARN.

Verified: RE-01.01 with uv.lock present returns PASS on 9974caf and WARN on this PR.

TOML cannot work around it — authority = "dispositive" on a step raises AuthorityViolation under the new no-loosening rule (control_loader.py:567). The fix has to land in each plugin's register_handlers().

Suggested follow-up: a regression test asserting no registered handler falls back to the default implicitly, so the next plugin doesn't reintroduce this.

2. `STAGE1-REF-SECURITY-01` can never PASS — and ships at level 1

openssf-baseline.toml:4386

llm_extract is ordered first; under the default stop_on_llm=True it returns PENDING_LLM, so the dispositive file_exists step never runs. verify_with_llm_response only looks for llm_eval, so the control lands on WARN. Verified with SECURITY.md present.

This is the PR's own demonstration of the two-step dispositive+suggestive pattern, and the ordering inverts it — and it ships at level 1 in the production baseline TOML, so it affects every real audit.

Should fix before merge

action_plan.py:349 — the collect_context branch of submit_result skips _validate_context_answer (the shell-metacharacter / newline / null-byte guard that graph.collect_context applies). The MCP tool then persists those values as USER_CONFIRMED into .project/project.yaml. Security-relevant given the values flow into generated command snippets.
driver.py:409 — --answers is inert. _collect_unanswered reads a feedback_questions key that sieve results never carry, and the resolved context_values are discarded by _run_body.
driver.py:467 — exit code is SUCCESS whenever fail == 0, so a run where every control ERRORed or stayed pending exits 0. For a fleet driver that's the difference between "clean" and "nothing ran."

Worth addressing

driver.py:229 — the consultation payload drops file_contents, gathered_evidence, analysis_hints, and confidence_threshold; the LLM judges from the bare prompt with no repository content.
driver.py:303 — _llm_continuation_loop hardcodes "openssf-baseline" when --framework is absent, while the initial audit resolves the framework from .baseline.toml. Mismatched frameworks leave PENDING_LLM in the report or raise ValueError → exit 3.
driver.py:531 — asyncio.to_thread for the initial audit is not preemptible, and asyncio.run joins the default executor at shutdown, so total_run_timeout_s cannot bound wall-clock time the way the comment claims.
control_loader.py:567 — _AUTHORITY_STRENGTH.get(x, 0) silently accepts unknown authority strings. A typo passes load-time validation and then quietly makes the step non-concluding — exactly the failure mode this feature exists to prevent. Reject unknown values at load time.
action_plan.py:384 — EvidenceItem.raw duplicates the full audit result list into state.evidence, and the whole state is the MCP wire format on every call. Payload grows quadratically with control count.

Stack note

#370 is branched off this PR, not #367 — so #367 and #370 are siblings, not a linear stack. Merge order will need a decision. Reviews on the other three follow.

…der + collect_context validation Reviewer pxp928 flagged two blockers and eight follow-ups on darnitdevorg#365. This commit addresses all of them. Blockers: - Plugin sieve handlers registered without `default_authority=` silently defaulted to `"suggestive"`, so a passing observation-based handler (gittuf verify, reproducibility repro checks, threat-model generation) produced a WARN instead of terminating the Check phase on PASS. `SieveHandlerRegistry.register(...)` is now keyword-only-required for `default_authority`; a plugin that forgets the argument gets a TypeError at registration time rather than a silent audit-status regression. All 8 plugin-owned handlers now register explicitly as `"dispositive"`. Regression test in tests/darnit/sieve/. - STAGE1-REF-SECURITY-01 could never PASS: llm_extract (suggestive) ran before file_exists (dispositive), and under stop_on_llm=True the sieve halted on the LLM step so file_exists never executed. Passes are reordered so the dispositive step runs first. Losing the "propose a contact string even when SECURITY.md is missing" property is documented in the TOML as a follow-up. Should-fix: - `submit_result`'s `collect_context` branch now runs the same shell- metacharacter guard the legacy agent-graph confirm_data flow used. The validator moved to `darnit.core.context_validation` so both call sites share it without a `core -> agent` import cycle. - `--answers` is no longer inert. `_collect_unanswered` now enumerates the framework's own `[context.*]` pending keys via `get_pending_context` and routes each through the `AnswerResolver`. Answers from `.project/project.yaml` and `--answers` are now actually consulted. - `exit_class` non-zero on any FAIL / ERROR / WARN. Previously an all-ERROR run exited 0 even though the harness verified nothing. Worth-addressing: - `ConsultationRequest` now carries `gathered_evidence`, `file_contents`, `analysis_hints`, and `confidence_threshold`. The harness stopped dropping them at the driver boundary, so the LLM sees the sieve's prior-pass evidence and the control author's hints. - LLM continuation resolves the framework via `load_effective_config_auto` (honoring `.baseline.toml`'s `extends` field) instead of hardcoding `"openssf-baseline"` as a fallback. - Control loader rejects unknown authority literals up front instead of silently coercing them to strength 0. - `EvidenceItem.raw` no longer copies `audit_results` / `feedback_questions` / `remediation_results` into the evidence log -- those already live on the state; the per-step raw payload had grown O(steps * controls) per run. Not fixed: `asyncio.to_thread(self._initial_audit)` is not preemptible by `asyncio.wait_for`, because Python threads can't be forcibly killed. The existing comment already documents the constraint; a real fix needs subprocess isolation and is out of scope for this review response. Workspace sweep: 2537 pass, 15 skip.

…flag Extends feature 026's harness with an active resolver seam. Where the `AnswerSource` chain is passive lookup ("here's a preloaded value"), `QuestionResolver` is active resolution ("go get me an answer somehow -- prompt a human, call an API, open an issue"). The two run in sequence: `.project/project.yaml` -> `--answers <file>` -> registered resolvers in chain order. Ships a hybrid registration surface (matching darnit's existing `darnit.frameworks` discovery pattern): - Python entry points under group `darnit.question_resolvers` for third-party packages. Discovery is lazy at CLI startup; broken entry points log a warning and are skipped, never crash the harness. - Direct injection into `HarnessRun.question_resolvers` for tests and library consumers. Reference implementation `InteractiveTerminalResolver` prompts on `/dev/tty` -- the private operator channel used by git, ssh, and sudo. Isolated from stdout (report body) and stderr (progress + exit summary), so feature 026's stream contracts stay intact. Empty and whitespace-only input treated as skip; Ctrl+C or Ctrl+D raises `InteractiveAborted` which the driver catches to stop further prompts while preserving already-collected answers. Test-injectable streams (`io.StringIO`) so unit tests never touch a real terminal. `--interactive` CLI flag on `darnit harness`. Default off. Fails fast in under 2 seconds with exit code 2 when stdin is not a TTY OR `/dev/tty` is not openable; the SC-005 guard runs BEFORE any control executes so a CI misfire cannot silently skip every question. Constitution IV property enforced at the type level: `Answer.authority` is a `Literal["asserted"]` with a fixed default. Resolver authors physically cannot construct an `Answer` with a different authority -- Pydantic raises `ValidationError` at construction time. Every answer that flows through this feature carries `authority: "asserted"`. Per-question `resolution_trail: list[ResolutionTrailEntry]` in the report captures which resolvers were offered a question and how each responded (`answered` / `skipped` / `errored`). Error summaries pass through feature 026's `_redact_secrets` and are truncated to 200 characters so credential material and runaway stack traces cannot leak into the report. New `AnsweredFeedbackEntry` list on the report surfaces per-answer provenance (origin + authority + trail) so an auditor can reconstruct how every user-judgment value was obtained. Feature 026's "no re-audit after collect" MVP invariant is preserved: a resolver-supplied answer is captured in the report but does NOT silently promote a FAIL to PASS. The audit's status stays what the sieve concluded; a subsequent audit run with the value persisted to .project/project.yaml re-evaluates it. Verified by extending the existing invariant test. Per-resolver `asyncio.wait_for` timeout wrapper (FR-011) available via `HarnessRun.per_resolver_timeout_s`. Default None (no timeout, matches the interactive resolver where a human may take arbitrary time). A timeout is captured as a `ResolutionTrailEntry(outcome="errored", error_summary="resolver timed out after Ns")` and the driver moves on. Tests: 44 new tests across 8 files (test_question_resolvers, test_protocol_conformance, test_interactive_resolver, test_resolution_trail, test_resolver_discovery, test_extensibility_sc002, plus additions to test_driver, test_cli, test_report). SC-002 (external resolver, no harness edits) is mechanically enforced by a fixture package that lives OUTSIDE packages/darnit/src/darnit/harness/ (in tests/.../fixtures/). Depends on PR darnitdevorg#365 (feature 026 + Stage 1 substrate).

…g#366) Adds a diagnostic test suite verifying the darnit audit's per-control output is consistent across the three consumers users care about: the direct MCP tool call (`audit_openssf_baseline`), the `darnit harness` end-to-end path, and the `/darnit-audit` coding-agent skill's summary. Motivated by the PR darnitdevorg#365 review where the skill was observed silently reclassifying WARN as PASS in its Markdown summary while the MCP tool and harness produced identical raw results. Tier 1 -- mechanical MCP-vs-harness parity (`tests/darnit/parity/tier1/`) runs on every PR. For each fixture in the corpus, invokes both audit paths in-process (harness uses `MockLLMStep` so no live API), normalizes to a common `AuditResult`, and diffs per-control status. Sole allowed drift: MCP leaves a control PENDING_LLM; harness resolves it via the LLM continuation loop to any non-PENDING_LLM status. Anything else is a hard failure with a human-readable fixed-width Markdown drift table (no ANSI). Full corpus runs in about 35 seconds, well under the 60s budget. Tier 2 -- coding-agent skill parity (`tests/darnit/parity/tier2/`) is manual-dispatch only via `.github/workflows/parity-tier2.yml`. The workflow declares `environment: parity-tier2` so a GitHub Environment with required reviewers gates every dispatch; `ANTHROPIC_API_KEY` is stored at the Environment level (never at repo scope). The runner captures raw MCP tool JSON, invokes the `/darnit-audit` skill via the Claude Agent SDK with an explicit prompt snapshot + turn cap + zero temperature, parses the skill's final assistant message, and diffs. Per-control status disagreement is a hard failure regardless of authority level (skill has no license to reinterpret). Unparseable skill output surfaces as a distinct failure class from disagreement so a maintainer can tell a broken parser from a real drift. Fixture corpus: `all_pass_repo`, `all_fail_repo`, `mixed_repo`, `pending_llm_repo`. Each ships with a `parity.toml` declaring category, expected counts, and a `control_ids` filter (both audit paths run all 66 OpenSSF Baseline controls today because neither auto-applies fixture-level `audit_profiles` from `.baseline.toml`; the filter is test-side and honest about scope). SC-008 corpus-inventory test verifies every category is represented. Governance property (FR-007a + SC-005a) enforced at the GitHub Actions Environment layer AND by a pytest test that iterates every `.github/workflows/*.yml` and asserts `ANTHROPIC_API_KEY` appears only in `parity-tier2.yml`. Pure-Python file iteration; no `grep` subprocess. FR-014 zero-product-code-changes enforced by `test_no_product_changes.py`: runs `git diff --name-only origin/026-harness-with-stage1...HEAD` and fails if any file under `packages/darnit/src/` or `packages/darnit-baseline/src/` is modified. Diagnostic finding surfaced during implementation: MCP tool and harness disagree on `OSPS-LE-03.02` (feature 026's `inferred_from` authority handling has a residual bug). Fixtures scope around it via `control_ids` in `parity.toml`. That divergence is now a discoverable finding for a follow-up feature to fix; the parity test surface makes it CI-visible from now on. 56 new tests (35 Tier 1 + 21 Tier 2). Full workspace sweep: 2589 pass, 15 skipped, 0 failures. `ruff check` clean; `validate_sync.py` PASSED. `claude-agent-sdk` added as a workspace dev-group dep only; product packages `packages/darnit/pyproject.toml` and `packages/darnit-baseline/pyproject.toml` are untouched. Closes darnitdevorg#366. Follow-up issues opened: - darnitdevorg#368: OpenAI SDK + other-provider parity checks (Tier-2-style, different provider) - darnitdevorg#369: scheduled cadence + governance-appropriate key sourcing Depends on PR darnitdevorg#365 (feature 026 + Stage 1 substrate).

…xternal resolvers Reviewer pxp928 flagged three blockers and a Constitution IV violation on darnitdevorg#367. This commit addresses all four. Blockers: - --interactive was inert in production because sieve results never carried `feedback_questions`. Fixed upstream in PR darnitdevorg#365 by wiring `_enumerate_framework_pending` into `_collect_unanswered`; the QuestionResolver chain now sees the framework's own [context.*] keys and can prompt for them. - Operator think-time was charged to `total_run_timeout_s`, so a 15-minute default budget would cancel any real interactive collect mid-question. Split `_run_body` into `_run_audit_and_llm` (bounded) and the collect phase (unbounded). Per-resolver preemption is still handled inside `_run_resolver_chain` via `per_resolver_timeout_s`. - Blocking `readline()` in an async def froze the event loop, so `asyncio.wait_for(coro, per_resolver_timeout_s)` couldn't fire. Wrap `readline()` in `asyncio.to_thread` so the event loop keeps running and the driver's per-resolver timeout actually preempts. Constitution IV violation: - Non-interactive runs silently invoked every installed third-party QuestionResolver and recorded their outputs as authority="asserted", which Constitution Principle IV reserves for human confirmation. External resolvers are now gated behind a new `--allow-external-resolvers` flag on `darnit harness`. Without it, discovered externals are logged at INFO ("N resolver(s) discovered but NOT invoked") so the operator can see what's available and opt in explicitly. Should-fix: - `_ensure_streams` no longer silently ignores a partially-injected stream pair. Passing input_stream XOR output_stream now raises ValueError, matching contract IR-25. - Exit-summary field order: `<PEND> pending` sits BEFORE `<A> answered` again so CI parsers written against the CLI-13 grep contract keep matching. The answered count is additive, appended after pending. - Post-abort counter: aborting mid-collect used to skip counting every remaining question. The early-abort branch now increments `counts["aborted"]` on each skipped-post-abort question. - Defensive `getattr(resolver, "name", "<unknown>")` in the three exception paths so a misbehaving resolver's missing `.name` attribute cannot crash the exception handler. Workspace sweep: 2607 pass, 15 skip.

… intended framework `.baseline.toml` was in `.gitignore`, so the minimal_llm_repo fixture had no framework config tracked. Locally the file exists and CI never noticed. In a fresh CI clone the harness auto-resolves whichever framework entry point loads first (darnit-testchecks in the current workspace) instead of openssf-baseline, so STAGE1-REF-SECURITY-01 never runs and `test_llm_suggestive_cannot_conclude_pass` fails with "STAGE1-REF-SECURITY-01 not in results" -- a CI-only failure that survived the 2537-passing local sweep on this branch. Add an `!tests/darnit/harness/fixtures/**/.baseline.toml` negation to `.gitignore` and force-add the fixture config.

…flag Extends feature 026's harness with an active resolver seam. Where the `AnswerSource` chain is passive lookup ("here's a preloaded value"), `QuestionResolver` is active resolution ("go get me an answer somehow -- prompt a human, call an API, open an issue"). The two run in sequence: `.project/project.yaml` -> `--answers <file>` -> registered resolvers in chain order. Ships a hybrid registration surface (matching darnit's existing `darnit.frameworks` discovery pattern): - Python entry points under group `darnit.question_resolvers` for third-party packages. Discovery is lazy at CLI startup; broken entry points log a warning and are skipped, never crash the harness. - Direct injection into `HarnessRun.question_resolvers` for tests and library consumers. Reference implementation `InteractiveTerminalResolver` prompts on `/dev/tty` -- the private operator channel used by git, ssh, and sudo. Isolated from stdout (report body) and stderr (progress + exit summary), so feature 026's stream contracts stay intact. Empty and whitespace-only input treated as skip; Ctrl+C or Ctrl+D raises `InteractiveAborted` which the driver catches to stop further prompts while preserving already-collected answers. Test-injectable streams (`io.StringIO`) so unit tests never touch a real terminal. `--interactive` CLI flag on `darnit harness`. Default off. Fails fast in under 2 seconds with exit code 2 when stdin is not a TTY OR `/dev/tty` is not openable; the SC-005 guard runs BEFORE any control executes so a CI misfire cannot silently skip every question. Constitution IV property enforced at the type level: `Answer.authority` is a `Literal["asserted"]` with a fixed default. Resolver authors physically cannot construct an `Answer` with a different authority -- Pydantic raises `ValidationError` at construction time. Every answer that flows through this feature carries `authority: "asserted"`. Per-question `resolution_trail: list[ResolutionTrailEntry]` in the report captures which resolvers were offered a question and how each responded (`answered` / `skipped` / `errored`). Error summaries pass through feature 026's `_redact_secrets` and are truncated to 200 characters so credential material and runaway stack traces cannot leak into the report. New `AnsweredFeedbackEntry` list on the report surfaces per-answer provenance (origin + authority + trail) so an auditor can reconstruct how every user-judgment value was obtained. Feature 026's "no re-audit after collect" MVP invariant is preserved: a resolver-supplied answer is captured in the report but does NOT silently promote a FAIL to PASS. The audit's status stays what the sieve concluded; a subsequent audit run with the value persisted to .project/project.yaml re-evaluates it. Verified by extending the existing invariant test. Per-resolver `asyncio.wait_for` timeout wrapper (FR-011) available via `HarnessRun.per_resolver_timeout_s`. Default None (no timeout, matches the interactive resolver where a human may take arbitrary time). A timeout is captured as a `ResolutionTrailEntry(outcome="errored", error_summary="resolver timed out after Ns")` and the driver moves on. Tests: 44 new tests across 8 files (test_question_resolvers, test_protocol_conformance, test_interactive_resolver, test_resolution_trail, test_resolver_discovery, test_extensibility_sc002, plus additions to test_driver, test_cli, test_report). SC-002 (external resolver, no harness edits) is mechanically enforced by a fixture package that lives OUTSIDE packages/darnit/src/darnit/harness/ (in tests/.../fixtures/). Depends on PR darnitdevorg#365 (feature 026 + Stage 1 substrate).

…xternal resolvers Reviewer pxp928 flagged three blockers and a Constitution IV violation on darnitdevorg#367. This commit addresses all four. Blockers: - --interactive was inert in production because sieve results never carried `feedback_questions`. Fixed upstream in PR darnitdevorg#365 by wiring `_enumerate_framework_pending` into `_collect_unanswered`; the QuestionResolver chain now sees the framework's own [context.*] keys and can prompt for them. - Operator think-time was charged to `total_run_timeout_s`, so a 15-minute default budget would cancel any real interactive collect mid-question. Split `_run_body` into `_run_audit_and_llm` (bounded) and the collect phase (unbounded). Per-resolver preemption is still handled inside `_run_resolver_chain` via `per_resolver_timeout_s`. - Blocking `readline()` in an async def froze the event loop, so `asyncio.wait_for(coro, per_resolver_timeout_s)` couldn't fire. Wrap `readline()` in `asyncio.to_thread` so the event loop keeps running and the driver's per-resolver timeout actually preempts. Constitution IV violation: - Non-interactive runs silently invoked every installed third-party QuestionResolver and recorded their outputs as authority="asserted", which Constitution Principle IV reserves for human confirmation. External resolvers are now gated behind a new `--allow-external-resolvers` flag on `darnit harness`. Without it, discovered externals are logged at INFO ("N resolver(s) discovered but NOT invoked") so the operator can see what's available and opt in explicitly. Should-fix: - `_ensure_streams` no longer silently ignores a partially-injected stream pair. Passing input_stream XOR output_stream now raises ValueError, matching contract IR-25. - Exit-summary field order: `<PEND> pending` sits BEFORE `<A> answered` again so CI parsers written against the CLI-13 grep contract keep matching. The answered count is additive, appended after pending. - Post-abort counter: aborting mid-collect used to skip counting every remaining question. The early-abort branch now increments `counts["aborted"]` on each skipped-post-abort question. - Defensive `getattr(resolver, "name", "<unknown>")` in the three exception paths so a misbehaving resolver's missing `.name` attribute cannot crash the exception handler. Workspace sweep: 2607 pass, 15 skip.

…g#366) Adds a diagnostic test suite verifying the darnit audit's per-control output is consistent across the three consumers users care about: the direct MCP tool call (`audit_openssf_baseline`), the `darnit harness` end-to-end path, and the `/darnit-audit` coding-agent skill's summary. Motivated by the PR darnitdevorg#365 review where the skill was observed silently reclassifying WARN as PASS in its Markdown summary while the MCP tool and harness produced identical raw results. Tier 1 -- mechanical MCP-vs-harness parity (`tests/darnit/parity/tier1/`) runs on every PR. For each fixture in the corpus, invokes both audit paths in-process (harness uses `MockLLMStep` so no live API), normalizes to a common `AuditResult`, and diffs per-control status. Sole allowed drift: MCP leaves a control PENDING_LLM; harness resolves it via the LLM continuation loop to any non-PENDING_LLM status. Anything else is a hard failure with a human-readable fixed-width Markdown drift table (no ANSI). Full corpus runs in about 35 seconds, well under the 60s budget. Tier 2 -- coding-agent skill parity (`tests/darnit/parity/tier2/`) is manual-dispatch only via `.github/workflows/parity-tier2.yml`. The workflow declares `environment: parity-tier2` so a GitHub Environment with required reviewers gates every dispatch; `ANTHROPIC_API_KEY` is stored at the Environment level (never at repo scope). The runner captures raw MCP tool JSON, invokes the `/darnit-audit` skill via the Claude Agent SDK with an explicit prompt snapshot + turn cap + zero temperature, parses the skill's final assistant message, and diffs. Per-control status disagreement is a hard failure regardless of authority level (skill has no license to reinterpret). Unparseable skill output surfaces as a distinct failure class from disagreement so a maintainer can tell a broken parser from a real drift. Fixture corpus: `all_pass_repo`, `all_fail_repo`, `mixed_repo`, `pending_llm_repo`. Each ships with a `parity.toml` declaring category, expected counts, and a `control_ids` filter (both audit paths run all 66 OpenSSF Baseline controls today because neither auto-applies fixture-level `audit_profiles` from `.baseline.toml`; the filter is test-side and honest about scope). SC-008 corpus-inventory test verifies every category is represented. Governance property (FR-007a + SC-005a) enforced at the GitHub Actions Environment layer AND by a pytest test that iterates every `.github/workflows/*.yml` and asserts `ANTHROPIC_API_KEY` appears only in `parity-tier2.yml`. Pure-Python file iteration; no `grep` subprocess. FR-014 zero-product-code-changes enforced by `test_no_product_changes.py`: runs `git diff --name-only origin/026-harness-with-stage1...HEAD` and fails if any file under `packages/darnit/src/` or `packages/darnit-baseline/src/` is modified. Diagnostic finding surfaced during implementation: MCP tool and harness disagree on `OSPS-LE-03.02` (feature 026's `inferred_from` authority handling has a residual bug). Fixtures scope around it via `control_ids` in `parity.toml`. That divergence is now a discoverable finding for a follow-up feature to fix; the parity test surface makes it CI-visible from now on. 56 new tests (35 Tier 1 + 21 Tier 2). Full workspace sweep: 2589 pass, 15 skipped, 0 failures. `ruff check` clean; `validate_sync.py` PASSED. `claude-agent-sdk` added as a workspace dev-group dep only; product packages `packages/darnit/pyproject.toml` and `packages/darnit-baseline/pyproject.toml` are untouched. Closes darnitdevorg#366. Follow-up issues opened: - darnitdevorg#368: OpenAI SDK + other-provider parity checks (Tier-2-style, different provider) - darnitdevorg#369: scheduled cadence + governance-appropriate key sourcing Depends on PR darnitdevorg#365 (feature 026 + Stage 1 substrate).

…flag Extends feature 026's harness with an active resolver seam. Where the `AnswerSource` chain is passive lookup ("here's a preloaded value"), `QuestionResolver` is active resolution ("go get me an answer somehow -- prompt a human, call an API, open an issue"). The two run in sequence: `.project/project.yaml` -> `--answers <file>` -> registered resolvers in chain order. Ships a hybrid registration surface (matching darnit's existing `darnit.frameworks` discovery pattern): - Python entry points under group `darnit.question_resolvers` for third-party packages. Discovery is lazy at CLI startup; broken entry points log a warning and are skipped, never crash the harness. - Direct injection into `HarnessRun.question_resolvers` for tests and library consumers. Reference implementation `InteractiveTerminalResolver` prompts on `/dev/tty` -- the private operator channel used by git, ssh, and sudo. Isolated from stdout (report body) and stderr (progress + exit summary), so feature 026's stream contracts stay intact. Empty and whitespace-only input treated as skip; Ctrl+C or Ctrl+D raises `InteractiveAborted` which the driver catches to stop further prompts while preserving already-collected answers. Test-injectable streams (`io.StringIO`) so unit tests never touch a real terminal. `--interactive` CLI flag on `darnit harness`. Default off. Fails fast in under 2 seconds with exit code 2 when stdin is not a TTY OR `/dev/tty` is not openable; the SC-005 guard runs BEFORE any control executes so a CI misfire cannot silently skip every question. Constitution IV property enforced at the type level: `Answer.authority` is a `Literal["asserted"]` with a fixed default. Resolver authors physically cannot construct an `Answer` with a different authority -- Pydantic raises `ValidationError` at construction time. Every answer that flows through this feature carries `authority: "asserted"`. Per-question `resolution_trail: list[ResolutionTrailEntry]` in the report captures which resolvers were offered a question and how each responded (`answered` / `skipped` / `errored`). Error summaries pass through feature 026's `_redact_secrets` and are truncated to 200 characters so credential material and runaway stack traces cannot leak into the report. New `AnsweredFeedbackEntry` list on the report surfaces per-answer provenance (origin + authority + trail) so an auditor can reconstruct how every user-judgment value was obtained. Feature 026's "no re-audit after collect" MVP invariant is preserved: a resolver-supplied answer is captured in the report but does NOT silently promote a FAIL to PASS. The audit's status stays what the sieve concluded; a subsequent audit run with the value persisted to .project/project.yaml re-evaluates it. Verified by extending the existing invariant test. Per-resolver `asyncio.wait_for` timeout wrapper (FR-011) available via `HarnessRun.per_resolver_timeout_s`. Default None (no timeout, matches the interactive resolver where a human may take arbitrary time). A timeout is captured as a `ResolutionTrailEntry(outcome="errored", error_summary="resolver timed out after Ns")` and the driver moves on. Tests: 44 new tests across 8 files (test_question_resolvers, test_protocol_conformance, test_interactive_resolver, test_resolution_trail, test_resolver_discovery, test_extensibility_sc002, plus additions to test_driver, test_cli, test_report). SC-002 (external resolver, no harness edits) is mechanically enforced by a fixture package that lives OUTSIDE packages/darnit/src/darnit/harness/ (in tests/.../fixtures/). Depends on PR darnitdevorg#365 (feature 026 + Stage 1 substrate).

…xternal resolvers Reviewer pxp928 flagged three blockers and a Constitution IV violation on darnitdevorg#367. This commit addresses all four. Blockers: - --interactive was inert in production because sieve results never carried `feedback_questions`. Fixed upstream in PR darnitdevorg#365 by wiring `_enumerate_framework_pending` into `_collect_unanswered`; the QuestionResolver chain now sees the framework's own [context.*] keys and can prompt for them. - Operator think-time was charged to `total_run_timeout_s`, so a 15-minute default budget would cancel any real interactive collect mid-question. Split `_run_body` into `_run_audit_and_llm` (bounded) and the collect phase (unbounded). Per-resolver preemption is still handled inside `_run_resolver_chain` via `per_resolver_timeout_s`. - Blocking `readline()` in an async def froze the event loop, so `asyncio.wait_for(coro, per_resolver_timeout_s)` couldn't fire. Wrap `readline()` in `asyncio.to_thread` so the event loop keeps running and the driver's per-resolver timeout actually preempts. Constitution IV violation: - Non-interactive runs silently invoked every installed third-party QuestionResolver and recorded their outputs as authority="asserted", which Constitution Principle IV reserves for human confirmation. External resolvers are now gated behind a new `--allow-external-resolvers` flag on `darnit harness`. Without it, discovered externals are logged at INFO ("N resolver(s) discovered but NOT invoked") so the operator can see what's available and opt in explicitly. Should-fix: - `_ensure_streams` no longer silently ignores a partially-injected stream pair. Passing input_stream XOR output_stream now raises ValueError, matching contract IR-25. - Exit-summary field order: `<PEND> pending` sits BEFORE `<A> answered` again so CI parsers written against the CLI-13 grep contract keep matching. The answered count is additive, appended after pending. - Post-abort counter: aborting mid-collect used to skip counting every remaining question. The early-abort branch now increments `counts["aborted"]` on each skipped-post-abort question. - Defensive `getattr(resolver, "name", "<unknown>")` in the three exception paths so a misbehaving resolver's missing `.name` attribute cannot crash the exception handler. Workspace sweep: 2607 pass, 15 skip.

… (stacked on #365) (#367) * feat(harness): add `QuestionResolver` Protocol + `--interactive` CLI flag Extends feature 026's harness with an active resolver seam. Where the `AnswerSource` chain is passive lookup ("here's a preloaded value"), `QuestionResolver` is active resolution ("go get me an answer somehow -- prompt a human, call an API, open an issue"). The two run in sequence: `.project/project.yaml` -> `--answers <file>` -> registered resolvers in chain order. Ships a hybrid registration surface (matching darnit's existing `darnit.frameworks` discovery pattern): - Python entry points under group `darnit.question_resolvers` for third-party packages. Discovery is lazy at CLI startup; broken entry points log a warning and are skipped, never crash the harness. - Direct injection into `HarnessRun.question_resolvers` for tests and library consumers. Reference implementation `InteractiveTerminalResolver` prompts on `/dev/tty` -- the private operator channel used by git, ssh, and sudo. Isolated from stdout (report body) and stderr (progress + exit summary), so feature 026's stream contracts stay intact. Empty and whitespace-only input treated as skip; Ctrl+C or Ctrl+D raises `InteractiveAborted` which the driver catches to stop further prompts while preserving already-collected answers. Test-injectable streams (`io.StringIO`) so unit tests never touch a real terminal. `--interactive` CLI flag on `darnit harness`. Default off. Fails fast in under 2 seconds with exit code 2 when stdin is not a TTY OR `/dev/tty` is not openable; the SC-005 guard runs BEFORE any control executes so a CI misfire cannot silently skip every question. Constitution IV property enforced at the type level: `Answer.authority` is a `Literal["asserted"]` with a fixed default. Resolver authors physically cannot construct an `Answer` with a different authority -- Pydantic raises `ValidationError` at construction time. Every answer that flows through this feature carries `authority: "asserted"`. Per-question `resolution_trail: list[ResolutionTrailEntry]` in the report captures which resolvers were offered a question and how each responded (`answered` / `skipped` / `errored`). Error summaries pass through feature 026's `_redact_secrets` and are truncated to 200 characters so credential material and runaway stack traces cannot leak into the report. New `AnsweredFeedbackEntry` list on the report surfaces per-answer provenance (origin + authority + trail) so an auditor can reconstruct how every user-judgment value was obtained. Feature 026's "no re-audit after collect" MVP invariant is preserved: a resolver-supplied answer is captured in the report but does NOT silently promote a FAIL to PASS. The audit's status stays what the sieve concluded; a subsequent audit run with the value persisted to .project/project.yaml re-evaluates it. Verified by extending the existing invariant test. Per-resolver `asyncio.wait_for` timeout wrapper (FR-011) available via `HarnessRun.per_resolver_timeout_s`. Default None (no timeout, matches the interactive resolver where a human may take arbitrary time). A timeout is captured as a `ResolutionTrailEntry(outcome="errored", error_summary="resolver timed out after Ns")` and the driver moves on. Tests: 44 new tests across 8 files (test_question_resolvers, test_protocol_conformance, test_interactive_resolver, test_resolution_trail, test_resolver_discovery, test_extensibility_sc002, plus additions to test_driver, test_cli, test_report). SC-002 (external resolver, no harness edits) is mechanically enforced by a fixture package that lives OUTSIDE packages/darnit/src/darnit/harness/ (in tests/.../fixtures/). Depends on PR #365 (feature 026 + Stage 1 substrate). * review-fix(pr-367): unblock --interactive + preempt readline + gate external resolvers Reviewer pxp928 flagged three blockers and a Constitution IV violation on #367. This commit addresses all four. Blockers: - --interactive was inert in production because sieve results never carried `feedback_questions`. Fixed upstream in PR #365 by wiring `_enumerate_framework_pending` into `_collect_unanswered`; the QuestionResolver chain now sees the framework's own [context.*] keys and can prompt for them. - Operator think-time was charged to `total_run_timeout_s`, so a 15-minute default budget would cancel any real interactive collect mid-question. Split `_run_body` into `_run_audit_and_llm` (bounded) and the collect phase (unbounded). Per-resolver preemption is still handled inside `_run_resolver_chain` via `per_resolver_timeout_s`. - Blocking `readline()` in an async def froze the event loop, so `asyncio.wait_for(coro, per_resolver_timeout_s)` couldn't fire. Wrap `readline()` in `asyncio.to_thread` so the event loop keeps running and the driver's per-resolver timeout actually preempts. Constitution IV violation: - Non-interactive runs silently invoked every installed third-party QuestionResolver and recorded their outputs as authority="asserted", which Constitution Principle IV reserves for human confirmation. External resolvers are now gated behind a new `--allow-external-resolvers` flag on `darnit harness`. Without it, discovered externals are logged at INFO ("N resolver(s) discovered but NOT invoked") so the operator can see what's available and opt in explicitly. Should-fix: - `_ensure_streams` no longer silently ignores a partially-injected stream pair. Passing input_stream XOR output_stream now raises ValueError, matching contract IR-25. - Exit-summary field order: `<PEND> pending` sits BEFORE `<A> answered` again so CI parsers written against the CLI-13 grep contract keep matching. The answered count is additive, appended after pending. - Post-abort counter: aborting mid-collect used to skip counting every remaining question. The early-abort branch now increments `counts["aborted"]` on each skipped-post-abort question. - Defensive `getattr(resolver, "name", "<unknown>")` in the three exception paths so a misbehaving resolver's missing `.name` attribute cannot crash the exception handler. Workspace sweep: 2607 pass, 15 skip.

…g#366) Adds a diagnostic test suite verifying the darnit audit's per-control output is consistent across the three consumers users care about: the direct MCP tool call (`audit_openssf_baseline`), the `darnit harness` end-to-end path, and the `/darnit-audit` coding-agent skill's summary. Motivated by the PR darnitdevorg#365 review where the skill was observed silently reclassifying WARN as PASS in its Markdown summary while the MCP tool and harness produced identical raw results. Tier 1 -- mechanical MCP-vs-harness parity (`tests/darnit/parity/tier1/`) runs on every PR. For each fixture in the corpus, invokes both audit paths in-process (harness uses `MockLLMStep` so no live API), normalizes to a common `AuditResult`, and diffs per-control status. Sole allowed drift: MCP leaves a control PENDING_LLM; harness resolves it via the LLM continuation loop to any non-PENDING_LLM status. Anything else is a hard failure with a human-readable fixed-width Markdown drift table (no ANSI). Full corpus runs in about 35 seconds, well under the 60s budget. Tier 2 -- coding-agent skill parity (`tests/darnit/parity/tier2/`) is manual-dispatch only via `.github/workflows/parity-tier2.yml`. The workflow declares `environment: parity-tier2` so a GitHub Environment with required reviewers gates every dispatch; `ANTHROPIC_API_KEY` is stored at the Environment level (never at repo scope). The runner captures raw MCP tool JSON, invokes the `/darnit-audit` skill via the Claude Agent SDK with an explicit prompt snapshot + turn cap + zero temperature, parses the skill's final assistant message, and diffs. Per-control status disagreement is a hard failure regardless of authority level (skill has no license to reinterpret). Unparseable skill output surfaces as a distinct failure class from disagreement so a maintainer can tell a broken parser from a real drift. Fixture corpus: `all_pass_repo`, `all_fail_repo`, `mixed_repo`, `pending_llm_repo`. Each ships with a `parity.toml` declaring category, expected counts, and a `control_ids` filter (both audit paths run all 66 OpenSSF Baseline controls today because neither auto-applies fixture-level `audit_profiles` from `.baseline.toml`; the filter is test-side and honest about scope). SC-008 corpus-inventory test verifies every category is represented. Governance property (FR-007a + SC-005a) enforced at the GitHub Actions Environment layer AND by a pytest test that iterates every `.github/workflows/*.yml` and asserts `ANTHROPIC_API_KEY` appears only in `parity-tier2.yml`. Pure-Python file iteration; no `grep` subprocess. FR-014 zero-product-code-changes enforced by `test_no_product_changes.py`: runs `git diff --name-only origin/026-harness-with-stage1...HEAD` and fails if any file under `packages/darnit/src/` or `packages/darnit-baseline/src/` is modified. Diagnostic finding surfaced during implementation: MCP tool and harness disagree on `OSPS-LE-03.02` (feature 026's `inferred_from` authority handling has a residual bug). Fixtures scope around it via `control_ids` in `parity.toml`. That divergence is now a discoverable finding for a follow-up feature to fix; the parity test surface makes it CI-visible from now on. 56 new tests (35 Tier 1 + 21 Tier 2). Full workspace sweep: 2589 pass, 15 skipped, 0 failures. `ruff check` clean; `validate_sync.py` PASSED. `claude-agent-sdk` added as a workspace dev-group dep only; product packages `packages/darnit/pyproject.toml` and `packages/darnit-baseline/pyproject.toml` are untouched. Closes darnitdevorg#366. Follow-up issues opened: - darnitdevorg#368: OpenAI SDK + other-provider parity checks (Tier-2-style, different provider) - darnitdevorg#369: scheduled cadence + governance-appropriate key sourcing Depends on PR darnitdevorg#365 (feature 026 + Stage 1 substrate).

#365] (#370) * test(parity): add two-tier audit parity test suite (Fixes #366) Adds a diagnostic test suite verifying the darnit audit's per-control output is consistent across the three consumers users care about: the direct MCP tool call (`audit_openssf_baseline`), the `darnit harness` end-to-end path, and the `/darnit-audit` coding-agent skill's summary. Motivated by the PR #365 review where the skill was observed silently reclassifying WARN as PASS in its Markdown summary while the MCP tool and harness produced identical raw results. Tier 1 -- mechanical MCP-vs-harness parity (`tests/darnit/parity/tier1/`) runs on every PR. For each fixture in the corpus, invokes both audit paths in-process (harness uses `MockLLMStep` so no live API), normalizes to a common `AuditResult`, and diffs per-control status. Sole allowed drift: MCP leaves a control PENDING_LLM; harness resolves it via the LLM continuation loop to any non-PENDING_LLM status. Anything else is a hard failure with a human-readable fixed-width Markdown drift table (no ANSI). Full corpus runs in about 35 seconds, well under the 60s budget. Tier 2 -- coding-agent skill parity (`tests/darnit/parity/tier2/`) is manual-dispatch only via `.github/workflows/parity-tier2.yml`. The workflow declares `environment: parity-tier2` so a GitHub Environment with required reviewers gates every dispatch; `ANTHROPIC_API_KEY` is stored at the Environment level (never at repo scope). The runner captures raw MCP tool JSON, invokes the `/darnit-audit` skill via the Claude Agent SDK with an explicit prompt snapshot + turn cap + zero temperature, parses the skill's final assistant message, and diffs. Per-control status disagreement is a hard failure regardless of authority level (skill has no license to reinterpret). Unparseable skill output surfaces as a distinct failure class from disagreement so a maintainer can tell a broken parser from a real drift. Fixture corpus: `all_pass_repo`, `all_fail_repo`, `mixed_repo`, `pending_llm_repo`. Each ships with a `parity.toml` declaring category, expected counts, and a `control_ids` filter (both audit paths run all 66 OpenSSF Baseline controls today because neither auto-applies fixture-level `audit_profiles` from `.baseline.toml`; the filter is test-side and honest about scope). SC-008 corpus-inventory test verifies every category is represented. Governance property (FR-007a + SC-005a) enforced at the GitHub Actions Environment layer AND by a pytest test that iterates every `.github/workflows/*.yml` and asserts `ANTHROPIC_API_KEY` appears only in `parity-tier2.yml`. Pure-Python file iteration; no `grep` subprocess. FR-014 zero-product-code-changes enforced by `test_no_product_changes.py`: runs `git diff --name-only origin/026-harness-with-stage1...HEAD` and fails if any file under `packages/darnit/src/` or `packages/darnit-baseline/src/` is modified. Diagnostic finding surfaced during implementation: MCP tool and harness disagree on `OSPS-LE-03.02` (feature 026's `inferred_from` authority handling has a residual bug). Fixtures scope around it via `control_ids` in `parity.toml`. That divergence is now a discoverable finding for a follow-up feature to fix; the parity test surface makes it CI-visible from now on. 56 new tests (35 Tier 1 + 21 Tier 2). Full workspace sweep: 2589 pass, 15 skipped, 0 failures. `ruff check` clean; `validate_sync.py` PASSED. `claude-agent-sdk` added as a workspace dev-group dep only; product packages `packages/darnit/pyproject.toml` and `packages/darnit-baseline/pyproject.toml` are untouched. Closes #366. Follow-up issues opened: - #368: OpenAI SDK + other-provider parity checks (Tier-2-style, different provider) - #369: scheduled cadence + governance-appropriate key sourcing Depends on PR #365 (feature 026 + Stage 1 substrate). * ci(parity-tier2): pin actions to SHAs + move context expressions into env: vars Addresses Kusari Inspector findings on PR #370: 1. Shell injection risk (HIGH impact / HIGH likelihood): `${{ github.actor }}`, `${{ github.sha }}`, and `${{ inputs.fixture_glob }}` were interpolated directly into `run:` blocks that had ANTHROPIC_API_KEY set in env. A crafted fixture_glob input containing shell metacharacters could execute arbitrary commands in the runner and exfiltrate the key. Fix: move each context expression into a step-level `env:` variable and reference it as a quoted shell variable inside the run: block. GitHub Actions substitutes ${{ ... }} at YAML-parse time; shell vars are substituted after the shell already parsed its own syntax, so a metacharacter in an input becomes a literal character in the value of the env var (harmless) rather than a shell operator. 2. Supply-chain risk from unpinned actions: actions/checkout@v4, actions/setup-python@v5, astral-sh/setup-uv@v6, and actions/upload-artifact@v4 all used mutable version tags. If any action owner's account were compromised, a silently repointed tag would execute in this workflow with ANTHROPIC_API_KEY in env. Fix: pin each action to a full 40-character commit SHA with a trailing comment identifying the human-readable version. Bumping an action now requires a PR edit -- reviewable, correlatable with any subsequent test-result changes. Pinned versions: actions/checkout 11d5960a326750d5838078e36cf38b85af677262 v4.4.0 actions/setup-python a26af69be951a213d495a4c3e4e4022e16d87065 v5.6.0 astral-sh/setup-uv d0cc045d04ccac9d8b7881df0226f9e82c39688e v6.8.0 actions/upload-artifact ea165f8d65b6e75b540449e92b4886f43607fa02 v4.6.2 Existing workflow-config tests continue to pass unchanged (they assert governance-critical structure -- trigger, environment, permissions, `if: always()` on artifact upload -- none of which this fix touches). * review-fix(pr-370): unblock parity discovery + tighten parser/differ + wire Tier 2 MCP Reviewer pxp928 flagged three blockers, three correctness bugs, an SDK wiring hole, and a packaging concern on #370. This commit addresses all of them. Blockers: - `.baseline.toml` was in `.gitignore`, so none of the four parity fixtures had their config tracked -- CI's Tier 1 collector saw zero fixtures. Added an `!tests/darnit/parity/fixtures/**/.baseline.toml` negation and force-added the four fixture configs. - Parity tests carried no pytest markers, so CI's `-m unit` / `-m integration` split silently deselected the entire suite. Auto-mark every test under `tests/darnit/parity/tier1/` as `integration` via a conftest `pytest_collection_modifyitems` hook. - `test_no_product_changes.py` silently skipped under shallow CI clones. Fail loudly with a clear pointer at `fetch-depth: 0` when `CI=1` and no base ref is reachable; keep the local-dev skip. Correctness bugs: - `comparator.is_allowed_drift` returned True for MCP=PENDING_LLM vs harness=<MISSING>, contradicting T1-3. Missing on either side is now always a hard failure. - `skill_markdown_parser` scanned status literals in _STATUS_LITERALS order, so a phrase like "WARN ... to reach PASS" parsed as PASS. Pick the LEFTMOST status literal instead so the first token in the line wins. - `tier2/diff.py` iterated only over the skill's claims; a skill omitting a FAIL control silently reported "success". Also walk the tool's controls and flag any the skill did not mention. Tier 2 SDK wiring: - `ClaudeAgentOptions` sent only model/max_turns/cwd. The agent had no MCP server, no allowed tools, and no permission mode -- so it couldn't actually call `audit_openssf_baseline` and Tier 2 was measuring "skill does nothing" rather than skill vs tool. Wire `mcp_servers={"darnit": {stdio "darnit serve"}}`, allow only `mcp__darnit__audit_openssf_baseline`, set `permission_mode="acceptEdits"`, and lock `setting_sources=[]` to keep the host's local Claude Code config out of the run. Packaging: - Moved `claude-agent-sdk>=0.1.0` out of `[project.optional-dependencies].dev` (published on `darnit-mcp[dev]`, ~90 MB) into a dedicated `[parity-tier2]` extra. Tier 2 CI now installs `--extra dev --extra parity-tier2` explicitly; a contributor installing `darnit-mcp[dev]` no longer pulls the SDK. Workspace sweep: 2593 pass, 15 skip. * fix(parity): auto-unshallow before failing FR-014 check under CI `_base_ref()` in `test_no_product_source_changes` fails loudly under `CI=1` when no base ref is reachable, but the default `actions/checkout` config leaves the repo shallow so the base ref (`origin/main` or the stack parent) is not initially reachable. The check would false-fail on every CI run until the workflow was updated with `fetch-depth: 0`. Attempt `git fetch --unshallow --tags origin` once before giving up, so the test resolves the base ref on its own if the runner has network access. Preserves the loud-fail semantics on a truly unreachable base (no network + no history).

mlieberman85 mentioned this pull request Aug 6, 2026

Test parity between coding-agent (MCP) audit path and harness audit path #366

Closed

5 tasks

mlieberman85 force-pushed the 026-harness-with-stage1 branch from 8a31801 to cf132cf Compare August 6, 2026 21:20

mlieberman85 force-pushed the 026-harness-with-stage1 branch from cf132cf to 559dd1e Compare August 7, 2026 18:20

mlieberman85 mentioned this pull request Aug 9, 2026

feat(harness): add QuestionResolver Protocol + --interactive CLI flag (stacked on #365) #367

Merged

6 tasks

This was referenced Aug 10, 2026

Parity: OpenAI (and other provider) skill-drift checks alongside the Claude Agent SDK version #368

Closed

test(parity): two-tier audit parity test suite (Fixes #366) [stacked on #365] #370

Merged

pxp928 approved these changes Aug 13, 2026

View reviewed changes

mlieberman85 merged commit edd92c4 into darnitdevorg:main Aug 13, 2026
8 checks passed

Navigation Menu

Uh oh!

feat: RFC-0001 Stage 1 authority + `darnit harness` fleet driver - #365

feat: RFC-0001 Stage 1 authority + `darnit harness` fleet driver#365
mlieberman85 merged 4 commits into
darnitdevorg:mainfrom
mlieberman85:026-harness-with-stage1

mlieberman85 commented Aug 6, 2026 •

edited

Loading

Uh oh!

mlieberman85 commented Aug 9, 2026

Uh oh!

mlieberman85 commented Aug 10, 2026

Uh oh!

pxp928 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

mlieberman85 commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What ships as user-visible

What's substrate (not user-visible)

Constitution notes

Test plan

Follow-up

Uh oh!

mlieberman85 commented Aug 9, 2026

Uh oh!

mlieberman85 commented Aug 10, 2026

Uh oh!

pxp928 left a comment

Choose a reason for hiding this comment

Blockers

1. Plugin handlers all default to suggestive → silent PASS→WARN regression

2. STAGE1-REF-SECURITY-01 can never PASS — and ships at level 1

Should fix before merge

Worth addressing

Stack note

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mlieberman85 commented Aug 6, 2026 •

edited

Loading

1. Plugin handlers all default to `suggestive` → silent PASS→WARN regression

2. `STAGE1-REF-SECURITY-01` can never PASS — and ships at level 1