Context
Feature 026 (#365) ships darnit harness, an async driver that calls the same run_sieve_audit code path as the audit_openssf_baseline MCP tool. In principle, the raw audit results from both paths should be identical -- the harness is a fleet-driver wrapper, not a re-implementation.
During feature 026 manual testing against a real repo, we confirmed this at the raw layer:
- Direct MCP tool call (
audit_openssf_baseline): PASS 51, FAIL 5, WARN 7, ERROR 2, PENDING_LLM 1
darnit harness output: PASS 51, FAIL 5, WARN 8, ERROR 2, PENDING_LLM 0
- (Same totals; harness resolved 1 PENDING_LLM to WARN via its own LLM step, which is correct per RFC-0001 Stage 1 authority rule.)
However, the /darnit-audit skill's user-facing summary showed PASS 56, FAIL 4, WARN 3 for the same repo. That is not what the tool returned. The skill is silently re-interpreting the audit results: Claude Code reads files, applies contextual judgment, and demotes some WARNs to PASS in its summary.
The concern
A user reading the skill's summary sees a headline like 56/66 PASS, 85% and reasonably believes that's what the audit found. It isn't; the audit found 51/66. The audit is faithful; the skill layer is not.
Constitution IV ("Never Guess User Values") is the sharper edge of this. If a control is FAIL because a user-judgment key is unconfirmed, and the skill infers the value from repo contents and reports the control as PASS, then the tool's safety property is being undone by the presentation layer.
What needs to happen
Not a fix. A test surface. We need to know when parity holds, when it doesn't, and where the drift lives.
Relationship to #365
#365 does not change any of this -- it exposes the substrate that makes it testable. The harness's report is a machine-readable artifact you can diff against MCP tool JSON, which is the mechanical part of building this parity suite.
Context
Feature 026 (#365) ships
darnit harness, an async driver that calls the samerun_sieve_auditcode path as theaudit_openssf_baselineMCP tool. In principle, the raw audit results from both paths should be identical -- the harness is a fleet-driver wrapper, not a re-implementation.During feature 026 manual testing against a real repo, we confirmed this at the raw layer:
audit_openssf_baseline): PASS 51, FAIL 5, WARN 7, ERROR 2, PENDING_LLM 1darnit harnessoutput: PASS 51, FAIL 5, WARN 8, ERROR 2, PENDING_LLM 0However, the
/darnit-auditskill's user-facing summary showed PASS 56, FAIL 4, WARN 3 for the same repo. That is not what the tool returned. The skill is silently re-interpreting the audit results: Claude Code reads files, applies contextual judgment, and demotes some WARNs to PASS in its summary.The concern
A user reading the skill's summary sees a headline like
56/66 PASS, 85%and reasonably believes that's what the audit found. It isn't; the audit found 51/66. The audit is faithful; the skill layer is not.Constitution IV ("Never Guess User Values") is the sharper edge of this. If a control is FAIL because a user-judgment key is unconfirmed, and the skill infers the value from repo contents and reports the control as PASS, then the tool's safety property is being undone by the presentation layer.
What needs to happen
Not a fix. A test surface. We need to know when parity holds, when it doesn't, and where the drift lives.
darnit harness, and (c) the/darnit-auditskill in a scripted mode. Diff their reported summaries and per-control statuses.dispositiveauthority evidenceaudit says XfromClaude thinks YRelationship to #365
#365 does not change any of this -- it exposes the substrate that makes it testable. The harness's report is a machine-readable artifact you can diff against MCP tool JSON, which is the mechanical part of building this parity suite.