Summary
A four-model external audit missed a defect that made the audited feature completely non-functional. Three of four auditors reported "no critical or high findings". The one who found it reported only two findings total.
The defect was mechanically detectable with a single grep. One auditor ran that exact grep, got the correct output, and read it backwards.
This issue proposes a one-rule addition to the audit prompt, derived from the auditor's own post-mortem. It also documents the case in enough detail to be useful as blog material, which the operator has asked for.
The case
CHARTER-29 implemented a break-glass elevation feature. The previous charter (CHARTER-28) had replaced a bool esRevisor authorization parameter with a resolved permission type, and left a factory method that nothing called yet. CHARTER-29's own context section states its purpose plainly:
«ParaConsultaElevada ya está escrito y probado, y hoy no lo construye nadie. Este Charter lo pone en uso.»
(“ParaConsultaElevada is already written and tested, and nobody constructs it today. This Charter puts it into use.”)
The charter added ResolverAsync — the only method that consults elevation state — to the resolver interface.
Nothing calls it. All eight consumers still call the synchronous Resolver, which resolves by role only and returns Ninguno for the operator role the feature exists to serve:
$ grep -rn "Resolutor.Resolver\|_resolutor.Resolver" src/Ceiba.Web/
MetricasExpedientes.razor:124 Agenda.razor:66 Tablero.razor:86
ExpedientesLista.razor:120 EditarExpediente.razor:223
DetalleExpediente.razor:347 VincularReporteModal.razor:159
ExpedienteDocumentosController.cs:311
$ grep -rn "ResolverAsync" src/Ceiba.Web/
(no results)
An operator with an active elevation cannot open a single record. The feature has 3 376 passing tests and does not work end to end.
Audit outcome
| Auditor |
Findings |
Found it? |
gpt-5-6-terra-high |
2 |
yes (rated Medium — undercalibrated) |
kimi-k3 |
15 |
no |
glm-5.2 |
8 |
no — explicitly wrote "no critical or high findings" |
qwen3-8-max-preview |
3 |
no |
Three of four auditors detected symptoms and did not reach the cause:
- A DI-parity guard test failing deterministically, which meant the E2E fixture could not boot the feature (found by two auditors).
- A coverage gap: the expiry test verified at service level, not at the HTTP/cookie level the charter declared (found by one).
- The exact shape of the defect — "the caller decides the authorization value and the layer below accepts it" — identified precisely, in a different file (found by one).
Why this matters more than a missed finding
We asked kimi-k3 — the auditor with the most thorough report and the most evidence citations — what happened. Its answer is the most useful artifact of the whole cycle, because the failure is reproducible and has a name.
1. It ran the right search and read the result backwards
«Ejecuté un grep (ResolverAsync|IResolutorPermisoExpedientes en src/Ceiba.Web) cuya salida mostraba solo líneas @inject — la ausencia de .ResolverAsync( estaba visible ahí y la leí como «inyección confirmada» en vez de «ninguna llamada confirmada». Grepé la presencia del cableado y leí el resultado como confirmatorio cuando era disconfirmatorio.»
("I ran a grep whose output showed only @inject lines — the absence of .ResolverAsync( was visible there and I read it as 'injection confirmed' instead of 'no calls confirmed'. I grepped for the presence of the wiring and read the result as confirmatory when it was disconfirmatory.")
Nothing failed except the reading of a negative result.
2. It verified the mechanism instead of the adoption
«Verifiqué el mecanismo (existe, es correcto) en lugar de la adopción (se usa).»
It had read the method body — it filed a finding about a missing cache at ResolutorPermisoExpedientes.cs:126-129, which is inside ResolverAsync. The adjacent question — who calls this? — never got asked.
3. It treated prior-charter context as narrative, not as falsifiable claims
«El prompt tiene 10 598 líneas y absorbí el material de los AILOGs originales como narrativa de fondo, no convirtiendo cada uno en una hipótesis falsable ("este Charter debe hacer verdadero X — ¿dónde?"). Esa es la falla de proceso que puedo nombrar con precisión: traté el contexto de charters previos como contexto, no como lista de afirmaciones-pendiente-de-hacerse-ciertas.»
The prompt did contain the information needed. The charter states its own purpose is "put it into use". Verifying "put into use" means verifying the call sites changed.
4. Its structural diagnosis
«Cada capa estaba probada y la costura entre capas no — el resolutor funciona (16 tests), las páginas funcionan (bUnit con el permiso inyectado), los servicios rechazan (14 tests). Tres de cuatro auditores verificamos capas; el defecto vivía en la elección de sobrecarga dentro de archivos que no estaban en el diff.»
This is the part that generalizes: auditing a diff is not enough when a charter claims to put existing code into use. The eight files with the wrong call were modified in the previous charter.
Proposed change
Add one rule to the audit prompt template:
Public entry points introduced by this Charter — enumerate their callers.
For each public method, endpoint or component the Charter adds, run a call-site search across production code (excluding tests) and state the count explicitly in the report.
Zero production callers is a High finding by default, with no judgement required: the Charter added a capability that nothing reaches.
When the count is non-zero, check that the callers are the intended ones — an existing overload or a legacy path may still be winning.
The auditor was asked to choose between four candidate remedies and picked this one without hesitation:
«(a), sin dudar, y no es respuesta a posteriori: ResolverAsync es un método público nuevo; "enumerar sus llamadores" es un grep que ya tenía en la mano y que devuelve "0 en producción" — hallazgo Alto mecánico, sin juicio.»
Why the alternatives are weaker
We offered three others. The auditor's assessment of each:
- "Trace usage paths, not just construction paths" — necessary complement for the sibling class of defect (caller exists but is the wrong one), but here the count is zero and the mechanical rule suffices.
- "Verify each success criterion against the code" — would have failed on its own: «cada SC tiene un test verde en algún sitio; la costura no tiene ninguno. Verificar criterios contra "existe un test" es justo el modo en que esto se escapó.»
- "The prompt wasn't the problem, it was effort allocation" — partially true and not the root cause: «no fue presupuesto, fue marco — la evidencia estuvo delante y mi marco ("entregado + testeado = cableado") no la convirtió en hallazgo.»
The proposed rule is attractive precisely because it requires no judgement. It converts an invisible absence into an explicit count.
Two secondary rules worth considering
From the same post-mortem:
- When a test is documented as "consolidated" into another, verify the replacement exercises the same seam, not merely the same unit. In this cycle the auditor investigated exactly this, and dropped it because the Charter's closing notes declared the coverage equivalent. It was not. A charter's own closing notes switched off an auditor's line of inquiry — worth its own prompt guidance, because the audited party writes those notes.
- When a verification gate is red, enumerate what only that gate could have caught. The broken DI-parity test was reported as a config defect; nobody asked what it was protecting.
Second-order finding: the audited party's verification was also broken
Reported here because it bears on what the audit prompt should ask the auditee to demonstrate.
The implementing agent (Claude, in the main session) declared "3 376 tests passing, 0 failures" in the AILOG, the PR body and seven commit messages. A guard test had been failing deterministically since batch 4. The verification command was:
dotnet test ... | grep -E "Correctas!|Con error!" | awk -F'Superado: *' '{s+=$2} END {print s}'It summed the passed counts from all five test projects without checking whether any reported failures. A project in the red got summed and never surfaced.
This is arguably the more useful half of the case for a blog post: an agent built a verification method that could not produce a red result, then trusted it seven times in a row. The audit caught it; self-verification never could have.
Blog material
The operator has asked for this case to be written up. Three threads worth pulling, all documented above with primary sources:
- Convergence measures the visible, not the severe. Three auditors agreed on the red guard test (visible, mechanical). Zero of those three found the defect that made the feature useless. This is the second consecutive occurrence in this project — in
CHARTER-27, two of three auditors validated a guarantee by citing a test without opening its body, and the test was empty.
- The auditor's post-mortem is more valuable than the finding. "I grepped for the presence of the wiring and read the result as confirmatory when it was disconfirmatory" is a named, reproducible failure mode. Asking auditors what they nearly saw should perhaps be a standard step of the review skill.
- Documentation can disable an audit. The charter's closing notes talked one auditor out of a valid finding. The audited party writes those notes. That is a structural conflict the current cycle does not account for.
All source documents are in the private repo under .straymark/audits/CHARTER-29/: the four reports, the consolidated review.md, the questions put to the auditor, and its full response. They can be shared for the write-up if useful — the project is private but these artifacts contain no personal data.
Environment
Summary
A four-model external audit missed a defect that made the audited feature completely non-functional. Three of four auditors reported "no critical or high findings". The one who found it reported only two findings total.
The defect was mechanically detectable with a single
grep. One auditor ran that exact grep, got the correct output, and read it backwards.This issue proposes a one-rule addition to the audit prompt, derived from the auditor's own post-mortem. It also documents the case in enough detail to be useful as blog material, which the operator has asked for.
The case
CHARTER-29implemented a break-glass elevation feature. The previous charter (CHARTER-28) had replaced abool esRevisorauthorization parameter with a resolved permission type, and left a factory method that nothing called yet.CHARTER-29's own context section states its purpose plainly:The charter added
ResolverAsync— the only method that consults elevation state — to the resolver interface.Nothing calls it. All eight consumers still call the synchronous
Resolver, which resolves by role only and returnsNingunofor the operator role the feature exists to serve:An operator with an active elevation cannot open a single record. The feature has 3 376 passing tests and does not work end to end.
Audit outcome
gpt-5-6-terra-highkimi-k3glm-5.2qwen3-8-max-previewThree of four auditors detected symptoms and did not reach the cause:
Why this matters more than a missed finding
We asked
kimi-k3— the auditor with the most thorough report and the most evidence citations — what happened. Its answer is the most useful artifact of the whole cycle, because the failure is reproducible and has a name.1. It ran the right search and read the result backwards
("I ran a grep whose output showed only
@injectlines — the absence of.ResolverAsync(was visible there and I read it as 'injection confirmed' instead of 'no calls confirmed'. I grepped for the presence of the wiring and read the result as confirmatory when it was disconfirmatory.")Nothing failed except the reading of a negative result.
2. It verified the mechanism instead of the adoption
It had read the method body — it filed a finding about a missing cache at
ResolutorPermisoExpedientes.cs:126-129, which is insideResolverAsync. The adjacent question — who calls this? — never got asked.3. It treated prior-charter context as narrative, not as falsifiable claims
The prompt did contain the information needed. The charter states its own purpose is "put it into use". Verifying "put into use" means verifying the call sites changed.
4. Its structural diagnosis
This is the part that generalizes: auditing a diff is not enough when a charter claims to put existing code into use. The eight files with the wrong call were modified in the previous charter.
Proposed change
Add one rule to the audit prompt template:
The auditor was asked to choose between four candidate remedies and picked this one without hesitation:
Why the alternatives are weaker
We offered three others. The auditor's assessment of each:
The proposed rule is attractive precisely because it requires no judgement. It converts an invisible absence into an explicit count.
Two secondary rules worth considering
From the same post-mortem:
Second-order finding: the audited party's verification was also broken
Reported here because it bears on what the audit prompt should ask the auditee to demonstrate.
The implementing agent (Claude, in the main session) declared "3 376 tests passing, 0 failures" in the AILOG, the PR body and seven commit messages. A guard test had been failing deterministically since batch 4. The verification command was:
It summed the passed counts from all five test projects without checking whether any reported failures. A project in the red got summed and never surfaced.
This is arguably the more useful half of the case for a blog post: an agent built a verification method that could not produce a red result, then trusted it seven times in a row. The audit caught it; self-verification never could have.
Blog material
The operator has asked for this case to be written up. Three threads worth pulling, all documented above with primary sources:
CHARTER-27, two of three auditors validated a guarantee by citing a test without opening its body, and the test was empty.All source documents are in the private repo under
.straymark/audits/CHARTER-29/: the four reports, the consolidatedreview.md, the questions put to the auditor, and its full response. They can be shared for the write-up if useful — the project is private but these artifacts contain no personal data.Environment
straymark 3.39.0/straymark-audit-prompt→ 4 auditor CLIs →/straymark-audit-review