Harden bounded RAG repair diagnostics

README.md

@@ -12,7 +12,7 @@ By default, runtime review data, embeddings, and prompts are processed by embedd
 ## Highlights
 
 - **Visual dashboard:** rating metrics, distribution chart, date and rating filters, and review browser.
-- **Validated citations:** short evidence citations map strictly to stable retrieved source IDs; invented or missing citations fail safely.
+- **Validated citations:** short evidence citations map strictly to stable retrieved source IDs; an invalid or premature-abstention response gets one bounded correction attempt, then invented or missing citations fail safely.
 - **Safe offline state:** analytics still load when Ollama is unavailable, while the app shows exact setup commands instead of crashing.
 - **Adaptive CSV upload:** automatically detect common headers, manually map unfamiliar names, and isolate every dataset in content-addressed Chroma storage.
 - **Reconciled indexing:** content-derived IDs survive reordering; additions, changed records, and deletions are synchronized safely.
@@ -140,30 +140,16 @@ uv run local-ai-agent evaluate --report-dir evaluation/results/my-run
 The versioned case manifest is tied to the dataset SHA-256 and uses immutable
 content-derived source IDs for gold relevance. The report records model tags
 and immutable Ollama digests, dataset and case-set hashes, retrieval limit,
-runtime versions, per-case RAG and BM25 rankings, and aggregate metrics. Each
-RAG observation also retains the first raw model response, any one-shot repair
-response, the initial and final structured validation reasons, and whether a
-repair was attempted. These diagnostics stay in the evaluation artifact; the
-CLI and dashboard continue to expose only validated answers or safe fallback
-messages.
-
-Compare answer models without changing the embedding model or case set:
-
-```bash
-uv run local-ai-agent evaluate \
-  --chat-model llama3.2 \
-  --report-dir /tmp/local-ai-agent-llama3.2
-
-uv run local-ai-agent evaluate \
-  --chat-model <installed-7b-or-8b-instruct-model> \
-  --report-dir /tmp/local-ai-agent-stronger-model
-```
-
-Keep the reports outside the repository so benchmark artifacts do not make the
-worktree dirty. Compare answer success, citation validity, abstention recall,
-latency, and the per-case failure reasons rather than optimizing one aggregate
-score.
-
+runtime versions, per-case RAG and BM25 rankings, and aggregate metrics. Report
+schema v3 additively extends v2 with answer diagnostics; consumers should branch
+on `schema_version` before reading those fields. Each RAG observation records the
+initial and final structured validation reasons and whether a repair was
+attempted. Raw initial and repair model responses are omitted by default; pass
+`--include-raw-responses` only when those diagnostics are needed. The CLI and
+dashboard continue to expose only validated answers or safe fallback messages.
+Raw model responses may echo source review text, so treat opted-in evaluation
+artifacts as potentially sensitive and review or redact them before sharing or
+committing.
 Retrieval quality is reported as recall@k, hit rate@k, and MRR@k for both
 semantic search and the BM25 baseline. Relevance judgments are known-positive,
 not exhaustive. The generated report is evidence for this fixed benchmark

agent.py

@@ -18,18 +18,22 @@
 CITATION_PATTERN = re.compile(r"\[([A-Za-z0-9][A-Za-z0-9_-]*)\]")
 INSUFFICIENT_EVIDENCE_TOKEN = "INSUFFICIENT_EVIDENCE"
 REPAIRABLE_FAILURE_REASONS = frozenset(
-    {"missing_citations", "out_of_range_citation", "unknown_citation"}
+    {
+        "clean_abstention",
+        "missing_citations",
+        "out_of_range_citation",
+        "unknown_citation",
+    }
 )
 
 ANSWER_PROMPT = """You are a review analyst.
 Answer the question using only the supplied reviews. Do not add facts that are not present.
-Always answer every part that at least one supplied review supports. Mixed or incomplete evidence is
-not a reason to abstain; describe the limitation and answer only the supported portion. Every
-factual claim must cite one or more retrieved evidence numbers exactly as shown, for example
-[1]. Never cite an evidence number that is not supplied. Source IDs are validation metadata;
-do not copy them into the answer. Reply exactly INSUFFICIENT_EVIDENCE only when no supplied
-review answers any part of the question. Never use INSUFFICIENT_EVIDENCE as prose or place it
-beside an answer.
+When evidence is mixed or limited, say so clearly. Every factual claim must cite one or more
+retrieved evidence numbers exactly as shown, for example [1]. Never cite an evidence number
+that is not supplied. Source IDs are validation metadata; do not copy them into the answer.
+Answer when at least one supplied review directly addresses any part of the question; partial
+or conflicting evidence does not by itself mean that evidence is insufficient.
+If the supplied reviews do not answer the question, reply exactly INSUFFICIENT_EVIDENCE.
 
 Question:
 {question}
@@ -40,25 +44,21 @@
 Answer:
 """
 
-REPAIR_PROMPT = """You are repairing one rejected review-analysis response.
-Using only the supplied reviews, rewrite it once so every factual claim has one or more valid
-evidence citations such as [1]. Use only evidence numbers that appear below. Preserve supported
-meaning, remove unsupported claims, and answer every supported part even when evidence is mixed
-or incomplete. If no supplied review answers any part, reply exactly INSUFFICIENT_EVIDENCE.
-Never use INSUFFICIENT_EVIDENCE as prose or place it beside an answer.
+REPAIR_PROMPT = """You are a review analyst correcting a response that could not be accepted.
+Re-evaluate the question against the supplied reviews. Answer when at least one supplied review
+directly addresses any part of the question, even when evidence is partial or conflicting. State
+those limits rather than abstaining. Use only supplied facts, and cite every factual claim with
+one or more supplied evidence numbers such as [1]. Never cite an unsupplied number or source ID.
+If no supplied review addresses the question, reply exactly INSUFFICIENT_EVIDENCE. Never include
+that control token alongside an answer.
 
 Question:
 {question}
 
 Supplied review records:
 {context}
 
-Rejected response:
-<rejected_response>
-{rejected_response}
-</rejected_response>
-
-Rewritten answer:
+Corrected answer:
 """
 
 
@@ -253,7 +253,7 @@ def answer_question(
         context=context,
     )
     raw_response = _response_text(answer_model.invoke(prompt))
-    validated, failure_reason = _evaluate_model_response(raw_response, matches)
+    validated, initial_failure_reason = _evaluate_model_response(raw_response, matches)
     if validated is not None:
         validated_answer, sources = validated
         return AnswerResult(
@@ -262,29 +262,19 @@ def answer_question(
             retrieved_source_ids=retrieved_source_ids,
             raw_response=raw_response,
         )
-    if failure_reason == "clean_abstention":
-        return AnswerResult(
-            answer=NO_MATCH_MESSAGE,
-            sources=(),
-            retrieved_source_ids=retrieved_source_ids,
-            abstained=True,
-            raw_response=raw_response,
-            failure_reason=failure_reason,
-        )
-    if failure_reason not in REPAIRABLE_FAILURE_REASONS:
+
+    if initial_failure_reason not in REPAIRABLE_FAILURE_REASONS:
         return AnswerResult(
             answer=CITATION_VALIDATION_MESSAGE,
             sources=(),
             retrieved_source_ids=retrieved_source_ids,
             raw_response=raw_response,
-            failure_reason=failure_reason,
+            failure_reason=initial_failure_reason,
         )
 
-    initial_failure_reason = failure_reason
     repair_prompt = REPAIR_PROMPT.format(
         question=normalized_question,
         context=context,
-        rejected_response=raw_response,
     )
     repair_response = _response_text(answer_model.invoke(repair_prompt))
     repaired, repair_failure_reason = _evaluate_model_response(repair_response, matches)
@@ -299,7 +289,11 @@ def answer_question(
             initial_failure_reason=initial_failure_reason,
             repair_attempted=True,
         )
-    if repair_failure_reason == "clean_abstention":
+
+    if (
+        initial_failure_reason == "clean_abstention"
+        or repair_failure_reason == "clean_abstention"
+    ):
         return AnswerResult(
             answer=NO_MATCH_MESSAGE,
             sources=(),
@@ -311,6 +305,7 @@ def answer_question(
             failure_reason=repair_failure_reason,
             repair_attempted=True,
         )
+
     return AnswerResult(
         answer=CITATION_VALIDATION_MESSAGE,
         sources=(),

evaluation.py

@@ -477,14 +477,22 @@ def run_rag_evaluation(
             limit=limit,
         )
         latency_ms = (perf_counter() - started) * 1000
-        if not result.retrieved_source_ids:
+        if result.failure_reason == "retrieved_source_missing_id":
+            outcome = "retrieved_source_missing_id"
+        elif not result.retrieved_source_ids:
             outcome = "empty_retrieval"
         elif result.abstained:
-            outcome = (
-                "model_abstention_after_repair"
-                if result.repair_attempted
-                else "model_abstention"
-            )
+            if not result.repair_attempted:
+                outcome = "model_abstention"
+            elif (
+                result.initial_failure_reason == "clean_abstention"
+                and result.failure_reason == "clean_abstention"
+            ):
+                outcome = "model_abstention_confirmed_after_repair"
+            elif result.initial_failure_reason == "clean_abstention":
+                outcome = "model_abstention_preserved_after_failed_repair"
+            else:
+                outcome = "model_abstention_after_repair"
         elif not result.sources:
             outcome = (
                 "citation_validation_rejection_after_repair"
@@ -641,6 +649,7 @@ def build_evaluation_report(
     provenance: Mapping[str, Any],
     baseline_observations: tuple[EvaluationObservation, ...] = (),
     generated_at: str | None = None,
+    include_raw_responses: bool = False,
 ) -> dict[str, Any]:
     timestamp = generated_at or datetime.now(UTC).isoformat().replace("+00:00", "Z")
     categories = Counter(case.category for case in cases)
@@ -689,9 +698,16 @@ def build_evaluation_report(
                 observation.repair_attempted for observation in observations
             ),
             "repair_success_count": outcome_counts["answered_after_repair"],
+            "raw_responses_included": include_raw_responses,
         },
         "observations": {
-            "rag": [_observation_as_dict(observation) for observation in observations],
+            "rag": [
+                _observation_as_dict(
+                    observation,
+                    include_raw_responses=include_raw_responses,
+                )
+                for observation in observations
+            ],
             "bm25_baseline": [
                 _observation_as_dict(observation)
                 for observation in baseline_observations
@@ -700,8 +716,12 @@ def build_evaluation_report(
     }
 
 
-def _observation_as_dict(observation: EvaluationObservation) -> dict[str, Any]:
-    return {
+def _observation_as_dict(
+    observation: EvaluationObservation,
+    *,
+    include_raw_responses: bool = False,
+) -> dict[str, Any]:
+    serialized = {
         "case_id": observation.case_id,
         "relevant_source_ids": sorted(observation.relevant_source_ids),
         "retrieved_source_ids": list(observation.retrieved_source_ids),
@@ -710,12 +730,14 @@ def _observation_as_dict(observation: EvaluationObservation) -> dict[str, Any]:
         "abstained": observation.abstained,
         "outcome": observation.outcome,
         "latency_ms": observation.latency_ms,
-        "raw_model_response": observation.raw_model_response,
-        "repair_model_response": observation.repair_model_response,
         "initial_failure_reason": observation.initial_failure_reason,
         "failure_reason": observation.failure_reason,
         "repair_attempted": observation.repair_attempted,
     }
+    if include_raw_responses:
+        serialized["raw_model_response"] = observation.raw_model_response
+        serialized["repair_model_response"] = observation.repair_model_response
+    return serialized
 
 
 def _metric(value: object) -> str:
@@ -733,7 +755,17 @@ def _report_markdown(report: Mapping[str, Any]) -> str:
     configuration = report["configuration"]
     provenance = report["provenance"]
     timing = report["timing"]
-    diagnostics = report["diagnostics"]
+    diagnostics = report.get(
+        "diagnostics",
+        {
+            "outcomes": {},
+            "initial_failure_reasons": {},
+            "final_failure_reasons": {},
+            "repair_attempt_count": 0,
+            "repair_success_count": 0,
+            "raw_responses_included": False,
+        },
+    )
     return "\n".join(
         (
             "# Evaluation report",

main.py

@@ -98,6 +98,11 @@ def build_parser() -> argparse.ArgumentParser:
         type=Path,
         help="write evaluation-report.json and README.md to this directory",
     )
+    evaluate_parser.add_argument(
+        "--include-raw-responses",
+        action="store_true",
+        help="include potentially sensitive raw model responses in the report",
+    )
     _add_runtime_arguments(evaluate_parser)
 
     chat_parser = subparsers.add_parser("chat", help="Start the interactive terminal")
@@ -314,6 +319,7 @@ def run(arguments: argparse.Namespace) -> int:
                     "dependency_versions": _dependency_versions(),
                     **_git_provenance(),
                 },
+                include_raw_responses=arguments.include_raw_responses,
             )
             json_path = arguments.report_dir / "evaluation-report.json"
             markdown_path = arguments.report_dir / "README.md"