EvidenceGate

EvidenceGate is a lightweight quality gate for retrieval-augmented AI systems. It measures whether an answer is supported by retrieved context and returns a deterministic pass/fail decision for release pipelines.

It is dependency-free, runs locally, and uses versioned JSON fixtures—useful when you need a transparent baseline before adding model-based evaluators.

What it checks

• Claim coverage: whether expected claims are represented in retrieved evidence
• Answer grounding: how much of the answer’s meaningful vocabulary appears in the evidence
• Citation validity: whether cited source IDs exist in the provided evidence pack
• Citation completeness: whether claim-bearing answers cite supporting evidence
• Regression detection: whether a candidate release drops below a baseline by more than a configured tolerance
• Release reports: JSON for automation and Markdown for pull-request comments

Quick start

git clone https://github.com/flokithecoder/evidence-gate.git
cd evidence-gate
python -m venv .venv
source .venv/bin/activate
pip install -e '.[dev]'
evidence-gate evaluate --input examples/release.json --min-score 0.80

Example output:

EvidenceGate: PASS | score=0.950 | threshold=0.800 | cases=2

Write a report for downstream tooling:

evidence-gate evaluate --input examples/release.json --min-score 0.80 --output report.json --markdown report.md

Input format

{
  "cases": [
    {
      "id": "refund-window",
      "answer": "Customers can request a refund within 30 days.",
      "sources": [
        {"id": "refund-policy", "text": "Refund requests are accepted within 30 days of purchase."}
      ],
      "expected_claims": ["refund within 30 days"],
      "citations": ["refund-policy"]
    }
  ]
}

Design

EvidenceGate intentionally starts with transparent lexical and citation-structure metrics. They are fast, cheap, deterministic, and easy to inspect in CI. They should complement—not replace—human review and semantic or model-based evaluation for high-stakes applications.

versioned evidence pack → score answer and citations → render reports → compare baseline → pass or fail release

Regression gate

After saving a baseline report from a known-good version, compare a new run:

evidence-gate compare --current candidate.json --baseline baseline.json --max-regression 0.02

This exits non-zero when the aggregate score falls by more than the allowed regression tolerance.

Development

pytest -q

The project includes unit tests and a GitHub Actions workflow across Python 3.10 and 3.12.

Safety

Use only synthetic, public, or appropriately authorized evaluation data. Do not commit API keys, customer content, employer code, or sensitive prompts.

License

MIT