EvidenceGate is a lightweight quality gate for retrieval-augmented AI systems. It measures whether an answer is supported by retrieved context and returns a deterministic pass/fail decision for release pipelines.
It is dependency-free, runs locally, and uses versioned JSON fixtures—useful when you need a transparent baseline before adding model-based evaluators.
- Claim coverage: whether expected claims are represented in retrieved evidence
- Answer grounding: how much of the answer’s meaningful vocabulary appears in the evidence
- Citation validity: whether cited source IDs exist in the provided evidence pack
- Citation completeness: whether claim-bearing answers cite supporting evidence
- Regression detection: whether a candidate release drops below a baseline by more than a configured tolerance
- Release reports: JSON for automation and Markdown for pull-request comments
git clone https://github.com/flokithecoder/evidence-gate.git
cd evidence-gate
python -m venv .venv
source .venv/bin/activate
pip install -e '.[dev]'
evidence-gate evaluate --input examples/release.json --min-score 0.80Example output:
EvidenceGate: PASS | score=0.950 | threshold=0.800 | cases=2
Write a report for downstream tooling:
evidence-gate evaluate --input examples/release.json --min-score 0.80 --output report.json --markdown report.md{
"cases": [
{
"id": "refund-window",
"answer": "Customers can request a refund within 30 days.",
"sources": [
{"id": "refund-policy", "text": "Refund requests are accepted within 30 days of purchase."}
],
"expected_claims": ["refund within 30 days"],
"citations": ["refund-policy"]
}
]
}EvidenceGate intentionally starts with transparent lexical and citation-structure metrics. They are fast, cheap, deterministic, and easy to inspect in CI. They should complement—not replace—human review and semantic or model-based evaluation for high-stakes applications.
versioned evidence pack → score answer and citations → render reports → compare baseline → pass or fail release
After saving a baseline report from a known-good version, compare a new run:
evidence-gate compare --current candidate.json --baseline baseline.json --max-regression 0.02This exits non-zero when the aggregate score falls by more than the allowed regression tolerance.
pytest -qThe project includes unit tests and a GitHub Actions workflow across Python 3.10 and 3.12.
Use only synthetic, public, or appropriately authorized evaluation data. Do not commit API keys, customer content, employer code, or sensitive prompts.
MIT