Skip to content

feat: add JUnit XML exporter and --fail-under CI quality gate - #41

Closed
brainsparker wants to merge 1 commit into
mainfrom
feature/2026-08-13-junit-ci-gate
Closed

feat: add JUnit XML exporter and --fail-under CI quality gate#41
brainsparker wants to merge 1 commit into
mainfrom
feature/2026-08-13-junit-ci-gate

Conversation

@brainsparker

Copy link
Copy Markdown
Owner

What this does

Adds CI-native quality reporting to PromptLens: a JUnit XML exporter and a --fail-under score gate, so eval runs can block a pipeline the same way unit tests do.

New output format:

output:
  formats: [html, json, junit]

New CLI gate:

promptlens run config.yaml --fail-under 3.5

Behavior:

  • One JUnit test suite per model, one test case per golden-set entry.
  • A test case whose judge score is below the threshold is a failure, a model API error is an error, and an unjudged case is skipped (so judging-disabled runs never report a false pass).
  • Suite properties carry run metadata CI dashboards can surface: model, provider, run id, golden set, average judge score, total cost.
  • If any model's average judge score falls below --fail-under, the run exits with code 2. Exit code 1 stays reserved for run errors, so CI can distinguish quality regressions from infrastructure failures.
  • promptlens export <run_id> --format junit re-exports any saved run.

Why now (market rationale)

  1. OpenAI acquired Promptfoo (announced March 9, 2026) and is folding it into its enterprise agent platform with a security and red-teaming focus (https://openai.com/index/openai-to-acquire-promptfoo/, https://techcrunch.com/2026/03/09/openai-acquires-promptfoo-to-secure-its-ai-agents/). Community coverage flags real anxiety about a formerly vendor-neutral OSS eval tool now owned by a model vendor. That is a genuine opening for lightweight, vendor-neutral eval tools, which is exactly PromptLens's positioning.
  2. JUnit XML output is now table stakes across the category. Promptfoo shipped and documented it as a flagship output format in 2026 after it sat as a long-standing feature request (https://www.promptfoo.dev/docs/configuration/outputs/, JUnit-style XML reports promptfoo/promptfoo#9071). DeepEval is winning the CI/CD lane largely on pytest-native and JUnit-compatible reporting, and newer tools (tribunal, ask-eval, evalci) all treat JUnit plus exit-code gating as baseline.
  3. The gate is what makes evals actionable: baseline-and-block patterns (fail the build when scores drop) are the recurring ask across 2026 eval tooling (see multivon-eval's ci_eval example and llm-eval-suite's regression flow). --fail-under is the minimal version of that, and it works with every existing format, not just junit.

This lane was chosen deliberately to avoid the repo's open-PR congestion: the change is a new exporter module plus small additive wiring in cli.py. It does not touch promptlens/models/config.py or the provider HTTP code, which is where the 30+ open nightly hardening PRs overlap. The formats list in OutputConfig is an unvalidated List[str], so no config model changes were needed.

Runner-up candidates considered today and passed on: a spec-compliance feature for MCP-Profiles around the new MCP list-result caching hints (SEP-2549 ttlMs/cacheScope, https://blog.modelcontextprotocol.io/posts/2026-07-28/), dropped because that repo has pivoted to the you-aware search server where gateway caching semantics no longer fit; and a regression baseline comparison command for PromptLens (promptlens compare --baseline), a strong follow-on that builds naturally on this PR's exit-code convention.

Implementation notes

  • promptlens/exporters/junit_exporter.py: new JUnitXMLExporter following the existing BaseExporter interface, built with xml.etree.ElementTree (proper escaping, no string templating). Configurable fail_under threshold, default 3.0 on the 1-5 judge scale.
  • promptlens/exporters/init.py: registers the new exporter.
  • promptlens/cli.py: junit entry in both the run formats map and the export command, new --fail-under option (click.FloatRange(1.0, 5.0)), and a _check_fail_under helper that reports every failing model with its average score. A model with no judge scores fails the gate explicitly rather than passing silently.
  • tests/test_junit_exporter.py: 18 new tests covering pass/failure/error/skipped mapping, custom thresholds, multi-model suites, XML escaping, suite properties, directory creation, and the gate logic (including the exactly-at-threshold and no-scores cases).
  • README.md: feature list, output config, and a new CI/CD Integration section with a GitHub Actions example.

Test status

  • Full suite: 27 passed (18 new, 9 existing), Python 3.9, pytest.
  • All touched modules compile clean and the CLI help renders correctly.
  • End-to-end verification: exported a saved run via promptlens export --format junit and validated the XML parses, with correct failure counts per suite.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copy link
Copy Markdown
Owner Author

Closing in favor of #43, the same change set rebased onto post-consolidation main (the cli.py conflict with #40 is resolved there). Full suite passes on the rebase: 54/54.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant